Conceptual Framework for Evaluating

the Impact and Effectiveness of HIA

Ben Harris-Roxas

Background

Health impact assessment (HIA) is increasingly recognised internationally as a mechanism to ensure that the potential health benefits of policies, programs and projects are maximised and that potential negative health consequences and health risks are minimised (WHO 2008a, WHO 2008b, IFC 2006). HIA has also been on the public health agenda in Australia and New Zealand for more than 15 years, though its use has been limited. This is changing however, as indicated by activity by the Australian Commonwealth Government, the New Zealand Government, every Australian state to develop HIA (CHETRE 2008, Mahoney 2005). In New South Wales provisions for HIA’s use have been incorporated into a number of government strategies including the recent State Health Plan (NSW Health 2007b, NSW Health 2007a).

 

Despite HIA being cited as a mechanism to improve consideration of health and health inequities in planning and implementation (Acheson 1998, WHO 1997) reviews have found that there has been only limited evaluation of the impact of HIA on decision-making and implementation (Taylor & Quigley 2002, Harris-Roxas et al 2004). The considerable challenges in evaluating HIA have been acknowledged in the literature (Quigley & Taylor 2004, Cashmore et al. 2004). Decision-making processes, contexts and the policies that are subjected to HIAs vary markedly, making it difficult to develop a “one size fits all” approach to evaluation (Wismar 2004, Wismar 2007), and it seems likely that efforts to strictly codify HIA would actually impair its effectiveness and usefulness across a range of settings.

About the Framework

This conceptual framework for evaluating the impacts and effectiveness of HIA was developed following a systematic literature review and a review of severn completed HIAs, which included document analysis and 20 semi-structured interviews. It enables evaluators to conceptualise and represent the broad range of factors that influence the impact and effectiveness of HIAs. Publication of this study and the framework is pending.

Conceptual Framework for the Impact and Effectiveness of HIA

Conceptual_Framework_Diagram

Click here for full sized version of the image

 

The framework has two major components - procedural elements and cross-cutting influences, with domains under each. These are outlined below.

Procedural Elements of the Framework

Parameters

These factors "set the scene" for the HIA. They may have been outside the process of the HIA itself but influenced the way the HIA was set up, the way the HIA's recommendation were or were not be acted upon, and the reasons for the HIA to be conducted.

Decision-Making Processes

An evaluation should consider the structure and attributes of external decision-making processes, beyond the HIA itself, including the transparency and openness of these processes. How stable were these decision-making arrangements and how were they governed? What related political events occurred that influenced the HIA?

 

It is important to consider what timeframe was available to conduct the assessment, what deadlines were imposed, and why?

Decision-Makers

Who were the decision-makers responsible for acting on the HIA's recommendations? (Note that these may be distinct from people responsible for making decisions within the HIA process itself) What role did they play in the commissioning and conducting the HIA? What were the personal characteristics of these decision-makers?

 

Additionally, what was the level of stakeholder and/or community interest in the proposal being assesses - support or opposition? To what extent were these "outsiders" involved in decision-making processes? For example did community representatives or stakeholders participate in planning or overseeing the development of the proposal being assessment

Type of HIA

An evaluation should detail the mandate or reason to undertake the HIA - was it mandated by law or regulations, was it being undertaken voluntarily for quality improvement, or was it being undertaken to advocate for changes to the proposal?

 

Was the HIA done as a stand-alone assessment or was it undertaken as a component of an integrated assessment or to contribute to other appraisal processes.

 

At what stage in the development of the proposal was the HIA undertaken? For example, was it undertaken while the proposal was at the conceptual stage or after a detailed proposal had been developed?

Inputs

These are the resources that were made available to conduct the HIA.

Proposal

What were the details of the proposal used to conduct the assessment? How much detail was included and how clear was it?

Capacity and Experience

How much experience and expertise did the assessors and steering group members have in conducting HIAs? What other skills and expertise were available? How much expertise and knowledge did the assessors and steering committee have in the proposal topic/area? Was there technical support available to undertake the HIA, including support for the HIA process or for specific data-gathering exercises?

Resources

What financial and human resources were made available to conduct the HIA, including voluntary and in-kind contributions.

Time

How much t ime was available to conduct the HIA?

Organisational Arrangements

Were there any existing partnerships or inter-organisational arrangements between stakeholder involved in the assessment? For example, was there a memorandum of understanding between the Ministry of Health and the agency developing the proposal?

Process

These are the factors relating to how the HAI was conducted.

Procedural Fidelity

Factors to be considered in evaluating an HIA include:

  • Was a structured, stepwise impact assessment process followed? (see Health Impact Assessment: A Practical Guide for more information)
  • Was the assessment conducted prospectively, i.e. before the proposal was implemented?
  • Did the HIA include predictions about health impacts and identify the confidence limits on these predictions?
  • Did the HIA examine differential impacts or health equity impacts? How?
  • Were cumulative health impacts assessed? How?
  • What guidance or best practice principles were followed?
  • Was the precautionary principle referred to in the HIA - assessment or recommendations?
  • Was the level of the HIA (desktop, rapid, intermediate, comprehensive - see FAQ for more information) appropriate for magnitude and significance of the impacts?
  • Did the s cope of the HIA include pertinent environmental, social and economic determinants of health? How were these determined?
  • Were potentially affected populations profiles in the assessment, including sub-populations and sub-groups?
  • Was baseline information on relevant health indicators and relevant determinants of health reported?
  • Were the impact identification methods rigorous and appropriate? Why/why not?
  • What a assessment methods were used to assess the impacts identified? For example was a matrix used or some other method (see Health Impact Assessment: A Practical Guide)
  • What were the nature and format of HIA recommendations, including recommendations for monitoring and management of health impacts? How were these developed?

Involvement of Decision-Makers and Stakeholders

What was the nature of involvement of decision-makers in HIA process and what was the level and nature of stakeholder involvement, including potentially affected communities.

Transparency

How was the process documented? Additionally how were decisions made during the HIA process and how was this documented, if at all.

Trade-Offs

Were areas of agreement and disagreement between sources of information/evidence identified? Were there areas of agreement and disagreement among the stakeholders involved in the HIA, including potentially affected communities, and how were these documented?

Review

    Were stakeholder given the opportunity to review the draft HIA report before it was finalised? What changes were made because of this? was there any peer or independent review of the draft HIA report, and what changes were made in light of these?

     

    Proximal Impacts

    These are the things that changed most directly as a result of the HIA .

    Informing decisions

    Were potential health impacts considered by decision-makers?

    Changing decisions and implementation

    Observable changed including:

    • Adoption of recommendations
    • Implementation of recommendations
    • Negative impact mitigation measures undertaken
    • Positive impact enhancement measures undertaken
    • Longer-term consistency of implementation and compliance with recommendations
    • Management and follow-up of health impacts

    Changes in Health

    Were there observed changes in health indicators and determinants of health attributable to the proposal, incorporating monitoring activities? For example did community health outreach services associated with a major project improve the health of potentially affected communities? How?

    Predictive Efficacy

    Did health outcomes or changes in the determinants of health fall within predicted ranges? Note: This issue remains controversial, with some suggesting that the conditions for predictions or modeling will never match up with what ends up being implemented. Others suggest that predictive efficacy is not a meaningful criteria for evaluation - would a correct prediction imply the HIA was effective or successful? This is related to the perceived purpose of the HIA.

    Achieving of Goals

    Were the proposal's goals achieved, even if the proposal was changed? For example, if the proposed dam was not built, was a way to generate the required energy found? The role of alternatives in the HIA, decision-making and implementation may play an important  role in evaluating this factor.

     

    In relation to the HIA, were it's stated goals or objectives met? Were these explicit or implicit - see objectives.

     

    Distal Impacts

    These are more indirect changes to come about as a result of undertaking an HIA.

    Understanding

    A frequently cited benefit of undertaking HIAs are improvement in understandings - understandings of health and the determinants of health, health equity , the way other organisations work, the way decisions are made, the way other sectors use and value information, etc.

    Participatory Learning

    HIAs may lead to group learning - changes in understanding about health issues and the ways decisions are made, amongst a group of stakeholder or communities involved in the process. Did this occur? How? Why?

    Influencing other activities

    HIA is often described leading to other activities. Did the HIA lead to involvement in other planning activities and decision-making to include consideration of potential health impacts and the determinants of health? Did the HIA result in any ongoing within-organisation and between-organisation collaboration? Were there any related activities that "spun off"? Did the use of evidence to inform other decision-making processes change? For example, was evidence from the HIA used to inform other decisions and implementation?

    Engagement

    Did stakeholder engagement in this decision and decision-making processes change as a result of the HIA? Did stakeholder engagement in other planning activities or decision-making, change, including the involvement of potentially affected communities or groups? Did other relationships change as a result of the HIA?

    Perception of HIA

    Did the perception of the HIA’s usefulness by decision-makers and other stakeholders, including potentially affected communities, change? What was their perception of the HIA’s credibility? What determined this? How transparent sis they view the assessment and subsequent decision-making.

    Cross-Cutting Influences in the Framework

    Broader Context

    HIAs are conducted in very different contexts across the globe. It is important that any evaluation of HA includes a description of the general governance, political and social context. Relevant recent and historical events should also be described and considered in the evaluation because these may play a significant role in determining whether an HIA is effective or not, independent of factors relating to how the HIA itself was conducted.

     

    Presumptions about the nature of decision-making in general, beyond the HIA being evaluated, should also be considered. For example:

     

    • What is the nature of policy development or planning processes in this context?
    • What are the planning, information-gathering and consultation processes that usually inform decision-making in this context?
      o Who are the stakeholders who generally influence decision-making in this context?

     

    These considerations are particularly relevant in situations where HIAs are being conducted or evaluated by external consultants, who often start out with only a limited understanding of the context in which the HIA was conducted.

    Values, Purpose and Goals

    There is still considerable debate about the purpose of HIA, though as has been noted elsewhere this debate reflects the dynamic nature of the field of HIA rather than being a problem to overcome (Parry & Kemm 2004). The diagram below illustrates some of these varying perspectives:

    Views on the Purpose of Health Impact Assessment

    Purpose of HIA

    Click here for a full sized version of the image

     

    Adapted from Cashmore M. The Role of Science in Environmental Impact Assessment: Process and procedure versus purpose in the development of theory, Environmental Impact Assessment Review, 24:403-426, 2004.

    In light of these differences it is necessary to characterise the purpose of the HIA that is being evaluated.

     

      An important criteria for evaluating any HIA is against its goals, stated or implicit. In order to do this it is necessary to what the HIA was trying to achieve in terms of:

      • Decision-making:
        • using information and evidence to inform decision-making;
        • recommending enhancement and mitigation measures for decision-making;
        • changing decisions and implementation;
        • changing the extent and nature of public participation;
        • generating debate; and 
      • Health:
        • protecting health and minimising exposure to health risks;
        • promoting health and enhancing health outcomes;
        • promoting intersectoral action for health and healthy public policy;
        • reducing health inequalities;
        • changing understandings of health and determinants of health.

       

      In addition to this it is necessary to describe the values and value-judgments underpinning the HIA, as this influences how an HIA is conducted. This includes:

       

      • the personal values of those involved in the HIA and those responsible for acting on recommendations;
      • the disciplinary values of those involved in the HIA and those responsible for acting on recommendations;
      • the social and cultural values of those involved in the HIA and those responsible for acting on recommendations
      • the role of evidence and what constituted evidence used in the HIA;
      • the role of judgment and what constituted judgments in the HIA;
      • how health was defined and what constituted a health impact and a determinants of health; and
      • how equity and differential impacts are defined and what constituted a differential impact.

      Considerations About the Framework

      No HIA will address all elements of this framework, not should they. The framework provides an overview of the broad range of factors that can determine whether an HIA is successful or not - critical considerations in evaluating any HIA.

      So What?

      This framework provides:

       

      • a conceptual basis for evaluating completed HIAs, moving beyond instrumental analysis of the direct effects of HIAs on decision-making and implementation to a more comprehensive understanding of the factors that enhance or reduce the effectiveness of an HIA;
      • a basis for the development of future tools that may be used to plan future HIAs to enhance their impact and effectiveness; and
      • an initial overview of which decision-making parameters, inputs and processes are linked with which proximal and distal impacts, which requires further investigation.

      Further Information

      For more information on evaluating HIA please look at the Evidence HIA Works page.

       

      For a full sized image of the framework click here.

       

      For more information on the framework please contact Ben Harris-Roxas at b.harris-roxas@unsw.edu.au

      References

       

      Acheson D (1998) Independent Inquiry into Inequalities in Health, Stationery Office: London. Access Page

       

      CHETRE (2008) HIA in Australia and New Zealand, Centre for Health Equity Training, Research and Evaluation (CHETRE): Sydney. Access Page

       

      Harris-Roxas B, Simpson S, Harris E (2004) Equity Focused Health Impact Assessment: A literature review, CHETRE on behalf of the Australasian Collaboration for Health Equity Impact Assessment: Sydney. Access PDF

       

      IFC (2006) Performance Standards on Social & Environmental Sustainability, International Finance Corporation, World Bank Group: Washington DC. Access PDF

       

      Mahoney M (2005) Health Impact Assessment in Australia, NSW Public Health Bulletin, 16(7-8): 113-114. Access PDF

       

      Parry J, Kemm J (2004) Future Directions for HIA in Health Impact Assessment (Eds Kemm J, Parry J, et al.), Oxford University Press: Oxford, 411-418. ISBN 0 1985 2629 6

       

      Quigley R, Taylor L (2004) Evaluating Health Impact Assessment, Public Health, 118(8): 544-552 doi:10.1016/j.puhe.2003.10.012.

       

      Taylor L, Quigley R (2002) Health Impact Assessment: A review of reviews, NHS Health Development Agency: London. Access Page

       

      WHO (1997) The Jakarta Declaration on Leading Health Promotion into the 21st Century, World Health Organisation: Geneva, accessed 28 February. Access PDF

       

      WHO (2008a) World Health Report 2008: Primary health care - Now more than ever, World Health Organization: Geneva. ISBN 9 7892 4156 3734. Access PDF

       

      WHO (2008b) Closing the Gap in a Generation: Health equity through action on the social determinants of health, Commission on the Social Determinants of Health, World Health Organization: Geneva. ISBN 9 7892 4156 3703. Access PDF

       

      Wismar M (2004) The Effectiveness of Health Impact Assessment, Eurohealth, 10(3-4): 41-43. Access PDF

      Wismar M, Blau J, Ernst K, Figueras J (Eds.) (2007) The Effectiveness of Health Impact Assessment: Scope and limitations of supporting decision-making in Europe, European Observatory on Health Systems and Policies, World Health Organization: Copenhagen