Impact from HTA-research for the Austrian health care system

Research area: HTA-methods & steering instruments

Project team: Ines Schumacher + cooperation with University of Bielefeld, UMIT, DIMDI
Duration: January 2010 - December 2010

Scientific advice: University of Bielefeld (Ansgar Gerhardus, Evelyn Dorendorf), UMIT (Ruth Schwarzer, Petra Schnell-Inderst, Uwe Siebert), Academy of Sciences (Alexander Bogner?)
Suggested by: LBI-HTA

Impact from HTA-research for the Austrian health care system.
Part 1: Overview - Update. HTA Project report 37a - https://eprints.aihta.at/877
Part 2: Results of the empirical survey. HTA Project report 37b - https://eprints.aihta.at/907

Background
In Austria research in HTA has been conducted since the 1990s. In the beginning, the Institute of Technology Assessment at the Austrian Academy of Sciences took a leadership role. Since the Ludwig Boltzmann Institute for HTA was established in 2006, importance of HTA research has increased considerably. Research in HTA aims at supporting an adequate and efficient use of health care resources in order to sustain a publicly financed and solidary health care system. Research results should provide independent information for decision makers. Eventually, HTA should improve the health care system (structures, processes, outcomes) and should result in improved population health in Austria. For legitimizing further research resources and for prioritizing future HTA research and guaranteeing value of future research, HTA research needs itself to undergo evaluation.

In part 1 of the project a literature review that addresses methods of measuring impact from HTA research has been conducted. The results are available online [1]. The report finishes with a framework for measuring impact in Austria that is based on former work by Gerhardus [2]. It will be used for structuring the empirical work as well as an analytical framework for data analysis. In the framework impact is classified into 7 categories: awareness, acceptance, policy process, policy decision, practice, final outcomes, enlightenment; furthermore, different levels of impact (micro, meso, macro) and different target groups are addressed.

Aims of project
1) To identify adequate methods for measuring the impact of HTA in Austria on the basis of existing literature (part 1)
2) To apply a mix of quantitative and qualitative measures in order to evaluate the HTA-Impact in Austria (part 2)

Research questions
For part 1
1) How has the impact from HTA been defined in the literature, what type of indicators and what type of instruments have been used to measure the impact?
2) Which methods are appropriate for measuring the impact from HTA in Austria?

For part 2
1) What is the impact of the HTA-research that has been conducted at the LBI-HTA and at the academy of sciences (ITA)? At which level and in which target groups can impact (not) be identified?
2) Which factors have influenced the effects?

Methods
For part 1
1) Summary of the existing methods for measuring HTA impact in terms of
a. Definition of HTA Impact + Indicators
b. Evaluation methods
2) Expert workshop (LBI HTA, UMIT, DIMDI, University of Bielefeld): defining an appropriate evaluation design for an empirical analysis in Austria
3) Defining a pool of evaluation tools that will be applied for the empirical analysis

For part 2

Rationale
For evaluating the multi-dimensional types of impact that have been presented in the literature, various methods have been identified: questionnaires, interviews, analysis of health care system data including economic data (cost data, price changes…) document analysis (legal papers, downloads of reports, guidelines, protocols etc.) and discourse analysis.

The methodical challenge for the empirical evaluation is how to gain an overall picture of the effects resulting from our research activities. This is in contrast to evaluate a single project and its impact. Hence, we will apply the available methods cross-sectional in order to address the question what pattern of impact can be identified. For example, we will analyse the downloads from our website according to frequencies of specific topics, types of reports etc. Similarly, we will analyse the media discourse for a defined period rather than analyse a single project and its impact on the media discourse.
The empirical approach reflects a qualitative approach which is explorative in nature and generates hypotheses rather than tests them. Hence the aim is to draw a pattern of HTA research impact which will help us to grasp the phenomenon ‘impact’ and the overall process where HTA-research is embedded in. This is in contrast to identifying causal relationships between HTA-research and its impact.

Operationalisation
A) A mix of qualitative and quantitative methods will be applied that should guarantee to identify the various types of impact from the framework in part 1.

Download-analysis: quantity and pattern of downloaded reportsAnalysis of scientific activities: Quantity and pattern of publications, conference papers/posters, invitations to discussions and presentations etc.Discourse analysis: Analysis of media discourse in a defined period and in defined mediaSemi-structured interviews of key-actors to identify how research products (e.g. decision support documents, full HTA-reports etc.) have been used in different institutions (hospitals, Ministry of health, social security institutions etc.)  and what their effect was according to the interviewees’ experienceEconomic analysis: analysis of cost/expenditure data (before-after), quantities of technologies used (before-after), price developments, efficiency gains, re-distribution etc.)Document analysis: supplementary analysis of laws, protocols, letters…

B) The empirical data gained will be analysed against the backdrop of the defined framework on a meta-level

Which types of impact in the framework are more/less observable? Can specific patterns in terms of project characteristics be identified (e.g. topics, type of reports, commissioner)In which target groups and at which levels can an impact be observed or can not be observed? Can specific patterns in terms of project characteristics be identified?

References
[1] Schumacher I, Zechmeister I. Auswirkungen der HTA-Forschung in Österreich auf das Gesundheitswesen. 1.Teil Methodenübersicht – Update. Wien: Ludwig Boltzmann Institut für Health Technology Assessment; 2010.
[2] Gerhardus A. Konzepte und Methoden zur Erhebung des Einflusses von HTA-Berichten auf das Gesundheitswesen. Hannover: Medizinische Hochschule Hannover; 2005.