Process and evaluation of digital health applications in the context of “Symptom Checkers”

Research area: HTA-methods & steering instruments

Duration: April 2021 – November 2021
Language: German (with English summary)

Publication: HTA Project Report Nr. 141: https://eprints.aihta.at/1348/
 

 

Background: mHealth („mobile Health“) is defined as a component of eHealth and describes the use of mobile devices such as cell phones, patient monitoring devices, personal digital assistants and other wireless devices in a medical context or in the context of (public) health care [1]. For these digital health applications (DiGA, „digitale Gesundheitsanwendungen“) there are high expectations for supporting the healthcare system, although for the majority of available DiGA there is little evidence on the actual benefit. Due to the iterative development and software-updates, challenges when evaluating DiGA arise [2]. The thorough evaluation of DiGA that are reimbursed or subsidized requires a comprehensive methodological approach.

In 2020, the AIHTA published a project report on mHealth assessment frameworks [3]. Based on this analysis, the AIHTA recommended a staged approach to the evaluation of DiGA in Austria. In a first step, the CE mark should be checked and the DiGA should be classified to the appropriate risk class according to the Medical Device Regulation EU MDR 2017/745 [4]. The DiGA should then be compared with the evidence requirements from the NICE Evidence Standards Framework [5]. The available evidence of the DiGA should be evaluated and relevant HTA aspects should be highlighted, taking into account the HTA Module for Mobile Medical Applications [6]and the Digi-HTA Framework [7].

The methodological concept for evaluating DiGA described in [3]has not yet been tested. In this follow-up project, a procedure is to be designed as to how this evaluation process could be embedded in the reimbursement process and what requirements DiGA must fulfill before they are integrated into medical care.To pilot the evaluation framework, a DiGA from the group of “Symptom checkers” will be evaluated based on the available evidence and compared with similar applications (“Symptoma” and possibly another mHealth application from the “Symptom checker” category) [8, 9]. As a result, a detailed process and evaluation conception for practical use should be established.

Project objectives and research questions: Based on an already elaborated evaluation model draft, the aim of this work is to further design the evaluation process for DiGAs in Austria by performing a pilot evaluation. The following research questions are to be answered:

RQ1: How can the potential benefits of DiGAs be categorized prior to evaluation regarding ASVG relevance (= “Allgemeines Sozialversicherungsgesetz”, general social security act): by application and user profile, comparators, expected and desired endpoints?

RQ2: Which technology-specific requirements must DiGAs meet before a comprehensive assessment can be performed?

RQ3: What evidence on (medical or organizational) effects is available for selected DiGA (from the group of “Symptom checkers”) and how does it compare to evidence requirements?

RQ4: Which further steps after adapting the assessment frameworks are to be integrated in a detailed conception?

Methods: The following methods are used to answer the research questions:

RQ1: Hand search of DiGAs included in public (German, Belgian, British) registries and DiGA developed, or under development, in Austria. Categorization and overview according to target group (users), indications, type of comparators, endpoints; supplementary expert consultation on DiGA in Austria.

RQ2: Selection of Austrian DiGAs identified in RQ 1, review of CE-mark and classification by risk class, overview of available information from DiGA on data security, interoperability (in particular electronic health record “ELGA”), and transparency of algorithms (according to HTA-Module for Mobile Medical Applications and Digi-HTA Framework).

RQ3: Systematic review of “Symptom Checkers”. Comparison with evidence and evaluation requirements of the evaluation model developed in Part 1 (Evidence requirements: NICE Evidence Standards Framework; Evaluation requirements: HTA Module for Mobile Medical Applications, Digi-HTA Framework).

RQ4: Revision of the process steps of the evaluation framework, analysis of the HEK (= “Heilmittel-Evaluierungs-Kommission”, cure evaluation commission) process or ÖGK (= “österreichische Gesundheitskasse”, Austrian health insurance) positive lists/ fee schedules (expert consultation, document analysis)

For RQ3, a systematic literature search in several databases (Medline, Embase, Cochrane Library, INAHTA-HTA, CRD, etc.) will be conducted. This will be supplemented by a targeted hand search (Webpages from Public Health-, HTA- and EbM-research institutions).

Inclusion criteria for the systematic literature search:

Population	Health personnel  Symptomatic individuals („potential patients“) Rationale: the population was selected on the basis of a systematic review [9].
Intervention	Symptom Checker (webbased and/or smartphone app): As an aid to guide patient flows (user: health personnel) As an aid for symptomatic individuals (user: insured/ „potential patients“)
Comparators	Conventional care/ Standard of Care (SoC) for guiding patient flow (e.g. phone triage services) Conventional care/ Standard of Care (SoC) (with or without doctor visit) of individuals with symptoms  (Studies with other comparators (or without a comparator), if they address the research question)
Outcomes	Effectiveness Accuracy of diagnosis suggestions (diagnostic sensitivity and specificity)  Generic endpoints (e.g. Quality of life, patient satisfaction) Systemic/ organizational endpoints (e.g. number of doctor/ outpatient visits)  eHealth-specific endpoints (e.g. Usability, engagement)  Safety Harms from false positive/false negative diagnosis suggestions Rationale: Relevant endpoints were selected on the basis of a systematic review [9].
Study design	Randomized controlled trials (RCTs) Non-randomized controlled trials (NRCTs) Observational studies, register studies, RWE Reviews Evaluation reports
Publication period	No restriction
Language	German, English

Time schedule:

Period	Activity
April – June	Scoping, preparation of the project protocol, hand search of digital health application registries, systematic literature search, selection of literature, expert consultation
July	Data extraction, data synthesis, document analysis
August - September	Writing/drafting the report
October - November	Internal and external review, layout, finalisation

References:

[1]        Kramer U., Borges U., Fischer F., Hoffmann W., Pobiruchin M. and Vollmar H. C. [DNVF-Memorandum - Health and Medical Apps]. Gesundheitswesen. 2019;81(10):e154-e170. Epub 2019/10/02. DNVF-Memorandum - Gesundheits- und Medizin-Apps (GuMAs). DOI: 10.1055/s-0038-1667451.

[2]        Tarricone R., Petracca F., Ciani O. and Cucciniello M. Distinguishing features in the assessment of mHealth apps. Expert Review of Pharmacoeconomics & Outcomes Research. 2021:1-6. DOI: 10.1080/14737167.2021.1891883.

[3]        Jeindl R. and Wild C. Framework zur Unterstützung von Refundierungsentscheidungen zu digitalen Gesundheitsanwendungen (mHealth) und dessen (retrospektive) Anwendung an ausgewählten Beispielen. AIHTA Projektbericht Nr.: 134. AIHTA - Austrian Institute for Health Technology Assessment GmbH: 2020 [cited 16.04.2021]. Available from: https://eprints.aihta.at/1279/1/HTA-Projektbericht_Nr.134.pdf.

[4]        Medical Device Coordination Group (MDCG). Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR. 2019. Available from: https://ec.europa.eu/docsroom/documents/37581.

[5]        National Institute for Health and Care Excellence (NICE). Evidence Standards Framework for Digital Health Technologies. 2019 [cited 07.08.2020]. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/our-programmes/evidence-standards-framework/digital-evidence-standards-framework.pdf.

[6]        Moshi M. R., Tooher R. and Merlin T. Development of a health technology assessment module for evaluating mobile medical applications. International Journal of Technology Assessment in Health Care. 2020:1-10.

[7]        Haverinen J., Keränen N., Falkenbach P., Maijala A., Kolehmainen T. and Reponen J. Digi-HTA: Health technology assessment framework for digital healthcare services. Finnish Journal of eHealth and eWelfare. 2019;11(4). DOI: 10.23996/fjhw.82538.

[8]        Symptoma GmbH. Symptoma Digital Health Assistant. 2021 [cited 16.04.2021]. Available from: https://www.symptoma.com/en/medical-device.

[9]        NIHR - National Institute for Health Research. Digital and online symptom checkers and assessment services for urgent care to inform a new digital platform: a systematic review. 2019 [cited 16.04.2021]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK545124/pdf/Bookshelf_NBK545124.pdf.