Artificial Intelligence in Health Care: Evaluation of the Clinical and Organizational Impacts of selected AI Applications in Hospitals

Duration: Mid- April – Mid- November 2025 (6 PM)
Language: English (with German summary)

Background:
Artificial intelligence (AI) is a field of computer science that aims to imitate human cognitive abilities. It is becoming increasingly important and is expected to have a significant impact on various areas of society, such as education, research, and medicine [1]. AI is already being used in various areas of healthcare [2]. In the majority of healthcare applications, AI can be conceptualized as a digital health technology (DHT), functioning either as a standalone medical device (software as a medical device) or as software integrated within a medical device (software in a medical device) [3]. As healthcare continues to digitalise, AI-enabled DHTs are expected to be integrated across various medical applications, especially in the hospital sector [4].

The Austrian Institute for HTA (AIHTA) published a report [4]on the methodological approaches for evaluating the benefits and risks of AI-enabled DHTs in hospital procurement decisions and provided an overview of assessments conducted by various international HTA organisations on such technologies. AI-enabled DHTs were used across a wide range of medical specialties. Most assessments (90%) focused on radiology and internal medicine, with diagnostics as the main application. Across the analysed 27 assessments in diagnostics, 134 individual AI-enabled DHTs were identified. In Austria, a 2022 report by the Austrian Health Institute (Gesundheit Österreich GmbH/ GÖG) identified 43 AI-enabled DHTs deployed either as pilot projects or already in routine hospital operation [5]. These were grouped into three main categories: risk prediction, treatment improvement and diagnostics. In this report most AI projects fell into the diagnostics category (54%), followed by treatment improvement (27%), and risk prediction (18%). Hence, the focus at both the international and national level lies primarily on the diagnostic use of AI.

Regarding the methodological aspect, the AIHTA report recommended to use existing frameworks for DHTs, supplemented with AI-specific components [4]. The ASSESS-DHT project, a European initiative, was highlighted as a structured starting point for such evaluations [6].

Based on the GÖG report [5]and the AIHTA report [4], AI-enabled DHTs in diagnostic imaging and in documentation/administration support were selected by the decision-makers for further assessment. Administrative AI tools, such as automatic notetaking, have already been implemented in several countries and show potential for improving efficiency by freeing up clinicians’ time. Meanwhile, diagnostic imaging AI-enabled DHTs continue to gain ground as technologies that have the potential to enhance diagnostic accuracy. Despite their considerable potential, these technologies also pose notable risks—such as documentation errors or hallucinations, and the potential for overdiagnosis and overtreatment in imaging diagnostics—that must be carefully weighed through a thorough benefit-risk assessment prior to clinical implementation [7].

While AI-enabled DHTs hold great promise for improving efficiency, accuracy, and accessibility, their use also raises critical questions. The adoption of AI is expected to affect many aspects of care, including resource allocation, staffing, patient outcomes, and the organization of health systems. Ethical and legal aspects such as data privacy, equity of access, and legal responsibility in AI-supported decisions represent additional considerations that may apply to certain use cases [7].

Project goals:
The aim of this project is:

1.     to provide an overview of AI-enabled DHTs in the field of documentation support and diagnostic imaging and prioritise those currently in use or considered most relevant in Austrian hospitals, and

2.     to evaluate the clinical and organisational impacts, as well as the types of resources to be considered, of two AI-enabled DHTs selected from this overview by Austrian healthcare experts.

Non-goals:
These are not within the scope of this project:

• To systematically evaluate AI-enabled DHTs other than the selected ones,
• To conduct a systematic search or comprehensive market analysis of AI systems (DHTs, products) in diagnostic imaging or documentation support,
• To systematically analyse ethical and legal aspects and cost impacts in monetary terms.

Research questions:
The following research questions (RQ) will be answered in the course of the report:

RQ1: Which AI-enabled DHTs in the fields of documentation support and diagnostic imaging are considered most relevant in Austrian hospitals by Austrian healthcare experts?

RQ2: What is the clinical and organisational impacts and what types of resources are needed for the implementation of selected AI-enabled DHTs in documentation support and diagnostic imaging? Specifically, sub-questions for the selected applications:

RQ2 A: Documentation support:

• How do the selected AI-powered documentation support systems affect the time healthcare providers spend on administrative tasks in hospitals?
• What are the potential benefits and challenges of AI documentation systems from the perspective of healthcare providers in terms of usability, accuracy, satisfaction, and possible disadvantages?
• What types of resources are needed related to acquisition and setting up the new technology?
• How does the technology modify the need for other technologies and use of resources?

RQ2 B: Diagnostic imaging:

• How do the selected AI applications in diagnostic imaging affect the accuracy and efficiency of diagnoses in hospitals, including any potential risks, limitations, or unintended consequences?
• What are the organizational impacts of integrating AI into diagnostic imaging practices in terms of workflow, staff training, and resource allocation?
• Which types of resources are affected by processes related to acquisition and setting up the new technology?
• How does the technology modify the need for other technologies and use of resources?

Methods:

RQ1:

• An overview of AI systems for diagnostic imaging and documentation support will be compiled using the AIHTA and GÖG reports as primary sources.
• A structured expert survey will be conducted with selected Austrian stakeholders (i.e., experts from the GÖG report, members from an internal expert pool, and selected representatives from Austrian hospital operators). Experts will be asked to prioritise AI applications of the long list based on criteria such as clinical relevance (novelty of the technology, addressing clinical need, potential to improve patient outcomes or clinical workflows, and availability of evidence), resource implications (frequency of use, costs, expected impact on healthcare resource use), and feasibility of implementation (including potential barriers such as organisational resistance, infrastructure limitations, data security concerns) and potential risks or unintended consequences (e.g. diagnostic errors, increased workload, ethical concerns).

Survey design (TBD): point allocation system (i.e. we provide each respondent with a fixed number of points (e.g., 100 points) and ask them to distribute these points among the list items based on their perceived relevance or importance. Items that are considered most important would receive more points.) OR Top N selection (Instead of ranking all items, ask respondents to select their top 3, 5, or 10 items from the list).

RQ2:

• Clinical and organisational evaluation, including the impact on types of resources of the selected AI-enabled DHTs using the EUnetHTA Core Model [8]as an established European evaluation framework. The AIHTA recommendations [4]and the new European HTA methodology [6](currently under development by ASSESS-DHT) will be applied as complementary resources. These assessments will serve as a pilot implementation of the ASSESS-DHT methodology.

The final protocols for RQ2 (A+B) will be developed once the prioritisation is complete, and these will be made public in June via OSF.

All steps (literature screening and selection, data extraction, and quality control) will be conducted by two researchers. Results will undergo an internal review by an AIHTA reviewer and an external peer review by at least one subject matter expert.

Timetable and milestones:

References:
[1] Al Kuwaiti A. N. K., Al-Reedy A., Al-Shehri S., Al-Muhanna A., Subbarayalu A. V., et al. A Review of the Role of Artificial Intelligence in Healthcare. J Pers Med. 2023;13(6):951.
[2] Bures D., Hosters B., Reibel T., Jovy-Klein F., Schramm J., Brendt-Müller J., et al. Die transformative Wirkung von künstlicher Intelligenz im Krankenhaus. Die Innere Medizin. 2023;64(11):1025-1032. DOI: 10.1007/s00108-023-01597-9.
[3] IMDRF SaMD Working Group. Software as a Medical Device (SaMD): Key Definitions. 2013 [cited 25.04.2025]. Available from: https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-131209-samd-key-definitions-140901.pdf.
[4]  Riegelnegg M., Giess D. and Goetz G. Artificial Intelligence in Health Care with a Focus on Hospitals: Methodological Considerations for Health Technology Assessment. A scoping review Vienna: 2024. Available from: https://eprints.aihta.at/1546/1/HTA-Projektbericht_Nr.164.pdf.
[5] Degelsegger-Márquez A., Dick D. and Trunner K. Telemedizin und Künstliche Intelligenz im intramuralen Bereich Österreichs. Ereignisbericht. Wien: 2022 [cited 02.04.2025]. Available from: https://jasmin.goeg.at/id/eprint/2443/1/Telemedizin_und_KI_in_Krankenanstalten_bf.pdf.
[6] ASSESS DHT. Development & harmonisation of methodologies for assessing digital health technologies in Europe. 2025. Available from: https://assess-dht.eu/.
[7] Canadian Journal of Health Technologies. 2025 Watch List: Artificial Intelligence in Health Care. 2025 [cited 02.04.2025]. Available from: https://www.cda-amc.ca/sites/default/files/Tech%20Trends/2025/ER0015%3D2025_Watch_List.pdf.
[8] EUnetHTA. HTA Core Model Version 3.0. 2016.