Artificial intelligence (AI) is increasingly being used to improve the speed and quality of diagnosis and enable more efficient use of resources in healthcare. As recently as last week, NHS England revealed the results of a study showing the positive impact of AI-driven tools on cardiovascular and cancer diagnoses. But with the pace of change so rapid, and healthcare needs so high, understanding the benefits and risks of different technologies and the wider factors that influence adoption is crucial.
The NHS AI Lab was set up in 2019, with government investment to advance AI-driven projects and position the UK at the forefront of health innovation. It explored the role of AI through multiple strands of work including infrastructure, communities of practice, AI ethics and regulation, as well as assessing the impact of specific projects and their potential for adoption at scale. This pioneering project by the Department of Health and Social Care and NHS England has been independently evaluated by the University of Edinburgh and NHS Arden & GEM CSU, to share insights about how we create a more conducive environment for future AI innovations.
Technology successes and challenges
The NHS AI Lab’s work has contributed to advancing AI development and scaling in health and care. Some technologies funded by the programme showed significant benefits, with promising and reliable estimates of returns on investment identified, particularly in relation to research investments in relatively mature technologies and where technological system implementation was accompanied by process and pathway changes.
In one case, a diagnostic tool providing a set of decision support tools that aided frontline clinicians to make time critical treatment decisions demonstrated an estimated cost saving of over £44 million across a cohort of 150,000 patients, far exceeding the £1.25 million cost of the project. Some projects also helped improve NHS clinical processes in line with the 2019 Long Term Plan, in one case increasing the rate of mechanical thrombectomy for stroke patients to 10%.
We found that many of the most successful projects involved clinicians with an intimate knowledge of the pathway the AI device was deployed into, and tackled issues in the existing pathway without radical redesign. This led to beneficial change that could be quantified and evaluated within short- to medium-term timeframes. However, other projects were unable to demonstrate full benefits within the five-year lifecycle of the programme. Underlying reasons included an underestimation of the deployment challenges associated with technology implementation and adoption.
Our ongoing analysis suggests that technologies that were targeted towards clinical priorities were more readily adopted. The biggest benefits arose where tools were geared towards service transformation rather than task automation.
Identifying and learning lessons
The primary benefit of the experimental work done by the AI Lab is learning – identifying transferable lessons for the development, implementation and adoption of AI technologies. Many of these lessons apply not just to AI but to wider digitalisation and transformation projects across healthcare. They reflect tensions that are not easily resolved.
- Transforming care: many projects in the AI Lab focused on automating and speeding up existing processes. However, those that focused on transforming ways of working or care pathways had greater potential for higher rewards. Although project timelines were often not sufficiently long to evidence these.
- Matching solutions to local needs: understanding the needs of those working in frontline care delivery can help prioritise the most impactful AI tools and maximise their potential. Technology needs to be informed by people, including patients, based on the realities of how care is delivered and received, so that AI tools solve real world problems.
- Balancing national direction with local choice: the AI Lab brought together what would otherwise have been disparate projects, enabling the team to develop a deep understanding of regulatory, ethical and stakeholder engagement needs, as well as deployment challenges. Co-ordinated, national support is needed to create the right infrastructure, governance and skills to manage AI and develop scalable technologies and deployment strategies. This includes simplifying and rationalising procurement to improve access to approved AI technologies which will support shared learning and minimise unnecessary duplication. However, offering choice and enabling service users to be involved in local decision-making is important in facilitating adoption and meeting local needs.
- Long term vision and leadership: the programme took place during a particularly turbulent time with the COVID-19 pandemic, political climate and funding pressures all impacting progress. Establishing a strategic framework and long-term vision for AI research and deployment, rooting projects firmly in service user and health system needs, as well as building in resilience to adapt to unexpected pressures, can all help to deliver consistent progress.
The evaluation challenge
How and when we evaluate AI tools directly impacts, adoption, funding and scalability. One of the main barriers to effective evaluation is timing. For NHS organisations to fully understand the impact of any intervention, we need to establish a baseline. In many cases, baselines were not available for AI Lab projects, limiting the potential to draw firm conclusions about impact and return on investment. This evidence gap makes it harder to secure funding to scale projects.
We also need to develop more agile methods of evaluation to keep pace with technology change. Our report highlights the significant value participants gained from sharing lessons from failures as well as successes within the AI Lab. Building in formative evaluation throughout a project enables teams to spot bottlenecks, identify issues quickly and adapt or even stop a project early if there are strong signs that it won’t succeed.
Most importantly, the NHS and wider healthcare system needs to actively learn from the NHS AI Lab evaluation. The greatest risk from this – and indeed any evaluation – is that we fail to act and build on the lessons learned.
AI is advancing quickly. The AI Lab has created a valuable resource of knowledge and experience which must be fully utilised to guide future strategy, enable more informed decisions, and increase safe and successful use of AI-driven technology. This will both support resolution of local operational challenges faced by frontline staff and, with strong strategic leadership, enhance patient care.
By Kathrin Cresswell, Professor of digital innovations in health and care, University of Edinburgh, and Rose Taylor, executive director of health and care transformation, NHS Arden & GEM CSU
