An AI project that has been using GP data to train an artificial intelligence (AI) model has been paused by NHS England, following concerns from GPs.
The BMA and the RCGP said they were not aware that GP data, collected for Covid-19 research, was being used by NHS England to train an AI model, Foresight.
According to NHSE, the Foresight project represents ‘a ground-breaking AI initiative’ to ‘transform predictive healthcare in the UK’.
It is an AI model ‘specifically designed for healthcare’ that has been trained in the NHS England Secure Data Environment (SDE), a secure data and research analysis platform, on de-identified NHS data from approximately 57 million people in England.
Researchers working on the model said that Foresight is being trained on ‘routinely collected’, de-identified NHS data, ‘like hospital admissions and rates of Covid vaccination’, to predict potential health outcomes for patient groups across England.
NHSE added that ‘like ChatGPT’ the model learns to predict what happens next ‘based on previous events’, working like an auto-complete function for medical timelines, and that predictions ‘are validated against real-world data’.
But GP leaders said that it was ‘unclear’ whether the ‘correct processes’ were followed to ensure that data was shared ‘in line with patients’ expectations’ and governance processes.
The BMA and RCGP joint IT committee asked NHS England to refer itself to the Information Commissioner over this issue ‘so the full circumstances can be understood’. The committee also demanded for NHS England to pause ongoing processing of data ‘as a precaution’.
NHS England told our sister title Pulse that it has agreed to pause the project while a review is conducted.
BMA GP committee England deputy chair Dr David Wrigley said: ‘For GPs, our focus is always on maintaining our patients’ trust in how their confidential data is handled.
‘We were not aware that GP data, collected for Covid-19 research, was being used to train an AI model, Foresight.
‘As such, we are unclear as to whether the correct processes were followed to ensure that data was shared in line with patients’ expectations and established governance processes.
‘We have raised our concerns with NHS England through the joint GP IT committee and appreciate their verbal commitment to improve on these processes going forward.’
An NHS England spokesperson said: ‘Maintaining patient privacy is central to this project and we are grateful to the Joint GP IT Committee for raising its concerns and meeting with us to discuss the strict governance and controls in place to ensure patients’ data remains secure.
‘The committee has asked us to confirm that GDPR principles are being upheld at all times and we have agreed to pause the project while our Data Protection Officer now conducts a review and recommends whether any further action is needed.’
Concerns around AI translation apps
It comes as NHS England has also raised concerns over the safety of AI translation apps and their use within healthcare.
In a new framework, published at the end of May, NHS England said there were ‘concerns about the appropriate use of AI translation apps’ that are currently widely used across the NHS to communicate with patients who have limited English.
It said: ‘While translation apps provide a convenient, familiar and timely means of translation, they can also carry risks, particularly regarding accuracy and the potential impact on patient safety.’
It added that if translation is inaccurate or not provided in its entirety, then a patient ‘cannot truly understand’ their choices or provide their informed consent to a treatment plan.
‘We must ensure that patient confidentiality and safety is a priority when using translation and interpreting services – and address the risks of using apps and informal interpreters as methods of communication with people with limited English proficiency,’ it said.
The framework, called Improvement framework: community language translation and interpreting services, recommended several actions for ICBs and primary care settings, including to ensure ICBs work with PCNs to apply quality improvement methods.
It suggested appointing a PCN lead to champion interpreting services, and to establish feedback systems for patients and staff to identify issues. As well as calling on PCNs to collaborate with ICBs when procuring new interpreting services and to establish clear escalation pathways within services to address incidents of discrimination.
Recommendations from the framework
Actions for ICBs
Leadership, quality and professional standards
- Ensure director-level leadership and accountability for the commissioning and contracting of translation and interpreting services for all services provided across the ICB’s footprint.
- Involve patients and communities in the development and improvement of local interpreting services through co-production, ensuring diverse voices are included to reflect local needs and address potential gaps in service provision.
- Work with PCNs to apply quality and service improvement methods to develop and strengthen services, using feedback mechanisms for patients and staff to help drive meaningful improvement.
- When procuring for a new service provider, ensure qualification and training standards are defined and interpreters registered (for example, with professional bodies such as the National Register of Public Service Interpreters). Build quality metrics that can be regularly monitored into contracts.
- Ensure any procurement of new interpreting services for primary care takes full account of local population needs and drives quality of service provision, not just cost factors.
Access and barriers to services
- Undertake a population-level needs assessment for community languages at system or place level, working with local community organisations and public health.
- Work with PCNs to review data on use of interpreting services, with a focus on improving access.
- Help improve awareness among practices and PCNs of local patient need for interpreting services and the procedures to access services.
Equity, cultural sensitivity and rights
- Involve patients and communities in the co-production and improvement of interpreting services in primary care, making sure they are culturally sensitive and inclusive.
Digital opportunities and challenges
- Capture and analyse patient data at the local level to identify trends and patterns in the use of interpreting services. Use these insights to optimise service delivery and meet demand.
Actions for primary care
Leadership, quality and professional standards
- Appoint a local or primary care network (PCN) lead to champion interpreting services, ensuring accountability, adherence to standards, and continuous quality improvement at the local level.
- Establish feedback systems for patients and staff to identify issues and provide ongoing input into the improvement of translation and interpreting services.
Access and barriers to services
- At practice or PCN level, collaborate with the ICB on any procurement of new interpreting services to ensure they respond to local population needs and ensure quality provision.
- Improve staff awareness of the right of patients to have interpreting support (including among receptionists and care co-ordinators).
- Use the primary care electronic record to record language need at the earliest opportunity and include language need in referrals to other services. Maximise recording of language need across the practice population in the electronic patient record (EPR).
- Support new and existing practice staff to understand and improve the operational processes for accessing interpreting services.
Equity, cultural sensitivity and rights
- Establish clear escalation pathways within the services to address and resolve incidents of discrimination, ensuring accountability at a local level, where language rights are incorporated.
- Involve patients and communities in coproducing the development and improvement of interpreting services and other equity-focused measures, ensuring cultural sensitivity and inclusivity.
Digital opportunities and challenges
- Record language need in the primary care electronic record as early as possible, include it in referrals, and maximise recording across the practice population in the EPR.
Safety, confidentiality and consent
- Develop or adopt translated consent and confidentiality forms at the local level to ensure patients with limited English proficiency fully understand and can participate in the consent process. Local consent forms should adhere to national standards.
- Confidentiality – interpreters should complete confidentiality forms to support patient trust and confidence.
Source: NHS England
A version of this story was first published on our sister title Pulse.
