Google’s DeepMind app deal with NHS brought into question

DeepMind’s deal with the NHS to develop an app that is able to detect possible kidney damage involved 1.6 million patients’ data has proved controversial. This is due to the fact that those whose data were involved did not know for what purpose they were being used.

DeepMind had launched an independent yearly review in 2016, after the agreement had been arranged, to look into the conduct of its DeepMindHealth department. The review suggested that "‘Good enough’ is not good enough for a company linked so closely to Google, a company that already reaches into every corner of our lives".

It suggested that more could be done to operate to ‘higher standards’, though noted that this could potentially make the company a "lightening rod for public concerns". The latter fact has already partially proved correct, after the Information Commission (ICO) ruled that the deal between Royal Free NHS Foundation Trust and the company was illegal – spurring a flurry of news articles on the deal.

The agreement saw vast amount of patient data change hands, particularly data that would not be relevant to the kidney app it was developing; this included information on whether patients had had an abortion or had tested positive HIV. The ICO expressed concerns that patients were not made aware that there data would be used by a subsidiary of Google to create an app.

Elizabeth Denham, Information Commissioner, released a statement on its inquiry: "Our investigation found a number of shortcomings in the way patient records were shared for this trial. Patients would not have reasonably expected their information to have been used in this way, and the Trust could and should have been far more transparent with patients as to what was happening."

The app that was developed from the collaboration is called Streams, and is used to identify acute kidney injury. DeepMind defended its use of patient data to say that it had created an app with the data that it said was saving nurses up to two per day.

Despite this, DeepMind released a statement acknowledging its own failures: "In our determination to achieve quick impact when this work started in 2015, we underestimated the complexity of the NHS and of the rules around patient data, as well as the potential fears about a well-known tech company working in health. We were almost exclusively focused on building tools that nurses and doctors wanted, and thought of our work as technology for clinicians rather than something that needed to be accountable to and shaped by patients, the public and the NHS as a whole. We got that wrong, and we need to do better."

The company also noted the changes it had made to rectify the errors, while the independent review board it set up said there were still further changes to be made.

DeepMind is known as an artificial intelligence company but, in this particular case, the company discovered the quality of data held by the NHS was not of the appropriate quality to use it in this context. Instead, the Streams app uses an existing NHS algorithm to identify risk.

Ben Hargreaves