Another step in the digital direction: NICE releases Evidence Standards Framework for Digital Health Technologies


As with traditional healthcare products, digital health technologies (DHTs) need to be clinically effective and economically viable in order to be provided on the NHS. For DHT developers, proving this can be challenging, due to factors such as difficulties in conducting traditional clinical trials and a dearth of existing evidence[1].

Therefore, it may have been something of a Christmas present for DHT developers when NICE released its Evidence Standards Framework for Digital Health Technologies in December 2018 (the Framework)[2]. The purpose of the Framework is to: provide advice to DHT innovators; assist NHS commissioners; and improve DHT development and commissioning. This article provides a brief overview of the Framework. It does not discuss the regulatory framework applicable to DHTs (such as the medical devices regulatory framework), which applies in tandem.

Scope: What is a DHT?

For the purposes of the Framework, NICE describes DHTs as: “apps, programmes and software used in the health and care system. They may be standalone or combined with other products such as medical devices or diagnostic tests”[3]. By way of example, ‘the Nervecentre platform’ is one of a number of DHTs that is already being used in the public health system. It is currently in operation at Nottingham University Hospitals to assist in the early detection of sepsis. The platform uses algorithms to detect signs of early sepsis and then, if necessary, triggers an intelligent alert system, to ensure clinicians and nurses are made aware immediately, and can escalate and prioritise care. This has the potential to reduce the time taken to diagnose sepsis, which in turn could lead to more targeted and streamlined treatment[4].

Overview of the Framework

The Framework is divided into two sections: Section A – Evidence for Effectiveness Standards and Section B – Evidence for Economic Impact.

Evidence for Effectiveness Standards

Section A includes the following steps to assist in establishing evidence for effectiveness:

Step one: Determine which functional classifications apply to the DHT. To this end, the Framework sets out a number of broad functional categories such as: ‘treatment’, ‘monitoring’ and ‘diagnosis’ and provides guidance as to which of these might apply to the particular DHT.

Step two: Identify the relevant evidence tier. Each functional classification corresponds with an evidence tier, ranging from Tier 1 – to Tier 3b. Active functions such as ‘treatment’ fall within the higher tiers and require the strongest evidence, while passive functions such as simple monitoring (i.e. symptom diaries) require the lowest. Where a DHT is multifunctional, the highest relevant evidence tier applies.

Step three: Apply the substantive standards. These are set out in order of tier in Tables 3 – 6 of the Framework. Users should apply the highest relevant standards, as well as those applicable to each lower tier. In the tables, the standards are split into ‘minimum’ and ‘best practice’. The Framework stipulates that DHTs without specific risks can rely on meeting minimum standards, whereas high risk DHTs should always follow best practice. The standards for evidence of effectiveness are as follows (in summary form)[5]:

For Tier 1 DHTs and above:

  • Credibility with UK health and social care professionals: requires demonstrating a plausible mode of action, viewed as useful by professional experts or groups in the field (through showing contributions from relevant clinical or social care professionals);
  • Relevance to current care pathways in the UK health and social care system: requires evidence to show the appropriateness of the DHT in the UK public health context;
  • Acceptability with users: requires developers to show engagement from relevant user groups in design, development and testing (and, for best practice, that those groups were satisfied);
  • Equalities considerations: requires demonstrating (when feasible) that a DHT could contribute to challenging inequalities (such as by being used by hard-to-reach populations);
  • Accurate and reliable measurements (if relevant): requires developers to provide evidence showing that the data recorded by the DHT is accurate, reproducible and relevant to the target population; and
  • Accurate and reliable transmission of data (if relevant): requires technical data showing that essential information is not changed during transmission and is not biased by the data value expected from the target patient population.

For Tier 2 DHTs and above:

  • Reliable information content: requires developers to show that any health information is valid, accurate, up-to-date, reviewed and comprehensive. Best practice requires evidence of endorsement of this by NICE, NHS England, a professional body, a patient organisation, or other recognised external validation;
  • Ongoing data collection to show usage of the DHT: requires a commitment to ongoing data collection to demonstrate usage, and reporting of that data to: commissioners and (for best practice) the public;
  • Ongoing data collection to show the value of the DHT: as above, but in respect of user satisfaction, and other user outcomes; and
  • Quality and safeguarding: requires developers to show that safeguarding measures are in place around peer-support and communication functions.

For Tier 3a and 3b DHTs:

  • Demonstrating effectiveness: this is the highest standard in the Framework and requires studies showing clinical measures, satisfaction and other patient outcomes. The parameters of the studies vary, up to the best practice for Tier 3b, which requires high quality randomised controlled studies, performed in a setting relevant to the UK health and social care sector[6]; and
  • Use of appropriate behaviour change techniques (if relevant): requires demonstrating that the techniques used are consistent and appropriate and, where necessary, based on published and recognised techniques.
Evidence for Economic Impact

Section B of the Framework consists of the following:

Key economic information: The key information that needs to be provided in order for a DHT’s economic impact to be assessed includes the following: user population size; current and proposed care pathways; and parameters for the economic model. Further guidance as to what these should consist of, and how they should be compiled, is set out in Table 7 of the Framework. The purpose of collecting the information is to enable assessments of the costs and benefits of the intervention(s) compared with existing practice.

Appropriate economic analysis: The type of analysis required should be determined by reference to the financial risk of the DHT being adopted and implemented. This is influenced by the total cost; strength of the effectiveness evidence; life-cycle stage of the DHT; and the quality of the economic information. For financially lower-risk technologies, the Framework states that a cost-consequence analysis should be provided, and stipulates how this should be undertaken. For those with a higher risk, the Framework provides that a cost-utility analysis should be done, which enables a deeper level of assessment as to how differences in cost between options can be justified on the basis of health related outcomes. Again, the Framework provides guidance on how such an analysis should be undertaken.

Economic analysis reporting standards: Finally, the Framework includes reporting standards in relation to specific matters such as: ‘time horizon’, ‘discounting’, and ‘equity analysis’, which should be adhered to when preparing an economic impact analysis.


Overall, the Framework appears to be a comprehensive and intuitive tool. It may enable DHT innovators to prepare for entry into the public health market in a streamlined way. By extension it could serve to reduce costs during the planning and preparation stages and could increase confidence (on the part of developers and investors) that DHTs will qualify for use on the NHS.

In addition to assisting developers, the Framework could potentially prove useful to NHS decision makers and care providers. The Department of Health and Social Care recently noted that confidence on the part of NHS commissioners and carers is essential to the increased use of DHTs in the public health sector[7]. The Framework contributes to this by potentially increasing the quality and clarity of evidence submitted, and by informing decision makers what level of evidence is required for each intervention. The Framework will hopefully be a positive step towards the potential benefits of DHTs for the NHS being realised.

[1] NICE itself recognised this as an important driver for developing the Framework:
[2] It is slightly unclear whether the current version of the Framework is the final one. It states that it has been published for comment and feedback – however, there is no other indication that further versions are due.
[3] Evidence standards framework for digital health technologies: frequently asked questions.
[4] The Nervecentre platform is Case Study 4 in the Department of Health and Social Care’s Policy Paper: The future of healthcare: our vision for digital, data and technology in health and care, 17 October 2018.
[5] These are not substitutes for a full reading of the standards.
[6] Note, ‘setting relevant to the UK health and social care sector’ is not defined.
[7] DHSC Policy Paper: The future of healthcare, 17 October 2018.

Zac Fargher

Related Articles