Covid Certification Service (CCS)

This document provides information on the decisions made around the procurement, development, and deployment of a multi-channel COVID Certification Service (CCS) that would, facilitate international travel (by virtue of vaccination or COVID testing, meeting EU and WHO requirements) and support domestic use in Northern Ireland (though confirmation of meeting selected conditions).

Background

1. Digital Health and Care Northern Ireland (DHCNI)

On taking up post in June 2019, the new Chief Digital Information Officer in the Department of Health, Dan West, reviewed the structures for delivery of digital services across the Department of Health and the HSC in Northern Ireland. This resulted in the formation of the Digital Health and Care Northern Ireland (DHCNI) partnership. It is led by the Department of Health and brought together the Health and Social Care Board’s eHealth and Care Directorate and the Public Health Agency’s Centre for Connected Health and Social Care.

The DHCNI partnership sets the vision and strategy for Digital Transformation in Health and Social Care, directs, and oversees the central budget for Digital investments and works in partnership with, and commissions projects and services from the Business Services Organisation which is the HSC’s shared services provider. DHCNI also aims to bring together the disparate Digital functions across the six Trusts: collaborating to develop a region wide strategy; facilitating shared leadership and decision making; setting policies and standards for the Trust teams to follow; and providing services and support.

2. Digital Co-ordination Cell

In April 2020, as part of the wider Departmental and HSC efforts to address the impact of the pandemic, the Department’s Chief Digital Information Officer (CDIO) convened a “Digital Co-ordination Cell” (DCC) to oversee the Digital response to COVID-19 in Northern Ireland. The development of the COVID Certification Service was one of the products of the Cell.

2.1. Purpose

The purpose of the DCC was to co-ordinate the digital support and enablement of the DoH & HSC response to prepare for the additional burden that COVID 19 was anticipated would place upon the system. Many of these were related to public health and operational considerations, however there was also a sub-set of the preparations that related to issues such as data collation, storage, and access (including governance) and use of appropriate technology. There were also a number of questions that required system-wide resolution and / or decision making as well as funding approvals at the overarching leadership level.

2.2. Objectives

• Understand, articulate, and where required, coordinate the Digital activities underway across the HSC.
• Identify duplication in the activities being undertaken or requested, bringing together the disparate parts of the system that are attempting to solve the same problem.
• Clarify expectations and define DHCNI strategy where required.
• Support DHCNI teams in provision of resources and authority; through coordination of funding and approvals and directing the unblocking of issues.

2.3 Reporting structure

In the first year the group reported to the Department of Health strategic decision-making team, “Gold Command” and worked in close liaison with other operational COVID cells in the Department and with HSC organisations. As the position stabilised, “Gold Command” was stood down but the DCC continued to meet under a revised remit to ensure efficient and effective delivery of digital services supporting the pandemic response.

3. Digital interventions identified by the DCC

The DCC identified a number of immediate priorities to help to cope with the impact on public health and the HSC. These included:

• enabling home working for DoH & HSC staff members
• the development of a COVID 19 web presence for healthcare advice
• the introduction of a clinical triage process to the Pandemic Helpline
• development of a capability to carry out remote consultations

As the situation evolved and it became clear that further rapid innovative interventions would be needed to help lessen the impact of the virus, the group identified and commenced development of a number of other initiatives which were subsequently deployed. These included:

• The COVIDCare NI App (symptom checker app providing immediate advice and links to trusted information on COVID 19).
• StopCOVID NI (an app to alert users if they have been in close contact with other users who have tested positive for COVID-19).
• Digital Self Trace (an automated system to enhance the contact tracing process operated by the PHA telephone Contact Tracing Centre).
• Vaccinations Management System (digital processes to help manage the delivery of the COVID 19 vaccines).
• Data analytics platforms such as the COVID Dashboard (digital systems to assist in the analysis of data to produce information which could help the HSC and other public bodies to respond more effectively to the spread of the virus). 
• COVID Certification Service (multi-channel service that facilitates international travel and domestic use in Northern Ireland). The solution is the subject of this paper.

A further benefit was realised during the COVID-19 pandemic with the delivery of British or Irish Sign Language (BSL) (ISL) Video Relay Service and Remote Interpreting Service, specifically:

• Providing real time interpretation using a video connection on an electronic device including laptop, PC, smartphone or a tablet for several benefit areas.

4. COVID Certification Service

4.1. Introduction

In June 2021, as the impact of the vaccination programme grew, pressure grew to ease restrictions. EU countries that rely on the travel industry, and tourism for a significant proportion of GDP, pressed for international travel to resume. In anticipation of a resumption of a degree of leisure travel it was anticipated that EU, and other countries, would require travellers to share immunity status and/or testing status as a condition of entry. The EU led on the development of a Digital COVID Certificates trust framework.

Northern Ireland citizens would have been unable to avoid quarantine restrictions when travelling without access to evidence of the required standard. As the requirements on the production of secure documentation were confirmed, it was clear that GPs and HSC Trusts would be unable to reliably access the healthcare information required to provide documentation without a significant impact on their ability to deliver their core services. Therefore, a centralised, multi-channel solution to meet agreed international standards was needed.

The DHCNI partnership, NICS NI Direct and key suppliers worked jointly on the development and delivery of a multi-channel COVID Certification Service (CCS) that would, by virtue of vaccination or COVID testing, facilitate international travel (meeting EU and WHO requirements).

4.2. Overview

The scope of the CCS requirements covers both a digital and non-digital, (telephone accessed, paper- based) solutions for people to obtain trusted, and internationally accepted, COVID status certification; initially for use in international travel settings, and later for “domestic use” in Northern Ireland, to meet the requirement that emerged in Autumn 2021. The travel  certificates produced were based on global standards and the emerging work of the World Health Organisation and EU. Domestic certification was delivered in line with ICO recommendations and GDPR compliance.

Integration with NICS Identity Assurance service (NIDA) was required to verify the identity of applicants for certification. It was considered essential that all reasonable efforts were made to ensure that citizens only received accurate certification relating to the correct person, preserving the confidentiality of medical records, and complying with GDPR obligations.

The DTS delivered NIDA solution had previously relied on a manual check process. This had been sustainable due to the low volume of applications made to access services. Due to the anticipated high volume of demand for digital registration, it was recognised as essential to automate the identity checking process, to the maximum extent possible, using AI developed processes. Automation was achieved to a level of 80% to 85%. Manual checking processes were in place to support the remainder.

As part of the CCS, the COVIDCert NI and COVIDCert Check NI apps were developed; and a Helpdesk established to assist with those having difficulty in passing automated checks, and for those unable to use digital channels:

• The COVIDCert NI app was developed to allow citizens to obtain and display a digital certificate satisfying the information requirements for international travel and later to support the requirements of the EU regulations and multiple bilateral agreements managed by the FCDO
• The COVIDCert Check NI app was developed to enable hospitality venues to quickly scan and verify COVID certificates presented without the need to examine personal information
• A Helpdesk was established as
  • the non-digital route (allowing for a secure printed certificate to be posted, following identity authentication),
  • to assist with those having difficulty in passing automated checks,
  • and allow citizens obtain assistance and a COVID certificate by telephone.

Key capabilities of the CCS were:

• Firstly, the ability for citizens of Northern Ireland to obtain a digital or paper COVID certificate to support international travel.
   
• Later the ability for citizens of Northern Ireland to obtain a digital or paper COVID status certificate to satisfy the NI domestic certification regulations for access to specified venues across NI.
   
• Later the ability for the hospitality sector across NI to read 2D barcodes presented by citizens of NI to validate their COVID vaccination status, exemption, recovery, or vaccine trial participant status.

A core team was established to undertake:

• Development and delivery of the COVID Certification Service.
• Procurement of experienced suppliers to develop the COVIDCert NI and COVIDCert Check NI applications.
• Consideration of potential enhancements of the products to meet revised guidance from NI policy and EU standards. Work to inform technical architecture.
• Setup and management of a call centre to support the non-digital channel for citizens of Northern Ireland to obtain a COVID certification.
• Provision of accurate, timely and consistent guidance and information (including frequently asked questions (FAQs) on the use of the COVID Certification Service via NI Direct website; and
• Provision of reports on development of CCS, citizen uptake and potential enhancements to CCS, to the Department of Health, Minister of Health and Northern Ireland Executive.

4.3. Objectives

The primary objective of the CCS was to provide a multi-channel COVID Certification Service (CCS) that would, facilitate citizens wishing to undertake international travel (meeting EU and WHO requirements) by validating vaccination status, or COVID test status and, in addition, support the “domestic use” policy in Northern Ireland, once framed.

Other objectives included:

• Supporting effective infection control, by reducing transmission rates of COVID.
• Contributing to the improved health / well-being of NI citizens, through supporting opportunities for increased social interaction.
• Supporting Economic recovery by providing reassurance (of reduced potential for COVID transmission) during opportunities for social interaction particularly in hospitality environments.
• To produce timely reports to inform policy decisions by Minister of Health and Northern Ireland Executive colleagues.
• And as a by-product, promote vaccine uptake.

DHCNI formally began the discovery work in preparation for development of a CCS product (Phase 1) in early June 2021, to meet international travel requirements whilst monitoring the potential for a “domestic use” policy at a future date. Travel regulations came into place end of June 2021, necessitating the rapid deployment of an interim service within 4 days. The intended digital service was delivered be the end of July 2021, including delivery of the COVIDCert NI app. Work on the development of a Phase 2, to support the emerging “domestic use” requirements began in early September ahead of the confirmation of the policy intent. A voluntary scheme was implemented start of November 2021. The policy regulations mandating domestic certification came into force from 29 November 2021 for Domestic Use as part of the Amended Health Protection (Coronavirus Restrictions) Regulations (Northern Ireland). The early engagement in the further development of the CCS product in advance of 29 November meant that additional features such as recovery certificates and medical exemption services were in place by 13 December when the grace period ended, and legislation (mandating checks for entry to large events and hospitality) came into force.

5. Procurement decisions

Given the impact of the pandemic on all aspects of society, the pace at which public health interventions could be deployed meant that the procurement approach needed to be open, collaborative, flexible, agile, and rapid.  Decisions, information, and designs needed to pass between the top-tier of management to the delivery teams quickly and the ‘traditional’ commissioning and delivery paradigm was not appropriate.

CCS development decision making was undertaken against a highly compressed timeframe due to an urgent need to rapidly implement a service that would enable citizens of Northern Ireland to travel internationally and later undertake activities domestically, that required a validation of COVID status.  

To deliver the complex requirements of CCS in the timeframe required, the use of external support was essential. Where suppliers were already part of existing public sector procurement framework agreements, these were used.

• The requirement to integrate CCS with NIDA meant that suppliers with experience in developing and supporting NIDA (engaged through the competitively tendered Digital Transformation Service framework ‘My Portal’) would be integral to expediting the successful development of CCS.
• DoH & HSC partners (DHCNI) used other public sector procurement frameworks to contract with other suppliers that offered solutions or components of the solution and/or capabilities which were identified as necessary and were not available internally or from existing suppliers. These are set out in the following paragraphs.

Civica delivered the design and development of the CCS digital solution architecture. They delivered an agile development service and provided support for the core solution. They were selected due to being already being contracted (tender process conducted by DTS) to deliver and support NIDA, (a critical component of CCS) and had previously demonstrated value for money over several years of delivering services to Digital Transformation Service (DTS).

Additional supporting services such as the provision of support and maintenance for CCS was procured via the Crown Commercial Services G-Cloud-12 framework (Agreement ID RM1557.12).

Big Motive provided digital service design and user experience (UX) services for the CCS solution. Big Motive, alongside Civica, have been the developers/designers for the NIDA product. Big Motive initially contracted through Civica, however they were later granted a direct award contract to continue supporting design and development, given their detailed and long-term involvement.

The Belfast Trust Azure Tenancy (a private Microsoft based cloud computing environment) was the preferred environment for the deployment and hosting of the CCS solution. The Belfast Trust already hosted a number of services which the COVID Status service interacts with including the various analytical services.

Cloudflare Inc., procured via Crown Commercial Services G-Cloud-12 framework, provided secure resiliency services for the external-facing web services that formed part of the overall solution through management of internet traffic to those services. This enhanced the overall user experience by managing queuing at times of high demand. The solution prevented demand driven system failure experienced in other jurisdictions.

Expleo Technology UK Limited provided Quality Assurance (QA) testing and were experienced in dealing with the solutions and services being used. CCS used an existing call off contract held by the NI Civil Service for this engagement.

Gartner UK Ltd were engaged to support Programme delivery and ensure robust Programme management. Gartner had already performed a similar role for DHCNI on the Vaccine Management Service and supported the Discovery phase of the CCS project. Their services were procured via Crown Commercial Services G-Cloud-12 framework.

HH Global (HH Associates) supply a secure printing service for COVID certificates (including necessary 2D barcodes) and supporting documentation (recovery certificates, exemption certificates, etc.). HH Global are a UK Government partner for secure printing and are available on Crown Commercial Services framework RM6170 (Lot 1) where direct award to HH Associates is the only permissible outcome.

IT Guarded provided cyber security Penetration Testing (Pen Testing). They have a proven capability with NI public sector over many years. IT Guarded were procured via Capita who are available on the DoH Technical Framework Agreement (TFA) framework which was leveraged for CCS.

Kainos provided data validation, data cleansing and data testing services, and had previously supported NI on the Vaccines Management System. CCS required to continue data cleansing of the VMS where 40,000 plus duplicate records needed to be addressed. It was proposed that Kainos continue to carry out this work given they have conducted this work on behalf of VMS to date. The services required from Kainos were procured via Crown Commercial Services G-Cloud-12 framework.

SureCert were selected to provide AI automated identity verification as they had demonstrated a proven capability with both UK government, other devolved administrations, and the private sector for. Their services were procured through the Crown Commercial Services G-Cloud-12 framework.

The original intention was to use NI Direct’s call centre capability to support the manual identity verification process and offer general support and advice to applicants. However due to the resources required to scale the service and the urgency of providing a solution, NI Direct lacked the capacity to undertake the work at that time and consequently external support was sought. A rapid market investigation was undertaken to identify an organisation with readily available resources to provide the required analytical and customer facing capabilities within the funding envelope. Only Ernst & Young LLP (EY) were able to meet NI needs, and subsequently their services were procured via their Crown Commercial Services G-Cloud-12 framework offering to establish a call centre by October 2021.

5.1. CCS Development Drivers and Decision Making

The DHCNI team used rapid, agile, parallel techniques to deliver selection, design, build and implementation of the CCS. Analysis of the traditional approach using linear, requirement capture, investment and procurement processes revealed that they could not deliver the required outcomes in the time available to meet public expectations.

The initial focus of CCS was the development of a service to support citizens of Northern Ireland to travel internationally. The service was later used to assist the introduction of “Domestic Use” policy in November 2021.

• Detail on development of international requirements began to emerge in early 2021 with sudden implementation of a certification requirement for travel in Europe at the end of June 2021;
• Formal discovery work in relation to developing the CCS started in June 2021;
• An interim service was available from 2^nd July 2021;
• The international travel digital solution was available from 19^th July; and
• Facilitating voluntary use of COVID Status certification for “Domestic Use” purposes was added 1^st November 2021.

Policy enforcing Domestic Use of COVID Certificates in Northern Ireland came into effect in 29 November 2021 with a two-week grace period, with mandatory checking in force from the 13 December 2021.

From 29 November, individuals had to prove their COVID status for access to nightclubs, licensed hospitality premises, including 'bring your own' alcohol venues, cinemas, theatres, conferences and exhibition halls,            indoor events with 500 or more attendees with some or all of the audience not normally seated, outdoor events with 4,000 or more attendees with some or all of the audience not normally seated and events where more than 10,000 people will be present regardless of whether they are seated.

From 26 January, individuals had to prove their COVID status for access to nightclubs and indoor events with 500 or more attendees with some or all of the audience not normally seated.

The legislation was amended in February 2022 removing the mandatory requirement for Domestic Use COVID Certificates in any setting. Guidance remains in place to encourage conducting a risk assessment for gatherings, events, entertainment, leisure, and sporting activities.

Ongoing updates were made to the CCS as new policy elements were introduced or amended, such as new vaccination requirements defined, the addition of the ability to display Booster Vaccines and the ability to “Apply On Behalf Of another individual”.

5.2. Information Governance

Throughout development of the CCS, the programme team engaged with the relevant Information Governance teams (IG) and Data Protection Officers (DPO). The CCS IG team was tasked to produce and develop the documents with advice and input from all three DPOs (Department of Health, Health & Social Care Board, and the Public Health Agency). Meetings were held weekly, and developers & IG professionals were in constant dialogue throughout.  

The Data Protection Impact Assessment (DPIA) for CCS was published on the relevant website (and accessible via the App) and refreshed as required.

5.3. Data Breach

In late July 2021, reports were received that some citizens applying for certification were seeing vaccination details of others, temporarily online, during the application process. As soon as the issue emerged, the ICO was informed, and new applications to the CCS were temporarily suspended (taken offline) for five days. During this period, certificates were manually processed for applications already received, while the cause of the issue was identified. A previously satisfactory software component (part of the existing NIDA infrastructure that was incorporated in the pre-existing NIDA solution) had failed under heavy load conditions. The failure resulted in one account holder seeing some information related to another, during the online application process. Although up to 1,300 accounts could have been vulnerable, only 6 complaints were received. The ICO was informed, immediately the risk of a data breach was confirmed, on 27 July 2021,.

The component that was identified as the source of the problem (provided by a developer that was not part of CCS), could not be reconfigured in a way that could guarantee data security in the context of high-volume demand. In the absence of sufficient data security assurances, a decision was taken to have CIVICA build, and stress test an alternative integrated component, providing the necessary assurances in relation to data security.  The CCS commenced receipt of new applications for certification on 30 July, with a managed service re-introduction, priority being given to those traveling soonest. In a managed process, restrictions on application for certificates were gradually eased, and ultimately removed.

The ICO later confirmed that they were content within the actions taken to contain the potential data loss and the additional mitigations deployed on the new service. No further action was deemed necessary by the ICO.

6. Assessing Value for Money in a Pandemic Use Case

The COVID Certification Service was established urgently to meet the emerging WHO & EU requirements for documentation to enable citizens to travel internationally

The CCS team therefore commissioned Gartner[1], the independent IT research company to provide ongoing programme assurance for the CCS work and reporting data for updates to the Minister of Northern Ireland and Northern Ireland Executives. Gartner also provided commercial support and advice to the CCS Director, and DHCNI, during negotiations and contract reviews. In leading negotiations with suppliers, the CCS Director sought to ensure VFM, driving down development and delivery costs in negotiation, wherever possible.

DHCNI, (on behalf of the CCS service delivery program,) utilised wherever possible, existing government procurement frameworks to shorten the procurement cycle. The procurement frameworks had already dealt with the selection process leaving only finalisation of the requirements and pricing for activities within the scope of the contracts.

• Civica/Big Motive and Expleo had already successfully competed for, and won, positions on the Digital Transformation Service (DTS) ‘My Direct’ framework, and a DoF call off contract respectively.
• HH Global (Associates) are the UK Government approved partner for secure print services and are the sole provider on the Crown Commercial Service framework RM6170 where a direct award is the only route permissible. Pricing provided by HH Associates was compared to that offered to other devolved administrations and deemed to offer VfM based on NIs required volumes.
• Cloudflare, EY, Gartner, Kainos and SureCert were procured using solutions and rates approved for inclusion in Crown Commercial Services G-Cloud-12 framework. Day rates are already agreed upfront as part of this framework agreement and further discounts are not allowed. Careful consideration was given to the scope and scale of each Statement of Work to ensure that it exactly met CCS needs. Pricing was compared to other NI government programmes, and where available, to other devolved administration agreements for VfM validation.
• IT Guarded were procured using the existing DoH Technical Framework Agreement (TFA). Competitive day rates were agreed up front as part of their proposals and submitted to the Department of Health for review.

All suppliers undertook to work at risk for several weeks while the contract paperwork went through the usual procurement processes. The CCS team observed that all suppliers worked longer hours and over weekends than their agreed rates permit. This additional effort has given DHCNI a de-facto discount.

6.1. Business Case development

From the beginning of the development of the service it was recognised that several processes, usually undertaken in a serial manner, had to be conducted in parallel if digital solutions were to be available to support he response to the COVID pandemic.

The Covid Certification Service (CCS) went live on the 20th July, with the App launched on 24th July, one month ahead of schedule. The final version of the Outline Business Case (OBC) for CCS was approved on 27th July 2021 following the approval of several interim submissions to fund discovery and accelerate solution design and development. The OBC approved and allocated a total of £11.85m revenue for 21/22 and £4.03m Capital. However financial tracking highlighted that the expenditure, to support policy developments and meet emerging requirements, would exceed the allocated amount.

An Addendum was prepared and submitted in October 2021 to acquire necessary additional funding of £4.17m.  Following changes in guidance on funding services to support the response to COVID, given on 7th December, a revised request (in the format known as Annex A) was submitted and reflected that the additional funding required had increased from £4.17m to a total of £5.58m.  This was due to the policy decision to introduce mandatory Covid Certificates for use for “Domestic purposes”.

The formal approval process for additional funding continued for several months, between Department of Health and Department of Finance, with extensive discussions on how best to capture and present the continued investment in COVID Digital assets. Particularly where there may be some future public health utility beyond the pandemic. This was concluded with agreement that the COVID spend would be justified through a COVID ‘Annex A’ for each product, covering an estimated period up to end of March 2023. The Annex A and a supporting, approved submission were formally submitted to DoF for approval in January 2022.

The length of time needed to finalise the approach to approvals combined with evolving requirements meant that the quantum of the quantum of the budget approval on 30th September was no longer sufficient. DoF approval was granted originally on 27 July 2021 for total expenditure of £15.88m on the COVID Certification Service. As further DoH expenditure had been committed, breaching the original conditions of approval, DoH sought retrospective approval for expenditure of £20.67m (£16.63m Revenue and £4.033m Capital) from DoF Supply on 31/3/2022.  

Upon review DoF Supply was content that it would have approved the additional expenditure had DoF been approached at the appropriate time, following the normal processes. The approval was granted on 11/4/2022.