Current analysis and data around COVID-19
Scottish Government (SG), Public Health Scotland (PHS) and National Records of Scotland (NRS) have been working together to produce a range of information to provide insight into the COVID-19 pandemic and the effect it is having in Scotland.
Scottish Government publish daily coronavirus data for Scotland (updated each day at 2pm) that shows number of new confirmed COVID-19 cases, new reported deaths of people who have tested positive, number of tests carried out (and whether results were positive or negative), and numbers of people in hospital. It also show the number of NHS24 calls, Scottish Ambulance Service Attendances, the number of people delayed in hospital, and numbers of suspected COVID-19 cases in Care Homes.
National Records of Scotland are publishing weekly National Statistics on deaths involving coronavirus, and Public Health Scotland are releasing weekly data on the cumulative number of positive COVID-19 cases by age, sex and deprivation, admissions to ICU, daily number of COVID-19 related calls made to NHS24 and daily number of consultations with COVID-19 Community Hubs and Assessment Centres. PHS also produces daily and weekly open data, a daily cases and deaths (including excess deaths) dashboard, a weekly wider impacts dashboard, and has plans to publish different topic area COVID-19 reports.
Public Health Scotland’s COVID-19 publications are accessible here: https://beta.isdscotland.org/covid-19-and-the-production-of-statistics/ and all their forthcoming publications are accessible here: https://beta.isdscotland.org/forthcoming-publications/.
Upcoming releases of analysis and data around COVID-19
During this public health pandemic, in line with guidance from the Office for Statistics Regulation, we are planning to publish a range of additional analysis on topics such as numbers of people identified as at high clinical risk of COVID-19 and advised to shield, numbers of tests carried out, causes of death, and other analysis of demographic characteristics. We know that there is a lot of demand for this data, and doing so will make sure that information relevant to the COVID-19 crisis is openly available to inform the ongoing public debate.
Some of the data being reported is management information (data that is gathered routinely by public bodies to inform their day to day operations). Management information produced by the Scottish Government is subject to voluntary application of the Code of Practice for Statistics, and our aim is to make sure it is published in a way that maintains trustworthiness, quality and value to users. For example, difficult choices may need to be made on whether data are of sufficient quality to support the use being made, and the professional analysts producing the analysis, will balance these risks to quality with making sure that they are published as quickly as possible.
When management information is used publicly to inform Parliament, the media and the public, it should be published in an accessible form, with appropriate explanations of context and sources. It is important therefore that this information is released in a transparent way to ensure that it is equally available to all. As part of that transparency, we preannounce our plans for what management information is going to be published, and when, noting that these analyses are often complex and involve multiple data sources, and plans may by necessity be updated at short notice.
Our current plans for release of new information are:
- Data on adult care homes and COVID-19 deaths reported to the Care Inspectorate: https://www.gov.scot/publications/coronavirus-covid-19-daily-data-for-scotland/ (released on Wednesday 3 June, weekly thereafter)
- Data on number of people identified as at high clinical risk of COVID-19 and advised to shield: https://www.gov.scot/publications/coronavirus-covid-19-daily-data-for-scotland/ (confirmed for Thursday 4 June)
- Blog Updated on 25/06/2020. Public Health Scotland’s weekly report will include updated COVID-19 data including Test and Protect data. PHS will be releasing new data on their COVID-19 wider impacts dashboard (and this will be signposted to in their weekly report): Data on stillbirths and infant mortality (confirmed for Wednesday 1 July). Other topic areas will follow in subsequent weeks: https://beta.isdscotland.org/covid-19-and-the-production-of-statistics/
- Additional analysis will be published by National Records of Scotland for May covering deprivation, age-standardised rates, leading causes of death, pre-existing conditions, and urban rural breakdowns. There are also plans to include age-standardised death rates for Health Boards and local authorities, deaths by intermediate zone and (possibly) a breakdown by occupation: https://www.nrscotland.gov.uk/statistics-and-data/statistics/statistics-by-theme/vital-events/general-publications/weekly-and-monthly-data-on-births-and-deaths-registered-in-scotland (on 17 June)
- NRS are also currently testing whether data from the 2011 Census can be used to produce an analysis of death registration data by ethnicity.
Open data around COVID-19
Scottish Government, National Records of Scotland, and Public Health Scotland are all publishing open datasets on the COVID-19 pandemic for people to use and reuse. This helpful blog post gives more details of what is available and where to access these sources of open data on covid-19.
The Office for National Statistics have also published their plans for further analysis of COVID-19 data.
I wanted to share an update on how Research Data Scotland is supporting the national response to COVID-19.
RDS’s mission remains to improve the economic, social and environmental wellbeing in Scotland by enabling access to and linkage of data about people, places and businesses for research in the public good.
In response to the national emergency, a new data taskforce, of which I am chair, has been established. While Scotland have been delivering high quality research in a secure and ethical way for a number of years, the ambition of the data taskforce is to enable evidence-based policy and operational decisions in response to the current COVID-19 situation.
This will be achieved by building on and repurposing our existing and emerging data infrastructure which includes resources, expertise and capabilities offered by service delivery partners and partner organisations, including The Scottish Government, PHS (eDRIS), NRS, EPCC and HDR-UK, alongside representatives from the University of Aberdeen, University of Dundee, University of Edinburgh, University of Glasgow, University of St Andrews and University of Strathclyde. This builds on existing, accredited infrastructure and methodologies that enable us to maintain privacy and security while delivering outputs quickly.
Researchers will be able to carry out collaborative research and analysis that provides evidence for important COVID-19 related decisions, initially to support decisions on lifting social restrictions.
This group has identified key datasets that will help research to support decisions on lifting social restrictions, and the development of a plan for securing these datasets. This includes data on;
- use of healthcare services
- care home residents
- vulnerable groups
- school pupils
- Census data on work, family structures and commuting
An infrastructure has also been established to hold and link data securely, and to provide secure online access for collaborative research. This has been developed by the Public Health Scotland and Edinburgh Parallel Computing Centre, and offers a virtual analytical workbench to allow research teams to work on the same datasets from different locations.
Work is well developed to secure these datasets, meanwhile forthcoming emergency data sharing legislation will allow us to operate quickly, while ensuring that the data is accessed legally and ethically. We’d also welcome suggestions for additional datasets that may support emerging sets of needs.
For more information about accessing this data please visit ResearchData.Scot
The use of data has greatly assisted us in understanding COVID-19 and will help our response to the pandemic. The use of data for this type of research is very important and gaining access to the appropriate data remains both legal and ethical.
To ensure that we balance our legal and ethical obligations whilst providing data for research in a timely manner, we have temporarily revised our process to access data for projects that will assist with the COVID-19 response.
The revised process can be found in this statement and details of the projects being approved using this process are available on request.
Scottish Government (SG), National Records of Scotland (NRS) and Public Health Scotland (PHS) are all publishing open datasets on the COVID-19 emergency for people to use and reuse. We know that there’s lots of interest in open data on COVID-19 with people keen to use it in briefings, data journalism, documents, apps and dashboards. It’s really encouraging to see such an engaged and lively open data community response and so the purpose of this blog is to help signpost users to relevant open datasets produced by these organisations. We intend to update this blog as more relevant datasets become available. Of course, given all of this interest, it is important to have accurate and robust information made available as open data for reuse and for decision making and the teams in SG, NRS and PHS are working hard to uphold these standards.
It is strongly advised that people using these data use the published metadata which are produced to support each dataset. The metadata describe the data and provide more information about it. This is vital for the effective use, reuse and interpretation of datasets, and any products that are created using these data.
We welcome feedback on the data and how you’re using it. Whilst our team cannot respond to detailed questions on the data itself, we’re happy to pass this on to the teams who produce it.
SOURCES OF OPEN DATA
http://statistics.gov.scot/data/coronavirus-covid-19-management-information. Data are available at Scotland level; some data are also available at Health Board level. We’d like to thank our contractors, Swirrl, for their help in supporting us to get these datasets published.
Our contractors at Swirrl have recently published a series of short videos to help users to explore the 5* linked open datasets on www.statistics.gov.scot: https://guides.statistics.gov.scot/article/53-video-guides
We have published a series of background helpguides for programmers to build apps and dashboards from www.statistics.gov.scot: https://guides.statistics.gov.scot/category/37-api
Our contractors at Swirrl have also published a blog on the reuse of the daily COVID-19 management information: https://medium.com/@northernjamie/covid-19-in-scotland-four-ways-to-get-the-data-e12c1b450a5. Their blog provides details on how to reuse the daily data from www.statistics.gov.scot, using 4 different packages – using R, Python, Google Sheets, and Tableau. There are examples of code which can be used to generate apps, dashboards and visualisations for each of the packages, as illustration of how the data can be retrieved. At the end of the blog, there is a section on the importance and benefits of publishing open data.
We have also published the R code used to generate these datasets as open data, as well as the 3* csv files on the DataScienceScotland GitHub repository : https://github.com/DataScienceScotland/COVID-19-Management-Information . We have done this to help users in the open data community who we know are building and updating apps and dashboards from these data.
Whilst we intend to publish the daily management information COVID-19 data by mid-afternoon each day, there may be, at times, unforeseen delays in getting the daily open data published. Please note that if this is the case and you need the data urgently, published data can usually be found in Excel format at 2pm on the Scottish Government website. We thank you in advance for your patience.
National Records of Scotland (NRS) are publishing weekly National Statistics on www.statistics.gov.scot on deaths involving coronavirus (COVID-19) http://statistics.gov.scot/data/deaths-involving-coronavirus-covid-19 .Data are available at Scotland level, and for some measures at council area and health board level.
This supports the weekly NRS publication: https://www.nrscotland.gov.uk/covid19stats . Further information and other relevant demographic data can be found within the NRS blog “Statistics relevant to COVID-19”: https://blog.nrscotland.gov.uk/2020/04/23/statistics-relevant-to-covid-19/
Public Health Scotland publishes daily and weekly open datasets on the Scottish health and social care open data platform as 3* open data files. Data are available on the daily and cumulative number of positive cases of novel coronavirus (COVID-19) in Scotland, including cumulative number of positive cases at health board and council area level, and deaths at Scotland level. Data are released weekly on the cumulative number of cases by age, sex and deprivation, admissions to ICU, daily number of COVID-19 related calls made to NHS24 and daily number of consultations with COVID-19 Community Hubs and Assessment Centres.
It is intended that the scope of COVID-19 related data within this open data platform will increase, as more evidence becomes available. The COVID-19 section of the portal can be accessed at: https://www.opendata.nhs.scot/dataset/covid-19-in-scotland and https://www.opendata.nhs.scot/dataset/weekly-covid-19-statistical-data-in-scotland
BACKGROUND AND USE CASES
Open data are those that are accessible, free of restriction on use or redistribution in its licensing conditions and in a digital, machine readable format for use with other data. They are published according to the 5* deployment scheme for open data.
The datasets which are referred to in this article are all published under the Open Government license. Given that these data are open, they are intended for anyone to use and build apps and dashboards from.
Below is an example of an official dashboard – the COVID-19 Scotland dashboard, produced by Health Protection Scotland and Public Health Scotland: https://www.arcgis.com/apps/opsdashboard/index.html#/658feae0ab1d432f9fdb53aa082e4130
We are also aware of some good examples of other dashboards which have been published using open data. These have been created with the support of some of our Open Government civil society partners and the open data community more generally.
This is a good example of what someone using open data can build with it. https://smazeri.shinyapps.io/Covid19_Scotland/ . This dashboard has been produced with support from the Epidemiology, Economics and Risk Assessment Group (EERA) within the Roslin Institute. It is important to remind people looking at public data and dashboards produced by individuals that different countries and organisations may gather data differently using different tests and definitions so we need to be careful about direct comparison, and methodologies can change over time.
Scotland is a member of the Open Government Partnership: https://www.gov.scot/policies/improving-public-services/open-government-partnership/ . In 2019, Scotland’s second Open Government Action Plan was launched in conjunction between Scottish Government with civil society representatives, and includes a commitment to improve how information and data are shared: https://www.gov.scot/publications/scotlands-open-government-action-plan-2018-20-detailed-commitments/pages/4/. This work has progressed with the help and input of our partners in the Open Government Network. We would like to thank our Open Government civil society partners and the open data community more widely for their continued support and enthusiasm in highlighting and promoting the value of using and reusing open data.
Rural and Environmental Science and Agricultural Services (RESAS) collect a range of statistical information on the agricultural sector which is the evidence base for the majority of our analysis on agriculture. However, in light of the current guidance from the Scottish Government and the Chief Medical Officer we have had to change our planned schedule of statistical surveys in agriculture for 2020.
These changes have been developed in consultation with our colleagues in DEFRA and the other devolved administrations. While Wales and Northern Ireland are still assessing the options for their June Census collection, we have all agreed to take a common approach to ensure UK figures can still be produced on all major agricultural surveys.
This approach ensures we comply with current legislation and guidance provided by both the Government Statistical Service and the UK Statistics Authority. The approach being taken is also in line with guidance being issued on all major and face-to-face surveys conducted by the Scottish Government.
For a full explanation of the changes to our proposed schedule for 2020, please see the attached letter to stakeholders..
In light of the extraordinary situation we are now in, in relation to COVID-19 and given the challenges we are all facing, this is an update on our contingency arrangements. As our analytical staff are supporting mission-critical work with other colleagues across government, we have had to make some difficult decisions in relation to current data access arrangements.
Due to the current increased pressures, we are not accepting any new bespoke data access requests for Scottish Government datasets or NRS census data, unless directly COVID-19 related. This will include variations (amendments) to existing data access permissions which have already been granted. We will review the situation over the course of the current public health emergency in light of Government advice and public health guidance.
We want to continue to support research, and those bespoke requests we received before 5pm on 24th of March will be reviewed and considered as normal, although please bear with us, as it may take longer than usual to reply to you. If your project is being handled by eDRIS, please also contact your research co-ordinator and refer to the eDRIS service users’ update of 19 March 2020.
Our data is still accessible via the Open Data Platform: www.statistics.gov.scot or in over the one hundred bulletins we publish each year, most of which include data that can be downloaded.
For further details please see our full 24 March statement.
Statisticians and researchers within The Scottish Government and other public bodies collect, analyse and publish official statistics and research reports to help government, business and the public make informed decisions.
The coronavirus illness (COVID-19) is a significant challenge to us all, and we are working with colleagues across the Government Statistical Service and Government Social Research profession, to ensure that Scotland along with the UK as a whole has the vital information needed to respond to the immediate and longer term impacts of this pandemic on our society and economy.
This means we will need to change our usual operations, to ensure we are able to support the government’s response to this crisis as well as providing the necessary information when it is needed the most.
We also need to consider what it is right and appropriate for us to be asking data providers and respondents to supply to us at this time, given the pressures on key areas and staff. We will be making urgent decisions around which data collections can be paused, or in some cases cancelled.
The changes to our work, and data collections and research activity, will mean that the production of some statistics and planned research outputs will need to be suspended. It could also affect the quality of some of our other statistics, in terms of accuracy, or the level of detail available, such as less commentary or fewer breakdowns. There will be new outputs related to COVID19.
Given that some statistics and research outputs will be even more important during this period, whereas others less so, decisions will be made on a case by case basis, prioritising what is relevant to the current situation, and we will be open and transparent about decisions made and the potential effects on our statistics and research. Throughout this process, we will remain guided by the Code of Practice for Statistics in publishing official statistics, that are of public value, are high quality and can be trusted, as well as the Government Social Research Code and publication protocol.
The latest information on our planned outputs or postponements will continue to be available on our release calendar.
The Office for Statistics Regulation have also issued a statement about their approach to regulating our statistical work at this time.
Roger Halliday, Chief Statistician
Audrey MacDougall, Chief Social Researcher
You will know that I was asked by the Cabinet Secretary for Social Security and Older People to put together guidance to support public bodies collect and present data on sex and gender. I have brought together experts in collecting and presenting data from across the UK. We met for the fifth time last week, and I wanted to give an update on this work.
Over the last month we have run two public events to hear from people about this issue, in particular examples where data on sex, on gender, and on the trans population is specifically needed or would be helpful. We talked about where people have seen this data collection done well or not so well. We also discussed the issues that someone collecting data about sex and gender needs to consider in practice.
Thank you to the 40 or so people who were able to attend: it was good to get a real mix of academics, members of the public, and representatives from public sector organisations. I got lots of important information. If I had to pick out some consistent themes across the events it would be…
• It is important for public bodies to be clear about the purpose of collecting the data and how it will be used;
• The need for clear definitions, simple language, and standardised questions;
• Collecting disaggregated data in order to highlight if there are inequalities between men and women generally which need to be tackled;
• That there are circumstances where information on sex, gender, and the trans population is needed by different public sector bodies;
• In seeking this information, it is important to ask this in ways that are respectful, where people are adequately trained in how to ask questions sensitively. This helps respondents answer the questions and avoid situations where organisations are concerned about collecting the data and therefore avoid doing so.
• Key is maximising the utility of the data whilst providing assurances around confidentiality and privacy
I have been reflecting on what I heard not just from the public events and the working group, but also from relevant organisations I’ve met and who have kindly shared views and evidence. I’m currently putting the final touches to a first draft of the guidance for public bodies. I’m expecting to seek feedback on this from those who have already shown an interest in this work.
My intention remains to provide advice to Ministers in the late Spring.
We’ve been making solid progress developing Research Data Scotland (RDS) over the last few months, working closely with our partners and stakeholders. I wanted to take this opportunity to answer some of the common questions that have been discussed in my recent conversations.
As developments continue, I’ll share further updates that will look in more detail at the progress being made around the user journey, legal status, governance and funding.
Once fully operational, what is RDS aiming to do?
RDS’s charitable mission is to improve the economic, social and environmental wellbeing in Scotland by enabling access to and linkage of data about people, places and businesses for research in the public good.
What are the principles on which RDS is founded?
There are six principles on which underpin RDS’s charitable mission;
- RDS will only enable access to data for research that is for the public good and improves our understanding of equalities
- RDS will ensure that researchers and RDS staff can only access data once an individual’s personal identity has been removed
- RDS will ensure that all data about people, businesses or places is always kept in a controlled and secured environment
- RDS will be transparent about all it does, with details of all applications to use data being published
- All income that RDS generates will be re-invested into services to help researchers continue to access data
- Firms that access public data for the public good through RDS will share any commercial benefits back into public services
How will RDS achieve this?
RDS will declutter data access in Scotland. Their data catalogue will help researchers understand what data is available for research, and their researcher service will offer a seamless pathway that brings together the various parts of the researcher access journey regardless of the type of data being requested.
What services will RDS provide?
RDS’s initial service offering will provide a single point of access to help researchers access a suite of key public sector data. The aim of RDS is to empower researchers and analysts to unlock value from public sector data. RDS will facilitate this by partnering with service delivery partners such as National Records of Scotland, University of Edinburgh/Edinburgh Parallel Computing Centre and NHS/NSS.
When fully operational, RDS will:
- Develop and maintain a data holdings portfolio that is de-identified and held securely
- Provide a catalogue of the various datasets and metadata available that will include Data Controllers requirements, sampling methodology, and any restrictions on data use
- Triage researchers’ enquiries and provide customer support service by offering a seamless journey from the point of enquiry to data delivery
- Provide a service allowing data controllers to bring their datasets and link these to established datasets in the RDS portfolio for research in the public good
- Provide a data analysis service
- Ensure data access via the Scottish network of safe settings
Will I have to pay for access to data?
RDS will charge for services to access datasets. However, RDS is a not for profit organisation. Income will be invested in the service to pay for the expertise needed to keep data safe and to make it as useful as possible for researchers doing work in the public good. Data will not leave the safe settings overseen by RDS.
If private sector organisations use RDS to access public sector data, they will pay for that access. In addition, RDS will agree a benefit sharing arrangement where a share of profit derived from this data access will be fed back into public services.
Why are you launching RDS now?
The Scottish Government is committed to facilitating data being shared securely for research that is in the public good. Currently it can be unclear what public sector data is available for use in research with the data being of unknown or poor quality. It can also take too long to access data as a result of it being dispersed between and within public sector organisations.
We want to work with data controllers and users to improve the quality of data for research use while also making access to data more cost-effective, faster and more streamlined. We need to ensure there is ongoing trust, support and feedback from the public.
The Scottish Government invites users of the Equality Evidence Finder to complete a short survey to help us better understand user needs.
The Equality Evidence Finder is the first point of call for people looking for equality statistics and social research in Scotland. It’s used widely across the public and third sectors for a range of purposes including Equality Impact Assessments (EQIAs) and reporting on the Public Sector Equality Duty (PSED). The site was modernised and relaunched in late 2018 and we’d now like to hear your views.
Your opinions and suggestions will help to shape the future development of the Equality Evidence Finder. An improved site will help policy makers to locate important equality evidence which will lead to better and more inclusive policy making, in turn leading to a fairer Scotland.
The survey will close on 20th March