Statistics
The Scottish Crime and Justice Survey, impact of the COVID-19 pandemic on the latest survey sweep
August 12, 2024 by lisaclark No Comments | Category Justice
The COVID-19 pandemic forced us to stop face-to-face interviews for the Scottish Crime and Justice Survey (SCJS) along with the other household surveys. When fieldwork restarted in November 2021, remaining restrictions meant we had to conduct the survey differently. But we didn’t know if this would affect the quality of the data collected, who would respond, and if it would still be comparable to our earlier surveys dating back to 2008.
This blog looks at how we addressed these concerns.
Change to the mode of interviewing and the potential impact to the results
Before Covid, a typical SCJS interview was done face-to-face in respondents’ homes. When we resumed fieldwork with pandemic restrictions, we used a ‘knock-to-nudge’ approach – interviewers called at an address and then arranged a telephone or video interview. From April 2022, respondents had the option of a face-to-face or remote (phone or video) interview. In line with other surveys, our response rates fell: from 63% in 2019/20 to 47% in 2021/22
We recognised that this change could impact the survey results. We risked having a significant shift in the profile of people taking part. For example, could individuals who typically experience less or more crime be more likely to respond to a remote interview? This could artificially change our estimates of the volume or prevalence of crime in Scotland.
Additional analyses
Alongside the regular quality assurance processes, we commissioned Ipsos, one of our contractors, to investigate the impact of changes. They looked at questions such as:
- did response rates differ by geography – such as police division areas across Scotland, or by urban vs rural or by deprivation?
- was there a change in the profile of respondents pre-and post-pandemic?
- were there different results for different interview types, and
- did respondent profile or results change again when face-to-face interviews became an option once more?
It showed that the changes in the composition of people surveyed were relatively small compared to before the pandemic, but there was a slight shift towards more owner-occupation of houses and fewer respondents with no educational qualifications. It found no significant differences between the different types of interview.
Keeping users informed
With these findings, we were content that the approach taken during the pandemic hadn’t adversely affected the quality of data collected and were comfortable presenting this as a continuation of the long-term time-series.
The investigation was published separately alongside our main report (link here). We also included a summary of the findings and why it was undertaken in the main report to make sure our users were properly informed.
Informing the Office for Statistics Regulation
We spoke with the Office for Statistics Regulation (OSR) before publishing: presenting on the issues we faced, the results of our analysis, our view that the results were comparable, plans to publish, and aim to retain an accredited official statistics designation. They were overall pleased with our proactive approach and provided useful and positive feedback before we continued.
Reflections and future work
While these findings don’t show a significant shift in types of people taking part in the SCJS in recent years, it does suggest there may be smaller changes (such as more home owners) which could have an impact over the longer term; and we need to remain aware of this.
This work was extremely valuable for us: from a data quality perspective; to help us and users understand the impact of COVID-19 on the survey; and also for our team, who were all going through the survey publication process for the first time, get a better understanding of who does and doesn’t respond to the survey.
If you would like to know more you can contact us at scjs@gov.scot or look at the key sources shown below.
RSS
Email
Leave a comment
You must be logged in to post a comment.