Digital

Tackling child poverty with machine learning

April 23, 2025 by Stewart Hamilton No Comments | Category Data Science and Innovation Accelerator, Digital Scotland

Blog post by Spencer Thompson, Economic Advisor, Scottish Government

The Data Science and Innovation Accelerator is a programme for public sector organisations in Scotland that supports the innovative use of data. In his blog, Spencer Thompson discusses his 2024 Accelerator project on child poverty.

“Over the past 6 years I’ve been working towards the Scottish Government’s mission of tackling child poverty. During this time I’ve been haunted by the most basic of questions – who are the children that live in poverty in our country? The question may be basic, but it’s by no means simple to answer.

According to the latest child poverty statistics, nearly one in four children in Scotland are living in relative poverty. Of those, around nine out of ten fall into at least one of the ‘priority groups’ as defined by age, ethnicity, disability, and family size and structure. Many other characteristics are also relevant, depending on the context – and all of these characteristics intersect.

But how can we make sense of the resulting complexity? Standard cross-tabulations only gets us so far. The number of possible combinations will increase exponentially with the number of characteristics, so it doesn’t take much before we have exceeded what the human brain can grasp at an intuitive level. The priority group characteristics alone contain 64 possible combinations – and that’s if we treat them as simple ‘yes/no’ binaries.

The Accelerator gave me the opportunity to seek answers to my questions. Having completed the ONS Data Science Graduate Programme the year before, it was the perfect follow-up, allowing me to put my learning to work in the service of one of the Scottish Government’s chief priorities – and learning a lot more in the process.

In particular, I set out to discover whether machine learning techniques can help us identify distinct ‘clusters’ of households in poverty. The idea is that each household with children in poverty is categorised into one, but only one, of a number of clusters. If the method has worked well, households within each cluster should be similar to each other, while households in different clusters should be dissimilar to each other. By characterising these clusters, we can provide a simplified but rigorous answer to my original question: who are the children that are living in poverty?

The idea is straightforward enough, but implementing it turned out to be more complicated than I’d envisaged. The guidance of my mentor, Chenglei Hu, was invaluable in navigating the various challenges that I encountered. We made a great team, with my knowledge of the data and the policy area combining with his knowledge of cutting-edge methods and approaches.

Besides picking up some valuable new skills, a key learning point for me was to be open-minded. I was initially intent on designing a fully unsupervised machine-learning task, where I ‘let the data speak’ with minimal instruction or intervention. But this led me down a number of dead ends; and it was only when I heeded Chenglei’s advice to introduce an element of supervision that I was able to find a way forward.

Another lesson for me was not to lose sight of the ‘why’ in the midst of the ‘what’ and the ‘how’. We ended up layering on several different methods, which allowed us to solve technical problems and produce a robust set of results. But by the same token, interpreting and communicating these results – and ultimately using them to inform policy – requires more care and attention.

And finally, to affirm that old cliché, I was assured that it’s okay to ask daft questions – in fact, it’s necessary. To put it somewhat loftily, in the supposed words of Erwin Schrödinger, the great physicist of quantum cat fame: ‘the task is not so much to see what no one has yet seen; but to think what nobody has yet thought, about that which everybody sees.’ If we are to achieve our ambitious goal of tackling child poverty, we will need all the daft questions we can get.”

If you’ve been inspired by Spencer’s quest for understanding and want to uncover the stories in your data, visit: Data Science and Innovation Accelerator – Scottish Digital Academy and apply by 28 April for our 2025 programme.

If you are interested in supporting people to progress the answers to important societal questions, we’d love you to consider volunteering as an Accelerator mentor. You’ll find all the information you need at Data Science and Innovation Accelerator – Mentor – Scottish Digital Academy.

Tags: child poverty statistics, Data Science and Innovation Accelerator, scottish government

Comments