All players need to work together to establish widely agreed standards for validating simulated data
The research industry is being transformed by AI in many ways but especially when it comes to simulation. Whether this is a dangerous development or a beneficial advance depends on whether clients can know if its outputs are reliable or not.
We want to know – and we want you to know – that you can trust YouGov’s AI twins (LLM-based simulations of real people’s attitudes). So, we have conducted a far-ranging and fully transparent test of one-to-one human to digital twin simulation. This groundbreaking study demonstrates scientifically that our simulated twin data can produce information of high value for a wide variety of business applications.
However, while we know that this is true of YouGov's simulated data, we can't be as confident about those who have not yet done the same. Clients and the public must know exactly how much they can trust AI twins from other providers and if and how they can use them.
So, today we are issuing a challenge. We are calling on the whole industry to join us in developing standards of verification and validation of simulated data. Our white paper shows what we think these standards should involve – including the ability to validate simultaneously.
In short: If the industry wants simulation to be trusted by clients and the public, it needs to earn it and only widely agreed standards that include verifying and validating will achieve this.
Our simulated data: H-twins and E-twins
Our approach to simulated data revolves around two groups: H-twins and E-twins. An H-twin is a genuine human – a person with actual experiences, opinions, behaviours and evolving identities. E-twins are computational representations, capable of simulation, prediction and interaction.
The question we wanted to answer in our scientific white paper – and the only one that truly matters to our clients and the public – is how accurately the computers reflect the views of the humans. In truth, the only way to know is to ask the real person. So, we did.
Working with OriginalVoices, an AI platform specialising in simulation, we took 1,008 government ID-verified YouGov panellists and built E-twins of them. We then put the same new questions to both. In this test, 69% of the real people (the H-twins) answered within 48 hours, meaning we could check the E-twin against its human. In regular operations this validation can be done in as little as an hour.
What our study showed
Our analysis is thorough and I urge you to read it in full. However, the key findings are:
- Rich data improves accuracy. We compared E-twins built only on our core profiling variables with E-twins that we also trained on each panellist's ratings of 4,000+ individual entities (covering everything from commercial brands to media and entertainment). Using this extra dimension of data saw person-level accuracy within one point on a five-point scale increase from 78% to 90%.
- Our E-twins can accurately answer things they were never taught. In the study we held one category – fiction books – out of training. However, when asked to rate them, 90% of the time the E-twin still came within one point of the H-twin’s response.
- Confidence level tracks with accuracy. H-twins agreed with 81% of the open-ended answers provided by E-twins who were rated high confidence, 72% medium and 67% low. This gives a practical guide to when users can rely on a twin’s output and when they need to speak to real people.
- Real people identify with their twin. 75% of the time, H-twins judged their E-twin's open-ended answers as describing their own views, giving confidence for directional insights but still requiring rapid H-twin validation.
- Less stereotyping. Increasing the amount of contextual training data from Ratings reduced E-twins’ reliance on demographics. The white paper found no pattern of biased e-twin performance by age, gender, race or education.
Why our findings matter
Other promoters of digital twins are not able to show genuine validation, based on a panel that can be reliably and instantly recontacted as needed to produce human answers to the same exact questions that the twins answered.
Since access to a panel and the ability to survey and recontact it as needed are difficult to obtain, most testing shared by simulators has been done by leveraging third party or publicly available datasets, which include ready-made variables and cannot repeatedly accommodate new questions that the twins may need to answer. Leveraging publicly available data also increases the risk of leakage in creating digital twins.
There are other issues around accuracy. Comparing aggregate distributions often hides whether each twin really resembles its human. Previous studies that use these datasets have shown twins flattening towards the average and have shown demographic stereotyping among other things. One found that adding detailed personal data barely improved individual accuracy.
In contrast, we have shown our workings out and have validation built in. Our study tested twins on topics they hadn’t been trained on and against the real people they were built from – and still demonstrated high levels of accuracy.
What we will test next
Our public commitment to showing our workings means we have also shown the gaps we need to look at next.
In this study, our E-twins gave exactly the same rating as their H-twin 58% of the time. We don’t yet know how close that is to the ceiling, because we haven’t measured how often our panellists repeat their own answers. That is the next thing we will test, and we will publish the result, whatever it shows.
The challenge
The detailed testing covered in this white paper shows what we think is the right approach. We are accurate and transparent. We have shown our workings.
We challenge everybody in the industry to now work together to establish standards for verifying and validating simulated data. If the industry wants simulation to be trusted by clients and the public, it needs to earn it and only widely agreed standards will achieve this.
-----------------------
YouGov’s approach to simulation
YouGov has an established relationship with millions of real people and uses thousands of real-time collected data points to instantly simulate answers for an extremely broad range of questions, making audiences insights instantly available on demand. This unique asset enables us to create and instantly test, in parallel to their creation, the validity of our simulations.
Read more about why our simulations are better here.
