Q: Did more people get vaccinated?
A: Overall, mixed results. Also: turns out it’s harder to answer that question than you might think! You need a counterfactual.
Since the announcement of Ohio’s “Vax-a-Million” sweepstakes on May 12, many states, counties and cities in the US have followed suit as public health officials grasp at President Biden’s goal of 70% of adults being vaccinated by July 4th. Lottery prizes as large as $1.5 million were offered in a slew of different sweepstakes.
Did the lotteries work?
Before we review some of the preliminary results, let’s nerd out for a moment about how you might go about figuring out whether the lotteries worked or not. In public health and policy speak, we want to know the “impact” of the lotteries: Did the lottery cause an increase in the vaccination rate?
To answer this question, we need to know what would have happened in the absence of the lottery. In impact evaluation, the “what would have happened” part is called the counterfactual. Without a valid counterfactual, we risk comparing apples to oranges and not getting a correct estimate of the impact of our program or policy.
In the case of a state vaccine lottery, what’s the counterfactual? Let’s take Ohio as an example. To estimate the impact of the Vax-a-Million lottery on vaccine doses administered in the state, we need to know how many vaccine doses would have been administered in the same time period if there had been no lottery.
In other words, we need to observe something that is not observable, because, well….it didn’t happen. There is no vaccination data for the period May 12-June 20, 2021 in the version of Ohio in which there was no lottery. We only have data during that time window for Actual Ohio — the Ohio in which the lottery was running.
So what’s an evaluator to do? Where feasible and appropriate, she can run a randomized trial. Randomized trials make really good counterfactuals — in fact, they are considered the “gold standard” for impact evaluation. Why? Because randomly assigning some states to a “treatment” (like a vaccine lottery) and some to a “control” condition (like whatever else that state was doing to promote vaccines) guarantees that any differences you observe in vaccination rates are due to the lottery.
But randomized trials are really hard to pull off “in the wild”. In the case of the lotteries, enough states would have had to agree to be in the trial and to implement whichever condition they were assigned. Hard to imagine, right?
Instead, most localities that ran lotteries just reported the change in the vaccination rate or the number of people vaccinated from the period right before the lottery was announcement to the period following the announcement.
If you’ve read this far, you can probably see why that’s not a great counterfactual. What if rates were going down before the lottery was announced (as they were in many places)? If after the lottery announcement the rates continued to go down but not as fast, did the lottery make difference? What if rates rocketed up after the announcement but there were other things going on at the same time that might also have boosted vaccination rates?
For all those reasons, it’s best to take reported impact of vaccine lotteries with a grain of salt, armed with your knowledge of counterfactual.
That said, here are the numbers (links below to news stories reporting these results):
Ohio: 43% boost in the week following the lottery announcement, which then trailed off pretty quickly.
New Mexico: 7-day average new vaccinations increased by just 85 per day in the week following the announcement compared to prior week.
California: 22% increase in vaccinations in the week prior to the grand prizes being awarded compared the prior week.
Oregon: Vaccinations were lower the week after the lottery announcement compared to the week before. Maybe the lottery slowed the decline?
New York: 10% increase the week after a lottery was announced, but vaccination rate has since fallen 40%.
Colorado: Vaccinations declined 37% the week after the lottery announcement compared to the week before
Many other lotteries are still underway or just announced, so stay tuned for more results. And when you hear about what the lottery did or didn’t do for vaccination rates, ask yourself, “but what was the counterfactual??”
Links
Our earlier posts on the lotteries:
