How to judge an election model

Still Counting

How to judge an election model

What would it mean for FLIPR to have a bad night? Plus, more on Haley Stevens and those NYT polls in Still Counting EP9.

Nate Silver

Nov 03, 2026

211

27

9

Share

FLIPR is looking pleased with himself, but what if his forecasts turn upside-down? Getty Images.

One more newsletter for you before we publish our final FLIPR forecast shortly after midnight. If you’re new here, we do

not

usually post this much: it’s typically ~3 items a week, not three a day. But we hope you’ll join us for our final flurry of coverage, including our

Election Night livestream and live blog tomorrow

.

And we have a bonus episode of Still Counting out:

Reviews, comments,

likes

and subscribes are always greatly appreciated. Audio version here:

The context

We covered a lot of ground on the show. But I wrote about

those NYT polls

and

Haley Stevens

yesterday, so I’m going to use this as an excuse to cover another topic that Galen, Clare and I were only able to scratch the surface of. I thought it would be nice to write about this ahead of time so it doesn’t feel like spin if a lot of FLIPR’s forecasts don’t come true.

I’ll sometimes get the question, “How do you know if a probabilistic forecast is ever really

wrong

?” which is an entirely reasonable thing to ask. In fact, let me give you a four-part answer. It’s certainly not quite as simple as saying “30 percent chances are

supposed

to happen 30 percent of the time.” That’s true insofar as it goes, but it’s incomplete. And if you’re being Bayesian about it, you can’t always use this as a get-of-of-jail-free card. A 0.03 percent chance happening is much more suspect for your model than a 30 percent hitting.

First,

the canonical way to answer this. You want your probabilistic forecasts to be well-calibrated over a sufficiently large sample of true out-of-sample predictions. In other words, if you’ve forecasted 100 events that you said had a 30 percent probability of happening, and 27 of them or 34 of them actually did “hit”, your model is doing well.

The “true out-of-sample predictions” part of the answer is important. Backtesting doesn’t count. Any halfway-competently-designed model should fit the past data well; that’s the easy part. But the model may be misfit or

overfit

, keying off of

essentially random factoids

in past data that have no true predictive power and aren’t likely to repeat themselves. You may have noticed that in documenting our models, we tend to not make claims like “ELWAY would have gotten the exact score right in the Super Bowl XVI” or “FLIPR would have nailed the 1994 midterm.” Who cares. We already know what happened. I generally think it’s a negative sign when forecasters brag about their backtested results instead of explaining why they think their models make structural sense.

In the case of the Silver Bulletin/FiveThirtyEight lineage of models, however, we have something fairly unique: an 18-year-track record of making true out-of-sample predictions in advance of elections and sporting events, which were published ahead of time, and indeed have received a ton of public scrutiny. At FiveThirtyEight, we actually had an interactive that

diligently tracked our forecasts

(although the Internet Archive version of the interactive is a little hinky and Disney

nuked the original version

). Our sports forecasts were extremely well-calibrated: over a large sample size, the 30 percent chances really did hit about 30 percent of the time, and so on. Our politics forecasts were pretty well-calibrated too, although actually a bit

underconfident

: there were actually

too few upsets

relative to what our models expected. It’s hard to know whether this is a statistically significant differential

1

, but it makes me a bit less worried when FLIPR is more confident about a race than the consensus.

For full access to

FLIPR,

ELWAY

, and our other models, please consider becoming a paid subscriber.

A

second

way is a very Nate way that some of you won’t care for. Would the model have made you money if you’d been betting on it — relative to other models, the conventional wisdom as far as it can be sussed out, or betting markets? If you trusted our model in 2016, you should

actually have made a large bet on Trump

, because you could get roughly 5-1 or 6-1 odds when our model thought he’d win around 1 time in 3 instead.

We could get into a more detailed discussion about things like

Brier scores

— you not only want your model to be well-calibrated, you also want it to discriminate clearly between relatively more and relatively less likely outcomes. You’ll be calibrated perfectly if you always forecast every race as 50/50 but the model doesn’t tell you anything.

But I actually think some of the most important ways to evaluate a model are qualitative.

The

third thing

I always look at is whether a model responds proportionally to new evidence. If either the magnitude or certainly the direction of changes in our model doesn’t pass a common-sense test in accordance with the new evidence, that’s often a sign it’s poorly constructed.

For instance, when that

set of very strong NYT/Siena polls for Democrats

came out last night, their win probability in the Senate improved by about 2 percentage points in FLIPR. That seems fairly rational. Our odds for Democrats were already pretty favorable. And the NYT/Siena polls have generally been on the higher side of the averages for Dems, so FLIPR was expecting some relatively good numbers. Still, they’re a

highly rated pollster

, and their polls weren’t just good for Democrats but

great

in some states like Michigan, often showing favorable trendlines.

By the way, you can also kind of combine #2 and #3. Here, for instance, are Democrats’ Senate odds this year in FLIPR as compared to

Polymarket

:

This doesn’t always happen, but FLIPR has been leading Polymarket this year, you might notice. They’re going to wind up in roughly the same place, but FLIPR got there faster. Although I know this is self-serving, I’d argue that’s already a little bit of a “W” for FLIPR. (You could, in principle, have bought Democratic positions and sold/hedged them now to lock in a profit.)

Fourth

and finally, you should make some effort to determine whether your model was lucky or good. This is sometimes easier in sports. With the Denver Broncos under Bo Nix, for instance, there have been some cases where ELWAY won its bet, but the team really had no business winning the game.

FLIPR has various hypotheses. Just to take a random few:

The incumbency advantage matters less than it once did.

State-by-state polling errors are correlated in midterms but not as much as in presidential years.

Gubernatorial races are less partisan than Congressional ones and produce more crossover voting.

Testable evidence for all of this exists in any given set of election results. If

candidate quality matters less than FLIPR expects

, maybe Ken Paxton and Roger Marshall hold on. If it matters more, though, maybe they lose but Susan Collins wins.

For all of this, by the way, margins ought to matter and not just the final calls. We’re quite skeptical about Democrats’ chances in Louisiana, for instance, but if Jamie Davis loses by just 1 point, that’s probably a

bad

forecast and not a good one even though our “call” will be right.

Conversely, some GOP Senate wins would be much worse for FLIPR than others. If Democrats pick up three Senate seats but lose recounts in Texas and Alaska, that’s much closer to FLIPR’s central distribution than if Roy Cooper loses North Carolina.

Possible FLIPR changes and improvements

What sort of things will we be looking to evaluate and potentially change in FLIPR and our other election models? Here are some things I’ve at least

though

t about. I always think it’s helpful to keep a mental list of these ahead of time so you’re not too overly driven by results and wind up fighting the last war.

Whether to treat polling errors as being somewhat persistent, e.g., Susan Collins should get a boost because she beat her polls last time around.

As

discussed in SBSQ

, whether to apply some sort of fancier adjustment for undecided voters, i.e., undecideds are presumed to be blue in Minnesota but red in Kansas.

Whether to account for

WAR

or past election margins for non-incumbents; currently we’re only doing this for incumbents.

Whether to include sources of fundraising other than individual contributions. Currently, we only include individual contributions and not things like PAC money. The individual dollars have been very lopsided in favor of Democrats this year, while the corporate and VIP money has

gone the other way

.

Whether there’s some feasible way to

account for candidate ideology

in our “fundamentals” ratings. Currently, we only do this for incumbents, and it’s not really based on ideology

per se

but rather party loyalty in Congressional voting records.

2

Whether improvements could be made in our statewide partisanship scores, like accounting for the differences between midterm and presidential electorates, or changes in voter registration or

migration patterns

.

Whether to

remove the expert ratings layer

from FLIPR Deluxe.

Whether to force our estimate of the national House popular vote to more closely match our generic ballot numbers — currently, there’s a

difference of about 1 or 1.5 points between these

. This is intentional, not a bug, but it’s something we might examine if Democrats considerably outperform their popular vote forecast.

Whether to account for early voting somehow.

And this would be the big one if polls have a bad night: whether generally to de-emphasize polling in favor of other factors, and how that would be implemented, exactly.

But in each case, there’s at least some reason that we aren’t doing these things already. #5 and #6 would be nice, and probably helpful, but it’s not necessarily easy to find the right data, especially if we also need it for lots of past elections to help validate the model. I have a

fairly strong prior

against #1. For #4, the individual contributions data is nice to have as one of the relatively few objective non-polling inputs in FLIPR. Incorporating early-voting data (#9) would be a beast. It’s

a moving target

, and you’d need a lot of state-by-state customization; it might take months to do right.

Still, I do try to look at whether the sort of reasonable changes we

could

make but

haven’t

made systematically tend to favor one party or the other. Changes #1, #2, #4 and #10 would probably help Republicans this year. But changes #3, #5, #8 and #9 would likely help Democrats. It looks pretty even to me. I was slightly more worried when our model was more D than the consensus, but now prediction markets and other models have mostly converged toward FLIPR.

Of course, #10 is arguably as important as the other nine combined. FLIPR has actually moved

slightly in the direction

of being more fundamentals-based given the changes we made this summer. But it’s a bit truer to the polls than some of our competitors’ models, especially in estimating the national environment. However, models that make stronger assumptions about fundamentals and let them override polls are easy to overfit, and frankly have

a middling-to-poor track record

. Nor is it necessarily clear that this would help Republicans anyway, so maybe I was too quick to put #10 in their bucket. As I’ll argue in our closing story, there is lots of non-polling evidence supporting a strong year for Democrats too.

Silver Bulletin news and notes

Join us for our

Election Night coverage

!

And so you know, we

are

updating

ELWAY

, it just got bumped down the homepage with all of our politics coverage this week.

1

It’s complicated to calculate because polling errors are correlated from state to state, which isn’t true for, say, two NFL games on opposite sides of the country.

2

Other things being equal, candidates who break with their parties more often, like Collins, tend to overperform.

211

27

9

Share

Previous