Print Story i hacked together an ugly and quick...
By gzt (Sun Nov 04, 2012 at 08:35:38 PM EST) gzt, mittens, simulations (all tags)
...simulation for TheophileEscargot. Update [2012-11-5 10:32:23 by gzt]: Now has color! I also made a non-skewed one here:

Update [2012-11-5 22:42:43 by gzt]: I added a simulation only using Rasmussen because they're a popular, big player, but they also are accused of being a little biased to the right.

It pulls directly off a site with poll data and runs of the RPubs server. I skew the polls in favor of Mittens and run a bunch of simulations. It's good fun. I show all the code because I might as well.

< Books I've Read This Year 2012 | Shocking election prediction! >
i hacked together an ugly and quick... | 29 comments (29 topical, 0 hidden)
so your simulation shows mittens wins? by the mariner (4.00 / 1) #1 Sun Nov 04, 2012 at 09:20:21 PM EST
all i need to know. 

yep by gzt (2.00 / 0) #2 Sun Nov 04, 2012 at 09:31:46 PM EST
it's got the source code and everything, you can verify it. he wins 99% of the time.

[ Parent ]
wow, i didn't know it was like that. by the mariner (2.00 / 0) #3 Sun Nov 04, 2012 at 09:44:08 PM EST
i might as well not even vote! 

[ Parent ]
I just hope by Herring (2.00 / 0) #10 Mon Nov 05, 2012 at 08:10:42 AM EST
that NASA get the funding for the probe to land on Kolob.

You can't inspire people with facts
- Small Gods

[ Parent ]
Heh by ucblockhead (4.00 / 2) #4 Sun Nov 04, 2012 at 09:50:33 PM EST
I suggest adding some google ads and sending the link to the unskewed polls guy.
[ucblockhead is] useless and subhuman
Interesting by TheophileEscargot (2.00 / 0) #5 Mon Nov 05, 2012 at 05:46:50 AM EST
First, this is in reference to the notorious Shy Tory Effect of the 1992 UK election, when the polls on average turned out to have been biased by about 8% towards the more left-wing party.

So I think what the simulation shows is that quite small systematic biases can still make drastic changes to the outcome of a state poll Monte Carlo simulation. So, state polls are not immune to this kind of effect.

But I think we need to look a bit closer at the really big election question: Is Nate Silver full of shit to be so certain Obama will win? The problem is, even if Obama wins, we don't know that he wasn't full of shit: the election might have been quite close and Silver might have just got lucky.

So let's take a step back and look at the possible reasons that might explain why national polls look close, but Nate Silver's simulation makes it look like a shoo-in for Obama. The reasons might be one or more of the following.

  1. Monte Carlo simulations of state polls might just produce apparent certainties on small swings
  2. The state polling might be missing an apparently safe state switching from Democrat to Romney this time
  3. State polling is more sensitive and can detect tiny leads that are below the thresholds of national polling
  4. A "Shy Republican" effect exists and is stronger in the swing states, so an Ohian is more shy about admitting a Romney preference than a Texan
  5. Romney has a highly disadvantageous distribution of a respectable overall vote share, but with huge majorities in his safe states and slight deficits in all the swing states
Of these, I would say only option 5 really validates Nate Silver's confidence. Option 3 even it's right this time, means his predictions are vulnerable to late shifts and systematic biases.

So what do we look for in the final results to check Nate Silver's methodology?

In an Option 2 scenario, if any safe Democrat state turns red, or shows a big swing that comes close to turning it red, it means we can't really trust this kind of state polling, and Nate Silver is wrong.

In an Option 5 scenario, if the overall vote share is close or has Romney ahead, yet Obama wins by a landslide in the electoral college, it's evidence that Nate Silver is correct and we maybe should rely more on state polls.

With other scenarios, we don't really know whether Nate Silver is correct. If Obama only just edges a win in the electoral college, then the race may have been closer than Silver thought. If Obama gets a landslide in both the electoral college and the vote share, we still won't know why the national polling didn't detect that landslide but state polls did.
It is unlikely that the good of a snail should reside in its shell: so is it likely that the good of a man should?

8 points would be huge by gzt (4.00 / 1) #6 Mon Nov 05, 2012 at 06:24:45 AM EST
Everybody is pretty sure that there can't be an 8% effect these days. An individual pollster might. But there are pollsters acknowledged by all to bias right and ones that are known to bias left these days, pretty much, and they don't differ by that much, usually, on average. 2% in certain states is certainly within the realm of possibility. Though I do admit, having looked at raw data for a bit, that the state data can be pretty noisy.

Silver does give detailed state-by-state election predictions. More to the point, there are maybe 20 contested states and he has confidence intervals for their results. A simple question is how many of the confidence intervals does he get right. He gives his guess of the popular vote with a confidence interval, was that within the interval? His results can also be compared with those of other pundits - a lot of people make predictions. There is also internal model validation - he attached a certain level of credence to some polling organizations over others, were they accurate or not? Did any states where he assigned a high probability (and most others did not) go one way or the other? Did he beat Rove?

But, judging performance is always going to be tricky here because everybody agrees that CA will go blue and AL will go red. And everybody in the reality-based community agrees that Bronco is ahead in the state polls in a bunch of swing states. The question is whether that gives him 60% odds, 70%, 80%, or what. Silver, as a pundit, has a large body of predictions. When he claims 90% confidence, is he right 90% of the time? When he claims 60%, is he right 60% of the time? The difference between him and almost every other pundit is that he puts a number on it, and that seems to get to a lot of people. People are fine with Rove saying, "Romney's got it, he'll get like 300EV." How sure is he? I don't know, you don't know, Rove doesn't know.

But I think your bringing up this question repeatedly just misses the point: "So let's take a step back and look at the possible reasons that might explain why national polls look close, but Nate Silver's simulation makes it look like a shoo-in for Obama." National polls don't matter at all because the election is counted at a state level. 2 of the last 3 elections came down to state elections. This election will come down to state elections. The popular vote might come down to how many people in CA, NY, or TX bother to vote (highly partisan and highly populated states). Frankly, if Obama gets a landslide in vote share, everybody will be confused because neither state polls nor national polls would suggest that. Nobody on the political scene who can do math is wondering why national polls look close but Silver has a shoo-in for Barry, they disagree with his confidence levels for certain states perhaps with national polls as an input for their criticism. And Silver also claims Barry is on top of the popular vote by some margin, which is also controversial.

[ Parent ]
Well by TheophileEscargot (2.00 / 0) #16 Mon Nov 05, 2012 at 12:03:38 PM EST
As you've shown very neatly, even a 2% systematic error across polls makes a huge difference to the probability outcome you get from this method. But in the last UK election "All of the companies... overstated the level of Lib Dem support, by between 2 and 5 points". Last French second-round Presidential election the poll average was about 2% different from the result. 2% systematic errors aren't unlikely. I think the reason the other pollsters don't claim to able to predict elections quoting probabilities to within 0.1% is that it isn't really possible to be that specific.

Looking back over previous elections, everybody who can do math likes to concentrate on whatever math supports what they want to believe.

If you check out the 2004 thread, everybody was debating "do undecideds break for the challenger or the incumbent". The theory was that undecideds always break for the challenger, so Kerry was bound to beat George W. Bush. I haven't checked, but probably that math was right at that point, there was probably a nice statistically significant association.

Other elections, everybody was crunching economic figures on GDP growth and unemployment rates, looking for associations that prove the candidate they wanted to win or lose would do so.

This election, it's Monte Carlo simulations of state polls. They offer a certain amount of scope for selecting the right date ranges and pollsters to help get the result you want, though maybe not enough to make it "certain" for either candidate.

But if they weren't providing the desired results, everybody "who can do math" would just be talking about some other set of statistics that did.

Unless Monte Carlo simulations of state polls happen to give the desired results in both 2012 and 2016, next time around everyone who can do math will have forgotten them as quickly as they did "breaking for the challenger". Instead they'll be lecturing me about how it's all about sunspot cycles or something: look at this 98.4% correlation...
It is unlikely that the good of a snail should reside in its shell: so is it likely that the good of a man should?

[ Parent ]
2% systematic difference by ucblockhead (2.00 / 0) #18 Mon Nov 05, 2012 at 12:16:05 PM EST
In most "safe states' (around 40 out of 50) the margins for one or another candidate are over 10%.  As such, a 2% systematic error is going to make no difference in any of these states.

You are missing the effect that the structure of elections in the US is fundamentally different than in the UK.

You'd do well to look at the current national reporting.  Very few expect extreme partisans are predicting Romney.  The Republicans are already starting to assign blame.
[ucblockhead is] useless and subhuman

[ Parent ]
UK elections by Scrymarch (2.00 / 0) #26 Tue Nov 06, 2012 at 09:41:39 AM EST
Being a parliamentary system with first past the post and major regional variation, the UK is much more prone to seat by seat swing effects than the US. The US electoral college has five hundred or so seats elected by fifty constituencies. The UK has five hundred constituencies. The only saver is that US state populations make it easier to get significant samples at lower cost.

Iambic Web Certified

[ Parent ]
the last 3 elections have been monte carlo by gzt (2.00 / 0) #19 Mon Nov 05, 2012 at 01:41:55 PM EST
though 2000 didn't really have enough state polls and things were too close in too many states. as for 2004, sure, people were debating that, but it's a factual question and it turns out to not matter - besides, the state polls were a pretty good indicator of the way they were going to break.

also, note that any 2004 analysis is done at the state level rather than the national level and for good reason.

I see that when you're looking at the UK polls and the French polls, they're both nationwide. Nothing in America is nationwide so nationwide polls and the popular vote are, quite literally, meaningless.

Polls are definitely flawed, there's a 9% response rate and polling methods are hitting a population that's certainly going to be less like the voting population each year, but I think that sort of thing still offers a much better insight than non-quant talk about "fundamentals". Quantitative talk about fundamentals might mean something. But it's definitely true that, while polls are flawed, if you're having the best polls say over and over that you're down 2% (which is what they're saying in Ohio, which is very likely to be the state that decides the election because Bronco has almost enough sure votes that OH will put him very close to 270), you have worse odds. And that's all Silver's odds say. In fact, even if you bump up all the polls by 2%, Barry might still have "better" odds. Mittens' odds rely completely on the polls being wrong in several states, which, while possible, is a low odds game.

[ Parent ]
2004 by ucblockhead (4.00 / 1) #20 Mon Nov 05, 2012 at 02:05:19 PM EST
Lots of people (including me) were in denial about the polls because of wishful thinking.  The liberal equivalent of the unskewed polls guy, though hopefully not quite so mathematically ignorant.

Obama can win without Ohio if he takes Florida.  Romney has to take both.
[ucblockhead is] useless and subhuman

[ Parent ]
very true. by the mariner (2.00 / 0) #21 Mon Nov 05, 2012 at 02:18:24 PM EST
i remember reloading three times a day with their nonsense about a .4% advantage being lean kerry or whatever. i can't stand to see that damn site to this day or even graphics sites use now that look substantially like it. 2004 was the hay day of bad liberal blogs, especially daily kos, pumping the interweb full of bullshit political analysis.

[ Parent ]
yeah by gzt (2.00 / 0) #22 Mon Nov 05, 2012 at 02:48:53 PM EST
i would definitely have gotten my stuff from a different site if other sites had a csv as convenient as their to work with.

[ Parent ]
cool NYT thing.. by infinitera (2.00 / 0) #24 Mon Nov 05, 2012 at 04:24:59 PM EST

[…] a professional layabout. Which I aspire to be, but am not yet. — CheeseburgerBrown

[ Parent ]
I was just looking at that by ucblockhead (2.00 / 0) #25 Mon Nov 05, 2012 at 04:41:31 PM EST
It'll be nice tomorrow night.
[ucblockhead is] useless and subhuman
[ Parent ]
Paddy Power pays out on Obama victory by Herring (2.00 / 0) #23 Mon Nov 05, 2012 at 03:47:42 PM EST
According to news sources and the Evening Standard

You can't inspire people with facts
- Small Gods

[ Parent ]
California by ucblockhead (2.00 / 0) #17 Mon Nov 05, 2012 at 12:06:30 PM EST
California is a deep blue state, and it is a state where nearly all the major races (President, Senate and most Congresspeople) are uncontested.  This is likely to depress turnout here.  Depressed turnout well happen across the board, so these people will all still win, but with lower vote totals.  This means that California will contribute less to the popular vote.

In a similar fashion, many populous deep-blue states were just slammed by Sandy and are facing a storm that may hit tomorrow.  This will depress the vote in those states.  Again, this will lower totals across the board, and so is unlikely to change the victor.  It will change vote totals.

This means there is very good reason to believe that Obama's popular vote totals will be significantly depressed in ways that have little or no effect on his chance of victory.

I trust Silver's electoral vote prediction much more than his popular vote prediction for these reasons.  These are factors that are hard to predict, yet don't effect popular vote totals.

I personally find "shy Republican theories" conservative wishful thinking.  We saw nothing like that in 2008 and these theories sound an awful like the "shy racist" theories that were bruted about in 2008.

Another factor some have bruted about is the cell phone effect: that because pollsters use landlines, they miss people who have cut the cord.  These people are disproportionately young and poor, and therefore more likely to vote Democratic.  I've no idea if this effect is at all real, but it seems to me more likely then a "shy Republican" effect as we did see this in 2008, though the effect was small.
[ucblockhead is] useless and subhuman

[ Parent ]
Silver actually believes that the race is close by lm (4.00 / 2) #12 Mon Nov 05, 2012 at 08:28:16 AM EST
``Is Nate Silver full of shit to be so certain Obama will win? The problem is, even if Obama wins, we don't know that he wasn't full of shit: the election might have been quite close and Silver might have just got lucky.''

Given that Silver is highly confident that the election will be close, I'm not certain where you're going with this.

It seems to me that perhaps you're conflating the confidence that Silver has in his prediction (85% that Obama will win) with the margin (50.5% to 48.5%) that he is predicting.

Also, it's interesting that most of the national polls are now falling into line with the various state polls. Given that, I do think you're right about one thing, that the facts at hand may not offer the scenario we need to performa a "crucial experiment" to see if Silver's theory is right. If the results fall exactly into line with Silver's predictions, the best we can say is that the facts did not contradict his model this cycle. But we'll be left without an event that proves his model correct where it differs from other models.

There is no more degenerate kind of state than that in which the richest are supposed to be the best.
Cicero, The Republic
[ Parent ]
I haven't seen the analysis yet, by wumpus (2.00 / 0) #27 Wed Nov 07, 2012 at 10:59:14 AM EST
but I think he was within a percent or so on every race he followed. It was about as definitive as you can expect for a claim of an "insurmountable 1% lead".


[ Parent ]
1% by gzt (2.00 / 0) #28 Wed Nov 07, 2012 at 02:29:22 PM EST
If you have a 1% lead, it's not "statistically significant", but that really does improve your odds a lot. If you have a lot of 1% leads in the few races where it counts, you get up to an 80% confidence of victory pretty quick.

[ Parent ]
I think it was really a 2-3% lead by wumpus (2.00 / 0) #29 Wed Nov 07, 2012 at 07:31:17 PM EST
that was consistent across enough of the battleground states to follow.  The point (and one that was an important post made on 538 right before the election) was that while multiple independent elections would be a dead certainty (see the Princeton prediction) there really wasn't a good reason to believe that they were independent.  Either the likely voter models were off, or one candidate appealed to those who would ignore or lie to pollsters (I canvassed one house in 2008 that claimed neutral until the woman of the house started spewing official fox derp).  I'm pretty sure that "likely voters" did not assume that voters 18-29 would vote be a higher chunk of the voters than in 2008.

But the slight lead wasn't budging, and Romney had to sweep the "tossup states".  I also suspected the race was just too close to predict that Nate Silver couldn't possibly have error bars that narrow.  Turns out I was wrong (or they've hid the mistakes on the page already).  Had 538's pro-Romney bias been instead a pro-Obama bias, Obama still would have won all but Florida (zero margin assumed) and won the election.:

                    538    Real    538    real    margin  538's

                    Obama  Obama    Romney    Romney    victory    error

Colorado            50.8    51.2    48.3    46.5    2.5    2.2

Florida             49.8    49.9    49.8    49.3    0      0.6

Iowa                51.1    52.1    47.9    46.5    3.2    2.4

New Hampshire      51.4     52.2    47.9    46.4    3.5    2.3

North Carolina      48.9    48.4    50.6    50.6    -1.7   0.5

Virginia            50.7    50.8    48.7    47.8    2       1


I'm giving up trying to get whitespace to line up in html.  Whoever came up with [code] didn't expect python, either.

[ Parent ]
My money's on reasons 4 & 5. by ammoniacal (2.00 / 0) #14 Mon Nov 05, 2012 at 10:09:54 AM EST

"To this day that was the most bullshit caesar salad I have every experienced..." - triggerfinger

[ Parent ]
Safe states by ucblockhead (2.00 / 0) #15 Mon Nov 05, 2012 at 11:57:55 AM EST
The state polling might be missing an apparently safe state switching from Democrat to Romney this time

I'm not sure how that could happen.  Most safe states have a particular candidate showing double-digit leads in all polling.  Do you really thing a poll that shows Obama up by 13% could really be so wrong as to end up seeing the state go Romney?

If this happens, it'll not cast doubt on just Silver's will cast doubt on political polling in the US in general.

Also, as others have said, Silver is specifically NOT predicting a landslide.   He's giving the chance of an Obama landslide of 0.3%.  An Obama landslide would cast major doubt on Silver's methodology; far greater doubt than a Romney victory.

For the record, Silver is currently predicting a 37 electoral vote margin and a 1.9% popular vote margin.

He also gives Obama an 86% chance of winning...roughly the chance of winning at Russian roulette.  Is that really "so certain"?  Would you play Russian roulette with these odds?

I am getting the impression that you haven't delved particularly deeply into his blog or actual predictions.
[ucblockhead is] useless and subhuman

[ Parent ]
Thanks for running your own simulation by jimgon (2.00 / 0) #7 Mon Nov 05, 2012 at 06:26:06 AM EST
I was hoping you would do that. 

Technician - "We can't even get decent physical health care. Mental health is like witchcraft here."
anyway, yeah by gzt (2.00 / 0) #8 Mon Nov 05, 2012 at 06:47:48 AM EST
so whether this is any good or not depends on whether the state polls are reliable predictors of state results. They're definitely noisy, they might be biased. If they're just mildly biased (+2 is actually pretty big to apply systemically, but maybe not on some individual states), Obama still has better than even odds based on current polling. So what subjective probability do we assign to the statement that the polls are skewed? If it's 50%, we can adjust our odds accordingly: weight the non-corrected model 50%, weight the corrected model 50%, and we'll get a probability of an Obama victory of maybe 68%. That still sounds too generous. Maybe it's 60% odds that the polls are skewed 2% in favor of Obama. Okay, that updates the probability to maybe 62% in favor of Obama. These results are all, in principle, testable...

also, as an aside... by gzt (2.00 / 0) #9 Mon Nov 05, 2012 at 06:50:43 AM EST
...using a normal rather than a uniform does give Bronco slightly better odds.

[ Parent ]
rpubs is very cool by clover kicker (2.00 / 0) #11 Mon Nov 05, 2012 at 08:25:34 AM EST
I hadn't seen that before, what a great idea.

I'll not comment on dirty foreigner elections, other then to say that I'm glad it's almost over.

it's a fantastic idea by gzt (2.00 / 0) #13 Mon Nov 05, 2012 at 09:23:59 AM EST
RStudio, the popular IDE for R, has one-click publishing to RPubs. very convenient.

[ Parent ]
i hacked together an ugly and quick... | 29 comments (29 topical, 0 hidden)