I don't think Silver claims to have data or knowledge that is better than what is available to the public.
What he does different - and his isn't the only model built off of polling data, nor are 538's probabilities that different than other models - is just trying to decipher all the available polling data to reduce the noise and focus on the signal.
I also don't think he sees himself as being in the business of picking a winner, as in saying "I believe X will win tomorrow".
One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different.
Polling aggregation and models are as much "science" as the practice of polling is in the first place.
One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different.
This really isn't true, because something like 35-40 of the states aren't even remotely up for grabs and anyone intelligent could predict them after 15 mins of looking at recent polls and previous election results. So he's trying to pick winners for 10-15 states where the race is closer, but for probably half of those, it's really not that close and you'd have a pretty good chance of getting them right if you just eyeballed it. So now we're talking about 5-7 really tight swing states, and you're basically saying that predicting 44/50 states is very different than predicting 49/50. Not so sure that it is, especially given that it was a 50/50 choice for each of those states, and he did it for one election. Also, lots of pundits and bloggers made predictions in 2008; you'd expect someone to be mostly right. Fooled by Randomness and all that.
For what it's worth, I think Silver probably is right and Obama will win (though I'd prefer he didn't), but his methods do lean towards being relatively unverifiable on a short timeline.
It seems like you are confusing predictions about this election with overall model accuracy.
The state by state predictions are the primary method by which you can judge the accuracy of his model. If his model forecasts a state as 60% for one candidate, you can assess that level of accuracy by looking at all 60% predictions.
Let's take some examples from todays date on FiveThirtyEight.
* Florida 54.8% chance of Romney win
* Virginia 67.0% chance of Obama win
* Nevada 67.9% chance of Obama win
* North Carolina 79.6% chance of Romney win
* New Hampshire 80.4% chance of Obama win
* Iowa 80.7% chance of Obama win
* Nevada 88.7% chance of Obama win
Other states like Texas, Utah, Idaho, Wyoming are projected at 100% for Romney and New York, California, Oregon, and Illinois at 100% for Obama.
If any of the 100% states go to the opposite candidate, that is a model problem. If some of the ones specified with extremely high percentages 95% go against his predictions with a high margin of victory, again that is a model problem.
Finally, some of those close races should go against the models prediction.
Let's take the 7 states listed above. 5 of the 7 are projected for Obama and 2 for Romney. However there is only about a 37.68% chance of that exact distribution happening. I break it down as follows:
Obama-Romney
* 0-7 0.02%
* 1-7 0.14%
* 2-5 2.26%
* 3-4 12.14%
* 4-3 31.48%
* 5-2 37.68%
* 6-1 16.00%
* 7-0 0.28%
Each of these numbers are probabilistic statements about the likelihood of the overall event occurring based on the probabilities. They are from a single simulation of the 7 state probabilities run 10,000 times. Each of them has it's own distribution, eg 5-2 was 37.68% in the first run, 37.38% in the next, then 38.03%, then 36.85%, etc.
This is a very simple model prediction, but by taking all 50 states into account you can get a very clear assessment of how well his model is actually predicting the outcome of the election. Complicating matters is the time series nature of the predictions.
This type of model is precisely the way you get away from "Fooled by Randomness and all that". His model is clearly articulating the amount of uncertainty in the forecast.
but his methods do lean towards being relatively unverifiable on a short timeline.
Perhaps. But I think a lot of his critics fall into this trap of thinking that what Silver thinks is that he is offering some sort of revolutionary and perfect prediction, or that he thinks he is doing some sort of groundbreaking science.
He's just trying to make a prediction based on all of the available information. If election predictions are inherently "relatively unverifiable", Nate Silver isn't capable of changing that - just working with what he is given (and doing a great job of explaining all of the ins and outs of this stuff to the layman).
In fact he says a good amount of this himself. I think some of his readers impute a greater level of "scientificness" to his numbers than he himself claims. He's had many posts throughout the fall explaining where his model is based on some assumptions that could turn out to be incorrect, and key parameters fit based on relatively limited data. For example, an important one is how you translate current poll leads to likelihood of winning on election day, i.e. why does an x% lead a week before the election give you y% chance of winning? His method is to look at the empirical distribution of poll misses in the 11 elections 1968-2008, make some normality assumptions, and use that to estimate poll->results mapping, which serves as a single estimate of a whole bunch of miscellaneous sources of error (likelihood the polls are systematically biased this year, likelihood of a last-minute change, etc.). But of course that's a small number of data points, and not IID ones, either, all of which he acknowledges. All he really claims is that this model is a reasonable attempt to integrate the available data.
The headline predictions aren't his only predictions.
Silver also publishes projected margins of victory (with margins of error), and he does so for every state and every senate race, not just the battleground ones. As such, it's possible to do a fairly substantial review of his model against actual results.
Statistics are never perfect, but his predictions are far more transparent and verifiable than anything else on the market.
What he does different - and his isn't the only model built off of polling data, nor are 538's probabilities that different than other models - is just trying to decipher all the available polling data to reduce the noise and focus on the signal.
I also don't think he sees himself as being in the business of picking a winner, as in saying "I believe X will win tomorrow".
One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different.
Polling aggregation and models are as much "science" as the practice of polling is in the first place.