Huh. That almost seemed like more of a probability skills quiz. "If people assign probability X% to event A and Y% to event B, what probability do they assign to both events occurring, and is that different from (X*Y)%?" kind of thing. Be interesting to see what comes out of it.

The "supposing" questions make it more interesting than that, of course, since the events aren't independent. (If Obama wins Louisiana, I expect there's a far-better-than-even chance he also wins Ohio, for instance.)

Yes, it's clearly a psychological study gauging peoples' ability to do statistics. This is typically how these studies are done - they "fool" the subject into thinking it's about something (the election) other than what it's actually about (how people think about probability); this helps mostly erase the effects of people over-thinking whatever thing they believe the study is testing them on.

That's not to say that I didn't take the test and put my email address down - I'd love to get free money. (I did also find the "supposing" questions interesting.)

I wonder if the survey changes based on which state you live in? I had "Obama wins NC OR McCain wins TN OR McCain wins Massachusetts" (or variations of that) on several questions.

I was asked about the chances for each candidate to win these states: NC, CO, TN, MA, MI, MT and SD. Was that the choices for everyone?

Mostly questions about TN, OR, HI, MA, RI, and AZ. It does seem like they picked states that were in different parts of the country from where I now live (VA). However, I am also originally from Massachusetts, so perhaps that screwed up my results :)

It's definitely a survey of familiarity with probability, not with politics. I was asked the same question twice in reverse a couple of times - eg, probability McCain wins Arizona, probability Obama wins Arizona.

I live in NJ and was asked about AZ, MD, IN, NY, AL, and MI.

I was asked about Illinois, Florida, Iowa, Arkansas, Alabama, Minnesota and Oregon (and I live in Massachusetts). It looks like they're use a mix of safe, battleground states, and I absolutely agree that they're testing people's ability to think statistically,not their political savvy.

They used language like "Probablility that Obama wins SUPPOSING THAT McCain wins SUPPOSING THAT McCain wins

Dave @2:
This is typically how these studies are done - they "fool" the subject into thinking it's about something (the election) other than what it's actually about (how people think about probability); this helps mostly erase the effects of people over-thinking whatever thing they believe the study is testing them on.

Oh, of course. That makes sense. I didn't really start analyzing it 'til about halfway through, so I guess they'll get useful information out of me for the first fourteen or so questions anyway.

I'm a Virginian, and I had mostly Wisconsin, Ohio, NJ, Connecticut, Delaware, Rhode Island, and Louisiana. Probably not purely state-based, then. Although I expect they leave out your state (and maybe its immediate neighbors).

The only point where I had to think more than two seconds was when they asked the probability of McCain winning Ohio--I had to back up and remember (I hope accurately) what probability I'd assigned to Obama doing so a few questions earlier.

Some of those questions are pretty alternate-world. Chance of Obama winning Wyoming in an election where McCain wins Delaware: ha ha ha.

Yes, it's clearly a test about probability. I used figures from fivethirtyeight.com as my basis for calculations. I'm not sure what they'll make of some of my answers, because "Probability of X supposing McCain wins Illinois" is a divide-by-zero-type situation, so I defaulted to 50%. That is, "supposing McCain wins Illinois" requires supposing that some huge, totally unexpected bizarre event occurs in the next 6 days -- and how do you estimate any other probabilities, given that you're already in some alternate probability space?

Those "supposing" questions were weird - I mean, I gave McCain a 0% chance to win MD. Then it asked the probability of McCain winning CT, supposing he won MD. How can that be a test on statistical ability? (Is that some sneaky "null set" thing? Dammit! There's a reason I dropped out of statistics!) But, ok, in the real world, if I had to consider the case of McCain winning MD, it would have to be because something huge and insane had just happened, so I gave him a good chance of winning CT.

I don't think it maps to just math, anyway. (But, as I said, I dropped out of stat, so what do I know?)

(And I'm in VA, and got mostly MD, CT, ND, LA, OR, NJ and... probably somewhere else that I forget.)

I'm in Massachusetts, got a lot of New England, Connecticut, Ohio, West Virginia, Tennessee. I did give McCain a 5% chance of winning Massachusetts. It could happen. It's not actually impossible, just exceedingly unlikely. I cannot imagine the circumstances under which he would win, but I'm sure they exist.

I'm in New York, and got Idaho, Illinois, Delaware, Ohio, Georgia, and Alaska.

I agree that it was a statistics quiz; I just hope I didn't screw up their results too badly, because I'm pretty sure I was Doing It Wrong.

I mean, are you supposed to average out the likelihood of something wildly improbable happening (e.g., Obama winning Idaho) when in conjunction with something almost certain happening (e.g., McCain winning Alaska)?

In my brain, that works out to about 50%, but that also seems less than useful as a measuring tool, as a result...

For the Bizarro-world postulates ("Suppose McCain wins Maryland..."), I defaulted to assuming independence. It was either that, or as Doctor Science suggested, the "shoulder shrug" 50%.\

I live in MD, and was asked about MD, OR, MO, FL, IA, GA and WI, three of which are electorally interesting.

I definitely must have been doing this wrong, then; whenever I was asked what % I would assign a statement with McCain winning Massachusetts, I put it at 0%, unless it was an "OR" question. If it was an "AND" or "SUPPOSED" I couldn't see any way that could happen unless I was in the Twilight Zone, so I put it down as 0%.

It definitely doesn't avoid your own state; WASHINGTON came up several times for me.

I figured that the "SUPPOSING THAT" questions actually *weren't* independent. If McCain wins Connecticut, then somebody's got a photo of Obama eating a baby or something, and that's going to damage his chances everywhere. Then again, I've gotten into arguments with statisticians more than once over the wording of things.

I figured that the "SUPPOSING THAT" questions actually *weren't* independent. If McCain wins Connecticut, then somebody's got a photo of Obama eating a baby or something, and that's going to damage his chances everywhere.

Yeah, exactly. I answered those questions accordingly. The "Obama wins OR supposing that McCain wins AR" was independent, but the reverse, not so much.

The only problem with my answers would be if I couldn't remember the exact probability I'd given to something in an earlier question. I suppose I should have used scratch paper.

I may be waaaay overthinking this, but:

I wonder if, in addition to testing how we estimate probability, it has anything to do with how things we associate values to (in this case, colours) affect how we estimate probabilities. I found myself blinking a bit at things like "If [Red]Obama[/Red] wins [Blue]Utah[/Blue]", when my hindbrain - imprinted, no doubt, by reading fivethirtyeight.com on a daily basis - associates Obama with Blue and Utah with Red. Sentences like "If [Red]McCain[/Red] wins [Blue]Wisconsin[/Blue]" caused far less cognitive dissonance. If, on the other hand, the choice of red in the text for candidate names and blue for states is just an accidental piece of design, they may find that it winds up dropping a confounding factor into their results. But, if it is just accidental, their use of a big map decorated with the same colours below is kind of distracting, to say the least.

If my hypothesis is right, it would also give a strong reason behind the initial questions asking for both party affiliation and where you live.

I'm in Ohio, and it asked me about Maine, Maryland, Tennessee, Montana, Kansas, Florida and Alaska. I think that was about all of them.

The fact that they claim that they're going to draw from the most accurate predictions, and then asked about probabilities. That was a huge red flag for me - after the election, is anyone really going to be able to say whether the chances of Obama winning Florida was closer to 50% or 90%? All we'll be able to tell is that he *did* (or *did not*) win it, surely?

I don't know if my answers were internally consistent, but between the impossibility of what they were asking for, and the same states repeatedly coming up, I think it only took a handful of questions for it to be obvious they were questions about statistics, rather than about politics.

Oh, and at the end, they asked something like "how familiar are you with the questions we asked?". Well, I'm 100% familiar with the fact that you just asked them, but until next Tuesday I'll be 0% familar with who wins which states... So I'll just go with the same thing I put at the beginning for how familiar I am with US politics...

It asked me about OR, AR, WV, IL, WI, PA, and FL (I think that was all of them). I'm in NC. It seems pretty random.

wintermute, yeah, I wondered about that too. I have to wonder if it will really be a random drawing.

As for the "How familiar are you with the scenarios in this survey?" question, I took it to mean "How familiar are you with Electoral College politics?" It was an oddly ambiguous question.

The "thank you" email I got after I finished the questionaire referred to it as the "Princeton Judgment Aggregation Study (PJAS)", for what that's worth.

I'm in NY and was asked about NV, AL, IN, WY, ME, WV, and KY.

I'd guess that the states asked about are randomly chosen.

It does seem like a statistics questionaire. If you've said that McCain has a 75% chance of winning IN and Obama has a 100% chance of winning ME, then you should, mathematically speaking, say that there's a 75% chance of McCain winning IN AND Obama winning ME. I'm guessing the test is to see if that's what you actually say.

If it is a stat quiz, I don't think it's a well-designed one. The math would be pretty negligible, but I'm supposed to remember that I gave McCain 3% on state A and 4% on state B, ten to fifteen questions ago? Yeah, right. I have forgotten to wear a shirt to work before. I'm not paying nearly that much attention to an online survey. (Had it been on paper, this wouldn't be a problem.)

Also, the whole "supposing" thing. Is there supposed to be a "right" answer for those? Alls I remember from logic was that you can say anything you like about something that doesn't exist. (Alls I remember from statistics is fleeing in terror, but I assume they don't get to make up a calculation that involves dividing by 0.)

But I also agree that it's not about "predicting election results" - that's just silly. A candidate either wins or not, there's no "well, he won here, but in 23% of alternate universes, he lost". I thought it might have something to do with the red/blue coloring, or the fact that all the states I was asked about were not at all in doubt - I haven't noticed real battleground states on anyone's list. (Then again, I'm not 100% sure which states are battlegrounds right now, except VA, so I may be wrong on that one.) Also went through the thought that asking about certain possibilities was supposed to influence our answers on other possibilites (a poll on push-polling?) but the states lists seem too random for that.

I really don't know.

It was odd... there were several repeats of questions (and the interface is whacked, I had to redo a couple because the pointer locked to the mouse. Finally went to entering percentages from the keypad), and some which required counterfactual thinking (if Obama wins Utah... well then he might win Montana, but he's not gonna win Utah, same for McCain winning Connecticutt).

From here in KCMO, I was asked about MI, MS, LA, UT, DE, CO, GA. I tried to pick prime numbers on principle, though the "McCain wins Delaware" and "Obama wins Utah" options got 0 every time they were tied to another outcome.

Cat Meadors: I got some (actually, about 1/3rd) questsions about Missouri. Right now it's a toss-up. I think it's leaning to Obama, but not by enough to take out of the margins of error (if you aggregate the polls.

Oddly designed survey. But hey, chance at Amazon gift card; I don't care if their results are lousy, I gave my five minutes with an honest heart.

They mostly asked me about Oklahoma, Nebraska, Louisiana, Pennsylvania, Nevada and Rhode Island. I'm in Oregon.

I'm in Oregon, and the states I was asked about included Louisiana, Oklahoma, Connecticut, Arizona, North Dakota, and Nevada. I have to admit that it was difficult wrapping my mind around the possibility of McCain winning Connecticut and Obama winning North Dakota supposing that Obama won Oklahoma. You'd have to presuppose a complete flip-flop of positions.

But I agree, it's mostly about decision-making/statistical understanding. Will be interesting to see if they do send out a real explanation of the study after the election.

mjfgates @15: Then again, I've gotten into arguments with statisticians more than once over the wording of things.

No arguments, but I've had to ask to have menus clarified ("choice of potato and salad or vegetable; does that mean 'choice of potato and salad' or 'vegetable'; or does it mean 'choice of potato' and 'salad or vegetable'?"). I refuse most surveys nowdays, because you cannot quiz them back.

Statistics, I'm a naif...

I'm in MD and got questions about OH, PA, AL, AZ and I forget what else, maybe IA and CA. I found all the "what if McCain loses AZ" questions odd, because that seemed highly unlikely, but it sounds like things were semi-randomized across quiz-takers.

By the "Judgement Aggregation Study" title, I would guess that they're comparing ways to take a large number of individual predictions and compute some overall prediction in more subtle ways than just averaging everything.

By the way, my answers to the questions about "How much do you know about these issues" will change substantially depending on whether I compare myself to some hypothetical voter in the street, to my idiot Uncle Fred and his brainless neighbors, or to the Fluorosphere. :-)

I used to design and implement market research surveys... This one had nothing to do with politics... I think Cat had it right up at #10 regarding the "null set" test... All of that "supposing" stuff was just to confuse us. Since we are all voting simultaneously, it just doesn't matter how Obama does in any one state when it comes to predicting another state (unless you're predicting off of past data).

I live in Minnesota, and my questions centered on WY, MO, MT, ND and VA... Not a one about some of the more popular ones listed above. I wish we would eventually get a report about what this was really for and what info was gleaned...

Since we are all voting simultaneously, it just doesn't matter how Obama does in any one state when it comes to predicting another state (unless you're predicting off of past data).

I disagree with that. The only way for Obama to win, say, Idaho would be for something nutty to happen like McCain flipping out and shooting someone on live TV; that's going to have an impact on the rest of the states.

When I did the survey, I think I put the chance of Obama winning WV at about 30%. But supposing Obama won Idaho, I'd say it's virtually 100% that he would win WV as well, because something drastic would have happened.

Edward @30, that's supposing every voter makes their decision in a vacuum, unaffected not just by the decision other voters make but by events and reality, and that there are no demographic trends in voting. All of which are, in a word, bad assumptions.

It is difficult to imagine Obama winning South Carolina while losing North Carolina; thus, in a situation where Obama wins SC, it is because something has happened (polls there are currently about 54/41 for McCain); whatever that would be, it would certainly also boost Obama's standings in NC (currently a toss-up). Thus, the real-world answer to "supposing Obama wins SC, what are his chances of winning NC" is "pretty damn close to 100%", not "still a toss-up". You can do similar analyses for "supposing McCain wins (insert safe blue state of your choice here)"; if that happens, it's because there's been a game-changer, and all current polls are meaningless.

If they want to do an analysis of how good people are at judging conditional probabilities, they should use events that are actually uncorrelated.

Yeah, I got a question along the lines of "...supposing McCain wins New York." I had to assume that meant Obama got caught with both the live boy and the dead girl.

I'm in GA, and got questions about MA, NY, WI, OK, CA and OR

I'm in MD and got questions mainly focused on CA, OK, MN, OR and WA.

Sara

Spam from 222.191.178.155

Something seems a bit off about a random "I like your website," nothing about the site or the specific thread, from a username in a foreign character set, with a link to a random-looking URL.

Hee, #35's name appears to translate to "electric heater".

Something about the Dan Rather clip on FiveThirtyEight reminded me strongly of this thread. Rather certainly seemed desperate to come up with something that might predict a win for McCain, no matter how unlikely a scenario he had to propose to do it.

Kinda boring even for Princeton.

I went at the survey with the same viewpoint as Jen Roth @ #31 (although Matt beat me to the "dead girl and live boy" line @ #33). My questions were about MI, TN, NC, NM, & AL. I figure that, in a world where McCain would win Michigan, there would be something really dreadful tied to Obama's campaign, and thus McCain would win most states. Same with Obama winning Tennessee, though I obstinately insisted that Obama has a fair chance of winning Alaska. I'm in Texas, and got questions about New Mexico, but it's been a toss-up the past couple of elections, and is still anybody's guess.

Well, I put non-US (because I'm in England) and found it rather more challenging than I'd expected because of my inability to remember where any of the States were (Virginia is the triangular one on the right hand side, right? In which case why is it labelled VA instead of, oh, I don't know, VI or something?) And of course it wasn't till half way through that I remembered you use the colours the other way round from the rest of us, so Obama would be expected to win the blue states, and not the red ones?

Um, sorry Princeton, but I think I just broke your survey!

Wendy Bradley @ 41: In which case why is it labelled VA instead of, oh, I don't know, VI or something?

VI is the (US) Virgin Islands (these are postal codes).

You've caused me to spend a pleasant few minutes reverse-engineering the state postal abbreviation algorithm. (I could look it up, but what fun would that be?) I think it goes something like this: (1) two-word locations get their initials; (2) other states get the first two letters if they don't conflict with (1); (3) remaining states get first/last, or if another state has the same first/last letters, first plus a "distinctive" letter (AK for Alaska, AZ for Arizona).

But there are exceptions: Kansas is KS despite being the only "K" state, Georgia is similarly GA, and Texas and Tennessee are TX and TN despite there not being a TE.

Tim W #42:

I was raised in *K*entucky (KY). I must protest.

joann @ 43: D'oh! My apologies to the Bluegrass State and all who sail in her.

And add Kentucky to the list of anomalies.

TX/TN are to avoid the sorts of confusion which would arise from two states with the same first two letters being confused for each other.

I am old enough hat I still have my AP Stylebook telling me to use the actual abreviations, not postal codes, when writing copy.

That changed a few years after I stopped doing professional journalism.

Terry @ 45: TX/TN are to avoid the sorts of confusion which would arise from two states with the same first two letters being confused for each other.

But then why not do the same for Massachusetts/Maryland, Michigan/Mississippi/Minnesota, Alaska/Alabama, Nebraska/Nevada, Arkansas/Arizona...

Siome of the odd abbreviations are holdovers from earlier days: MO, KY, PA, CT, GA.

Does AL for Alabama predate Alaska's statehood?

And what could be used for Massachusetts that would *not* be confused with something else? MS is Mississippi.

It looked like an exercise in how people intuitively handle Bayesian reasoning (the statistical techniques that allow you to predict the probability of A if you know the probability of B and the probability of B given A for example). I got into and was trying to remember what I already answered so I could do back of the envelope math. After a while I gave up.

Bayesian results can be very counter-intuitive. If you know any statisticians ask them about the Monty Hall problem.