"dismissing" it I have devoted a large chunk of the last couple of years investigating it. You might like to run a search on my posts.

But I'll do a very quick summary:

I was so interested in the discrepancy at precinct level (because at precinct level you have the most statistical power) that when the Edison-Mitofsky report came out, I pored over it in minute detail to try to figure out what it was telling us. And I figured out that they probably hadn't measured it in a very fair way - it underestimated the discrepancy in extreme precincts, particularly high Dem precincts. I figured out a better way, and published a paper on the web, and posted some DKos diaries. Warren Mitofsky got to hear about it, realised I probably had hit on an important problem, re-ran some of his analyses, then hired me to run some more, which I did. Then Mark Lindeman, Rick Brady and I worked some more on the problem of how to measure the discrepancy, and I ran them all again. What I was looking for were "correlates" of discrepancy - what factors were associated with bigger differences, particularly differences favoring Bush (what Jonathan Simon, I think it was, termed "redshift"). And I found that redshift was greater under certain conditions - mostly conditions in which it would have been more difficult to get a good unbiased sample. For example where the interviewing rate was low, or where the interviewers were standing a long way from the precinct. And I found that just a few of these factors were enough to account for all the significant redshift. However, there is a fair bit of wiggle room in the data, so I also looked to see whether there was any evidence of the discrepancy being greater in precincts where Bush had done suspiciously well. And I had data on his vote share in 2000 (a year in which the exit poll discrepancy was small). So I tested to see whether the discrepancy was suspiciously large in precincts where he had increased his vote by a large amount, and perhaps smaller where he had done fairly badly. And I found absolutely no tendency for that to be the case, no matter how I sliced and diced it.. The discrepancy was simply not correlated with any apparent benefit to Bush.

So, far from "dismissing" the difference I've probably looked harder at those discrepancies than anyone else on the planet. Sure, I was lucky to have that opportunity, so I'm not bragging about it. But it certainly was precisely because I didn't "dismiss" the difference that I ended up analysing those differences in such enormous detail. And I didn't find evidence that the discrepancies were due to fraud. But I DID find evidence that they were associated with methodological factors in the poll.

So to get back to your question: I think it is likely that people reported their vote correctly (although it is of course possible that not all of them did - there are anecdotal reports of people deliberately telling interviewers the wrong answer). I also don't think that there was a very marked tendency for Bush voters to refuse more than Kerry voters, although there is some evidence that certain groups of Bush voters may have refused more. What the data suggest, strongly, is that bias crept into the poll at the level of respondent selection. Kerry voters were more likely to be selected than Bush voters. This could have been because they looked more enthusiastic, they were more likely to volunteer; or because the Bush voters avoided making eye contact; or the Bush voters were more likely to evade selection in the first place. The data can't tell us which of those it was. But it can tell us that where that where sticking to strict random sampling would have been more difficult, there was greater redshift in the poll.

It certainly doesn't rule out fraud, although it does mean that widespread massive vote-switching is unlikely. And it does suggest that the exit poll data itself is not strong evidence for (indeed it is actual evidence against) widespread vote-switching fraud. But it makes no difference, IMO, to the case for election reform; and it makes no difference to the strong evidence for voter suppression of various forms. And it makes no difference at all to the case that paperless electronic voting is insecure, non-transparent, unreliable and unauditable.

Hope this clears things up. Thanks for asking.

Lizzie

here it is:




1250 precincts, with raw data: on the vertical axis is the swing to Bush (the amount by which he increased/decreased his share of the two-party vote) and on the horizontal axis is the RAW PRECINCT LEVEL EXIT POLL DISCREPANCY measured in z scores that represent how far from zero the discrepancy was relative to what would be expected if the only error was sampling error (see TIA's posts to find out what sampling error is, if you don't know). Roughly, anything with a z score of more than 2 or less than -2 shouldn't happen in more than 95% of precincts IF the only source of error is sampling error.

As you can see, there are a fair number of precincts with z scores lower than -2 (means that Kerry did better than the exit poll suggested) but a great many more precincts with z scores higher than 2 (means that Bush did better than the exit poll suggested). From this we can conclude that there was something other than sampling error going on (could be non-sampling error, could be fraud) and that more of it was going in Bush's direction than Kerry's.

But the point about this plot is that there is NO TENDENCY for Bush to DO BETTER RELATIVE TO 2000 where he DID BETTER RELATIVE TO THE EXIT POLL.

So the "tell-tale" precinct level RAW data (no weighting, no nothing) tells the tale that whatever was causing the exit poll discrepancy didn't seem to have much effect on Bush's vote. Which rather suggests to me that that something wasn't fraud (which would).

So there you have it, PP. Tell-tale precinct raw data. And the tale it tells. If it's telling anything else (and it could be, although I've interrogated it pretty thoroughly), then perhaps you'd like to suggest what.

Pages 1, 2 and 5 reference either Lindeman, myself, or both; I have also commented on pages 3 and 5. Page 6 refers to a largely methodological paper by Michael McDonald.

Page 1.

Dopp uses quotation marks to frame a statement attributed to Lindeman and myself (footnote 1): "If there is vote fraud, then there will be a positive correlation between Bush vote increase from 2000 to 2004 and the exit poll discrepancy." This statement does not appear in the source given, I am not an author of the source, and it is not true. It is crucially not true. As Dopp correctly demonstrates, it is perfectly possible for there to be vote fraud and for there to be no positive correlation between swing and shift. The relevant question is not whether such a counter-case it is possible, but how probable it is, given the variance in each variable. That, of course, is what is tested when a correlational hypothesis is tested. Indeed it is exactly what Dopp advocates testing when she suggests that fraud is indicated by a negative correlation between WPE and Kerry's exit poll share and/or by a positive correlation between WPE and Kerry's vote-share. It's how correlations work.

My argument is therefore not invalidated by Dopp's because she simply invalidates (correctly) an argument I do not make. What I conclude (by calculating the confidence limits of the regression line) is that the maximum shared variance between swing and shift is very small. However, translating this into votes is less easy, as the amount of variance shared will depend on the variance in fraud. If fraud was uniform in extent and magnitude then there would be no variance, and thus no shared variance. However, by the same token the correlations between shift and other factors from which others have deduced fraud would also be absent or meaningless. Fraud would correlate with nothing. Moreover, one would have to ascribe virtually all variance in redshift over and above sampling variance to non-sampling error in the poll. There is no problem with doing so, but it flies in the face of most fraud arguments, including many of those advanced by Baiman and Dopp. Practically pertinent, however, is the fact that a range of voting methods are represented in the poll, rendering substantial variance inevitable. In order to estimate the maximum number of votes likely to be stolen (for a given probability threshold) we therefore have to make some assumptions as to the maximum likely proportion of corrupt precincts, and also the likely variance in fraud magnitude. I have been modelling various scenarios, and I cannot squeeze anything like popular-vote margin scale theft into the plot without making heroic assumptions about the distribution and uniformity of the fraud, and the more generous my assumptions the more all other fraud arguments (Freeman's "neglected correlations" for example) are invalidated.

Dopp states on this page that "ESI's analysis method is equally invalid as a statistical model or trend <....>". She appears misunderstand the term "statistical model". The way correlational analysis works is that a "model" is fitted to data. This model is sometimes called the "model hypothesis" to distinguish it from the "null hypothesis". If the data is explained better by the model hypothesis than by the null hypothesis (as tested, for example, by an F test), the model is supported. If it is not, the null is retained. When the null is retained, it could well be that the hypothesis is true - in other words that what we are observing may be a "counter-example", to use Dopp's phrase. This is why, when we retain the null, we compute confidence limits for the best model fit. In other words, we do not claim (ever) that the null is "true" (actually a two tailed-null is never true). What we do is to compute the largest effect that is probable (to a given degree of probability, say 95%) given the data. As the statistical power in the Ohio sample is small, the confidence intervals are wide, and could accommodate fraud without stretching probability unduly. However, as the statistical power in the nationwide sample is large, it places strong constraints on the likely prevalence and variance of fraud, as I argue above. (Elsewhere, Dopp has accused Lindeman and myself of using the term "statistical model" as some kind of retreat from the word "hypothesis" - this, clearly, is not the case. All hypotheses in statistics are tested using a "statistical model". It's how inferential statistics works).

As for the word "trend" - this term is usually reserved for a model that narrowly fails to achieve signficance. There is no "trend" for swing and shift to be correlated, either in the ESI's Ohio dataset or in the nationwide dataset.

(I should also point out that the other apparently verbatim quotation on this page, attributed to ESI: "there is no exit poll evidence of fraud..." also does not appear in the source given, although unlike the statement attributed to myself and Lindeman, it is a reasonable paraphrase of ESI's conclusion.)


Page 2.

The idea behind the measure is not to "eliminate or 'straighten out' the influence of exit-poll-partisan-response bias". I refer you to my original paper (www.geocities.com/lizzielid/WPEpaper.pdf ) and to the Lindeman, Liddle and Brady paper here: http://inside.bard.edu/~lindeman/ASApaper_... . The idea was to find a way of quantifying "bias" (whether in poll or count) that would be invariant with regard to vote-share. It was not easy to do, but I think we managed it.

What Dopp's simulations demonstrate is the nature of the problem we were trying to solve. WPE has greater variance in the centre of the plot than at the end. For a given magnitude of "bias" (in count or poll) the WPE value is smaller at the extremes than in the centre. Our suggested measure ("tau") is uniform for a given degree of bias. However, because of the asymmetric nature of the Poisson distribution for all vote-share proportions except 50%, sampling error results in a non-symmetrical distribution of both WPE and tau at the extremes of the vote-share distribution, and in the case of tau, the variance due to sampling error a) increases at the extremes and b) the expected value of the mean departs from zero (the magnitude of the deviation being a function of sample size). This was why we developed "tau prime" and recommended that a Weighted Least Squares regression technique be used to tame the remaining heteroskedasticity. It has nothing to do with eliminating "the influence of exit-poll partisan-response bias". I can only think Dopp has not read our papers. I would also note, while I am on this page, that the paper she cites in footnote 5 as being a "mathematically correct way to evaluate exit poll data" advocates, inter alia, correlating WPE with exit poll share; as any error in the poll (including sampling error) will then appear on both sides of the regression equation, absolutely nothing can be deduced from such a correlation. Far from being "mathematically correct", this is a fundamental statistical error.

Dopp also alleges on this page that "Liddle also drops the most suspicious (and most indictative of vote miscounts) precincts from her calculations as "outliers". I am not sure what she is referring to here, and she has not provided a citation. I have not dropped any outliers from any analysis, although I have certainly conducted leverage diagnostics and outlier analyses, as any competent and diligent data analyst should.

Page 3.

NEDA's equations to derive Kerry and Bush exit poll response rates unfortunately do not work, for several reasons. One of these relates, again, to the asymmetry of the Poisson distribution when events are rare (such as encountering a Kerry voter in a high Bush precinct), and it can be readily shown that using the equations given, even if overall response rates were known with accuracy (which they are not) and for each precinct (the plot merely shows aggregate values), that sampling error alone will commonly generate impossibly high response rates (>1) for the group of voters who are in the minority.

It is simply not possible to infer differential response rates from these, or any, data. However, differential participation (or "representation") rates can be computed directly from the data (tallied responses for each candidate/tallied votes for each candidate), and the ratio between them is what I termed "alpha", from which "tau" and "tau prime" are derived. (See Lindeman, Liddle and Brady for further details.)

I would agree with NEDA that differential non-response may be a relatively minor factor in producing the overall discrepancy. The exit poll data are more strongly supportive of selection bias as the key to the discrepancy, and this will be reflected in the ratio between participation (representation) rates for each set of voters, as of course will vote-miscounts favoring one candidate. However, both differential non-response and selection bias may arise from the same latent variable - a greater willingness for Kerry voters than for Bush voters to participate in the poll.

Page 4.

The confidentiality issue is far more serious than Kathy appears to realise, and "vote-counts" can't be blurred, at least by Scheuren's method. "Vote-share" can be blurred by collecting data from a large number of precincts (all, I think), banding them by vote-share, and assigning each NEP precinct to the mean of the band in which it falls. I know of no method of blurring a quantity like vote counts that would preserve the statistical properties of the data, and yet render precincts unidentifiable. The more variables provided, blurred or otherwise, the greater the chance of precincts being identified.

As for the claim that



Firstly no scientific hypothesis is ever "proven"; it is merely supported. Secondly, in general, in science, investigators do not "compete" over the same data. If a hypothesis is supported by one group, then the challenge is for the same or other groups to replicate the finding on new data. In the social sciences, data are not generally released because of confidentiality issues. That is why methods and results sections of papers are so important - other investigators need to know exactly what was done and what was found. They do not need to do the same thing again, although reviewers might legitimately demand that other analyses be carried out on the data. However, I will agree with Dopp that the E-M evaluation document was not couched as a scientific report, and lacked the kind of information, such as t tests, F tests and standard deviations, that would be required to critically evaluate the document. That was one of the reasons I wrote my original paper.

Page 5.

Regarding the swing shift correlation, see my comments to page 1. Curiously, on this page, Dopp avers that



even though at least two of the referenced papers (footnote13) plot linear trend lines between voteshare and redshift (the latter measured by WPE) and one states that "evaluating exit poll data" involves, inter alia, "Plotting the pattern of actual and significant precinct discrepancies according to their official vote share and exit poll shares" (my italics). I can only think Dopp has misunderstood Lindeman's statement, which remains perfectly valid. If a single counter example invalidates a correlational hypothesis, Dopp's method itself would be rendered invalid, as would her advocated correlation between redshift and exit poll share, were it not invalid anyway owing to the error term confound mentioned above.

I will take the opportunity to point out that Lindeman's paper does not characterise "anyone who believes that exit polls are correct as 'fundamentalists'". From his abstract:



(my bold).

which is a direct contradiction of Kathy's paraphrase.


edited for errors - I expect some remain

it may be wrong (although I wouldn't be making it if I thought it was) but it isn't silly.

But before I explain, let me say: I DO NOT TRUST THE VOTE COUNT. If I did, I wouldn't have spent over a year of my life poring over exit poll data to try and find out who won.

The fact that the election result for the leader of the world's most powerful nation can even be considered a matter for statistical inference and not a simple fact is what is wrong, what we are fight for, and, too often, about.

So I am with you all the way on that. I am also with you all the way on voter suppression, as well as forms of vote miscounting that differentially disadvantages Democrats, and, indeed, cost Gore the presidency.

So, we are left with trying to figure out from the exit polls whether Bush really won the popular vote (they are useless for figuring out who won the electoral college vote, as they don't have enough statistical power at state level, and in any case, Ohio, the closest, is likely to have cost Kerry far more in suppressed votes, which wouldn't show up in the exit poll data).

Because the exit polls can tell us something about whether the votes people thought they'd cast were counted. They are problematic, of course, because they are only a sample, and there is no guarantee that the sample was not biased. So you do need to do some clever stats on them to control for bias.

And, as you no doubt know, I actually found myself in a position to do those clever stats. And I looked extremely hard for fraud, and also for bias, and I found bias, a little evidence for vote-miscounting on older technologies (levers, punchcards) which may indicate old-fashioned differential residual vote rates disadvantaging Democrats in urban areas. However, I did not find evidence for vote-switching fraud; more importantly, I found rather powerful evidence against it.

Fraud, as you know, would be reflected in "redshift", and fraud on the scale of millions of votes would result in substantial "redshift". Pro-Kerry bias in the poll will also show up as redshift, so we need to tease them apart. And the way to do that is this:

• Redshift due to pro-Kerry bias in the poll will be unrelated to Bush's performance
  
• Redshift due to pro-Bush fraud will.

And because we can use 2000 as a reasonable baseline (redshift was considerably less than in 2000) we can test whether redshift in 2004 is correlated, or not, with "swing" towards Bush relative to 2000. In other words, if Bush's apparent popular vote improvement in 2004 over 2000 was largely due to vote-switching fraud, then we would expect to see greater redshift where that improvement was greatest, and lesser redshift, or blueshift, where his improvement was least, or negative. And this is a relatively simple statistical test.

And the answer is an emphatic no. There is no such correlation, not even a sniff. Even when I controlled for mean state WPEs, when I improved the signal to noise ratio by accounting for possible other sources of variance in redshift, and in swing, there was still no correlation. It remained stubbornly slightly, but insignificantly, below zero. Even when I looked at interactions between factors we might expect to be associated with fraud (we might, for example, postulate that fraud was only present in large precincts, or Kerry precincts) - there was still no tendency for redshift to be correlated with swing to Bush.

I then attempted to model, in various ways, distributions of vote theft, of varying magnitudes, that might even be consistent with that finding. And of course, because all data is noisy, I can't rule out some fraud, and indeed, as I said I DO NOT TRUST THE COUNT. But it is possible to put fairly severely probabilistic constraints on the kinds and extent of vote-switching fraud that are consistent with the finding, and it is small. I can't give you a precise figure, because I can smuggle various amounts of fraud into the plot depending on the assumptions I make, but sticking with some fairly generous, but nonetheless realistic parameters, there is no room for millions, and problems for hundreds of thousands, of stolen votes. And the MOST probable number, statistically speaking, is zero.

Now, you may not be convinced by this, and of course you are perfectly at liberty to disbelieve it. But it is not SILLY, and it is certainly an argument that will be taken seriously in some informed quarters. Which is why I keep saying, that if you want to make a good case for election reform, when it comes to the transparency, reliability, auditability and security of voting methods, those are your best arguments - not the case that millions of votes were stolen in 2004, because although it is mathematically possible it is very unlikely.

Which, as I keep saying, is GOOD NEWS, because it means the mountain Democrats face is climbable.

As long as you fix voter suppression, which IMO, probably cost Kerry more votes than any other form of disenfranchisement in 2004. And as long as you fix the voting machinery so that not only is it much more difficult to steal votes with, but so that TRUST IN THE COUNT is restored, because without that you don't have a democracy anyway.

It seems that you don't want to read the FAQ.

OK, here is a summary of their summary:

The polls are designed to do two things:

1. Produce "tabulations" of who voted for whom and why.
   
2. Make "projections" of the result.

Note that providing an independent check on the integrity of the election does not appear in that short list.

So how do they achieve these two purposes? They conduct a survey of voters leaving the precincts, and telephone surveys of absentee and early voters, during which they ask voters to complete a detailed questionnaire. From these answers they produce tabulations of who voted for whom and why. However, the accuracy of those tabulations will depend on getting an unbiased sample, which is very difficult, and a problem for all surveys. However, because Edison-Mitofsky, possibly foolishly, assume that America is a civilised democracy and not a banana republic, they have an additional tool at their disposal, which is a cross-check on the validity of their sample - the presidential count itself. Assuming this to be correct, they weight the cross-tabulations to their the presidential vote-count, giving them not only an extremely rich set of survey data on what issues influenced the election, but, assuming the vote count is correct, an unusually accurate one.

As for their second purpose - again, their assumption is that the vote-count is correct. Their purpose is to "project" it, in advance of the final numbers. They make their projections initially from their survey data (exit polls, absentee/early voter polls) cross-checked against pre-election polls (which, inter-alia, give them early warning of unusual bias in their sample), and then, as the vote-returns come in, this data-stream is incorporated into the projections. Essentially, as each precinct returns its vote, the estimate of the vote count in that precinct is replaced by the counted vote for that precinct; and as county tabulations begin to complete the picture for each state, these too contribute to the projections. The projections thus converge on the count (surely you've watched Peter Snow's "swingometer" in the UK? - it's exactly the same principle). If confidence in the "projection" reaches a critical level (determined by a statistical test) the networks may decide to "call" the state on the basis of the projection, even if the vote-count is not yet complete. For close run states this is unlikely to happen until the vast majority of the vote-returns are in.

So, you say, (or should say, because this is the right question), but what if we can't assume the vote count is correct? Indeed, which is why, if we have reason to suspect that the vote count is NOT correct, the estimates made by the pollsters at close of polls ("Call 3") are useful, as is the pollster's analysis as to which part of their process (selection of precincts; selection of voters; weighting of samples) might have led to the discrepancy, and, as they nailed the discrepancy to "within precinct error" (WPE) then we have evidence that while it could have been a voter selection problem (and they had good reason to think that it was), it could also have been fraud.

This is why so much of the debate over the exit poll discrepancy has centred on "within precinct error" data.

So there you have it. US Exit polls are not designed to check on "the integrity of the election" - the polls are a survey of voters, designed to incorporate the vote-count as a check on the poll. This is because all surveys are prone to bias, and exit polls, uniquely amongst surveys, come with a built in check on the bias. Provided you can rely on the count.

If you want, post hoc, to use the exit poll as a check on the count, you have to reverse-engineer it to do so, and retro-fits are never easy. However, analysis of the discrepancies at precinct level indicates that discrepancy in the poll was not correlated with benefit to Bush, suggesting that indeed the count was more accurate than the poll, not the other way round.

But no amount of analysis of exit poll data will tell you anything about voter suppression, which could, conceivably, have cost Kerry Ohio, and is, in any case, unjust.


edited for typo

I think that's a little unjust. We are bothtrying to be scrupulously fair about this. We are not acting for the defence. We would both, I think, prefer to be acting for the prosecution. Think of us as the poor guy in your office who has to probe the weaknesses in your client's case, not the guy you actually face in court.

If I had to defend the case that the exit polls proved that there was no fraud (bear with me, I do not believe this) the argument is very simple.




Well, you get my drift. I'm not very good at this. But what I am trying to say is that the latest analysis gives your opponent a rather better argument against fraud rather than it gives you an argument for it. Stack that up with copious academic works on the problems of non-response bias and measurement error in survey data, and I think you have just got yourself a proportionally spaced-font memo.

Use it if you want. But to me, it's worth checking out that memo pretty thoroughly and so far, to me, it looks like a fake.

So, to turn to your earlier point, where I think you are implying that these things should not be publicly rehearsed - well, you may be right. But we don't have much option. It is on forums like this that we have an opportunity to probe the case for holes. Several posters have suggested ways in which Mitofsky's alibi for the polls could be busted (and to give myself credit, I've been fairly active in soliciting them and trying to work them through logically - it's a dirty job, but someone's got to do it). But I have to say, I'm not convinced yet, or anything close. Mitofsky's evidence certainly suggests that something OTHER than fraud was responsible for a major part of the exit poll discrepancy. And once that's conceded, the case looks pretty thin.

On the other hand (boy, this is catching...) if we bite the bullet, and say, well, perhaps fraud was NOT the cause of the exit poll discrepancy, THEN perhaps we can develop hypotheses, given that we still have every reason to believe it was a filthy election, that will lead us to evidence that WILL stand up in court.

(Edited to remove snarky comment. Sorry, I'm still a bit cross.)

I don't think it is in the exit polls.

Fraud, I mean. Let me explain in one post what I have been trying to explain on many.

As most of you will know, I have been an exit poll skeptic for some time. My background, though in all sorts of odd things, is largely in social sciences, so while I was highly suspicious of Bush's apparent win (and devastated) last November, and seriously wondered whether the exit polls were an indication of massive fraud (especially in the light of the Ukraine story that followed close in its heels) I did not share the assumption of many DUers that the discrepancy could not have been due to polling bias. I am not a statistician as such (i.e. it is not my primary speciality) but I have had a rigorous statistical training, and use statistics heavily, daily. I also teach statistics at university level. And I know enough about statistics to know that the "margin of error" refers to "sampling error" - in other words, it is the margin of error that could have occurred simply by chance. If I measure my son's height today, and again tomorrow, I may find he's grown. And he sure is growing. But because he wriggles a bit, I have to allow for random wriggles. A jump beyond the margin of wriggles will mean he's grown. A jump (or a drop) inside the margin of wriggles will just indicate he's wriggling.

OK. So I knew that the discrepancy between poll and count wasn't chance. So what was it? It could have been fraud. It could have been bias in the poll - i.e. the poll might have had a biased sample. And I was perfectly prepared to believe in sampling bias. I meet it every day. You can do what you can to minimize it, but you can't avoid it. So when I read the E-M report in January, and saw that apparently the discrepancy WAS at precinct level (not because of poor selection of precincts) I realised the only alternatives were: fraud; or biased sampling of voters. And although the report was frustrating to read (short on geeky bits like standard errors, F values, probability estimates, degrees of freedom) the apparent findings that "redshift" was associated with precinct characteristics likely to make adherence to strict random sampling protocol difficult, I was convinced that sampling bias probably played a role. However, nothing in the E-M report indicated how much, except the assertion that the evaluation had determined that "non-response bias" (which CAN mean that one group of voters was more likely to agree to participate than the other, but can also mean that one group of voters was more likely to selected than the other) accounted for the discrepancy.

So far, so not very good. Unsatisfactory in fact. In order to establish whether non-response bias was sufficient to account for the exit poll discrepancy a number of analyses need to be done (and though I would like to say I'd thought of all these months ago, I didn't and they've been growing on me slowly - however, the first was obvious).

The first would be a proper test of the polling bias hypothesis. A multiple regression analysis needed to be done, in which all the precinct/interviewer characteristics hypothesised to be contributors to the discrepancy were entered into the same model, ie. not a series of separate analyses where WPE in one kind of precinct is compared with WPE in another. What needs to be done is an analysis that takes into account the fact that in some precincts several of these factors may be present together, and may even interact. If such a model could be shown to account for the discrepancy - or if, to put it differently, after accounting for factors likely to give rise to polling bias, there was still a residual discrepancy, then one might deduce fraud. And actually, even if they did account for the full discrepancy, one might also wonder whether some of the variables were proxies for fraud variables. So that wouldn't be conclusive, but it would be of interest. In my paper, a few months back, I called on this kind of analysis to be done.


But we also neeed fraud hypotheses to test. I am aware of two, both important.

1. This was a test of a hypothesis originally formulated by USCV (aka NEDA) as the "Bush Strongholds have more Vote-Count Corruption" hypothesis. This in itself seemed an odd hypothesis (why, a priori, would we expect more corruption in Bush precinct strongholds?). However, a later formulation expressed by USCV, and suggested to me by Josh Mitteldorf, was that if fraud was responsible for the exit poll discrepancy, you would expect a "bunching" of precincts with highly discrepant poll results at the "high Bush" end of the spectrum. In other words, precincts that ought, if the vote count had been honest, to have been in the centre of the spectrum where the distribution of precincts is fattest, would have been shifted Bush-wards by fraud. As the distribution of precincts by Bush's vote share is roughly bell-shaped (actually there are more moderately Bush precincts than moderately Kerry precincts, but more extreme Kerry precincts than extreme Bush precincts) then if fraud was randomly distributed across the spectrum - a thickish swarm of precincts from the middle of the plot should move to the right (literally and metaphorically) and give rise to a positive correlation between discrepancy in the poll and Bush's share of the vote.

Unfortunately this test was complicated by the fact that the traditional measure of exit poll discrepancy (WPE)does weird things in relation to the way the vote count is distributed. I devised a measure that I think does a better job - and Mitofsky performed the correlation. The hypothesis was not supported - there was no linear tendency for the discrepancy to be greater at the Bush end of the plot.

Plots can be viewed at the links in this DKos diary by HudsonValleyMark (Mark Lindeman):

http://www.dailykos.com/story/2005/5/24/21...

We cannot, however, conclude from this null finding that fraud was not responsible for the total discrepancy. Simply that it does not seem to have been randomly distributed through the precincts. Maybe it was concentrated at the Kerry end.

2. So here is a second hypothesis, this time formuated by ESI for Ohio. This hypothesis says: if fraud was responsible for the exit poll discrepancy, as well as for Bush's apparent increase in support in the election (the presumed purpose of fraud) then precincts with greater "redshift" in the poll ought to have a greater shift to Bush in the count. In other words the two effects should be correlated. but how do we measure Bush's gain? One way of doing it is to baseline it from 2000, a year in which the exit polls were relatively accurate (and therefore a year in which fraud, if it occurred, seems to have played relatively minor role in Bush's vote count - after all, Gore won).

ESI performed the correlation for Ohio, and found no association between what Brits call "swing" to Bush and redshift in the poll. But the trouble is that states, in exit poll terms, are small, and therefore do not give you a lot of statistical power. And to demonstrate a null you need A LOT of statistical power. In fact you can never demonstrate a null. What you can do instead is to demonstrate that if there was an "effect" it was, to a given degree of probability, less than a certain size. And the power in the Ohio study would have left fairly wide confidence limits for the "true" association between redshift and swing to Bush. (We have yet to see a proper peer-reviewed report of the ESI study - it is apparently in the pipeline).

So Mitofsky repeated the ESI analysis on the entire dataset, and it was the results of this analysis that he presented at the debate with Steve Freeman in Philadelphia last week. He presented it in the form of two scatter plots, which are posted here, by Mark Lindeman, with some informative text. If you click on the plots, you can examine them more closely.

What they indicate is that there is no discernable association between redshift and Bush's performance relative to 2000 (remember, a year in which the exit poll was fairly accurate, although there was a small net red-shift). Confidence limits are not given, but geeks among you can ballpark it given the precinct N which is 1250. The limits are fairly tight.

So what's with the Cat in the Hat?

I DON'T want to prove that Bush won a fair election. I WANT to prove that he didn't. But I think it is very hard, given that plot, to see view fraud as the explanation for the exit poll discrepancy. But we don't NEED to demonstrate that fraud was the cause of the exit poll discrepancy. We need to demonstrate that fraud was the cause of Bush's victory. Actually, as far as I'm concerned, we don't need to demonstrate even that - what we need to do is to demonstrate that he did not win a fair race. And he didn't. The race was unfair from beginning to end, from the moment he stepped into his Poppy's size nines, to the attack ads, to the lies about WMD, to the felon purges, to the voter suppression tactics, to the rationing of voting machines to Democratic precincts in key states, to the monkeying about with regulations on voter challenges, to the monkeying about with provisional ballot regulations, to the refusal to expedite a recount in Ohio, to the media mockery of those who doubted as "tinfoilers" (a new word to me) - and maybe to the electronic corruption of the actual vote, made absurdly possible by the absurdly insecure software installed on the voting machines.

But if we want to find that certain something - the smoking gun, the evidence that Bush, far from spreading freedom and democracy is the president of a democracy only in name - we need to FIND OUT WHERE IT'S NOT.

And if it's in the exit polls, it's bloody well hidden.


On edit: link added, and I should also say that my name is Elizabeth Liddle, for those who don't know, aka Lizzie.

it's a bit long, and it's taken me a while, but FWIW:

First the good news:

I do think Steve makes a good case for investigation. I agree with him that Ohio stinks, voter suppression is rampant, whether structural or intentional (and it is hard not to believe intentional), and that the exit poll discrepancy cannot be due to chance. Given the first two, it is reasonable to suspect that the exit poll discrepancy may have been due to fraud.

And I like his term "Precinct Level Discrepancy"(PLD) “Within Precinct Error” (WPE) has become synonymous with a very particular measure of PLD, which I think Steve agrees is not the best, although inevitably it is the one that will feature in his analyses as it is the one used in the E-M report. And it is important as it appears that precinct level was where the red-shift occurred.

So, thus far I’m with Steve. The case for investigation is good, and PLD is where the bodies appear to be buried.

Steve also presents some interesting observations. Although state level aggregates are difficult to interpret, it is of interest that completion rates show a bit of a tendency to be higher in redder states, and that PLD appears to be higher in states with higher African American populations. (I identified a possibly related pattern which is that WPE was higher in bluer states). State level is too coarse a grain with which to investigate potential causal factors in these relationships, but they are, nonetheless, interesting, and worthy of further investigation.

Steve also says in one slide that the only two causes of PLD are "non-response bias" and "count corruption". I agree with this as long as you take a fairly broad definition of "non-response bias" to include actual sampling bias i.e. people who do not respond because they are not even approached. There is evidence that the actual sampling may have been biased (PLD was greater where sampling rate was low). This point is important in interpreting some of his other observations.

Because here is where Steve and I start to part company.

He then goes on to say that interviewer effects cannot account for all the discrepancy - although he then says that it cannot be ascertained without looking at the data. This latter thing is true. You need a multiple regression model to determine whether interviewer effects can or can't account for all the PLD. But from what is publicly available we do not know whether it can or can't. And it is also true that some polling factors may be collinear with fraud. So although I accept that with the data publicly available we cannot be assured that interviewer effects account for all the discrepancy – without the data we can also not be certain that they do not. There is a difference between saying that “interviewer effects cannot account for all the discrepancy” and saying that “on the data given, we cannot know whether effects account for all the discrepancy”.

The completion rate by precinct partisanship argument is of interest, but IMO not at all conclusive. At the risk of sounding like an apologist for the exit polls, there are many reasons why the completion rate patterns are perfectly consistent with non-response bias as an explanation for the exit poll discrepancy (particularly if this is taken to include sampling bias). Firstly we know, from a scatterplot presented by Mitofsky at AAPOR (and again at this debate) that variance in completion rates was vast, and neither negatively nor positively correlated with Bush’s share of the vote. This means that staring at aggregate values tells you very little about whether those aggregates are meaningfully different. Secondly, total completion rate may vary independently from differential completion rate. For example, voters of from both groups may be more likely to respond in, say suburban than urban precincts; however, for a given total completion rate, there may nonetheless be a tendency for Bush voters to respond less. Thirdly, non-response bias may operate at the level of voter selection rather than voter refusal. If Bush voters were less likely to be selected, but equally likely to participate IF selected, completion rate would not vary by precinct partisanship; red-shift would nonetheless be manifest in the PLD. And the fact that PLD was greater where interviewing rate was lower (and therefore where there was more opportunity for non-random sampling) strongly suggests that bias in voter selection was a factor in producing redshift.

Steve presents a plot in support of his claim of “implausible” Kerry response rates which is simply wrong. I believe it was produced by Ron Baiman for UScounts votes, and plots a calculated value for Kerry and Bush response rates based on a formula that can be readily shown to give “impossible” values for response rates for voters for the minority candidate under conditions of sampling error alone. A full rebuttal of this plot is way to geeky for this post, but if anyone’s interested I can demonstrate that Ron’s formula can easily generate computed “response rates” for voters for one candidate that exceed 100%, even if the only source of error is sampling error. And it is not possible to conduct a poll without sampling error.

And now we come to the apparent rise in PLD in “high Bush” precincts. This is, I believe, an artefact of the WPE as a measure, and of the instability of aggregate measures where Ns are relatively small. I believe that Steve agrees with me that the measure I proposed, and called ln(alpha) is a better measure. However, even when WPE is used as the measure of PLD, Steve’s statement that “there’s no PLD at all in the Kerry strongholds” is simply untrue – the mean PLD may be zero in the Kerry strongholds, but there is plenty of it. Moreover if ln(alpha) is used as the measure, it is clear that the only sense in which PLD is”lower” in Kerry strongholds than in Bush strongholds is that in the high Bush category (incidentally, not “quintile” – which would imply equal numbers in each category) is that there is a shortage of extremely blue shifted precincts – and in fact one very blue-shifted precinct only escapes the arbitrary “high Bush” category line by a single percent. And in fact, as Mitofsky demonstrated at the Miami AAPOR meeting in May, there is no significant tendency for PLD to be higher at the Bush end of the spectrum than at the Kerry end. The correlation coefficient between ln(alpha) and Bush’s vote-share is insignificantly different from zero.

A few more miscellaneous comments:

The swing state thing is interesting. Could be evidence of fraud. Could be a spatial manifestation of the temporal phenomenon suggested in the E-M report whereby red-shift appears to be greater in years in which interest in the election is high.

Voting technology. I simply don’t think anything can be drawn inferred from this. As Steve draws attention to, the number of urban precincts with paper ballots is absurdly small – too small to do any statistics on. As with many of these analyses, it’s kind of interesting, but you need to know what is collinear with what else to draw any conclusions.

Reported Vote: Steve seems to have cottoned on to TIA’s case. My response is the same as my response to TIA’s. I won’t repeat it here, except to say that it strikes me as being in no sense a slam dunk for fraud, despite popular belief.

Confidentiality. I simply disagree with Steve that the data should be publicly released. Quite apart from being a violation of the ethical guidelines of the professional organisation of pollsters, it is, simply, unethical to make data public in a form in which respondents could be identified – and here we are talking about the sanctity of the secret ballot. And I also believe that E-M owe their interviewers (casual employees) a duty of care. If precincts can be identified, so could interviewers. The demographics of both responders and interviewers are important variables. It is simply wrong to allow these variables to be released in a way that make either identifiable. “Blurred” data on Ohio was released. Perhaps other sets could be similarly prepared and released. But confidentiality matters. I would be dismissed from my own job if I published data in a form in which my respondents could be identified. You just can’t do it.

So to summarise my response to Steve’s talk, for anyone who is interested, and doesn’t think I’m a freeper:

Freeman and Mitofsky agree that there are only two possible causes of the Great Exit Poll Discrepancy: Fraud and Non-response Bias (and I’ll qualify that by including selection bias in Non-response bias).

So whodunnit? Fraud or Non-response Bias:

Steve, as a good prosecuting counsel builds a nice case for Fraud as the killer of the Exit Poll. His evidence that a crime was committed is sound. His evidence that Fraud was the culprit is suggestive. Some is flawed. Some is wrong. But regardless of which is which, it is built into a solid-looking edifice. Things look bad for Fraud.

But even solid looking cases can come apart with a watertight alibi. And I think Mitofsky as defence counsel presented the alibi for Fraud.

Here it is:

If Fraud was responsible for the Great Exit Poll Discrepancy, then where the PLD is greatest, the boost to Bush’s vote should be greatest. After all, the point of fraud is to win the election. Cui bono?

If Non-response Bias was responsible for the Great Exit Poll Discrepancy, PLD should vary independently of Bush’s advantage in the vote count.

Mitofsky correlated a measure of PLD with a measure of advantage to Bush, namely the increase in Bush’s share of the vote relative to 2000 (a year in which mean PLD was near zero).

And there was no relationship. PLD is not correlated with boost to Bush’s vote share.

So, carefully as Steve’s case was built, I simply don’t think it stands.

Of course Fraud could still have stolen the election. But it looks to me that with regard to the Great Exit Poll Discrepancy, he’s innocent.



(edited for grammar and typo - I'm sure there are more)