Philip B. Stark, 4 December 2010

I prepared the following testimony for a hearing before Judge Ira Warshawsky (New York Supreme Court) on 4 December 2010. Some portion was read into the record, but not the entire statement. I have since discovered that I overlooked one error that the audit had found, which favored Mr. Johnson; I have corrected this statement to take that error into account.

Judge Warshawsky denied the request to expand the audit or have a full hand count. The plaintiff appealed; the appellate court upheld Judge Warshawsky's decision, 4–0. The Court of Appeals upheld the decision as well, with an unsound statistical argument.

The 3% audit gives very little statistical confidence that a full hand count of the ballots would show Mr. Martins to be the winner. The audit results would not be surprising even if a full hand count would show Mr. Johnson to be the winner.

Three of seven audited machines had errors: roughly 43%^{[1]}.
Net, the errors favored Mr. Martins: correcting them decreases the apparent margin.

Because the audit examined only 7 machines, there is a substantial possibility that the machine with the largest error was not one of the machines that was audited. Indeed, there's a 97% chance that auditing 7 of 249 machines won't check the machine with the largest error.

An average of less than two errors per machine could account for the apparent margin of about 450 votes. An average of one error per 200 ballots could account for the apparent margin.

In my experience, this level of error in precinct-count optically scanned ballots would not be surprising. And it is consistent with the errors the audit did find, within the statistical variability expected from "the luck of the draw."

The potential for error in this contest is large: In total, the 242 unaudited machines
could hold enough error to account for the apparent margin 186 times over.
Sixty-six of the 242 unaudited machines could *individually* hold enough error to account for
the apparent margin.

There is a substantial chance that a 3% or 8% audit would find little or no error even if Mr. Johnson is the true winner.

For instance, if 30 of the 249 machines have errors of 15 votes or more--enough to account for the apparent margin--the chance that the 3% audit would have found any of those machines is under 60%.

If 20 of the 249 machines have errors of 23 votes or more--enough to account for the apparent margin--the chance that the 3% audit would have found any of those machines is under 45%.

If 20 of the 242 unaudited machines have errors of 23 votes or more (enough to account for the apparent margin) and an additional 5% of the machines are audited, the chance that the additional audit would find any of those 20 is under 69%.

Because the margin is so small compared to the possible errors, a very large percentage of machines needs to be audited to give strong evidence that Mr. Martins is indeed the winner. 3% is not sufficient. 8% is not sufficient. To have 90% statistical confidence that Mr. Martins won requires auditing a minimum of 90% of the machines selected randomly: an additional 218 machines.

This is true if the audit finds that those 218 machines have counted perfectly. If the audit of those 218 machines found many errors, still more machines would have to be audited.

[1] One of the seven audited machines had two errors that favored Mr. Martins, one had one error that favored Mr. Martins, and one had one error that favored Mr. Johnson.

Here are some links to news reports:

- http://portwashington.patch.com/articles/supreme-court-certifies-7th-senate-race-for-martins-2
- http://www.newsday.com/long-island/nassau/it-s-official-judge-declares-martins-victor-over-johnson-1.2516962
- http://online.wsj.com/article/APdcf6dc2872e7425aaa027804b561aabe.html
- http://www.nytimes.com/2010/12/05/nyregion/05elect.html
- http://www.newsday.com/columnists/other-columnists/appellate-judges-deny-johnson-martin-recount-1.2545833?qr=1
- http://www.nycourts.gov/ctapps/decisions/2010/dec10/237PC10.pdf
- http://www.bolipari.com/boblog/2010/12/paper-ballots-ny-courts-dont-get-it/
- http://www.newsday.com/opinion/oped/paper-ballots-offer-certainty-1.2556022
- http://www.nycourts.gov/ctapps/arguments/2010/Dec10/Dec10_OA.htm

P.B. Stark. Last modified 9 March 2011.