Logic and Accuracy Testing of DREs in Santa Clara County

On October 21, 2004, seven members of the Stanford Freshman Introductory Seminar Digital Dilemmas and Professors Armando Fox and David Dill visited the office of the Santa Clara County Registrar of Voters to watch the Logic and Accuracy Testing (LAT) of the county's electronic voting machines. We hoped to learn more about how electronic voting machines, also known as direct-recording electronics, or DREs, work. We were interested in the pre-election testing procedures in their connection with the security of such systems. The following documents the testing procedures as we observed them.

There is another photo essay on setup, election opening, voting, and election closing on election day.

Warehouse Security

Access to the 'Cage' where the DREs are stored is restricted
Security cameras keep tabs on much of the warehouse
Security systems keep track of who has swiped into and out of the DRE testing area

When we first entered the election office, we were required to sign in as visitors and wear name tags, which served as passes for all election observers. We were then escorted by a team of three election officials, all of whom stayed with us for the remainder of our visit in the secure areas. To enter the warehouse where the DREs were stored and tested, the officials escorted us through a door locked by a keypad. As we signed in a second time, we were confronted by an enormous cage enclosing the area where the most of the Pre-LAT testing took place. Upon closer inspection, we noticed numerous security cameras, which, we were informed, by which people were constantly monitoring the premises. Finally, the project manager led us into an inner room of the warehouse that was used as the troubleshooting room, scanning her badge to unlock the door. She explained that her badge, unlike those of our other escorts, was the only one that the swipe-card scanner, which served as a lock, would admit; only certified employees were allowed in the troubleshooting room.


There are two types of workers staffing the DRE storage and testing operation: permanent employees and seasonal workers. The warehouse employs six permanent workers who are certified by Sequoia to troubleshoot the machines. However, many of the approximately forty-five seasonal workers return year after year; around 90% of them have been working with DREs since their inception. These seasonal workers often hear about the job by word of mouth or through friends. During the weeks before the election, all employees may work up to twelve hours per day in order to test the daily quota of machines. Unlike workers in other counties, these employees will test all of Santa Clara's approximately 5500 machines—up to 610 machines per day.

The Sequoia AVC Edge


The Sequoia AVC Edge

Santa Clara County purchased the Sequoia AVC Edge machines as part of a contract with Sequoia Voting Systems. Although the machines are priced at $3500 each, the County received a discount for buying such a large quantity. As part of the County's contract with Sequoia, it will receive a printer for each DRE once the California Secretary of State's office certifies the printers. These printers print out a paper version of what is stored in the machine's memory to create a voter verifiable paper trail that makes meaningful recounts possible by allowing election officials to detect a discrepancy between the paper trail and the electronic totals. If a vote is initially recorded incorrectly by a DRE and there is no such paper trail, it will be impossible to detect that the cartridge recorded the vote inaccurately.



DREs with cartridges loaded One of several laptops used to burn the ballot images onto cartridges
Closeup of cartridges loaded into the DRE Bundles of burned result cartridges identified by precinct number. Each cartridge also has a unique barcode to identify it.



Each ballot cast on a DRE is stored on a cartridge inserted in the back of the machine. It should be noted here that there are two cartridges: one that uploads the ballot onto the DRE and one that records the results on Election Day.

Testing Procedures


Testing menu
Instructions to employees using laptops to prepare result cartridges
The 'Protective Counter' lists how many votes have been cast on the machine in it's entire lifetime
Schedule for testing DREs
DREs being tested
Buttons which switch the machine between voting and testing mode are secured under caps

To begin the Logic and Accuracy Test, cartridges are inserted into the back of the DREs in order to load the ballots and the voter simulation scripts onto the machine, a process that takes between fifteen and twenty minutes. A check is run to make sure the software is properly installed and has been updated from version 4.2 to 4.2a. At this time, the Zero Proof Screen pops up to report that no votes have been cast prior to the opening of the polls. During testing, the machines' protective counter, a feature that records the total number of votes that have ever cast on it, is checked, and the time and date are set. Then, upon the insertion of an activation card, which will be discussed later, the machine runs a practice vote through automated scripts. These scripts go through a pattern of votes: one for the first candidate, two for the second, and three for the third. This makes it easy to see if votes are going to the wrong candidate because of a faulty machine. Though in Logic and Accuracy Tests for previous elections, fifty-five votes were simulated on each machine, now only thirteen ballots are mock-cast to speed up the process. Additionally, a manual test is performed to make sure that the screens work properly, ensuring that voters' ballots are cast for their intended candidates. The DREs are in test mode for all pre-LAT testing. Employees record the results of each test on a publicly-available paper record that stays with the DRE through the entire life of the machine. This record also indicates any machine failures the DRE experiences during an election.

According to a certified employee, five to ten percent of the machines in Santa Clara County fail the Logic and Accuracy Test. While most fail because of human error such as an incorrectly programmed ballot cartridge, 1-2% of the machines fail because of observable hardware failure. This failure is recorded on the Pre-LAT Inspection Form, and these machines are taken to the troubleshooting room, where certified personnel attempt to diagnose the problem and will replace components of the machine if necessary. Subsequent testing will be done to determine if the machine has been successfully repaired, and no machine will leave the warehouse without passing the audit.

Employees then prepare the machines that have passed the audit to be delivered to the precincts. Cartridges are labeled with barcodes that match each machine's individual barcode so that each cartridge can be traced to a specific machine. Tamper-proof seals that also have barcodes and serial numbers that match them with their respective machines are used to provide pre-election security. These seals display "VOID" on their tape if they are removed or disturbed. Employees place tamper-proof seals on each machine after recording the corresponding machine and seal barcodes. One seal is placed over the open/close polls door, another goes over the results cartridge, and a third is sealed inside the open/close polls door for later use on Election Day. The machines are then unplugged and stored in the warehouse until they are placed on carts to be delivered to each precinct. Each cart is pre-labeled with specific delivery and contact information as well as the number of the cartridges included. After the testing, the machines remain unplugged until polling begins unless the battery needs to be charged.

After we observed the testing procedures, the election official explained how the DREs will work on Election Day. Please note that the following is information conveyed to us by Santa Clara County and is not based upon our personal observations.



Sign on cart of ready-to-go DREs with information about the polling place they will be used at on election day
Cart of voting machines ready to be transported to their polling place
Empty cart used for moving and delivering the DREs

Approximately 4100 DREs will be delivered to the 822 polling places in Santa Clara County in preparation for Election Day; each precinct will receive five DREs. Workers will transport the machines to precincts up to five days in advance of the election; thus, many of these machines will be stored in the polling places for several days. In order to ensure the machines' security, the people working at the precincts will be required to sign a form promising that the voting machines will be locked until the election. Furthermore, the delivery staff will check on facility security during DRE drop-off.

Prior to November 2, 2004, the poll workers at each precinct must undergo three hours of training. Among other things, they are specifically told that they must return the cartridge from each machine to election officials at the end of the day. These cartridges carry the results that will be used in vote tallying.

Tamper-Evident Seals


Tamper-evident seals ready to be applied to the machines
Closeup of the seals. The barcode is unique to each seal.

To open the polls, the poll workers remove the tamper-proof seal on the open/close polls door and turn on the machine. They then place a new seal over the covering so that they will be able to monitor if someone turns the machine on or off or tampers with the DRE during the day. After the polls open on Election Day, poll workers verify hourly that the seals on the cartridges remain intact over the results cartridge. If the seal has been disturbed, the machine will be taken out of service pending investigation. There are questions, however, as to whether legitimate votes cast between the last positive check and the detected manipulation will be counted.

In case of a power failure, each machine has built-in battery backup so that the election can continue for two to four hours. If this occurs, the county election office will be notified immediately and additional batteries with two hours of power will be delivered to polling places.

How to Vote Using the Sequoia EVC Edge


The 'Smart Card' which allows the voter to use the machine
A page of the ballot, as presented to the voter
Voter card activator used by ballot officers to give each person access to a DRE exactly once


For a voter to cast a ballot, a poll worker must first activate a Smart Card for the voter on the Activator Machine. The card will stay active for thirty minutes or until a vote is cast. A deactivated card cannot be used to cast a vote. To cast a vote, a voter inserts the Smart Card into the DRE. Subsequently, the ballot pops up onto the screen. The voter is prompted to select the language he wishes to use and then selects the candidates he will vote for via a touch screen. On these machines, it is impossible to vote for more than one candidate for a particular position, but the voter does not have to make a selection for every office. If a voter touch-selects a candidate on a DRE but wishes to change his selection before casting his ballot electronically, the voter may do so by re-touching the name of the candidate they wish to deselect and then touching the name of the candidate for whom they want to vote. Poll workers are available to instruct voters how to change their votes if necessary. Before the vote is recorded, a screen displays all the voter's choices and lists each issue on which he has not voted. The voter can then confirm his vote or go back and make changes. If the vote is confirmed, the card immediately deactivates, and the time of the vote is recorded on the card. If the voter wishes to check whether or not his ballot was cast, the Activator Machine will determine whether the card is activated or deactivated. However, the card will not reveal how the vote was recorded on the DRE results cartridge.

Help America Vote Act


Impaired voters can use the keypad and headphones to vote without assistance
Certain DREs are designated for audio voting, although all machines are capable of supporting audio voting


One of the potential benefits of DREs is that they comply with the requirements Help America Vote Act (HAVA) of 2002 to provide equal access to voters with disabilities. Some of the DREs we saw included audio sets and keypads with large, colorful, uniquely shaped buttons for the blind, illiterate, and manually impaired. Additionally, the ballot is read to the voter in many languages including English, Spanish, Vietnamese, and Chinese. In Santa Clara County, there will be one Audio Voting Machine at each precinct, and the Smart Cards are indented in one corner so blind voters can independently insert the card with the proper orientation. Voters do not have to show that they have a disability to use the Audio Voting Machine, though the voting process takes longer on the Audio Voting Machine than on the regular machines. Santa Clara County intends to place one Audio Voting Machine at each polling place.

Tallying the Results


Display of totals
Laptops which will be used to tally the votes from the cartridges

After the polls close, the precinct managers remove the tamper-proof seal from each cartridge, band all the cartridges together, and transport them to the County Registrar that night. If any of the cartridges are not returned, a search and rescue team will retrieve missing cartridges and determine whether or not they can be counted. As an additional security measure, the identifying barcodes on the machine and result cartridges are cross-checked to see if the total number of votes cast on the machine may be wrong. However, it is impossible to distinguish between intentional "fleeing voters" who activate the machine but choose not to cast a ballot and voters whose ballots were improperly recorded.

Back at the Registrar of Voters office, results from the cartridges are copied onto a Dell Latitude laptops running Windows XP, which tally the votes. These computers have no access to the Internet; they are on a private local area network only. The final vote tallies are copied onto a USB key drive, and then copied onto another machine that is connected to the Internet so the public can review the results.

By observing electronic voting machine testing and by speaking with those in charge of running Election Day in Santa Clara County, we gained a greater understanding of how DREs work and the processes used to prepare them to function in elections.

We would like to extend a special thanks to Ruth Chavira-Lopez, Marion Dillon, Raquel Goodson, Matt Henderson, Elaine Larson, Joe Le, Mike Lopez, Albert Martinez, Al Salcedo, Mark Stamps, and everyone who works at the Santa Clara County Registrar of Voters.