Accessibility Review Report for California Top-to-Bottom Voting Systems Review

• Table of Contents
• Executive Summary
• Introduction
• Purpose of the Review
• VVSG Provisions on Usability and Accessibility, Including A Brief History
• Usability and Accessibility
  • Definitions of Usability and Accessibility
  • The Goal of Good Usability and Accessibility
  • Scope and Limits of the Review
  • General Methodology
    • Heuristic Analysis
      • Group Walkthroughs
      • Individual Walkthroughs
      • Review of User Videos
    • Methodology for User Testing
      • Physical Layout
      • Testing Protocol
  • Major Findings
    • Physical Access to the Voting Machines
      • Positioning for Wheelchair Users
      • Interference with Wheelchair Controls and Armrests – Frontal Approach
      • Interference with Wheelchair Controls and Armrests – Lateral Approach
      • Adjustability for Standing and Seated Use
      • Machine Support Stand Stability Concerns
    • Manual Dexterity Accommodation Concerns
      • Voter Access Cards
      • Physical Controls
      • Touch Screen Controls
    • Access User Interface Connectors
    • Visual Display Concerns
      • Touch Screen Parallax
      • Screen Rendering Time
    • Touch Screen Controls
    • Non-Touch Screen Input Controls
      • The three systems have very different physical controls
        • Hart eSlate
        • Diebold TSX
        • Sequoia Edge
      • Counting of Keystrokes or Other User Control Input Actions
        • Selecting Votes and Casting a Ballot
        • Comparison of Touch Screen versus Wheel Selector Voting for Sighted Voters
    • Dual-switch Input Control Concerns
    • Speech Interface Concerns
      • Speech Quality and Rate
        • Speech Rate
      • Volume
        • Speech Volume
      • Pauses
      • Concatenation Effects
    • Alternative Language Concerns
    • VVPAT Accessibility for Verifying Print Record
    • Total In-booth Voting Times
    • Voting Accuracy
    • The Audio Interface in Complex Tasks
    • Write-in Concerns
      • Mispronunciation
      • Confusing Wording
    • Orientation, Help and Instruction Messages for the Audio Interface
    • Privacy Concerns
      • Eavesdropping on the Visual Display
      • Blanking the Visual Display
      • Privacy Shields
      • Vulnerability to Quick Glances or Camera Shots
      • Eavesdropping on the Audio Voting with Radio Receivers
      • Eavesdropping on Audio Voting with Recorders and Wireless Bugs
    • Personal Safety Concerns
    • Public Health and Sanitation
    • Reliability Concerns
    • Poll Worker Training and Assistance Concerns
  • Near Term Mitigations for Major Findings
    • Improving Physical Access to the Voting Machines
      • Provide Chairs and Flat Surfaces for Voters
    • Improving Manual Dexterity Accommodations
      • Voter Access Cards
      • Physical Controls
      • Touch Screen Controls
    • Improving Interface Connectors
    • Improving the Visual Display
      • Single Race per Page
      • Eliminate In-Screen Scrolling
      • Accelerate Video Performance
    • Improving the Touch Screen Controls
      • Reduce Touch Screen Parallax
    • Improving Non-Touch Screen Input Controls
    • Reduce Fatigue from Audio Interface Use or Arduous User Control Input Actions
    • Improving the Speech Interface
      • Speech Rate
      • Volume Control
    • Improving Alternative Language
    • Improving VVPAT Accessibility
    • Addressing Total In-Booth Voting Times
    • Improving Voting Accuracy
    • Improving the Audio Interface in Complex Tasks
      • Verification
    • Ballot Verifying and Casting Concerns
    • Confusing Wording and Use of Grammar
    • Orientation, Help, and Instruction Messages
      • Missing the Startup Orientation Help or Instructions
      • Outreach to Inform Voters about Accessible Voting
      • Information Available in Braille and Large Print
    • Privacy Concerns
      • Eavesdropping on the Visual Display
      • Blanking the Visual Display
      • Privacy Shields
        • Privacy curtains
        • Vulnerability to Quick Glances or Camera Shots
        • Eavesdropping on the Audio Voting with Radio Receivers
        • Eavesdropping on Audio Voting with Recorders and Wireless Bugs
    • Personal Safety Concerns
    • Public Health and Sanitation
    • Reliability Concerns
    • Pollworker Training and Assistance Concerns
  • Conclusions
  • Acknowledgements
  • References
  • Appendices
    • Section: 3.1.2.a.
    • Section: 3.1.2.b.
    • Section: 3.1.2.c.
    • Section: 3.1.2.d.
    • Section: 3.1.2.e.
    • Section: 3.1.2.f.
    • Section: 3.1.2.g.
    • Section: 3.1.4.a.
    • Section: 3.1.4.b.
    • Section: 3.1.4.b.i.
    • Section: 3.1.4.b.ii.
    • Section: 3.1.4.c.i.
    • Section: 3.1.4.c.ii.
    • Section: 3.1.4.c.iii.
    • Section: 3.1.4.d.
    • Section: 3.1.4.e.
    • Section: 3.1.5.a.
    • Section: 3.1.5.b.
    • Section: 3.1.5.c.
    • Section: 3.1.5.d.
    • Section: 3.1.5.e.
    • Section: 3.1.5.f.
    • Section: 3.1.5.g.
    • Section: 3.1.5.h.
    • Section: 3.1.6.a.
    • Section: 3.1.6.b.
    • Section: 3.1.6.c.
    • Section: 3.1.6.d.i.
    • Section: 3.1.6.d.ii.
    • 3.2.2.1 Partial Vision
    • Section: 3.2.2.1.b.
    • Section: 3.2.2.1.d.
    • Section: 3.2.2.1.e.
    • Section: 3.2.2.1.f.
    • 3.2.2.2 Blindness
    • Section: 3.2.2.2.b.
    • Section: 3.2.2.2.b.i.
    • Section: 3.2.2.2.b.ii.
    • Section: 3.2.2.2.b.iii.
    • Section: 3.2.2.2.b.iv.
    • Section: 3.2.2.2.b.v.
    • Section: 3.2.2.2.c.i.
    • Section: 3.2.2.2.c.iv.
    • Section: 3.2.2.2.c.v.
    • Section: 3.2.2.2.c.vi.
    • Section: 3.2.2.2.c.vii.
    • Section: 3.2.2.2.c.viii.
    • Section: 3.2.2.2.c.ix.
    • Section: 3.2.2.2.d.
    • Section: 3.2.2.2.e.
    • Section: 3.2.2.2.f.
    • Section: 3.2.2.2.g.
    • Section: 3.2.3.b.
    • Section: 3.2.3.c.
    • Section: 3.2.3.d.
    • Section: 3.2.3.e.
    • Section: 3.2.4.a.
    • Section: 3.2.4.b.
    • Section: 3.2.4.b.i.
    • Section: 3.2.4.b.ii.
    • Section: 3.2.4.b.iii.
    • Section: 3.2.4.b.iv.
    • Section: 3.2.4.b.v.
    • Section: 3.2.4.b.vi.
    • Section: 3.2.4.c.
    • Section: 3.2.5.b.
    • Section 3.1.2.a.
    • Section 3.1.2.b.
    • Section 3.1.2.c.
    • Section 3.1.2.d.
    • Section: 3.1.2.e.
    • Section: 3.1.2.f.
    • Section: 3.1.2.g.
    • Section: 3.1.4.a.
    • Section: 3.1.4.b.
    • Section: 3.1.4.b.i.
    • Section: 3.1.4.b.ii.
    • Section: 3.1.4.c.i.
    • Section: 3.1.4.c.ii.
    • Section: 3.1.4.c.iii.
    • Section: 3.1.4.d.
    • Section: 3.1.4.e.
    • Section: 3.1.5.a.
    • Section: 3.1.5.b.
    • Section: 3.1.5.c.
    • Section: 3.1.5.d.
    • Section: 3.1.5.e.
    • Section: 3.1.5.f.
    • Section: 3.1.5.g.
    • Section: 3.1.5.h.
    • Section: 3.1.6.a.
    • Section: 3.1.6.b.
    • Section: 3.1.6.c.
    • Section: 3.1.6.d.
    • Section: 3.1.6.d.i.
    • Section: 3.1.6.d.ii.
    • Section: 3.2.2.1 Partial Vision
    • Section: 3.2.2.1.b.
    • Section: 3.2.2.1.d.
    • Section: 3.2.2.1.e.
    • Section: 3.2.2.1.f.
    • 3.2.2.2 Blindness
    • Section: 3.2.2.2.b.
    • Section: 3.2.2.2.b.i.
    • Section: 3.2.2.2.b.ii.
    • Section: 3.2.2.2.b.iii.
    • Section: 3.2.2.2.b.iv.
    • Section: 3.2.2.2.b.v.
    • Section: 3.2.2.2.c.i.
    • Section: 3.2.2.2.c.iv.
    • Section: 3.2.2.2.c.v.
    • Section: 3.2.2.2.c.vi.
    • Section: 3.2.2.2.c.vii.
    • Section: 3.2.2.2.c.viii.
    • Section: 3.2.2.2.c.ix.
    • Section: 3.2.2.2.d.
    • Section: 3.2.2.2.e.
    • Section: 3.2.2.2.f.
    • Section: 3.2.2.2.g.
    • Section: 3.2.3.b.
    • Section: 3.2.3.c.
    • Section: 3.2.3.d.
    • Section: 3.2.3.e.
    • Section: 3.2.4.a.
    • Section: 3.2.4.b.
    • Section: 3.2.4.b.i.
    • Section: 3.2.4.b.ii.
    • Section: 3.2.4.b.iii.
    • Section: 3.2.4.b.iv.
    • Section: 3.2.4.b.v.
    • Section: 3.2.4.b.vi.
    • Section: 3.2.4.c.
    • Section: 3.2.5.b.
    • Section: 3.1.2.a.
    • Section: 3.1.2.b.
    • Section: 3.1.2.c.
    • Section: 3.1.2.d.
    • Section: 3.1.2.e.
    • Section: 3.1.2.f.
    • Section: 3.1.2.g.
    • Section: 3.1.4.a.
    • Section: 3.1.4.b.
    • Section: 3.1.4.b.i.
    • Section: 3.1.4.b.ii.
    • Section: 3.1.4.c.i.
    • Section: 3.1.4.c.ii.
    • Section: 3.1.4.c.iii.
    • Section: 3.1.4.d.
    • Section: 3.1.4.e.
    • Section: 3.1.5.a.
    • Section: 3.1.5.b.
    • Section: 3.1.5.c.
    • Section: 3.1.5.d.
    • Section: 3.1.5.e.
    • Section: 3.1.5.f.
    • Section: 3.1.5.g.
    • Section: 3.1.5.h.
    • Section: 3.1.6.a.
    • Section: 3.1.6.b.
    • Section: 3.1.6.c.
    • Section: 3.1.6.d.i.
    • Section: 3.1.6.d.ii.
    • Section: 3.2.2.1 Partial Vision
    • Section: 3.2.2.1.b.
    • Section: 3.2.2.1.d.
    • Section: 3.2.2.1.e.
    • Section: 3.2.2.1.f.
    • 3.2.2.2 Blindness
    • Section: 3.2.2.2.b.i.
    • Section: 3.2.2.2.b.ii.
    • Section: 3.2.2.2.b.iii.
    • Section: 3.2.2.2.b.iv.
    • Section: 3.2.2.2.b.v.
    • Section: 3.2.2.2.c.i.
    • Section: 3.2.2.2.c.iv.
    • Section: 3.2.2.2.c.v.
    • Section: 3.2.2.2.c.vi.
    • Section: 3.2.2.2.c.vii.
    • Section: 3.2.2.2.c.viii.
    • Section: 3.2.2.2.c.ix.
    • Section: 3.2.2.2.d.
    • Section: 3.2.2.2.e.
    • Section: 3.2.2.2.f.
    • Section: 3.2.2.2.g.
    • Section: 3.2.3.c.
    • Section: 3.2.3.d.
    • Section: 3.2.3.e.
    • Section: 3.2.4.a.
    • Section: 3.2.4.b.
    • Section: 3.2.4.b.i.
    • Section: 3.2.4.b.ii.
    • Section: 3.2.4.b.iii.
    • Section: 3.2.4.b.iv.
    • Section: 3.2.4.b.v.
    • Section: 3.2.4.b.vi.
    • Section: 3.2.4.c.
      • All Races and Choices for the Accessibility Testing Ballots
      • List of All Possible Contests and Choices:
      • List of Contests for Each System's Ballots

Accessibility Review Report for California Top-to-Bottom Voting Systems Review

Noel Runyan and Jim Tobias
July 26, 2007

Table of Contents

Executive Summary

Three voting systems, the Diebold AccuVote TSx, Hart eSlate and Sequoia Edge I and II, were evaluated for usability and accessibility for voters with disabilities and voters with alternate language needs, using both heuristic and user testing techniques. Although each of the tested voting systems included some accessibility accommodations, none met the accessibility requirements of current law and none performed satisfactorily in test voting by persons with a range of disabilities and alternate language needs. In some cases the accessibility or usability deficits could be partially or wholly mitigated. Some of these mitigations would not require new federal and state certification testing.

Introduction

Voting system accessibility surveys and reviews by Cook County (Illinois), Access World and others have shown that the voting systems previously used in California have significant limitations in accessibility for many voters with disabilities and alternative language needs.

Recently, the designs of some of the voting systems have been changed to try to improve both their accessibility and their security. In some cases, the changes made to add voter verified paper audit trail (VVPAT) printers and other security and privacy improvements have had a negative impact on the accessibility of the systems for voters with disabilities.

Because it is impossible to affirm overall accessibility and usability conformance merely by examining documentation for voting products, and because there have not been in-depth accessibility studies performed for these voting systems, rigorous testing was required to assess the accessibility and usability of California's voting systems. However there are currently no rigorous methodologies or standards defined, in the 2005 Voluntary Voting System Guidelines (VVSG) or elsewhere, for testing accessibility of voting systems. The Federal Election Assistance Commission (EAC) and the National Institute of Standards and Technology (NIST) are still in the early stages of development of standards for voting system accessibility testing.

Purpose of the Review

This review was undertaken primarily to identify whether the three systems were sufficiently accessible for voters with a range of different disabilities, and whether they were generally usable as well. Alternative language accessibility was also assessed.

Additionally, this access review was tasked with identifying specific accessibility and usability concerns and reporting options for near-term mitigations appropriate for the 2008 elections, as well as longer-term mitigations including voting system design changes.

The results may be used to guide future actions by the Secretary of State's office regarding certification. In addition, vendors and local election officials may find useful information in this report that will improve the usability and accessibility of voting technologies in both the near and long terms.

Finally, the authors hope that the larger community of stakeholders concerned with voting technologies will find both the methodology and results useful in developing advanced practices of design, testing, and implementation.

VVSG Provisions on Usability and Accessibility, Including A Brief History

The Help America Vote Act of 2002 (HAVA) established the United States Election Assistance Commission (EAC), the agency responsible for federal funding and technical assistance in voting technologies. HAVA section 301(a)(3) sets forth accessibility requirements. EAC inherited previous guidelines, the 1990 Performance and Test Standards for Punchcard, Marksense and Direct Recording Electronic Voting Systems, and the 2002 Voting Systems Standards (VSS). EAC empanelled the Technical Guidelines Development Committee (TGDC) to develop further standards, supported by the National Institute for Standards and Technology (NIST). The work of the TGDC, comments from the public and other experts at public hearings, and the EAC itself as well as other authoritative reviewers, resulted in the release of new Voluntary Voting System Guidelines (VVSG) in December 2005. The VVSG supersedes the 2002 VSS effective in December 2007.

The VVSG contain a completely new section on usability and accessibility requirements. These reflect the HAVA 301(a)(3) accessibility requirements.

As already indicated, there is not nor should there be an absolute line drawn between accessibility and usability. The Section 3 requirements in the VVSG contain references to both, and provide specific guidance regarding the goals of accuracy, efficiency, and satisfaction. Using a framework from the domain of accessible technology, they address the needs of users with functional limitations in vision, hearing, mobility, dexterity, speech, and cognition. For all of these dimensions of functional limitation, VVSG addresses both perception and interaction where relevant. Additional issues are also included: privacy, protection of voters who use alternate formats or methods for voting, and the use of alternate languages.

In this report we will use the VVSG requirements in two ways. First, they form the framework in which we identify overarching issues we found in our testing. Second, they are the reporting method we use in the appended results of the review for each of the voting systems tested.

Usability and Accessibility

The scope of this accessibility review is primarily limited to human factors issues, meaning we are concerned with the entire process of the voter casting a ballot as they intended.

This process involves not only the voting system interface directly experienced by the voter, but also includes usability issues pertaining to ballot design, the influence of the polling place environment on accessibility and usability, as well as the setup, operation, and support of accessible voting systems by election administrators and pollworkers.

Definitions of Usability and Accessibility

The International Standards Organization defines usability as:

"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use."

It could be said that accessibility consists in defining those "specified users" as inclusively as possible. That is, the goals of effectiveness, efficiency, and satisfaction are unchanged, but equal attention is paid to making sure that those goals are met for people with disabilities. Accessibility can be further clarified by categorizing the accommodations used to include these "additional" users: some are extensions to usability features (such as magnification) with significant benefit to non-disabled users, and some provide compatibility with assistive technologies rarely useful to people without disabilities (such as sip and puff switches).

The Goal of Good Usability and Accessibility

The goal of good usability in voting equipment for voters and pollworkers is to easily, accurately, and intuitively use the equipment with minimal instruction and training and to successfully complete the voting process.

Accessibility can be seen as the additional accommodations, used primarily by people with disabilities, that help them to successfully use the voting equipment. For this reason, this report most often treats accessibility as a subset of usability.

Testing voting equipment for both usability and accessibility is potentially complex, but necessary for two key reasons:

1. Any usability problems that a piece of voting equipment has is likely to impact voters with disabilities more seriously than voters without disabilities.
2. A piece of voting equipment might have the legally required accessibility accommodations, but still may not be very usable by voters with disabilities, if the accommodations were poorly designed or improperly implemented.

Scope and Limits of the Review

This study was undertaken over an extremely brief period of time from its approval to its completion, due to the rescheduling of the California primary and the resulting compression of the election preparation schedule. This, as well as other factors have unfortunately limited the scope of this access review, and we wish to be explicit about those limitations.

Only three systems were included, although several more, using different designs, have been certified for use in California.

The versions of these three systems may not have been the only versions used in California, and new models of one or more of the systems may already be undergoing testing for approval by the federal Election System or the Secretary of State.

The ballot definitions used on the three machines were not identical, and were not based on a ballot design optimized for usability testing.

We were not able to test a multiple member race in which the voter may choose more than one candidate.

The alternate language ballot definitions supplied by the vendors for the machines we tested may not have been as complete or optimal as those used in a normal election.

The 45 volunteers who served as test voters ("voters" or "users") were selected to represent a broad range of disabilities. They cannot be assumed, however, to be perfectly representative of all possible voters with disabilities in degree and type of functional limitation, experience with voting, or pre-existing attitude toward voting technologies.

Finally, because successful accessibility of the voting system by the voter depends, in many ways, on the ability of the pollworkers and elections administrators to set up, operate, and support the voting systems properly, it is important to also review and address the usability of the voting systems for pollworkers. This was outside the scope of the current access review.

General Methodology

Heuristic Analysis

The two authors of this report, assisted by a volunteer with expertise in accessible technology and voting systems, served as expert reviewers of the three systems. The goal of their analysis was to identify as many potential accessibility and usability issues for voters as possible in advance of the user testing, and afterwards to confirm and clarify issues identified during the user testing.

These expert analyses took three forms:

Group Walkthroughs

All three experts worked together in a single session, engaging in a dialogue as one expert navigated through the scenario. These were recorded by note taking, video/audio recording, or both.

Individual Walkthroughs

A single expert exercised a specific interface, with his verbalized interactions and findings recorded on audio and video.

Review of User Videos

After the user testing was complete, we reviewed a subset of the sessions for several purposes. First, we wanted to confirm the accuracy of the session timing results. Second, we wanted to be sure that particular issues noted in the session data forms appeared similarly in the recording. Third, we wanted to identify any additional information that would help explain usability or accessibility issues, or deepen our analysis.

Methodology for User Testing

Physical Layout

Cameras were positioned in the cubicles to record the contents of the screen, the user's actions at the controls, and the user's face to capture any emotional expressions. A microphone recorded any speech by the user and "pollworker", and an additional sound track captured any speech from the voting system.

Testing Protocol

We developed a brief intake instrument for the study, focused on basic demographic information, disabilities, and voting experience. Project staff assisted the users in completing this form and the consent forms as needed.

For privacy protection, the users were not referred to by their names during the testing. To better simulate a real polling place experience, the test subjects were referred to as "voter" and the experimenters were referred to as "pollworker". To limit user bias for or against particular vendors or product names, the voting systems were only referred to as 'A', 'B', or 'C', and the cubicles were so labeled.

Users were assigned to the three systems in a randomized order. For each system, users were assigned a pollworker who provided a standardized amount of orientation and assistance in getting started. For users using the audio interface this often meant adjusting the volume. For users who used wheelchairs it was usually necessary to re-arrange the equipment.

Users were encouraged at all points to perform as many tasks as possible independently. This included entering the four-digit voter access code used by the Hart eSlate and inserting the voter access cards into the slots on the Diebold TSX and Sequoia Edge.

There were five timed segments to each trial. The first was the orientation provided by and with the pollworker. Next, the voter was allowed to complete the ballot however he/she wished, moving through the ballot "freestyle" as if in an actual election, and making choices according to his/her own preferences. Once the voter reached the end of the ballot, we asked him/her to go back to a specific race and change his/her selection. Then we asked the user to enter a write-in name for another specific race. Then we began a sequence leading up to actually casting the vote, which included reviewing the ballot, printing a verification, rejecting the ballot and making a change, and finally printing another verification and casting. We collected the elapsed time for each of these five segments.

We asked the users to state out loud what selections they were making for each contest, as they were making their selections. This was to assist us in determining their intended choice, so we could later determine their voting accuracy.

We also encouraged the users to verbalize their thoughts as they were working with the system. Several of the users were able to give us extremely useful verbal stream-of-consciousness observations.

After each user completed voting on all three systems we interviewed the user to collect specific reactions on a data collection sheet. We asked the user to rate the system on several factors, and asked whether he/she would be willing to use that system in a real election. Additionally, we encouraged a full discussion of the user's reaction to the system, including specific features he/she thought were important as well as any suggestions. When possible we conducted these discussions while the user was still in front of the voting system, so they were able to look at and/or touch the system while commenting about it.

Major Findings

Physical Access to the Voting Machines

Physical access concerns arose in four contexts:

• Positioning for wheelchair users
• Interference with wheelchair controls and armrests, both frontal and lateral
• Standing and seated use
• Machine support stand stability

Positioning for Wheelchair Users

The VVSG requires a minimum of 30 inches of toe and knee clearance. No machine provided that clearance. This deficit posed a problem to almost every wheelchair-using voter in this study.
Sequoia Edge II Diebold TSX
Hart eSlate

Interference with Wheelchair Controls and Armrests – Frontal Approach

The bottom edge of the eSlate interferes with the wheelchair joystick and even with the armrest.
The same joystick barely cleared the VVPAT on the Diebold after adjusting the angle of the machine.
The front part of this wheelchair control barely makes it under the bottom edge of the eSlate; that edge is too low for the joystick itself.
Side view of the same wheelchair control approaching the Diebold; the VVPAT interferes with the joystick.
The user's knees meet the bottom of the eSlate while the user is still out of range of the controls.
This wheelchair control juts forward of the right armrest. The Diebold VVPAT prevents enough forward progress for the user to reach the touch screen.
The narrow clearance between the two front legs of this machine prevents the user from approaching from the front, as the wheelchair footplates are wider than the legs would allow.

InteInterference with Wheelchair Controls and Armrests – Lateral Approach

The Diebold VVPAT interferes with armrest when user approaches from the left.
The lower edge of the machine is too close to the wheelchair control when this user approaches from the right.

Adjustability for Standing and Seated Use

The systems we tested had little or no height adjustment capability; two were able to adjust the angle at which the screen was presented. This lack of adjustability posed a problem principally for voters who would prefer to sit, but also posed a problem for voters who wanted to stand but were limited in their ability to bend over to read the screen and/or VVPAT.

The range of angle adjustment was occasionally insufficient. In some cases, we needed to place blocks under the back legs of the voting machine stand, in order to tilt the unit forward enough to give a voter in a wheelchair access to the touch screen.

Machine Support Stand Stability Concerns

One machine's support structure seemed prone to tipping or breaking if it was subjected to uncontrolled movements of an individual or a wheelchair.

The lack of adjustment flexibility in the support stands of the current systems severely limits physical access to the voting machines for many voters with motor impairments.

Manual Dexterity Accommodation Concerns

Users with impaired dexterity and reach had some difficulty using these systems. Concerns arose in three specific areas regarding dexterity: voter access cards, physical controls, and touch screen controls.

Voter Access Cards

Voters with impaired grip strength had difficulty picking up the cards (if they were placed on a table), holding the cards, orienting them properly, reaching the card slot (especially on the Diebold TSX), applying enough force to engage the card in the card reader, and retrieving the card after voting. The Hart eSlate, which uses an access code instead of a card, did not pose any accessibility concerns here for voters with dexterity limitations.

Physical Controls

All physical controls pose some degree of difficulty for people with impaired dexterity. The Hart eSlate controls are fairly well designed, with a good dish on the Select wheel that would aid use by finger or head- or mouth-stick. Note, however, that the circular motion required by the Select wheel is difficult for most stick users. The other buttons are large enough, and placed far enough apart (except for the Previous and Next buttons) for use by all but the most severely physically disabled voter, who would probably prefer the external switches. However the lack of relief on the built-in controls means that head- or mouth-sticks might pop out before activating the intended control.

The other two systems are intended to be controlled by touch screen, not by their physical controls (which are intended for blind users). We will briefly mention the possible difficulties these controls would pose to users with both dexterity and vision impairments. The Sequoia Edge controls are too stiff for some users, with a large degree of travel and sideways displacement. The controls protrude more than a quarter-inch from the shell of the control box, making them especially difficult for voters using head- or mouth-sticks. This is also the case for the Diebold TSX. The Diebold keypad keys are more stiff than necessary, and do not offer any friction surface.

Touch Screen Controls

The two touch screen systems pose a reach/range problem for voters with impaired range of motion, and reduced strength near the limits of their range. Some users had difficulty reaching the very top of the touch screen – the distance from top to bottom is quite large, by design. If the touch screens were oriented in landscape mode this range problem would be much reduced.

The other problem lies in the size and shape of the screen targets, and the separation between them. In most cases the active targets are low, long rectangles with little or no space between them in the vertical dimension. The optimal design for people with even moderately reduced accuracy would be square or circular targets with large, evident dead spaces between them.

Access User Interface Connectors

On the Sequoia Edge, the supplied sip and puff head-mounted control plugs into the keypad unit properly. However, the jelly switches also supplied by Sequoia are not compatible, as they have a right angle plug and the Sequoia Edge keypad needs straight-in plugs. Because the dual switch jacks on the tactile keypad are too close together and tightly surrounded by the case housing, it is not easy to find extension cables or extension adapters that can plug into the two jacks, side-by-side.

Additionally, the opening around the Sequoia Edge audio headphone jack is in a channel that is too narrow for some 1/8 inch headphone plugs to fit, making it difficult or impossible for voters with assistive listening devices or voter-supplied personal headphones to plug in.

Both of the Sequoia Edge I and II units we were testing required an on-site visit from factory representatives, due to the units freezing up or "hanging" without helpful diagnostic message when keypads were plugged into an unused improper jack on the back of the units identical and immediately adjacent to the jack intended for connection of the keypad. Instead of being labeled "Keypad", the correct jack was labeled "Audio" and no color coding was used to help pollworkers match the plug and jack. When the keypad was plugged into the incorrect jack and the voting machine was loaded with a voter card, the unit would hang, and would not report that it could not find a keypad plugged in to support the audio voting specified on the voter card.

Another time, the Sequoia Edge II test unit began making random sounds that could be described as an "angry chipmunk chattering". Sequoia engineers advised us that the noises were probably the result of the same keypad plug coming loose, according to Sequoia engineers. Subsequent daily checks by the access team leader found this same keypad plug often was not plugged in firmly and could be removed with only the slightest pulling on the cable near the plug.

The Hart eSlate tablet can be removed from its stand for curb-side voting, and when installed on the stand, the clearances between the inside wall of the stand and the jacks on the side of the tablet unit are very tight and only allow space for right angle plugs in those jacks. Therefore, the jacks will not accept a straight-in headphone plug. If headphones supplied by a user do not have a right angle plug they must be connected through an extension cable with a right angle plug.

Additionally, the headphone cable must be routed up through the stand and out the flap openings at the top edge of the tablet. This routing may make the remaining cable on a user's headphones too short.

During our testing, the eSlate unit reported printer failures several times. In each case, the failure appeared to be caused by pulling the tablet out of the stand and then not getting it settled and plugged back into the stand properly. These printer failures exacerbated the inconvenience of having to remove the tablet from the stand every time the headphone or dual-switch input jacks need to be accessed.

Visual Display Concerns

Touch Screen Parallax

Touch screen systems are typically calibrated for use by an average height voter in a standing position. Voters positioned lower in chairs may experience an optical parallax effect that makes the screen's visible buttons appear to be slightly higher than the touch sensing area for that visible button. Touching near the boarder of a visible screen button may cause selection of the adjacent button.

Screen Rendering Time

We measured the time it took both Edge I and Edge II in magnified mode to completely render a screen – the "screen write time".

• On the Edge I, 14 seconds to set up the next page and 24 seconds to set up a horizontal scroll.
• On the Edge II, 7 seconds to set up the next page and 8 seconds to set up a horizontal scroll.

The screen write times of both systems are long; such individual delays may confuse or frustrate a voter.

Touch Screen Controls

Our testing revealed that touch screen controls on all three tested systems pose insuperable barriers for sighted voters who cannot see the screen well enough to identify where the active surfaces are. All three systems provided alternative input devices for these voters.

Non-Touch Screen Input Controls

Several voters in our testing found that some of the voting systems require so many repeated button presses or wheel movements that they were feeling exhausted and sore. Some of the input control methods take several times as many keystrokes or actions as the other systems.

The three systems have very different physical controls

Hart eSlate

The controls of the Hart eSlate are built into the tablet, all along the lower edge directly beneath the screen. At the extreme right is a rotating Select wheel (radius = 31 mm). The wheel has radial ridges and a single dished depression for a finger or mouthstick. To its left is the Enter button, 50 mm tall by 32 mm wide, with a scooped right edge echoing the Select wheel's circumference. To its left are three buttons: a rounded rectangular Help button (11 mm tall by 32 mm wide) above a pair of triangular buttons Previous and Next. These latter buttons are roughly equilateral, 20 mm tall and 20 mm wide, oriented as arrowheads pointing in opposite directions. At the extreme left is the Cast Ballot button, round (radius = 17 mm) with a truncated top. Distances between the controls range from 11 mm to 25 mm. The Cast Ballot button is red with white letters; the others are off-white with black letters. Braille appears below the buttons. There are no volume or speech rate controls.

The Select wheel is used as a scrolling device within and between races, with approximately 20 tactile click points per rotation.

All buttons have a very slight bevel, but there is no relief or reveal. That is, aside from the bevel, the buttons are flush with the shell of the tablet. The key tops are not dished or recessed, to make it easier to position and keep a mouthstick on them. Some users found the buttons difficult to navigate by touch; one commented, "The keys should be raised to make them more obvious."

The braille on the Previous and Next keys is not horizontal like the others, but follows the diagonal line of the bottom of the arrowhead alignment. This was disorienting to some braille users.

Some users liked the wheel, but some found it too easy to turn inadvertently. Some found it at least initially confusing. One voter said she started off thinking she would not like the wheel and would prefer arrow keys, but liked the wheel for write-ins. One voter commented, "[the] wheel makes it more intuitive". At least one user was confused about which direction to turn the wheel.

Some users found that there was not enough mechanical feedback from the buttons; they were concerned that their input had not been accepted.

Having a Cast Ballot button made that function clear and evident, but it did cause some user error when voters pressed it before they were really ready to vote. Some confused the Cast Ballot button with merely activating their current choice in a race.

Voters did not use the Next and Previous keys very often.

Diebold TSX

The controls are laid out like a conventional telephone keypad: 4 rows of 3 columns, with the digits, '*', and '#'. There is a nib on the '5' key. The keys are 13 mm high by 21 mm wide, and are separated by 5 mm horizontally and 9 mm vertically. Print on the keys is white on a black background. There is no braille.

The control keypad is tethered to the machine and nests above the VVPAT. The keypad can be moved and repositioned without difficulty. However, it was difficult for voters to replace the keypad properly. Its undersurface has rubbery pads for stability when used on a flat surface such as a table or wheelchair lap tray. The laterally dished underside does not facilitate connection to a voter's leg or armrest. One voter asked for "the keypad mount [to] be a horizontal slide to let you position it left, middle, or right." Because of its width and lack of an underside grip, it was awkward and tiring for users to hold the keypad in one hand, for the long voting process, while pressing keys with the other.

In theory a familiar keypad is an advantage. However, this may have been offset by the fact that there was no direct mapping between the keys and many of the functions the voter wanted to perform, and no braille, color, or shape indication of their functions.

Frustration was expressed by several voters during the write-in task. Letters were entered via the '2' through '9' keys. Editing and selection functions were arbitrarily assigned to the other keys. Key assignments changed from one function to another, further confusing some users. One user asked for a "summary of which letters go on which keys"; another complained that "Key '5' is sometimes used for cancel and sometimes confirm; that seems contradictory."

Sequoia Edge

At the top of the control box are small (8 mm) buttons for volume control and speed of speech. The volume buttons are separate and identical; the speed controls are connected to each other. Below these, near the right edge, is a blue square (21 mm) Help button. Below this is a pair of arrow buttons, Back and Next, yellow and green respectively, 21 mm high by 24 mm wide. Below these, near the right edge, is a round (radius = 12 mm) red Select button. The distances between controls range from 16 to 18 mm.

Although there are braille labels for the keys, the labels are shallow and too close to key caps to facilitate reading by some voters' fingers.

The Sequoia Edge ATI keypad is tethered to the voting machine by a heavy data cable. With a Velcro cloth adapter added to its back, the keypad can be stored by sticking it to the Velcro claw patches on the outer and bottom edge of the left privacy panel. However, because of the keypad size and weight, it does not appear to be stored securely when it is stuck onto any of the Velcro patches of the privacy panel. Voters with manual dexterity impairments can encounter problems attempting to independently remove a Velcro-attached keypad from or reattaching it to the system's privacy panels. "Parking" the keypad on the privacy panel for operation puts the keypad at a height and angle that makes its operation extremely uncomfortable for visually impaired voters who need to keep their fingers on the keys, for touch typing rather than operation by single finger poking. The force required to press the keys of the keypad would make the flexible privacy panel sway or wobble back and forth substantially, if voters attempted to use it mounted there. It appears that the Velcro cloth roll adapter on the back of the keypad supplied for testing is a new feature that has not been generally available on currently fielded Sequoia Edge systems in California. Although it may be of some use for storing the keypad between voting sessions, sticking the keypad to the privacy shield isn't a reasonable solution for a parking place to support the keypad for single handed operation.

Several voters commented that the controls were easy to use, but might have better instructions.

One user found that pressing the Select button off center could catch the key cap under the shell of the control box, at least temporarily.

Counting of Keystrokes or Other User Control Input Actions

Note: This is for the non-touch-screen input control alternatives such as keypad or selector wheel.

Minimum keystrokes needed for a write-in of "John Smith":

• 103 for system A, Hart eSlate, wheel clicks and Enter keystrokes. Multi click wheel spinning might decrease the effort.
• 32 keystrokes for system B, Diebold TSX, using telephone-style text messaging entry.
• 130 keystrokes for system C, Sequoia Edge, using back and forth selection in a circular alphabetic list.

Note: The Sequoia write-in character selection list has numbers 0-9 and several other punctuation characters that make it a much longer list to navigate than that of the shorter eSlate character list, and takes about 4 times more keystrokes than the phone text-messaging approach of the Diebold TSX.

In the case of write-ins, the cognitive load of phone-style text entry may cause many voters to prefer the lower cognitive load of the selector wheel for write-in tasks.

Also, because sip and puff or other dual-switch input controls are not supported with a Previous Selection control, write-in with dual-switch control could take several hundred strokes or sip/puffs.

Selecting Votes and Casting a Ballot

In addition to an added single keystroke or wheel click to move to any next candidate in any race, there would be:

• 1 stroke or action to start, 2 per race and 4 to cast -- on Hart eSlate (with an additional action per race, if using dual-switch control).
• 2 strokes per race and 2 to cast ballot -- on Diebold TSX.
• 4 strokes per race and 3 to cast ballot -- on Sequoia Edge.

Comparison of Touch Screen versus Wheel Selector Voting for Sighted Voters
(From the Brennan Center Usability Report)

Hart InterCivic's eSlate

• required 3.92 actions per task
• 10.56 minutes on average for a voter to complete the voting process

Diebold's AccuVote-TSX touch screen

• required 1.89 actions per task
• 4.68 minutes to complete the process

By comparison, when the Brennan Center's non-disabled usability testers voted on systems that were similar to those we were testing, they needed about half the actions and half the time if they were using the touch screen, instead of the selector wheel control. It should not be too surprising that they generally felt that they were more efficient with the touch screen voting.

However, lower action counts and shorter time-on-task should not be the only measures of usability. As shown in the Brennan Center's Usability report, lower keystroke or action counts and shorter time-on-task may be associated with less voter error checking and lower accuracy scores.

Dual-switch Input Control Concerns

Only the eSlate and the Sequoia Edge systems support dual-switch input controls such as jelly switches and sip and puff switches. These controls allow the user to control the voting system by actuating one or the other of the two switches. Generally, one switch moves the system to the next choice in a list, and the other switch acts like an Enter or Select key. A serious limitation of the dual-switch input controls for the eSlate and Edge is that neither system has significantly redefined their help messages, instructions, or prompts to reflect use of the dual-switch controls (with minor exceptions for the eSlate). This leaves the dual-switch user with irrelevant help messages.

Speech Interface Concerns

Audio interface users mentioned several problems with the speech interface:

• Speech quality and rate
• Volume
• Pauses
• Concatenation effects

Speech Quality and Rate

The audio quality of the recorded speech was inconsistent. It was not possible to determine if the low quality arose during recording or playback.

Speech Rate

Speech rate measurements were made using help messages to generate audio output.
The Hart eSlate speech rate is not adjustable. In the help and instruction messages it is often too fast for average voters.
The Diebold TSX speech rate is 70% to 200% of normal speech rate, which complies with the VVSG range specification.
The Sequoia Edge speech rate is 71% to 135% of normal speech rate. This does not meet the VVSG upper rate requirement of 200% of normal.

Discussion: Both the Diebold TSX and the Sequoia Edge rate control technology causes significant "chipmunk" or pitch distortion, like speeding up or slowing down a simple tape player. The fact that some blind voters might be used to and not bothered by "chipmunk" speech pitch distortion does not mean that it is appropriate for all audio voters. Because the VVSG does state that the speech should be undistorted, we feel that pitch distorting rate controls are not in full conformance with the VVSG rate control requirement.

Although range as a percentage of normal speech rate is an easy parameter to measure for conformance checking, the work of Emerson Foulke and others indicates that it would be more meaningful to specify and measure the desired speech rate range in words per minute. High speed comprehension of speech is not limited so much by how much faster than some arbitrary slower rate it is being played. Rather, its comprehension tends to be limited by maximum word absorption rates nearing 300 words per minute (for the average listener).

Volume

Two of the three systems had integrated volume controls. That is, the user interface included a software-driven volume control. The third system used an inline volume control built into the headphone.

Speech Volume

Speech volume measurements were made using help messages to generate audio output, with the headphones provided by the vendors. Other headphones may produce different results.

The Hart eSlate volume was measured from Hart headphones with their in-line volume control.

• Lowest volume was nearly inaudible,
• Highest volume was 100 dB (controlled only by in-line volume control).

The inline volume control is difficult or impossible for many voters with neuropathy or other dexterity impairments to adjust. An additional reason that external, in-line volume controls are not in conformance with the VVSG is that a voter may inherit an overly low or overly high volume setting from the previous voter. This might make it too quiet and cause the second voter to miss initial messages. Likewise it may damage the hearing of the next voter if it is set too loud.

The Diebold TSX volume was measured from Diebold headphones with in-line volume control.

• Lowest volume is unintelligible and barely audible.
• Normal volume is 70 dB.
• Highest volume is 90 dB.
• Volume step size is approximately 10 dB per step and has 5 steps.

This maximum volume level is not in conformance with the VVSG range.

Despite the fact that the Diebold TSX has built-in volume control, with a non-inherited return to normal level for each new voter, Diebold supplied their test system with headphones using an in-line volume control. This oversight or misguided attempt to provide additional controls defeats the whole reason for requiring built-in volume controls with minimum, normal, and maximum levels and a reset to avoid inheritance problems.

When Diebold TSX units are used with headphones having in-line volume controls, they are not in conformance with the VVSG.

The Sequoia Edge volume was measured from their headphones with no in-line volume control.

• Lowest volume is unintelligible and barely audible.
• Normal volume level is approximately 75 dB.
• Highest volume is 100 dB.
• Volume step size is approximately 5--6 dB per step, and has about 10 steps.

The Sequoia Edge volume controls are more in line with what was intended by the VVSG, although the sound level meter we had for the testing was not able to read as low as the 20 dB lower limit defined in the VVSG.

The apparent reason for the lowest volume limit in the VVSG is to assure that the speech might be heard, by some voters, even if it gets set to the lowest level in the range.

Pauses

One concern was the long pauses in speech where none should have occurred.

Visual representation of speech showing pause. Transcript: "State Senator; District 35; Vote for 1" [almost 4 seconds of silence] "Please wait while the list of choices is read …"

Some audio does not contain enough prosodic pausing, which makes the interface hard to use when the voter cannot control the speech.

Visual representation of speech showing pause. Transcript: "Write-in help page. [short pause] 1. Turn the wheel to pick a letter. [short pause] 2. Press 'enter' to add the letter to complete the write-in turn the wheel until you hear 'accept' and press the 'enter' button. [short pause] To reject the write-in turn the wheel until you hear 'cancel' and press the 'enter' button. [short pause] To add a space between first and last names turn the wheel until you hear 'space' and press the 'enter' button press 'enter' to continue press help again for assistance." In about 20 seconds this audio interface presents 7 functions requiring 10 discrete actions, using pauses inconsistently if at all.

Concatenation Effects

Concatenation is a technique used by fixed vocabulary speech systems that speaks individually recorded words and phrases one after the other in order to create a complete utterance. Since the individual segments were not recorded in any context, concatenated speech can sound strange and disconnected. We found such effects in these systems. In addition, the volume of some individual words or phrases varied, which some users found additionally confusing.

Alternative Language Concerns

We tested the three systems for their ability to be used in Spanish and Chinese, by people with or without disabilities. In general, our findings may be somewhat limited by the specific ballot definitions loaded onto the three machines. In addition, without an opportunity to investigate the process for creating a non-English interface, we could not be sure if any deficiencies were the result of the technical capabilities of the machines themselves or of drawbacks in the content of the ballot definition.

Here are our findings:

Only the Hart eSlate rendered Chinese characters on the screen. The other two systems only displayed an empty square graphic for each missing character.

During the language selection task, the Sequoia Edge used an incorrect word (possibly the word for "Korean") as the audio equivalent of "Chinese".

No system contained a complete and accurate Chinese translation for any of the propositions on the ballot, either visually or auditorially. The propositions were abbreviated, and the phrases were not well chosen.

The Diebold TSX had placeholders for most of the spoken Chinese and Spanish, with no real content in those languages at all, making it impossible for a blind voter preferring non-English to vote independently.

The Chinese instructions for using the Hart eSlate did not clearly explain the keys and how to use them.

Incorrect or misleading Spanish words and phrases were common. For example, the Spanish equivalent of "Race 1 of 7" on the Hart eSlate was incomprehensible.

The Hart eSlate uses two different words for "select" in Spanish.

The introductory text and write-in instructions did not match the Spanish audio in the Diebold TSX.

The audio interface for the language selection task on the Sequoia Edge is only in English, making it impossible for a blind non-English user to select the alternate language.

Additionally, it was extremely easy for English voters to inexplicably find the Sequoia Edge system in the initial language menu. This has happened so often that it should be considered to be a bug or design flaw, regardless of what advantages the vendor may offer for its frequent occurrence. In the case of one test voter, she didn't realize that she had accidentally managed to select Chinese language and became totally lost and confused.

VVPAT Accessibility for Verifying Print Record

Most voters with severe visual impairments will not be able to privately verify that the printout is an accurate representation of their intended vote selections. Some voters with normal eyesight may not be able to read the VVPAT printout if they cannot position themselves where they would be able to see the paper printout.

During our testing, there were some of the audio voters who came to the mistaken conclusion that the VVPAT printer was actually allowing them to read back and verify the printed vote record directly from the VVPAT paper tape. They said that they were led to believe this because the audio messages of the systems seemed to be saying that, and because, in some cases, the audio interface began reading aloud the ballot verification selections while the printer was printing out their paper record.

Total In-booth Voting Times

In general voters with visual impairments take from 3 to 4 times as long to vote as voters without disabilities. Some of the voters with visual impairments took over an hour to complete their vote.

Voting Accuracy

Accuracy for voting systems is the ability of the system to capture, record, store, consolidate, and report the specific selections and absence of selections, made by the voter for each ballot position without error.

For the purposes of this test, the error rate is based on contests that the voter purposely decided to not undervote and in which the voter's announced selection intentions did not match the actual selections cast on their final ballot. This error rate has both a user selection error component and a machine recording error component. For the purposes of this review, it was assumed that the machine's vote recording error rate is essentially zero.

When the voters chose to skip or undervote a race on purpose, that was not considered an error. Similarly, if the voter got too frustrated with the voting task and decided to skip voting on the later portion of the ballot, those undervoted races were not considered to be errors for the accuracy score. Actually, when the voter skipped the final races, that was considered to be the voting system's failure, not a user failure.

Write-in names were not considered to be in error if a reasonable elections official would be able to discern the name that the voter likely intended. Misspellings, lack of separator blanks between first and last names, repeated or missing letters, and even improperly imbedded digits or special characters were considered acceptable, as long as the intended name could be discerned. Elections officials normally have to be tolerant of misspelled or partially illegible write-in names.

In this test, errors also do not include the number of times a voter inadvertently attempted to overvote, unintentionally undervoted a ballot, changed their selection in a race or was unsure of the next step in a process, assuming that these conditions were corrected before their ballot was cast.

In this testing, the voters were asked to announce each of their vote selections, as they made each selection. In the case of some voters, they announced their votes by signing their choices to an interpreter, who announced their choice out loud. The spoken announcements of intended vote selection were picked up with a microphone and recorded, along with any audio output from the voting machine, on the same DVD recorder that was being used to record the three camera views in each voting booth. One of the three cameras in each voting booth was recording the voting machine display screen.

In this manner, the voter's intended ballot selections and the selections actually made on the voting machine were recorded for later analysis.

Additionally, the VVPAT paper roll from each of the voting systems was collected and archived, as a backup method for checking the votes. Because the VVPAT records are designed to maintain voter privacy, it would be very tedious to try to correlate the paper trail print outs with the actual voter for purposes of accuracy scoring or checking .It is easier to monitor the votes and score for accuracy by watching and listening to the DVD audio/video records. Because of the extremely tight time schedule for data analysis and writing of this report, there will not be time for the accessibility review team to analyze the approximately 135 DVD records of the test sessions. This will have to be done at a later date.

During the testing by voters, our pollworker team members observed few voting selection errors that were made by the voters.

At the current time, only a very rough estimate for the overall voting error rate is possible. We analyzed parts of 51 sessions in which 137 discrete choices were made and verbalized so that we were confident as to what the voter's intention was. We discovered 9 errors, giving a rough error rate of approximately 6.6 percent. Confidence in this accuracy estimate should improve as more of the DVD record is analyzed to improve the quality of the voting error rate data.

Additionally, because our total number of subjects does not represent a very large sample set, readers should be cautioned against assigning any great statistical relevance to any quantitative accuracy score based on this testing.

Perhaps the most useful finding for the voting accuracy of this test is simply the qualitative conclusion that the general error rate for the voting of most subjects was low.

The Audio Interface in Complex Tasks

Audio interface users found the more complex tasks such as reviewing, making changes, write-in voting, verifying, and casting the ballot very confusing. Some of the confusion may be due to the nature of those processes themselves. Although moving from race to race and making selections is somewhat intuitive, the acts of reviewing, modifying, verifying, and casting are not. It was difficult for voters using the audio interface to understand and navigate through these abstract modes. Sighted voters often had visual cues about the mode they were in, such as a page header saying "Ballot Review". Although this phrase may have been spoken to the audio interface user, it was only spoken once per screen and thus did not serve as a mode reminder.

Write-in Concerns

The Hart eSlate and the Sequoia Edge write-in functions allowed the voters to spell out a candidate's name by selecting one letter at a time from an alphabetical list of letters and other choices. Neither the eSlate nor the Edge system offered the audio voters enough guidance when they initially entered the write-in function. The process of moving back and forth in the lists was very tedious. In the case of the Edge system, it took a minimum of 130 keystrokes to write-in "John Smith". For the eSlate, it took 103 single clicks of the wheel, or quite a few less, if the wheel was spun through several clicks for each move.

The Diebold TSX system allowed the voter to spell out a candidate's name by multiple presses of the keypad keys with letters associated with the keys as they often are for text message entry on a telephone keypad. For some voters this was familiar and easy to use. For others, it was cognitively challenging. Many blind voters have no idea of which letters are associated with which telephone keypad keys.

The Diebold TSX write-in of "John Smith" took a minimum of 32 keystrokes. The low number of keystrokes may be at least partially offset by the higher cognitive load. That is, although the telephone keypad interface was less physically demanding, it required more cognitive processing by the user to identify and select which key was associated with which bank of letters.

In general, both of these approaches to write-ins were challenging and tedious for most of the voters.

Mispronunciation

The audio ballot contains some candidate names mispronounced. One example is "Darryl Issa", which is pronounced "Ice –a".

Confusing Wording

In some cases, messages caused confusion because they could be interpreted either as an instruction or as an indication of the current status. For example, "You are finished voting" can be interpreted by the voter to mean that you are done voting and can walk away. Alternatively, it might be a prompt, asking you if you feel you are done voting and ready to cast your ballot, or merely that you have reached the end of the ballot. Another example is the use of the phrase "Review Choices" in isolation: does this mean that the following are your choices, or that you are supposed to review your choices at that moment?

Orientation, Help and Instruction Messages for the Audio Interface

All three systems provide "Help" functions, although they are not identically implemented or activated. The Hart eSlate and Sequoia Edge provide separate Help buttons; the Diebold TSX uses its telephone-style keypad.

There were four sets of concerns regarding orientation, help, and instruction messages for audio users. First, the amount of information presented at one time was often overwhelming. Without a way to pause and resume the audio help function, users were not able to reflect on the meaning of the information and absorb it at their own pace. Second, the information sometimes covered several complex sequences. For example, the Hart eSlate ballot review help function tells the user that he/she is in review mode, how to go back to a race, how to make a change, then how to return back to review mode. Third, some users found the audio pace too fast and without sufficient pauses. Fourth, the audio interface carries over some of the visual interface elements, such as the concept of a "page". The help information provided on a page basis is not easy for audio interface users to apply.

Privacy Concerns

Eavesdropping on the Visual Display

Voters with low vision or others who use the magnified or high contrast text on their displays may be reasonably concerned that eavesdroppers might easily compromise their privacy from across the room or through a window or open doorway. Even without magnified or high contrast text options, the ballot on the screen may be quite exposed to other people walking by, or standing around behind the voter. This is especially a problem if the voter is sitting in a chair, rather than standing and blocking the display screen with their body, or if the VVPAT printer adds enough width to the polling station that the voter can not effectively block both the printout window and the visual display screen with their body.

Blanking the Visual Display

Some voting systems have a control to allow the voter to quickly hide their displayed ballot, by blanking the visual display, in case they are worried about eavesdroppers or if they wish to ask for help from a pollworker.

Privacy Shields

Given the use of large print by some voters, privacy from eavesdropping is an important feature. The privacy panels of both the Sequoia Edge and the Diebold TSX were small fold-out panels that may not adequately shield the screen from view by eavesdroppers.

Because the left hand privacy panel of the Sequoia Edge would block viewing of the VVPAT viewing area for a voter sitting down to vote, a large window was cut in the left privacy panel. This reduces the utility of the privacy panel.

Vulnerability to Quick Glances or Camera Shots

With the text ballot displayed clearly on a text screen, a quick glance might give an eavesdropper a lot better idea of the voter's ballot selections. Additionally, the prevalence of compact digital micro cameras and cell cameras, means that only a brief exposure of the voter's visual display screen may leave their ballot vulnerable to capture by a camera.

Eavesdropping on the Audio Voting with Radio Receivers

Although not designed to do so, many electronic devices with an audio interface for the user are also unintentionally broadcasting the same audio signal over the radio. In our access review testing, a simple shortwave receiver was used to monitor some of the radio spectrum, to check for obvious audio transmissions from the voting systems (not for electromagnetic interference). Although this radio listening check was brief, informal, and not using sophisticated radio monitoring equipment, it was surprising to find that the Hart eSlate DAU, whose audio interface is always active, was transmitting its audio messages strongly across a broad AM band that peaked around 6.75 megahertz.

Eavesdropping on Audio Voting with Recorders and Wireless Bugs

Modern miniature recorders or radio transmitters could actually be hidden on or inside a set of headphones or in its cable connectors, to permit eavesdropping on the headphone audio.

Alternatively, a simple compact solid state audio recorder or a wireless transmitter with a microphone pickup might be disguised or hidden close to the area on the voting unit that is used to hold the headphones, when not being worn by a voter. In the case of the Hart eSlate or Diebold TSX, a voter using the screen might be unaware that the headphones were simultaneously talking and being overheard or picked up by the recorder or wireless bug.

Personal Safety Concerns

The thin plastic privacy shields used on two of the systems project outward and upward from the screen surfaces. The outer edges of the shields are sharp, and pose a hazard to people who cannot see them and people who cannot fully control their hand or arm movements.

Public Health and Sanitation

There are public health concerns about the public sharing of headphones, touch screens and controls, sip and puff tubes, and other parts of a voting system that may not be kept in a sanitary condition. No headphone covers were provided, and not all of the pollworker materials provided enough information about maintaining sanitary conditions.

Reliability Concerns

Both the Sequoia Edge I and II hung whenever the keypads were plugged into the wrong jacks on the backs of the units.

At least seven times during the access testing, the Sequoia Edge II emitted unintelligible audio chattering; this problem was later diagnosed as a loose keypad plug.

Sequoia Edge and its card programmer did not work together whenever they were not initialized with the proper date and time.

When the Diebold printer was bumped on its bottom by a voter's knee, it stopped working.

On another occasion, the Diebold VVPAT paper jammed and had to be cleared.

On one occasion the Diebold keypad malfunctioned and had to be replaced.

The Hart printer would not work when the tablet was not seated completely in the stand. This occurred when the eSlate tablet was pulled out to plug in the audio or tactile switches.

When the Hart eSlate was removed from its stand to test curb-side voting, it turned itself off and refused to come back on while out of the stand. After being installed back in its stand, the eSlate indicated that it didn't have any battery.

Poll Worker Training and Assistance Concerns

Accommodating some of the voters required ad hoc changes and adjustments to the machines and their immediate surroundings, especially for voters who used wheelchairs and had impaired dexterity. In addition, the orientation to the audio interface for blind and low vision voters requires significant understanding of the needs and expectations of those users. This raises innumerable concerns about the types of training offered to typical pollworkers and the materials used for training.

Near Term Mitigations for Major Findings

The purpose of this section is to describe possible improvements to the concerns stated above. All of these options are actions that could readily be taken by poll workers or other election officials, or by manufacturers, using materials and techniques we believe they may have at hand. The sources include both user suggestions during testing and the expertise of the project staff.

We do not assert any specific timeframe in which these actions could be taken, but believe them to be easy enough to be put in place soon.

We want to clearly express that there are also longer-term improvements for all of the concerns we raised above, but that these would constitute a significant re-design of the systems and are therefore excluded in this section.

In some cases all of the options should be used; in others only one option is necessary.

Improving Physical Access to the Voting Machines

• Separate the machines from their current stands and use a regular table instead. Many large folding tables would meet the height, floor space, and other clearances necessary.
• Procure a mounting/positioning framework that would meet the dimensional clearance and adjustability requirements and that could be used by all the machines. Specialty furniture suppliers have many such adjustable frames used for therapy or other purposes.

Obviously any solution should be sturdy enough to provide a stable platform.

Other tables or stands would also make it easier to use free-standing privacy curtains or panels that would not block the voter's arm movements and approach to the voting units. Using less confining privacy shields should also make it possible to assure adequate lighting on any print instructions and on the VVPAT printout.

• Provide pollworker training and reference materials with suggestions for physical accommodation alternatives, including how to change the height or angle of the system, how to remove the voting tablet unit (from Hart or Diebold) and place it tilted up in the voter's lap or how to work with a voter who wants to try alternative controls such as keypads, jelly switches, or sip and puff switches.
• Double-stick tape, Velcro straps, and similar supplies can go a long way towards helping a voter over many physical access hurdles such as dexterity, stamina limitations, or gripping problems. These types of low tech accommodations should be documented and added to the pollworker training and reference materials.

Provide Chairs and Flat Surfaces for Voters

Many voters with disabilities have difficulty standing for the long booth time some will experience. Also, some voters will need to sit down, with an audio keypad in their laps. The current stand for the Hart eSlate is so low that a typical voter must be sitting to operate its controls.

The parking cradle that normally holds the Diebold keypad is at a height that makes it very awkward to use for a standing voter who is tall, and the keypad is too awkward to hold for long times while voting. Thus, many of the Diebold TSX voters will also need to vote sitting down.

Some blind users with ballot marking notes in braille or other forms may also need to sit down to vote and may even need an adjacent table or flat surface on which they can place and read their voting notes. Other voters may need a flat surface to hold their purse, bag, or other materials while they are voting.

Improving Manual Dexterity Accommodations

Voter Access Cards

• Train pollworkers to offer assistance to voters in handling, inserting, and retrieving access cards, but to allow them to perform these tasks independently if they prefer.

Physical Controls

No near-term mitigations are possible for these concerns.

Touch Screen Controls

• Train pollworkers to adjust the height and angle of the touch screen to match the most effective range and reach of users with impaired dexterity.

As we did not evaluate the ballot design tools for these systems, we are unable to say whether those tools can address this problem either directly or by changing target size and/or shape and the distance between targets.

Improving Interface Connectors

• Provide a physically compatible pair of extenders or adapter cables to allow the jelly switches or other input control switches to be connected to the Sequoia Edge keypad. It would be helpful if color coded labels were added to make it obvious which plug goes in which jack.
• Provide the Sequoia Edge with an audio extender adapter or extension cable that would allow the headphone jack to be connected to audio headphones or assistive listening devices whose plugs are not slim enough to fit down into the Edge audio jack housing channel.
• Cover jacks with a masking label or block with a filling plug cap to prevent Sequoia keypad plugs from being plugged into the wrong jack on the back of the Edge unit. Also, the "Audio" label over the correct jack area should be changed to "Audio Keypad".
• Replace the jack for the keypad with an adapter with at least 3 micron gold plated contacts.
• Add stereo audio-type cables with right angle male plugs for the tablet jack ends and female 1/8 inch jacks on the other to the Hart eSlate access connection. This will eliminate removing and reinserting the tablet in the stand every time someone wishes to plug in a headphone or dual-switch input. The headphones or dual-switch plugs can be either straight-in or right angle plugs. Finally, the extension cables should be at least six feet long, with an adequate storage method to prevent tangling.

Improving the Visual Display

Single Race per Page

• Limit ballot layouts to one race per screen.

As we did not evaluate the ballot design tools for these systems, we are unable to say whether this option is near-term or not; we include it here for the sake of clarity.

Eliminate In-Screen Scrolling

• Even in magnified mode, all content and certainly all controls should be visible at all times. There may need to be an exception for races with many candidates.

As we did not evaluate the ballot design tools for these systems, we are unable to say whether this option is near-term or not; we include it here for the sake of clarity.

Accelerate Video Performance

• Eliminate long screen write times that confuse the user.

We are unable to estimate whether this is feasible near-term; we include it here for the sake of clarity.

Improving the Touch Screen Controls

Reduce Touch Screen Parallax

Voters positioned low in chairs may experience an optical parallax effect that makes the screen's visible buttons appear to be slightly higher than the touch sensing area for that visible button.

As we did not evaluate the ballot design tools for these systems, we are unable to say whether those tools can address this problem either directly or by increasing target size and the distance between targets.

• To compensate for parallax, voters should be instructed to be careful to try to touch only in the middle of the visible buttons.
• Alternatively, on systems that will primarily be operated by a voter sitting down, the touch screen should be recalibrated for a seated voter.
• In either case, poll workers should be informed of this effect and that the best way to counter it is to position the voter so that eye and hand are as perpendicular as possible to the touch screen.

Improving Non-Touch Screen Input Controls

• Make braille and large print materials available with clear reference cards that identify the names of each of the keys. This might be particularly helpful where the controls have no braille labels or function identifiers.

Reduce Fatigue from Audio Interface Use or Arduous User Control Input Actions

• Make sure that a chair is available to help make the voter comfortable and minimize fatigue during tedious voting on long ballots, for voters using the audio interface, or where many user input actions must be performed. Train poll workers to expect some long voting times, and to offer these chairs.

Improving the Speech Interface

All of the following solutions should be adopted:

• Use only speakers with clear, unaccented voices to record any audio ballots.
• Use proper audio recording studio equipment and procedures.
• Normalize speech volume levels to assure that all messages have the same volume level.
• Use a pitch queue tone or other technique to help voice talent keep the pitch of their voices normalized.
• Test audio ballots with a thorough walkthrough on an actual voting system.
• Rescript the text for help, instructions, and prompts; and record new audio for the systems to improve their accessibility and usability. According to at least one of the vendors, the "fixed" messages that are not part of the variable ballot definition can be changed without having to recertify the voting system.

Speech Rate

No near-term mitigation is possible for speech rate.

Volume Control

• Provide headphones with a large, easy to manipulate control, preferably one with a light but tactilely discernible index or click stop to indicate when it is set for the middle of the volume range. If possible, use volume controls that also adjust to a lowest level that is not zero, to attempt to meet the minimum volume requirement.

Improving Alternative Language

• Check both the audio ballot and the visual ballot for translation accuracy, proper pronunciation of candidate names, and absence of confusing or inconsistent wording.
• Perform complete walkthroughs of the ballot with both the audio and the visual interfaces.

Improving VVPAT Accessibility

• For voters with severe manual dexterity impairments, it may be necessary for a pollworker to prop the privacy cover of the VVPAT printer window open before the voting session begins. This may need to be done with a piece of adhesive-backed tape. For some voters, the fresnel lens of the VVPAT may also need to be propped open.
• Provide better lighting for the text on the VVPAT printout and printed instructions. In the case of the eSlate, it may help to fold back the glare guard flap at the top of the unit.
• Clearly inform all audio voters that the VVPAT is not being read back and verified directly from the printout.
• Warn Sequoia Edge audio voters that the system will print out their VVPAT without giving them an opportunity to verify it themselves or have anyone else verify it for them.

Addressing Total In-Booth Voting Times

Because voters using the audio interface to vote may take 3 to 4 times longer than other voters to vote, chairs should be provided for the voters, and pollworkers should be trained to expect longer in-booth times, typically half an hour to an hour.

Improving Voting Accuracy

• Establish standards and train elections officials that process write-in ballots to be more tolerant of misspelled or partially illegible write-in names, including names without separator blanks and sometimes even names with repeating characters.

Improving the Audio Interface in Complex Tasks

Verification

• Warn Sequoia Edge audio voters that the voting system will not offer them the opportunity to check the VVPAT printout or have someone else check it for them.
• Write-in Concerns: Provide a braille and large print summary of which letters are associated with which keys of the Diebold telephone-style keypad. Note: Although this might fit on the underside of the keypad, being on the underside may not be desirable as its inversion may cause confusion about which labels go with which keys.
• Revise the write-in character list to remove any other special characters that are not truly necessary for name write-in.

Ballot Verifying and Casting Concerns

No near-term mitigations are available for these concerns.

Confusing Wording and Use of Grammar

• Check both the audio ballot and the visual ballot for proper pronunciation of candidate names, freedom from confusing wording, and bad grammar.
• Perform complete walkthroughs of both the audio and visual ballots.

Orientation, Help, and Instruction Messages

• Rescript the text for help, instructions, and prompts; and record new audio for the systems.

Missing the Startup Orientation Help or Instructions

• Instruct the voter to put on the headphones and/or make any other necessary preparations before the voting session begins, so they will be ready to follow the initial messages of the voting machine.

Outreach to Inform Voters about Accessible Voting

• Produce and distribute informative materials that explain that accessible voting systems are available in the local polling places, whom they assist, and how they work. Circulate this information to organizations for people with disabilities, retirement homes, medical clinics, and other community organizations, via meetings, printed material and mailings, websites, phone lines, and to the media as press releases and public service announcements.

Information Available in Braille and Large Print

• Provide voter access cards with intuitive tactile and large print markings to aid in proper insertion orientation.
• Provide operating instructions and other usual information in braille and large print. Also post this information on the county's web site and elections info phone line.

Privacy Concerns

Eavesdropping on the Visual Display

• Assure that the accessible voting station is in an open corner or other area that has good wheelchair access but is not exposed to potential eavesdroppers walking or standing nearby or through a window.

Blanking the Visual Display

• For systems that do not have a blank-screen or hide-display mode, provide voters with a simple, opaque, flexible flap, hinged cover plate, or simple paper or cardboard sheet to allow voters to quickly cover the screen when necessary.

Privacy Shields

Privacy curtains

• Use softer curtains or other privacy screens where possible to prevent possible injury and interference.
• Provide fully surrounding privacy for all voters, or at least for voters using a magnified interface.
• Use curtains that do not come all the way down to the floor, but stop around table top level so that voters in wheelchairs can back out through surround curtains easily and independently.

Vulnerability to Quick Glances or Camera Shots

• Plan out the polling place so that it is not possible to view any tablet or display from any common area.
• Encourage voters to tilt their tablet or display unit down (if possible) to a more horizontal position, unless this might cause more multiple point touching problems (by hand, jewelry, etc.) on touch screen displays.
• Do not allow cameras or cell phones with cameras to be used in the polling place.

Eavesdropping on the Audio Voting with Radio Receivers

• Do not allow the use of radios and other nonessential electronic equipment in the polling area.
• Develop and use signage that informs voters and others at the polls of the ban on use of cameras and nonessential electronics in or around the polling place.

Eavesdropping on Audio Voting with Recorders and Wireless Bugs

• Inform pollworkers about the possibility of bugs and recorders, and train them to keep the area around the voting machine clear of odd gadgets, bogus headphones, laptops or any other nonessential electronics, or items that might contain electronics.

Personal Safety Concerns

• Warn voters about sharp edges. Cover the worst edges and corners with soft materials.

Public Health and Sanitation

• Supply sip and puff controls with sanitary throw-away tube straws.
• Supply disposable covers for headphones. Note: disposable headphones are not an acceptable option as they would cause significant distortion in the audio speech reproduction.
• Provide pollworkers with the proper materials and vendor-authorized sanitizing procedures for sanitizing touch screens, tactile controls, and other parts of the systems.

Reliability Concerns

For mitigations affected by access interface cables (including some eSlate printer problems), see the section on mitigation for Access Interface Connections.

Pollworker Training and Assistance Concerns

• Correct errors and omissions in the vendor-supplied pollworker training and reference materials
• Provide training videos and other materials that show pollworkers working with voters with disabilities.
• Provide a list of frequently asked questions/answers about usability and accessibility to every pollworker.

Conclusions

Accessible technology is a mature field, with over thirty years of practical success in making information and communications technologies usable by people with disabilities. Inclusive design solutions are well tested, reliable, and inexpensive. Information about them is publicly available and non-proprietary. Every day more accessible products enter the market, driven by technological improvements, market demand, and policy insistence.

As a result of this access review, we have concluded that the three tested voting systems are all substantially noncompliant when assessed against the requirements of the HAVA and specified in the 2005 VVSG guidelines.

This report has documented these accessibility concerns and offered options for short-term mitigations for near term elections, and also offered system design changes and other longer-term mitigations possible for voting systems.

The usability and accessibility of voting encompasses far more than just the design of the voting machines. Election officials should analyze voting as an integrated system of technologies and social practices. For example, running public service announcements that include a description of the accessible voting system at the polling place would improve outreach to voters with disabilities and prepare them for their experience at the polls. Providing pollworkers with more exposure to people with disabilities as well as more training in how to use the accessibility features of voting technologies could open up the voting process to people with disabilities. Working with grassroots organizations, as some counties do already, can aid local election officials in their efforts to improve their inclusiveness and maintain efficiency at the same time.

This project looked primarily at the voter interface. There should be formal, rigorous analyses of the other interfaces in voting systems, especially the ballot design interface and the interfaces used to set up, test, and administer the individual machines as well as any collection and tallying interfaces.

The EAC and NIST are in the process of developing accessibility and usability testing methodologies and certification practices. We can see from our experience that this area clearly needs a lot of development. We hope this report will be useful in driving that work.

Clearly, it will be important to use a balance of expert heuristics and user testing. As long as the design of voting technologies remains more art than science, objective testing methodologies will need to be supplemented with insight-based analytical techniques.

Much remains to be done to open up the channels of communication regarding usable and accessible voting. For example, voting equipment manufacturers and election officials could benefit from a greater understanding of how people with disabilities actually use voting equipment. To this end the Secretary of State might consider putting online some of the video of the walk-throughs and the user voting sessions.

Most of all, we want to express how important it is to remember that we are in the early days of electronic voting systems. As with all new technologies, innovation and reliable functionality do not yet have a stable relationship. As a technology driven by the needs of public policy, voting technologies are subject to political as well as technological and economic storms. The best way to weather those storms is to build trusting collaborations among manufacturers, public officials, experts, advocates, and testers in a manner that is open to the public and is communicated clearly. The explicit creation of a 'community of practice' would go a long way towards guaranteeing progress while reducing unnecessary confusion and concern.

We are grateful for the California Secretary of State for giving us the opportunity to evaluate these systems, not only because this report may help the Secretary make near-term decisions about the systems themselves, but, more importantly, some of the information within it can be shared with the larger community.

Acknowledgements

This study was undertaken with the assistance of many people, without whom it could not possibly have proceeded. We would like to take this opportunity to thank them.

The Office of the Secretary of State
Lowell Finley, Deputy Secretary of State, Voting Systems Technology and Policy
Deborah O'Donoghue
Ryan Macias
Jason Heyes
Miguel Castillo
Michael Lakey
Jane Howell
Nancy Arceo

Matt Bishop of the Department of Computer Science, University of California, Davis, Principal Investigator

The management and staff of the grassroots organizations who helped recruit users:

• California Council of the Blind
• NorCal Center on Deafness
• Resources for Independent Living
• Society for the Blind

The users, who were much more than subjects:

• Keith Alcombright
• Nancy Arceo
• Beth Berenson
• Nancy Bingham
• Leslie Birchell
• Regina Brink
• Lois Brooks
• Heather Campbell
• Warren Cushman
• Charlene Davis
• JDD Doran-Jammer
• Paula Douglas
• Claude Farinha
• Shirley Farinha
• Bob Foss
• Ramona Garcia
• Jan Garrett
• Sally Garza
• Rose Gasser
• Frank Gaul
• Ralph Gingrich
• Norma Guerrero
• Sheela Gunn-Cushman
• Don Lee Hanaumi
• Kristene Hunsinger
• Ann Jacobson
• Bob Jones
• Stepheny Kersten
• Ann Kysor
• Dan Kysor
• Hugh Lafler
• Michael Lakey
• Connie Leblond
• Kathy Ludwig
• Joanie Patche
• Paul Patche
• Alfonso Nguyen
• Eddie Rea
• Jean Riker
• George Sharp
• Garth Sturgill
• Leslie Thom
• Maureen Thompson
• Dick Yee
• Russ Zochowski

Stanley Chan and John Bartle of Onetake Productions, our videographers
Deborah Runyan, for consulting, data processing, and report editing
And finally, to Scott Luebking, whose dedicated assistance and insight were essential to the planning and implementation of this study.

References

Burton, D & Uslan M. (2002, November). Cast a Vote by Yourself: A Review of Accessible Voting Machines, Access World, 3(6), November, 2002.

CalTech-MIT (2001). July, Voting: What is, What Could Be, https://www.vote.caltech.edu/ .

Center for Independence of the Disabled in New York; Fields, C. V., (2003), Voting Technology for People with Disabilities.

Election Assistance Commission, Voluntary Voting System Guidelines, Volume I Version 1.0 at §§ 3.1 (2005), available at https://www.eac.gov/VVSG%20Volume_I.pdf .

Federal Election Commission (2003). Developing a User-Centered Voting System.

Federal Election Commission (2003). Usability Testing of Voting Machines.

ISO/TS 16071 (2003). Ergonomics of Human-System Interaction Guidance on Accessibility for Human-Computer Interfaces.

ISO 13407 (1999). Human Centered Design Processes for Interactive Systems.

Jones, D. (2002). Voting Systems Standards: Work that Remains to be Done, Testimony before the Federal Election Commission, Washington D.C., April 17, 2002.

Jones, D.W. (2001). Problems with Voting systems and the Applicable Standards, Testimony before the U.S. House of Representatives.

Mercuri R. (2002). Humanizing Voting Interfaces, Presentation to the UAPA Conference, Orlando, FL.

Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, p. 81-97

Norden, L., The Brennan Center for Justice Voting Technology Assessment Project, (2006), The Machinery of Democracy: Voting System Security, Accessibility, Usability, and Cost, www.brennancenter.org.

Pierce, K., Cook County State's Attorney's Office, (2005), Accessibility Analysis of Four Proposed Voting Machines.

Quesenbery, W. (2001). Voting for Usability: A Backgrounder on the Issues, Talk presented at TECH*C0MM 2001 in Washington DC.

Roth, S.K. (undated). Human Factors Research on Voting Machines and Ballot Designs: An Exploratory Study.

Section 508 Electronic and Information Technology Accessibility Standards (2000). Architectural and Transportation Barriers Compliance Board, 36 CFR Part 1194.

Runyan, N.H. (2007). Improving Access to Voting Systems.

Tadayoshi, K.; Stubblefield, A; Rubin, A.D. & Wallach, D.S. (2003). Analysis of an Electronic Voting System.

Appendices

1. Background of the Authors
2. Glossary
3. VVSG Conformance Notes
4. Timing Results for Phases of the Voting Process
5. Comments on All Systems from Study Participants
6. Subject Recruiting and Screening
7. Intake Form
8. Post-Test Form
9. Ballot Design Summary
10. Testing Script

Background of the Authors

• Noel Runyan

  With his degree in Electrical Engineering and Computer Science, Noel Runyan has been working in human-factors engineering for over 35 years, primarily developing access technologies for helping persons with visual impairments use computers and other electronic devices.

  During the 5 years he worked for IBM, he was involved in the design and testing of the security systems for both BART ticket machines and ATM credit card systems.

  After starting his own company to supply access technologies, he designed and manufactured the Audapter speech synthesizer, to enable computers to talk to visually impaired users. He also authored the EasyScan, BuckScan, and PicTac programs that made it easier for visually impaired users to read print books, identify dollar bills, and convert print pictures into raised line tactile drawings.

  To help their visually impaired customers access and make use of computer systems, the author and his wife, Deborah, have personally built over 500 custom-integrated personal computers with speech, braille and/or large print interfaces.

  More recently, he has been involved in the development of talking internet radios and talking pill bottles and other medical equipment for persons who have difficulties reading print labels and displays.

  For several years, the author has been studying and testing accessibility features and the usability of all the major voting systems used in this country. He has worked with the Santa Clara County Voter Access Advisory Committee, voting rights advocates, and manufacturers to make voting systems more accessible for all folks with disabilities or special language needs. In February of 2007, he published Improving Access to Voting, A Report on the Technology for Accessible Voting Systems.

• Jim Tobias

  Jim Tobias has thirty years experience in technology and disability in both the public and private sectors. He began his career at Berkeley's Center for Independent Living, was a Member of Technical Staff at Bell Labs and Bellcore, and their principal liaison with the Baby Bells on accessibility, aging and education. He is now President of Inclusive Technologies, a technology and marketing consulting firm for large information and communication technology companies. Clients have included AOL, Cisco Systems, HP, IBM, Microsoft, Panasonic, Verizon and Xerox.

  Jim was appointed to the Access Board's Telecommunications Accessibility Advisory Committee responsible for drafting Section 255 regulations, and the FCC's Consumer/Disabilities Technical Advisory Committee. He was been re-appointed to the FCC's Consumer Advisory Committee for 2005-2007. He is co-Chair of the Access Board's Telecommunications, Electronic and Information Technologies Advisory Committee, currently revising the Section 255 and 508 Standards and Guidelines.

  He is Chair of the Alliance for Telecom Industry Solutions' Interactive Voice Response (IVR) Accessibility Forum. He is a member of the International Standards Organization's JTC1 Special Working Group on Accessibility. He coordinates the accessibility component of the Usability Professionals Association's World Usability Day.

  Mr. Tobias and Inclusive Technologies have developed several accessibility projects: an innovative deaf relay service that integrated speech detection and text-to-speech; a network-based talking PIM for blind users; and a database-driven customized interface for voice mail and IVR accessibility. In addition, Inclusive Technologies performs market analysis and accessibility management.

Glossary

• Accessibility: Measurable characteristics that indicate the degree to which a system is available to, and usable by, individuals with disabilities. The most common disabilities include those associated with vision, hearing and mobility, as well as cognitive disabilities and alternative language needs.
• Accessible Voting Station: Voting station equipped for individuals with disabilities or alternative language needs.
• Alternative Format: The ballot or accompanying information is said to be in an alternative format if it is in a representation other than the standard ballot language and format. Examples include, languages other than English, Braille, large print, and recorded audio.
• Audio Ballot: a ballot in which a set of offices is presented to the voter in spoken, rather than written, form.
• ATI: Audio-Tactile Interface, Voter interface designed to not require visual reading of a ballot.
• CIF: Common Industry Format, Refers to the format described in ANSI/INCITS 3542001 "Common Industry Format (CIF).
• Claim of Conformance: Statement by a vendor declaring that a specific product conforms to a particular standard or set of standard profiles; for voting systems.
• Conformance: Fulfillment of specified requirements by a product, process, or service.
• Corrective Action: Action taken to eliminate the causes of an existing deficiency or other undesirable situation in order to prevent recurrence.
• COTS: Commercial off-the-shelf.
• Disability: With respect to an individual; a physical or mental impairment that substantially limits one or more of the major life activities of such individual.
• DRE: Direct-recording electronic.
• EAC: Election Assistance Commission, https://www.eac.gov .
• Early Voting: Broadly, voting conducted before election day where the voter completes the ballot in person at a county office or other designated polling place or ballot drop site prior to election day.
• Election Definition: Definition of the contests and questions that will appear on the ballot for a specific election.
• Election Officials: The people associated with administering and conducting elections, including government personnel and pollworkers.
• Electronic Voter Interface: Subsystem within a voting system which communicates ballot information to a voter in video, audio, or other alternative format which allows the voter to select candidates and issues by means of vocalization or physical actions.
• Electronic Voting Machine: Any system that utilizes an electronic component. Term is generally used to refer to DREs. See also voting equipment, voting system.
• Electronic Voting System: An electronic voting system is one or more integrated devices that utilize an electronic component for one or more of the following functions: ballot presentation, vote capture, vote recording, and tabulation device.
• FEC: Federal Election Commission, https://www.fec.gov .
• FFBS: Full-face ballot system.
• HAVA: The Help America Vote Act of 2002, Public Law 107-252. Full text at https://www.fec.gov/hava/hava.htm .
• Human Factors (Ergonomics): "The scientific discipline concerned with the understanding of interactions among humans and other elements of a system, and the profession that applies theory, principles, data, and methods to design in order to optimize human well-being and overall system performance." (Source: International Ergonomics Association).
• Human-Computer Interaction: A discipline concerned with the design, evaluation, and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them. Also, a collection of behaviors and responses that occur between a computer and a human attempting to accomplish a task.
• IEEE: Institute of Electrical and Electronics Engineers, https://www.ieee.org .
• ISO: International Organization for Standardization, https://www.iso.org .
• ITA: Independent testing authority (private test labs).
• Marksense: System by which votes are recorded by means of marks made in voting response fields designated on one or both faces of a ballot card or series of cards.
• Multi-Seat Contest: Contest in which multiple candidates can run, up to a specified number of seats. Voters may vote for no more than the specified number of candidates.
• NASED: National Association of State Election Directors, https://www.nased.org/ .
• NASS: National Association of Secretaries of State.
• NIST: National Institute of Standards and Technology.
• OS: Optical Scan, System by which votes are recorded by means of marks made in voting response fields designated on one or both faces of a ballot card or series of cards.
• Overvote: Voting for more than the maximum number of selections allowed in a contest.
• PCOS: Precinct-count optical scan.
• Personal Assistive Device: A device that is carried or worn by an individual with some physical impairment with a primary purpose of helping to compensate for that impairment.
• Privacy: The ability to prevent others from determining how an individual voted.
• Residual Vote: The total number of votes that cannot be counted for a specific contest (e.g., overvoting a contest, failure to cast ballot before leaving polling place).
• Risk Assessment: The process of identifying the risks to system security and determining the probability of occurrence, the resulting impact, and safeguards that would mitigate this impact.
• Rolloff: The difference between number of votes cast for contests in the higher offices on the ballot and the number cast for contests that are lower on the ballot. It is sometimes referred to as voter fatigue.
• T-Coil: Inductive coil used in some hearing aids to allow reception of an audio band magnetic field signal, instead of an acoustic signal. The magnetic or inductive mode of reception is commonly used in conjunction with telephones, auditorium loop systems and other systems that provide the required magnetic field output.
• TGDC: Technical Guidelines Development Committee.
• Touch Screen Voting Machine: A voting machine that utilizes a computer screen to display the ballot and allows the voter to indicate his or her selections by touching designated locations on the screen.
• Undervote: Occurs when the number of choices selected by a voter in a contest is less than the maximum number allowed for that contest or when no selection is made for a single choice contest.
• Usability: Effectiveness, efficiency and satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment. Usability in the context of voting refers to voters being able to cast valid votes as they intended quickly, without errors, and with confidence that their ballot choices were recorded correctly. It also refers to the usability of the setup and operation in the polling place of voting equipment.
• Usability Testing: Encompasses a range of methods that examine how users in the target audience actually interact with a system, in contrast to analytic techniques such as usability inspection.
• Voter Verifiable Audit Record: Human-readable printed record of all of a voter's selections presented to the voter to view and check for accuracy.
• Voting Equipment: All devices, including the voting machine, used to display the ballot, accept voter selections, record voter selections, and tabulate the votes.
• Voting Machine: The mechanical, electromechanical and electric components of a voting system that the voter uses to view the ballot, indicate their selections, verify their selections. In some instances, the voting machine also casts and tabulates the votes.
• Voting Station: The location within a polling place where voters may record their votes. A voting station includes the area, location, booth, or enclosure where voting takes place as well as the voting machine.
• Voting System: Combination of environment, equipment, ballot, voters, and other persons (e.g., poll workers and election officials) involved in the voting process.
• VRA: Voting Rights Act of 1965.
• VSS: Voting System Standards, the Federal guidelines for voting systems, last revised by the FEC in 2002, freely available from https://www.fec.gov/pages/vssfinal/vss.html. Conformance to the VSS is a prerequisite for certification by some states.
• VVPAT: Voter-verified paper audit trail.
• VVPR: Voter-verified paper record.
• VVSG: Voluntary Voting System Guidelines.
• Write-in Voting: To make a selection of an individual not listed on the ballot. In some jurisdictions, voters may do this by using a marking device to physically write their choice on the ballot or they may use a keypad, touch screen, or other electronic means to enter the name.

C. VVSG Conformance Notes
C.1. VVSG Conformance for the Diebold AccuVote-TSX

Vendor: Diebold (DESI)

Device Model: AccuVote-TSX with AccuView Printer Module

Ballot Station firmware version: 4.6.4

Supported with Vote Card Encoder version: 1.3.2

3.1.2 Functional Capabilities

Section: 3.1.2.a.

The voting system shall provide feedback to the voter that identifies specific contests or ballot issues for which he or she has made no selection or fewer than the allowable number of selections (e.g., undervotes)

Conformance: Yes

Section: 3.1.2.b.

The voting system shall notify the voter if he or she has made more than the allowable number of selections for any contest (e.g., overvotes)

Conformance: N/A

Comments: Not needed, because it does not permit overvote selections.

Section: 3.1.2.c.

The voting system shall notify the voter before the ballot is cast and counted of the effect of making more than the allowable number of selections for a contest

Conformance: N/A

Comments: Not needed, because it does not permit overvote selections.

Section: 3.1.2.d.

The voting system shall provide the voter the opportunity to correct the ballot for either an undervote or overvote before the ballot is cast and counted

Conformance: Yes

Comments: Does not permit overvote selections.

Section: 3.1.2.e.

The voting system shall allow the voter, at his or her choice, to submit an undervoted ballot without correction

Conformance: Yes

Section: 3.1.2.f.

DRE voting machines shall allow the voter to change a vote within a contest before advancing to the next contest.

Conformance: Yes

Section: 3.1.2.g.

DRE voting machines should provide navigation controls that allow the voter to advance to the next contest or go back to the previous contest before completing a vote on the contest currently being presented (whether visually or aurally).

Conformance: Yes

3.1.4 Cognitive Issues

Section: 3.1.4.a.

Consistent with election law, the voting system should support a process that does not introduce any bias for or against any of the selections to be made by the voter. In both visual and aural formats, contest choices shall be presented in an equivalent manner.

Conformance: Yes

Section: 3.1.4.b.

The voting machine or related materials shall provide clear instructions and assistance to allow voters to successfully execute and cast their ballots independently.

Conformance: Partial

Comments: Many voters in the testing had difficulty learning to deselect any previously selected choice in a fully voted race, before trying to select another. Several had to seek pollworker assistance to clarify how to accomplish this.

As one typical voter said, "Needs instruction on how to change your vote."

Suggestions: Another voter suggested, "I would have a braille instructions sheet about making changes or modifying a vote."

Several voters asked for a braille and large print sheet that could identify the keypad keys.

Some voters asked to be able to read the help or instructions one sentence at a time, and be able to repeat any message they had just heard.

Another voter felt, "Help instructions should be more contextual."

Several blind voters who did not know which letters were assigned to which number keys on telephone keypads asked to have the system's initial write-in help announce a summary list of which letters are on each of the keys.

Judging by the areas where we observed the most frequent and frustrating challenges for voters, the processes where voters could most benefit from better help and instructions are usually in modifying vote selections, write-ins, and ballot verifying/casting.

Section: 3.1.4.b.i.

Voting machines or related materials shall provide a means for the voter to get help at any time during the voting session.

Conformance: Yes

Suggestions: In addition to help messages from the system, consider adding a Summon Pollworker option that would discreetly signal for help from a pollworker.

Section: 3.1.4.b.ii.

The voting machine shall provide instructions for all its valid operations.

Conformance: Yes

Section: 3.1.4.c.i.

The voting equipment should not visually present a single contest spread over two pages or two columns.

Conformance: Partial

Comments: It depends on proper ballot layout, because it permits multiple races per page if the ballot layout has it.

Suggestions: Proper ballot layout may need to be controlled by configuration of the county's ballot layout program and training of elections officials who do ballot design.

Section: 3.1.4.c.ii.

The ballot shall clearly indicate the maximum number of candidates for which one can vote within a single contest.

Conformance: Yes

Section: 3.1.4.c.iii.

There shall be a consistent relationship between the name of a candidate and the mechanism used to vote for that candidate.

Conformance: Yes

Section: 3.1.4.d.

Warnings and alerts issued by the voting system should clearly state the nature of the problem and the set of responses available to the voter. The warning should clearly state whether the voter has performed or attempted an invalid operation or whether the voting equipment itself has malfunctioned in some way.

Conformance: Yes

Comments: There is no audio feedback if the system has not yet begun talking.

Section: 3.1.4.e.

The use of color by the voting system should agree with common conventions: (a) green, blue or white is used for general information or as a normal status indicator; (b) amber or yellow is used to indicate warnings or a marginal status; (c) red is used to indicate error conditions or a problem requiring immediate attention.

Conformance: Partial

Comments: Yellow – Yes. Green – Yes. Blue – Yes. Red – No. Some color usage is not appropriate. There is no color use on the keypad.

3.1.5 Perceptual Issues

Section: 3.1.5.a.

No voting machine display screen shall flicker with a frequency between 2 Hz and 55 Hz.

Conformance: Yes

Section: 3.1.5.b.

Any aspect of the voting machine that is adjustable by the voter or poll worker, including font size, color, contrast, and audio volume, shall automatically reset to a standard default value upon completion of that voter's session.

Conformance: No (Using the vendor supplied headphones with its in-line volume control.)

Conformance: Yes (Using other headphones with no in-line volume control.)

Comments: Use of an in-line volume on the vendor-supplied headphones means that its volume control setting can be inherited from the previous voter and won't be automatically reset.

Suggestions: Provide headphones without an in-line volume control.

Section: 3.1.5.c.

If any aspect of a voting machine is adjustable by the voter or poll worker, there shall be a mechanism to reset all such aspects to their default values.

Conformance: No (Using the vendor supplied headphones with its in-line volume control.)

Conformance: Yes (Using other headphones with no in-line volume control.)

Comments: Headphone in-line volume control appeared to have no tactile detent for the center or normal volume level.

Suggestions: Although manual volume controls are inappropriate, if a headset inline volume control is used, it should have a tactile indicator for the center volume position.

Section: 3.1.5.d.

All electronic voting machines shall provide a minimum font size of 3.0 mm (measured as the height of a capital letter) for all text.

Conformance: Yes

Comments:

Overall minimum is 2.8 mm

Overall maximum is 10 mm

"card entry picture page": 5 mm to 3.8 mm

"insert card to begin voting": 5.7 mm to 4 mm

Language Selection Page Title: 4 mm to 3 mm

Language Selection Page Language Selection Button: 6 mm to 4.5 mm

Instruction Page Large Text: 10 mm to 5 mm

Normal Text: 5.6 mm to 2.8 mm

Official Ballot Races Large Text: 7 mm to 5 mm

Official Ballot Races Normal Text: 7 mm to 3.1 mm

Race Large text: 7 mm to 5 mm

Race Normal text: 5 mm

Candidate Large text: 7 mm to 5 mm

Candidate Normal text: 5 mm to 4 mm

Occupation Large text: 6 mm to 4.7 mm

Occupation Normal: 4.5 mm to 3.2 mm

Write-in Large text: 8 mm to 6.8 mm (on ballot page 7 mm)

Write-in Normal text: 4 mm to 8 mm

Judges & Propositions Large text: 7 mm to 4.8 mm

Judges & Propositions Normal text: 5 mm to 3.1 mm

Summary Page Large text: 7 mm to 5 mm

Section: 3.1.5.e.

All voting machines using paper ballots should make provisions for voters with poor reading vision.

Conformance: Partial

Comments: Printed text is too small and the fresnel lens magnifier distorts the image so bad that it was considered to be of no use by test voters with low vision.

Paper verification page: 2.8 mm unmagnified

Suggestions: Consider providing a much better magnifier or change to use a printer with wider paper, to fit a larger print font.

Section: 3.1.5.f.

The default color coding shall maximize correct perception by voters with color blindness.

Conformance: Yes

Comments: It is not easy to test this, as the brightness and other color aspects can help voters with color blindness to distinguish differences. One of the self-identified color blind subject said that he was not aware of any problems in this system's choice of colors.

Suggestions: The colors should be tested on subjects with severe color blindness conditions or subjected to objective testing.

Section: 3.1.5.g.

Color coding shall not be used as the sole means of conveying information, indicating an action, prompting a response, or distinguishing a visual element.

Conformance: Yes

Section: 3.1.5.h.

All text intended for the voter should be presented in a sans serif font.

Conformance: Yes

3.1.6 Interaction Issues

Section: 3.1.6.a.

Voting machines with electronic image displays shall not require page scrolling by the voter.

Conformance: Yes

Comments: Except for summary page.

Section: 3.1.6.b.

The voting machine shall provide unambiguous feedback regarding the voter's selection, such as displaying a checkmark beside the selected option or conspicuously changing its appearance.

Conformance: Yes

Section: 3.1.6.c.

If the voting machine requires a response by a voter within a specific period of time, it shall issue an alert at least 20 seconds before this time period has expired and provide a means by which the voter may receive additional time.

Conformance: Yes

Comments: No timeouts

Section: 3.1.6.d.i.

On touch screens, the sensitive touch areas shall have a minimum height of 0.5 inches and minimum width of 0.7 inches. The vertical distance between the centers of adjacent areas shall be at least 0.6 inches, and the horizontal distance at least 0.8 inches.

Conformance: Yes

Comments: Some voters felt that they were having to press this system's screen harder. It appears that they may actually have needed to press longer, rather than with more force. Usually, pressing harder increases a voter's touch dwell time.

Suggestions: Consider decreasing the minimum dwell time for accepting screen touches.

Section: 3.1.6.d.ii.

No key or control on a voting machine shall have a repetitive effect as a result of being held in its active position.

Conformance: Yes

3.2.2 Vision

3.2.2.1 Partial Vision

Section: 3.2.2.1.b.

The accessible voting station with an electronic image display shall be capable of showing all information in at least two font sizes, (a) 3.0-4.0 mm and (b) 6.3-9.0 mm, under control of the voter.

Conformance: Yes

Section: 3.2.2.1.d.

An accessible voting station with a color electronic image display shall allow the voter to adjust the color or the figure-to-ground ambient contrast ratio.

Conformance: Yes

Section: 3.2.2.1.e.

Buttons and controls on accessible voting stations shall be distinguishable by both shape and color.

Conformance: Partial

Comments: Uses a 3 x 4 telephone-like keypad, no color usage.

Section: 3.2.2.1.f.

An accessible voting station using an electronic image display shall provide synchronized audio output to convey the same information as that which is displayed on the screen.

Conformance: Yes

3.2.2.2 Blindness

Section: 3.2.2.2.b.

The accessible voting station shall provide an audio-tactile interface (ATI) that supports the full functionality of the visual ballot interface, as specified in Subsection 2.3.3.

Instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if this is normally performed by the voter on comparable voting stations

Conformance: Yes

Comments: The VVPAT makes it difficult for some voters and impossible for others to insert the voter cards themselves.

Suggestions: Many of the voters said that they would like to have the voter card put down lower in front, where it would be much easier for more voters to reach and be able to insert and remove the card.

Instructions and feedback to the voter on how to operate the accessible voting station, including settings and options (e.g., volume control, repetition)

Conformance: Yes

Instructions and feedback for navigation of the ballot

Conformance: Partial

Comments: The 4, 5, 6, 7, 9, and 0 keys are frequently used for different purposes. In some functions they mean "Proceed", while in other functions the same key has a negation sense. '9' is used to confirm sometimes, but to move on in others. the '5' key is sometimes used positively to select, like an Enter key, and sometimes to cancel. '0' is also used to accept, like an Enter key. This makes it very confusing and hard for the voter to anticipate which key will be used in a given sense, for any process. It tends to force the voter to proceed very slowly, waiting for the system to pause and then time out and offer the voter instructions. Speech users like to be able to move quickly, interrupting messages and skipping to more important information or processes. However they can't do this well if they have to always keep stopping and waiting impatiently to hear instructions about their next command key options.

Suggestions: Assign keys in a more intuitive manner with only a single sense for each keys navigation and selection/rejection use. For example: a simpler example layout might be a "Cross" in which 2 and 8 are to move up and down through a list of choices , 4 and 6 move back and forward by contest, and 5 accepts/rejects. Sticking with the sense of these keys to do all the ballot navigating and select/reject type functions could simplify the navigation and cut down on the common feeling that this system has way too many confusing and hard to learn navigation control keys.

Instructions and feedback for contest choices, including write-in candidates

Conformance: Yes

Comments: The phone-pad text messaging approach used for write-in names entry can be reasonably fast for voters who are either sighted or, if blind, who already know which letters are on which phone keys. However, for some voters, such as blind voters with no idea of the letter assignments on the phone pad, write-ins can represent a tremendous challenge.

Many of Many of these voters do not even know which of the phone keys are for '*' and '#'.

Suggestions: Several voters suggested having the assignment of letters on the phone pad be summarized in the initial instructions or write-in help. It may be helpful to provide the voter with a large print and braille map of the key pad layout with the letter assignments for the phone keys.

Instructions and feedback on confirming and changing selections

Conformance: Partial

Comments: Many of the test voters were confused and not aware that the process of selecting another choice in an already fully voted contest required them to first deselect a previous selection, before attempting to select their new choice.

In this review, the available test ballots unfortunately did not have any multiple member contests, so we did not test changing with multiple selectables.

Suggestions: The instructions should be changed to give the voter a better overview or understanding of what they generally need to do in the selection change process. The requirement for deselecting the current choice before selecting a new choice is considered useful by some to prevent accidental changes to the ballot on touch screens. For audio interface users accidental changes are not as much a problem. It would simplify operation for audio voters if they weren't required to deselect first, in a manner similar to the eSlate.

Instructions and feedback on final submission of ballot

Conformance: Partial

Comments: It was not obvious to some voters that they would need to move contest by contest down to the end of the ballot, before being allowed to cast their ballot. Many were reading help and searching around to try to find a way to "jump" to the ballot casting.

In the ballot casting, it tells the voter that the vote selections that are being printed on the paper record are the same as the information in the electronic memory. This actually misled a few voters into thinking that the system was scanning and verifying the vote selections back and verifying the contents of the paper printout.

Suggestions: This message should be corrected to say that the information printing on the paper should be the same as the ballot stored in the electronic memory.

Section: 3.2.2.2.b.i.

The ATI of the accessible voting station shall provide the same capabilities to vote and cast a ballot as are provided by other voting machines or by the visual interface of the standard voting machine.

Conformance: No

Comments: Voters who can not visually read the VVPAT printout cannot verify the ballot directly from the selections the system was supposed to have just printed on the paper.

Section: 3.2.2.2.b.ii.

The ATI shall allow the voter to have any information provided by the voting system repeated.

Conformance: No

Comments: There is no direct method to have it simply repeated. Many users asked for a repeat control because the timeout repeats take too long and don't always repeat the last spoken message.

Suggestions: There should be a control that lets the voter request that the last message be repeated. At a minimum, the message should automatically repeat (letting you know that it is repeating itself) after a time out of maybe 5 seconds. No additional user action should be required as a result of the repeated message.

Section: 3.2.2.2.b.iii.

The ATI shall allow the voter to pause and resume the audio presentation.

Conformance: No

Comments: Many audio voters requested a pause/resume control.

Section: 3.2.2.2.b.iv.

The ATI shall allow the voter to skip to the next contest or return to previous contests.

Conformance: Yes

Section: 3.2.2.2.b.v.

The ATI shall allow the voter to skip over the reading of a referendum so as to be able to vote on it immediately.

Conformance: Yes

Section: 3.2.2.2.c.i.

The ATI shall provide its audio signal through an industry standard connector for private listening using a 3.5mm stereo headphone jack to allow voters to use their own audio assistive devices.

Conformance: Yes

Section: 3.2.2.2.c.iv.

A sanitized headphone or handset shall be made available to each voter.

Conformance: No

Comments: No sanitizable headphone covers were supplied by the vendor.

Suggestions: Throw-away headphones have such poor sound reproduction quality that they would not be acceptable.

Throw-away sanitary headphone covers should be provided, but they should be of a type that does not distract the voter by sliding off too easily, as some are known to do.

Section: 3.2.2.2.c.v.

The voting machine shall set the initial volume for each voter between 40 and 50 dB SPL.

Conformance: No (Using the vendor supplied headphones with its in-line volume control.)

Conformance: Yes (Using other headphones with no in-line volume control.)

Comments: The built-in audio volume control can be overridden by the headphone inline control.

Difficult for voters with manual dexterity impairments to operate, and because the setting of the previous voter is inherited and might be much too low or too high.

Suggestions: Headphones could be provided that have no in-line volume control.

Section: 3.2.2.2.c.vi.

The voting machine shall provide a volume control with an adjustable volume from a minimum of 20dB SPL up to a maximum of 100 dB SPL, in increments no greater than 10 dB.

Conformance: No (Using the vendor supplied headphones with its in-line volume control.)

Conformance: Yes (Using other headphones with no in-line volume control.)

Comments: We were unable to measure this system's lowest volume level as 20 dB is below the lowest level of our meter and the testing would have required better sound proofing isolation of ambient noise.

Using the vendor-supplied headphones with their in-line volume control:

lowest volume is unintelligible and barely audible (less than 50 dB).

normal volume is 70 dB.

highest volume is 90 dB.

volume step size is approximately 10 dB per step and has 5 steps.

the Maximum volume level was measured as about 90 dB.

Suggestions: Near term, providing headphones with no in-line volume control might result in higher output maximum volume levels and conforming on maximum level, as well as on the resetablity requirements.

Section: 3.2.2.2.c.vii.

The audio system shall be able to reproduce frequencies over the audible speech range of 315 Hz to 10 KHz.

Conformance: Not tested.

Comments: Difficult to test in our test environment. It would be better to examine circuit design or test audio output in an electronics lab.

Section: 3.2.2.2.c.viii.

The audio presentation of verbal information should be readily comprehensible by voters who have normal hearing and are proficient in the language. This includes such characteristics as proper enunciation, normal intonation, appropriate rate of speech, and low background noise. Candidate names should be pronounced as the candidate intends.

Conformance: Yes

Comments: Volume levels on some of the write-in letters and various other messages are not normalized well.

The audio has an approximately 1 Hz "heart beat" noise click that can be heard in the audio and is also a strong radio signal.

Suggestions: Whenever messages are recorded, they should be done in a proper sound studio with professional equipment and procedures. Additionally, native speakers should be used and volume and pitch should be normalized. Like singers, a reference pitch tone cue can be used to help the speaker stay on pitch.

Reduce the 1 Hz "heart beat" noise in the audio channel and confirm EMI testing results for RF interference from this "heart beat".

Section: 3.2.2.2.c.ix.

The audio system shall allow voters to control the rate of speech. The range of speeds supported should be at least 75% to 200% of the nominal rate.

Conformance: No

Comments: The speech rate control technology sounds as though it is just a simple time base compression that causes unacceptable "chipmunk" pitch distortion. Variable Speed Control (VSC) technologies for speech rate change without pitch distortion have been readily available for over 30 years and are simple and inexpensive to implement in modern microprocessor systems.

While some skilled voters who have been reading pitch distorted taped books for years may be able to understand the pitch distorted speech of this system at higher or lower rates, other voters can find it very difficult to understand.

Diebold TSX speech rate range is 70% to 200% of normal speech rate.

Based on time to read Diebold help message with audio

43 seconds for slowest speed is 70% of normal.

30 seconds for normal speed

15 seconds for fastest speed is 2 times faster.

Suggestions: Consider implementing a standard VSC technology for varying the speech rate.

Section: 3.2.2.2.d.

If the normal procedure is to have voters initialize the activation of the ballot, the accessible voting station shall provide features that enable voters who are blind to perform this activation.

Conformance: Yes

Comments: There are noticeable card insertion concerns for voters with manual dexterity impairments.

For voters using a wheelchair or otherwise seated, it is awkward to reach up over the VVPAT and keypad, holding a small card at an awkward angle, and manipulate it sideways into a thin slot. Most of the voters using wheelchairs could not insert the access card independently.

Comments from several test voters:

• "The access card input is too high"
• "Move access card slot to bottom, and don't recess it."
• "Have access card at bottom for easier input"
• "Could not insert card."

Section: 3.2.2.2.e.

If the normal procedure is for voters to submit their own ballots, then the accessible voting station shall provide features that enable voters who are blind to perform this submission.

Conformance: Yes

Section: 3.2.2.2.f.

All mechanically operated controls or keys on an accessible voting station shall be tactilely discernible without activating those controls or keys.

Conformance: Yes

Comments: The keys are tactilely discernable only by position in a 3 by 4 matrix of keys that resembles a phone key pad. The individual keys do not have tactile texture or shape clues for discerning their function. Additionally, the keys are not braille labeled. There is a nib on the '5' key to facilitate use.

Suggestions: Consider providing a large print and braille map that labels all keys and the letters of the alphabet that are associated with each key.

Section: 3.2.2.2.g.

On an accessible voting station, the status of all locking or toggle controls or keys (such as the "shift" key) shall be visually discernible, and discernible either through touch or sound.

Conformance: Yes

3.2.3 Dexterity

Section: 3.2.3.b.

All keys and controls on the accessible voting station shall be operable with one hand and shall not require tight grasping, pinching, or twisting of the wrist. The force required to activate controls and keys shall be no greater 5 lbs. (22.2 N).

Conformance: Partial

Comments: Except for a short person standing and using the keypad in its parking place on top of the printer, other ATI users will need to hold the key pad with hand and operate it with their other hand. Because of the pad's extra width and due to lack of any thing on the underside to grip. it can become uncomfortable to hold the key pad for very long, to vote in a long ballot election.

The shape of the underside of the keypad does not conform to the shape of a leg of a seated user.

The keys are not spaced far enough to allow for reliable operation by a closed fist.

The key caps are flat, so they are not dished or sunken to help voters with mouthsticks keep their mouthstick from sliding off the key caps.

Section: 3.2.3.c.

The accessible voting station controls shall not require direct bodily contact or for the body to be part of any electrical circuit.

Conformance: Yes

Section: 3.2.3.d.

The accessible voting station shall provide a mechanism to enable non-manual input that is functionally equivalent to tactile input.

Conformance: No

Comments: System does not support dual-switch input controls.

Section: 3.2.3.e.

If the normal procedure is for voters to submit their own ballots, then the accessible voting station shall provide features that enable voters who lack fine motor control or the use of their hands to perform this submission.

Conformance: Yes

Comments: Some users may not be able to independently insert and remove the voter access card.

3.2.4 Mobility

Section: 3.2.4.a.

The accessible voting station shall provide a clear floor space of 30 inches (760 mm) minimum by 48 inches (1220 mm) minimum for a stationary mobility aid. The clear floor space shall be level with no slope exceeding 1:48 and positioned for a forward approach or a parallel approach.

Conformance: N/A

Comments: Requirement for voting station setup space.

Section: 3.2.4.b.

All controls, keys, audio jacks and any other part of the accessible voting station necessary for the voter to operate the voting machine shall be within reach as specified under the following sub-requirements:

Section: 3.2.4.b.i.

If the accessible voting station has a forward approach with no forward reach obstruction then the high reach shall be 48 inches maximum and the low reach shall be 15 inches minimum.

Conformance: N/A

Comments: VVPAT printer represents both a side obstruction and a forward reach obstruction for the card slot.

Section: 3.2.4.b.ii.

If the accessible voting station has a forward approach with a forward reach obstruction, the following requirements apply:

The forward obstruction shall be no greater than 25 inches in depth, its top no higher than 34 inches and its bottom surface no lower than 27 inches.

Conformance: No

Comments: The top of the printer is 42 inches high, the bottom clearance of the printer is 25.5 inches, the printer front edge to tablet front edge is 9.5 inches, and printer front edge to card slot depth is about 13 inches, depending on the tilt of the tablet. The printer blocks the reach to the card slot above it. The bottom of the printer and stand are too low to get wheelchairs under it so it forces users to approach in the area to the left of the printer.

Note: The VVSG should include a specification for side obstructions that block elbows when reaching to the work surface. The printer represents a right elbow obstruction to the touch screen which blocks access to the right area of the screen. Similarly, the left privacy shield (9 inches) is a hard left side obstruction that cannot be moved away and blocks the voter's left elbow and access to the left area of the screen.

Suggestions: Move the printer back, move the card slot down to the lower front, and move the keypad parking area down and tilt it back (closer to horizontal) because taller voters have to bend their wrist at an awkward angle to be able to "touch type" on the keypad. Pollworkers could be trained to be aware that moving the keypad off the top of the printer might help some voters with reach impairments to better reach the card slot.

If the obstruction is no more than 20 inches in depth, then the maximum high reach shall be 48 inches, otherwise it shall be 44 inches.

Conformance: No

Comments: May be able to reach top of screen but cannot reach card slot. VVPAT top in front of and below card slot is 46 inches high. VVPAT to tablet front edges approximately 9.5 inches. The voter has to reach over the printer to slide the card to the left into a recessed slot. This requires most voters to hold their hand with knuckles to the right and elbow down so it makes it difficult to flex their elbows as they reach over the printer and keypad to the recessed card slot area.

Suggestions: See previous requirement.

Section: 3.2.4.b.iii.

Space under the obstruction between the finish floor or ground and 9 inches (230 mm) above the finish floor or ground shall be considered toe clearance and shall comply with the following provisions:

Toe clearance shall extend 25 inches (635 mm) maximum under the obstruction

Conformance: N/A

Comments: Knee clearance, not toe clearance is the determining factor limiting the approach.

Suggestions: Provide a better thin topped, height adjustable, stand or table that permits better approach. You want flexibility in knee clearance height under the table but want it to be as low as possible so the voter can reach the top of the screen and the card slot. The thin top avoids wasting vertical clearance.

The minimum toe clearance under the obstruction shall be either 17 inches (430 mm) or the depth required to reach over the obstruction to operate the accessible voting station, whichever is greater

Conformance: N/A

Comments: See previous requirement.

Suggestions: See previous requirement.

Toe clearance shall be 30 inches (760 mm) wide minimum

Conformance: No

Comments: 19 inches between legs at front.

Suggestions: See previous requirement.

Section: 3.2.4.b.iv.

Space under the obstruction between 9 inches (230 mm) and 27 inches (685 mm) above the finish floor or ground shall be considered knee clearance and shall comply with the following provisions:

Knee clearance shall extend 25 inches (635 mm) maximum under the obstruction at 9 inches (230 mm) above the finish floor or ground.

Conformance: No

Comments: About 25.5 inches high,

Suggestions: Use alternative stands or tables.

The minimum knee clearance at 9 inches (230 mm) above the finish floor or ground shall be either 11 inches (280 mm) or 6 inches less than the toe clearance, whichever is greater.

Conformance: No

Between 9 inches (230 mm) and 27 inches (685 mm) above the finish floor or ground, the knee clearance shall be permitted to reduce at a rate of 1 inch (25 mm) in depth for each 6 inches (150 mm) in height.

Conformance: N/A

Knee clearance shall be 30 inches (760 mm) wide minimum.

Conformance: No

Comments: 19 inches between legs at front and

Suggestions: Use alternate stands or tables.

Section: 3.2.4.b.v.

If the accessible voting station has a parallel approach with no side reach obstruction then the maximum high reach shall be 48 inches and the minimum low reach shall be 15 inches. See Figure 3.

Conformance: N/A

Comments: Has left privacy panel and right printer side obstructions.

Section: 3.2.4.b.vi.

If the accessible voting station has a parallel approach with a side reach obstruction, the following sub-requirements apply.

The side obstruction shall be no greater than 24 inches in depth and its top no higher than 34 inches.

Conformance: No

Comments: VVPAT printer blocks access to card slot and top of screen. The left privacy panel also blocks access to the screen.

Suggestions: Mount the printer further back and use flexible privacy shield curtains or panels that can be moved aside or folded back accommodate elbow room.

If the obstruction is no more than 10 inches in depth, then the maximum high reach shall be 48 inches, otherwise it shall be 46 inches.

Conformance: No

Comments: VVPAT printer blocks access to card slot and top of screen. The left privacy panel is 9.5 to 12.5 inches deep, depending on the tilt of the tablet.

Suggestions: Mount the printer further back and use flexible privacy shield curtains or panels that can be moved aside or folded back accommodate elbow room.

Section: 3.2.4.c.

All labels, displays, controls, keys, audio jacks, and any other part of the accessible voting station necessary for the voter to operate the voting machine shall be easily legible and visible to a voter in a wheelchair with normal eyesight (no worse than 20/40, corrected) who is in an appropriate position and orientation with respect to the accessible voting station

Conformance: Partial

Comments: The VVPAT is blurry and hard to read.

3.2.5 Hearing

Section: 3.2.5.b.

If voting equipment provides sound cues as a method to alert the voter, the tone shall be accompanied by a visual cue, unless the station is in audio-only mode.

Conformance: Yes

3.2.7 English Proficiency

For voters who lack proficiency in reading English, or whose primary language is unwritten, the voting equipment shall provide spoken instructions and ballots in the preferred language of the voter, consistent with state and federal law. The requirements of 3.2.2.2 (c) shall apply to this mode of interaction.

Conformance: No

Comments: The Diebold system did not actually handle real alternative language ballots for either Spanish or Chinese. For the visual portion of the Chinese ballot, there were only character place holder boxes, no actual Chinese characters. It was not clear if the tested system is capable of displaying non-Roman fonts.

Neither the Spanish nor the Chinese ballot was functional enough to be used by our test voters to complete a vote. At the time of this report, Diebold officials have not explained how much of the lack of alternative language support is due to incomplete elections or ballot definitions and how much was due to an inability of the system hardware/software to support the languages properly.

Suggestions: Because non-Roman fonts for languages such as Chinese may require performance from the voting system's video and graphics handling hard/software, it is important to confirm and test this system's capability to actually fully support the non-Roman languages.

As was observed on the audio and visual ballots of the other tested systems, there also may be translation errors, as well as speech recording concerns on this system as well. When developing its alternative language support, it would be good for the vendor to have professional language translators try voting on the system, with only the audio output and then to vote again with only visual display. Then ask them to help translate and make translation scripts more accurate and less confusing.

C.2. VVSG Conformance for the Hart eSlate

Vendor: Hart InterCivic

Voting System: System Hart InterCivic 6.2.1

Device Model: eSlate/DAU, version 4.2.13

VVPAT: Built-in

Supported by: JBC, version 4.3.1

3.1.2 Functional Capabilities

Section 3.1.2.a.

The voting system shall provide feedback to the voter that identifies specific contests or ballot issues for which he or she has made no selection or fewer than the allowable number of selections (e.g., undervotes).

Conformance: Partial

Comments: Inadequate visual feedback, but it does notify the audio voter.

Section 3.1.2.b.

The voting system shall notify the voter if he or she has made more than the allowable number of selections for any contest (e.g., overvotes).

Conformance: N/A

Comments: System prevents overvote selections.

Section 3.1.2.c.

The voting system shall notify the voter before the ballot is cast and counted of the effect of making more than the allowable number of selections for a contest.

Conformance: N/A

Comments: Not necessary, as system does not allow overvotes.

Section 3.1.2.d.

The voting system shall provide the voter the opportunity to correct the ballot for either an undervote or overvote before the ballot is cast and counted.

Conformance: Yes

Comments: Overvote correction isn't necessary, as it can't happen.

Section: 3.1.2.e.

The voting system shall allow the voter, at his or her choice, to submit an undervoted ballot without correction.

Conformance: Yes

Section: 3.1.2.f.

DRE voting machines shall allow the voter to change a vote within a contest before advancing to the next contest.

Conformance: No.

Comments: This system automatically moves the voter to the next race when a selection is made. Conforms only when using dual-switch input control, where it adds an Exit Race option on the menu.

Section: 3.1.2.g.

DRE voting machines should provide navigation controls that allow the voter to advance to the next contest or go back to the previous contest before completing a vote on the contest currently being presented (whether visually or aurally).

Conformance: Yes

Comments: Except when in dual-switch input control.

Suggestions: Consider adding a Previous Contest option on menus.

3.1.4 Cognitive Issues

Section: 3.1.4.a.

Consistent with election law, the voting system should support a process that does not introduce any bias for or against any of the selections to be made by the voter. In both visual and aural formats, contest choices shall be presented in an equivalent manner.

Conformance: Yes

Section: 3.1.4.b.

The voting machine or related materials shall provide clear instructions and assistance to allow voters to successfully execute and cast their ballots independently.

Conformance: Partial

Comments: Help messages run together too fast, without prosodic breaks to help the voter parse and absorb the separate elements of the messages. One of the help messages has seven different sub messages all run together without any noticeable breaks for parsing or absorption time. The speech rate in this and other help messages is so high that it is difficult for slow listeners to understand the speech or comprehend the messages. Fast listeners prefer fast speech like this but they too need appropriate prosodics.

Suggestions: There should be more help messages and they should be shorter with fewer sub messages in each one and should be more context appropriate.

Section: 3.1.4.b.i.

Voting machines or related materials shall provide a means for the voter to get help at any time during the voting session.

Conformance: Yes

Comments: Needs a method to request help, as well as summon pollworker.

Suggestions: For dual switch voters, could add Help, as well as Summon Pollworker options on menu or "Press" Help by time out on a long sip or puff.

Requiring a dual switch voter to press both switches at the same time for help or alternate commands might work for some users with jelly-switch-type controls, but is not a good idea because non coincident dual switch systems, such as sip and puff, can not activate both switches at the same time. Activating a single switch for a reasonably "long" time is easier for most dual switch voters.

A long sip or puff could bring up a special Dual-switch menu with options such as: Help, Summon Pollworker, Cast Ballot, Next Race, Previous Race or other options.

Section: 3.1.4.b.ii.

The voting machine shall provide instructions for all its valid operations.

Conformance: Partial

Comments: Most of the help and instructions are not modified for dual-switch mode operation.

Additionally, there are times when pressing certain keys or turning the wheel does nothing, but the user is not given feedback about why the control was inactive or the input was ignored.

The fundamental action-response closure rule says that the system should always give some response for any control input action, even if only to give a warning beep. It is usually better to give more helpful clues.

Suggestions: When dual-switch input is used, the help, instructions and prompts should only mention dual-switch controls, not all the other controls that the dual-switch voter doesn't need or want to hear about.

For audio voters in general, and specifically for those with neuropathy which limits their ability to sense tactile activation feedback in buttons or the wheel, use of a unique, short sound for affirming activation can substantially improve voter confidence.

Section: 3.1.4.c.i.

The voting equipment should not visually present a single contest spread over two pages or two columns.

Conformance: Partial

Comments: It depends on proper ballot layout, because it permits multiple races per page if the ballot layout has it.

Requiring one race per page would make the Next and Previous keys much more useful and lighten cognitive loading, as those switches would not have the unpredictable function of moving sometimes by race and other times by several.

Suggestions: The Hart ballot definition software should be configured to refuse to format ballots with multiple races per page.

Section: 3.1.4.c.ii.

The ballot shall clearly indicate the maximum number of candidates for which one can vote within a single contest.

Conformance: Yes

Section: 3.1.4.c.iii.

There shall be a consistent relationship between the name of a candidate and the mechanism used to vote for that candidate.

Conformance: Yes

Section: 3.1.4.d.

Warnings and alerts issued by the voting system should clearly state the nature of the problem and the set of responses available to the voter. The warning should clearly state whether the voter has performed or attempted an invalid operation or whether the voting equipment itself has malfunctioned in some way.

Conformance: Yes

Comments:

Suggestions: All messages, even those generally meant for pollworkers should talk. Battery status, printer status and other "system" configuration or status messages can be surprisingly helpful, even for non-technical voters, assuming the messages talk.

Section: 3.1.4.e.

The use of color by the voting system should agree with common conventions: (a) green, blue or white is used for general information or as a normal status indicator; (b) amber or yellow is used to indicate warnings or a marginal status; (c) red is used to indicate error conditions or a problem requiring immediate attention.

Conformance: Partial conformance

Comments: No green is used. Black outlines the ballot, and when a selection is made, it turns red. Conforms for use of Red, white and blue.

Suggestions: Keeping common red-green color blindness in mind, color use should be made more consistent with the intuitive conventional color associations.

3.1.5 Perceptual Issues

Section: 3.1.5.a.

No voting machine display screen shall flicker with a frequency between 2 Hz and 55 Hz.

Conformance: Yes

Comments: The blinking red "Cast Ballot" message is considered to be too slow and too small a fraction of the over-all screen flux, for it to represent a problem for triggering seizures.

Section: 3.1.5.b.

Any aspect of the voting machine that is adjustable by the voter or poll worker, including font size, color, contrast and audio volume, shall automatically reset to a standard default value upon completion of that voter's session.

Conformance: No

Comment: Only the volume is controlled by the voter, and it is not reset. It is only controlled with the manual in-line headphone control and its previous voter setting can be inherited.

Suggestions: In the near term, it could help to considering replacing the current headphones with others that have a larger in-line volume control that is easier for voters with limited dexterity to adjust, has a light tactile indicator or index for the middle volume position, and hopefully even has a lowest setting that complies with the 20 dB minimum volume guideline.

Pollworker training materials could suggest that the pollworkers try to always return the volume to the middle volume position before the next voting session.

Section: 3.1.5.c.

If any aspect of a voting machine is adjustable by the voter or poll worker, there shall be a mechanism to reset all such aspects to their default values.

Conformance: Yes

Comments: Assuming that they also reset the volume level of the in-line headphones volume control, as mentioned above.

Suggestions: A middle volume setting tactile index on the volume control would make it easier for pollworkers to reset the volume to normal.

Section: 3.1.5.d.

All electronic voting machines shall provide a minimum font size of 3.0 mm (measured as the height of a capital letter) for all text.

Conformance: No

Comments:

The smallest capitals are < 3.0 mm and the overall minimum is 2 mm

The overall maximum is 6 mm

President: 5 mm

Names: 4 mm

Party: 3 mm to 2 mm

Occupation: 4 mm to 3 mm

Judge: Text 3 mm to 2 mm

Write-in Letters 6 mm

Write-in On ballot (after typed in) 3.6 mm, but low contrast, grey text

Title of Prop: 3 mm to 2 mm

Summary Page Instructions: 6 mm to 4.7 mm

Summary Page Title of Race: 4 mm

Summary Page Selection: 4 mm

Language Selection Page: 6 mm to 4.7 mm

Section: 3.1.5.e.

All voting machines using paper ballots should make provisions for voters with poor reading vision.

Conformance: Partial

Comments: Measurement of fonts.

Paper verification page: 5 mm

Selection/Race: 3.5 mm

The 3.5 mm font for choices is pretty small but within the 3 mm guideline.

Because of the shading of the privacy shield, some low vision readers were not able to read the poorly lit printout.

Suggestions: A small spot lamp might be installed inside of the privacy shield, where it could shine on the VVPAT window (also good to have light for the print instructions on the right side).

Near term, pollworkers could fold back the glare guard at the top, provide an over-head light or even provide a wide beam flashlight. In any of these cases, care must be taken to assure that the added lighting is not generating glare problems on the visual display.

Section: 3.1.5.f.

The default color coding shall maximize correct perception by voters with color blindness.

Conformance: Yes

Comments: It is not easy to test this, as the brightness and other color aspects can help voters with color blindness to distinguish differences. One of the self-identified color blind subject said that he was not aware of any problems in this system's choice of colors.

Suggestions: The colors should be tested on subjects with severe color blindness conditions.

Section: 3.1.5.g.

Color coding shall not be used as the sole means of conveying information, indicating an action, prompting a response or distinguishing a visual element.

Conformance: Yes

Section: 3.1.5.h.

All text intended for the voter should be presented in a sans serif font.

Conformance: Yes

3.1.6 Interaction Issues

Section: 3.1.6.a.

Voting machines with electronic image displays shall not require page scrolling by the voter.

Conformance: No

Comments: It does not require horizontal page scrolling.

Section: 3.1.6.b.

The voting machine shall provide unambiguous feedback regarding the voter's selection, such as displaying a checkmark beside the selected option or conspicuously changing its appearance.

Conformance: Yes, for both visual and audio.

Section: 3.1.6.c.

If the voting machine requires a response by a voter within a specific period of time, it shall issue an alert at least 20 seconds before this time period has expired and provide a means by which the voter may receive additional time.

Conformance: Yes

Comments: Has no obvious timeouts that require user action.

Section: 3.1.6.d.

Input mechanisms shall be designed to minimize accidental activation.

Section: 3.1.6.d.i.

On touch screens, the sensitive touch areas shall have a minimum height of 0.5 inches and minimum width of 0.7 inches. The vertical distance between the centers of adjacent areas shall be at least 0.6 inches, and the horizontal distance at least 0.8 inches.

Conformance: N/A

Comments: It is not a touch screen, but the mechanical keys measure. We include notes here because there are no VVSG requirements regarding the size and separation of mechanical controls.

Cast Ballot button: 28 mm by 34 mm

Help button: 11 mm by 32 mm (not tall enough)

Previous, Next buttons: 20 mm by 20 mm

Enter button: 50 mm by 32 mm

Select Wheel: 52 mm across

11 mm to 25 mm between buttons Some are too close, for example, Help and the Next or Previous keys are close enough that it is too easy to press the wrong one, and a careful voter must go slowly to make sure that he/she is lining up on the center of either of these 3 keys before pressing. We observed a lot of "scrubbing around" time of motion whenever blind voters were trying to find Help, Next or Previous.

Suggestions: The closeness of adjacent keys presents serious key location and registration problems that would not be as severe if the keys were easier to distinguish tactually. Adding relief (dishing), reveal (raising) or texturing the keys could help. To help the voters with mouthsticks, it would be better if the top surface of the key caps was dished to help "trap" the end of a mouthstick (or other key pressing tool or finger). For voters with visual impairments, raising the key cap above the surface would be preferable. However the raising should not be with a gentle curve or dome. It is better to have a "sharp" step up at the edge of a tactile key for visually impaired voters. The needs of the voters with dexterity impairments are not necessarily incompatible with those of voters with visual impairments. The answer to compatible key tops is to make the keys both raised at the edges and sunken or dished within the surrounding ridge of the key edges. It appears that there is room to raise the height of the key caps by almost 2 mm or a tenth of an inch.

In the near term, adhesive-backed or stick-on key cap tops could be added onto the tops of at least some of the keys. The Cast Ballot key would need it mostly to help voters with dexterity impairments.

We feel that prototyping new key surface "stickers" and testing them with live subjects will probably give surprising results. Similar to adding tactile indicators on computer keyboards that experienced keyboard users couldn't imagine they would want has, in the past, yielded surprises for those users. The users felt much more confident and they had markedly reduced wasted time of motion "scrubbing" or hunting to line up and register on keys.

Section: 3.1.6.d.ii.

No key or control on a voting machine shall have a repetitive effect as a result of being held in its active position.

Conformance: Yes

3.2.2 Vision

Section: 3.2.2.1 Partial Vision

Section: 3.2.2.1.b.

The accessible voting station with an electronic image display shall be capable of showing all information in at least two font sizes, (a) 3.0-4.0 mm and (b) 6.3-9.0 mm, under control of the voter.

Conformance: No

Comments: there is no font or magnification control.

Suggestions: When adding larger fonts or magnification, the ballot should be formatted so that test is not hidden off to the side and requiring horizontal scrolling.

Section: 3.2.2.1.d.

An accessible voting station with a color electronic image display shall allow the voter to adjust the color or the figure-to-ground ambient contrast ratio.

Conformance: No

Comments: No voter controls.

Section: 3.2.2.1.e.

Buttons and controls on accessible voting stations shall be distinguishable by both shape and color.

Conformance: Partial

Comments: Inadequate use of color marking. Keys have different shapes, but, as they are flush with the console surface, the shapes are very difficult to feel. As mentioned above, these keys are hard to locate and identify from their shape.

Suggestions: Intuitive coloring of the key caps should be added to most of the keys, except the Cast Ballot button.

Raising the keys or giving them clear, "sharp" tactile edges would help to reveal their shapes better.

If add-on key tops are used in the near term, adding color should be considered, in addition to tactile definition.

Section: 3.2.2.1.f.

An accessible voting station using an electronic image display shall provide synchronized audio output to convey the same information as that which is displayed on the screen.

Conformance: Yes

Comments: The speech was often lagging far behind the visually displayed information.

Suggestions: Especially for voters who are actively listening to the audio speech while they read, it is important to have the audio and video tightly in synch. Voters with visual or cognitive impairments find it very helpful to have the words that are being spoken highlighted on the screen at the same time, with a moving highlight position indicator.

3.2.2.2 Blindness

Section: 3.2.2.2.b.

The accessible voting station shall provide an audio-tactile interface (ATI) that supports the full functionality of the visual ballot interface, as specified in Subsection 2.3.3.

Instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if this is normally performed by the voter on comparable voting stations.

Conformance: Yes

Comments: The pollworker uses the Judge's Booth Controller to print a unique 4-digit access or authentication number on a small slip of paper. Reading this number off the paper will require pollworker assistance for blind voters and others with reading impairments. Having to accept in reading the access code from the paper slip does present a barrier to independent voting for some voters, but it does not impact the voter's ability to perform a private vote.

Suggestions: It is important for the voters to try to do the actual entering of the access code for themselves, if for no other reason than it gives them an excellent practice opportunity for learning how to use the wheel and the Enter key. It should also be used as an opportunity to get the volume set properly for audio users.

Pollworkers should be trained to accompany the voters with reading impairments to the booth, to get them set up and ready to enter their access code before reading the access code to the voter.

Pollworkers should be trained to support and encourage the voters to enter their access code for themselves.

Because braille printers currently cost well over $1,000, and because most voters with visual impairments do not use braille, most counties are not likely to produce brailled access code slips. There are inexpensive braille sticky tape embossers that are hand-held, can be used by workers with minimal training, and could be provided for supplying braille access code "slips".

Instructions and feedback to the voter on how to operate the accessible voting station, including settings and options (e.g., volume control, repetition).

Conformance: Yes

Instructions and feedback for navigation of the ballot.

Conformance: Partial

Comments: Generally, for audio voters, when moving out of one race and into another, it did not announce "Next Race" to alert the voter that it was moving to a new race. This easily leaves the voter unaware that they are in a different race.

The race title and description all run together too fast, without prosodic breaks to help the voter parse and absorb the separate elements of the messages.

Suggestions: The contest announcements have too many separate items that should be separated with pauses in the speech and scripted as plain prose, instead of such numerical statements. It is better to say "There are no selections", instead of "0 candidates selected" or say, "There are 3 choices". It could say, "You can choose Yes or No.".

Add a "Next Contest" or "Previous Contest" message and/or a unique, short, uninterruptible sound when changing contests.

There are too many repeats of information in the contest header message. Some of this may be due to bad ballot layout. If so, the ballot layout software should be changed to avoid repetitious messages.

High functioning audio users may ask for fast messages without pauses. It is better to error in having too much delay for the hotshots then have slower voters totally miss the content of a message due to its fast pace.

Instructions and feedback for contest choices, including write-in candidates.

Conformance: Partial

Comments: Several voters seemed to be oblivious to the "Current Selection" message, even though it was at the beginning of a choice, where it should be. They could be actively listening to try to find their current selection, but roll right on past it as the system announced "Current Selection". Evidently this is because the "current selection" status message is in the same voice and sounds just like all of the candidate-name-"babble" the voter is only half listening to.

Suggestions: One suggestion is to use a strikingly different voice for the status message. A different speaker, at a very different pitch might help.

A preferred approach might be to add a unique, short, uninterruptible sound. This should be heard even if the user rolls the wheel rapidly, interrupting announcing of candidate names. This is an example, similar to change of contest notification that ought to be heard, despite other speech interrupts. Obviously, this sort of uninterruptible sound needs to be short, if it is not interruptible.

Instructions and feedback on confirming and changing selections.

Conformance: Yes

Comments: In this review, the available test ballots unfortunately did not have any multiple member contests, so we did not test changing with multiple selectables.

Having the "Current Selection" status announced at the start of a choice is good (despite the comments above).

Write-in for the audio voter had inadequate information initially. Write-ins are done so rarely by the typical voter, so the system can risk being too verbose. It may be obvious to a visual voter, but the audio voter does not realize that they have just been placed in a list of the alphabet and some other options such as "Accept", Cancel", etc.

The only clue the voter has is that the system simply announces "Ah", which really means 'A'. Only if and when the voter begins turning the wheel experimentally, do they catch on that they are in an alphabet.

All audio voters complained that they did not have a way to review what they had entered as a write-in. Visual voters can see the write-in name on the screen, but it does not speak.

Suggestions: A little more instruction at the start of write-in would go a long way towards improving the voters' write-in experience. This message should alert them to the list wrap around and the presence of "space", delete last letter and other choices at the ends of the alphabet.

There should be a way of reviewing the write-in name, either automatically (after a time out with no voter input), or at least as an option on the menu. When the voter is out of write-in mode and reviewing the candidate choices, the write-in choice should spell out the write-in name (of course announcing any "space" entries).

Instructions and feedback on final submission of ballot.

Conformance: partial

Comments: Despite the big red "Cast Button" and help messages about it, some users forgot about it. The design of the red blinking "Cast Ballot" message on the screen certainly drew visual voters' attention, but it appeared to some voters to be a button they should push with a finger (forgot it wasn't a touch screen) or that they should try to get the pointer on by rolling the wheel so they could press Enter on it.

The ballot verification review appears to keep chatting on uninterrupted, even when the voter rolls the wheel to select "Accept" or "Reject".

There was not appropriate when the voter pressed the Cast Ballot button. Some times it seemed, to the audio user, that the button was being ignored. There actually were times, in ballot casting, when input controls did have an effect (visually confirmed), but had no audio feedback. Long time delays before the ballot was heard to start printing left voters wondering what to do next, as it seemed that they must have done something wrong.

Suggestions: The "Cast Ballot" message on the screen should be changed to help the voter recognize that it refers to the mechanical button on the front left side.

In ballot verification review mode, pressing keys or rolling the wheel should interrupt the list of contests and choices.

Any time the voter does a key press or wheel turn, there should be a response, even if just a message or sound "earcon" that lets them know that the control input is being ignored. This is just good basic human factors. There were voters who hit the Cast Ballot key several times because "It didn't seem to notice the first press."

Before starting the long delay that leads up to the sound of the VVPAT starting to print, there should be a message that lets the voter know that it is going to start printing after a short delay.

Section: 3.2.2.2.b.i.

The ATI of the accessible voting station shall provide the same capabilities to vote and cast a ballot as are provided by other voting machines or by the visual interface of the standard voting machine.

Conformance: No

Comments: Voters who can not visually read the VVPAT printout cannot verify the ballot directly from the selections the system was supposed to have just printed on the paper.

Suggestions: Some voters appeared to believe that the system was directly reading the text of the ballot from the paper of the VVPAT printout. They believed this because the audio verification review started speaking after the printer started printing, continued (even after speech was interrupted) as the printer kept printing, and because the system made a statement that seemed to indicate that it was reading the actual ballot from the paper record.

The incorrect verifying-from-the-paper-record message should be rewritten to clearly state that the system is reading, from memory, what should now be printing on the paper record.

Section: 3.2.2.2.b.ii.

The ATI shall allow the voter to have any information provided by the voting system repeated.

Conformance: No

Comments: There is no direct method to have it simply repeated.

Suggestions: There should be a control that lets the voter request that the last message be repeated. At a minimum, the message should automatically repeat (letting you know that it is repeating itself) after a suitable timeout. No additional user action should be required as a result of the repeated message.

Section: 3.2.2.2.b.iii.

The ATI shall allow the voter to pause and resume the audio presentation.

Conformance: No

Section: 3.2.2.2.b.iv.

The ATI shall allow the voter to skip to the next contest or return to previous contests.

Conformance: Yes

Comments: Not if using the dual-switch interface.

Suggestions: Add "Next Contest" and "Previous Contest" options on the dual-switch menu.

Section: 3.2.2.2.b.v.

The ATI shall allow the voter to skip over the reading of a referendum so as to be able to vote on it immediately.

Conformance: Yes

Section: 3.2.2.2.c.i.

The ATI shall provide its audio signal through an industry standard connector for private listening using a 3.5mm stereo headphone jack to allow voters to use their own audio assistive devices.

Conformance: Yes

Comments: Voter cannot easily plug in their own headphones or assistive listing device. The tablet of the eSlate must be removed from the support stand in order to reach the audio phone jack, and only a right angle plug can be plugged in their. Many voters would not be able to see where to plug in their own headphones and give up. Maybe worse, others might follow the vendor-supplied headphone cable and attempt to remove the tablet themselves. From the experience in the accessibility review testing, it would be safe to assume that the voter probably could not get the tablet re-installed properly, causing a Can't Find Printer error. Even if the pollworker does the tablet removal and re-insert in the stand, there is a good chance that the tablet may not be inserted properly and have problems.

Suggestions: Near term, at a minimum, the systems should be provided with a right angle adapter that would allow a standard straight-in plug from alternative headphones or listening devices to be plugged into the tablet.

Preferably, an extension cable with a right angle plug should be plugged into the tablet and routed out through the upper flap hole, to offer the voter a less complicated and less failure prone way to plug into the audio output.

Long term: The design of the tablet and/or the support stand should be modified to allow access to an audio jack without having to pull the tablet from the stand. Note: The same solutions are needed for the dual-switch input control.

Section: 3.2.2.2.c.iv.

A sanitized headphone or handset shall be made available to each voter.

Conformance: No

Comments: No sanitizable headphone covers were supplied by the vendor.

Suggestions: Throw-away headphones have such poor sound reproduction quality that they would not be acceptable.

Throw-away sanitary headphone covers should be provided, but they should be of a type that does not distract the voter by sliding off too easily, as some are known to do.

Section: 3.2.2.2.c.v.

The voting machine shall set the initial volume for each voter between 40 and 50 dB SPL.

Conformance: No.

Comments: The audio volume control is only by the manual in-line volume control on the headphones. This is unacceptable because it is too difficult for voters with manual dexterity impairments to operate, and because the setting of the previous voter can be inherited and might be much too low or too high.

Suggestions: Near term, a larger in-line volume control might be supplied that would be easier to use with manual dexterities, has a center volume position detent, and limits the minimum volume setting to no less than 20 dB.

Long term, built-in, software-controlled volume control should be added to the design, to offer audio volume reset to normal. An initial configuration menu could be implemented to offer choices over access controls such as volume, speech rate, magnification, etc.

This menu should also be compatible with dual-switch input control.

Section: 3.2.2.2.c.vi.

The voting machine shall provide a volume control with an adjustable volume from a minimum of 20dB SPL up to a maximum of 100 dB SPL, in increments no greater than 10 dB.

Conformance: Partial

Comments: The minimum volume is nearly inaudible, and the maximum is 100 dB. Additionally, it does not conform with adjustable in steps.

Suggestions: Near term, a different in-line volume control might offer a 20 dB lower volume limit.

Long term, implementing software-controlled volume level adjust is necessary to conform with the range and step requirements, as well as to allow dual-switch users and voters with other manual dexterity impairments to adjust the volume through a software interface.

Section: 3.2.2.2.c.vii.

The audio system shall be able to reproduce frequencies over the audible speech range of 315 Hz to 10 KHz.

Conformance: Not tested.

Section: 3.2.2.2.c.viii.

The audio presentation of verbal information should be readily comprehensible by voters who have normal hearing and are proficient in the language. This includes such characteristics as proper enunciation, normal intonation, appropriate rate of speech and low background noise. Candidate names should be pronounced as the candidate intends.

Conformance: No.

Comments: The speech is too fast for slow listeners.

Proper prosody is not supported in messages, especially help or instructions that are concatenated from several message strings. The main help message is an interesting example, because it strings 7 messages together with out breaks to indicate phrase boundaries and without breaks between messages to offer the voter absorption and processing time. One of the female voices on this ballot seemed to be distorted enough that several voters commented about the poor quality.

Additionally, voters mentioned that some of the candidate names were not pronounced properly.

Suggestions: Nearly all of the messages should be re-recorded at slower rate. If done properly, this might be done by using good quality Variable Speed Control (VSC), so there wouldn't be any pitch distortion. The help and other multi- message audio should be re-concatenated with appropriately longer pauses between the messages. Whenever messages are recorded, they should be done in a proper sound studio with professional equipment and procedures. Additionally, native speakers should be used and volume and pitch should be normalized. Like singers, a reference pitch tone cue can be used to help the speaker stay on pitch.

Section: 3.2.2.2.c.ix.

The audio system shall allow voters to control the rate of speech. The range of speeds supported should be at least 75% to 200% of the nominal rate.

Conformance: No

Suggestions: The nominal rate should be lower than the current fixed rate. It is better to start with the system talking too slow and let skilled users increase the speed, than to leave slow listeners straining or unable to understand the default speech.

Section: 3.2.2.2.d.

If the normal procedure is to have voters initialize the activation of the ballot, the accessible voting station shall provide features that enable voters who are blind to perform this activation.

Conformance: Yes

Section: 3.2.2.2.e.

If the normal procedure is for voters to submit their own ballots, then the accessible voting station shall provide features that enable voters who are blind to perform this submission.

Conformance: Yes

Section: 3.2.2.2.f.

All mechanically operated controls or keys on an accessible voting station shall be tactilely discernible without activating those controls or keys.

Conformance: Yes

Comments: It is difficult to tactilely discern many of the keys, as they are not textured and are flush with the console surface.

Suggestions: Near term, add on key cap tactile and colored tops.

Section: 3.2.2.2.g.

On an accessible voting station, the status of all locking or toggle controls or keys (such as the "shift" key) shall be visually discernible, and discernible either through touch or sound.

Conformance: Yes

3.2.3 Dexterity

Section: 3.2.3.b.

All keys and controls on the accessible voting station shall be operable with one hand and shall not require tight grasping, pinching or twisting of the wrist. The force required to activate controls and keys shall be no greater 5 lbs. (22.2 N).

Conformance: Yes

Section: 3.2.3.c.

The accessible voting station controls shall not require direct bodily contact or for the body to be part of any electrical circuit.

Conformance: Yes

Section: 3.2.3.d.

The accessible voting station shall provide a mechanism to enable non-manual input that is functionally equivalent to tactile input.

Conformance: Yes

Comments: Many of the help, instructions and prompts are not appropriate for dual-switch voters.

Suggestions: Use separate dual-switch oriented messages when being operated by dual-switch voters. Messages for dual-switch users should not bother the voter with mention of other controls that the voter cannot operate.

Section: 3.2.3.e.

If the normal procedure is for voters to submit their own ballots, then the accessible voting station shall provide features that enable voters who lack fine motor control or the use of their hands to perform this submission.

Conformance: Yes

3.2.4 Mobility

Section: 3.2.4.a.

The accessible voting station shall provide a clear floor space of 30 inches (760 mm) minimum by 48 inches (1220 mm) minimum for a stationary mobility aid. The clear floor space shall be level with no slope exceeding 1:48 and positioned for a forward approach or a parallel approach.

Conformance: N/A

Comments: Requirement for voting station setup space.

Section: 3.2.4.b.

All controls, keys, audio jacks and any other part of the accessible voting station necessary for the voter to operate the voting machine shall be within reach as specified under the following sub-requirements:

Section: 3.2.4.b.i.

If the accessible voting station has a forward approach with no forward reach obstruction then the high reach shall be 48 inches maximum and the low reach shall be 15 inches minimum. See Figure 1.

Conformance: Yes

Comments: 41 inches

Section: 3.2.4.b.ii.

If the accessible voting station has a forward approach with a forward reach obstruction, the following requirements apply:

The forward obstruction shall be no greater than 25 inches in depth, its top no higher than 34 inches and its bottom surface no lower than 27 inches.

Conformance: N/A

Comments: No forward obstruction.

If the obstruction is no more than 20 inches in depth, then the maximum high reach shall be 48 inches, otherwise it shall be 44 inches.

Conformance: N/A

Section: 3.2.4.b.iii.

Space under the obstruction between the finish floor or ground and 9 inches (230 mm) above the finish floor or ground shall be considered toe clearance and shall comply with the following provisions:

Toe clearance shall extend 25 inches (635 mm) maximum under the obstruction.

Conformance: N/A

The minimum toe clearance under the obstruction shall be either 17 inches (430 mm) or the depth required to reach over the obstruction to operate the accessible voting station, whichever is greater.

Conformance: N/A

Toe clearance shall be 30 inches (760 mm) wide minimum.

Conformance: N/A

Section: 3.2.4.b.iv.

Space under the obstruction between 9 inches (230 mm) and 27 inches (685 mm) above the finish floor or ground shall be considered knee clearance and shall comply with the following provisions:

Knee clearance shall extend 25 inches (635 mm) maximum under the obstruction at 9 inches (230 mm) above the finish floor or ground.

Conformance: Partial

Comments: 26.5 inches high

The minimum knee clearance at 9 inches (230 mm) above the finish floor or ground shall be either 11 inches (280 mm) or 6 inches less than the toe clearance, whichever is greater.

Conformance: Partial

Comments: 29.5 inches wide

Between 9 inches (230 mm) and 27 inches (685 mm) above the finish floor or ground, the knee clearance shall be permitted to reduce at a rate of 1 inch (25 mm) in depth for each 6 inches (150 mm) in height.

Conformance: No

Comment: Knee clearance only goes up to 26.5 inches above the floor.

Knee clearance shall be 30 inches (760 mm) wide minimum.

Conformance: Partial

Comment: Knee clearance was only 29.5 inches.

Section: 3.2.4.b.v.

If the accessible voting station has a parallel approach with no side reach obstruction then the maximum high reach shall be 48 inches and the minimum low reach shall be 15 inches.

Conformance: Yes

Comments: Max high reach required is approximately 41 inches.

Section: 3.2.4.b.vi.

If the accessible voting station has a parallel approach with a side reach obstruction, the following sub-requirements apply.

Conformance: N/A

The side obstruction shall be no greater than 24 inches in depth and its top no higher than 34 inches.

Conformance: N/A

If the obstruction is no more than 10 inches in depth, then the maximum high reach shall be 48 inches, otherwise it shall be 46 inches.

Conformance: N/A

Section: 3.2.4.c.

All labels, displays, controls, keys, audio jacks and any other part of the accessible voting station necessary for the voter to operate the voting machine shall be easily legible and visible to a voter in a wheelchair with normal eyesight (no worse than 20/40, corrected) who is in an appropriate position and orientation with respect to the accessible voting station.

Conformance: Partial

Comments: The privacy shield shades instructions and VVPAT too much. Can't see or reach the audio jack.

Suggestions: Consider providing extension cables to extend the audio and the dual-switch jacks out of the stand enclosure so they would be visible and accessible.

3.2.5 Hearing

Section: 3.2.5.b.

If voting equipment provides sound cues as a method to alert the voter, the tone shall be accompanied by a visual cue, unless the station is in audio-only mode.

Conformance: Yes

3.2.7 English Proficiency

For voters who lack proficiency in reading English, or whose primary language is unwritten, the voting equipment shall provide spoken instructions and ballots in the preferred language of the voter, consistent with state and federal law. The requirements of 3.2.2.2 (c) shall apply to this mode of interaction.

Conformance: Yes

Comments: Some messages were not properly translated. The Chinese ballot did not have good help for location of the control keys and wheel.

Suggestions: Have professional language translators try voting on the system, with only the audio output and then to vote again with only visual display. Then ask them to help translate and make translation scripts more accurate and less confusing.

C.3. VVSG Conformance for the Sequoia Edge I and II

Vendor: Sequoia

Model: AVC Edge Model I and II

Firmware Version 5.0.24

VVPAT: VeriVote Printer

Supported by: Card Activator, Version 5.0.21

3.1.2 Functional Capabilities

Section: 3.1.2.a.

The voting system shall provide feedback to the voter that identifies specific contests or ballot issues for which he or she has made no selection or fewer than the allowable number of selections (e.g., undervotes)

Conformance: Yes

Section: 3.1.2.b.

The voting system shall notify the voter if he or she has made more than the allowable number of selections for any contest (e.g., overvotes)

Conformance: N/A

Comments: Not needed, because it does not permit overvote selections.

Section: 3.1.2.c.

The voting system shall notify the voter before the ballot is cast and counted of the effect of making more than the allowable number of selections for a contest

Conformance: N/A

Comments: Not needed, because it does not permit overvote selections.

Section: 3.1.2.d.

The voting system shall provide the voter the opportunity to correct the ballot for either an undervote or overvote before the ballot is cast and counted

Conformance: Yes

Comments: Does not permit overvote selections.

Section: 3.1.2.e.

The voting system shall allow the voter, at his or her choice, to submit an undervoted ballot without correction

Conformance: Yes

Section: 3.1.2.f.

DRE voting machines shall allow the voter to change a vote within a contest before advancing to the next contest.

Conformance: No

Comments: It automatically advances to next contest after current is fully voted.

Section: 3.1.2.g.

DRE voting machines should provide navigation controls that allow the voter to advance to the next contest or go back to the previous contest before completing a vote on the contest currently being presented (whether visually or aurally).

Conformance: Yes

Comments: Except for switched input use.

3.1.4 Cognitive Issues

Section: 3.1.4.a.

Consistent with election law, the voting system should support a process that does not introduce any bias for or against any of the selections to be made by the voter. In both visual and aural formats, contest choices shall be presented in an equivalent manner.

Conformance: Yes

Section: 3.1.4.b.

The voting machine or related materials shall provide clear instructions and assistance to allow voters to successfully execute and cast their ballots independently.

Conformance: Partial

Comments: The initial help is too long for most voters to absorb and remember for the rest of the session. There were only a few other help or instructions messages for an extremely limited number of contexts.

Additionally, Dual-switch input control users need help and instructions that reflects the use of dual-switches. None of the system's help, instructions, or prompts appear to be written with any awareness of dual-switch control possibilities.

Suggestions: Rescript all of the current help and instructions and prompts. Have a larger variety of context-dependent messages that are shorter and relevant to the immediate context.

Preferably, make the system's messages dual-switch context aware. If the system is being controlled by dual-switch input controls, the help, instructions, and other messages should only mention the switch control options.

Dual-switch aware menus could have extra options.

Consider using long-switch-closure time out as an alternative way to request help or to bring up a special dual-switch menu. An extra long sip or puff time of perhaps 4 or 5 seconds might be used to indicate a request for help, pollworker assistance, or the dual-switch menu. A special dual-switch menu might include Help, Summon assistance, change volume, change rate, etc.

Requiring a dual switch voter to press both switches at the same time for help or alternate commands might work for some users with jelly-switch-type controls, but is not a good idea because non coincident dual switch systems, such as sip and puff, can not activate both switches at the same time. Activating a single switch for a reasonably "long" time is easier for most dual switch voters.

Section: 3.1.4.b.i.

Voting machines or related materials shall provide a means for the voter to get help at any time during the voting session.

Conformance: Partial

Comments: The help may be available, but it may be imbedded somewhere within strings of many other contextually irrelevant messages.

Additionally, it is difficult for dual-switch input control users to request any contextual help.

Suggestions: Have a larger variety of context-dependent messages that are shorter and relevant to the immediate context.

Consider using long-switch-closures for a dual-switch control help request (as mentioned in previous requirement).

Section: 3.1.4.b.ii.

The voting machine shall provide instructions for all its valid operations.

Conformance: Yes

Section: 3.1.4.c.i.

The voting equipment should not visually present a single contest spread over two pages or two columns.

Conformance: No

Comments: It may, in magnified mode despite good ballot layout

Section: 3.1.4.c.ii.

The ballot shall clearly indicate the maximum number of candidates for which one can vote within a single contest.

Conformance: Yes

Section: 3.1.4.c.iii.

There shall be a consistent relationship between the name of a candidate and the mechanism used to vote for that candidate.

Conformance: Yes

Section: 3.1.4.d.

Warnings and alerts issued by the voting system should clearly state the nature of the problem and the set of responses available to the voter. The warning should clearly state whether the voter has performed or attempted an invalid operation or whether the voting equipment itself has malfunctioned in some way.

Conformance: Yes

Comments: Except there is no audio feedback if it is not yet talking.

Suggestions: All messages, even those generally meant for pollworkers should talk. Battery status, printer status and other "system" configuration or status messages can be surprisingly helpful, even for non-technical voters, assuming the messages talk.

Section: 3.1.4.e.

The use of color by the voting system should agree with common conventions: (a) green, blue or white is used for general information or as a normal status indicator; (b) amber or yellow is used to indicate warnings or a marginal status; (c) red is used to indicate error conditions or a problem requiring immediate attention.

Conformance: Partial

Comments: Blue – No/Yes. Yellow – No. Red – Yes. Green – No. Screen colors are OK. Button colors are not well chosen.

3.1.5 Perceptual Issues

Section: 3.1.5.a.

No voting machine display screen shall flicker with a frequency between 2 Hz and 55 Hz.

Conformance: Yes

Section: 3.1.5.b.

Any aspect of the voting machine that is adjustable by the voter or poll worker, including font size, color, contrast, and audio volume, shall automatically reset to a standard default value upon completion of that voter's session.

Conformance: Yes

Section: 3.1.5.c.

If any aspect of a voting machine is adjustable by the voter or poll worker, there shall be a mechanism to reset all such aspects to their default values.

Conformance: Yes

Section: 3.1.5.d.

All electronic voting machines shall provide a minimum font size of 3.0 mm (measured as the height of a capital letter) for all text.

Conformance: Partial

Comments: Generally does, but not consistently.

Overall minimum is 2 mm

Overall maximum is 14.1 mm

Language Selection: English is 3.1 mm; Spanish is 6 mm; Chinese is 9 mm

Race Large text: 9 mm to 5.2 mm; Normal text: 4.6 mm to 3 mm

Candidate Large text: 8 mm to 5.2 mm; Normal text: 4 mm to 3 mm

Occupation Large text: 6 mm to 4 mm; Normal text: 3 mm to 2 mm

Write-in Large text: 4.4 mm; Normal text: 4.4 mm

Write-in typed on ballot page Large text: 5 mm; Normal text: 2.2 mm

Directional buttons at bottom of each page Large text: 14.1 mm to 10 mm; Normal text: 7 mm to 5 mm

Judges & Propositions Title & Description Large text: 8 mm to 4 mm; Normal text: 4 mm to 2 mm

Answer "Yes" or "No" Large text: 8 mm; Normal text: 4 mm

Summary Page

Title "Touch the office…" Large text: 7 mm to 5 mm; Normal text: 3 mm to 2 mm

Races Large text: 8 mm to 6 mm; Normal text: 4 mm to 3 mm

Cast Ballot

Multiple fonts: 8 mm to 3 mm; no choice of large or normal

Section: 3.1.5.e.

All voting machines using paper ballots should make provisions for voters with poor reading vision.

Conformance: Partial

Comments: Paper Verification Page: Printed font is 5 mm. If voter is using audio interface to vote, they are not given an opportunity to read the printout or have someone read it for them.

Suggestions: Consider allowing voter to choose whether or not they wish an opportunity to review the paper printout.

Section: 3.1.5.f.

The default color coding shall maximize correct perception by voters with color blindness.

Conformance: Yes

Comments: It is not easy to test this, as the brightness and other color aspects can help voters with color blindness to distinguish differences. One of the self-identified color blind subject said that he was not aware of any problems in this system's choice of colors.

Suggestions: The colors should be tested on subjects with severe color blindness conditions.

Section: 3.1.5.g.

Color coding shall not be used as the sole means of conveying information, indicating an action, prompting a response, or distinguishing a visual element.

Conformance: Yes

Section: 3.1.5.h.

All text intended for the voter should be presented in a sans serif font.

Conformance: Yes

3.1.6 Interaction Issues

Section: 3.1.6.a.

Voting machines with electronic image displays shall not require page scrolling by the voter.

Conformance: No

Comments: Scrolling is used in magnified mode

Scrolling in magnification caused test voters to say:

"Large print did not show entire screen, so some races were missed."

"scrolling was too slow when using magnified text - need to have both audio and video."

"far too long to scroll and screen was jumping"

"scrolling bothered my eyes"

Section: 3.1.6.b.

The voting machine shall provide unambiguous feedback regarding the voter's selection, such as displaying a checkmark beside the selected option or conspicuously changing its appearance.

Conformance: Yes (for visual)

Conformance: No (for audio)

Section: 3.1.6.c.

If the voting machine requires a response by a voter within a specific period of time, it shall issue an alert at least 20 seconds before this time period has expired and provide a means by which the voter may receive additional time.

Conformance: No

Comments: forces voter back into Help or language selection menu.

Section: 3.1.6.d.i.

On touch screens, the sensitive touch areas shall have a minimum height of 0.5 inches and minimum width of 0.7 inches. The vertical distance between the centers of adjacent areas shall be at least 0.6 inches, and the horizontal distance at least 0.8 inches.

Conformance: Yes

Comments:

Cast Ballot: 28 mm by 34 mm

Help: 11 mm by 32 mm

Previous, Next: 20 mm by 20 mm

Enter: 50 mm by 32 mm

Select: 52 mm across

11 mm to 25 mm between buttons

Some screen buttons, like Help are not tall enough, and others are too close vertically.

Some voters felt that they had to press this system's screen too hard and said:

"had to press too hard to make choices".

It appears that they may actually have needed to press longer, rather than with more force. Usually, pressing harder increases a voter's touch dwell time.

Suggestion: Consider increasing the size of short keys and vertical separation of those that are too close. Also, consider decreasing the minimum dwell time for accepting screen touches.

 

Section: 3.1.6.d.ii.

No key or control on a voting machine shall have a repetitive effect as a result of being held in its active position.

Conformance: Yes

3.2.2 Vision

Section: 3.2.2.1 Partial Vision

Section: 3.2.2.1.b.

The accessible voting station with an electronic image display shall be capable of showing all information in at least two font sizes, (a) 3.0-4.0 mm and (b) 6.3-9.0 mm, under control of the voter.

Conformance: No

Comments: See Section: 3.1.5.d. for list of font sizes. Cannot complete the voting process in the larger font size.

The screen write lag times of the Sequoia Edge and Edge II are so long that they seriously degrade the voters' patience and cause fatigue and frustratingly long voting times.

The Sequoia Edge magnified mode screen update timing (using Proposition 1A race screen)

• On Edge I, 14 seconds to set up the next page, and 24 seconds to set up a horizontal scroll.
• On Edge II, 7 seconds to set up the next page and 8 seconds to set up a horizontal scroll.

Section: 3.2.2.1.d.

An accessible voting station with a color electronic image display shall allow the voter to adjust the color or the figure-to-ground ambient contrast ratio.

Conformance: Yes

Section: 3.2.2.1.e.

Buttons and controls on accessible voting stations shall be distinguishable by both shape and color.

Conformance: Yes

Comments:

Keypad, top to bottom

Volume: 8 mm

Speed: 8mm by 21 mm

Help: 21 mm by 21 mm

Back/Next: 24 mm by 21 mm

Select: 25 mm

Distance between buttons: 16mm to 18 mm

Section: 3.2.2.1.f.

An accessible voting station using an electronic image display shall provide synchronized audio output to convey the same information as that which is displayed on the screen.

Conformance: No

Comments: One test voter refused to test the system when he realized that it would not give him audio and visual output simultaneously.

3.2.2.2 Blindness

Section: 3.2.2.2.b. The accessible voting station shall provide an audio-tactile interface (ATI) that supports the full functionality of the visual ballot interface, as specified in Subsection 2.3.3.

Instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if this is normally performed by the voter on comparable voting stations

Conformance: Yes

Comments: The location of the card slot is difficult to find and is so close to the stand below that it is difficult to get hand into the area while holding the card. The card does not have any clear tactile orientation label for insertion help.

Test voters and poll workers said:

"Access card was easy to insert."

"card reader hard to access."

"case itself is an obstacle for inserting card"

"should alert voter to remove card after ballot is cast"

Suggestions: Consider adding intuitive tactile and large print indicators on the card as clues for proper insertion orientation.

Instructions and feedback to the voter on how to operate the accessible voting station, including settings and options (e.g., volume control, repetition)

Conformance: Yes

Suggestions: Consider providing large print and braille instruction materials.

Instructions and feedback for navigation of the ballot

Conformance: Partial

Comments: Voters found navigation difficult because the Next and Previous keys were multimodal, sometimes moving up and down through a contest, and other times moving by contest. The Select key is multimodal and could change function mode in the middle of an audio message. For example, changing from select/deselect key to Exit Contest. Test voters said:

"too much waiting through listing of each contest over and over - couldn't just directly go to end of contest you wanted to change, help button didn't offer any info on how to do that"

"Design is not helpful or intuitive, cannot move through things quickly."

Suggestion: Consider changing to use a keypad with up and down arrow keys to move through choices in a contest, and left and right keys to move back and forth by contest. The Select key could be at the center of this "cross". This approach prevents the mode switching of keys for different navigation functions, keeps voters from accidentally backing out or proceeding out of a contest, and limits the modes of the select key to a select/deselect.

Instructions and feedback for contest choices, including write-in candidates

Conformance: Yes

Test Voter Comments: "write-in took too long."

"write-in was very difficult."

" [Screen] write-in keys need to be farther apart"

"keyboard wasn't clear for write-in - 'V' looked like 'U'."

"Wouldn't have known how to backspace over entry to avoid redoing write-in."

"had to go to end of letter/number sequence to finish. "

Suggestion: Consider removing all non-essential options such as numbers from the write-in menu. Give more instructional help upon start of a write-in.

Instructions and feedback on confirming and changing selections

Conformance: Partial

Comments: Many voters had trouble understanding the procedure for deselecting a current choice before choosing another. The deselect-before-change approach may be helpful for preventing accidental touch screen modifications of choices, but audio voters should be able to simply choose a new choice and have any previous choice automatically deselected, as it is done on the eSlate.

Test voters and poll workers said:

"need better instructions on touching to deselect."

"Need to hit select before actually hearing candidates."

Suggestions: Considering providing voters with more helpful instructions for how to make or change a choice. Consider change to not require manual deselect. Consider adding a unique, uninterruptible short earcon sound when moving onto the current choice in a list. Announce the Selected status at the beginning of a choice, rather than at the end.

Instructions and feedback on final submission of ballot

Conformance: Partial

Comments: There should be a simple way to cast ballot without plodding slowly down through the whole ballot.

As test voters said:

"a little hard to figure out during casting ballot at end."

"at end, instructions were unclear."

"no help button when needed, it didn't say when you were final and couldn't make any more changes."

"It wouldn't go to last screen easily"

"The final menu is confusing and misleading, as it seems to be informing you that you are done voting, rather than letting an audio user know that they have choices of reviewing the ballot or wrapping up and casting the ballot."

Regarding the help and instructions test voters and poll workers said:

"Need better instructions ."

"Poor and confusing."

"too many instructions in the beginning and nothing during to guide you"

"synopsis of propositions was very helpful but it was hard to tell what was going on with the instructions given and easy to get lost."

Suggestions: Considering rewriting all the scripts, incorporating the script suggestions of the Cook County Sequoia committee.

Avoid ambiguous messages that can be confused as status or option choices.

Section: 3.2.2.2.b.i.

The ATI of the accessible voting station shall provide the same capabilities to vote and cast a ballot as are provided by other voting machines or by the visual interface of the standard voting machine.

Section: 3.2.2.2.b.ii.

The ATI shall allow the voter to have any information provided by the voting system repeated.

Conformance: No

Comments: No direct control to repeat

Suggestions: Add a "repeat last message" control, or least automatically repeat it if there is a time out from a long voter inaction. Note the automatic time-out-triggered instructions or message repeat should not require voter action to ignore and continue. The current time outs that return voter to help or the language menu could probably be removed.

Section: 3.2.2.2.b.iii.

The ATI shall allow the voter to pause and resume the audio presentation.

Conformance: No

Comments: This was requested by many test voters.

Section: 3.2.2.2.b.iv.

The ATI shall allow the voter to skip to the next contest or return to previous contests.

Conformance: Yes

Comments: This would be much simpler if a cursor "cross" approach was used, as suggested above.

Section: 3.2.2.2.b.v.

The ATI shall allow the voter to skip over the reading of a referendum so as to be able to vote on it immediately.

Conformance: No

Comments: Selection status comes at the end of the whole proposition, so voter must wait to hear it.

Suggestions: Consider announcing if selected at the beginning, also possible use of a short earcon sound.

 

 

Section: 3.2.2.2.c.i.

The ATI shall provide its audio signal through an industry standard connector for private listening using a 3.5mm stereo headphone jack to allow voters to use their own audio assistive devices.

Conformance: Yes

Section: 3.2.2.2.c.iv.

A sanitized headphone or handset shall be made available to each voter.

Conformance: No

Comments: No sanitizable headphone covers were supplied by the vendor.

Suggestions: Throw-away headphones have such poor sound reproduction quality that they would not be acceptable.

Throw-away sanitary headphone covers should be provided, but they should be of a type that does not distract the voter by sliding off too easily, as some are known to do.

Section: 3.2.2.2.c.v.

The voting machine shall set the initial volume for each voter between 40 and 50 dB SPL.

Conformance: Yes

Comments: 75 dB with supplied headphones.

Section: 3.2.2.2.c.vi.

The voting machine shall provide a volume control with an adjustable volume from a minimum of 20dB SPL up to a maximum of 100 dB SPL, in increments no greater than 10 dB.

Conformance: Yes

Comments: Sequoia Edge Volume Measurements from supplied headphones with no in-line volume control:

The lowest volume is unintelligible and barely audible. Normal volume level is 75 dB. The highest volume is 100 dB. Volume step size is 5--6 dB per step, and has about 10 steps.

Section: 3.2.2.2.c.vii.

The audio system shall be able to reproduce frequencies over the audible speech range of 315 Hz to 10 KHz.

Conformance: Not tested.

Section: 3.2.2.2.c.viii.

The audio presentation of verbal information should be readily comprehensible by voters who have normal hearing and are proficient in the language. This includes such characteristics as proper enunciation, normal intonation, appropriate rate of speech, and low background noise. Candidate names should be pronounced as the candidate intends.

Conformance: No

Comments: Test voters and poll workers said:

"Constant hum is annoying."

"Constant hum is annoying."

Wants "less distortion in voice."

"quality of speech okay, just a little fuzzy sometimes. "

"poor quality speech, lots of hiss."

The speech volume of different messages was not well normalized to keep it constant.

"Volume fluctuates too much."

Suggestions: Consider better normalizing of volume and pitch of recordings.

Section: 3.2.2.2.c.ix.

The audio system shall allow voters to control the rate of speech. The range of speeds supported should be at least 75% to 200% of the nominal rate.

Conformance: No

Comments: Sequoia Edge speech rate is 71% to 135% of normal speech rate.

Time to read Sequoia help instructions with audio: 38 seconds for slowest speed is 71% of normal; 27 seconds for normal speed; 20 seconds for fastest speed is 1.35 times normal.

Rate control technology caused significant "chipmunk" pitch distortion. Several test voters complained about audio distortion.

Suggestions: Consider using VSC (Variable Speech Control) methods of rate control, to avoid pitch distortion.

Section: 3.2.2.2.d.

If the normal procedure is to have voters initialize the activation of the ballot, the accessible voting station shall provide features that enable voters who are blind to perform this activation.

Conformance: Yes

Comments: But dexterity problems for card insertion.

Suggestions: Consider raising position of slot so it is not easier to find by touch, as it almost seems to be recessed.

Section: 3.2.2.2.e.

If the normal procedure is for voters to submit their own ballots, then the accessible voting station shall provide features that enable voters who are blind to perform this submission.

Conformance: Yes

Section: 3.2.2.2.f.

All mechanically operated controls or keys on an accessible voting station shall be tactilely discernible without activating those controls or keys.

Conformance: Yes

Comments: The keys of the ATI do not have dished or high friction tops, which makes it difficult for mouthsticks to be aligned on them without slipping off or accidentally activating.

Suggestions: Consider dishing, "fencing", or non-skid surfacing button tops.

Section: 3.2.2.2.g.

On an accessible voting station, the status of all locking or toggle controls or keys (such as the "shift" key) shall be visually discernible, and discernible either through touch or sound.

Conformance: Yes

Section: 3.2.3 Dexterity

Section: 3.2.3.b. All keys and controls on the accessible voting station shall be operable with one hand and shall not require tight grasping, pinching, or twisting of the wrist. The force required to activate controls and keys shall be no greater 5 lbs. (22.2 N).

Conformance: Partial

Comments: Keypad is awkward to hold, but must be held in one hand and pressed with the other.

Suggestions: Consider changing to use a "cursor cross" style keypad, and developing a proper parking place on the system, where it is at the correct height and angle for "touch typing" operation by tall or short voters.

Section: 3.2.3.c.

The accessible voting station controls shall not require direct bodily contact or for the body to be part of any electrical circuit.

Conformance: Yes

Section: 3.2.3.d.

The accessible voting station shall provide a mechanism to enable non-manual input that is functionally equivalent to tactile input.

Conformance: Yes

Comments: Plug interference/clearance does not permit standard jelly switches.

Section: 3.2.3.e.

If the normal procedure is for voters to submit their own ballots, then the accessible voting station shall provide features that enable voters who lack fine motor control or the use of their hands to perform this submission.

Conformance: Yes

3.2.4 Mobility

Section: 3.2.4.a.

The accessible voting station shall provide a clear floor space of 30 inches (760 mm) minimum by 48 inches (1220 mm) minimum for a stationary mobility aid. The clear floor space shall be level with no slope exceeding 1:48 and positioned for a forward approach or a parallel approach.

Conformance: N/A

Comments: Requirement for voting station setup space.

Section: 3.2.4.b.

All controls, keys, audio jacks and any other part of the accessible voting station necessary for the voter to operate the voting machine shall be within reach as specified under the following sub-requirements:

Section: 3.2.4.b.i.

If the accessible voting station has a forward approach with no forward reach obstruction then the high reach shall be 48 inches maximum and the low reach shall be 15 inches minimum.

Conformance: No

Comments: Highest is 52 inches in vertical position

Suggestions: Provide a better thin topped, height adjustable, stand or table that permits better approach. You want flexibility in knee clearance height under the table but want it to be as low as possible so the voter can reach the top of the screen and the card slot. The thin top avoids wasting vertical clearance.

Section: 3.2.4.b.ii.

If the accessible voting station has a forward approach with a forward reach obstruction, the following requirements apply:

The forward obstruction shall be no greater than 25 inches in depth, its top no higher than 34 inches and its bottom surface no lower than 27 inches.

Conformance: N/A

If the obstruction is no more than 20 inches in depth, then the maximum high reach shall be 48 inches, otherwise it shall be 44 inches.

Conformance: N/A

Section: 3.2.4.b.iii.

Space under the obstruction between the finish floor or ground and 9 inches (230 mm) above the finish floor or ground shall be considered toe clearance and shall comply with the following provisions:

Toe clearance shall extend 25 inches (635 mm) maximum under the obstruction

Conformance: N/A

Comments: Knee clearance, not toe clearance is the determining factor limiting the approach.

Suggestions: Provide a better thin topped, height adjustable, stand or table that permits better approach. You want flexibility in knee clearance height under the table but want it to be as low as possible so the voter can reach the top of the screen and the card slot. The thin top avoids wasting vertical clearance.

The minimum toe clearance under the obstruction shall be either 17 inches (430 mm) or the depth required to reach over the obstruction to operate the accessible voting station, whichever is greater

Conformance: N/A

Comments: See previous requirement.

Suggestions: See previous requirement.

Toe clearance shall be 30 inches (760 mm) wide minimum

Conformance: No

Comments: 26.5 inches between legs at front.

Suggestions: See previous requirement.

Section: 3.2.4.b.iv.

Space under the obstruction between 9 inches (230 mm) and 27 inches (685 mm) above the finish floor or ground shall be considered knee clearance and shall comply with the following provisions:

Knee clearance shall extend 25 inches (635 mm) maximum under the obstruction at 9 inches (230 mm) above the finish floor or ground.

Conformance: No

Comments: Not wide enough.

The minimum knee clearance at 9 inches (230 mm) above the finish floor or ground shall be either 11 inches (280 mm) or 6 inches less than the toe clearance, whichever is greater.

Conformance: No

Comments: Not wide enough.

Between 9 inches (230 mm) and 27 inches (685 mm) above the finish floor or ground, the knee clearance shall be permitted to reduce at a rate of 1 inch (25 mm) in depth for each 6 inches (150 mm) in height.

Conformance: No

Comments: Not wide enough.

Knee clearance shall be 30 inches (760 mm) wide minimum.

Conformance: No

Comments: Not wide enough.

Section: 3.2.4.b.v.

If the accessible voting station has a parallel approach with no side reach obstruction then the maximum high reach shall be 48 inches and the minimum low reach shall be 15 inches. See Figure 3.

Conformance: No

Comments: Top of screen is 52 inches

Suggestions: Use alternative stands or tables.

Section: 3.2.4.b.vi.

If the accessible voting station has a parallel approach with a side reach obstruction, the following sub-requirements apply:

The side obstruction shall be no greater than 24 inches in depth and its top no higher than 34 inches.

Conformance: No

Comments: Top of screen is 52 inches

Suggestions: Use alternative stands or tables.

If the obstruction is no more than 10 inches in depth, then the maximum high reach shall be 48 inches, otherwise it shall be 46 inches.

Conformance: No

Section: 3.2.4.c.

All labels, displays, controls, keys, audio jacks, and any other part of the accessible voting station necessary for the voter to operate the voting machine shall be easily legible and visible to a voter in a wheelchair with normal eyesight (no worse than 20/40, corrected) who is in an appropriate position and orientation with respect to the accessible voting station

Conformance: Partial

Comments: The VVPAT is hard to read.

3.2.5 Hearing

Section: 3.2.5.b.

If voting equipment provides sound cues as a method to alert the voter, the tone shall be accompanied by a visual cue, unless the station is in audio-only mode.

Conformance: Yes

Test Voters' Comments: "want photos of candidates."

"would be nice to have pictures of candidates for identification -- all voters benefit"

"Want colors used as a party ID coding."

3.2.7 English Proficiency

For voters who lack proficiency in reading English, or whose primary language is unwritten, the voting equipment shall provide spoken instructions and ballots in the preferred language of the voter, consistent with state and federal law. The requirements of 3.2.2.2 (c) shall apply to this mode of interaction.

Conformance: No

Comments: The Sequoia Edge II system was not able to demonstrate the ability to practically handle real alternative language ballots for either Spanish or Chinese. For the visual portion of the Chinese ballot, there were only character place holder boxes, no actual Chinese characters. It was not clear if the tested system is capable of displaying non-Roman fonts.

Neither of the Spanish or the Chinese ballots were functional enough to be used by our test voters to complete a vote. At the time of this report, vendor representatives have not explained how much of the lack of alternative language support is due to incomplete elections or ballot definitions and how much was due to an inability of the system hardware/software to support the languages properly.

Suggestions: Because non-Roman fonts for languages such as Chinese may require high performance from the voting system's video and graphics handling hard/software, it is important to confirm and test this system's capability to actually fully support the non-Roman languages.

As was observed on the audio and visual ballots of the other tested systems, there also may be translation errors, as well as speech recording concerns on this system as well. When developing its alternative language support, it would be good for the vendor to have professional language translators try voting on the system, with only the audio output and then to vote again with only visual display. Then ask them to help translate and make translation scripts more accurate and less confusing.

Jumping into the Language Menu

There were serious concerns about the systems' propensity to mysteriously force the voter back into the language menu. This happened so often, even late in the voting session, that it was considered to be a bug or at least a design flaw. In the case of one of our voters, who hadn't expected to be placed back in the language menu, she ended up getting Chinese selected. Because the audio ballot was speaking the names in English, she didn't realize she had switched languages. She initially thought she was just having trouble understanding the messages.

It appears that the system can jump back to the language menu from timeouts (without warning the user) as well as from simply pressing the Next or Select keys at certain times.

As one voter said: "It shouldn't be so easy to get to language choice again."

D. Timing Results for Phases of the Voting Process

We measured the time it took each user to perform separate tasks within the total voting process:

• Orientation
• Free voting (user-paced voting from the beginning to the end of the ballot)
• Returning and changing a vote
• Write-in vote
• Reviewing and casting the ballot

The three measures that are the most useful are free voting, write-in, and changing a vote.

Free voting averaged 8.95 minutes. The Hart eSlate averaged 9.97 minutes; the Diebold TSX averaged 6.67, and the Sequoia Edge averaged 10.27 minutes. Note: Free voting times should not be strictly compared across the systems because their ballot definitions were not identical.

Changing a vote averaged 1.98 minutes. The Hart eSlate averaged 2.02 minutes; the Diebold TSX averaged 1.64, and the Sequoia Edge averaged 2.29 minutes.

Write-in voting averaged 3.27 minutes. The Hart eSlate averaged 2.90 minutes; the Diebold TSX averaged 3.25, and the Sequoia Edge averaged 3.68 minutes.

The Sequoia Edge took the longest time for all three separate functions.

As expected, there was a significant difference between the times required by voters using the audio interface and those using the visual interface. Audio interface users took 2.43 times longer to free vote on the Hart eSlate, 3.12 times longer on the Diebold TSX, and 4.04 times longer on the Sequoia Edge.

E. Comments on All Systems from Study Participants

E.1. Subjects' and Pollworkers' Comments on the Diebold AccuVote TSX (System B)

Subject 01

"I vote absentee, but if I had to use this system I would be happy to use it"

"home key, #5 to be more pronounced; worked well"

Subject 02

"pretty easy" [note: voter used keypad and touch screen simultaneously]

"no changes; clear and concise, contrast and colors made it easy to read; good repetition"

Subject 03

prefers 'A'

"audio wasn't suitable which led to me becoming confused; I wish I could see the screen better because keypad was difficult and large print was not large enough."

Subject 04

I have confidence in this machine; good that the keypad is hand held

would like to see paper trail more visibly; need wider leg stance or machine should be on a table; slow instructions makes my mind wander -- too slow

Subject 05

rather use system 'A'

"too much chance for errors; keypad hard to use; hard to memorize; hard to tell proper position.

He wants a pause key.

In write-in, he several times inadvertently cancelled his whole write-in by pressing the wrong key (probably the '#' key).

He did not know which keys have what letters on them.

"Audio was good."

Keypad was his biggest concern.

System A is his favorite.

"Very few people know the phone keypad letters."

He actually didn't finish accepting the casting, but that was maybe the poll worker's fault.

Subject 06

"should announce first and last names, space bar, help file for write-ins to identify letters

"The write-in should tell you what keys have which letters to start with."

"Write-in review should announce space."

Wanted bigger delays between messages, especially in Help Information.

Voter liked being able to interrupt and skip over the rest of a message.

Subject 07

"very simple, no need for improvement, very accessible and easy to use"

In write-in, it does not speak 'space' when reviewing.

pound key is for cancel write-in and star is for cancel character.

In write-in, he thought he had to delete a character, although he hadn't accepted with '0'.

In changing a Yes/No judge race, it referred to the yes and no choices as candidates, instead of choices.

"Pauses may be helpful to give you time to think, but just not quite so long."

Subject 08

write-in is difficult; key assignments should be consistent

Subject 09

"had a little difficulty inserting access card, took several tries, had to go through entire ballot, no key enabling you to go directly to review. could have sped up speech a little, but didn't get around to it; overall very easy to use."

Subject 10

"needs a cover to reduce glare; like B better than A or C; needs a pause, fast forward, rewind buttons"

Subject 11

space button should be clarified; hard to do write-in

Subject 12

not comfortable sitting at the machine; card slot inconvenient; speech speed either too fast or too slow

Subject 13

prefers absentee

Subject 14

"I like electronic voting, it's easier"

"keypad works easier, but touch screen is good too"

Subject 15

"needs pause button, skip contest, FF/REW; during write-in audio can't be slower or speeded functions 9,7,5 a little confusing; candidate names are clear but other audio is muffled"

Subject 16

"increments of speed control too wide; user should be able to choose male or female voice, I had trouble remembering the keypad, as keys were used for different purposes -- inconsistent; took a lot of concentration. prefers synthesizer, wants to hear all of proposition, "

Subject 17

No comments

Subject 18

would like loading screen at beginning and rejection point;

"confusing because it is new; for review you should be able to change the size of the font so you can see the whole page without scrolling"

Subject 19

get rid of VVPAT - not for voters with impairment; unable to insert or remove card -- VVPAT box obstructed; had to change angle to accommodate wheelchair

Subject 20

card slot should be located at bottom; once you understand the select button is used repeatedly it is easier.

Subject 21

"propositions are difficult to read and compare with verification ballot"

"they all take too much time to learn and you only vote infrequently, instructions have to be easy, clear steps with simple language"

Subject 22

"too complex, confusing; it wasn't easy to pick a category-too hard to move between races; C was easier"

he accidentally entered a character during write-in because his hand was still on the touch screen when the touch screen keyboard came up

Subject 23

machine should be a little lower for seated person

Subject 24

needs instruction on how to change your vote

Subject 25

"needs instructions on how to change vote, or automatically change; pictures of candidates; should explain propositions, should say 'help' and 'instructions'"

Subject 26

have access card at bottom for easier input

Subject 27

"concerned about computer voting machines, worries about fraud, partisan mfr."

wants to see pictures of candidate - head shot -- icon of party

Subject 28

"good to have large print"

Subject 29

"not easy to change a vote, can it have options, tools to point on the screen; is using the touch screen healthy (contagion)?"

Subject 30

poor angle for using touch screen for seated voters back and forth too low on screen and hard to press didn't cast ballot on 2 propositions as no info was presented

Subject 31

Within a race, she would like to have a control to announce the name of the race.

It calls the Yes and No candidates.

"This was pretty good."

"I liked being able to speed it up."

"Once you knew to hit 6 it went okay."

"It was very clear on which button to press."

"Using number keypad for write-in was pretty easy for me."

"After reviewing, there should be a simpler way to cast ballot without having to move down to the end."

"It's pretty easy."

Subject 32

He is blind and might be able to use audio and video.

He would like the keypad parked down lower.

Party statement after candidate name should say 'party' after the name of the party.

"To cancel your vote for Barbara Boxer," (big pause) "Press 5." makes it sound as though you are being told tpress 5, as the pause makes the following seem like an unrelated command.

The way it says, 'There are 3 candidates and 1 write-in in this race.' is better than systems that would just say, '4 choices'.

Also, saying, 'Vote yes or no.' is more helpful than saying, 'You have 3 choices.'.

'No votes have been cast in this race.' is similarly better than other systems that say something like, '0 candidates selected.'

He would like a control to ask for the name of the current race.

In modifying, the big pause before 'Press 5' caused him to press 6 to move on and miss the 'Press 5.' message.

Pound key cancels the write-in function.

There should be a summary of which letters go on which keys.

"7" is to continue with write-in, too close to the cancel last.

Key '5' is sometimes used for cancel and sometimes confirm, that seems contradictory, so causes confusion.

Needs a way to abort the help instructions.

"Do I keep the print out?"

"Moving through candidates of a race should be with up and down arrows and the list should wrap around."

"Write-in would be difficult for people, unless they text message a lot. Maybe could have a full keyboard."

"Help instructions should be more contextual."

He would like to be able to read help line by line.

He would like a repeat last line or message control.

He was able to follow the screen position visually some, but was counting mostly on the audio for reading text.

He would prefer white on black.

He would like a linear ballot with up and down arrows to move through all the candidates.

The keypad mount should be a horizontal slide to let you position it left, middle, or right.

Subject 33

Needs braille instructions

She is totally blind.

The system does announce the space during write-in review.

"I like it. I like the way it lets me move through the races and hear my selections. I like this machine, it is clear. I can increase the rate, which is important. I like the female voice. It doesn't give me all the detailed descriptions of the propositions, maybe needs a verbosity control. The keypad is familiar, like a phone. My concern is the big expense to tax payers. I would have a braille instructions sheet about making changes or modifying a vote. On line or telephone instructions material for pre-voting learning would be good. I would use this system for voting."

Subject 34

"more choices in contrast, esp. white letters on black background; extra magnification

She is a high partial who uses touch screen and does not want to use audio.

She stood up to read part of the vote and sat to read the lower screen choices.

"I have only one negative thing to say about it."

She would have liked larger magnification.

She would have liked more choices for magnification and contrast.

She did use high contrast.

"The magnification lens on the VVPAT is chintzy."

"The fresnel lens wouldn't have helped."

Subject 35

"audio detail can be confusing, esp. different between yes/no and candidates; size of font needs to be consistent; magnifier doesn't work for me; should have multiple settings for magnified text, audio is clear; should have more space between text and lines/borders; text should be bold"

Subject 36

use 'yes' and 'no' for propositions

Subject 37

speed up announcements of controls; keypad was easy; didn't need all keys

Subject 38

"legs too close together, access card input too high"

Subject 39

printer should be pushed back some; should have a pause option

Subject 40

second and third voices need to be clearer

Subject 41

"should be clearer how to go back to previous race

The system started up by rejecting the ballot card and requiring another card.

In the instructions screen, it looks like the instructions could be selected by touching.

She had to start over because she whizzed through with no feedback verbalizing and cast before she could do other tests.

The deselecting a candidate requirement was not obvious and confused her for a minute.

She needed the back of the unit tilted so she could reach it to use touch screen.

We had to put blocks under the back legs to tilt it up enough.

She wants a previous race control to review it.

She feels strongly that there are serious security concerns about the Diebold DREs.

Subject 42

"could have been wider, privacy screen is sharp, kept hitting with elbow"

The stand for the machine is not wide enough for her to get close enough.

She ended up lining up sideways.

She will try voting touch screen and using high contrast.

Touching is harder than the C system she tried, because of needing higher pressure and needing longer dwell.

The privacy shield is getting in the way of her left elbow.

"The privacy shield should fold back."

She did not know how to switch a vote selection and had to ask for help.

She had a lot of trouble writing in 'E' because letters on the left cause her to hit the left privacy panel with her elbow.

She is going to try to also vote with audio and keypad

She has neuropathy, so she doesn't get tactile feedback from keypad.

She is finding keypad control easier than touch screen.

"I'm not particularly crazy about this machine."

"Over all, I prefer the other touch screen."

"This took longer to move from race to race."

If she had just the tablet in her lap, she could have voted better on touch screen.

A non skid rubber sheet might help keep it from sliding off her lap.

She would want to be able to tilt it up in her lap.

If she had to use the unit not in her lap, she would rather use the keypad and audio.

She would prefer absentee voting over using this machine.

Maybe should include an accessible voting polling place notice with the sample vote mailing.

"Voting independently can make you feel more "normal":

Subject 43

"change printer to side of machine to make screen more accessible; printer privacy cover not accessible

He is a quad and having to have the machine tilted way forward to be able to reach the top of the screen. We put blocks under the back legs to tilt it forward more.

He cannot get his chair and arm control under the machine.

He had quite a bit of trouble hitting the right letters in write-in, because the targets were too small.

"I like the sensitivity of the touch screen."

The printer blocks his elbow and makes it too hard to touch the Next Screen Icon, in the lower left corner.

The printer makes it hard for someone to get close enough, especially if they have to approach sideways.

"I would move the printer to the side further, or to the rear, or inset more."

"I'd move the Next key to the center, to avoid the blocking of the printer or left privacy panels."

"I'd allow for more range on the tilt of the machine."

The claim the ballot printed on the paper trail is the same as in the electronic memory is misleading.

Subject 44

"could not insert card; printer was in my way; keypad and printer location awkward; should be lower; privacy could be better; move access card slot at bottom, no recess; move printer; require less pressure on touch screen improve magnifier - blurry

She uses a mouth stick.

We had to put blocks under the back legs to tilt it forward enough for her to touch top of screen

To make touches, she said, "You have to hit it pretty hard."

"The lens over the VVPAT kind of makes me want to throw up, because of the stretching."

"The numpad and VVPAT is in the way and awkward."

"Also inserting and removing the voter card is impossible for me."

"The machine is high so I have to do a stretch to reach the top of the screen on some choices."

"The magnifier was very stretched."

She could open VVPAT cover, but she could not get or keep the lens out of the way.

"It seemed to me that I had to hit the screen pretty hard. Maybe if I had just stayed longer it would have worked."

"I would remove the VVPAT and numeric keypad."

I would move the voter card down and not recessed."

"I would make it easier to press the screen buttons."

"I would make the magnifier better and easier to move out of the way with a mouth stick."

"The privacy on this is not as good as the A system."

Subject 45

"include under keypad what function each button does would be quicker than waiting for audio

He has a learning disability, dyslexia.

He is using both the audio and video, and mostly keypad, but a little touch screen.

He would like a key identifier function or more on-screen reminder of possible key functions.

He used Next when he was trying to change the judge selection and it took him into final ballot cast.

"This was my favorite of all 3 systems."

"It was pretty easy."

"It helps me and others with disabilities to have both audio and visual output."

"I would put a possible key function list on screen."

It was not always obvious if Next meant page of ballot, race, or candidate."

E.2. Subjects' and Pollworkers' Comments on the Hart eSlate/DAU (System A)

Subject 01

speed of speech instruction was too fast; need to pause before next instruction; Voice is distorted and too fast.

Help runs without a pause.

Too much help info, too quick, can't absorb it.

Said that it needs to have help slower and with pauses to think.

He was going to press "cast ballot" to try to select a candidate, because he didn't absorb or remember it telling him to use Enter, so poll worker interrupted him to prevent premature casting.

Prop 70 Tribe Gambling Compacts -- could not be understood.

Female voice is fuzzy and distorted.

In the write-in, he could not review or play back what he had already put in.

Asked him, "What should be changed?"

"Instructions should be slower and pause between messages."

Subject 02

"make female voice clear and distinct; black and white hard to read; should be bolder; blue was easier; font too small; She said she didn't think she was using the wheel enough.

Her arm got tired from holding up to the keys.

"It was pretty easy."

Most of the print is not bold and big enough.

Text highlighted in blue was readable.

Contest names were readable.

Wants bigger font.

Female voice was harder to understand. Male was clear.

Wants female voice sharper.

Subject 03

"if only I had a dress rehearsal for practice; a lot of things coming at you at once -- speed of sound, dial use, etc.; speech was too fast and hard to understand"

"switch turned too quickly, missed some choices and dial too sensitive to touch -- unintentionally moved if knuckle touched dial; got frustrated with a lot of things to focus on and remember."

"I'm having a hard time hearing it."

I had her turn up the volume a bit.

She pressed the red key prematurely, so jumped into review mode.

"Speech is too fast."

"Does not give a chance to breathe and think." "Almost so frustrated that I want to quit."

"I'm sort of like clinching my hand."

"It really makes you concentrate."

"It makes you prepare before so you won't be as confused, like go over your ballot at home."

The wheel is too easy to turn inadvertently.

She maybe skipped some propositions because of bumping the wheel accidentally.

"The wheel is "kind of scary".

She wanted slower speech.

Subject 04

"make the written instructions easier to see and find written instructions need to match the machine language. too low for wheelchair access, needs better lighting

"After making a selection, it should leave the highlighted choice and pause for a few seconds for you to catch that it took the selection."

In casting verification modify, he entered on 'No' choice of judge. It left it undervoted, so he had to go back and select 'Yes'.

"The printer is only showing print out through Prop 67, not 72."

That was because he did not realize that it was page 1 of 2.

Audio intersperses reviewing contests with 'Accept' or 'Reject' prompts.

Voter instructions are in the dark, because of privacy screen shading.

He likes the glare guarding above.

Voter instructions should be out in the light.

If looking for them, he thinks he wouldn't have found Spanish or Chinese instructions under the English.

The Spanish menu select should give help that has directions for finding the Spanish print instructions on the side of the unit.

He felt the unit was too close to the floor.

Knee clearance is not high enough, so he tilted back the unit to get knees under it.

Subject 05

There are no instructions on how to changes pages in review mode

Subject 06

possibly have screen turned off; like tactile buttons with braille was not clear how to change vote during review

Subject 07

"It's good the way it is."

Subject 08

Tried Next and Prev to go through candidates.

"So far it is very clear."

Wanted full text of propositions.

What I like about the wheel is that is easier than old versions to select without accidentally selecting than older units.

Counter-clock wise seems to cause undervote.

It is confusing that it does not give contest number in one direction, but does in the other.

He was not sure that it would let him know about undervotes.

It should have a write-in name review.

Casting review is different and confusing because it deselects the 'candidates' (choices).

Casting button is sort of doing what he uses the Enter for, so it seems redundant or contradictory, and causes confusion.

"The voice is clear."

"Does not need help much."

"Write-in is the best I've ever seen."

"Write-in should have review."

"There should be a turbo mode everywhere."

"The machine is not consistent."

"Need beep if you try to turbo past an important screen."

"Write-in should have an option for a review request."

Subject 09

liked that the ballot read out as it printed; using select wheel was faster than system c; didn't need a lot of verbal instruction, found it to be clear, concise, succinct. layout and speech were good and it did what it said it would do. felt confident

The Propositions talk about 2 choices per contest, instead of just choosing Yes or No.

Repeatedly missed and seemed oblivious to the 'Current selection' message.

It isn't obvious that you use the Cast Ballot button when you want to start the casting.

Had to press the cast ballot button more than once.

When retrying the cast ballot, had to press the cast button twice again.

"Generally, I liked it."

In write-in, the wheel selects and skips faster than buttons.

There was not too much verbiage.

"It was more concise."

She felt that she forgot to use the Next and Previous keys more.

"Speech was pretty good."

"It was pretty fast and I felt pretty confident."

Arrows might be hard to find for someone with Neuropathy.

Because it spoke out the choices in verification of casting while it was printing, she thought it was verifying from the paper.

Subject 10

"larger screen and text"

He needs light on dark on the screen.

"Cannot read the first screen."

"I like this system."

"I would have to use the audio."

"I like this machine."

"The voice was very clear."

"It was very explanatory."

"Older people would be very comfortable with this, because it tells them every step to take."

"I wouldn't really change anything.""

Subject 11

"help should have more detail"; liked privacy shroud; very blind friendly

"Spinning is hard for some."

Maybe should have a practice mode.

"Write-in should have had a little more startup help."

Subject 12

Immediately expressed extreme dislike for the privacy enclosure. She feels some folks would find it very claustrophobic.

She thinks proposition descriptions are too sparse.

She had real trouble noticing that it said, 'Current Selection', so she couldn't tell when she was sitting on her current selection choice.

An earcon sound on current selection might help.

She is using her eyesight some, as well as the audio.

It was hard for her to get the system to accept the cast ballot button, had to get out of help first by pressing Enter.

"I would never use a machine like this to cast my ballot."

This is a little easier to read than her previous test machine.

She would not bother trying to correct mistakes, because it's too difficult to correct.

"The wheel wasn't too neat."

"The write-in wasn't too easy."

She needed both the audio and video.

Subject 13

She needs to get up close to read the screen. She is starting out to use the audio with the video.

"I could, as long as I have the headphones and the screen."

Subject 14

"I liked this it's so much easier than absentee, I can do it myself"

The VVPAT and the print instructions are too dark.

He cannot use the screen for anything but spatial position, so he will rely on audio.

In write-in, can't read the text on the right side.

Should change pronouncing of 'M' to sound like 'emma'.

The text is clear but not big enough.

There should be a reminder that you can cast your ballot after you move to the end of the ballot.

"The system is not hard to use."

Seeing the printed instructions is a problem because of shading.

If the camera was not in the way, he would have leaned into the screen and read a little more on screen. He needs to get up to a few inches from the screen.

"The buttons are easy to use."

He was feeling his way to the keys.

He would like to see the print instructions and VVPAT lighted or back lighted.

"No problems with the speech."

 

Subject 15

It says, 'Natural Law Cand.', and that is too cryptic for some folks with visual impairments.

He hit Next when trying to hit Enter, due to lack of tactile difference.

Initially, he spent quite a bit of time wandering around through races, trying to get a feel for how the wheel control worked.

He really likes that he has control to move forward and back through the summary review and interrupt the speech.

He would like a review of the write-in name spelling.

In casting verification, he selected Reject, but it kept talking the review of the page, without canceling or interrupting the review speech. This left him confused.

After he accepted the first page, it started reading the next page without announcing that it was the second page. This confused him. It needs a page transition message.

He really likes the privacy shield tent.

He likes the access code, instead of a voter card.

"I love the braille."

He wanted a speed control to speed it way up. He thinks that he personally could use the fast speech with some chipmunk distortion.

It might help to have a message that it is moving to the next contest.

More help instructions with sentence by message control would be good.

"The keys should be raised to make them more obvious."

Subject 16

"Use human speech next/prev controls for next race, not page - page is irrelevant, write-in did not confirm when done, liked wheel but it didn't always click, would move when I didn't want help option didn't help much.

Wants to speed it up.

Thinks the speech should be higher quality.

She would like the controls of synthesized speech, instead of natural human voice.

Wants the write-in screen to let her know when she has cleared out all entries.

"The 'of' sound is softer."

Daryl Isa's name is mispronounced and should be correct.

She felt that it skipped from race 4 to 5 automatically, without her specific control.

She used the Previous key to try to go back.

She is not going to use Next and Prev keys anymore, because they are unreliable.

She likes when it comes back to the same position in a list, rather than resetting to the beginning.

She started off thinking she would not like the wheel and would prefer arrow keys.

Now she likes the wheel for write-ins.

Wants to have a review of her write-in selection.

She entered on write-in to get back into write-in, to see if she had done it right, and that deselected it and cancelled her previous entry.

"If you lose your place on this system, you are so screwed."

"Sometimes when I press Enter, it doesn't take it."

It should always respond audibly to any keystroke, even if just to tell you why the key entry will be ignored.

She pressed 'Cast ballot' before being prompted and it didn't work.

On reviewing Judgeships, it doesn't give the judge's name, just 'Judge'.

"I'm afraid to try something because it doesn't let me know what is going to happen."

Cast ballot seemed to need to be done at end of ballot, when prompted.

"The wheel is not always taking all my clicks."

"Switching around amongst 3 voices may be a problem for some cognitively impaired folks."

Does not like wheel moving her into a next contest.

Next and Prev keys should move by race (not page).

"I don't like being moved out of the race without my control or expecting it. Maybe auto move to next contest is alright when you've just made a selection."

She would like to cast immediately at any time.

Subject 17

"The voice is hard to understand, should say what happens to ballot after done"; prefers printed ballot

"I find this system quite intuitive. I like the idea of the wheel -- it makes voting more efficient. The voice is quite clear. I'm glad that there is no [voter access] card. I like the idea of a paper ballot."

"I like the braille labels. It helps orient me.

He has no problem with the angled key labels.

Write-in should review with a spell out of the selected letters.

He felt a little unsure about pushing the cast ballot button.

The system said, 'Listen to what is printed', which is misleading on the VVPAT.

"Write-in is slicker with a wheel. The wheel makes it more intuitive"

"I want the controls as integrated as possible."

"Voice interrupted nicely without hicking."

"Some things could be more descriptive in the instructions."

Subject 18

She is hard of hearing.

In review, she would like to have some simpler way to double Enter or something to go back to the beginning of the ballot.

In write-in, maybe arrow keys with up down and left right would be better than wheel.

In casting, when accepting the long response delay before printing makes voters think it did not take the command.

She suggested that it should maybe use Next and Prev to move around in write-in.

Subject 19

didn't like wheel, not enough dexterity - prefer touch screen

Her wheelchair cannot get close enough, because of the arm rests.

She finds it easier to do touch screen then to use the wheel.

She thought that she could not press the cast ballot button hard enough, because it seemed to be ignoring her presses.

"Too much hand movement for me." "it's too hard."

"Can't get close enough."

"It's difficult to change a selection if you change your mind."

Subject 20

liked how buttons were laid out and didn't have to de-select to change

"I like this system. It's much better than the first system I tested (C)."

"I like when it says, 'You've selected Yes for this contest.'"

He would like to hear the longer descriptions on the propositions.

"I like the layout of the buttons on this machine."

"Want propositions described."

"I didn't have to use the help key once."

"I don't know that I'd change anything on this machine."

Subject 21

"I got confused on the wheel, but then I got the hang of it."

Subject 22

"turning the dial is distracting"

He is hard of hearing and moves slowly.

"It's hard to switch between pushing buttons and turning the dial."

Subject 23

"needs lighting on printer"

She is hard of hearing.

"I actually like it. Its pretty cool. It needs a bit more light in the booth."

She had trouble reading the VVPAT, because it needs better light.

Subject 24

"It needs sign language video on ballot, photos of candidates; change wheel to an arrow, like on a remote control"

"It takes a long time to use the wheel for Write-in."

"I'd like to use a hand-held remote control."

Subject 25

"Use an arrow instead of dial; add pictures of candidates"

"It is a lot of dialing. It would be nice to have arrows to move around."

"Many deaf folks don't get the sample ballot in the mail."

"I'd like to have the system communicate with video of ASL signing. It should also have pictures of candidates."

Subject 26

"I like that it blocks the glare and that it is private.

I like the touch screen better than the dial."

Subject 27

Note: He had to leave testing early, before the session on this system.

Subject 28

had to press enter too many times - dial too much movement, very distracting

"The dial seemed to miss some clicks."

She has some ADD and dyslexia, so she has trouble knowing which way to turn the dial.

Subject 29

It needs a light by the printer

She is deaf and uses interpreters.

She was using Next and Prev in review, but did not use the wheel to find race 6.

Not enough light.

"I prefer touch screen. It seemed to take longer than touch screen.

The instructions were easy to follow."

Subject 30

Prefers voter card, didn't like the wheel; instructions for multi-page ballot and casting need to be redone; flashing 'cast ballot' icon is bad, don't need the top part of the privacy screen; help is good

Wants more description of propositions.

"Doing write-in on wheel is hard for people with dexterity impairments."

"This is the most cumbersome of the 3 machines."

"The print instructions were not clear."

She thought that some instructions are better than on the other machines.

She is concerned that the access code might not be private

"The instructions should clarify that the access code is private."

"The weakest point is the ballot casting and verification."

Subject 31

"Easy to move around; speech was clear; instructions were good but not at end"

It needs to say more during cast ballot - didn't say anything about more than 1 page; help is not helpful enough; liked the wheel for moving around and typing in a name

Subject 32

"It seemed strange at first, different controls; instructions not very clear; at first wheel is intimidating, may need to get more comfortable with it before being left alone; instructions unclear at end -- doesn't announce enough, unclear how to vote"

Subject 33

should be able to increase speech rate; braille instructions

Subject 34

"Cast Ballot icon is too small to see, should say what shape, size of button to press, should be more tactile. The access code is not reasonable; should have a card or larger print on paper"

Subject 35

Similar to reading; doesn't jump from page to page

"could improve contrast, more muted colors; LCD was too bright

She is a native Spanish speaker.

The blue background color is too light. She would prefer green.

"There is too much glare."

She put on yellow lens glasses and that cut down the glare. She uses the glasses to prevent glare fatigue and takes them off to be able to read the screen text better.

She was turning the wheel so fast that she moved on to the next race. She was also trying to press down on the wheel hard.

"The Spanish is very clear and good quality. Most other information systems with Spanish do not have such clear Spanish. The introductory part of the candidate choices is a bit stilted and hard to understand. For the contest entries message, It is saying, 'Career of...', and that's not right. 'Carera' can mean 'Race', but it's the 'Career' sense. It should use 'Contesta' and 'Oprima' would be better for 'press'."

There is too much abbreviation on the propositions, she just wanted a little more info about each proposition.

The red outline helps her follow which is her choice on the screen.

"I love this way of voting."

The female Spanish voice is much lower in volume, compared to the male voice.

"I can interrupt it without having to listen to all the instructions for the page."

'Tribal gambling contracts' is not translated properly into Spanish.

In the Modify test, she had to play Help to try to figure out how to change a selection.

If she didn't have the screen to see the options for space and accept on the end of the menu she wouldn't have known to select them and would have thought to try Enter to accept.

In the ballot summary, she noticed that she had several undervoted races.

"It's easier to trouble shoot with this."

She could use her lighted magnifier to spot check the paper trail, but she couldn't read the whole thing.

"The instructions aren't complicated."

"It's nice to not have to put a card in it."

Subject 36

"should be informed that you are moved to the next race automatically"

Didn't know you could turn wheel both directions

Would like demo before election day

She is totally blind.

It should say, 'Next race', when moving to the next race.

In ballot summary, she kept rotating backwards and backing up a race, instead of rotating forward through the candidates

She was not sure if she was on ballot summary page or not, needed a way to announce that.

"At first I didn't know what the wheel did. After I got the hang of it it went okay."

"I might change it to, in the beginning instruct you which way to turn the wheel to do what."

Subject 37

He is partially blind, and can't read the screen.

He can see well enough to track his ballot position on the screen but not well enough to read the text visually.

In ballot marking, the 'Contest ## of 23' should not be the first thing you hear. The first thing should be the name of the contest, followed by the current selection.

To start cast ballot, he was trying to use the wheel to get the screen pointer down to the flashing 'Cast Ballot'.

"I found the dial to be really easy."

"I would use any of these systems, instead of voting absentee."

"I would change it by having more instruction on casting ballot and how to make a selection change."

Subject 38

additional options; directional keypad

He is a C5-C6 quad.

He is spinning the dial just for single clicks. He would prefer a 4 key cursor cross, instead of the wheel. He has neuropathy in his fingers, so he did not get tactile feedback on the wheel.

"In terms of the interface, it looks pretty good."

"If this was available, I would go to vote at the polls."

Subject 39

"The audio is muffled; needs directional arrows, better instructions on summary pages, does not say there are 2 pages, cast ballot button is not round"

Subject 40

Audio was not consistent -- goes from high to low

Subject 41

"It's good that it turns page automatically, but otherwise 'scrap it'"

Subject 42

"privacy screen should be wider, needs up/down adjustment; touch screen would be easier because of little mobility with hands"

Subject 43

No option for using keypad because of privacy shroud

Subject 44

Height should be adjustable so wheel could be turned more easily; use directional buttons instead of wheel; enter key requires too much force

Subject 45

no comments

E.3. Subjects' and Pollworkers' Comments on the Sequoia Edge I and II (System C)

Subject 01

"better instructions -- poor and confusing never able to enter a vote unfamiliar keypad, not like a phone"

Subject 02

"scrap it!"

"large print did not show entire screen, so some races were missed; casting ballot was a problem because it reverted back to small print; far too long to scroll and screen was jumping; scrolling bothered my eyes"

Subject 03

wants to be able to view and hear at same time.

shouldn't be so easy to get to language choice again.

Subject 04

large print screens are too difficult to navigate; small print is too small

Subject 05

need better instructions on how to use the keypad -- took too long to figure out how to navigate; audio was good

Subject 06

"synopsis of propositions was very helpful but it was hard to tell what was going on with the instructions given and easy to get lost; should be able to skip ahead; too frustrating, unable to de-select vote and get back without starting over; kept repeating instructions after every vote and it was way too confusing"

Subject 07

"more precise; too lengthy; operation was confusing, but keypad was simple"

Subject 08

"inconsistent command structure, poor quality speech, lots of hiss; no 'expert mode' menu structure was poor, machine is bad in general"

 

Subject 09

"needs more consistent instruction throughout; write-in took too long, had to got to end of letter/number sequence to finish. access card was easy to insert; quality of speech okay, just a little fuzzy sometimes. wouldn't have known how to backspace over entry to avoid redoing write-in."

Subject 10

black background white letters all the way through

Subject 11

"reduce pauses and let voter know what they've done and how to undo it; write-in was very difficult; access card slot hard to locate; no braille; big gap between instructions, unsure how long to wait - no prompt and out of sequence. 'back' takes you out of contest, no to previous vote and it selected an unwanted candidate. lag time makes it tedious; no way to shut it up."

Subject 12

cancelled due to lack of audio

Subject 13

experienced confusion as to how to cast ballot.

Subject 14

likes being able to listen and see at same time

scrolling was too slow when using magnified text - need to have both audio and video; cannot see paper printed ballot

Subject 15

would like a third faster speed; less distortion in voice; red button got stuck in the box when pressed

Subject 16

Wants to use screen and print at same time

"The control box could be half the size -- junk it."

trouble reading braille on input box; buttons too far apart; needs 'shut up' button; too much information; constant hum is annoying; hands got tired using select button all the time

Subject 17

"design is not helpful or intuitive, cannot move through things quickly; no place to put hand box; need to hit select before actually hearing candidates."

Subject 18

Needs scrolling; should be able to touch and drag cursors

Subject 19

"keyboard wasn't clear for write-in - 'v' looked like 'u'; directions weren't printed clearly, had to press too hard to make choices"

Subject 20

change position of card insert to easier location

Subject 21

unclear what field to press to cast ballot

Subject 22

No comments

Subject 23

"a little hard to figure out during casting ballot at end; print should be slightly larger on text of propositions, but title font size was ok"

Subject 24

No comments

Subject 25

"no help button when needed, it didn't say when you were final and couldn't make any more changes; somewhat open for privacy, liked having text of propositions visible, instructions didn't always match what was on screen; would be nice to have pictures of candidates for identification -- all voters benefit"

Subject 26

"make English easier, especially complicated propositions; anti-glare screen"

Subject 27

"hard to read screen; want photos of candidates and color party id, candidates speaking voice with sign language interpreter or capability to sign back during voting"

Subject 28

"at end, instructions were unclear; should say 'processing' so user knows something is happening; should alert voter to remove card after ballot is cast"

Subject 29

needs more privacy

Subject 30

instructions should include what to do with card after voting

Subject 31

"too much waiting through listing of each contest over and over - couldn't just directly go to end of contest you wanted to change, help button didn't offer any info on how to do that"

Subject 32

forward and back buttons are confusing -- they took me out of the race I was in

Subject 33

volume fluctuates too much; needs braille instruction

Subject 34

instructions need to be more precise and audio needs to follow ballot and be more intuitive

Subject 35

No comments

Subject 36

yes and no key for propositions; too much audio

Subject 37

finish or end button

Subject 38

The case itself is an obstacle for inserting card

Subject 39

"different style keypad, not clear in ballot what buttons to use unless you use help; too many instructions in the beginning and nothing during to guide you"

Subject 40

audio needs to be consistent and directions need to be clearer

Subject 41

"shouldn't have to adjust screen tilt manually, should be automatic"

Subject 42

access card problem

Subject 43

"needs to be a little lower, write-in keys need to be farther apart [on screen]"

Subject 44

"card reader hard to access; need better instructions on touching to deselect, wouldn't go to last screen easily"

Subject 45

needs audio and visual at same time

F. Subject Recruiting and Screening

Human Subject Research (HSR) Approval

The Principal Investigator and the Access Review Team Leader for this project were required to take the NIH HSR course and become certified to lead a project involving human subject research. The test protocol for this access review testing was submitted to and approved by the UC Institutional Review Board.

As required by the IRB, all subjects for the accessibility testing were required to sign the IRB consent form, the Experimental Subject's Bill Of Rights form, as well as a subject's agreement/consent form. Some subjects read these forms for themselves in print or braille and others had one of our team members read the forms to them before signing.

Note: Throughout this Report we refer to the subjects as "users", or "voters".

Recruitment

In order to perform effective and valuable user testing of the voting systems, we needed to recruit representative individuals with a range of different functional limitations that may affect their ability to vote: impairments in vision, hearing, mobility, dexterity, and cognition. One goal was to oversample among people who are blind or have low vision, as vision loss is a particularly frequent source of barriers in using all forms of voting technologies. However, we did not intend to ignore the other disability categories, and planned our recruitment correspondingly.

We recruited all of our users through grassroots disability organizations and senior groups in the Sacramento area, using telephone, email, and personal visits. In all cases we encouraged those organizations to publicize the testing widely, and to focus on typical members rather than high-profile advocates. This latter point is important in recruiting voters who are more representative of the user populations.

We succeeded in recruiting 43 individuals with different disabilities. We added 2 users without disabilities to assist in identifying pure usability issues, as distinct from accessibility.

Demographics

Below is a description of the disabilities of the users. Note that the numbers add up to more than 43 because 13 individuals had more than one impairment.

Blind 19

Low vision 5

Deaf 5

Hard of hearing 8

Mobility 11

Dexterity 11

Cognition 2

22 of the 43 indicated that one or more of their disabilities has affected their ability to vote. Note that 15 of the other 21 voted absentee most recently instead of in person.

The gender distribution was 19 male, 26 female.

Below is the distribution by age:

under 40 11

40-60 21

over 60 13

Voting Experience and Attitudes

Only two users had never voted before. They were asked what voting method they had last used; 15 had voted absentee, 13 had used an electronic voting system, and most of the rest had used a punch card method.

We asked the users to rate their recent voting method for confidence of accuracy, ease of use, and privacy. Electronic voters were less confident that their vote had been accurately recorded. Absentee voters rated that method slightly lower for privacy: blind and low vision voters may have required the help of another individual. There was no significant difference between those who had voted absentee and those who had used an electronic system regarding ease of use.

I. Ballot Design Summary

Ideal usability and accessibility testing of voting systems should be performed with ballots specially designed for testing a full set of different race types. William Killam has helped to develop a standardized test ballot for NIST. This is a neutral ballot without political bias loading. It's candidate names are not recognizable politicians, and the political parties are made up parties with only colors for names.

To support testing of voting on very long races, the NIST test ballot even has one extremely long contest race that should overflow on to multiple screens or pages.

We would have preferred to have been able to use the NIST test ballots for our accessibility testing, however, several practical limitations prevented use of the NIST test ballot. Sequoia only had the NIST ballot defined for their Edge II+ systems, not the Edge I and Edge II we were testing in this TTB review. Designing new ballots is a nontrivial task, involving language translators, visual ballot layout, and audio studio work to record the sound files for the audio ballot.

We wanted to be able to test all three of our voting systems with very similar ballots, containing all types of races and at least three languages (English, Spanish, and Chinese). Chinese was requested in order to test the systems' ability to handle the non-Roman character graphics font handling capabilities of the systems. Supporting non-Roman fonts has been problematic for the video hardware and software of some voting systems.

Given the time constraints of this TTB accessibility review, we found our best solution was to use the California test ballots already done up for the 2004 general elections. These ballot designs did not contain any multiple member race samples, had no extremely long races, and were not perfectly identical for the three voting systems, but the closest we could come to good matching sample test ballots.

The ballots differed in how well they supported alternative languages, as well as in the number of propositions they included.

As it turned out, for the types of testing we decided to employ in our review, these ballots served us well.

The following is a summary of the 3 vendor's test ballots we used in our 2007 TTB accessibility review.

All Races and Choices for the Accessibility Testing Ballots

Note: The ballots of the 3 tested systems were the same for the first 7 contests, but they varied in how many of the propositions they included.

All 3 systems had the same first 7 contests.

• US PRESIDENT
• US SENATE
• US Representative District 49
• STATE SENATE District 35
• ASSEMBLY District 66
• JUDGE: Carlos Moreno
• JUDGE: Marvin Baxter

Diebold TSX used the first 2 propositions, Sequoia used the first 7 propositions, and Hart eSlate had all 16 possible propositions.

Additionally, Sequoia Edge had extended information about each of its propositions.

List of All Possible Contests and Choices:

• US PRESIDENT
  • Leonard Peltier / Janice Jordan (Peace and Freedom)
  • Michael Anthony Peroutka / Chuck Baldwin(American Independent)
  • Michael Badnarik / Richard Campagna (Libertarian)
  • George W Bush / Dick Cheney (Republican)
  • David Cobb / Pat LaMarche (Green)
  • John F. Kerry / John Edwards (Democratic)
• US SENATE
  • Don J. Grundmann (American Independent)
  • Bill Jones (Republican)
  • Barbara Boxer (Democratic)
  • Natural Law Cand (Natural Law)
  • Marsha Feinland (Peace and Freedom)
  • James P. "Jim" Gray (Libertarian)
• US Representative District 49
  • Lars Grossmith (Libertarian)
  • Darrell Issa (Republican)
  • Michael P Byron (Democratic)
• US Representative District 50
  • Francine P. Busby (Democratic)
  • Randy 'Duke' Cunningham (Republican)
  • Brandon C. Osborne (Libertarian)
  • Diane Templin (American Independent)
  • Gary M. Waayers (Green)
• US Representative District 51
  • Michael S. Metti (Libertarian)
  • Daniel C. Ramirez (Democratic)
  • Michael Giorgino (Republican)
• STATE SENATE District 35
  • John Campbell (Republican)
  • Timothy Johnson (Libertarian)
  • Rita B. Siebert (Democratic)
• STATE SENATE District 37
  • James F. Battin (Republican)
  • Pat Johansen (Democratic)
• STATE SENATE District 39
  • Christine Kehoe (Democratic)
  • John Murphy (Libertarian)
  • Lawrence W. Stirling (Republican)
• ASSEMBLY District 66
  • David G. Brostrom (Democratic)
  • Ray Haynes (Republican)
  • Jack N. Lee (Libertarian)
• ASSEMBLY District 74
  • Paul King (Libertarian)
  • Karen R. Underwood (Democratic)
  • Mark Wyland (Republican)
• ASSEMBLY District 75
  • Karen Heumann (Democratic)
  • George A Plescia (Republican)
  • Richard J Senecal (Libertarian)
• ASSEMBLY District 76
  • Patricia Rae Hunter (Republican)
  • Jennifer Osborne (Libertarian)
  • Lori Saldana (Democratic)
• ASSEMBLY District 77
  • Virgil (Randy) Hall II (Libertarian)
  • Chris Larkin (Democratic)
  • Jay La Suer (Republican)
• JUDGE: Carlos Moreno
  • YES
  • NO
• JUDGE: Marvin Baxter
  • YES
  • NO
• PROP 1A: Protection of Local Government Revenues
  • YES
  • NO
• PROP 59: Public Records, Open Meetings
  • YES
  • NO
• PROP 60: Election Rights of Parties
  • YES
  • NO
• PROP 60A: Surplus Property
  • YES
  • NO
• PROP 61: Children's Hospital Projects
  • YES
  • NO
• PROP 62: Elections. Primaries
  • YES
  • NO
• PROP 63: Mental Health Services
  • YES
  • NO
• PROP 64: Limits on Private Enforcement
  • YES
  • NO
• PROP 65: Local Govt Funds, Revenues
  • YES
  • NO
• PROP 66: Limitations on 3 Strikes
  • YES
  • NO
• PROP 67: Emergency Medical Services Funding
  • YES
  • NO
• PROP 68: Non-Tribal Commercial Gambling
  • YES
  • NO
• PROP 69: DNA Samples. Collection
  • YES
  • NO
• PROP 70: Tribal Gambling Compacts
  • YES
  • NO
• PROP 71: Stem Cell Research. Funding
  • YES
  • NO
• PROP 72: Health Care Coverage
  • YES
  • NO

List of Contests for Each System's Ballots

• Diebold TSX Test Ballot:
  • US PRESIDENT
  • US SENATE
  • US Representative District 49
  • STATE SENATE District 35
  • ASSEMBLY District 66
  • JUDGE: Carlos Moreno
  • JUDGE: Marvin Baxter
  • PROP 1A: Protection of Local Government Revenues
  • PROP 59: Public Records, Open Meetings
• Sequoia Edge Test Ballot
  • US PRESIDENT
  • US SENATE
  • US Representative District 49
  • STATE SENATE District 35
  • ASSEMBLY District 66
  • JUDGE: Carlos Moreno
  • JUDGE: Marvin Baxter
  • PROP 1A: Protection of Local Government Revenues
  • PROP 59: Public Records, Open Meetings
  • PROP 60: Election Rights of Parties
  • PROP 60A: Surplus Property
  • PROP 61: Children's Hospital Projects
  • PROP 62: Elections. Primaries
  • PROP 63: Mental Health Services
• Hart eSlate Test Ballot:
  • US PRESIDENT
  • US SENATE
  • US Representative District 49
  • STATE SENATE District 35
  • ASSEMBLY District 66
  • JUDGE: Carlos Moreno
  • JUDGE: Marvin Baxter
  • PROP 1A: Protection of Local Government Revenues
  • PROP 59: Public Records, Open Meetings
  • PROP 60: Election Rights of Parties
  • PROP 60A: Surplus Property
  • PROP 61: Children's Hospital Projects
  • PROP 62: Elections. Primaries
  • PROP 63: Mental Health Services
  • PROP 64: Limits on Private Enforcement
  • PROP 65: Local Govt Funds, Revenues
  • PROP 66: Limitations on 3 Strikes
  • PROP 67: Emergency Medical Services Funding
  • PROP 68: Non-Tribal Commercial Gambling
  • PROP 69: DNA Samples. Collection
  • PROP 70: Tribal Gambling Compacts
  • PROP 71: Stem Cell Research. Funding
  • PROP 72: Health Care Coverage

J. Testing Script

Escort voter to booth indicated on clipboard cover sheet. Tell videographer to roll.

Show voter report cover (with ID and booth sequence numbers) to lap and screen cameras.

Introduction to their First System

Say, "This is not a real election, but we would like to try to test these voting systems as if it is a real election. You can make any choices you want. You don't have to vote every race."

"You can stop to take a break if you get too tired or frustrated. You can even completely stop the test voting on this machine, if you get too uncomfortable, and you will still be allowed to continue with testing the other systems. Keep in mind that it's the machines we are testing, not you. If the system is wearing you out, we need to learn that."

Orientation and Configuration

Say, "This voting system can magnify the text on the screen and can also speak your choices to you through headphones. How would you like to work with it?"

If the voter's input control needs are not obvious, ask, "The system can be controlled with a touch screen, with a keypad, with dual-switch buttons, or sip and puff. Which method of control would you like to use?"

Introduce voter to voting system and help them orient to system controls and output components (screen and headphones).

If they are using headphones, have them put them on. Help them to find volume and rate controls.

Say, "Please announce each of your selections, when you make them, so we can check to make sure the machine gets your choice correctly. Of course, in a real election, you wouldn't need to speak your vote out loud." Gently encourage the voter to feel free to speak their thoughts out loud as they go, but don't push this if they are too nervous about it.

Note all input/output configurations.

Starting to Vote

Note free voting start time.

Offer them the voter card in proper orientation for insertion. Tell them where to put the card in the machine and assist if necessary. If using the screen for output, help them find the video controls.

In Booth A, encourage them to select language and enter the access code by using the wheel and Enter key. Check to make sure that volume, speech rate, magnification, and contrast are alright for them.

If the voter makes selections without verbalizing their choice, remind them to speak their choice out loud, or just ask them which choice they just made.

Free Voting Stop Time

Note free voting stop time.

Review Start

If review does not start automatically, ask them to select and start review.

Note time of review start.

If possible, ask them to review to the first Senate race and modify their choice.

Note time of modify-choice start.

If they have not made a choice in that race, ask them to make a choice and then to change it.

Note time of modified-choice stop.

Review Stop

Note time of review stop.

Write-in Start

Ask them to go back to the first House of Representatives race and change to write-in "Joe Smith".

Note time of write-in start.

Write-in Stop

Note time of write-in stop.

Casting Start

Note time of casting start.

Have them review their ballot, reject it, and make a change in the first Yes/No contest.

Let them finish the casting as they wish.

Casting Stop

Note time of end of ballot casting.

Post-test questions form:

If the voter wants help with the post-test questionnaire, read them the questions and help fill it out for them. Otherwise let them fill it out.

Thank the voter for helping us to test this voting system.