Chapter 12 Measuring Throughput in a Text Entry Task..161
Chapter 13 Conclusions and Future Work..182 Bibliography....184 Appendix A Phrase Sets for Use as Presented Text in Text Entry Studies...196 Appendix B Data from the Second Experiment of Chapter 12.208

List of Tables

Table 1 - Examples of Minimum String Distance and Error Rate.. 37 Table 2 - Results of character-based analysis of MSD matrix.. 46 Table 3 - Minimum String Distance versus Character-Level Error Rates.. 48 Table 4 - A Character-Level Error Analysis of Empirical Data.. 52 Table 5 - Frequency of Letters in the Phrase Set.. 60 Table 6 - The Five Most Frequent Words in the Phrase Set.. 61 Table 7 - Readability of the Phrase Set... 62 Table 8 - Comparison of Corrected, Uncorrected, and Total Error Rates. 69 Table 9 - Comparison of Error Statistics... 73 Table 10 - Results for Disappearing Presented Text condition.. 83 Table 11 - Results for the Remaining Presented Text condition.. 83 Table 12 - Application Usage by Percent Received Keystrokes.. 107 Table 13 - Frequencies of the Fifteen Most Common Keystrokes.. 109 Table 14 - Frequency of Unrepresented Characters.. 110 Table 15 - English Language Models Used to Analyse Empirical Data from Experiment 2... 175

List of Figures

Figure 1 - Speed and accuracy as distinct performance measures.. 6 Figure 2 - Example of synchronisation in a form-based text input experiment. 20 Figure 3 - The matrix D after initialisation... 34 Figure 4 - The completed D array... 34 Figure 5 - Several examples of the matrix D... 35 Figure 6 - Completed D array... 44 Figure 7 - Substitution and Deletion Error Rates by Character. 53 Figure 8 - Confusion matrix for Character-Level Error Analysis.. 54 Figure 9 - Constituents of the input stream... 71 Figure 10 - Classifying the keystrokes in an example.. 73 Figure 11 - The modified Sharp EL-6053 showing the circuit board and protective cover underneath the keyboard... 79 Figure 12 - A screen-print of the experimental software.. 80 Figure 13 - Information transmission paradigm... 120 Figure 14 - Depiction of Shannons fundamental theorem for a discrete channel with noise.... 123 Figure 15 - The relationship between throughput and source entropy. 125 Figure 16 - Depiction of throughput as a function of source entropy and equivocation... 127 Figure 17 - Depiction of Shannons fundamental theorem for a discrete channel with noise.... 143 Figure 18 - Choice Reaction Time Apparatus from Fitts (1966).. 145 Figure 19 - Throughput versus equivocation data adapted from Fitts (1966). 146 xiv

(14).. 75
Utilised Bandwidth = Wasted Bandwidth = Total Error Rate =
C C + INF + IF + F INF + IF + F C + INF + IF + F INF + IF

(15).. 75

(16)... 76

C + INF + IF

100%. (17)... 76 INF
Not Corrected Error Rate =

100% , and, (18).. 76

Corrected Error Rate =

IF C + INF + IF

100% , (19).. 76
Corrected but Right IFc = 100% , Error Rate C + INF + IF Corrected and Wrong IFe = 100% , Error Rate C + INF + IF

H ( p ) = log 2 ( p ).

(20).. 90

(21).. 90

(22)... 114
H (' fair dice throw ') = log 2 ( ) = 2.585 bits... 114
H (' heads ') = log 2 ( 0.75 ) = 0.415 bits, H (' tails ') = log 2 ( 0.25 ) = 2 bits.
H ({ pi } ) = ( pi H ( pi ) ) = pi log 2 ( pi )

i i =1 N

.. 114

(23)... 115

H (' fair coin toss ') = pi log 2 ( pi ) = 1 bit per toss.
= 0.5 log 2 ( 0.5 ) 0.5 log 2 ( 0.5 ).. 115
H (' biased coin toss ') = pi log 2 ( pi )
= 0.25 log 2 ( 0.25 ) 0.75 log 2 ( 0.75 ).. 115 = 0.811 bits per toss.
H ( A + B ) = ( ai b j ) H ( ai b j )
= ai b j log 2 ( ai b j )
= ai b j log 2 ( ai ) + log 2 ( b j )
= ai b j log 2 ( ai ) ai b j log 2 ( bi )

i, j i, j

(24).. 116
= b j ai log 2 ( ai ) ai b j log 2 ( b j ) i j j i = ai log 2 ( ai ) b j log 2 ( b j )

= H ({ai } ) + H

({b }) = H ( A) + H ( B )
H (' two coin tosses ') = H (' fair coin toss ') + H (' biased coin toss ') = 1 bit per toss + 0.811 bits per toss. 116 = 1.811 bits per toss.
H PA ( A, B ) = ai log 2 ( PA ai ) (1 PA ) b j log 2 ( (1 PA ) b j )
= PA ai log 2 ( PA ) + log 2 ( ai )
(1 PA ) b j log 2 (1 PA ) + log 2 ( b j )
= PA ai log 2 ( PA ) PA ai log 2 ( ai ) i i (1 PA ) b j log 2 (1 PA ) (1 PA ) b j log 2 ( b j ). 117 j j = PA log 2 ( PA ) (1 PA ) log 2 (1 PA ) PA ai log 2 ( ai ) (1 PA ) b j log 2 ( b j )
= H ( PA , 1 PA ) + PA H ({ai } ) + (1 PA ) H

({b })

= H ( PA , 1 PA ) + PA H ( A ) + (1 PA ) H ( B )
HPA ( A, B) = H ( PA , 1 PA ) + PA H ( A) + (1 PA ) H ( B)
The entropy of choosing between A and B. Entropy of A weighted by probability PA Entropy of B weighted by probability (1-PA)

(25). 118

H () = H (' biased coin toss ') + 75% H (' die throw ') + 25% H (' fair coin toss ').. 118 = 0.811 + 0.75 2.585 + 0.3 bits

R = I E, (26).... 121

I E R= 0

if I E > 0 otherwise.

(27).... 121
C = Max ( R ) = Max ( I E ) ,

(28)... 122

(29).... 122
Emin = I C. (30).... 122 I = C + S.... 124
Emin = I C = (C + S ) C.... 124 = S. R = I Emin = (C + S ) S = C.

R = I E. (31)... 124

(32).... 142 (33)... 142

Card et al. (1980) report that up to one fourth of an experts time can be spent correcting errors. In Chapter 8 we will describe a study in which we have found that the backspace key is the second most common keystroke (following the space bar, but more common than the letter e) in typical desktop computer keyboard text entry.
Organisation of this Thesis
A large portion of this thesis concerns finding and then evaluating a solution to the problem of measuring error rates in text entry studies. 3 The development of the methodology described here was not instantaneous. Rather it evolved and was refined over time. The presentation of the text entry error rate methodology in this thesis mirrors chronologically the development of the methodology. A review of the literature is presented in Chapter 2. Solutions to the problem of measuring errors in text entry studies are presented in Chapters 3 to 7, tracing the series of publications through which the methodology was developed. standardised text phrases. Chapter 8 presents a study that explores a different approach to observing text entry behaviour. In the preceding chapters text entry performance is measured by observing participants performing text copy tasks under experimentally controlled conditions. The weakness inherent in this approach is that it does not generalise well to typical real-life text entry behaviour. Outside of the lab, most text entry occurs as a part of text generation instead of copying tasks, and the generation of text is a difference process to which the analytical approach described in Chapters 2 through 7 cannot be applied. The work in these early chapters does not take into account issues related the text creation process which are relevant but beyond the scope of the thesis. Chapter 8 presents a preliminary study of real-life text entry that, although not conclusive, is as interesting for the methodology used as the results garnered. These chapters are primarily focused on the evaluation of key-based text entry methods using predefined
Because the primary focus of this dissertation is the measurement of error rate, the reader may be tempted to presume falsely that our position is that the typical users goal is the production of errorfree text. This is not so. Imperfect text is excusable in many casual circumstances, and even desirable by the artistic and/or rebellious. And, due to the inherent redundancy of Human natural languages, errors do not necessarily inhibit the comprehensibility of inaccurate text. Our objective is the development of a means to determine the error rate, as is this an important factor in the evaluation of text entry technologies.

where: NCW TCh

is the number of correct words, is the total number of characters in the presented text,
is the total number of characters in words spelled incorrectly (Alsio & Goldstein 2000, p. 92), and,
is the number of extra characters. 8
Once the NCW has been calculated, the ratio of correct words is found by dividing the NCW by the total number of words in the presented text. The number of correct words per minute is found by multiplying the ratio of correct words by the uncorrected words per minute rate. Note that one problem with this definition of error rate (Equation 1) is that extreme values of EC can cause NCW to become negative. For example:
Presented Text: Transcribed Text:
the the quick brown fox jumped over ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

23 + 0 = 1.6 words. 9 5

The problems hindering the character-level error rate (that error rates greater than 100% are possible) and the word-level error rate (that negative error rates are possible) demonstrate that, despite the best efforts and intentions of researchers, error analysis is difficult.
Alsio & Goldstein (2000) do not define exactly what an extra character is. Presumably, EC represents the number of characters in words that do not appear in the presented text, but do appear in the transcribed text. It is not clear whether the spaces in between the incorrect words should be counted in the ChWW figure in the numerator. We have not included them.
2.3.5 Avoiding Error Analysis
The previous sections have illustrated difficulties in measuring error rate. It is not surprising, then, to find that some researchers try to avoid errors. For example, Goldberg & Richardson (1993, p. 84) report peak error-free writing speeds in their evaluation of Unistrokes. With this approach, the reported text entry speed makes no allowance for the time wasted making errors and required to fix them. In a study comparing a standard keyboard, a predictive keyboard, and perfect handwriting recognition, Lewis (1999b) instructed participants to produce completely accurate text. But even diligent participants will make errors. Lewis states that participants were required to produce completely accurate text, so input times include the time for both initial text production and error correction. (Lewis 1999, p. 426). Presumably experimental software or an examiner verified the transcribed text, prohibiting the participant from completing the trial until perfection had been achieved. This approach could increase the text entry time if the participant overlooks a small error and is forced at length to find it. We hesitate to single out work that has completely disregarded errors; however, one does not have to search the literature too hard to find examples.

A Brief Note Regarding Speed
Typically experimenters record the number of characters (and/or keystrokes) entered and the time elapsed while a quantity of text is entered. The primary statistic used for quantifying the speed of text entry is words per minute (WPM),
Characters . Elapsed Time 5
Note that, typically it is impossible to measure the elapsed time for the first character, because it is not known when a participant begins to try to enter it. So 26
the elapsed time is usually measured between the first and final characters. For example: the quick brown fox jumpes over the lazy dog (44 characters) ^ ^ t=0s t = 20 s The example shows a 44-character phrase entered over a span of twenty seconds. The entry speed calculated by Equation 2 is (44 1) / 20 = 2.15 characters per second (cps). It is common practice to transform the units from characters per second to words per minute. This is accomplished by multiplying the cps rate by 60 seconds per minute, and dividing by 5 characters per word (the common typists definition of a word is five characters including space), 10 so, 2.15 cps (60 / 5) = 25.8 wpm. An element of text entry that is not reflected in the above definition of speed is text correction. Users of text entry devices make errors, and often try to correct them. If text is entered with a keyboard, the user may use the backspace, delete, or cursor keys to correct their errors, and these keystrokes are not present in the final text produced. So a subsequent definition of text entry speed capturing this important information, is the keystroke speed,
This definition of entry speed can be generalised to other (non-keyboard) modes of text entry via appropriate modification of the numerator of Equation 3, for example,
10 Regardless of the fact that, for example, the average word length in the British National Corpus (www.natcorp.ox.ac.uk) is 4.59 characters.
Strokes 1 would be a logical choice for the numerator if a stylus-based text entry method such as Graffiti was being investigated.

Conclusions

demonstrate that the MSD error rate defined by Equation 4 is well-defined over a wide range of input conditions. Even difficult situations, such as extremely long or short transcribed text (compared to the presented text) are handled gracefully.
Because the MSD is bounded by zero and the length of the longer of the two character-strings.
Table 1 - Examples of Minimum String Distance and Error Rate Five examples are given. The first two were given earlier; they demonstrate that the MSD error rate yields reasonable results for these examples. The third is the inverse of the second (the presented and transcribed texts have been reversed). The fourth example demonstrates that when no errors have been made, the error rate is indeed 0%. The fifth example demonstrates that when the input text is completely incorrect, the error rate is 100%. The last example is typical of the results that occur during text entry experiments.
Presented / Transcribed Text
the quick brown the quickk brown ^ the

a b a b

Length 26 3

Error Rate 6.3%

the quick brown fox jumped ^^^^^^^^^^^^^^^^^^^^^^^ the quick brown fox jumped the
^^^^^^^^^^^^^^^^^^^^^^^ the quick brown fox jumped the quick brown fox jumped

(No error)

the quick brown fox jumped zzzzzzzz

100.0%

^^^^^^^^^^^^^^^^^^^^^^^^^^ the quick brown fox jumped thequick broen fox jjumped ^
Presented text, b Transcribed text
Methodological Implications
The MSD error rate statistic enables text entry evaluations to employ a methodology that is free from artificially-imposed constraints upon the text entry task. Participants may be directed to enter text as they normally would, correcting the errors that they notice as they go; synchronicity with the presented text. Because the MSD approach allows researchers to use methodologies wherein participants are allowed to correct their errors as they enter text, and because participants will not necessarily notice and correct all of their errors, two distinct concepts of error rate emerge: the errors that participants correct, and those that remain uncorrected. The MSD error rate (Equation 4) is appropriate for the latter. Note that the corrected errors do not appear in the transcribed text the only way to observe the corrected errors is to analyse the stream of editing actions (or, the input stream) that the participant generated during the text entry process. dependent measure. 15 Formally, the KSPC statistic is defined, To quantify the corrected errors, we propose using keystrokes per character (KSPC) as a there is no need for the use of restrictive procedures such as ignoring errors, correcting all errors, or maintaining
Note that Keystrokes per character has played a role in the literature as a descriptive statistic used to describe text entry methods. We first proposed using KSPC as a dependent variable in empirical studies in Soukoreff & MacKenzie 2001. The KSPC statistic has been used in several recent text entry studies (Sears & Zha 2003, Wobbrock Myers & Kembel 2003, Gong & Tarasewich 2005).

Chapter 5 Phrase Set for the Evaluation of Text Entry Technologies 18
In evaluations of text entry methods, participants enter phrases of text using the text entry technology of interest while performance data are collected. controlled across separate studies. We have published (via the web) a collection of 500 phrases for such evaluations. (Note that the complete phrase set collection is reproduced in Appendix A.) Utility programs are also provided that compute statistical properties of the given phrase set, or any other candidate phrase set. The merits of using a pre-defined phrase set are described along with methodological considerations, such as attaining results that are generalisable. This represents an experiment variable that heretofore has not been standardised nor

Text Entry Evaluations

Among the desirable properties of experimental research are internal validity and external validity. Internal validity is attained if the effects observed are attributable to controlled variables. External validity means the results are generalisable to other participants and situations. Simple as this seems these attributes are typically at odds with one another; too strictly attending to one tends to compromise the other. This paper pertains to one such point of tension between internal and external validity the text entered by the participants in text entry studies.
This chapter is based on: I. S. MacKenzie & R. W. Soukoreff (2003). Phrase sets for evaluating text entry techniques. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems - CHI 2003, 754-755.
Text entry research typically pits one entry method against another. Thus, entry method is the independent variable, and it is manipulated over two or more levels. Allowing participants to freely enter whatever comes to mind seems desirable, since this mimics typical usage. Such a procedure improves external validity since the results are generalisable. However, such a methodology has problems. Because the source of text is uncontrolled measurements might not be representative of text entry performance, due to the effects of spurious behaviours such as pondering. Thus, sources of variation may potentially be present in the dependent variables (viz., speed or accuracy) that are not attributable to the controlled variable. This compromises internal validity because variations in measurements are, in part, due to other effects. On balance, the preferred procedure would seem to be to present participants with randomly-selected text, taken from a predefined representative set.

The parallel between the MSD error rate and KSPC statistic pair, and the corrected and not corrected error rates, is apparent. The new statistics proposed in this paper seem to perform similarly to their older counterparts. In particular, the numerical values of the not corrected error rate, and both new and old MSD error rates, are all very similar across participants. The values in the corrected error rate columns of Table 10 and Table 11 represent errors that would have gone unaccounted for, prior to the introduction of the new error measures. Notice that there is no apparent relationship between the total error rate and conscientiousness. Correlations between total error rate and conscientiousness, for both conditions, do not exceed 0.31. This suggests that there is some factor that affects each participants error rate that is independent of their desire for correctness. Participants performed similarly with respect to wasted bandwidth in both conditions. The correlation of the wasted bandwidth figures for both conditions was 0.89.
Conclusions and Future Work
In this chapter we have presented criticisms of the MSD error rate and KSPC statistic. A framework has been developed that provides a new perspective on error analysis. Several new statistics have been proposed including total error rate, 85
corrected error rate, and not corrected error rate. The latter two are proposed as replacements for the MSD error rate and KSPC. These new error rates have the following useful properties: The total error rate reflects all errors committed by a participant (corrected and not). Total error rate cleanly separates into two constituents, corrected error rate, and not corrected error rate. The error rates are device independent. They are directly comparable and do not misstate the performance of text entry methods with inherently different KSPC characteristics. We have also presented the results of a study to determine the effect the persistence or absence of presented text has on text entry. By hiding the presented text from participants once the first keystroke of a trial begins, experimenters can expect a higher text entry speed, accompanied by a higher not corrected error rate. While we are satisfied that we were able to detect the speed-accuracy trade-off, further work is needed to determine the relationship between these error rate metrics (i.e. accuracy), and speed. A means to measure the bandwidth of the interaction between humans and machines during text entry, that encompasses both speed and accuracy, remains to be found. Such a measure would be the ideal way to compare text entry methods. We would like to see this framework extended to account for non-keyboard-based text entry methods. In typical desktop text entry, the mouse may be used to move the cursor, or select text. A means to include this rich form of text interaction into these analyses is highly desirable.

Empirical Study of a Typical Text Input Stream
We constructed software that records what users type as they go about their regular activities using computers. We used this software to collect keystroke data from four participants for approximately one month.

8.4.1.1

Materials and Method

Software

To capture raw input text data, we wrote the KeyCapture software using Visual C++, for the Windows operating system. This software takes advantage of several hooks in Windows to log user interaction at a low level. The hooks enable KeyCapture to receive keystroke data (both key presses and releases), mouse movement and mouse button clicks, before the active application receives this data. KeyCapture also monitors the window focus to record the active application during typing. The users keyboard and mouse activity are recorded to a log file, and each 105
entry is time-stamped with millisecond resolution. The KeyCapture software was designed so that when it is operating (recording the users actions) it is essentially invisible, it has no open windows and no entry in the task bar, so the user is not aware of it or disturbed by it. We anticipate that the KeyCapture software will be of interest to others, and so the software with its source code are available at
http://www.dynamicnetservices.com/~will/academic/textinput.
The reader may be surprised to find that we used desktop computers to gather our text input data, after identifying handheld computers as the focus of contemporary text input research. We found it very difficult to find participants willing to have their keystrokes logged. As well as mundane text communications, our software captured passwords, personal e-mails, and confidential letters. using their PDAs, for an exploratory study, led us to this decision. The difficulty in finding enough willing participants, with similar PDAs, that entered sufficient text

8.4.1.2

Five volunteer participants were solicited, although one coincidentally stopped using his computer just as we began our study and so we considered his data insufficient. Of the remaining four participants, 3 were male, 1 was female; ages ranged from 25 to 45 years. All participants are computer literate and use their computers on a regular basis. Three participants were frequent users of standard desktop applications, such as email, word processing, and web browsing. One participant spent almost all of his time using Visual Basic; the three others split their time between word processing and e-mail. Although this is a small number of users, we feel the data collected is sufficient for this preliminary study of text-input-based language modelling.

Chapter 12 Measuring Throughput in a Text Entry Task
In this chapter we present the results of two studies performed to gather empirical data with three purposes in mind: 1. To determine whether it is possible to estimate source entropy, equivocation and throughput from human text entry data, 2. To observe the speed-accuracy trade-off from the perspective of information theory, and, 3. To determine the shape of the Eh function, so as to confirm or refute our hypothesis concerning the linearity of Eh. The first study we shall present is essentially a repeat of Fitts (1966) choice reaction time study. Fitts had participants enter four-bit chords in response to four stimulus lights (see Figure 18). Because the stimulus light codes can be presented randomly in a statistically independent way, exact measures of source entropy, equivocation and throughput can be made. The purpose of this experiment is to address points (2) and (3) above to observe the informatic speed-accuracy trade-off, and to determine the shape of the Eh function. The second study follows the standard text-entry paradigm, and provides estimates the informatic quantities source entropy, equivocation and throughput. The purpose of this study is primarily to address all three of the above points.

12.1 12.1.1

Experiment 1 Fitts 1966 Redux Participants
Twelve unpaid volunteers participated in this study (nine females, three males). They ranged in age from 24 to 40, with an average age of 33.25 years. Eleven were right-handed; one was left-handed (as reported by the participants).

12.1.2

Java software that mimicked Fitts (1966) choice reaction time task was presented to the participants on an HP laptop (see Figure 23).
Figure 23 - A Laptop was Used for the Fitts (1966) Redux Experiment 162
A Java program presented stimuli to participants and time-stamped and recorded their responses to a log file for subsequent analysis. In writing the software particular attention was paid to ensure the accuracy of the final time-stamps. The softwares user interface was designed to be easy to interpret with high stimulusresponse compatibility (for example, the trial being displayed in Figure 24 is instructing the participant to simultaneously depress the x, v and n keys). Most of the softwares interface used a grey colour scheme; the activated stimulus indicators, however, displayed red letters on a bright yellow background, so as to be highly visible and stimulating.
Figure 24 - A screen-print of the Fitts (1966) Redux Experiment software

12.1.3

Prior to performing the experiment, each participant received an orientation session in which the task was explained verbally. Some survey data was collected regarding the participant including their handedness, age, gender, and whether they frequently played any musical instruments, or video games. During the orientation session, the purpose of the study was explained to the participants as an 163

8 Throughput, R (bits/s)

10 Equivocation, E (bits/s)
Figure 27 - Data from Experiment 1 Plotted as Throughput versus equivocation for comparison with Fitts (1966) Data (see Figure 19)
The discussion of this data is postponed until after the presentation of the data from the second experiment.

12.3 12.3.1

Experiment 2 An Informatic Analysis of Text Entry Participants
Ten unpaid volunteers participated in this study (eight females, two males). They ranged in age from 24 to 40, with an average age of 30.70 years. Nine were righthanded; one was left-handed (as reported by the participants).

12.3.2

A Java program was created supporting the usual text entry experiment paradigm. Text phrases were randomly selected from the set described in Chapter 5, and were presented to the participants as the software monitored the participants editing actions, time-stamping and recording each keystroke. These data were saved to a data file for subsequent analysis. The Java program provided the look-and-feel of a simple text editor, so the participants typing on the keyboard received visual feedback and confirmation of keystrokes. See Figure 28. After each trial the software calculated and displayed feedback statistics such as the speed and error rate, that were displayed to encourage the participant.
Figure 28 - A screen-print of the Informatic Text Entry Experiment software

12.3.3

Prior to performing the experiment, each participant received an orientation session in which the task was explained verbally, and data was collected regarding the participant such as their handedness, age, and gender. During the orientation 170
session, the purpose of the study was explained to the participants as an investigation to determine how their rate of errors changes as they speed up or slow down. Participants were seated before the computer and encouraged to make themselves comfortable. Each trial consisted of a randomly selected text phrase being presented to the participant. The participant responded by typing the text. Elapsed time was measured starting with the first keystroke, and ending with the participant depressing the Enter key. Participants were free to take breaks between trials as they wished.

88. 89.

91. 92. 93. 94.

95. 96. 97. 98. 99. 100.

Quastler H. (Ed.) (1954) Information theory in Psychology: Problems and methods. The Free Press Publishers, Glencoe, IL Rabbitt P. (1978) Detection of errors by skilled typists. Ergonomics, 21, 945958. Reza F. M. (1961) An introduction to information theory. McGraw-Hill Book Company, New York. Rheingold H. (2000) Tools for Thought: The History and Future of MindExpanding Technology. The MIT Press, Cambridge, MA. Rissanen J. (2007) Information and Complexity in Statistical Modelling. Springer, New York. Sears A. & Zha Y. (2003) Data entry for mobile devices using soft keyboards: Understanding the effects of keyboard size and user tasks. International Journal of Human-Computer Interaction, 16, 2, 163-184. Shaffer L. H. & Hardwick J. (1968) Typing performance as a function of text. Quarterly Journal of Experimental Psychology, 20, 360-369. Shaffer L. H. & Hardwick J. (1969) Errors and error detection in typing. Quarterly Journal of Experimental Psychology, 21, 209-213.

101. 102.

Shannon C. E. (1948) A mathematical theory of communication. The Bell System Technical Journal, 27, 379-423 and 623-656. Republished as: See Shannon & Weaver (1949) below. 104. Shannon C. E. (1974) A mathematical theory of communication. In Key papers in the development of Information Theory, Slepian D. (Ed.). IEEE Press, New York. Shannon C. E. (1993) A mathematical theory of communication. In Claude Elwood Shannon collected papers, Sloane N. J. A., and Wyner A. D. (Eds.). IEEE Press, New York. Shannon C. E. (1998) A mathematical theory of communication. Bell Labs.
http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html,
Note that page numbers appearing in references to Shannon 1948 refer to the 1998 Bell Labs version a.PDF file available from the web link given above. 107. 108. 109. Shannon C.E. (1951) Prediction and entropy of printed English. Bell System Technical Journal, 30, 50-64. Shannon C. E. & Weaver W. (1949) The Mathematical Theory of Communication. University of Illinois Press, Urbana, Illinois. Silfverberg M. (2007) Historical Overview of Consumer Text Entry Technologies. In Text entry systems: Mobility, accessibility, universality, MacKenzie I. S. & Tanaka-Ishii K. (Eds.). Morgan Kaufmann Publishers, San Francisco. Silfverberg M., MacKenzie I. S. & Korhonen P. (2000). Predicting text entry speed on mobile phones. Proceedings of the ACM Conference on Human Factors in Computing Systems CHI 2000, 9-16, ACM, New York. Soukoreff R. W. & MacKenzie I. S. (1995) Theoretical upper and lower bounds on typing speeds using a stylus and soft keyboard. Behaviour & Information Technology, 14(6), 370-379. Soukoreff R. W. & MacKenzie I. S. (2001). Measuring errors in text entry tasks: An application of the Levenshtein string distance statistic. Companion Proceedings of the ACM Conference on Human Factors in Computing Systems CHI 2001. 319-320, ACM, New York. 193

129. 130. 131. 132. 133.

Appendix A Phrase Sets for Use as Presented Text in Text Entry Studies
In this appendix we include 500 English phrase sets, designed to be used for presented text in text entry studies. These phrases are discussed in Chapter 5 and were originally published in: MacKenzie I. S. & Soukoreff R. W. (2003). Phrase sets for evaluating text entry techniques. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems CHI 2003, 754-755.
my watch fell in the water prevailing wind from the east never too rich and never too thin breathing is difficult I can see the rings on Saturn physics and chemistry are hard my bank account is overdrawn elections bring out the best we are having spaghetti time to go shopping a problem with the engine elephants are afraid of mice my favorite place to visit three two one zero blast off my favorite subject is psychology circumstances are unacceptable watch out for low flying objects if at first you do not succeed please provide your date of birth we run the risk of failure prayer in schools offends some 196
he is just like everyone else great disturbance in the force love means many things you must be getting old the world is a stage can I skate with sister today neither a borrower nor a lender be one heck of a question question that must be answered beware the ides of March double double toil and trouble the power of denial I agree with you do not say anything play it again Sam the force is with you you are not a jedi yet an offer you cannot refuse are you talking to me yes you are very smart all work and no play hair gel is very greasy Valium in the economy size the facts get in the way the dreamers of dreams did you have a good time space is a high priority you are a wonderful example do not squander your time do not drink too much take a coffee break popularity is desired by all the music is better than it sounds starlight and dewdrop the living is easy fish are jumping the cotton is high drove my chevy to the levee but the levee was dry I took the rover from the shop movie about a nutty professor come and see our new car coming up with killer sound bites I am going to a music lesson the opposing team is over there 197
soon we will return from the city I am wearing a tie and a jacket the quick brown fox jumped all together in one big pile wear a crown with many jewels there will be some fog tonight I am allergic to bees and peanuts he is still on our team the dow jones index has risen my preferred treat is chocolate the king sends you to the tower we are subjects and must obey mom made her a turtleneck goldilocks and the three bears we went grocery shopping the assignment is due today what you see is what you get for your information only a quarter of a century the store will close at ten head shoulders knees and toes vanilla flavored ice cream frequently asked questions round robin scheduling information super highway my favorite web browser the laser printer is jammed all good boys deserve fudge the second largest country call for more details just in time for the party have a good weekend video camera with a zoom lens what a monkey sees a monkey will do that is very unfortunate the back yard of our house this is a very good idea reading week is just about here our fax number has changed thank you for your help no exchange without a bill the early bird gets the worm buckle up for safety this is too much to handle protect your environment 198