ACM Computing Surveys 28(4es), December 1996, http://www.acm.org/pubs/citations/journals/surveys/1996-28-4es/a138-jacob/. Copyright © 1996 by the Association for Computing Machinery, Inc. See the permissions statement below. This article derives from a position statement prepared for the Workshop on Strategic Directions in Computing Research.
Abstract: This paper gives some of the technical background for this area along with a brief survey of the range of input devices currently in use and emerging. Then it attempts to predict future trends in input. Some of those trends point toward devices and interaction styles that are different enough from those in current use that they will require research into whole new types of interfaces, new interaction techniques, and new user-interface software architectures. For example, some new styles of interaction revolve around parallel, continuous streams of input, while most current user-interface software techniques assume a single stream of discrete events or tokens.Categories and Subject Descriptors: H.5.2 [Information Interfaces and Presentation] User interfaces - input devices and strategies; I.3.6 [Computer Graphics] Methodology and Techniques - interaction techniques; H.1.2 [Models and Principles]: User/Machine Systems - human factors; B.4 [Input/Output and Data Communications]: Input/Output Devices
General Terms: Human Factors
Additional Key Words and Phrases: Human-computer interaction, input, input devices, interaction techniques
All aspects of human-computer interaction, from the high-level concerns of organizational context and system requirements to the conceptual, semantic, syntactic, and lexical levels of user-interface design, are ultimately funneled through physical input and output actions and devices. The fundamental task in computer input is to move information from the brain of the user to the computer. Progress in this discipline attempts to increase the useful bandwidth across that interface by seeking faster, more natural, and more convenient means for a user to transmit information to a computer.
This paper gives some of the technical background for this area along with a brief survey of the range of input devices currently in use and emerging. Then it attempts to predict future trends in input. Some of those trends point toward devices and interaction styles that are different enough from those in current use that they will require research into whole new types of interfaces, new interaction techniques, and new user-interface software architectures. For example, some new styles of interaction revolve around parallel, continuous streams of input, while most current user-interface software techniques assume a single stream of discrete events or tokens.
A designer looks at the interaction tasks necessary for a particular application [Foley 1990]. Interaction tasks are low-level primitive inputs required from the user, such as entering a text string or choosing a command. For each such task, the designer chooses an appropriate interaction device and interaction technique. An interaction technique is a way of using a physical device to perform an interaction task. There may be several different ways of using the same device to perform the same task. For example, one could use a mouse to select a command by using a pop-up menu, a fixed menu (or palette), multiple clicking, circling the desired command, or even writing the name of the command with the mouse.
User performance with many types of manual input depends on the speed with which the user can move his or her hand to a target. Fitts' Law provides a way to predict this and is a key foundation of input design [Card 1983]. It predicts the time required to move based on the distance to be moved and the size of the destination target. The time is proportional to the logarithm of the distance divided by the target width. This leads to a tradeoff between distance and target width: it takes as much additional time to reach a target that is twice as far away as it does to reach one that is half as large. Different manual input devices give rise to different proportionality constants in the equation. Some thus give better overall performance, and others better performance either for long moves or short moves, but the one-for-one tradeoff between distance and target size remains.
Another way of characterizing many input devices is by their control-display ratio, the ratio between the movement of the input device and the corresponding movement of the object it controls. For example, if a mouse (the control) must be moved one inch on the desk in order to move a cursor two inches on the screen (the display), the device has a 1:2 control-display ratio. A high control-display ratio affords greater precision, while a low one allows more rapid operation and takes less desk space. An accelerator can be provided, so that the ratio increases dynamically when the user moves faster. This allows more efficient use of desk space, but can disturb the otherwise straightforward physical relationship between mouse movement and cursor movement. Of course, with a direct input device, such as a touchscreen, the C-D ratio is always unity.
To survey the principal types of interaction devices in use today and emerging, we begin around the "output" mechanisms of the user's body rather than the device technology, since the former are more likely to remain constant over time.
Keyboards attached to workstations, terminals, or portable computers are one of the principal input devices in use today. Most use a typewriter-like "QWERTY" keyboard layout, typically augmented with additional keys for moving the cursor, entering numbers, and special functions. There are other layouts and also chord keyboards, where a single hand presses combinations of up to five keys to represent different characters.
A much wider variety of devices is in use for continuous input from the hands. A number of studies and taxonomies attempt to organize this range of possibilities [Foley 1990], [Mackinlay 1990]; most devices used for manual pointing or locating can be categorized in these ways:
Foot position, head position (with a 3-D tracker), and even the direction of gaze of the eyes [Bolt 1981], [Jacob 1991] can also be used as computer inputs.
Another type of input is the user's speech. Carrying on a full conversation with a computer as one might do with another person is well beyond the state of the art today--and, even if possible, may be a naive goal. Nevertheless, speech can be used as input with unrecognized speech [Schmandt 1993], discrete word recognition, or continuous speech recognition. Even if the computer could recognize all the user's words in continuous speech, the problem of understanding natural language is a significant and unsolved one. It can be avoided by using an artificial language of special commands or even a fairly restricted subset of natural language. But, given the current state of the art, the closer the user moves toward full unrestricted natural language, the more difficulties will be encountered.
Virtual reality systems rely combinations of the 3-D devices discussed above, typically a magnetic tracker to sense head position and orientation to determine the position of the virtual camera for scene rendering plus a glove or other 3-D hand input device to allow the user to reach into the displayed environment and interact with it.
Given the current state of the art, computer input and output are quite asymmetric. The amount of information or bandwidth that is communicated from computer to user is typically far greater than the bandwidth from user to computer. Graphics, animations, audio, and other media can output large amounts of information rapidly, but we do not yet have means of inputting comparably large amounts of information from the user.
One way to predict the future of input is to look at some of the characteristics of emerging new computers. The desktop workstation seems to be an artifact of past technology in display devices and in electronic hardware. In the future, it is likely that computers smaller and larger than today's workstation will appear, and the workstation-size machine may disappear. This will be a force driving the design and adoption of future input mechanisms. Small computers are already appearing--laptops, palmtops, and wearables--and straining the limits of keyboard usage. These are often intended to blend more closely into the user's other daily activities. They will certainly require smaller input devices and may also require more unobtrusive input mechanisms, if they are to be used in settings where the user is simultaneously engaged in other tasks, such as talking to people or repairing a piece of machinery. At the same time, computers will be getting larger. As display technology improves and as more of one's tasks become computer-based, a office-sized computer can be envisioned with a display that is as large as a desk or wall (and has resolution approaching that of a paper desk). Such a computer leaves considerable freedom for possible input means. A large, fixed installation could accommodate a special-purpose console or "cockpit" for high-performance interaction. In collaborative work, the large display may be fixed, but users move about the room, interacting with each other and with small, mobile input devices.
Another trend, seen in the emergence of virtual reality is that computer input and output, is becoming more like interacting with the real world. For input, this means attempting to make the user's input actions as close as possible to the user's thoughts that motivated those actions, that is, to reduce the "gulf of execution" [Hutchins 1986], the gap between the user's intentions and the actions necessary to input them into the computer. Doing so exploits skills humans have acquired through evolution and experience. Direct manipulation interfaces [Shneiderman 1983] have enjoyed great success, particularly with new users, largely because they draw on analogies to existing human skills (pointing, grabbing, moving objects in space) rather than on trained behaviors. Virtual reality interfaces, too, gain their strength by exploiting the user's pre-existing abilities and expectations. Instead of inputting strings of characters, users interact with a virtual reality in more natural and expressive ways--moving their heads, hands, or feet. Future input mechanisms may continue this trend toward naturalness and expressivity by allowing users to perform "natural" gestures or operations and transducing them for computer input. More parts or characteristics of the user's body can be measured for this purpose and then interpreted as input.
Yet another way to predict the future of input devices is to consider the progression that begins with laboratory devices used to measure some physical attribute of a person. As they become more robust, they may be used as practical medical instruments. As they become convenient, non-invasive, and inexpensive, they may find use as future computer input devices. The eye tracker is such an example. Measurements such as blood pressure, heart rate, respiration rate, eye pupil diameter, and galvanic skin response, and even EEG (electroencephalogram) signals are possible candidates for inputs in the future. Perhaps the final frontier in user input and output devices will someday be to measure and stimulate neurons directly, rather than via the body's transducers.