Sunsoft Developer Products 2550 Garcia Ave MS MPK16-303 Mountain View CA 94043 anant.mithal@eng.sun.com |
Department of Computer Science University of Oregon Eugene, OR 97403-1202 douglas@cs.uoregon.edu |
This paper describes a study comparing the movement characteristics of the mouse and the velocity-control isometric joystick. These characteristics are called the microstructure of movement. The comparison found random variations in the velocity of the isometric joystick that make it hard to control. The study shows that the microstructure of movement can explain differences in performance among devices.
For instance, a number of studies have shown that the mouse is substantially faster than the isometric joystick [2, 3, 5, 13]. For example, in an earlier study, we, reported that the isometric joystick was 70% slower than the mouse [4], but none of the studies explains the reason for the performance difference.
A possible explanation for this is that all these studies examine the gross characteristics of movement, such as target width, target distance and movement time, which are the variables in the Fitts' law equation. In doing so, they ignore the path the cursor follows in its movement towards the target, and so cannot describe the manner in which movement occurs.
For example, Figure 1 illustrates two
hypothetical movements. It plots the velocity towards the target
on the Y-axis, and distance from the start on the X-axis. From
the plot, we can see the speed at which the cursor was moving
at any point towards the target. Both trials have the same target
distance and width, and take the same amount of time. From the
point of view of a Fitts' law analysis, they are identical, but
they have different movement microstructures. The first trial
is made up of three submovements, all in the direction of the
target. The second trial has two submovements, the first of which
overshoots the target, and the second doubles back.
In addition, to ignoring the structure of movement, the Fitts' law analysis is performed by averaging the movement times for a given target and width combination. What happens during a single movement of the cursor towards a target is ignored both by the type of data gathered and by the analysis technique used. As a result, current research on pointing devices has no way to explain why performance differences exist between pointing devices.
In building models that describe movement, psychomotor research examines the microstructure of movement. As the cursor moves towards the target, time and displacement data is gathered for each trial. Velocity is then derived and plotted as a function of time as shown in Figure 1. From this the pattern of submovements is studied.
This methodology can be used to study pointing devices as shown in this paper. We describe here the results obtained from comparing the movement microstructure of two devices, the mouse and the isometric joystick. Our analysis shows that there are random variations in the velocity of the cursor which are evident in the microstructure of the joystick but not the mouse. These variations would make fine control difficult with the isometric joystick. Thus differences in performance can be attributed directly to differences observed in their movement microstructures.
The mouse is the ubiquitous pointing device. Early work [2] showed that it was very fast and had few errors. Since then numerous studies have supported its superiority and only the touch tablet seems to have similar performance [10].
The isometric joystick is a device that does not move when force is applied to it, but is able to sense the applied force and translate that into a velocity of the cursor on the screen. It has attracted a great deal of research interest because its small size makes it ideal for notebook computers where space is at a premium making it popular in current PC-compatible notebook computers.
However, research dating back to the early Card study has consistently shown it to be slower and have more errors than the mouse. Epps [5] and Rutledge and Selker [13] similarly found poor performance with the joystick. Subjects in one study complained that the isometric joystick "was hard to control" [4]. Thus there is ample evidence that there is a substantial difference in performance between mice and isometric joysticks, but the literature does not explain why.
There are three major differences between the mouse and the isometric joystick that can account for differences in performance, the isometricity of the joystick, the order of control and finger vs. wrist control. Because the isometric joystick does not move when pressure is applied to it, the amount of kinesthetic feedback that the user gets is drastically reduced. [12]
The second major difference is that the joystick controls the velocity of the cursor on the screen [12]. The harder that a user pushes, the faster the cursor moves, like a car's accelerator. This is referred to as first order control, while the mouse's mapping of mouse displacement to cursor displacement is called zero order control. Jagacinski showed that for isotonic joysticks, displacement control is better for stationary targets, while velocity control is better for targets moving at constant velocity [7]. These results have not been tested for isometric joysticks, possibly because it is difficult if not impossible to construct a displacement control isometric joystick.
In addition, most isometric joysticks in use today are controlled by the tip of the index finger, while the mouse is controlled by the wrist. Langolf showed that the finger had a higher Fitts' law index of performance than the wrist, but again, he was studying isotonic movement. Any of these differences could cause the performance differences between the devices.
The characteristics of movement have been of interest to psychomotor researchers for a while, and were first used in the Langolf study [8]. The term microstructure of movement was coined by Jagacinski in his study of isotonic joysticks [7]. Figure 1 indicates what the microstructure might be. The upper figure shows a trial consisting of a rapid submovement that covers most of the distance to the target, followed by two smaller, relatively less accurate submovements. Jagacinski's data showed that a single movement towards a target is made up of a sequence of one or more submovements (rarely more than three), and the first submovement tends to be much larger and faster than the rest of the submovements.
Meyer et al. developed a model, the stochastic optimized submovement model based on the idea that a movement is made up of submovements, and optimizes the conflicting goals of rapid movement and low error rates [11]. The model predicts a close approximation to Fitts' law and also predicts that the first submovement is faster and more accurate than the rest of the submovements. This was verified for wrist movements, supporting Jagacinski's observations, and subsequent research has extended this model to movement with the mouse [15].
The present study gathered data on the microstructure of movement for mouse and isometric joystick to see if the observations about movement characteristics of the mouse were applicable to the isometric joystick, and to see whether differences in the movement microstructure could be used to explain differences in performance.
Both devices were connected to an MS-DOS compatible 90 Mhz Pentium running Windows NT 3.5. An MS-DOS computer has one connection for keyboard data. A pointing device can be connected either through a PS/2 mouse port, or through a serial port (Figure 3). The TrackPoint keyboard has two cables that come out of it, one going to the PS/2 keyboard port, and the other going to the PS/2 mouse port. In addition, there is a PS/2 mouse port on the keyboard that allows a user to attach a mouse to the keyboard for use in addition to the TrackPoint joystick. This configuration was used in the study, and is shown in Figure 3.
After the completion of a trial, the screen was blanked, and the data from the trial was written to disk and then the next trial was presented. If the subject missed the target, the computer beeped, and the subjects was made to complete the trial. The trial was marked as an error and written to the data file, and presented again to the subject at the end of the block.
The cursor was re-centered on the home square prior to each trial, and the home square always appeared in the same location. The width and location of the target ribbon were varied as indicated in Table 1. These values generated the following six amplitude to width ratios (A/W): 48, 24, 12, 6, 3, 1.5.Table 1. Values distance to target and width of target used.
| 768 | 232.5 | 16 | 4.84 | ||
| 384 | 116.3 | 32 | 9.69 | ||
| 192 | 58.1 | 64 | 19.38 | ||
| 96 | 29.1 | ||||
One area of concern was that the subjects on the average had more experience with the mouse than the isometric joystick. In order to counteract this, subjects were given a large number of trials with each device, 600 randomly presented trials per session, or 1800 trials with each device. Earlier work indicates that after approximately 1200 trials, subjects do not have significant improvement in performance from one block of 120 trials to the next [2, 4]. Subjects were given 1800 trials so that the data from the last block of 120 trials would represent practiced performance.
An analysis of the last block of 120 trials for all subjects showed that both devices followed Fitts' law, with an r2 of .998 for the mouse, .991 for the joystick, and IP of 8.13 for the mouse, and 5.75 for the joystick.
A better way of looking at these data is shown in Figure 6, with displacement on the x-axis, and velocity on the y-axis. From this graph, we can see the speed of the cursor at any point along the path to the target. This graph shows that the movement is a sequence of three submovements. The first submovement covers most of the distance to the target and reaches a high peak velocity. The next two submovements are smaller, covering smaller proportions of the distance to the target. The second submovement overshot the target and the third submovement was a correction, bringing the cursor back to the target.
The plot in Figure 6 is a prototypical trial, that supports earlier findings that the first submovement tends to be larger and faster than the subsequent submovements. It is representative of the data obtained for a number of different types of mice in the pilot study, as well as in the data from the main study.
Mouse data from the main study was similar. Figure 7 and Figure 8 represent mouse movement data from the main experiment. They trials have been selected because they are representative of mouse trials from the main study.
Figure 7 shows that the subject made a rapid primary submovement towards the target that overshot. The subject doubled back with the second submovement and overshot again. The third submovement also overshot, and the subject oscillated around the target till it was reached. This trial also has a large primary submovement, followed by smaller secondary submovements. Figure 8 deviates from the prototype in that there appear to be two large movements followed by smaller movements.
These figures show that in general, mouse movements have a large primary submovement, followed by smaller submovements, which is consistent with the data from earlier studies. This characteristic held throughout the experiment, from the early trials to the end of the study, and across all subjects. Note that each graph has been picked out of 3600 such graphs of mouse movement for each session. (Each of 6 subjects did 600 trials per session, and only one graph from each session has been shown.)
Figure 9 and Figure 10 show a joystick trial from the pilot study. The distance vs. time sub-plot in Figure 9 does not reveal anything out of the ordinary, but the velocity vs. time sub-plot shows a strange pattern of jitter in the velocity.
This jitter is more clearly evident when the velocity vs. distance plot (Figure 10) is examined. The jitter appeared in all the isometric joystick trials. This particular trial was one of the first attempted by the test subject, but persisted after 50 trials, and also occurred in trials by others.
There are a number of things that could account for this jitter. Among other things, it could be caused by an interaction between the joystick and the computer, it could be an artifact of the program gathering the data, it could be the subject, and it could be caused by the computer.
To eliminate the possibility that the error was caused by an interaction between the computer and the joystick, the TrackPoint keyboard was connected to another computer, with the same result, i.e., the jitter was still present.
Figure 3 showed the different ways in which pointing devices can be connected to MS-DOS computers. To eliminate the possibility that the jitter was due to the PS/2 connection used by the isometric joystick, tests were performed with four different pointing device configurations with the TrackPoint keyboard or a separate keyboard as appropriate. These configurations were the TrackPoint keyboard by itself, the PS/2 mouse by itself, the PS/2 mouse connected through the TrackPoint's PS/2 port, and an identical RS-232c serial mouse. In all cases, graphs for the mouse did not show jitter, with graphs similar to Figure 6, and jitter was present in the TrackPoint joystick.
To eliminate the possibility that there was a flaw in this particular joystick, the software was recompiled to run on a smaller screen (640x480, the standard size for notebook computers), and the test carried out on two IBM ThinkPads (which have built-in TrackPoints) and on an NEC Versa notebook (that has a similar isometric joystick, but is not called a TrackPoint), and the tremor was still present.
The last isometric device tested was a PortaPoint. It is a separate isometric joystick that uses the same electronics that are used in the Home Row 'J' mouse keyboard [4, 3]. The PortaPoint is mounted in a housing similar in shape to a mouse, and attaches to the computer through the PS/2 mouse port. Figure 11 is a trial from this device, showing jitter.
In all the tests conducted, none of the mouse configurations showed jitter, and all of the isometric joystick configurations showed jitter. If we go back to the base condition, the PS/2 mouse, by being connected to the TrackPoint's PS/2 connector, feeds data to the CPU through the same cable to the same software. Therefore there is something about the isometric joystick that causes the jitter.
After carefully considering all possible sources that could cause this jerkiness, the only explanation we found that fit all the available data was that the isometric joystick, which is force sensitive, was picking up physiological tremor in the finger. Tremor is a series of random variations in force that occur in the finger (and the rest of the human body), and the isometric joystick translates these changes into variations in cursor velocity. Tremor occurs during the application of force, during steady movement, and even at rest. It tends to increase with increased force [14].
The effect of tremor on the joystick's movement microstructure was a totally unexpected phenomenon. When the main study was conducted, jitter continued to be evident in joystick trials throughout the experiment.
The graphs shown in Figure 12, Figure 13 and Figure 14 are representative samples of joystick trials from the main study. Figure 12 shows that the movement towards the target was made up of a sequence of movements that had rapidly changing velocity. Figure 13 shows a period in the beginning of the movement which might represent a primary submovement with rapid movement towards the target, relatively unaffected by tremor. The rest of the movement is quite badly obscured by tremor.
Joystick trials were thus characterized by jerkiness that is visible only when the velocity of the trial is studied either over time or over movement distance. The jerkiness was evident in early trials in the pilot study and persisted throughout the duration of the main study.
The presence of the jitter in the isometric joystick trials suggests higher frequency components in joystick trials than mouse trials. This was verified through spectral analysis. Figure 15 presents the frequency data for the mouse for all subjects compared to the frequency data for the joystick all subjects. It clearly shows a that there is a greater percentage of the power for higher frequencies (between 3 and 10 Hz) for the joystick than for the mouse, which is consistent with higher frequencies in the jitter.
Consider Figure 16 which shows a block diagram of an isometric joystick. The joystick is a force sensitive device that translates input force into changes in cursor velocity. Note that because the joystick is using the same hardware connections and software drivers as the mouse, it must send data to the OS in the same form as the mouse. The mouse sends the OS data about displacement, and so the isometric joystick performs an integration of velocity over time in order to send similar displacement information.
When there is tremor present in the finger, the tremor is perceived as changes in applied force. These in turn are mapped onto changes in velocity which of course get integrated into displacements and sent to the OS. As long as we look at plots of displacement, nothing is apparent. The moment we look at plots in the velocity domain, the tremor appears.
The isometric joystick happens to be unique in the manner in which it picks up tremor and the way in which it is affected by it. Consider for a moment the finger that controls the joystick. It is applying a lateral force on the joystick, and it is unsupported. It also does not move. Therefore there is little to damp the tremor. In other types of movement, tremor is damped by two things, inertial mass and friction. Because the finger does not move, inertial mass does not play a role and so does not damp tremor. Again, because nothing moves, friction does not play a role. In fact, the only damping factor that we have been able to uncover is the friction that exists between adjacent muscle fibers in the finger [14].
Tremor also occurs during mouse movement, but it does not have the same effect for two reasons. First, the mouse operates in the displacement domain, and is not sensitive to changes in applied force in the manner that the joystick is. It responds only to changes in displacement. The mouse is therefore not directly affected by tremor. Second, when a mouse is moving across a mousepad, any tremor is damped by the friction of the mouse and the hand on the mousepad, as well as by the mass of the hand, wrist and mouse (and perhaps the forearm). Due to these reasons, tremor does not appear in plots of mouse movement. So the physical construction of the joystick makes it susceptible to tremor in a manner that devices such as the mouse are not.
The effect of tremor on other pointing devices is still an open question. Two devices that use relatively unsupported limbs are the trackball and the touch tablet. The trackball is typically controlled by a single thumb or single finger, while touch tablets are controlled either by a finger, or by a stylus. While inertial mass and friction still play a role in these devices, the lack of support for the controlling limb suggests that a similar study needs to be performed on these devices to determine how they are affected by tremor.
The joystick also accentuates tremor. This tremor is picked up by the joystick and passed on to the joystick's hardware controller. The joystick's controller employs an 'accelerated' design, i.e. it employs a non-linear translation between input force and output displacement. This transfer function accentuates the effect that changes in input have on the output. So the changes caused by tremor are exaggerated in the displacement data that is sent to the software driver. The software driver is in turn accelerated, and accentuates changes in displacement into larger changes in the position of the cursor. The net result is that the velocity of the cursor varies widely as it moves across the screen. Thus the design of the joystick makes it unique in the way that it amplifies tremor.
The effect of tremor might be reduced by introducing digital signal processing routines into the joystick's hardware or in its software drivers. These routines could filter out the tremor. Making the joystick slightly deformable might introduce enough inertia into the system to damp the tremor. Additionally, a support for the finger might also reduce tremor.
To summarize, studying the movement microstructure for the joystick revealed that tremor causes involuntary changes in cursor velocity. Taking a closer look at the characteristics of the joystick and its connections to the computer showed that the isometric joystick is unique in the way that it is affected by tremor and the way in which it accentuates the effects of tremor.
The reason for this is now easy to understand. Tremor causes involuntary changes in the velocity at which the cursor moves. This makes it difficult for users to achieve fine control over the cursor making it difficult to stop the cursor at a desired point on the screen and explains why isometric joysticks are hard to control.
A simple lesson from this study is that when a pointing device is being studied, we need to examine its effect on the variable it controls. The isometric joystick controls the velocity of the cursor (even though it integrates the velocity over time and sends the computer displacement information). The effect of tremor was visible only when velocity was plotted against time or displacement.
Taking this argument to the next stage, the implication is that we should look at graphs of the appropriate variable. For example, if we control acceleration, we should look at graphs of acceleration.
This study opened avenues of additional research, some of which has been completed and is outside the scope of this paper. A more extensive Fourier analysis has been completed, which revealed patterns of individual differences and learning. One of our originals goal was to model movement with the isometric joystick, but we were confounded by the problems that we were unable to differentiate voluntary from involuntary submovements in the case of the isometric joystick. We believe that studying muscle activations and neuron firings will help in this area. While our goal was understanding movement with the isometric joystick, our study has suggested ways in which it can be improved. These improvements have to be incorporated and tested.
The microstructure level analysis was extremely effective in providing information about what happened during joystick trials. The next step in the development of research on pointing devices should be to study movement microstructure in the manner that a Fitts' law analysis is routinely done. The manner in which this study uncovered tremor in the isometric devices strongly supports this suggestion.
Fitts' law, which is the tool most often used in understanding pointing devices, is excellent for comparing devices, but does not help explain why some devices perform better than others. The study of the microstructure of movement of a device helps explain device performance.
We need inventories of knowledge of the microstructure of movement with different devices just like we have a corpus of knowledge about gross movement and the applicability of Fitts' law.
Research on pointing devices will feed off research on the human psychomotor system, and is likely provide an impetus for new discoveries about how humans move. In this effort, Fitts' law must be augmented by microstructure level analysis as the primary research and design tool for pointing devices [9]. This work is the first step in that direction.