Stanford University Study

Professor Clifford Nass (Stanford University, Palo Alto)
Ing-Marie Jonsson, Kwan Lee Courtney Bennett
Published in the Proceedings of CHI 2002

ABSTRACT
It is now possible to present interfaces designed for PCs or handhelds to be displayed on either device. In this experimental within-participants study (N = 39), participants used an interface designed for either a PC or a handheld on a PC, a handheld with keyboard, or a handheld with a virtual keyboard/pen input. The context was an interactive natural language query system used for financial and entertainment inquiries. When interface matched device, the application was perceived as easier to use. Applications on the PC were perceived as easier to use, less impersonal, and made users feel more in control. The handheld interface was perceived as better on all dimensions. Implications for cross-platform interface design are discussed.

Keywords
Handheld Device, Platform Differences, Interface Paradigms, Keyboard, Natural Language, Screen Size, Perceived Competence.

INTRODUCTION
The traditional world of computing has usually involved presenting a user interface on a 12 to 18 inch (diagonal) stationary display monitor. There are a small but growing number of larger desktop displays in use [7]. User input has been dominated by the combination of mouse and keyboard.

A newer generation of mobile, handheld computing devices with small screens (less than 5 inches diagonal) have appeared on the scene. Input was based on a reduced size keyboard, a full-sized detachable keyboard, a virtual keyboard and touch-screen activated with a stylus, or a text-like input system using a stylus and touch-screen.

Because of the differences in functionality between desktop and handheld devices, there was a fundamental separation in the applications and interfaces that were used on the two platforms. In general, handheld devices were limited to a very small set of tasks. Even when the same functionality appeared on both types of devices, e.g., scheduling appointments, the applications and interfaces were very different.

Recently, we have entered a third-generation of computing in which the distinctions between devices have blurred. Handheld devices now have access to the Internet via wireless communication, and they have fast processors and sufficient memory to do many of the same applications that could only be performed on desktops. Conversely, people used to handhelds want the same interface and applications on both their desktop and handheld units. The desire for interface consistency has obvious implications on learning (reinforcement through repetition and redundancy) as well as the amount of learning required to operate the two system. Thus, the distinctions between desktops and handhelds, at least as far as software, have become blurred: Individuals now demand the ability to have all functionality on all platforms.

Under this vision, there is a world of anytime, anywhere, any-device access. This vision depends upon factors such as infrastructure, wireless technology, legislation, devices, and middleware technology [8, 9]. The most important factors here, however, will be the consumer's trust and perceived usefulness of the devices and applications.

This trend presents a dilemma to interface designers. On the one hand, it would seem optimal to rewrite each application for each device, capitalizing on the various opportunities and constraints of screen size and input modality. This approach, however, is incredibly time-consuming and expensive, leading to the likelihood that many applications will appear on a second platform only after an extended period of time, if at all (similar to the creation of dual versions of software for PCs and the Mac).

A seemingly more economical approach is to design the interface only once, and to allow the same interface to appear on multiple platforms [6]. Theories for decoupling applications from interaction models [3] as well as commercial products offering access from different devices are indications to this trend [1]. This option is feasible given that handhelds now have sufficient memory and processor speeds, the possibility of entering text, and the ability to scroll when the visual output will not fit within the smaller screen. Conversely, desktops can display the visual output of an application written for a handheld, and input normally done with a handheld pointer can be input by a mouse. Designers have been reluctant to pursue this course of action, however, because it has seemed to lead to interfaces that would be unacceptably sub-optimal for a given platform.

This study explores the effects on user's attitudes when a natural interaction user interface is matched or mismatched to the device; when a device is used for different types of applications; and when different methods for data input is used. Specifically, we focused on three parameters that can influence perceptions of the interface when used on a desktop and on a handheld device: input method, presentation format and interaction style.

KEY DIMENSIONS FOR THE RESEARCH
Input Methods
There are a number of different input methods available for both PCs and handheld devices. For typed data these range from full-sized keyboards to pen entry via a software keyboard displayed on the screen. There are also methods such as handwriting recognition and graffiti. However, these input methods require significant training and were beyond the scope of the present research. We also excluded spoken input since these systems also tend to require initial training, and performance is hampered by the immature status of the technology [5].

Screen Size and Presentation Format
Common practice and intuition would have it that the user would prefer a graphical interface matched to screen size. Using a large graphic interface on a small screen would deny the user an overview of the interface at a glance. It would force the user to scroll, vertically and horizontally, to identify the functionality of the interface and to receive outputs from the interface. Conversely, using a small interface on a large screen would seem to be a failure to take advantage of the technology.

Interaction Style
An interaction model is based on the combination of presentation format, screen-size, application and communication technology. A design with short interactions, short messages and limited results is suitable when the screen real estate is small and the bandwidth is restricted. There is an emphasis on filtering on the application side, resulting in more interaction and smaller chunks of data presented to the user. For large screens and high bandwidth, the design of filtering can be more relaxed. This moves the responsibility for filtering to the user; often resulting in fewer interactions and a large volume of results to browse through.

Goals of the Experiment
Despite the importance of the link between input method, screen size/presentation format and interaction style, little research has been done to understand the trade-offs in importing an interface designed for the PC to a handheld device, or importing an interface designed for a handheld to a PC (for an exception, see [2]). In this paper, we discuss an experiment in which users are presented with an interface designed for either a PC or a handheld. Users interact with the interface as presented on a traditional desktop monitor with a full-size keyboard, a handheld (iPAQ) with a full-size keyboard and a handheld (iPAQ) using a stylus to tap on an on-screen virtual keyboard. This design enables us to distinguish between the effects of screen size and keyboard.

We examine users' perceptions of the interfaces and their feelings about the interaction to answer three questions:

1. Does the device (PC vs. handheld) influence perception of the application and the interface?
2. Does the interface (PC vs. handheld) influence perception of the application and the device?
3. Does the interaction between device and interface influence perception of the application?

METHOD
Experimental Design
The basic task in the experiment involved the use of a natural language query interface. To increase reliability and external validity, we had users perform queries in two different domains: financial and entertainment (examples of the task are provided in Appendix A). The motivation for using these two applications is that they represent formal versus informal queries, and that the timeliness as well as the positive/negative consequences of getting the desired information differs. Task domain was treated as a within-participants repeated factor. If an initial query by the user did not provide the requisite information, participants continued to query the system until the correct information was found. All participants were able to find answers to all of the questions.

The experiment was a 2x3x2 mixed design. The between-participants factor was whether the user worked with an interface that was designed for a PC (assumption of large screen real estate) or a handheld (assumption of small screen real estate). The key within-subjects factor was the device type: a PC, a handheld with an external keyboard, or a handheld with pen interface. We included two different methods for working with the handheld to independently control for screen size and input method. As noted above, the second within-subject factor was task domain.

Participants
A total of 39 adults were recruited to participate in the study. All participants had prior experience with computers and the vast majority did not have experience with handheld devices.

Each of the participants was randomly assigned to one of the two user interfaces: PC or handheld.

Experimental Apparatus
Each participant in the experiment used two applications: 1) a business application that is part of Dejima Direct Enterprise, that allowed users to request data about the stock market; and 2) an entertainment application that is part of Dejima Direct Mobile, that allowed users to find movies and movie theaters. Both applications were running on external servers, and they were accessed through an interface that allowed natural language queries and dialogues. The applications were chosen because of their natural language interfaces, with the intent to reduce the interaction between interaction style and type of application. Informal studies show that the graphic interface tends to disappear in favor of a focus on the results when natural language is used.

Three devices were used in the experiment: 1) a laptop with a 14" screen, resolution set to 1024 x 768 with a keyboard and two-button touch-pad; 2) an iPAQ with a 3.7" screen, resolution of 240 x 320, a touch-screen pen interface; and an attached fold-away keyboard; and 3) an iPAQ with a 3.7" screen, resolution of 240 x 320,a touch-screen pen interface, and an on-screen keyboard used with the pen. All participants used all three devices; the order in which the devices were used was randomized.

The PC interface was designed for a desktop/laptop environment with the following parameters: 13" screen and up, resolution of 800 x 600 and up, and keyboard and mouse as interaction tools described above (see Figure 1).

Figure 1. The PC interface to a natural interaction application

The handheld interface (see Figure 2) was designed for a small handheld device. Specifically, it was designed for a device that had a 3.7" screen, resolution of 240 x 320, and a keyboard/pen touch screen as interaction tools.

This interface was an adaptation of the PC interface, with as few changes as possible. The aim was to see if it is possible to shrink the interface - without major redesign - when the properties of the smaller device allowed for this.

Figure 2. The handheld interface

The size and resolution of the iPAQs screen, paired with its ability to handle high bandwidth communication, resulted in an interface with minor alterations. The interface together with some of the resulting graphics and tables where shrunk to fit the screen,; although scrolling was still required; no changes were made to the interaction model. The result was that the interface was designed to fit the screen; it minimized the need for scrolling left-to-right, both for input and for viewing results. When the handheld interface was presented on the PC, it simply took up a small fraction of the screen.

Procedure
The participants were told that the goal of the study was to test two new applications. The study was conducted in one room where all three devices were set-up. When participants had completed the business and entertainment questions on one device, they moved to the next device (order of device presentation was randomized across participants).

The participants were instructed that they were to finish a set of tasks for two different applications, and that they would do this on three different devices. They were told to finish the task but not to write down answers, as many of the tasks involved multiple queries, answers, and graphics. They were also asked not to change the size of applications windows (a potential problem in the PC device case).

After completing the first set of tasks with a given device, the participant filled out a paper-and-pencil questionnaire that asked for an evaluation of the application and the participant's feelings about the interface. This process was repeated for the second and third device.

The time to complete the study session was approximately 45 minutes.

Measures
All dependent measures were based on items from the paper-and-pencil questionnaire administered after the participants addressed the ten questions on each device. The first set of questions asked: "For each of the following adjectives, please indicate how well each of the adjectives describes the application?" The second set of questions asked: "For each of the following adjectives, please indicate how well each of the adjectives describes how you felt?" Each adjective was presented next to a ten-point Likert scale ranging from "Describes Very Poorly" (=1) to "Describes Very Well" (=10). All indices were created based on factor analysis and prior theory; all were highly reliable.

Competence of the application was an index of eleven adjectives: accurate, competent, complete, efficient, expert, intelligent, reliable, skillful, thorough, trustworthy, and valuable. The index was very reliable (Cronbach's alpha ranged from .92 to .94).

Ease of use of the application was an index of five items: easy to use, easy to understand, confusing (reverse-coded), frustrating (reverse-coded), and tricky (reverse-coded). The index was very reliable (alpha ranged from .78 to .91).

Impersonalness of the application was an index of two items: cold and impersonal. The index was reliable (alpha ranged from .57 to .71).

Feel in control was an index of seven items: at ease, calm, comfortable, relaxed, safe, secure, and tense (reverse-coded). The index was very reliable (alpha ranged from .89 to .94).

RESULTS
All analyses were based on a full-factorial 3 (Device Type) X 2 (Interface Type) X 2 (Task domain) mixed ANOVA model. However, because task domain only had an impact as a main effect (there was a preference for the entertainment task over the finance task), we only report the Device Type by Interaction Type results here.

Users perceived the application to be more competent when presented on the PC as compared to either handheld, F(2, 74) = 15.23,p < .001, h² = .29 (see Figure 3). This result is striking given that the performance of the application was identical in both applications. The handheld interface was perceived to be more competent than the PC interface, F(1, 37) = 4.59, p < .04, h² = .11, even though the performance was again identical. The interaction was not significant.

Figure 3. The effect of interface and device on perceived competence (10 point Likert scale)

The handheld interface was easier to use than the PC interface, F(1, 37) = 11.24, p < .01, h² = .23 (see Figure 4). There was a significant interaction such that the handheld interface was easier to use on both of the handheld devices, while the PC interface was easier to use on the PC, F(2, 37) = 5.21, p < .03, h² = .12. There was no main effect for device.

Figure 4. The effect of interface and device on ease of use (10 point Likert scale)

There was a significant device effect with respect to impersonalness, F(2, 74) = 15.23, p < .001, h² = .29 (see Figure 5). Post-hoc analyses reveal that the PC is perceived as more impersonal than the handheld without keyboard, with the handheld with keyboard intermediate. The PC interface was more impersonal than the handheld interface, F(1, 37) = 4.63, p < .04, h² = .11. The interaction was not significant.

Figure 5. The effect of interface and device on impersonalness of the application (10 point Likert scale)

There was a significant device effect with respect to feeling in control, F(2, 74) = 42.86, p < .001, h² = .71 (see Figure 6). Post-hoc analyses reveal that the PC gave the user a greater sense of control than the handheld with keyboard, which in turn gave the user a greater sense of control than the handheld with pen input. The handheld interface gave users a greater sense of control than the PC interface, F(1, 37) = 47.11, p < .01, h² = .17. The interaction was not significant.

Figure 6. The effect of interface and device on feeling in control (10 point Likert scale)

DISCUSSION
The present results suggest both optimism and pessimism with respect to simplifying design by placing the same interface on multiple devices. On the positive side, there was no device/interface interaction with respect to perceived competence of the interface, impersonalness of the application, nor the user's feeling of control. On the down side, there was a significant, though not unexpected, interaction between ease of use and device, such that the handheld interface made interactions with the handheld iPAQs easier, while the PC interface made the PC seem easier. While ease of use is certainly a critical consideration, favoring device-specific design when interfaces are individually matched to a device, other factors such as timeliness and cost suggests that it may be feasible, though not optimal, to create a single design.

One of the most striking results of the study is that the application appeared more competent on the PC than on either of the handhelds. The functional intelligence of the application was identical for all three devices, however, the results indicate that the perception was quite different. This result suggests that the particular device on which the application appears can actually influence the perceived intelligence of the application. (for similar results, see [2, 4]).

With the exception of changing the user's sense of control, there were no differences between the handheld with external keyboard as compared to a virtual keyboard. There are two possible explanations for this. First, screen size has been shown to be a critical aspect of interfaces, perhaps overcoming the differences in input [7]. Second, technologies are often associated with particular characteristics defined by the culture; if both handhelds were assigned the same social category, they could elicit the same response [4]. That is, a small screen is psychologically a handheld even if it is equipped with a conventional keyboard.

In general, the handheld interface was perceived much more positively than the PC interface regardless of presentation device. One possible explanation for this is that users are typically not used to formulating queries to applications using natural language. The natural language approach allowed for a simple user interface with few buttons and switches at the expense of the initial complexity in formulating and entering the input. The simplicity of the handheld interface may have facilitated thinking by removing extraneous images. It is very possible that in tasks that have an entertainment rather than cognitive orientation, the richness of the traditional PC interface would have been perceived as a significant positive.

There are many limitations to this study. First, the task involved heavy use of typing words, rather than the use of icons. It is possible that with a more graphical input model, in which the user would have to scroll across the screen even to enter inputs, the results of the study could very well change. With that said, acquisition of data via queries is likely one of the most frequently performed tasks across platforms, so the results here should certainly have relevance to a wide variety of situations.

Another limitation is that the user community from which the present study was drawn did not, for the most part, have experience with handhelds, while they all had experience with PCs. Familiarity likely explains the feelings of being in control, greater ease of use, and greater feelings of personalness in the PC case. However, the link between familiarity with the device and perceived competence of the application seems much more tenuous.

A third limitation of this study is that it was restricted to keyboard-style input. Future research should explore the use of gesture based pen input, especially in the case of the handheld. Because of the growth of speech interfaces to the Web on both handheld devices (most notably cell phones) and PCs, exploring speech input seems important as well.

A final limitation involves interaction style. The present study involved few interactions and filtering by the user. If a model with more interactions and filtering at the application end is used, it is possible that the limited use of screen real estate by the handheld interface, when presented on a PC, would be perceived negatively. Thus, it is an open question how the more communication-oriented interface model used on handheld devices would differ compared to the question-browsing model commonly used in applications on PCs.

There is little question that there will be an ever-increasing variety of intelligent and connected devices which will allow for interaction and information acquisition [10]. This variety will include properties such as, screen size and shape (including whether there is a screen), input modality(s), output modality(s), and portability.. If each application must be recreated for each device, there will be an increasing balkanization of functionality, eventually ending in chaos as software designers make choices about which technologies they will support, and when. The present study gives cause for at least some optimism, suggesting that while there are definitely problems in cross-platform implementations, particularly with respect to ease-of-use, it may be possible to meet the dream of "write one, use on everything."

APPENDIX A
Example of the Finance Task
You were recently hired as a research analyst at a prestigious investment banking firm. Your boss has asked you to retrieve information about a series of companies that the first is considering as a potential investment. The information that your boss needs is:

• A chart that compares the performance of IBM and Intel over a three-month period
• A complete report or listing for Compaq
• A list of all companies with a price/earnings ratio over 20
• A list of all companies with a price greater than 100

APPENDIX B
Example of the Entertainment Task
You and a group of friends want to see a movie on Saturday. You know that the theaters get crowded on the weekends, and want to plan ahead. The information that you want is:

• Whether there are any thrillers showing in San Jose
• Whether there are any comedies showing in Mountain View at 8 PM
• Whether there are any movies starring Brad Pitt that are showing in Mountain View
• What movies start before noon at the Shoreline Boulevard location

REFERENCES

1. DiGiorgio, R., An introduction to the URL programming interface. in JavaWorld, (San Francisco, 1999), Sun Corporation.
2. Goldie, H. Does size matter? Size and multi-functionality in consumer electronic devices Communication, Stanford University, Stanford, CA, 1999.
3. Jonsson, I., The decoupled application interaction model: DAIM. in Proceedings of HCII '99, (Munchen, Germany, 1999).
4. Kernal, H., Effects of computer/television convergence on user evaluation of content and equipment. in CHI 99, (Pittsburgh, PA, 1999).
5. Lai, J. and Srinivasan, S., Designed for enablement or disabled by design? Choosing the path to effective speech application design. in Proceedings of HCII '01, (New Orleans, LA, 2001).
6. Myers, M., Hollan, J. and Cruz, I. Strategic directions in human-computer interaction. ACM computing surveys, 28 (4). 794-809.
7. Reeves, B., Detenber, B. and Steuer, J., New televisions: The effects of big pictures and big sound on viewer responses to the screen. in Annual conference of the International Communication Association, (Washington, D. C., 1993, May).
8. Saulsbury, A., Wireless spaces: Enabling technological freedom and creating invisible borders. in Proceedings of HCII '01, (New Orleans, LA, 2001).
9. Tang, J.C., Yankelovich, N. and Begole, J., ConNexus to awarenex: Extending awareness to mobile users. in Proceedings of CHI '01, (Seattle, WA, 2001), ACM Press, 221-228.
10. Weiser, M. The computer for the 21st century. Scientific American, 265 (3). 94-104.