Summary: This article discusses how eye-tracking can be used to supplement traditional usability test measures. User performance on two usability tasks with three e-commerce websites is described. Results show that eye-tracking data can be used to better understand how users initiate a search for a targeted link or web object. Frequency, duration and order of visual attention to Areas of Interest (AOIs) in particular are informative as supplemental information to standard usability testing in understanding user expectations and making design recommendations.
Eye-tracking has been used in usability testing for many years, beginning primarily with cockpit design testing (Fitts, Jones and Milton, 1950). Such early studies were valuable in establishing some assumptions about the relationships between eye-movement data and the cognitive activity of a user. Frequency of fixations was assumed to be related to the importance of the control, while the duration of fixations was related with ease of interpreting information.
In recent years, eye-tracking has been adapted to usability studies involving web-based stimuli (for review see Duchowski, 2003; Hyönä, Radach and Deubel; 2003). Usability professionals and independent research groups have been attempting to identify the specific contributions of eye-tracking to website design and usability. Many such studies involve the identification of the specific targets of the users’ visual attention to different parts of the interface, or Areas of Interest (AOI). The data relative to these AOIs can then be compared in different ways, including order, number and duration of fixations to each AOI.
As part of a continuing investigation into the contributions of eye-tracking to usability testing, user eye-movement data was gathered during testing of three e-commerce websites. The data was examined in terms of various AOIs on the screens and was used to uncover more information about how the users interacted with these interfaces.
Thirty-six undergraduate and graduate students at Wichita State University (26 female, 10 male) at Wichita State University participated in this study. Seventy-two percent of the participants were between the ages of 18 and 26 years. A Pentium IV-based PC computer with 96 dpi, 17″ monitor with a resolution setting of 1024 x 768 pixels was used. The monitor was integrated with the Tobii 1750 eye-tracking system, which was used to detect and collect participant eye-gaze data during testing. The Tobii 1750 eye-tracker samples the position of the user’s eyes on an average of every 20ms (i.e., 50Hz) and is characterized by the unobtrusive addition of the eye-tracking hardware (e.g., high resolution camera and near infra-red light-emitting diodes) to the monitor frame. This design aspect helps promote more natural user behavior by not placing unnatural restrictions on participants (e.g., helmets, head-rests, etc.). Three e-commerce websites specializing in the sale of educational toys were used in the study: Mastermind Toys, Toys to Grow On, and Wonder Brains.
Participant performance data was compared on three usability measures: time on task, number of pages visited, and perceived difficulty (Likert-type scale, 1 = “Very Easy”, 5 = “Very Difficult”). The eye-movement data collected during testing included the number and duration of visual fixations relative to different AOIs on each site’s homepage. The results discussed below are for two of the tasks performed during usability testing of the sites.
Task 1: Find Contact Information
This was a simple information search task that requested users to find contact information for the toy company. On this task Site 2 (MasterMindToys.com) had significantly higher scores for both perceived difficulty (F (2,33) = 5.459, p. = .009) and for Time on Task (F (2,33) = 9.841, p. < .001) (see Table 1).
Table 1. Usability Measures across Sites for Task 1
|Measure||Site 1 (Toystogrowon)||Site 2 (MastermindToys)||Site 3 (WonderBrains)|
|Difficulty||1.08 (0.29)||1.81 (0.60)||1.67 (0.77)|
|Time on Task (sec)||40.12 (10.59)||61.94 (17.1)||42.66 (10.84)|
The specific links that lead to the company contact information were located in different areas of the homepages for each respective site. Site 2 (MasterMindToys.com) had the proper link located among a list of other hyperlinks on the right hand side of the page. By way of contrast, Site 1 (Toystogrowon.com) had the link at the bottom of the page, and Site 3 (WonderBrains.com) had the link located in multiple locations at the top and bottom of the page. The elevated time and difficulty scores for Site 2 suggest that users may have had some preconceived notions of where to find certain common webpage components on a homepage (i.e., contact information), which resulted in poorer performance when the link was in an unexpected location. This view is substantiated by research into web objects and user schemas (Bernard, 2001; Bernard 2002).
Eye-tracking data for this task was used to explore how users went about searching for these particular hyperlinks on the different homepages. Based on the average time to first fixation received to the individual AOIs on the homepage, the users of Site 1 started their visual search for the appropriate link in the center of the page then examined other web components along the edges the page, such as the top shopping cart links and left navigation bar (Figure 1).
Figure 1. Order of first fixations per AOI for Task 1 (Toystogrowon.com)
A similar bias for searching for the contact information link at the top of the page is shown by the AOI data for Site 2, suggesting that expectations as to this type of link’s typical location worked against the users when the link is located elsewhere, such as the links on the right side of the page, which were typically fixated upon last by users (Figure 2).
Figure 2. Order of first fixations per AOI for Task 1 (MasterMindToys.com)
The ClearView eye-tracking software can represent the areas of the screen receiving either more fixations or receiving the longest dwell times in a color-coded “hotspot” image of the interface, with the closer to red the more fixations occurred in an area of the interface and intensity decreasing with movement down the spectrum.
Inspection of these hotspots on the homepages for this task relative to the AOI definitions reveals a very distributed pattern of user eye-movement on Sites 1 and 2 (Figures 3 and 4). This suggests that users directed their visual attention to several areas other than that where the correct link was located. For whatever reasons, it was these areas that users searched most frequently and longest for the correct link in addition to the area where they eventually found it.
Figure 3. Homepage fixation hotspots on Task 1 (MasterMindToys.com)
Figure 4. Homepage fixation hotspots for Task 1 (Toystogrowon.com)
On the other hand, the hotspots for Task 1 on Site 3 were more concentrated and directed primarily at the correct links located at the top and bottom of the page (Figure 5).
Figure 5. Homepage fixation hotspots on Task 1 (WonderBrains.com)
Task 2: Shopping by Age
This task asked participants to find a puzzle for children ages 5 and up. There were significant differences observed for difficulty (F (2,33) = 3.593, p. = .039), time on task (F (2,33) = 5.564, p. = .008), and number of pages (F (2,33) = 4.949, p. = .013) (see Table 2), with Site 1 (Toystogrowon.com) having the highest scores. These results suggest that there was something about Site 1 that made it more difficult for users to shop by age than on the other two sites. These findings are corroborated by a review of the AOI data for this task.
Table 2. Usability Measures across Sites for Task 2
|Measure||Site 1 (Toystogrowon)||Site 2 (MastermindToys)||Site 3 (WonderBrains)|
|Difficulty||2.58 (1.08)||1.72 (1.00)||1.67 (0.65)|
|Time on Task (sec)||108.09 (36.59)||61.31 (27.4)||67.54 (45.64)|
|Number of Pages||8.00 (4.11)||5.08 (1.5)||5.17 (0.94)|
This task could be completed in two different ways on all three sites: (1) using the categories on the left navigation bar to find a puzzle and narrow the search from there; or (2) using the shop by age option available from the homepage. On Site 1, the shop by age option was accessed through a single hyperlink, a solution used by only one participant. Based on the average time to first fixation for each AOI, the area where this link was located was typically attended to later than many other areas (Figure 6).
Figure 6. Order of first fixations per AOI (Toystogrowon.com)
Examination of the fixation and dwell time data in terms of hotspot maps of the homepage during this task assists with the interpretation of this data (Figure 7). Although the left navigation array received the most attention, the “shop by age” option in the upper navigation array was minimally noticed, primarily by the one user that clicked on this link. Higher fixation rates or dwell times in this area would be indicative of users finding the option but perhaps not recognizing its function, and thus failing to choose it. The relative lack of visual attention to that area suggests that few users other than the one who chose the “shop by age” option ever fixated upon it.
Figure 7. Hotspot map of the homepage showing areas of highest fixation for Task 2
On Site 2 (MasterMindToys.com), all the participants but two used the “shop by age” option which was broken down into a range of age-related links on the top of the left navigation bar. This area of the homepage received the majority of the users’ visual attention (Figure 8). The product category links on the left navigation bar and the “shop by age” links above them were attended to relatively late on average, but this did not seem to impact performance for most participants. There were some aspects of the page not useful in this task that received considerable visual attention, including the large picture of the store interior and the center links to the right of the picture, similar to Task 1 on this site (discussed above). This observation is confirmed by the hotspot map of the homepage for Task 2 (Figure 8), and may indicate that these links are distracting and/or listed in a manner that is difficult to quickly glance through and eliminate as potentially useful during a search task. However, the overall design of the page and placement of the age-related links was conducive to a successful search for products by age, an assessment supported by the eye-tracking data.
Figure 8. Homepage fixation hotspots on Task 5 (MasterMindToys.com)
The users of Site 3 (WonderBrains.com) were more evenly split on the use of the “shop by age” option, though it was less popular and the age-related links received less attention than the standard toy categories higher up on the navigation bar (Figure 9). In fact, examination of this hotspot map confirms that the most of the visual attention directed toward the homepage during this task involved the appropriate links—either categorical or age-related—and little else. This implies that the design of Site 3’s homepage was conducive to shopping (at least for this task) and held few visual distractions for participants.
Figure 9. Homepage fixation hotspots on Task 2 (WonderBrains.com)
Much can be learned about the nature of the user experience in the comparison of user visual attention afforded to different areas of web interfaces. Specifically, the use of eye-tracking data allows usability professionals to gather more information about what areas of interest (AOIs) are potentially: (1) most eye-catching; (2) most informative; (3) most frequently ignored; and (4) most distracting. In this way eye-tracking can be used to determine not only whether users are successful at navigating to a particular page of a website, but also where and how they search the interface for a targeted link or web object. This use of eye-tracking data as supplemental information to standard usability testing can be very informative as to both user expectations and resulting design recommendations.
The results of this study emphasize the importance of interpreting user eye-movement data in usability testing within the context of the interface being evaluated. In future studies, AOI eye-tracking could also be utilized not only within a single webpage or interface screen, but also across different pages of a site (see Pan et al., 2004), multiple exposures to a site (see Josephson and Holmes, 2002), and between different stages of a complex usability task. These comparisons could also be performed both in terms of iterative studies between different versions of web page/site and comparative studies between entirely different websites.
Bernard, M. (2001). Developing schemas for the location of common web objects. Usability News 3.2. [Online]. Available: http://usabilitynews.org/developing-schemas-for-the-location-of-common-web-objects/
Bernard, M. (2002). Examining user expectations for the location of common e-commerce web objects. Usability News 4.1. [Online]. Available: http://usabilitynews.org/examining-user-expectations-for-the-location-of-common-e-commerce-web-objects/
Duchowski, A. T. (2003). Eye tracking methodology: theory and practice. London: Springer.
Fitts, P. M., Jones, R. E. & Milton, J. L. (1950). Eye movements of aircraft pilots during instrument-landing approaches. Aeronautical Engineering Review, 9(2), 24-29.
Hyönä, J., Radach, R. and Deubel, H. (2003). The mind’s eye: Cognitive and applied aspects of eye movement research, Amsterdam: Elsevier Science.
Josephson, S. and Holmes, M. E. (2002). Visual attention to repeated images on the World-Wide Web: Another look at scanpath theory. Behavior Research Methods, Instruments, & Computers, 34(4), 539-548.
Pan, Hembrooke, Gay Granda, Feusner & Newman, (2004). Determinants of web page viewing behavior: An eye-tracking study, Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications, 2004, San Antonio, Texas.