by Mark Russell
Summary: This study discusses the contributions of eye-tracking data to traditional usability test measures for first-time usage of websites. Participants viewed the homepages of three different websites. Results showed that eye-movement data supplemented what users verbally reported in their reactions to a site. In particular, the eye-tracking data revealed which aspects of the website received more visual attention and in what order they were viewed.
As eye-tracking systems have become both more sophisticated and more affordable, there has been an increasing interest in the use of eye-tracking within the software usability testing domain (see Byrne, Anderson, Douglass, & Matessa, 1999; Kotval & Goldberg, 1998; and Hornof & Halverson, 2003).
Eye-tracking studies that have focused on web-based stimuli have gathered eye-movement data while participants were engaged in typical information search tasks on web pages (Cowen et al., 2002; Goldberg, Stimson, Lewenstein, Scott & Wichansky, 2002; Josephson & Holmes, 2002). Despite an increased amount of research in this area, it is still not entirely clear exactly how much eye-movement data contributes to the evaluation of a user interface.
As a part of a comparative usability test between three websites of a similar domain, participant eye movements were recorded during the users’ initial introduction to each site’s homepage. The eye-tracking data was examined to gain additional insight into how users initially viewed a site and what page elements attracted their visual attention.
Thirty-six undergraduate and graduate students (26 female, 10 male) at Wichita State University participated in this study. Seventy-two percent of the participants were between the ages of 18 and 26 years. A Pentium IV-based PC computer with 96 dpi, 17" monitor with a resolution setting of 1024 x 768 pixels was used. The monitor was integrated with the Tobii 1750 eye-tracking system, which was used to detect and collect participant eye-gaze data during testing. The Tobii 1750 eye-tracker samples the position of the user’s eyes on an average of every 20ms (i.e., 50Hz) and is characterized by the unobtrusive addition of the eye-tracking hardware (e.g., high resolution camera and near infra-red light-emitting diodes) to the monitor frame. This design aspect helps promote more natural user behavior by not placing unnatural restrictions on participants (e.g., helmets, head-rests, etc.).
Participants were seated approximately 60 cm away from the computer monitor. After calibration, participants were presented with the home page of one of the three e-commerce websites used in the study and were asked to take about 1 minute to look over the site and verbally report their impressions as to the site’s purpose and what audience the site was targeted toward. (Participants completed a series of tasks with each site, but only the analysis of this initial task is discussed in this article.)
All three of the websites used in this study were e-commerce sites that specifically dealt with the sale of educational toys. During this exercise, the participants’ responses were often little more detailed than, "it is a site for toys" and "it is aimed at parents", but this was to be expected. In a standard website usability test, this would simply be a means of determining whether users are able to recognize the purpose or domain of the site simply from the homepage. However, under a typical usability study (that does not gather eye-tracking data), the only data gathered during such an exercise is whether or not the design and information of the homepage sufficiently communicates the purpose of the site to users. The addition of the eye-tracking data allows designers to better understand exactly what users are looking at in order to make sense of the site.
ClearViewTM was used with the Tobii 1750 to examine the users’ eye-movements by means of a post-hoc analysis of visual fixations falling within different areas of each homepage. Specifically, captured images of each homepage are manually divided into natural segments of interface real estate, e.g., the company logo, search field, or navigation menu. The eye-tracking data is then examined in terms of fixations recorded within these segments, or areas of interest (AOI). The fixation data for these AOIs can be compared in a number of ways, including: (1) the order in which each AOI received its first fixation; (2) the number of fixations recorded in each AOI; and (3) the cumulative dwell time of fixations recorded in each AOI. This provides a way to determine what aspects of the homepage users generally look at first, for how many times and for how long when they first encounter the site.
Figure 1. Toys To Grow On homepage (www.toystogrowon.com) with AOIs outlined in blue
The Toys to Grow On homepage was divided into 10 different AOIs (Figure 1). User eye-movement data was reviewed, and it was determined that some AOIs were fixated upon earlier, more often, and longer than others (Table 1).
Table 1. Area of Interest data for homepage
|AOIs||Average Fixation Order||Average Number Fixations (SD)||Average Dwell Time in seconds (SD)|
|Picture||1||11.5 (2.89||329.85 (0.84)|
|Big logo||2||8.5 (2.50)||385.89 (2.06)|
|Text||3||14 (2.3)||771.21 (3.96)|
|Lakeshore logo||4||6 (3.09)||192.63 (0.45)|
|Upper logo||5||1 (1.15)||48.3 (0.26)|
|Left navigation||6||3.5 (5.25)||63.21 (0.34)|
|Shopping cart||7||0.5 (1.41)||265.98 (7.02)|
|Upper navigation||8||1 (1.73)||70.38 (0.29)|
|Bottom navigation||9||0 (0.50)||10.74 (0.00)|
|Search fields||10||1 (1.26)||23.88 (0.53)|
The first AOIs typically receiving fixations from this group of users were the large areas in the center of the page: (1) the picture of the child on the bike; (2) the big Toys To Grow On logo to the right of the picture; (3) the block of text directly beneath the picture; and (4) the Lakeshore logo below the text. These areas are not only the largest areas and those dominating the center of the page, but they also generally provide the most information about the nature of the site, which follows the purpose of this orientation task. Figure 2 shows the general order of first fixation to the different AOIs.
Figure 2. Average order of first fixation per AOI. Participants were asked to state the purpose and target audience of the site.
Another output of the eye-tracking data is a representation of the areas of the screen receiving either more fixations or receiving the longest dwell times in a color-coded "hotspot" image of the interface (see Figure 3). Specifically, the closer to red the more fixations occurred in an area of the interface, with intensity decreasing with movement down the spectrum (see Figure 4). Inspection of these hotspots relative to the AOI definitions reveals that, although the large picture in the center of the page was fixated upon earliest and received the most fixations on average, the portion within that AOI most fixated upon was the child’s face, and not the toy motorcycle he is riding. This indicates two things: (1) just because certain AOIs are larger than others, that does not mean that visual attention (i.e., concentration of fixations) is evenly distributed across the AOI; and (2) it may be that photos of toys posted on the homepage to generate interest in the company’s products might be more effective if they emphasize the toy over the child model.
Figure 3. Hotspot map of the homepage showing areas of highest fixation count
Figure 4. Example legend for hotspot maps; red = more fixations, green = fewer, etc.
It is interesting to note, however, that the Lakeshore logo, which was typically fixated upon 4th in sequence, received an average of 6 fixations per user and was dwelled upon for an average of 3 seconds, actually provides very little in the way of information about this particular site. If it were the intention of the website’s owners to emphasize their company’s relationship with Lakeshore, this strategy was successful. However, if it is more important for users to focus on the explanatory text or the navigation menu options, this logo may be considered a distraction.
Figure 5. Mastermind Toys homepage (www.mastermind.com) with AOIs outlined in blue
The Mastermind Toys homepage was divided into 18 different AOIs (see Figure 5). User eye-movement data was examined in a manner similar to the first site. Table 2 shows a summary of the fixation data by AOI.
Table 2. Eye-tracking data for homepage AOIs
The first AOIs on the homepage typically receiving fixations from this group of users were those running along the top of the page: (1) the "live help" information—not a link but merely a 1-800 number listing); (2) the top "Mastermind Toys" logo; and (3) the search engine (see Figure 6). This was generally followed by fixations on the large areas in the center of the page: (4) the picture of the inside of the store; and (6) the main listing of links to the right of that picture. All these areas were not fixated upon very many times or for very long, however, as the areas given the most visual attention—in addition to the large picture of the inside of the store—were the various products pictured down the center of the homepage. Remarkably, even those items appearing far "below the fold" were extensively examined.
Figure 6. Average order of user visual fixations on Homepage AOIs
It is interesting to note that users depended primarily on the pictures of the products displayed on this homepage in order to determine the purpose of the site, rather than focus on the listing of links on the left navigation bar. This supports the apparent intent of the site’s owner(s)/designer(s) to generate interest in the site by drawing attention to some available products, but also brings into question the effectiveness of the lengthy left navigation bar. Hotspot analysis of this page showed that the portion within the large store AOI most fixated upon was the text "10 stores to serve you", and not the toys in the picture (see Figure 7). Again, this indicates two things: (1) just because certain AOIs are larger than others, that does not mean that visual attention (i.e., concentration of fixations) is evenly distributed across the AOI; and (2) if text in this area is quickly and frequently attended to, this might be a good place to provide more critical information (e.g., sales, popular toy names) rather than a generally descriptive statement about the number of stores in the chain.
Figure 7. Hotspot map showing areas of highest fixation count
Figure 8. Wonderbrains homepage (www.wonderbrains.com) with AOIs outlined in blue
The WonderBrains homepage was divided into 18 different AOIs (see Figure 8). User eye-movement data was examined in a manner similar to the first site. Table 2 shows a summary of the fixation data by AOI (Table 2).
Table 2. Eye-tracking data for homepage AOIs
|AOIs||Average Fixation Order||Average Number Fixations (SD)||Average Dwell Time in seconds (SD)|
|Logo||1||12.92 (9.89)||0.08 (0.22)|
|Description||2||17.92 (11.31)||2.92 (1.31)|
|Header||3||16.08 (6.17)||3.16 (1.71)|
|Promo 1||4||32.08 (16.82)||1.63 (2.03)|
|left nav categories||5||20.58 (14.71)||6.15 (3.57)|
|shopping cart||6||5.67 (5.07)||0.69 (0.77)|
|right shipping||7||9.00 (6.56)||2.69 (1.43)|
|promo 2||8||19.42 (9.52)||6.99 (4.00)|
|top nav bar||9||6.25 (5.55)||3.88 (2.87)|
|right special||10||9.08 (9.81)||2.79 (2.42)|
|left nav ages||11||8.33 (12.57)||0.59 (0.78)|
|bottom nav||12||0.33 (0.89)||1.05 (1.17)|
|search engine||13||3.17 (3.24)||0.33 (0.67)|
|promo 3||14||13.42 (10.35)||2.10 (1.32)|
|right join||15||2.92 (3.73)||1.87 (1.27)|
|left nav wb||16||1.92 (1.88)||0.43 (0.46)|
|right math||17||2.92 (3.34)||0.90 (0.79)|
|right resources||18||0.92 (1.73)||0.96 (0.74)|
The first AOIs on the homepage typically receiving fixations from this group of users were those that gave the most information with the least amount of reading: (1) the logo in the upper-left corner; (2) the brief description of the purpose of the website; and (3) the header above that description. Otherwise, there is no readily recognizable pattern of user visual fixations across the page (see Figure 9). This may indicate that there were few areas of the page that attracted a great deal of attention compared to another during later exploration of the page. This idea is supported by the fact that much of the page is filled with blocks of small text, creating a look of uniformity across the page. By the same token, there were no areas of the homepage that were ignored during this initial exposure to the site.
Figure 9. Average order of user visual fixations on Homepage AOIs
Typically the areas receiving attention first during this task received more attention in terms of both number of fixations and dwell time than others, with the exception of the left and top navigation bars and the promotional information running down the center of the page. This is in keeping with the goals of this task and justifies the inclusion of this information for the elucidation of the site’s purpose to new visitors.
Analysis of this site’s homepage in terms of "hotspots" relative to the AOI definitions reveals a summary of the user eye-movement data similar to that explained above (see Figure 10). For instance, the upper-most options on the left navigation menu received the most fixations, probably because the users only needed to see a few to understand the general trend of products offered through the site. Interestingly, one area that received a high number of fixations but is not necessarily useful in determining the purpose of the site is the "Free Shipping" blurb located just above the shopping cart. It could be that this text attracted more attention due to it being a different color than most the other text on the page. This might be an indicator to designers that this could be a means of drawing users’ attention to other aspects of the page in the future.
Figure 10. Hotspot map showing areas of highest fixation count
This study demonstrates that eye-movement data gathered during an initial encounter with a website’s homepage supplements what users are able to verbally report in their reactions to a site. In addition, it provides objective data as to what aspects of the website receive more visual attention. Breaking down web pages into specific areas of interest (AOI) can provide an indication of where visitors may be looking first, most often and longest on the page. This information can be visually summarized in the form of "hotspot" maps of the page, showing a color-coded representation of where users look most or longest.
Future research examining user eye-movement data should investigate a number of different aspects of website design. For example, designers could investigate how users change their visual attention to different aspects of the page depending on: (1) the nature of the task; (2) repeated exposures to the site; and (3) the relative arrangement of the page’s content.
Byrne, M. D., Anderson, J. R., Douglas, S. & Matessa, M. (1999). Eye tracking the visual search of click-down menus. In Proceedings of CHI ‘99,15-20. Pittsburg: ACM Press.
Cowen, L., Ball, L.J. & Delin, J. (2002). An eye movement analysis of web page usability. In Proceedings of the annual HCI 2002, 317-335.
Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N. & Wichansky, A. M. (2002). Eye tracking in web search tasks: Design implications. In Proceedings of the symposium on ETRA 2002: Eye tracking research & applications symposium, 51-58. New Orleans: ACM Press.
Hornof, A. J. & Halverson, T. (2003). Cognitive strategies and eye movements for searching hierarchical computer displays. In Proceedings of CHI 2003, 5(1), 249-256. Ft. Lauderdale, FL: ACM Press.
Josephson, S. & Holmes, M. E. (2002). Visual attention to repeated images on the World-Wide Web: Another look at scanpath theory. Behavior Research Methods, Instruments, & Computers, 34(4), 539-548.
Kotval, X. P. & Goldberg, J. H. (1998). Eye movements and interface component grouping: An evaluation method. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 486-490.