By C. Siu & B. Chaparro
Summary. Previous research has shown that users scan traditional text-based Web pages using an “F-shaped” pattern. Evidence of this pattern exists because of the hierarchical structure of search engine result pages (SERP). In Windows 8, search results are presented as a horizontal grid rather than a list. Due to the structural differences between the list and grid layouts, it is reasonable to postulate that users may exhibit different scanning patterns between layouts. This study compared the eye-tracking data of two SERP layouts (grid vs. list) with two types of tasks (informational vs. navigational) to observe differences in gaze patterns. Results indicated that users viewed the top left portion of the grid layout the most. However, there was little consensus as to how the results were ordered in the grid, unlike the list layout.
This is a summary of the article titled “First Look: Examining the Horizontal Grid Layout using Eye-tracking” in the Proceedings of the Human Factors of Ergonomics Society Annual Meeting in Chicago, IL 2014.
Search results are traditionally displayed as a list in which the most relevant results are located at the top of the page. The rank-ordered structure guides users to view traditional search engine result page (SERP) layouts linearly. Past eye-tracking research has shown that users follow an “F-shaped” pattern of scanning within the traditional list layouts (Nielson, 2006; Shrestha et al., 2007; 2008).
Implications of the “F-shaped” gaze pattern are less prominent in alternative SERP layouts. Shrestha (2012) investigated users’ interaction with three SERP layouts—list, tabular, and grid (See Figure 1). In comparison to the list and tabular layout, the linear scanning pattern was less noticeable within the grid layout. However, Shrestha (2012) noticed users treated the two columns as individual lists. That is, users vertically scanned down the left column before moving onto the right column. Similarly, Kammerer and Gerjets (2014) found differences in gaze patterns between the grid and list layouts when trustworthiness of the search results were manipulated in descending (most to least) and ascending (least to most) order.
Figure 1. Example of the traditional and alternative SERP stimuli used in Shrestha (2012).
Compared to the list layout, the layout of Windows 8 may reveal unique differences in how users view a different SERP layout. Unlike the list interface which naturally guides users to search in a top-down manner, Windows 8 incorporates a grid tiled layout that may display unclear ranking positions and require users to navigate horizontally. While Shrestha (2012) and Kammerer and Gerjets (2014) found differences in scanning patterns for a grid layout, the stimuli that were used were not fully representative of the grid layout presently implemented in a Windows 8 interface.
The goal of this study is to understand how users view the grid SERP layout currently implemented in Windows 8. This study compared two SERP layouts (grid vs. list) and two types of tasks (informational vs. navigational). Eye-tracking data was gathered to see if differences in gaze patterns existed when users perform two types of search tasks on the two SERP presentations
A total of 45 participants (16 males and 29 females) aged 18 to 46 (M = 23.05, SD = 6.71) were recruited through the Wichita State University SONA system. Course credit was granted for their participation. All participants reported that they owned some form of computing device. A majority of participants (69%) reportedly used their laptop as their primary source of computing. More than half of the participants (58%) used Windows 7 as their primary operating system. The remaining users used Windows 8 (n = 9), Mac OS X (n = 8), or other (n = 2).
Hardware. A Tobii X120 eye-tracking system was used to capture eye-tracking measures—first fixations, fixation count, visit count, and total fixation durations
Stimuli. Two different layouts were used in this study. A grid layout was designed to mirror the SERP interface of Windows 8 and a list layout was designed to reflect the traditional SERP interface of Google. Results presented on each layout were for either an informational or navigational search task. Figures 2 and 3 show an example of the stimuli.
Tasks. Search tasks modeled after Shrestha (2012) were used. There were 12 tasks total (6 informational; 6 navigational). The informational search task (represented as info task) was synonymous with an exploratory search task and participants were able to choose an answer from multiple sources (e.g., “You had heard that Facebook was once sued by twin brothers. What are their names?”). The navigational search task (represented as nav task) was specified at a particular area on the SERP layout— the 2nd, 4th, 5th, 6th, 7th or 9th search results (e.g., “Why should you love the Skinny Caramel Macchiato according to the Starbucks website?”). Figures 2 and 3 show an example of an info task and nav task on the two layouts.
The experiment lasted approximately 60-75 minutes. Prior to the experiment, participants were asked to complete a consent form and a background questionnaire. Participants were calibrated using Tobii X120 to accurately capture their eye movement. Participants completed 6 info tasks and 6 nav tasks (12 tasks total) on two SERP layouts (grid and list). Task type and layout were counterbalanced for each participant. Tasks within each layout were randomized.
The 12 search tasks were presented on the computer screen. Participants were asked to read aloud the search tasks before starting the search task. Each participant was given a practice trial for the grid and list layout. They were allotted 5 minutes to complete the search task. If the task was not completed within the given time interval or if a participant quit the task, it counted as a failure. Upon completion of each layout, participants were also asked to complete a satisfaction survey and were asked to give a layout preference and answer a few open-ended questions on search strategies post-experiment.
Viewing Patterns of Task Type and Layout Type
Areas of Interest (AOIs) were specified on each search hit for the two layouts. Based on the official ordering of the search results found on the grid and list layouts, three areas of the grid and list layouts were grouped into these three sections. Figure 4 provides a visualization of how these AOIs were divided.
Figure 4. Sections of AOIs for Grid and List layouts: Blue represents search hits 1-3 (GridLeft; ListTop), purple represents search hits 4-6 (GridCenter; ListCenter); green represents search hits 7-10 (GridRight; ListBottom).
Order of Fixations. Results revealed that fixation patterns within the grid were not as systematic as fixation patterns the traditional list layout. Overall, regardless of the type of tasks, the order of fixations for the search results on the traditional list layout remained relatively similar. The order of fixations on the grid was also relatively similar regardless of the type of task. Users fixated first on the results located on upper left quadrants of the grid layout for the info and nav task condition. Figure 5 shows the gaze orders and Figure 6 shows an example of a heatmap generated from the gaze data.
Comparisons of the Grid and List Layout by Task Type
A series of one-way ANOVAs were conducted to look at the number of fixations and total fixation duration for the AOIs on grid & info task, grid & nav task, list & info task, and list & nav task. Table 1 displays the means and standard deviations.
Fixation Count. Overall, results showed significance for the info task within the layouts, F(1, 43) = 101.92, p < .001, partial η2 = .70. Users fixated more on the grid layout than the list layout when performing an info task.
Significant differences were found for the grid layout, F(1, 43) = 122.23, p < .05, partial η2 = .74, and list layout, F(1, 43) = 6.65, p <. 05, partial η2 = .13, for the tasks. Users fixated more when performing an info task than a nav task on the grid and also list layout.
Total Fixation Duration. Significant differences were found for the list layout for the tasks, F(1, 43) = 6.13, p <. 05, partial η2 = .13. Users fixated longer when performing an info task than a nav task on the list layout.
Table 1. Mean (SD) of Eye-Tracking Measures for Task Type (Info/Nav) and Layout Type (Grid/List)
Comparisons of Specific AOIs on the Grid Layout
Results from the eye tracking data indicated that users view the grid layout differently than the list layout. Unlike the list layout, users were not consistent in reporting the location of the ‘top three’ results. When users were prompted (post-experiment) to indicate where they believed the ‘top three’ results were located, 57% of the participants reported that the ‘top three’ search results would be located in the top row and 32% indicated they would be in the left column (see Figure 7). 11% of the participants were unsure or thought there was no pattern.
Overall AOIs Comparisons of Left Column & Top Row
To better understand where users viewed the top results on the grid layout when performing an info task or nav task, multiple dependent t-tests were conducted to compare the number of fixations, visit count, and total fixation duration between the results located on the top row and left column of the grid layout. AOIs for these two sectioned areas were created and represented in Figure 8. Table 2 displays the means and standard deviations. A Bonferroni correction was implemented to control for family-wise type I error (p = .013).
Fixation Count. Results indicated no differences in fixations for the info task between the Left – Top, t(22) = -2.04, p > .05, d = .29. There was no significant differences for the nav task between Left – Top, t(21) = -2.33, p > .01, d =.33.
Visit Count. No significant differences were found in revisits for the info task between Left – Top, t(22) = -.83, p > .05, d =.10, and nav task between Left – Top, t(21) = -1.27, p > .05, d =.10.
Total Fixation Duration. Results indicated significant differences in total fixation duration for Left – Top when performing an info task, t(22) = -2.75, p < .013, d =.33, and nav task, t(21) = -3.14, p < .01, d = .35. Users fixated longer on the top row than the left column when performing an info task and nav task.
Table 2. Means (SD) of Eye-Tracking Measures for Left Column and Top Row for Task Type (Info/Nav) on the Grid Layout
Overall Satisfaction and Preference. An adapted version of SUS scale was used to gather an overall satisfaction rating for the two layouts. The SUS score ranges from 0 to 100 with a higher score indicating higher satisfaction. A dependent t-test revealed that were no differences in satisfaction between the grid and list layout, t(44) = -1.76, p > .05 (see Figure 9). When asked which layout they preferred, 47% of participants chose the grid over the list layout.
This study provides some insight as to how users view a grid layout in comparison to a traditional list layout when performing two types of tasks with search results. Similar to Kammerer and Gerjets (2014), results from this study showed that users viewed the grid layout differently from the list layout. The “F-shaped” pattern was less prominent on the grid in comparison to the list layout. Dense gaze fixations were found on the upper left quadrants of the grid and users fixated on these areas earlier and for a greater length of time. The results located on the bottom row and right column of the grid were typically scanned less and last.
The lack of a formal structure in the grid layout may have confused the users’ perception of search result order. When users were asked where they thought the ‘top three’ results were located most indicated that it was either the left column or the top row. Users tended to look at the top row more and for a longer length of time than the left column regardless of task type. This seems to imply that users abided to a top-to-bottom rather than left-to-right scanning pattern. Had users known of the correct ordering of the search results, it could be argued that differences would have been found and users would have fixated more on either result hits based on this prior knowledge of the search results order.
Windows 8 is currently used by millions (Warren, 2014). This study provides a better understand how users may be approaching this new horizontal grid-based interface. Designers should consider these results to optimize the horizontal grid layout. Due to the ambiguous ordering of the search results on the grid layout, designers cannot assume that all users will view the results in a similar manner. If the upper left quadrants are where users tend to focus first, it would be important to place the most relevant information in that specific area. Otherwise, the potential important information may be ignored similarly to the bottom results on the list layout. On the other hand, Kammerer and Gerjets (2014) postulate that perhaps there are some benefits to the confusion. If users do not realize the true ordering on the grid interface, the nontraditional viewing patterns may be good for information processing. Instead of relying on the search engine’s ranking algorithm, users would have to actively evaluate the sources in accordance to trustworthiness. It is possible that as users become more familiar with the layout, a bias on the upper left portion of the grid may result in users ignoring other areas on the layout.
Brooke, J. (1996). SUS: A quick and dirty usability scale. In P. Jordan, B. Thomas, B. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189-194). London, UK: Taylor & Francis.
Kammerer, Y., & Gerjets, P. (2014). The role of search result position and source trustworthiness in the selection of web search results when using a list or a grid interface. International Journal of Human-Computer Interaction, 30, 177-191.
Nielsen, J. (2006). F-Shaped Pattern for Reading Web Content. Retrieved November 1st, 2013 from http://www.nngroup.com/articles/f-shaped-pattern-reading-web-content/
Shrestha, S., Lenz, K., Owens, J. & Chaparro, B. S. (2007). F-Pattern Scanning of Text and Images on Web Pages. In Proceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Society, (pp.1200-1204).
Shrestha, S., Owens, J. & Chaparro, B. S. (2008). Eye Movements on a Single-Column and Double-Column Text Layout. In Proceedings of the 52nd Annual Meeting of the Human Factors and Ergonomics Society, (pp. 1599-1603).
Shrestha, S. (2012). The effect of search engine results page presentation style on user satisfaction and eye movements. (Doctoral dissertation, Wichita State University). Retrieved August 2013, from SOAR: Shocker Open Access Repository. (Publication No. d12039).
Warren, T. (2014). Windows 8 passes 2 million license sales. Retrieved July1st 2014 from http://www.theverge.com/2014/2/13/5408666/windows-8-200-million-license-sales