Summary: This study reports on the usability test of three weight loss websites. In addition, eye tracking patterns were observed for initial exposure to each site home page. Results indicate that participants were able to search the Atkins diet site more efficiently than the Jenny Craig website or Weight Watchers website and preferred this site overall. Analysis of eye-tracking data suggests users first fixate on graphics and large text even when looking for specific information. Interface issues contributing to overall satisfaction and preference are discussed.
As 2004 begins, millions of Americans have resolved (once again) to lose weight. MyGoals.com listed “to lose weight” as the most popular resolution for 2004 (2004 New Year’s, n.d.). In a meta-analysis of dieting programs and long-term weight loss success stories, reviewers for the International Journal of Obesity reported that a correspondence program was the most effective way to assist in long-term weight loss and improved fitness. A study conducted by Tate, Wing, and Winett (2001) found that the Internet and email were reliable methods for assisting in weight loss programs. At the 2003 annual meeting of the American Dietetic Association, experts supported the value of online resources for complementing dieting efforts. A panel of experts at the meeting unanimously agreed that online programs should be used as tools to help dieters “achieve permanent weight loss and improved fitness” (Burke, 2003).
Advertising in popular media suggests there are currently many popular weight-loss programs—Jenny Craig, Weight Watchers, South Beach Diet, Atkins Diet, Zone Diet, among others. In this study, we chose to evaluate the websites promoting three of these popular weight loss programs: Jenny Craig, Weight Watchers, and the Atkins Diet (see Figures 1-3 for images of these sites’ homepages). This usability test evaluated first-time users’ satisfaction, navigational efficiency, and general preferences for the three sites.
We also began exploring the usefulness of gathering user eye-movement data in conjunction with web usability testing. On virtually any given e-commerce website there is a war being waged for user attention. The collection and analysis of eye-tracking data would logically benefit web designers and advertisers who need to know where users look and why. What draws attention and what inhibits it? What aspects of a site are looked at most often and the longest? Can reading behavior be influenced by the page design? These and many other questions still remain to be answered as we continue to investigate the difference between eye-movement, attention, and comprehension.
Figure 1. JennyCraig.com homepage
Figure 2. WeightWatchers.com homepage
Figure 3. Atkins.com homepage
Nine participants (5 female, 4 male) with an average age of 24.6 volunteered for this domain study. All participants were familiar with the Internet (67% reported using it daily); none reported having visited the online diet sites before. Five participants reported purchasing products online 1-5 times in the last year, 2 reported zero online purchases in the past year, and the remaining reported buying more than six times. Five of the participants indicated they were somewhat interested in diet information, while 3 reported they were not interested at all, and 1 participant was neutral in opinion.
A Pentium IV-based PC computer with a 75 Hz, 96 dpi, 17” monitor with a resolution setting of 1024 x 768 pixels was used. In addition, the monitor used during testing was an integrated component of the Tobii ET-17 eye-tracking system. The Tobii ET-17 was used to detect and collect participant eye-gaze data prior to usability testing. This system is characterized by the unobtrusive addition to the eye-tracking hardware (e.g., high resolution camera and near infra-red light-emitting diodes) to the monitor. This aspect helps promote more natural user behavior by not placing unnatural restrictions on participants (e.g., helmets, head-rests, etc.). Participants answered the System Usability Scale (SUS) instrument (Digital Equipment Corporation Limited 1986), which was slightly adapted for web usage and consisted of 10 satisfaction questions using a 1-5 Likert scale. A final questionnaire was used to solicit the participants’ overall preference for one of the three websites.
Participants completed a background questionnaire about their typical Internet usage. Participants were seated approximately 60 cm away from the monitor. The eye tracker was calibrated for each participant. Once the eye-tracking apparatus was calibrated, participants were asked to view an image of each website home page and to report their impressions as to the site’s purpose and the types of information likely to be available on the site. During this time (20 seconds per image), participants’ eye-movements were recorded as they visually inspected the image and gave verbal reports. After this initial task, participants were given a total of seven tasks for each of the three sites (site order and task presentation were counterbalanced across participants). The tasks included finding details of the diet plan, recommended types of exercise to accompany the plan, success stories, and advice when eating out at restaurants.
Participants’ search efficiency, or ‘lostness’ was measured by the number of pages traversed beyond the optimal number of pages to complete a task as measured by the tracking program Ergobrowser™. In order to better analyze the efficiency and intuitiveness of the site structure, participants were instructed to find the answers to the tasks without the assistance of the site search engine. After completing the tasks on each site, participants answered the SUS. After tasks were completed on all three sites, participants were asked to indicate their favorite website of the three.
Average time taken to complete the seven tasks ranged from 12.24 minutes on the Weight Watchers site to 12.90 minutes on the Jenny Craig site. The time taken on the Atkins site (M = 12.87 minutes) was similar to that taken on Jenny Craig. Results of a repeated measures one-way ANOVA revealed no significant differences across the three sites for time taken to complete the tasks.
Significant results were found for overall search efficiency or “lostness” on the seven tasks using a repeated measures one-way ANOVA, [adjusted F (1.347, 10.779) = 4.718, p < .05]. Paired samples t-tests indicated the Atkins website (M = 21) was significantly more efficient than both Jenny Craig (M = 30) and Weight Watchers (M = 31.44). Figure 4 illustrates the search efficiency for the three sites.
Figure 4. Mean search efficiency for all seven tasks
Figure 5 shows participants’ rate of successful task completion across all three sites. Participants were most successful in completing the tasks on the Atkins site, and least successful on the Jenny Craig site. No significant differences were found using a repeated measures one-way ANOVA.
Figure 5. Successful task completion on diet sites
After completing each task, participants indicated their subjective satisfaction with each website. Atkins received a mean score of 30 (maximum possible = 50) which was slightly higher than Jenny Craig (M = 29.22) and Weight Watchers (M = 28). Results of a repeated measures one-way ANOVA revealed no significant differences across the three sites for satisfaction.
After all testing was completed, participants were asked to indicate which site they preferred overall. Figure 6 shows more participants chose the Atkins website as their favorite site.
Figure 6. Participant preference for diet websites (# participants choosing site as first choice)
“Hot spot” maps summarizing the gaze position for each website home page were generated across users (though only 7 of the 9 participants were included due to calibration difficulty). The maps are color-coded similar to what you might expect from thermal imaging, with red being the most viewed area, graduating down to light yellow to indicate less fixation time. These hot spot maps are useful for a quick assessment of the areas of the page most frequently fixated on by participants. We also studied the user visual fixation data for experimenter-defined areas of interest (AOI’s) on each image. This enabled the collection of multiple visual fixations in a large area to be combined, so that a more precise estimation of user attention could be obtained.
Participants were asked to view the image of each site and then verbally report “what information they could expect to find on this website.” According to this data, the hot spot map of the Atkins.com site (Figure 7) indicates which specific areas received the most visual attention by participants. The most frequently and longest fixated upon AOI (M=2748ms, SD=1366ms) on this page was the rectangular area in the center of the page with the title “4 steps to a healthy new lifestyle.” The attention afforded this AOI is reasonable as it contained a block of text that explained the philosophy of the Atkins diet plan (and was targeted information). Despite the presence of the color red on the hot spot map, the ‘My Atkins’ AOI was actually fixated upon for less time (M=1365ms, SD=1412) than the navigation bar (M=2474ms, SD=2129ms) and the Atkins logo (M=1717ms, SD=1162ms). This was most likely because the other AOI’s were larger and fixation times collected in those areas exceed that for the smaller areas.
Figure 7. ‘Hot Spot’ map of the Atkins homepage
The hot spot map and AOI dwell time data for the Jenny Craig home page (Figure 8) indicate the area most fixated on to be the ‘Internet-only’ advertisement (M=2346ms, SD=1235ms), followed by the ‘Joy lost 23 lbs’ text (M=1544ms, SD=1102ms). It is interesting to note that details about the Jenny Craig diet plan and clues as to what information is available on this website located in the lower left portion of the screen was fixated upon an average of 1,000ms less than the ‘Internet-only’ ad (M=1306, SD=1108). The results for the Jenny Craig page are very similar to those of the Atkins page; participants apparently focused primarily on very specific areas of the screen for the majority of the 20 seconds. However, in both cases, it does not appear as though the areas of the page that actually contained the most information about the website and/or diet plan received the most fixation time.
Figure 8. ‘Hot Spot’ map of Jenny Craig homepage
The hot spot map for the Weight Watchers home page (Figure 9) was somewhat different. The most popular areas were the ‘Flex Points’ caption (M=2459ms, SD=1424ms), the ‘Meetings’ text directly under it (M=1985ms, SD=1229ms), and the information for online meetings below that (M=1925ms, SD=874ms). It is interesting to note, however, that on this site participants apparently fixated on areas on the left side of the screen considerably more than the right, and their hot spot areas cover more area on the screen than the other two diet sites. This suggests that no specific areas of the screen (other than the entire left side of the page) attracted the users’ attention which may have resulted in a less efficient visual search.
Figure 9. ‘Hot Spot’ map of Weight Watchers homepage
It is difficult to draw any specific conclusions from the preliminary eye-tracking data. A general observation would be that participants focused more on very specific areas of the Atkins and Jenny Craig home pages. However, upon further inspection, these areas are not necessarily those that would assist the participant in objectively determining what kinds of information might be found on the sites. Data from the Weight Watchers home page was somewhat different, in that participants seemed to both favor the left side of the screen over the right, yet did not have a specific set of targets on that side that held their attention.
Results from this study showed that users were able to complete the tasks successfully and were equally satisfied by all three sites. However, participants searched for the information more efficiently on the Atkins site and preferred this site overall.
Participants found some tasks more challenging than others across the three sites. Five users failed to correctly explain the purpose of the Jenny Craig program. This finding is complimented by the eye-tracking data, which showed participants (at least initially) fixated on graphics or large text on the home page more than the main navigation area or information near the ‘fold’. The home page was not the only problem, however, as participants did not look deep enough into the site to locate the fact that you must buy your food from Jenny Craig centers.
Many participants commented that the Atkins site was easier to use. The sitemap was a benefit noted by one user. The site was labeled as “well planned out” and “more professional” than the other sites. The titles and links were viewed as more specific resulting in an easier navigation experience. Participants commented negatively about the many areas of the Weight Watchers site that require a paid membership. The participants did not feel they could get the full scope of the site from the limited areas made available to all users.
Areas noted by participants to impact overall preference included the following:
Navigation Location: Locating the navigation at the top of the screen was seen as beneficial by many participants. However, on the Weight Watchers site, clicking the top navigation resulted in a sublevel menu directly below the clicked tab. Many users commented on the small size of the font and most missed the sublevel navigation completely on the first few tasks. Jenny Craig uses a similar structure after the home page, but the sublevel navigation is further below the clicked top level navigation. On the Atkins site, the home page has the main navigation links located at the top of the screen as well. On subsequent pages, the sublevel navigation is located to the left of the main information area. Users repeatedly ignored the left navigation. Possible explanations for this include the fact that on the home page this space is occupied by a picture; this area is also a different color and may have appeared to be an advertisement; or the user may have expected the sublevel navigation to be closer to the top level navigation.
Navigation Wording: Users complained about the wording used for the links on all three sites. The task asking about exercise was particularly troublesome because there was no consistent location for it on the sites. Several participants were looking for the key word “Exercise” which was not on any of the sites. Weight Watchers used the term “Healthy and Fit” which was more intuitive to the users than the wording used on the other sites. Wording also affected results on the task about eating out; again users expected this to be worded as “eating out” or “dining out.”
Membership Requirements: The membership requirement was a major factor affecting the opinion of the Weight Watchers’ site. The site uses a different model than the other two and requires a paid membership to access many features. The users did not feel they could get adequate information about the weight loss program without paying for a membership. Some users commented they thought they would have to go to a meeting to get information.
Page Layout: Many users preferred the home page of the Atkins site over the other two sites. The initial page of the Jenny Craig site contained little information and had two large buttons reading “Client” and “Guest.” Almost all of the participants started their activities by clicking the guest button. For tasks such as locating a recipe, they completely ignored the link labeled “Recipes” and clicked the Guest button. The Flash features on the Jenny Craig home page were also described as distracting and meaningless. The favorite feature on the Jenny Craig site was the search functions within sections of the website. The search boxes were located in a highly visible area to the right of the main text and were easy to use when locating recipes and success stories. The Weight Watchers site was reported to be “cluttered and unorganized.” Most users liked the large blue buttons located at the top of the screen on the Atkins site. As mentioned, most users completely disregarded the sublevel navigation to the left of the main text area. Other complaints on the Atkins page layout regarded the amount of space used immediately below the navigation bar. Participants did not understand the purpose of this portion of the page and often found it led them to information they were not seeking.
2004 New Year’s resolutions bode well for economy. (n.d.). Retrieved December 21, 2003, from MyGoals.com Web site: http://www.mygoals.com/about/HF
Digital Equipment Corporation Limited (1986). System Usability Scale. Reading, United Kingdom: Digital Equipment Co. Ltd. ©.
Burke, Susan L. (2003). Losing weight online: The new diet evolution. Retrieved December 21, 2003, from http://www.ediets.com/news/article.cfm?article_id=8675&pg=1
Tate D.F., Wing, R.R., Winett, R.A. (2001). Using Internet technology to deliver a behavioral weight loss program. Journal of the American Medical Association, 285(9), 1172-1177.