By Barbara Chaparro, Vanessa Pereira, and Shawn P. Padgett
OfficeMax and Staples have been touted as the top online office supply sites for small businesses (Konrad, 2001). Staples.com, in particular, has been acknowledged for its heavy focus on site usability and demonstrated increases in the number of repeat customers, reduced drop-off rates, increased traffic, and improved shopping experiences by customers after a site renovation in mid-2000 (CHI_India, 2000). Office supply sites are a challenge to web designers because of the huge volume of items available to the consumer (> 30,000 items). The task of categorizing them into a clear, concise, and usable site can be overwhelming. In this study we asked users to work with three different office supply sites to see how they compared to one another.
We evaluated participants’ user satisfaction, navigational efficiency, and general preference for three office supply sites – staples.com, officemax.com, and vikingop.com. Participants’ search efficiency, or ‘lostness’, was measured by the number of pages traversed beyond the optimum number of pages to complete a task. This efficiency data was gathered by the tracking program ErgobrowserTM.
After completing the tasks with each site, participants answered the End-User Computing Satisfaction (EUCS) instrument (Doll, Xia, & Torkzadeh, 1994), which was adapted for web usage and consisted of 12 satisfaction questions using a 1-5 Likert scale. After completing the tasks with all three sites, participants ranked the sites in order of preference. A Pentium II based personal computer, with a 60 Hz, 96dpi 17" monitor with a resolution setting of 1024 x 768 pixels was used.
Nine participants (5 male, 4 female) with an average age of 20.7 volunteered for the usability study. All participants were familiar with the web (67% reported using it daily, 33% reported using it at least a few times per week), but had not visited any online office product sites before. Six participants reported buying from the internet 1 – 5 times in the last year, 1 reported buying more than 30 times, and 2 reported never buying from the internet before. Participants were asked to find seven items on each site and place them in their shopping cart (the order of the items was randomized and the order of the sites was counterbalanced across participants).
Your boss would like you to purchase the following office supplies online. Find each item and place it into your shopping cart:
a. 12 yellow legal notepads
b. 1 electric pencil sharpener
c. 1 color and 1 black inkjet cartridge for an Epson 740 printer
d. 1 letter opener
e. 4 black Permanent markers
f. 4 Post-It memo cubes
g. 6 rolls of PoloroidTM film
At the end of the tasks, the participants were asked to write down the total cost from the shopping cart. For the purposes of this study, participants were asked to find the items without using the site search engine. This was done to better analyze the efficiency and intuitiveness of the site structure.
Figure 1-3 show the average satisfaction, lostness, and success scores for the three sites. Results from a one-way ANOVA revealed no significant difference across the sites [F (2,16) = .124, p > .05]. In addition, no site was significantly preferred over another [Mean ranks: Staples 2.2, OfficeMax 1.8, Vikingop 2.0; Friedman c2 (2, N = 9) = .89, p > .05]. Analysis of the navigational efficiency, or lostness, showed Staples.com and Vikingop.com to be superior to Officemax.com [F(2,14) = 7.06, p < .01] (see Figure 2). Interestingly, however, participants were found to be more successful in finding the items within Officemax.com (92%) than within Staples.com (79%) and Vikingop.com (76%).
Figure 1. Reported Satisfaction of Office Supply Sites
Figure 2. Efficiency of Office Supply Sites
Figure 3. Task Success with Office Supply Sites
It is important to note that none of the users really had significant problems with any of the sites and task success was fairly high. The items most difficult to find on all sites were the Polaroid film and letter opener. In fact, at the time of this test, the only letter opener available on OfficeMax was an industrial, mailroom letter opener for more than $1000. (As a result, the above efficiency and success data were analyzed using only the remaining 6 tasks.) A separate card sorting exercise of 208 office products showed that users expected film to be under a general "Camera Supplies" category and a letter opener to be under "General Office Supplies." Only Vikingop.com categorized the letter opener this way; while Staples.com placed it under "Scissors, Rulers, and Trimmers." Both Vikingop.com and Staples.com placed the Polaroid film three levels deep under "Presentation" materials (which stumped several users), while OfficeMax.com categorized it under Technology.
While our results showed that no site was really preferred over another (for the items tested), the following were noted as important features that contributed to their overall satisfaction of the site:
Preset Quantities – Staples.com offered preset quantities to many items, which saved the users one step when ordering a single unit of an item. In addition to increasing efficiency, this also seemed to make the user more aware of the quantity being ordered. On Vikingop.com, several users ordered 144 yellow legal pads rather than 12, not noticing that they were packaged in ‘dozen’ units.
Pictures of Items – finding the yellow legal pads was much easier when viewing the item thumbnail pictures than by reading the item descriptions. OfficeMax made the thumbnails readily apparent. Staples.com and Vikingop.com required users to click on a View Picture/Image button or link to see them. Interestingly, very few users took advantage of this option – they did not notice the button or link and searched through the item descriptions instead (most commented later that pictures would have made it easier). This is one example of weighing the pros and cons of optimal page design. With no images, a page downloads faster; however, if the user cannot quickly find the desired item, time (and perhaps a sale) is sacrificed.
Visual Feedback when Adding Items to Shopping Cart – Staples.com was the only site of the 3 that did not navigate the user to a shopping cart page when an item was added. While this feature can enhance efficiency to the experienced user, it proved to be problematic to the novice user. First-time users did not notice that the shopping cart summary on the right of the page was updated when an item was added. As a result, multiples of the same item were added to the cart until the user either explicitly went to the cart to see its contents, or eventually noticed the cart summary. In fact, several users did not realize how to view the cart contents until they were forced to find it to give the final cart total at the end of all tasks.
Grouping ‘Like’ Items under Different Categories – finding both of the Epson inkjet cartridges posed a challenge to the users of Vikingop.com, which categorized the black cartridge under "Epson Ink-jet Cartridges I" and the color cartridge under "Epson Ink-jet Cartridges II"! Participants had to guess which category might contain the cartridge they desired. (Since this test, Vikingop.com has changed the path to the cartridges to be more intuitive.)
Special Feature Items – occasionally, some items appeared on a page as a special deal or "Featured Item." It was surprising to us that most users did not choose to select a Featured Item and instead navigated further into the site to choose another instead. In the case of Vikingop.com, a significant portion of the home page was dedicated to advertisements for special offers and deals. None of the users we tested ever showed any interest in even looking at these ads for their desired item, thus, showing some support for "banner blindness" (Benway, 1998).
Figure 1. Vikingop.com home page.
In summary, participants were more efficient with Staples.com and Vikingop.com but were generally more successful in finding the prescribed items with OfficeMax.com. No single site was preferred or showed higher reported satisfaction over the others. Preset quantities, visible thumbnails of the products, and visual feedback when adding to the shopping cart were all cited as factors enhancing overall site satisfaction. Non-intuitive categorization of items (letter opener, film) caused some user-confusion in all three sites. Designers are encouraged to use activities, such as card sorting, to determine users’ categorization of items so that they can be matched in the interface.
Benway, J.P. (1998). Banner blindness: the irony of attention grabbing on the world wide web Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, USA, 1, 463-467.
Chi_India (2000). The Challenge: Staples.com. Retrieved 7/13/02: http://www.indiachi.com/case_studies.htm
Doll, W. J., Xia, W., & Torkzadeh, G. (1994). A confirmatory factor analysis of the end-user computing satisfaction instrument. MIS Quarterly, December, 453-461.
ErgobrowserTM, Ergosoft Laboratories © 2001.
Konrad, R. (2001). OfficeMax, Staples top list of e-tail survey. CNET News.com. Retrieved 7/13/02: http://news.com.com/2100-1017-254741.html?legacy=cnet