By C. V. Lauterbach, A. Duvall & E. Pherigo
Summary. This study evaluated performance and subjective perceptions of three iPad pizza ordering apps (Pizza Hut, Papa John’s and Domino’s). Sixteen users (8 male, 8 female) performed a series of tasks on each app while measures for first impressions, number of taps, perceived difficulty, satisfaction, and preference were obtained. Results show that users reported Pizza Hut to be the most appealing app initially; however, it was reported to be the least preferred after usage. Users preferred the Papa John’s app overall and reported it to be the easiest to use.
Websites are not the only means for college aged individuals to navigate or browse the internet. In 2010 Apple introduced the iPad, which featured a multi-touch screen and allowed users to download apps ranging from video games to document creation tools. Currently, the iPad is selling at a rate of 4.5 million per quarter, making it the most popular tablet (Jacobsson-Purewal, 2010).
The most common uses for the iPad are reading, gaming, and surfing the internet (Nguyen & Chaparro, 2010). According to Kovach (2010), it is also used for entertainment purposes, such as watching videos or movies. College students enjoy the versatility, light-weight, quick response, and using it gives them the ability to refer to texts in class (Marimelli & Ringle, 2011).
There have been studies comparing the usability of iPad apps. One such study by Budiu and Neilson (2011) reported that not all apps allow for the same navigation and gesturing features. More specifically users were frustrated when they learned a gesture (e.g., swiping across the screen to go to the next page) on one app that did not carry over to a different app.
The purpose of this study was to investigate the usability of three iPad pizza ordering apps among college students.
Participants were 16 volunteers (8 male, 8 female; M Age = 21.81, SD=7.88, Range=18-50) from Wichita State University. They reported using the internet and various mobile devices for a variety of uses and at different times. Fifteen participants reported previous experience using a touch-screen cell phone and all users reported owning a mobile computing device (e.g., laptop, iPad, smartphone).
A first-generation Apple iPad with downloaded Pizza Hut, Papa John’s, and Domino’s apps was used in this study. Morae™ 3.1 was used to record number of taps and participant comments. The satisfaction survey was a modified version from the Satisfaction Usability Scale (SUS) (Brooke, 1996).
The study was conducted at the Software Usability Research Lab at Wichita State University. Participants were shown the opening screens for each of the three pizza ordering applications (see Figures 1-3) and asked to rank them based upon first impressions. Users were asked to complete a series of tasks with each of the apps:
- Add a pepperoni pizza to an order
- Add a cheese pizza (alter the order to customize toppings on the pizza)
- Order a variety of items using a $100 budget
- Delete an item from the order
After each task, users rated the perceived task difficulty on a 5-point Likert scale, (1=very difficult, 5=very easy) and completed a satisfaction survey pertaining to their interaction with that particular app. After completing all tasks, users were asked to rank their preference of the apps. The order of the apps and the tasks were counterbalanced to avoid order effects.
Figure 1. Pizza Hut Figure 2. Papa John’s Figure 3. Domino’s
The percent of successful attempts is shown in Table 1.
Table 1. Percent of users who successfully completed each task.
|Application||Add a Pizza||Alter an Order||Order Variety of Items||Delete Item|
Participants were the least successful with the task which asked them to add a cheese pizza to their cart and then change the order to customize its toppings. On Pizza Hut, users spent a significant amount of time trying to figure out how to add selected toppings to only a specific side of the pizza (see Figure 4). On the Domino’s app, users had difficulty initially choosing the cheese pizza. Domino’s offered many pizza types but a simple cheese pizza was not one of them.
Figure 4. Some users had difficulty moving toppings to one side using the Pizza Hut app. The proper method was to drag the topping picture to the picture of the pizza in the background. This was not intuitive.
Figure 5 shows the screen where users would adjust the toppings in the Papa John’s and Domino’s app.
Figure 5. Papa John’s (left) and Domino’s (right) used a similar and more intuitive graphic to customize toppings for a pizza.
After users completed each task they were asked to rate its perceived difficulty on a 5-point Likert scale, (1 = very easy, 5 = very difficult). Results from a one-way ANOVA showed that users reported the task of “Altering an Order” significantly more difficult on the Pizza Hut app than on the Papa John’s app F(2, 30) = 7.89, p < .01, η2 = .34 (Figure 6). In addition, it was found that deleting an item was more difficult on the Domino’s app than on both Papa John’s and Pizza Hut apps, F(2, 30) = 10.13, p < .01, η2 = .40. Figure 7 shows the screens where users could delete an item from their order. Users reported the red circular button used in the Papa John’s app was more intuitive than the “swipe to delete” method required by Domino’s. Figure 7 shows the screens where users could delete an item from their order. Users reported the red circular button used in the Papa John’s app was more intuitive than the “swipe to delete” method required by Domino’s.
Figure 6. Mean difficulty rating per task on each application (1=Very Easy, 5 =Very Difficult). The higher the bar, the more difficult the task.
Figure 7. Pizza Hut (left), Papa John’s (center) and Domino’s (right) had different methods of deleting an item. Pizza Hut and Papa John’s required tapping the red circular button to the left of the item. Domino’s required swiping above the item to display a delete button.
Number of Taps
A measure of task efficiency was determined by subtracting the user’s number of finger taps from an optimal number of finger taps based on the correct path (see Table 2 for optimal path values). Results from a one-way ANOVA showed a statistically significant difference across applications for the “Alter an Order” task F(2, 28) = 9.82, p < .01, η2 = .41 (see Figure 8). Post-hoc analyses showed that users used a higher number of taps for this task with all of the apps but most notably with the Pizza Hut app than both Papa John’s and Domino’s. The number of taps also significantly differed across applications for the “Delete an Item” task F(2, 28) = 5.93, p < .01, η2 = .30. Post-hoc analyses revealed that use of the Domino’s app resulted in more extra taps than Pizza Hut and Papa John’s.
Table 2. Number of taps for each task by application.
|Application||Select a Pizza||Alter an Order||Delete Item|
Figure 8. Average deviation from the optimal path for number of taps by application.
Once users had completed all tasks on an application they were asked to complete a modified version of the SUS (Brooke, 1986). Results from the SUS yielded a score between 0-100, with higher scores indicating higher user satisfaction. Results from a one-way repeated measures ANOVA indicated no significant differences in satisfaction scores across the three applications F(2, 30) =2.15, p > .05, η2 = .13.
Users were asked to rank their preference for the three pizza apps before they used them. A Friedman test indicated a significant difference in preference, χ2 (2, N = 16) = 19.63, p < .001 (see Figure 10). Post-hoc analysis showed that the initial preference for Pizza Hut was higher than Papa John’s and Domino’s (p < .01). Users liked Pizza Hut’s high definition graphics and maps of store locations. Many reported that this gave it a professional appearance.
Figure 10. First impressions ranking for each application.
After completion of tasks, users were asked to rank the applications for preference. Results from a Friedman test revealed a significant difference, χ2 (2, N = 16) = 11.38, p < .01 (see Figure 11). Post-hoc analysis showed that the Papa John’s app was significantly more preferred than both the Pizza Hut and Domino’s applications, p < .05. Users said that the Papa John’s app was quick and easy to use. They also indicated that they liked being able to see images of the items before they were added to their order.
Figure 11. Final ranking for each application.
Usability Issues: Pizza Hut
Gestures. Users had difficulty altering the toppings on their pizzas. Many did not read the instruction and found the gesture to move the toppings not very intuitive (see Figure 4).
Navigation. Users expressed that there were a lot of screens to navigate to complete a task. Each time the users completed a task they had to select whether it was a carryout, select a store location, and order an item. The app did not remember the store location from the home screen unlike Papa John’s and Domino’s.
Crashes. The Pizza Hut app frequently crashed, with 75% of the users experiencing at least one crash during their interaction. When users switched between item menus, users also noticed a slight delay in loading the screen.
Usability Issues: Papa John’s
Pizza Cart. Users had to go to the cart button at the bottom of the home page each time they wanted to check their order (see Figure 12). For Pizza Hut and Domino’s the cart would automatically appear after each item was selected. This was especially bothersome to users when they were completing the task to order $100 worth of items. Users commented that this caused them to lose track of where they were in the ordering process.
Figure 12. Users had to tap the cart button every time they wanted to check their order total.
Usability Issues: Domino’s
Item Names. Users reported difficulty in locating a simple pizza. Unlike Pizza Hut and Papa John’s, Domino’s gives unique names to their pizzas (i.e., “the ultimate pepperoni feast” instead of a pepperoni pizza) (see Figure 13). Although Papa John’s had specialty pizzas these were placed in a separate menu and users were also able to select pizzas that had more common names.
Altering an Order. Users spent more time locating toppings in the menus under the Domino’s app than they did using the Papa John’s app. Many users commented that all of the toppings should be listed under one menu (i.e., meats and vegetables in one menu) (see Figure 13).
Figure 13. Users commented that they would like to have all toppings under one menu instead of 4 separate menus.
Deleting Items. Users did not notice the instructions to swipe across the item to delete it, unlike the Papa John’s and Pizza Hut apps that had a button that allowed for quick deletion of an item.
The study revealed two important design issues of iPad apps that impact usability:
Gestures. The users in this study had difficulty with tasks which required hidden gestures (e.g., to delete an item in the Domino’s cart or to place toppings on half of a pizza). This was due to the fact that each app did not use gestures consistently throughout the interface and written instructions informing users of the gesture went unnoticed. On the Pizza Hut app, the graphic of the toppings and pizza halves did not afford drag and drop to the end users. On the Domino’s app, users had no expectation that a swipe gesture would reveal a delete action button since no other swipe gestures were used in the rest of the app. These results closely resemble those by Budiu and Neilson (2011) in that users felt some frustration in the lack of consistency in gesturing features across the apps.
Impressive Graphics. Impressive graphical design was shown to influence first impressions in this study, but ultimately did not overshadow poor usability. The Pizza Hut app graphical design was reported to be “slick” and more professional but also was reported to impact response time and user performance with the app.
Brooke, J. (1996). SUS: A quick and dirty usability scale, in P. Jordan, B. Thomas, B. Weerdmeester, & I. L. McClelland (eds.), Usability evaluation in industry, (pp. 189-94). London, UK: Taylor & Francis. Retrieved from http://usabilitynet.org/trump/documents/Suschapt.doc
Budiu, R., & Nielson, J. (2011). Usability of iPad apps and websites. 2nd ed. Nielson-Norman Group, (1-134). Freemont, Ca. Retrieved from http://www.nngroup.com/reports/mobile/ipad.
Jacobsson-Purewal, S. (2010, October 5). iPad, fastest-selling electronic device…ever [Web log message]. Retrieved from http://www.pcworld.com/article/206953/iPad_fastestselling_electronic_deviceever.html
Kovach, S. (2010, December 10). The Most Popular Free iPad Apps of 2010. Business Insider. Retrieved from http://www.businessinsider.com/the-most-popular-free-ipad-apps-of-2010-2010-12
Marmarelli, T. & Ringle, M. (2011). The Reed College iPad study. The Reed Institute. Retrieved from http://184.108.40.206/cis/about/ipad_pilot/Reed_ipad_report.pdf
Nguyen, B. & Chaparro, B. (2010). Survey results: iPad is best for reading, communicating, and gaming. Usability News, 12(2). Retrieved from http://usabilitynews.org/ill-take-a-large-pepperoni-with-a-side-of-usability-ordering-pizza-on-an-ipad/