Summary: An Electronic Flight Bag (EFB) is a mobile computing device intended primarily for use in the cockpit for pre- and post-flight planning. Experiment 1 compared information retrieval times with an EFB and with paper documentation for trained pilots. Results indicate that information is retrieved slower when using the EFB but only when the user had to manipulate the information depicted on the screen via paging or scrolling. Experiment 2 compared information retrieval times for EFB and paper documentation in a real-world flight simulation at three workload levels. Results indicate no difference in retrieval time when the workload was easy but significantly slower retrieval times with the EFB at moderate or difficult workload.
An Electronic Flight Bag (EFB) is a personal computing device intended primarily for use in the cockpit for pre- and post-flight planning. This includes such tasks as: calculating aircraft performance, entering and updating dispatch information, and displaying flight planning documents all of which are traditionally found in volumes of paper binders that are carried onboard the airplane during operation. The EFB has also been proposed as an alternative for displaying instrument approach charts, which are primarily used during flight for navigation and decision-making during arrivals, approaches, and departures. This feature of the EFB makes the design and evaluation of EFBs crucial since they will be used primarily during the phases of flight associated with a majority of aviation accidents (Nagel, 1988).
Using the EFB to display instrument procedure charts offers two primary advantages. First, the quantity of paper documents currently required onboard aircraft consume valuable storage space and subtract from the available payload for flight operations. Successful integration of the EFB will eliminate the need for paper documents facilitating the realization of a “paperless cockpit.” Second, the EFB allows automatic updates of critical documents, which can be required as often as every 28 days, as is the case with the instrument navigation charts. Updating these charts on paper is a tedious and time consuming task that all too often results in misplaced or lost charts, a mistake typically not realized until the chart is needed.
Pilot demand for Electronic Flight Bags (EFB) is increasing and the Federal Aviation Administration (FAA) has responded to operator’s requests for guidance towards airworthiness certification with advisory circular AC120-76A (DOT, 2003). The advisory circular provides certification guidelines for those who wish to use an EFB in the cockpit. Other publications address usability considerations for the design and testing of EFBs (Chandra, 2003; Chandra & Mangold, 2000); however, these documents rely on existing research primarily derived from studies involving desktop computers.
Previous research investigating the usability of EFBs is limited and the results are somewhat misleading. Shamo, Dror, and Degani (1998) found that performance calculations and document retrieval were faster and more accurate with the EFB. A follow-up study (1999) found that searching documents using EFBs require significantly less workload than searching through paper documents. It should be noted however, that the device used in both of Shamo’s studies was unique and is not necessarily representative of the commercial devices currently available. Most notably the device used a 10.8 inch diagonal display screen, which is much larger than most off-the-shelf EFBs which use 6.8 – 8.2 inch diagonal screens. In addition, the device used a custom-made user interface. A study by Hamblin (2003) evaluated a commercially available EFB and software package (JeppView FlightDeck) and showed that EFBs with small display screens increased information retrieval times primarily due to the additional attention required to manipulate the device. This study investigates the effect of different screen manipulations on information retrieval performance (Experiment 1) and user workload (Experiment 2).
Ten instrument rated pilots (nine male and one female) ages 26 to 48 years (M = 35.5) participated in the study. Pilot experience ranged from 120 to 2000 total flight hours (M = 653.9 hours) and 40 to 180 instrument hours (M = 96.8 hours). All the participants had obtained their instrument pilot rating and four had obtained their commercial pilot certificate. All of the pilots’ ratings and certificates were current at the time of their participation.
The study was conducted using paper copies of instrument approach plates printed by Jeppesen as well as JeppView FlightDeck (JPFD) software loaded on a CT-1000 manufactured by Northstar Avionics. Interaction with the CT-1000 was done using either a one-inch square touch screen with a pen-type stylus or 30 multifunction keys located around the perimeter of the 6.4-inch diagonal screen. For this experiment, only the multifunction keys were used.
The pilots were asked to complete a consent form, privacy statement, and a demographic questionnaire. Each pilot was briefed on the format of the paper-based instrument navigation charts published by Jeppesen regardless of the pilot’s experience. The pilots were also given a tutorial explaining how to use the EFB and were given up to 60 minutes of practice to get acquainted with the EFB.
The pilots were required to find the answers to 60 explicit questions found on instrument navigation charts. The pilots answered 30 questions using charts displayed on the EFB and 30 questions using charts displayed on paper. The questions were matched between sets based upon the location of the answer and the relative density of the information displayed on the map, thus each question in the first set was similar to a question in the second set. In addition, because the EFB did not display the chart in its entirety, questions were designed so that 1/3 of the questions required scrolling to find the correct answer, 1/3 required paging, and 1/3 required no manipulation. The order of the chart presentation was counterbalanced so that half of the participants used the EFB first and half used the paper charts first. Information retrieval time for each question was recorded.
Response times were averaged across the 30 questions for each presentation mode. A 2×3 within-subjects analysis of variance (ANOVA) revealed a significant main effect for Display Type F (1, 10) = 8.029, p = .018. Participants were significantly slower to find the answers to questions when using the EFB (M = 22.20 sec) as compared to using the paper charts (M = 14.50 sec).
There was also a significant main effect for Manipulation Type, F (2,20) = 16.437, p <.001. A priori contrast analysis (t-tests) of information retrieval time revealed there was no significant difference between searches that required no manipulation of the EFB and paper charts (t = .294, p = .770) suggesting that the increase in time was not due to the resolution of the display but rather the paging and scrolling. Searches that required paging through information displayed on the EFB took an average of 11.35 seconds longer (p < .001) and searches that required scrolling through information displayed on the EFB took 7.37 seconds longer (p < .001) than the paper charts (see Figure 1).
Figure 1. Mean information retrieval time (IRT) for paper and Electronic Flight Bag device.
The results showed a significant increase in information retrieval time when using the EFB but only when the user had to manipulate the information depicted on the screen. This suggests that the user interface, originally designed for use on a desktop computer with a mouse and keyboard, is not optimized for the mobile device using multi-function buttons. As a result, the user must dedicate more time and attention to manipulating the EFB in order to extract the necessary information. Experiment 2 determines if this increase in attention increases perceived workload.
The data from the Experiment 1 was acquired with the participant sitting at a desk, and finding the correct answer was the lone objective, an obvious contrast to the complex task of flying an airplane. Previous research has demonstrated that task performance and workload are not monotonically related (Hart & Staveland, 1988). With this in mind, a second experiment was conducted to determine user performance given various levels of workload. We also wanted to know if using the EFB resulted in an increase in perceived workload.
The pilots performed six instrument approaches on a PI-135 PC-based aviation-training device (PCATD) manufactured by Elite Simulation Solutions. The PCATD was equipped with a control console, rudder pedals, throttle quadrant, functional navigation, and communication avionics, and dual 17” monitors, one for the pilot and one for the instructor. The pilot’s monitor displayed the aircraft’s instrument panel and view looking forward outside the cockpit, while the instructor’s monitor tracked the aircraft’s progress and position.
Ten instrument rated pilots from Experiment 1 flew six distinct instrument approaches, which started 17-20 miles from the destination airport. During the approach, pilots were required to manage the airplane’s systems, communicate with an air traffic controller, navigate the airplane per instructions, and respond to 15 explicit questions regarding the approach using the appropriate instrument approach plate (IAP). The questions were designed to force the participant to use all relevant parts of the IAP. The six approaches were divided into three levels of difficulty (difficult, moderate, easy,) by manipulating four variables: quantity of controller issued commands, cloud ceiling height, crosswind velocity, and systems failure. Each participant flew two approaches at each level of difficulty; one approach using paper-based copy of the approach plate and one approach using an approach plate displayed on the EFB. Technical demands of each approach were of equal difficulty. The approach plate order was systematically randomized between the paper-based approach plates and the EFB displayed plates.
Pilots were given their location and altitude of their starting point and were allowed to review the approach and set the navigation equipment as necessary prior to beginning. During each simulation, one experimenter monitored the approach from the instructor’s station, acted as the air traffic controller, and asked the test questions. A second experimenter recorded response times using a stopwatch and operated a video camera.
Information retrieval time was defined as the elapsed time from the completion of the verbalized question to the completion of the verbal response. In order to insure that all 15 questions were asked during each approach, pilots were given a maximum of 40 seconds to correctly respond to each question. If the pilot did not respond correctly in 40 seconds, a no-response was recorded.
Effects of Display Type
Information retrieval time was analyzed using a 2 (display) x 3 (workload) within-subjects analysis of variance (ANOVA) and was found to increase significantly when using the EFB as opposed to paper, F(1,9) = 16.892, p = .003. There was also a significant effect for workload as retrieval time increased, workload increased, F(2,18) = 5.663, p = .012 (see Figure 2). The was no significant interaction between display type and workload.
Figure 2. Mean information Retrieval time (IRT) for paper and Electronic Flight Bag by workload.
Effect of Manipulations
A 2 x 3 within-subjects ANOVA of EFB retrieval time revealed a marginal main effect of manipulation, F(1,9) = 4.395, p = .065 and a significant main effect for workload, F(2,18) = 6.483, p = .008. There was also a significant interaction between manipulation type and workload F(2,18) = 5.606, p = .013 (see Figure 3).
Figure 3. Mean information retrieval time (IRT) for paging and no-paging by workload.
Examination of error rates showed that participants committed more errors and more no-answers when using the EFB; however, these differences were not significantly different F(1,188) = .136, p = .713. Interestingly, average error rates when using paper remained constant across levels of workload, while average error rates when using the EFB increased as workload increased; however, again the differences were not significant F(2, 188) = 1.641, p = .197.
The results from both studies show that response times for finding information on paper instrument approach plates and electronic instrument approach plate displayed on an EFB are similar when manipulation is not required; however, the need to page or scroll for information significantly increased the amount of time required to find information. This is also most likely the reason for increases in perceived workload. Results of the second experiment show that the interaction with the EFB during levels of moderate and high workload levels significantly increased the information retrieval time making the retrieval of vital information more difficult at a time when pilots need it the most. The interaction between display type and workload indicates that there is no difference between response times when workload is easy; however, retrieval times are significantly effected by the need to manipulate the EFB as workload becomes moderate or difficult.
The results from both studies demonstrate that hardware and software found to be usable on a desktop computer is not necessarily appropriate for mobile devices, especially those used in high a workload, complex environment like a cockpit. The results highlight the need for the usability engineer to consider the task and environment that the hardware and software are to be used in. For example, a usability engineer might quickly decide that the correct solution is to simply use a device with a screen large enough to display the information in its entirety that will eliminate the need to page or scroll for information. With the emergence of tablet PCs, this is certainly a viable solution for a Boeing 777 that has a spacious cockpit; however, not a good solution for smaller business jets that leave little room for an additional 10-14 inch display.
Chandra, D. C. (2003). A tool for structured evaluation of electronic flight bag usability. In Proceedings of the 22nd Digital Avionics Systems Conference. 12-16 October 2003, Indianapolis, IN.
Chandra D. C., & Mangold S. J. (2000). Human factors considerations in the design and evaluation of electronic flight bags (EFBs) Version 1: Basic functions. (Report No. DOT-VNTSC-FAA-00-22). Cambridge, MA: USDOT Volpe Center.
Department of Transportation. (2003). Guidelines for the certification, airworthiness, and operational approval of electronic flight bag computing devices. DOT-FAA-AC120-76A. Washington, DC: Department of Transportation.
Hamblin, C.J. (2003). Electronic Flight Bags (EFBs) with small displays significantly increases information retrieval times. Proceeding of the 12th Annual Symposium on Aviation Psychology. Dayton, Ohio.
Hart, S.G., & Staveland,L.E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Hancock, Peter A. & Meshkati, Najmedin (Eds), Human mental workload. Advances in psychology, 52. (pp. 139-183). Oxford, England
Nagel, D.C. (1988). Human Error in Aviation Operations. In E.L. Wiener & D.C. Nagel (Eds.), Human Factors in Aviation (pp. 263-303). San Diego: Academic Press.
Shamo, M.K., Dror, R., & Degani, A. (1998). Evaluation of a new cockpit device: The integrated electronic information system. Proceedings of the Human Factors and Ergonomics Society Meeting 42nd Annual Meeting. Chicago, IL.
Shamo, M., Dror, R., & Degani, A. (1999). A Multi-Dimensional Evaluation Methodology for New Cockpit Systems. Proceedings of the Tenth International Symposium on Aviation Psychology. Columbus, OH.