ABSTRACT

Altitude deviations are the most common incident reported to the Aviation Safety Reporting System. Altitude deviations are reported to the ASRS at the rate of about one per hour. The reporting pilot's narrative is usually the only source of information about what happened in these incidents. In a line oriented simulator study, twenty-two airline crews flew a realistic two hour mission in DC-9 and MD-88 aircraft. This paper will describe and analyze, two of the five altitude deviations that were observed during this study. The paper focuses on the flight crews' use of the autopilot and autothrottle during these incidents and explores some of the reasons why "automation surprises" occurred to these airline crews in situations where the automation worked exactly as designed.

EXPECTATIONS, MENTAL MODELS AND AUTOMATION SURPRISES

In learning to operate a system, operators form expectations about how the system normally behaves in response to control inputs and to environmental disturbances. The collection of these expectations of system behavior is an important component of the operator's "mental model" of the system. Accurate expectations of system behavior allow quicker and more fluent performance. Operators can use observed deviations from expected behavior as a way to detect their own errors or system malfunctions. On the negative side, if the expected behavior "always" occurs, experienced operators may be less likely to check that the expected behavior did in fact occur.

An "automation surprise" occurs when the automation behaves in a manner that is different from what the operator is expecting. In both of the following cases, a reasonable sequence of pilot actions performed in a high workload situation resulted in an unusual and undesirable result - an automation surprise. Sometimes surprising behavior is due to a system malfunction but in these two cases the automation worked as designed.

This paper will examine in detail the automation surprises that occurred on two simulated MD-88 flights that were part of a line oriented simulator study of cockpit automation (Wiener, et al, 1991). These incident descriptions were reconstructed from data from two cockpit video cameras. One camera provided a wide angle view of the cockpit and the other provided a close up view of the altimeter and the Flight Mode Annunciator (FMA) display (see Fig. 1).

Case 1: Narrative Description.

The crew had just attempted a Category II approach; did not see the runway at the decision height and made a missed approach. After climbing to 2100, Air Traffic Control told them, "You're cleared to the 0-0-6 radial at 1-5 miles, climb now and maintain 5,000 feet, let me know when you're ready to copy holding instructions." Their airspeed was 160 knots and the flaps were set to 15 degrees. The altitude window on the Mode Control Panel (MCP) was set to 5000 and the Captain, the pilot flying (PF), pushed first the EPR/LIM button on the MCP and then selected a vertical speed of about 2000 feet per minute. The Flight Mode Annunciator (FMA) now showed the thrust mode to be EPR/GA, altitude capture armed and the pitch mode to be VERT/SPD (Fig. 3.D). The aircraft was

climbing at a vertical speed of 2,000 feet a minute and accelerating rapidly because the engines were set to go around thrust. As the aircraft approached 4000 feet, the airspeed was about 240 knots, and the aural voice warning "STABILIZER MOTION" began to repeatedly sound[1]. The Captain mistakenly interpreted the "STABILIZER MOTION" warning as a malfunction - a runaway stabilizer. The Captain changed the pitch mode to IAS (Fig. 3.F) and disconnected the autothrottles and reduced power (Fig. 3.G). The aircraft now had insufficient thrust to maintain 240 knots with flaps at 15 degrees and also climb. The autopilot pitched the aircraft down to maintain 240 knots and the aircraft began to descend before reaching the cleared altitude of 5000 feet. The descent away from the 5000 foot target altitude triggered the "ALTITUDE - ALTITUDE" voice warning. As the aircraft started to descend, the First Officer (FO) said, "You're descending now." After descending to approximately 4000 feet, the Captain pushed the MCP button ALT/HOLD and remarked, "Did we ever ... we never got a capture on that." During the climb the crew had also received holding instructions for an unpublished holding pattern on the 0-0-6 radial and the FO had programmed the new fix and the holding pattern into the Flight Management System. The remainder of the flight was uneventful except that at about 20,000 feet the crew changed the Thrust Reference Panel from GO-AROUND to CLIMB thrust. Figure 2 shows the altitude profiles for the two cases.

Case 2: Narrative Description.

The second incident takes less than 20 seconds to play out. The crew had just made a missed approach and had climbed to and leveled at 2,100 feet. They received the clearance to "... climb now and maintain 5,000 feet ..." After some communication confusion on the cleared altitude (5,000 or 15,000) and which radial to hold on (0-0-6 or 0-6-0), the Captain set the MCP altitude window to 5,000 feet, set the autopilot pitch mode to vertical speed with a value of approximately 2,000 ft per minute and the autothrottle to SPD mode with a value of 256 knots (Fig. 4.D) . Climbing through 3,500 feet the Captain called for flaps up and at 4,000 feet he called for slats retract. As the aircraft climbed from 4,000 to 5,000 feet, the first officer was copying the holding clearance. Climbing through 4,000 feet, the Flight Mode Annunciator (FMA) showed: [SPD/255 || ALT || VOR/TRK || VERT/SPD] (Fig. 4.E). Passing through 4000 feet, the Captain pushed the IAS button on the MCP. The pitch mode became IAS and the autothrottles went to CLAMP mode (Fig. 4.F). Altitude capture was still armed. Three seconds later the autopilot automatically switched to altitude capture mode. The FMA arm window went blank and the pitch window showed ALT/CAP (Fig. 4.G). A tenth of a second later, the Captain adjusted the vertical speed wheel to a value of about 4000 feet a minute. This caused the pitch autopilot mode to switch from altitude capture (ALT/CAP) to vertical speed (Fig. 4.H). Climbing through 4500 feet, the Flight Mode Annunciator (FMA) showed: [SPD/ATL || blank || VOR/TRK || VERT/SPD] and the approaching altitude light was on. As the altitude passed through 5,000 feet at a vertical velocity of about 4,000 feet per minute, the Captain remarked, "Five thousand. Oops, It didn't arm." He pushed the MCP ALT/HOLD button and switched off the autothrottle. The aircraft continued to climb to about 5,500 feet and the "ALTITUDE - ALTITUDE" voice warning sounded repeatedly.

What Saved The Day?

Before attempting to describe what went wrong in these two cockpits it is interesting to ask as Degani, Chappell & Hayes (1991) did , "What saved the day?" The aviation system is very much an error tolerant system (Palmer, et al, 1993) with almost all errors being rapidly detected and corrected. In the first case, the FO quickly detected that the aircraft was descending. In the second case, the Captain immediately detected that the aircraft did not level at 5000 feet. In both cases, the altitude alerter also sounded. The Captains both recovered from the errors by selecting altitude hold on the MCP. Woods ,et al, (1994) have observed that in many incidents involving automation that the error is first detected not by autoflight displays such as the Flight Mode Annunciators that tell the state of the automation but by the basic aircraft displays such as the altimeter and the vertical speed indicator. This happened in both of these cases. The crews were apparently aware of the state of the aircraft but not aware of the state of the automation. The errors were detected by observing the unexpected state of the basic aircraft displays, not the automation display.

Case 1: What Went Wrong?

The immediate cause of the aircraft descending before reaching the cleared altitude of 5000 feet can be attributed to two actions taken by the Captain: 1) his selection the IAS pitch autopilot mode, and 2) his decision to disconnect the autothrottle and reduce thrust. In IAS pitch mode, the autopilot adjusts the aircraft's pitch angle in order to maintain the target airspeed set in the MCP speed window. Usually this pitch mode is used during climb with the thrust set to climb thrust, or during descent with the thrust at idle. In this case, the thrust was not sufficient to climb and maintain the MCP target speed of 240 knots and the aircraft started to descend.

A little detective work showed that the root cause of this incident actually occurred earlier, when the Captain configured the autoflight system for the climb from 2100 to 5000 feet. The Captain chose to climb with throttles at EPR/LIM thrust and with the pitch autopilot maintaining a climb rate of 2000 feet per minute in VERT/SPD mode. Climbing in this combination of thrust and pitch modes requires close monitoring because no speed target is specified; the airspeed is unconstrained, and the pilot must intervene when the aircraft has accelerated to the desired speed. In this case, the Captain selected the IAS pitch mode when the airspeed reached 240 knots. If the Captain had not then disconnected the autothrottle in response to the "STABILIZER MOTION" warning, the aircraft would have continued to climb and then leveled off as desired at 5000 feet.

The problem occurred when the Captain selected the initial autothrottle mode. He pushed the EPR/LIM button on the MCP in order to have the autothrottles command climb limit thrust. He apparently failed to notice, on the Flight Mode Annunciators, that the autothrottle was commanding go-around (EPR/GA) thrust instead of climb (EPR/CL) thrust[2]. Normally during a climb when the EPR/LIM button is pushed, climb thrust would be commanded by the autothrottles; Simply pushing the EPR/LIM button usually gives the desired thrust setting. However, consistently correct use of the EPR/LIM button requires that the pilot check to see what mode the Thrust Reference Panel is set to; before pushing the EPR/LIM button, and that after pushing the EPR/LIM button that the thrust Flight Mode Annunciator is checked to insure that the desired thrust mode was achieved.

Edwin Hutchins (1993) has refered to buttons such as EPR/LIM as a control with "meta-meaning." The EPR/LIM button has the meta-meaning, "set the reference thrust to the value selected on the Thrust Reference Panel". In this case, pushing the EPR/LIM button resulted in the incorrect thrust reference being set for the autothrottles (EPR/GA instead of EPR/CL). If the design had required the crew to explicitly pick the thrust limit or if the label on the MCP button dynamically changed to display the currently selected thrust i.e., EPR/GA instead of EPR/CL, it seems less likely that they would have selected go-around thrust in this situation. It is certainly possible to do the task correctly with the EPR/LIM meta-button but consistently correct task performance requires that the pilot check the Thrust Reference Panel before pushing the EPR/LIM button and then check the FMA to confirm that the desired mode and target occurred.

A second design factor may have contributed to this incident was the location of the relevant autoflight information. The EPR/LIM button is on the MCP which is located on the glare shield. The Thrust Reference Panel is located on the center instrument panel near the engine instruments. After pushing the EPR/LIM button, the pilot should check the FMA display to insure that the correct mode has been selected. The FMA display is located near the primary flight displays near the altimeter. Either of these two checking tasks could have prevented this error, but performing each task requires the pilot to look at a different display. The use of a button with meta-meaning requires the pilot to check other displays to insure correct task performance. A design that collocated the desired thrust reference and the system response would make it less likely that the pilot would select the wrong thrust reference^.

CASE 2: What Went Wrong?

In the second case, an automatic mode transition leads to a mode error. This is such a common problem that it has been given a name, a "kill-the-capture" bust. Hundreds of similar altitude deviations have been reported to the ASRS. The only thing unique about this incident is that it occurred during a full-mission simulator study and was recorded for later analysis.

In the MD-88 vertical speed is the basic pitch autopilot mode; pushing the MCP VERT/SPD button or adjusting the vertical speed wheel causes the pitch autopilot to switch to the vertical speed pitch mode. Two different mode configurations occur during an autoflight climb to a new target altitude. When a higher target altitude is entered in the MCP altitude window, altitude capture becomes armed. This is indicated by the letters, "ALT", appearing in the arm column of the FMA. (see fig. 4.B) The pilot next selects a pitch autopilot mode and thrust autothrottle mode for the climb; typically the pitch mode selected is either vertical speed (VERT/SPD) or IAS. During the climb, the autopilot continually checks the difference between the MCP target altitude and current altitude and the vertical velocity to see if the conditions are satisfied to switch to the altitude capture (ALT/CAP) pitch mode. When the autopilot automatically (and silently) switches to ALT/CAP mode, the autopilot smoothly pitches the aircraft to level out at the target altitude. When the altitude is acquired, the pitch mode switches to altitude hold (ALT/HLD). When the pitch mode switches to ALT/CAP, the letters, "ALT", disappear from the ARM mode annunciator and altitude capture is no longer armed (see Fig. 4.G). Figure 5 illustrates how the autopilot pitch and altitude capture arm modes interact^.

The autopilot pitch mode can be changed during a climb to a target altitude without disarming the altitude capture. The system is primed for error only after the pitch autopilot automatically switches to the altitude capture (ALT/CAP) mode. Adjusting the vertical speed wheel causes the pitch autopilot mode to change from its present mode - in this case, ALT/CAP - to vertical speed (VERT/SPD). Since altitude capture is no longer armed, the capture has been "killed," and if the pilot takes no further action the autoflight system is configured to climb the aircraft through the target altitude set in the altitude window[3].

This is a classic example of a mode error. Mode errors occur when the same pilot action - in this case, adjusting the vertical speed control wheel - results in different system behavior depending on what mode the machine is in. During the climb while altitude capture is armed, the pilot can change the pitch autopilot modes at will and altitude capture will remain armed and the autopilot will level the aircraft at the set altitude. However, once the autopilot has transitioned to the altitude capture pitch mode (ALT/CAP), adjusting the vertical speed wheel will cause the autopilot to switch to the vertical speed mode and the aircraft will no longer level off at the MCP target altitude. Adjusting the vertical speed wheel always puts the pitch autopilot mode in the vertical speed mode.

There are two different mode configurations during an automatic altitude change. In the first configuration, altitude capture is armed; in the second, altitude capture in no longer armed and the pitch mode is altitude capture (ALT/CAP). While the autoflight system is in the first configuration, it is difficult to disarm the altitude capture; During the second, it is all to easy to "kill-the-capture." The mode change that causes the problem in this type of altitude deviation incident is the altitude capture arm mode.

This is what some have called a problem due to a "silent" mode transition. In column two, the ARM annunciation - ALT - disappeared. In column four, the PITCH mode changed from ALT/CAP to VERT/SPD. But as can be seen by comparing figure 4.E with figure 4.H, after the pilot had adjusted the vertical speed, the only difference in the FMA is the absence of the letters - "ALT" - in the second column of the FMA display. If the pilot had checked the FMA and observed that the letters - "ALT" were absent, he could have inferred that the automation was set up to fly the aircraft through the 5,000 foot target altitude; an error likely sequence of observations and inferences.

DISCUSSION

These two altitude incidents illustrate some of the problems that humans can have in interacting with automation to control dynamic systems. Woods, et. al., (1994) have coined the term, "automation surprise", to describe this type of incident. An "automation surprise" occurs when the automation behaves in a manner that is different from what the operator is expecting. From the two Captains remarks when they detected the autoflight problems, "Did we ever .. we never got a capture on that" and "Five Thousand, Oops, It didn't arm", we can reasonably infer that the pilots were surprised. In both of these altitude incidents, the automation worked correctly but it did not respond in the "usual" expected manner. In the first case, pushing EPR/LIM usually results in CLIMB power. In the second case, adjusting the vertical speed usually just adjusts the vertical speed, it does not usually kill-the-capture. In both cases, the automation display, the Flight Mode Annunciator, indicated what was actually happening; however, the immediate response of the aircraft and the primary aircraft instruments was normal. The unusual and unexpected aircraft behavior occurred later. In case 1, rapid acceleration leading to the stabilizer in motion warning; and in case 2, not capturing the target altitude in the MCP altitude window. Woods, et al., (1994) have observed that most errors that result from the use of automation are detected by observing the system response and not the automation mode display[4].

Why might this be the case? What makes it difficult to use the information on the FMA to verify the correct autoflight mode? The cases described in this paper suggest a number of possible reasons why this might be happening. First, the FMA must be read and its meaning interpreted. Sometime what must be "read" and interpreted is the absence of information. Second, the FMA's physically location away from the MCP require that the pilot act in one place and check the outcome of the action in another place. Finally, the FMA does not provide a direct display of what the pilot needs to know to stay ahead of the aircraft - i.e., What trajectory have I set up the automation to fly the aircraft on?

CONCLUSIONS

The purpose of this paper has been to demonstrate in detail how "automation surprises" occurred to two airline crews during a high workload simulated flight segment. A lesson of these two incidents for pilots is to reinforce the importance of monitoring the automation displays - the Flight Mode Annunciators. In these incidents, they provided the only timely and correct indication of what the automation was actually doing. A lesson for autoflight designers should be to develop designs that either eliminate or provide more salient displays of normal but unusual automation modes. What is needed is a "what you see is what you will get" (WYSIWYWG) type of display of the of the aircraft's predicted vertical path.

REFERENCES

Degani, A., Chappell, S. L., and Hayes, M. S. (. (1991). What saved the day: A comparison of traditional and glass cockpits. In R. S. Jensen (Ed.), Proceeding of the Sixth International Symposium on Aviation Psychology Conference (pp. 227-234). Columbus, OH: The Ohio State University.

Hutchins Edwin, Personal communication, 1993.

Palmer Everett, Hutchins Edwin, Ritter Richard & vanCleemput Inge, Altitude Deviations: Breakdowns of a Error Tolerant System, NASA TM 108788 (1993).

Wiener, E. L., Chidester, T. R., Kanki, B. G., Palmer, E. A., Curry, R. E., & Gregorich, S. E. (1991). The Impact of Cockpit Automation on Crew Coordination and Communication: I. Overview, LOFT Evaluations, Error Severity, Questionnaire Data, NASA Contractor Report No. 177587 (1991).

Woods David D, Johannesen Leila J, Cook Richard I, Sarter Nadine B. Behind human error: cognitive systems, computers, and hindsight. Wright Patterson Air Force Base, Dayton, OH: CSERIAC, (1994).