NASA's Air Transportation Safety Survey: Light, Heat, and Fog

Regular readers of this blog know that NASA is not my favorite government agency. Once upon a time in the 1960s, it had a clear mission, attracted some of the world's best professionals, and landed men on the moon. But since then the organization has swayed between focused, clear projects (space telescopes such as Hubble come to mind), and disasters ranging from the tragic (the Challenger and Columbia disasters) to the merely expensive (I could cite numerous space-probe projects that went awry here). The disasters have won a place of prominence for NASA in most engineering ethics textbooks, which usually use the Challenger disaster as an example of how bad management can kill people.

Well, it is my sad duty to comment on yet another episode of what looks like a good idea gone awry due to internal conflicts, bickering, and bad management inside NASA, plus possibly a little help from the media. After NASA started to implement what promised to be a great idea about how to improve airline safety (there's the light), the agency got in a tussle with freedom-of-information-act requestors, NASA head Michael Griffin intervened and took hostile questions from Congress and other agencies (there's the heat), and finally released the data in a close-to-unusable form (there's the fog).

First, the light. Everybody familiar with engineering ethics problems knows that for every major disaster (a bridge that actually falls down, a spaceship that crashes), there are dozens to hundreds of lesser problems and issues that, if noticed and properly acted upon, can serve as warnings about some truly major problems that can then be prevented. Knowing this, some clever people at NASA and outside it (notably a questionnaire expert at Stanford named John Krosnick) organized a big telephone survey of thousands of airline pilots and did interviews from 2001 to 2004, asking them about potentially hazardous incidents that they had personal experience with. This was called the National Aviation Operation Monitoring System.

The normal way that the Federal Aviation Administration (an agency separate from NASA) finds out about near-misses and so on is when pilots file reports on them. Apparently there are rules about when a pilot is supposed to report a near accident, but if pilots are human (most of them are, anyway), they probably don't always follow such rules. If other pilots are involved, the whole process smacks somewhat of ratting on one's colleague, and I suppose there is no reward for reporting these things other than the knowledge that you're following the rules. Anyway, to my knowledge, that is the only current mechanism for detecting incidents that might tell us about dangerous trends having to do with new equipment or procedures, for instance, that might lead to serious accidents in the future.

The NASA-sponsored survey project was an advance on this method. It didn't just wait for pilots to report incidents—it went out and asked about them in random phone surveys. In the nature of things, this kind of survey will turn up more data than one that relies upon the pilot's initiative to write up and submit a report. But there are ways of calibrating out that difference and arriving at something close to the truth, if the survey is checked by other means and completed under the supervision of qualified experts such as Prof. Krosnick.

Well, that didn't happen. Or if it did, we don't know about it yet. Evidently, when the numbers of incidents reported by pilots through the phone survey turned out to be a lot higher than the numbers the FAA was getting, some news media people got wind of the information and submitted requests for it under the Freedom of Information Act. Now if I were in NASA's shoes, this might give me some pause, admittedly. It takes a certain amount of time to process and analyze data, but it seems like with computer-aided methods, a year or two should be enough for the survey investigators to write up and issue a report. No report was issued. Why is not clear, except that NASA is quoted as saying it didn't want to harm the airline industry. Well, fine, but crashes harm the airline industry too, and if this data can be used to improve the already good airline safety record further, it's a shame that NASA has sat on it so long.

In congressional hearings about the matter held last October, NASA head Griffin promised to release some data from the project by the end of the year. He kept the letter of his promise, anyway, by posting a 16,000-page .pdf document somewhere on NASA's website on Dec. 31, 2007. A number of indications show that NASA was not especially eager for people to do anything with this data.

For one thing, the news release announcing the document said it was to be found at NASA's website, "http://www.nasa.gov." For anyone familiar with NASA's huge and almost Byzantine website, that's like saying "It's in Arkansas." Your scribe spent fifteen minutes looking for it there and with Google, without success. This is not to say it's not there—the Associated Press people found it, but they're paid to do things like that. A search with NASA's own website search engine under "National Aviation Operations Monitoring System" done while I was looking at that very phrase in one of their own news releases on their own website—turned up zero results. Go figure.

What I figure is what many news outlets have concluded: for some reason, possibly the one NASA stated about fear of scaring customers away from airlines, they are reluctant to make these results public or useful in any meaningful way that could actually serve the original purpose of the survey, which was to come up with a better way of catching potential airline accidents before they become real ones. So we have a situation where $11 million of the taxpayer's money has been spent on a media flap and a release of data in a form that one of the survey's own designers—Prof. Krosnick—says is intentionally designed to mislead anyone who tries to use it.

After one of the old movie comedy team Laurel and Hardy's epic screwups involving ropes, stairs, ladders, cream pies, a piano, and a goat, Oliver Hardy would turn to Stan Laurel and say, "Well, Stanley, this is a fi-i-i-ne mess!" That about covers this latest NASA episode. The best thing I can say about it is that nobody got killed, although if it had been done better, we might have been able to prevent some fatalities in the future.

Sources: Two news reports on the NASA data release are at the Houston Chronicle website http://www.chron.com/disp/story.mpl/front/5414060.html and the Chicago Tribune website http://www.chicagotribune.com/news/local/sns-ap-air-safety-secrets,0,3362253.story. NASA's own news release announcing the data is at http://www.nasa.gov/home/hqnews/2007/dec/HQ_M07191_NAOMS_advisory.html. If a sharp-eyed or patient reader locates the actual URL where the NASA survey data is available, I would appreciate it if you could send it to me so I could mention it in a revised blog.