Monday, January 07, 2008

NASA's Air Transportation Safety Survey: Light, Heat, and Fog

Regular readers of this blog know that NASA is not my favorite government agency. Once upon a time in the 1960s, it had a clear mission, attracted some of the world's best professionals, and landed men on the moon. But since then the organization has swayed between focused, clear projects (space telescopes such as Hubble come to mind), and disasters ranging from the tragic (the Challenger and Columbia disasters) to the merely expensive (I could cite numerous space-probe projects that went awry here). The disasters have won a place of prominence for NASA in most engineering ethics textbooks, which usually use the Challenger disaster as an example of how bad management can kill people.

Well, it is my sad duty to comment on yet another episode of what looks like a good idea gone awry due to internal conflicts, bickering, and bad management inside NASA, plus possibly a little help from the media. After NASA started to implement what promised to be a great idea about how to improve airline safety (there's the light), the agency got in a tussle with freedom-of-information-act requestors, NASA head Michael Griffin intervened and took hostile questions from Congress and other agencies (there's the heat), and finally released the data in a close-to-unusable form (there's the fog).

First, the light. Everybody familiar with engineering ethics problems knows that for every major disaster (a bridge that actually falls down, a spaceship that crashes), there are dozens to hundreds of lesser problems and issues that, if noticed and properly acted upon, can serve as warnings about some truly major problems that can then be prevented. Knowing this, some clever people at NASA and outside it (notably a questionnaire expert at Stanford named John Krosnick) organized a big telephone survey of thousands of airline pilots and did interviews from 2001 to 2004, asking them about potentially hazardous incidents that they had personal experience with. This was called the National Aviation Operation Monitoring System.

The normal way that the Federal Aviation Administration (an agency separate from NASA) finds out about near-misses and so on is when pilots file reports on them. Apparently there are rules about when a pilot is supposed to report a near accident, but if pilots are human (most of them are, anyway), they probably don't always follow such rules. If other pilots are involved, the whole process smacks somewhat of ratting on one's colleague, and I suppose there is no reward for reporting these things other than the knowledge that you're following the rules. Anyway, to my knowledge, that is the only current mechanism for detecting incidents that might tell us about dangerous trends having to do with new equipment or procedures, for instance, that might lead to serious accidents in the future.

The NASA-sponsored survey project was an advance on this method. It didn't just wait for pilots to report incidents—it went out and asked about them in random phone surveys. In the nature of things, this kind of survey will turn up more data than one that relies upon the pilot's initiative to write up and submit a report. But there are ways of calibrating out that difference and arriving at something close to the truth, if the survey is checked by other means and completed under the supervision of qualified experts such as Prof. Krosnick.

Well, that didn't happen. Or if it did, we don't know about it yet. Evidently, when the numbers of incidents reported by pilots through the phone survey turned out to be a lot higher than the numbers the FAA was getting, some news media people got wind of the information and submitted requests for it under the Freedom of Information Act. Now if I were in NASA's shoes, this might give me some pause, admittedly. It takes a certain amount of time to process and analyze data, but it seems like with computer-aided methods, a year or two should be enough for the survey investigators to write up and issue a report. No report was issued. Why is not clear, except that NASA is quoted as saying it didn't want to harm the airline industry. Well, fine, but crashes harm the airline industry too, and if this data can be used to improve the already good airline safety record further, it's a shame that NASA has sat on it so long.

In congressional hearings about the matter held last October, NASA head Griffin promised to release some data from the project by the end of the year. He kept the letter of his promise, anyway, by posting a 16,000-page .pdf document somewhere on NASA's website on Dec. 31, 2007. A number of indications show that NASA was not especially eager for people to do anything with this data.

For one thing, the news release announcing the document said it was to be found at NASA's website, "" For anyone familiar with NASA's huge and almost Byzantine website, that's like saying "It's in Arkansas." Your scribe spent fifteen minutes looking for it there and with Google, without success. This is not to say it's not there—the Associated Press people found it, but they're paid to do things like that. A search with NASA's own website search engine under "National Aviation Operations Monitoring System" done while I was looking at that very phrase in one of their own news releases on their own website—turned up zero results. Go figure.

What I figure is what many news outlets have concluded: for some reason, possibly the one NASA stated about fear of scaring customers away from airlines, they are reluctant to make these results public or useful in any meaningful way that could actually serve the original purpose of the survey, which was to come up with a better way of catching potential airline accidents before they become real ones. So we have a situation where $11 million of the taxpayer's money has been spent on a media flap and a release of data in a form that one of the survey's own designers—Prof. Krosnick—says is intentionally designed to mislead anyone who tries to use it.

After one of the old movie comedy team Laurel and Hardy's epic screwups involving ropes, stairs, ladders, cream pies, a piano, and a goat, Oliver Hardy would turn to Stan Laurel and say, "Well, Stanley, this is a fi-i-i-ne mess!" That about covers this latest NASA episode. The best thing I can say about it is that nobody got killed, although if it had been done better, we might have been able to prevent some fatalities in the future.

Sources: Two news reports on the NASA data release are at the Houston Chronicle website and the Chicago Tribune website,0,3362253.story. NASA's own news release announcing the data is at If a sharp-eyed or patient reader locates the actual URL where the NASA survey data is available, I would appreciate it if you could send it to me so I could mention it in a revised blog.

1 comment:

    January 2, 2008

    Tom Sullivan, “Quiet Rockland”: 1-845-480-1088, “”
    John J. Tormey III, Esq.: 1-212-410-4142


    Rockland County, NY - January 2, 2008: Livid that NASA and the FAA now appear to have acted in concert towards a common goal of concealing vital air traffic safety information from flyers and others on the ground, and in solidarity with a call for further hearings by Chairman of House Science and Technology Committee, Rep. Bart Gordon (D-TN), suburban New York activist group “Quiet Rockland” today called upon Congress and its investigative arm the Government Accountability Office (GAO) to examine and compel correction of NASA’s just-issued “Air Safety Survey”.

    John J. Tormey III, attorney with “Quiet Rockland”, said: “NASA Administrator Michael Griffin admitted that his agency’s release of the Survey’s data occurred late on New Year’s Eve. He then assured all of us that NASA ‘didn’t deliberately choose to release on the slowest news day of the year’. Griffin and NASA doth protest too much. The NASA survey data was issued in a redacted and deliberately-indecipherable manner. NASA previously sought to withhold the totality of this same data at least once before, when NASA rejected a prior AP FOIA request for it. Of course NASA sought to bury its New Year’s information-release amongst the champagne corks and the dropping ball. Griffin’s suggestion otherwise insults the intelligence of the American public.

    “In response, Quiet Rockland schedules this press release to arrive on what should be one of the busiest back-to-work news days of the new year. 2008 will be the year that we mandate transparency of government. We cannot trust NASA management to communicate fairly or candidly to the American people. It is pathetic that this once-majestic agency of the Apollo era, no longer able to put astronauts on the Moon, and facing difficulty keeping a number of its recently-launched spacecraft intact, now cannot even terrestrially adopt precision or seriousness of purpose beyond that of Captain Anthony Nelson, Major Roger Healy, and Barbara Eden’s ‘Jeannie’. How dare NASA play space games with our safety!

    “The organizational ineptitude of NASA management is particularly threatening in light of yet another recent runway incident between two planes over the Holidays, once again at LAX, involving pilot miscommunications with an air traffic controller. NASA’s ostensible collaboration with its cousin-agency FAA towards concealing safety information from Americans, is confluent with the overall objective of the aero-mercantile complex to over-schedule flights and over-saturate our skies. With focus only upon the almighty buck, these un-checked rogue agencies continue to act at the expense of citizen and environmental safety and health. FAA’s “NY/NJ/PHL Airspace Redesign” is another component of this same harmful aviation special-interest plan. That Redesign must be and will be defeated by citizen outcry such as that voiced by ‘Quiet Rockland’, not to mention the pending federal court litigations and Congressional action against it, taken in the interests of making our skies and our homes safer.

    “NASA and Administrator Michael Griffin indicate that they have no intention to analyze or study, much less further report to the public or press upon the 16,000-plus pages of raw data in the ‘Air Safety Survey’. ‘Quiet Rockland’ therefore asks that Congress and the GAO: (1) audit and investigate NASA’s purposeful mishandling and cheeky and contemptuous New Year’s Eve issuance of purposefully-obfuscated and misleading data; and (2) order NASA to marshal and digest the Survey data and report to Congress, the GAO, and the media on it, in a fully-intelligible writing, within thirty calendar days after the date of this press release. Given NASA’s proclivity to hide from the truth, ‘Quiet Rockland’ suggests Groundhog Day as the most fitting date imaginable for that next report’s issuance.

    “Of the current Survey, Griffin says ‘It’s hard for me… to see any data the traveling public would care about or ought to care about’. ‘Quiet Rockland’ assures Griffin and NASA that anecdotes extracted from the current Survey such as “pilot difficulties in talking to controllers in busy airspace’; air traffic control “capacity inadequate to handle traffic load”; “too many people on the frequency…causing a safety problem”; and perhaps worst of all, “pilots asleep” on the “flight deck”, are most definitely “cared about” by the traveling public - and will indubitably also be “cared about” by the many travelers who comprise Congress, the GAO, and the federal judiciary.”