Monday, May 06, 2019

Faked Test Result Cost $700 Million, Says NASA


Last Tuesday, April 30, NASA announced the results of a years-long investigation by its Office of the Inspector General (OIG) and its Launch Services Program (LSP).  Back in 2009 and 2011, two climate-change-observing satellites failed to reach orbit and were lost at a total cost of $700 million.  In both cases, the payload fairing—the dome protecting the payload during launch—failed to separate on command, throwing the flight dynamics out of whack and ultimately crashing the satellites before they reached orbit.  After a long investigation in which the U. S. Department of Justice was involved, NASA's OIG found that a supplier of aluminum extrusions used to hold the fairing together had been faking materials-testing results on the extrusions not once, not twice, but literally thousands of times over a period of nineteen years.

At least, that is what the revised launch-failure report by NASA says.  Understandably, the company involved disputes some of these findings.  At the time the extrusions were supplied, it was known as Sapa Profiles Inc. (SPI), of Portland, Oregon, although it is now part of a multinational corporation called Hydro.  The report makes for chilling reading. 

To allow the fairing to separate into its clamshell halves at the right moment during the launch phase of a flight, explosive charges are set off to sever the aluminum extrusions that hold the halves of the fairing together.  But the aluminum has to have the right properties to break cleanly, it appears, and so NASA required supplier SPI to do certain materials tests on their extrusions, probably things like tensile strength and so on.  Given the right equipment, these are straightforward tests, and even low-level engineers and engineering students such as I teach know that faking test results is one of the worst, but at the same time one of the more common, engineering-ethics lapses. 

According to the NASA report, such fakery became routine at SPI, so much so that a lab supervisor got in the habit of training newcomers how to fake test results.  NASA found handwritten documents showing how the faking was done.  And this was no now-and-then thing.  For whatever reason, the extrusions failed tests a lot, and so lots of faking went on, not only for NASA's extrusions but for products bound for hundreds of other customers.  But not all of them had the investigative resources and motivation of $700-million launch failures to check out what was happening.

The investigation took years to complete, and once SPI was confronted with its results, the company agreed to pay $46 million in restitution to the U. S. government and other customers as a part of a settlement of criminal and civil charges.  That's a lot of money, but clearly a drop in the bucket compared to what the firm's malfeasance cost NASA and all the people who put years of work and ingenuity into the launches of the satellites which were doomed by the faulty extrusions. 

Seldom does a clear-cut violation of engineering ethics principles have such an equally clear-cut result that makes it into the public eye.  Fortunately, none of the flights affected by the faulty extrusions were manned, but losing $700 million of hardware is bad enough.  What we don't know is how SPI's other customers were adversely affected by the falsified tests, but didn't have the resources to trace the problem back to its true source.  It's entirely possible that this new information will inspire other SPI customers to look back into mysterious failures of their products to see if faulty SPI extrusions may be at fault there as well.  At any rate, NASA has suspended the firm from selling anything to the U. S. government and is thinking about proposing a perpetual bar.

I can only speculate what went through the minds of the engineers who were asked to falsify the test results.  Clearly, a culture of falsification had to be in place for the problem to go on as long as it did.  And numerous psychology experiments have shown that we are much more creatures of our environment than we think we are.  Stanford professor of psychology Philip Zimbardo showed that in 1971 when he set up a mock prision with college-student volunteers who were randomly assigned to be either prisoners or guards.  Within six days, the guards were treating the prisoners so badly that Zimbardo prematurely terminated the experiment for fear that someone could get injured or killed. 

If ordinary law-abiding students can conform to an environment they know is not real, but nevertheless demands that they act a certain way that is contrary to their everyday behavior, it is no great surprise that engineering students newly hired into a company where systematic corrupt practices are in place find it all too easy to conform to the expectations of their supervisor and fall in with the practice of test-result fakery.  As an educator, I don't know what we can do other than to repeat that faking test results is never, under any circumstances, a good thing to do.  Adhering to this advice requires that the listener believe in at least one moral absolute, and that itself can prove to be a challenge these days.

So sometimes, telling stories is more effective than just reciting rules.  The SPI extrusion episode will probably make it into the annals of engineering-ethics textbooks, as it should.  Maybe telling the story of how fake test results led directly to the loss of satellites will make an impression that will stick in the minds of students.  At any rate, this debacle deserves to be more widely known, as it serves as an object lesson for anyone who is responsible for testing hardware, or software, for that matter.

Sources:  I referred to a report on the NASA investigation at the website Space.com carried at https://www.space.com/nasa-determines-cause-satellite-launch-failures.html.  The revised NASA failure report is available at https://www.nasa.gov/sites/default/files/atoms/files/oco_glory_public_summary_update_-_for_the_web_-_04302019.pdf.  I also referred to the Wikipedia article "Stanford prison experiment."
-->

No comments:

Post a Comment