Doctors, Data, and Doomsday

Nearly every business, government office, and organizations of any size down to the local barber shop have made the transition from paper records to computers—except doctors and hospitals. Go into any doctor's office and you will still see big file cabinets filled with cardboard folders bearing colored tabs. The system of keeping a file for each patient was an innovation when the Mayo Clinic came up with the idea in the early 1900s. As Robert Charrette reports in a recent article in IEEE Spectrum, the Clinic is one of the few medical facilities so far to make a successful transition to all-electronic records. But he warns that while we aren't necessarily facing a medical Doomsday, troubles lie ahead along the way to converting the entire U. S. medical system to computerized recordkeeping

As Charrette points out, the history of large-scale software projects is littered with the bones of huge, expensive failures. One of the most egregious was the FBI's attempt to computerize their elaborate system of case files, which had been kept on paper since the days of J. Edgar Hoover in the 1930s. After spending over $100 million, the FBI gave up on the project altogether. Why is it that society tolerates such disasters in software engineering? If banks lost your money as readily as some software firms do, people would still be keeping their cash in mattresses.

Software engineering differs from almost every other kind of engineering in two fundamental ways. In electrical, mechanical, civil, and chemical engineering, the subject matter of the discipline is something physical: steel dirt, chemicals, or electromagnetic waves. But in software engineering, the "material cause" (as Aristotle would put it), the matter out of which the discipline emerges, is thought. And thoughts are notoriously hard things to pin down. Secondly, most large-scale software projects invariably deal with the largely undocumented and tremendously variable behavior of thousands of people as they do comparatively complex intellectual tasks. This is nowhere more true than in the medical profession, where some of the most highly educated and individualistic professionals deal daily with life-or-death situations. These two factors make software engineering the most unpredictable of engineering disciplines, in that despite the best plans of competent engineers, projects often run off the rails of budgets and schedules to crash in the woods of failure (metaphorically speaking).

To what extent are software engineers morally culpable for the failure of a major software project they are involved in? Failures are a normal part of engineering. And it can be said in behalf of most software project failures that no one dies or is seriously injured, at least directly. A building that collapses usually takes someone with it, but a failed software project's worst consequences for individuals are usually the loss of jobs, not life itself. But the expenditure of millions of dollars toward an end that is ultimately never realized is hardly a social good, either.

Despite such notable failures, no one seems inclined to give up on the idea that computerizing paper medical records, if we can do it, will be better than the situation we have now, where the present limited access to data results in thousands of misdiagnoses and hundreds of deaths every year. Of course, along with the promises of better access for those who need to know medical records comes the threat of abuse by unscrupulous businesses and criminals. Patient advocacy groups have already weighed in to oppose the present versions of health information technology legislation which do not protect the privacy rights of patients enough, in their opinion. This is a problem that can be dealt with, as the largely successful effort to put private banking records on the Internet has shown. But the challenges are greater with medical records, and it would be easy to promulgate a system that would have as many security holes as a Swiss cheese if things aren't done right.

Some people advocate an increased role for the federal government in this area, pointing out that many medical practices are small and simply don't have the resources to adapt on their own. The track record of government involvement in medicine in this country is excellent with regard to research, problematic with regard to large-scale social programs such as Medicare, and largely unknown with regard to standardized software. As with anything else, if enough good people of good will are put to the task, it could be made to work. But in the present political atmosphere in which government is often regarded as the enemy of the free market and the good in general, it is hard to imagine how enough public and professional support for a government-sponsored project could be raised.

The field of software engineering itself is only about a generation old, and its practitioners are increasingly aware of the need to borrow from fields such as sociology, ethics, and psychology to do their jobs better. The old days of a geeky nerd sitting alone in a cubicle churning out code that no one else can understand are passing, if not completely over. Good software engineers study the project's intended users as thoroughly as anthropologists observe primitive tribes, in order to figure out not only what the customers say they want, but in order to discover existing methods and connections that the users may not even know about themselves and their organizations. The ideal paper-to-software transition in the medical profession will still be a lot of work. But if it is staged properly, using good examples such as the Mayo Clinic as paradigms and checking results in each new case before proceeding, it could work as smoothly as the introduction of computers into banking. But in this case, it won't be your money, it will be your life.

Sources: The article "Dying for Data" in IEEE Spectrum's October 2006 issue is available online at http://www.spectrum.ieee.org/oct06/4589. Charrette also wrote about the FBI project failure in "Why Software Fails" at http://www.spectrum.ieee.org/sep05/inthisissue. An example of one organization advocating in favor of better patient privacy rights can be found at http://www.patientprivacyrights.org.