By Jack Ganssle

Medical Upgrades

Reader Rick Schrenker, Systems Engineering Manager Massachusetts General Hospital Department of Biomedical Engineering, and I corresponded recently about a trend in the embedded systems used in hospital settings. His comments are insightful and should impact the way we build networks of systems. I'll quote extensively from his emails:

"Just over a year ago I presented a talk listing some potential hospital use cases for RF based wireless technologies at an IEEE 1073 meeting. At the end I went off topic to throw in the following tidbit: Of the 30,000+ medical devices currently in use in our operation, about 9,000 had software versions entered in our equipment management database. We have somewhere between 1,000 and 2,000 models of equipment in our system last time I ran a query. The system is complex to manage.

"Twenty years ago, I could go to the service manual of any of device in our inventory and determine what it was supposed to do and how it did it. The key, of course, was the schematic diagram. What made it so useful was its use of standard symbols to represent components. The abstractions were consistent across the industry. A schematic from HP for a patient monitor was as meaningful as a schematic from Tektronix for a patient monitor was as meaningful as a schematic from Siemens for a ventilator was as meaningful as a schematic from Baxter for an infusion pump. Based on the schematic and related information, I could train technicians how to provide service to the device, or clinicians how to use it. I could develop tests to verify operation, and I could validate that the device could do what it was supposed to do, even in concert with other devices."

Rick then complained that we might have UML or other design documents to guide us during development. but none of that information flows to the people in the field doing maintenance. His comment "There are not only no documentation standards, apparently there are no standards for anything" closely mirrors my experience in working with developers on all sorts of projects. Working code - more or less working code - no matter how badly structured and hacked together, is all that counts. The quality is invisible to the customer and our boss, and only becomes apparent when bugs surface or maintenance becomes difficult or impossible.

Rick went on to talk about the problem of upgrading software in all of these embedded applications, some of which are truly safety-critical. He wrote: "The vision I described at the IEEE and other forums goes like this: Hospitals will have a lab with some number of workstations lying around it, each configured to support the update of one or more types of medical devices. Technicians will bring the devices in, connect them to the workstation, magic will happen, and the technician will return the devices to the floor. Hopefully nothing will go wrong, either to the device or in the context of the system in which they are being returned to be applied. After my IEEE presentation, an engineer from a medical equipment manufacturer came up to me to say he'd never considered what it might be like on the receiving end of one of his upgrades."

Rick notes that his colleagues who actually perform the upgrades report that they may come only with installation instructions, i.e., they may not include a list of changes that were made or a procedure describing a comprehensive test of the upgraded device. Rick has heard of cases where a call to the manufacturer's help desk reveals that they may not have the information, either.

Scary stuff. It's tough for us developers to really understand how our customers use the equipment. I've always advocated sending the engineers to the customers' sites, in uncomfortable support situations, because there's no better way to see what the user really needs.

Rick says the FDA is working to address a myriad of issues related to software-based medical devices but believes its ability to address current problems may be constrained by the scope of its authority. One topic he cites that has been kicking around for about a decade is when does a network become a medical device? As Rick understands it, FDA is limited to regulating medical devices, and a network does not meet the strict definition thereof, although it may be a component of or accessory to a medical device. Medical devices communicating over networks is nothing new, but generally hospitals have partitioned IS and device networks via the use of cable. The advent of wireless effectively does away with this method of avoiding the issue. While Rick admits to not being sure he groks the breadth of regulatory authority in this area, he is concerned that if the FDA is indeed constrained to regulating devices and not systems, then it will become increasingly difficult for hospitals to evaluate the risks they accept in implementing a system. As more and more devices share spectral space, whose responsibility is it, i.e., who will bear the cost and risk, to ensure that the entire mesh works correctly. and safely, both in routine day-to-day operations and when subjected to the rigors of software upgrades?

I find it interesting that a network turns even the simplest device is part of a huge system. Change part of it, via an upgrade, and what effect does that have on a connected device or database perhaps in an entirely different ward?

And user interfaces vary all over the map as well. When a harried doctor or nurse needs to adjust some setting that might help or kill the patient, she has to dig through what might be unfamiliar menus - in a crisis situation - to program the device. The patient is crashing and one unit beeps. Just what does that single beep from that machine mean? Oh, it's the same thing as three beeps from the other unit.

Rick recalls seeing an IV pole with three infusion pumps that were being evaluated for user preference. "Not only did each have a different UI, I couldn't intuit any of them. Back when I practiced on the floor that was a key criterion for me. There was only so much time for training, and if a device didn't provide at least some guiding context of itself, I found it relatively more difficult to teach. It's even more of a concern today, given the accelerating rate of change in technology. What strikes me as least defensible in all of this is that throughout my life I have been able to go into any auto showroom and test drive a car within minutes, but licensed practicing clinicians almost always require fundamental user training every time they encounter a new model of types of devices that have been around for 25 years. I'm not talking about training for new features. I'm talking about just learning the UI."

Rick cites the Institute of Medicine's 1999 report "To Err is Human", where it was estimated that as many as 98,000 Americans were dying annually as a result of medical errors. "In the section titled "Why do Accidents Happen?" the report states "People . become accustomed to design defects and learn to work around them, so often they are not recognized" and "Accidents are more likely to happen in certain types of systems. When they do occur, they represent failures in the way systems are designed. The primary objective of systems design ought to be to make it difficult for accidents and errors to occur and minimize damage if they do occur." While the report's scope was much broader than devices alone, the implications for systems that include devices are clear. And the number of what are argued to be preventable deaths is staggering. At the high end, it's the equivalent of greater than two September 11ths every month. Yet five years after publication, by which time the report targeted that a 50% reduction in the number of deaths should have been achieved, one of its coauthors stated, "The evidence of improvement is indeed unimpressive." ("Five Years Later, Medical Errors Still a Leading Killer", Boston Globe, November 9, 2004). And some of us believe it could be worse if not for the absolutely incredible expertise and dedication of the medical staff who compensates for the many inadequately designed systems within and with which they have to work."

Rick concluded: "This brings me back to the need for technology standards that extend beyond design into application, including maintenance. Ultimately, in the software engineering domain, only the software engineering profession can fix this. Manufacturers do not have incentive to do so, especially with the money to be made in providing proprietary maintenance systems (manufacturers just love service contracts, or as they now call them `service support agreements', especially when there is so little information available from which to discern alternatives that essentially there is no choice but to purchase them). I certainly hope that your colleagues will address the problem, but I'm not hopeful, at least in the short term. As much respect as I have gained for software engineers in terms of the nature of the fundamental technical problems they address, I am concerned at how little attention I see them giving to life cycle systems issues. My sense is many if not most are so focused on the technical that they rarely if ever stop to consider the breadth of impact of their products.

"A profession arises from the application of technical expertise in service to social interest. Software engineers may construct good, useful artifacts, but if they want their profession to rise to the level of others, they have to address issues like these."

I agree. We need to think about much more than the code. We developing systems of unimaginable complexity. How will our customers use, maintain and support this equipment?