An OS in a Can

Published in Embedded Systems Programming, January 1994

By Jack Ganssle

I have a confession to make: until recently, I'd never used a canned Real Time Operating System (RTOS). Over the years I've developed a lot of embedded systems built around RTOSes, but all were kernels I kludged together myself.

One of the first systems I did with an RTOS was a X-ray big gauge used to measure the thickness of hot (2000 degree) steel moving at high speed. One of the folks involved in the initial design suggested that we include an RTOS to handle the vast number of interrupts and asynchronous activities going on. My partner and I dismissed this; we knew that a simple polled loop would be more than adequate. Besides, we took this suggestion as an attack on our professional competence, which, from the distance of a dozen years, I guess was maybe not so competent. The nail in the coffin of the RTOS idea was that neither of us had any experience with these, and were frankly concerned we'd be in over our heads.

Well... the polled scheme didn't work too well. We frantically shoehorned a simple RTOS written over a couple of weekends into tens of thousands of lines of extant code, creating a nightmarish problem of correcting all the documentation to reflect the new, radically different, program structure. The RTOS was a simple round robin task scheduler with no messaging facilities or semaphores, but it performed well and made the code much more organized. Now, each logical activity could be grouped as an independent task that ran without worrying much about what else was going one, instead of being one of hundreds of routines a massive polling driver had to sequence.

After this experience I looked back and realized that a number of systems that I'd built would have been easier to code and more robust had I included an RTOS. Fear of the unknown is a terrible thing. I started including RTOSes in more and more products, finally writing, of all things, a multitasking Basic compiler for CP/M and later DOS that made Basic itself a tasking language. What a bust! This is another story, but suffice to say that despite selling 10,000 copies it never made much money.

My last mistake (that I'll admit here!) was to write a chapter about rolling your own RTOS in the book I had published a year or so ago. Mea Culpa.

Anyway, in conversations with vendors I hear that my experiences were not unusual. Apparently 80% of all operating systems in embedded systems are home-brew or downloaded freebies. I'm told that the remaining 20% of the market, that fulfilled by commercial real time operating systems, is worth about $20 million. Most developers feel the kernel is so small and "simple" that they might as well code one up in a spare hour or two. The result is often a buggy mess that takes far too long to fix and that offers only a handful of basic services. Any port to a new processor is a struggle, as every CPU has its own interrupt structure.

I've learned, through many mistakes, that the best solution for any software problem is to buy code in a can whenever possible. As a programmer I always either delivered code late or got it out on time through heroic efforts. As a manager I've learned to be wary of time estimates from any software person. Why increase your problems by writing proprietary code when a simple, cheap, solution is available off the shelf?

Cost is the usual objection. A decent RTOS can cost thousands of dollars. A little math shows just how cheap this is: suppose it takes a week to code, test, and thoroughly prove your home-brew RTOS. At a salary of, say, $50k, you earn about $1000/week - before benefits. Most companies figure overhead at about twice salary, so the raw cost of the one-week OS that will do little and is unproven will be $2000. Shop around - many commercial products cost less than this; others, though perhaps a pricier, offer lots of value for the buck. And each has been proven in hundreds of applications.

Now I'm seeing more and more people interested in buying rather than building. Perhaps the industry is maturing. Certainly the promise of OOP will never be realized till we programmers decide to buy software components rather than reinvent them. In the embedded world there are few such components that are widely used regardless of what system is being designed - the RTOS is probably the most common of these.

An RTOS

One of the neat things about my job is that lots of vendors send copies of their latest compilers, OSes, and other tools for me to look at. I wish I had the time and hard disk space to try them all! I decided to play with a commercial RTOS, and selected A. T. Barrett's (now known as Embedded System's Products) RTXC simply because, frankly, they were the most persistent. Others looked great. Most came with wonderful documentation. However, this is not a review of any one OS, but is rather an account of features found in RTXC and in most off-the-shelf operating systems. Most of what I list here will be found in any vendor's product..

Like many such products RTXC comes with complete C source. Though an RTOS is conceptually pretty simple, real products offer so much functionality that their code is substantial and non-trivial. I quickly decided I had no interest in learning how it worked, so decided to just include the code with my application and not make changes to it.

Many users expect the source for maintenance purposes. This makes sense to me as insurance against the vendor's demise, but I'd be hesitant to make changes unless there was an overwhelming business reason to do so. Any change will make your version incompatible with future upgrades, and will probably invalidate your warranty. Make changes in your application to deal with the peculiarities, if any, of all purchased software modules.

RTOSes are little more than a collection of services that you invoke in some manner. Some use software interrupts to isolate the OS from application code - you issue the interrupt, passing parameters. The OS intercepts the interrupt, performs the function, and then returns to your code. This has the advantage of making the RTOS a separate binding, so you never recompile or relink it.

Others make the services available as public functions which your program calls. Generally you link this sort of RTOS into your executable, simplifying ROM creation and making the RTOS internals available to debuggers.

To the user, the basic element of an RTOS is the "task", a functionally complete activity that can run more ore less independently of other code. In RTXC each task is encapsulated within a C function, though that function (task) may be very complex, and may call many other C and assembly functions.

The RTOS's primary job is to assign computer time to each task. In a wonderful world the operating system's context switcher would start task 1, let it run to completion, then run task 2, etc. Real time applications are never so simple, however; usually, each has it own peculiar execution requirements. Some never really terminate - often a polled keyboard handler is always more or less active. Others terminate but need to be reborn - a routine to pound on the watchdog timer, for example, might want to reincarnate once a second. Still others start only when a specific event occurs - a serial character arrives and generates an interrupt.

Most home-brew operating systems have a time slice scheduler that - task 1 gets 1 millisecond and is then suspended, task 2 gets a millisecond, then 3, 4, until all have had a chance to run and task 1 starts again. This is called pre-emptive tasking because any task might be temporarily suspended at any time.

All reasonable commercial OSes support this form of preemptive multitasking, but carry the concept further. First, each task has a priority assigned to it. The highest priority tasks get all of the execution time until complete, blocked by a need for some sort of resource, or their priority is changed . Priority allocation is always dynamic so a critical action can gain more access to the CPU when something important is going on. In a medical instrument the "oh my god we're gonna zap the patient" routine probably gets the highest priority when critical X-ray levels are detected.

Asynchronous events can change execution order. Generally an embedded system thrives on multiple interrupt sources. A "character received" interrupt might spawn off a task to queue the character to system buffers and then terminate until the next character comes. Commercial RTOSes also allow for software events: a "buffer full" condition could create an event that starts an error task going.

Homemade operating systems generally provide little more than some sort of tasking capability. Remember, though, that tasks will want to talk to each other. Without some sort of inter task communication protocol, they'll have no option other than using a zillion global variables. Yuk. Globals are the source of world hunger, the deficit and ozone depletion. Or, at least of crummy, impossible to debug code. Any routine can step on any global, so it's all but impossible to guarantee that any routine will not corrupt any other.

The best solution is a repertoire of robust inter task communication resources. Decent RTOSes all pass data using some form of messaging system, where a task cleanly defines and sends a message to another task. The message's transmission, reception, and synchronization is all maintained by the operating system itself.

This is important and non-trivial. I see lots of applications that crash and burn because a home-made RTOS uses several instructions to load a variable being passed. An interrupt comes in the middle of the load, and the receiving task gets garbage.

In RTXC, and I presume in most other RTOSes, each task gets one or more mailboxes for incoming messages. Each task exclusively owns its mailboxes, so you can guarantee that a message meant for one task is not accidentally received by another. The mailbox is a variant of a simple FIFO, where the task reads the oldest message first. Unlike normal FIFOs, the operating system can reorder the mailbox's contents when high priority messages come in.

It's just not enough to send messages. Sometimes a task might need to wait until the message has been accepted by the receiving task before proceeding. If one task sends a message to set the gain of an amplifier feeding the system A/D converter, then it really doesn't want to continue executing if the next instruction is a read A/D until the message has been received and processed. Therefore, RTOSes support of synchronous messaging system that suspends the sender and leaves it suspended until the receiver asserts an "I'm done" message.

Some kernels use semaphores to synchronize activities. A semaphore is typically a one bit flag used to handshake between tasks, so task A knows when task B is done with a certain event, or so the system knows when a resource (such as an I/O device) has been freed up.

Everyone worries about getting off-the-shelf software for an embedded system and finding it is too big to fit in the narrow confines of a ROM. As you might imagine, the mailboxes, semaphores and queues can eat up a lot of space. Further, each task uses its own share of stack room as well. Most vendors address this by offering some sort of system generation routine. When you define your system, you specify the number of each resource needed, and let the SYSGEN build data structures into header or include files that allocate just enough (but no extra!) storage for your application.

An Example

For fun I wrote a system that controls a train moving around a track. Magnetic sensors in the track detected the train passing overhead and send radio messages to a receiver inside the train, stopping and stopping it at each sensor. Then, my 6 year old and I assembled a really cool Lego train set, cannibalized a cheap radio controlled car for the transmitter and receiver, and rewired the train to respond to the radio. Actually, we had more fun assembling the Lego pieces than writing the code...

To show just how easy this is, here is a simplified version of the code. I've cut out some of the tasks to fit limited magazine space. The idea is to show just how easy it is to write RTOS-based code. Don't write complaining that the code could be simpler! This is illustrative only.

/* main module - starts tasks off */
main()
{
   int i;
   rtxcinit(); /* initialize RTXC */
   clkstart(); /* start the clock */
/* start each task */
   KS_execute((TASK) SWITCH); /* looks for switch closure */
   KS_execute((TASK) START); /* starts motor */
   for (i=0;;) KS_yield();  /* yield CPU time to other tasks */
}
switch()  /* wait for switch closure */
{  int i,j;
   for (i = 0;;)
    {j=inport(0x290);
     if ((j & 1) == 0)
      {KS_execute((TASK) STOP);
        KS_delay(SELFTASK,50);
       };
   KS_delay(SELFTASK,2); /* give some CPU time to other tasks */
   }
}
start() /* task to start the motor */
{
   outport(0x290,0xff);       /* start motor */
   KS_terminate((TASK) 0); /* end task */
}
stop() /* task to stop the motor for 1 second */
{ outport(0x290, 3);	     /* stop motor */
   KS_delay(SELFTASK, 100); /* leave motor off for 1 second */
   KS_execute((TASK) START); /* restart motor */
   KS_terminate(SELFTASK); /* end task */
 }

It's interesting to note that all of the time management is taken care of by the RTOS. From a programmer's standpoint you make simple "delay for 100 msec" calls and let the kernel handle the rest.

Evaluating an RTOS

Which RTOS is right for your application? Consider your application and ask yourself and the vendor a few questions:

1. Do you really need the source code? Don't get it just to make changes. Live, preach, and practice encapsulation.
2. Does the RTOS support all of the features you'll need? If you are new to using an RTOS you'll see no immediate use for many of the features each product offers. However, with time and experience you'll want ever more. Get lists of functions from the vendors and make sure they are complete enough. Some companies sell the RTOS in different flavors with different functions included.
3. Does the RTOS cost structure fit your requirements? Some provide it royalty-free; others charge for each use, presumably reducing the up-front price.
4. The holy grail of RTOSes seems to be context switching times, yet these are only rarely advertised and are widely misunderstood. Don't base all of your decisions on this, as no matter how bad an RTOS is, most of the time is burned up in your application. Optimize the code there.
5. Be sure that a configuration that is usable to you will fit in your ROM. Kernels are growing as vendors add more and more features requested by users. Your 8051-based system with a 2k ROM might be too small for a realistic RTOS!