Embedded Muse 131 Copyright 2006 TGG August 3, 2006

EDITOR: Jack Ganssle, jack@ganssle.com

CONTENTS:
- Editor’s Notes
- EDN Turns 50
- The Show Myth
- Failure Stories
- On VCS & Continuous Improvement
- Jobs!
- Joke for the Week
- About The Embedded Muse

Editor’s Notes

Many outfits run surveys to try and understand this fragmented embedded systems industry. The results are often hard to fathom. One (http://www.embedded.com/showArticle.jhtml?articleID=187203732 ) suggests Linux use is *down* over the last year! Another (http://linuxdevices.com/articles/AT7070519787.html ) says the opposite.

I’m trying to understand other aspects of the industry, and have created a somewhat different form of survey. Please, take a minute and go to http://www.surveymonkey.com/s.asp?u=208902414966 and answer the 14 quick questions. It’s a two minute job that might help paint a more complete picture of the state of the industry.

All answers will be anonymous, unless you opt to fill in your email address at Question 14. I won't correlate the other answers to that address, but will offer an iPod Shuffle to one lucky respondent. I only need the email addresses to have a drawing and contact the winner!

Mark Bereit wrote a fascinating article about a proposed radical change in the way we build systems: http://www.embedded.com//showArticle.jhtml?articleID=186700597 . Readers have posted responses here: http://www.markbereit.com/devel/paradigmtrap.html . It’s quite thought-provoking.

The IEEE reports (http://www.spectrum.ieee.org/jul06/4108 ) that salaries for EEs are up a bit over the previous year. Not a lot, and considering pay tumbled in 2004, not enough to keep up with inflation. But it seems the specter of the dot-com bust is over.

EDN Turns 50

EDN was born long after the invention of the vacuum tube ushered in the electronics revolution. But think about the change that magazine has seen! Only two years earlier the first commercial silicon transistor became available – for $100 each. 1956 was the year Shockley, Bardeen and Brattain were awarded the Nobel Prize for their invention. Though transistors existed and were used in some products in 1956, computers still relied on vacuum tubes. In fact, an EDN article that year (http://www.edn.com/article/CA6317063.html ) described the RAMAC, the first commercial computer that used hard disks, which sported nary a single transistor. The machine offered a breathtaking 5 Mb of storage spread across 50 – count ‘em – two foot-diameter disks. See http://www.magneticdiskheritagecenter.org/MDHC/SANJOSE.MPG for a video of the unit in action or http://www.answers.com/topic/ramac for pictures.

In EDN’s first year only 29 million transistors were produced world-wide. Today, according to http://semiconductormuseum.com/Museum_Index.htm, the semiconductor industry cranks out some 10**18 transistors per year (mostly in ICs), or almost 200 million for each person on the planet.

The IC was invented when the magazine was a toddler, Jack Kilby filing his patent in 1959. Noyce et al produced the first RTL logic ICs for Fairchild in 1961. See
http://www.electronballet.com/DataSheets/Fairchild%20Micrologic%201/RTL%20uL900-914/1.jpg for the datasheet of a typical logic IC of the time, which contained a whopping four transistors arranged in a very simple manner.

EDN was 7 when TTL technology appeared.

I discovered the magazine around 1967 while visiting my Dad’s company. EDN was, and still is, free for practicing engineers. A bit of creative fiction on the qualification form soon brought this 14 year old a subscription. Though much of the content went over my head, I perused every issue and learned a lot about electronics and the state of the art.

Some events are so stunning that the details surrounding them are forever burned into your brain. Those of us over 50 can remember where we were when we heard of JFK’s assassination. Similarly, I remember sitting in the kitchen and reading the January 15, 1972 issue. EDN reported about Intel’s 4004, the first “computer on a chip.” See http://edn.com/article/CA6351283.html?spacedesc=features . The data sheet is here: http://smithsonianchips.si.edu/ice/4004thb.htm . Though the story is very dry and matter-of-fact, the idea of a miniaturized computer seemed science fiction. Yet this was the space age; America was routinely landing men on the moon, so the impossible never looked more likely.

The 4004 cost $100 in 1972. Today we can buy a 32 bit processor for a few bucks, yet, ironically, the 4004 now brings in as much as $1000 on eBay. Its (at the time) startling 2300 transistors seems quaint compared to the 370 million in some Pentiums.

I can remember one grizzled old engineer claiming the EDN article about the 4004 was just marketing hype. Computers were big and clunky, and would always be big and clunky.

Over the following years the magazine followed the transition of the computing industry from big iron to flecks of silicon that today form the fabric of our technological society. The magazine, too, has evolved, having added Dilbert cartoons, a snazzier layout, and short columns by industry experts. But it continues to report on new trends and products, with detailed design guidelines on emerging technologies. Every electrical engineer simply must have a subscription.

Happy birthday, EDN! I hope to be reading it when the magazine turns 75. And I cannot even imagine what astonishing technology will be described in that issue.

The Show Myth

This is the Show Myth: the hysterical response of the engineering director to marketing’s panicked call for a new or upgraded product.

At first blush it’s almost a reasonable schedule-driver. The company will, after all, spend astronomical sums on The Show; why shouldn’t engineering deliver a new widget to help justify those costs? Why not introduce a new product to make a splash rather than a whimper?

Any capriciously-assigned deadline is a joke, an exercise in futility. 80% of embedded systems are delivered late, even in those (rare?) cases where careful estimates override marketing’s giddily optimistic promises to customers. When The Show’s inflexible date sets the project’s completion time, we’re creating a recipe for disaster.

Hey, go to The Show one year, whatever show your industry primarily patronizes. Watch the demonstrations of those fancy new products. The average demo lasts maybe 10 seconds. 20 tops.

How much needs to work for a successful 10 second demo? Face it – if the red light comes on when the salesman presses the blue button, you don’t need an awful lot of working firmware.

If The Show is indeed important for business reasons – as they often are – then maybe we should rethink the schedule. Instead of saying “let’s complete this by then”, isn’t it more reasonable to implement a subset of features? Just a few things that demo very well, that look cool. Identify features knowing that the goal is passing a 10 second demo, not an in-depth evaluation that could only really take place at a customer’s site over the course of weeks.

If you do go to that show, get an exhibitor’s badge and visit the day before the event opens. Walk the floor and watch the chaos as vendors scramble to get booths assembled, carpet down, phone lines installed.

And look very carefully at the sales people pulling the latest new shiny widget from its packing crate. Watch the fear in their eyes, the sweat beading on their brows. They know, with certain clarity, that this is yet another product rushed to The Show with inadequate testing via crazy deadlines. Salespeople have huge egos; bigger even than engineers. All fear the crash when even that 10 second demo fails.

So identify the significant show features, implement those, and only those, very carefully.

And get them right.

Failure Stories

Recently, at a new employ a coworker told me of an experience where a time related problem was traced to the start-up cycle of the mercury vapor lamps used in a facility. When the factory lighting was switched on at around 5:00 AM, the lamps' warm-up cycle would result in unusual voltage and current loads on the mains. And these would impact on safety equipment designed to monitor the loading on the mains.

And another story:
About 20 years ago, a previous employer sent me from our R&D facility near Toronto, Ontario, Canada to a customer location near Seattle, WA. to look at a data communications problem that seemed to manifest itself an the morning, between 9:30 AM and 10:00 AM.

The pre-PC & pre-LAN equipment involved, a small mini-mainframe computer, a couple of data terminals and voice/data PBX manufactured by my employer. Over a few days, while looking at power distribution loads and time related cycles in the facility and similar technically focused equipment cycles I noted the behavior of the employees using the terminals. Specifically, the terminal where the problem, a lock up, was occurring was used to enter data from time cards from the customer's machine shop. The amount of data on a time card was limited to the usual machine number, operator number, operation number, start time, end time and the like.

The clerk entering these time cards would come in to work at about 8:30 AM and chat with her co-workers while entering a few cards. After a while she would break for her morning coffee and chat a bit more while entering a few more cards. She would slowly build up her data entry speed to what I thought was close to 300 wpm using the numeric keypad on the terminals. And it would take the clerk an hour or so to build up to this data entry rate.

While the clerk was entering data at the 300 wpm rate, occasionally the terminal would lock up. Investigation showed that delays in passing the asynchronous data characters through the PBX to the computer and back would exacerbate a "bug" in the terminal that would cause the terminal to lock up.

I will admit that in the few days I was at the customer site monitoring for the failure, I did not immediately see how the clerk's behavior and data entry speed would have an impact. It took a while for this realization, the formulation of a hypothesis and confirmation of the latter.

Note: 300 wpm is not an error. A numeric-only data entry rate of 300 wpm is roughly equivalent to a normal typing speed of 60 wpm.

The Lesson Learned: Like looking at the sunlight during the sunrise, this situation involved looking at the human behavior. One has to learn that in troubleshooting one might have to look up from the bits and bytes, the hardware and the software involved and consider other factors, however far from the "usual suspects".

On VCS & Continuous Improvement

I concur with Leland Hamilton where he said: "I prefer to place software in some sort of VCS system as it is being developed ... " And I would add that, I believe that the work, source-code at all, from the start or close to the start, should be available, e.g. on a LAN, for all in the organization to see and review should they have the need to do so.

I had difficulty at a previous employer getting coworkers to buy into the concept to tracking changes in development.
Their thought was that revision tracking was important only after release - to production.

I maintained that with incremental releases during the development process the organization needs to be able to track the process between internal development releases and the "final" release.

A related issue was that while I thought it important to put my day-to-day development work, embedded source code and all, on the company LAN for all to see others preferred to "hoard" or hide their work on their own hard drives only. The other developers would, from time to time, "release" an embedded executable to the other hardware developers without letting them see the source code. Until it had to be released, these developers would not let others see their source code.

I agree with Hamilton's comment that the use of the VCS "helps to see the development progression" and will add that I feel this type of information can be used with the philosophy of "Continuous Improvement". The detail of the work involved in designing, prototyping, testing, debugging software can disappear if one is simply left with the calendar dates such as start date, planned finish date, actual finish date and the delta between the last two dates. With no idea of what caused the delta in the first place one is hard pressed to come up with any ideas for "Continuous Improvement".

Jobs!

Joke for the Week