You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• On Management
• The 1802 Processor
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here and here.

In another blow for Boeing, 737-NG aircraft can't land going west. If the runway is facing 270 degrees (true) all six display units blank. Happily, there are only 7 such runways in the world. FAA Airworthiness Directive here.

Dave Gutow won last month's contest, for a PSoC dev kit. There's no contest this month due to a few weeks of travel. Stay tuned for March's giveaway.

From Eric Smith: My favorite quote regarding programming languages in general is C.A.R. Hoare on Algol 60: "Here is a language so far ahead of its time, that it was not only an improvement on its predecessors, but also on nearly all its successors."

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Brian Cuthie likes Visual Studio Code and the Saleae logic analyzer:

Tomasz Kos wrote:

We've made some improvements to our tools page, which is comprised of short reviews and comments of various hardware and software tools we embedded people use.

The voice on the other end of the phone was quiet, furtive. In guarded tones "Joe" recounted the sordid details of his story, as if I were an investigative reporter. "I can send you documents, copies of our listings, memos, you name it", he murmured.

He worked for a small under-capitalized firm which was intolerant of any sort of failure. Joe's boss had a rule: software bugs were unacceptable. At first I mistakenly assumed he meant delivered products were to be defect-free, a noble goal I'd hope we all share.

"Nah, we're not allowed to do any debugging," he almost whispered. "The boss says we've got to write code correctly the first time, and he's not spending any money on fixing our mistakes. So there are no tools and we all work OT to fix problems."

I was reminded of my teenaged technician years, when the QA manager decreed that we were to replace only defective components, under threat of dismissal. From time to time we techs screwed up, of course, removing parts that in fact were just fine. Since Mr. QA had us send all of these extracted components to an outside lab for failure analysis, we learned to manually fry those that we had replaced in error. His in-basket consequently received some quite puzzling failure reports from the lab. Once I killed as op amp in a metal can by applying 110 VAC to the leads. It blew a hole through the can. I never saw the failure analysis for that part but imagine much head-scratching ensued.

Part of me wanted to believe that Joe's boss was a man of deep insight, who had created a dynamic development environment where the product was so well designed all - or most - bugs disappeared before test started. Unhappily the man was as capricious as my old QA tormentor, as he mandated a terribly difficult goal without providing any system or framework in which to achieve it.

Developers almost universally complain about managers, for good reasons and bad. The number one complaint I hear is about fickle decision making. Schedules with imposed end-dates, or those arbitrarily revised. Disappearing resources. Marketing overrides very real technical requirements, and even impose plainly impossible technical goals.

Too many of today's bosses are often preoccupied with generating reports, preparing for shows, and endlessly meeting with vendors. Though these are important roles, managers have two critical functions that are often neglected. First, managers must manage. This means tracking the day by day, week by week progress of an engineering effort, which presumes there's an honest schedule which forms the project's roadmap. When something changes - and things always change - then an effective manager revises the roadmap.

Management means acting as a buffer between the doers and upper level decision makers, insulating developers from distractions from on high, and waging battles to secure needed resources.

Management means looking forward and anticipating changes and problems, clearing minor potholes before they become roadblocks. When disaster strikes it means working with the team to find reasonable solutions, and negotiating with the manager's superiors to adapt to the new realities, without imposing impossible requirements on the development team.

But there's another aspect to proactive management. Your boss is immediately tasked with building a product, but much more important his role is to create an effective, constantly improving, way of building products. Keep your shoulder to the grindstone and you will eventually fail.

Truly enlightened managers transcend an incessant focus on "shipping the damn thing now" to look a year or more ahead, and to develop the skills and technologies that will surely be required.

This is an extension to the philosophy promoted in The E-Myth Revisited by Michael Gerber, one of the better books around for people building businesses. Gerber suggests that too many small companies (and by extension, engineering departments) fail because the leader spends too much time doing productive work, and not enough time inventing a company (department) that does better work in less time. We've all discovered a tool that could save us weeks of effort, but being trapped by an impossible deadline we feel we can't spend time to install and learn how to use it.

Then there's us.

I've observed that all too many developers seem to reach the zenith of their abilities in college. Yes, building real products in industry hones our skills though repetitive practice, but the raw knowledge of how we should design systems and write code seems cast in concrete when the diploma arrives.

How many engineers over 30 ever truly master a new concept like OOP or UML? For every success story there are too many who stayed locked into the concepts with which they have long been comfortable.

We can acquire new skills only by dint of struggle and practice. With today's development effort always overwhelming every other consideration it's no surprise we careen from project to project with little chance to improve. Home life gets ever more complicated with time; children and spouses require attention that detracts from after-hours study opportunities.

Yet this field changes daily. Those not changing with the times are doomed to professional lives in ever more restrictive corners of the field. My dad told a story of a mechanical engineer he knew in the 1960s who was the world's expert at designing wheels for lunar roving vehicles. Surely that poor fellow became unemployable during the engineering crash of the early 70s.

We should take charge of our own careers, and manage those careers aggressively. Figure out what you need to know, and then take action.

But managers, too, must enhance their peoples' skill sets. A manager's most expensive resource is his or her people. Let them stagnate and the entire team will degrade.

We know how to improve software development. The Capability Maturity Model (CMM) is one such process. Though the CMM is no panacea, it does describe a disciplined way to create software products in a predictable manner: predictable bug rates, predictable schedules, and the like.

Like ISO9000, the CMM requires a major commitment from management as it changes the nature of the entire organization. Effective? Yep. Hard? You bet.

In my opinion software development is so hard that we need every asset possible to help us. Pursuing the CMM is one strategy.

Another is Watts Humphrey's Personal Software Process (PSP). Where the CMM requires a total management buy-in from the top down, the PSP is an approach individuals and small teams can use.

Yet when I talk to firmware developers just about 0% work for a company pursuing the CMM. Another 0% know anything about the PSP.

Managers must manage their people's skills. They've got to cajole, plead, push and order their folks to study these and other ideas to improve. Stagnation is death.

And we need to learn from our mistakes. There is no industry I admire more for their management of failure than the commercial airplane business. Every crash brings out an army of investigators who are tasked with finding the accident's cause. The planes themselves are equipped with the notorious "black boxes", tools designed to record and report the parameters that lead up to the crash. When possible, investigators recommend changes in training, documentation or plane design so the chances of a similar problem will be reduced.

In other words, the system has a built-in learning process which is highly targeted towards making changes

Why don't we work as hard to learn from our mistakes? Failure will always be part of doing anything difficult. We'll miss schedules, deliver products whose quality falls below expectations, run over budget, and even suffer massive redesigns due to our own problems and changes imposed from without.

A wise manager simply has to understand that if we don't manage failure, if we don't study our mistakes and turn them into learning experiences, we'll repeat the same old dysfunctional problems forever.

The airplane industry has a feedback loop that improves the process. If we don't emulate their example via aggressive failure management, we'll also have a loop: Repeat forever: the usual mistakes. End Program.

I've often wondered what happened to Joe. Did his boss resign him to he scrap heap of failed employees?

How many of us either manage only to meet just a project's goals, or manage to build an awesome engineering department?

Back in the late 70s processors consumed a fair amount of power. It wasn't unusual for one to need two or even three power supplies, one of which was generally a negative voltage. RCA took a different approach: they bet heavily on CMOS technology, which seemed a backwater to most cutting-edge designers.

CMOS was both extremely low power and highly radiation resistant, factors critical for space and a few other applications. CMOS, though, was slow, very slow. Laughably slow. Switching speeds could be in the hundreds of nanoseconds, a far cry from the 10-20 nsec speeds achievable by reasonably fast TTL. Most of us considered CMOS an interesting idea that would soon die off. We were all so wrong; today CMOS's speed problems are gone, and it's the logic of choice for most applications. In fact, only the use of CMOS now avoids complex chips from self destructing.

RCA joined the microprocessor fray with their 1802, a fully CMOS 8 bit device with a very unusual architecture. It was intensely register based, having many times the number of internal registers as any of the competing NMOS devices. Any one of those could be the program counter. Today we find that one of the tenants of RISC technology is to similarly maximize registers. RCA was clearly ahead of its time.

Its greatest flaw, though, was a total lack of a stack. No push, no pop. No call; no return.

Indeed, the 1802's innate lack of call instructions was a source of continuous grief. We worked around this by building a non-reentrant pseudo-subroutine structure: you do a call by loading a fixed register with the return address, another with the destination of the subroutine, and then jumping to a "subroutine" that stored the return in memory on a manually-managed stack before finally vectoring off to the destination. Returns were managed in a reverse manner. Rather like a primitive version of Arm's R14, aka LR (link register).

The overhead was enormous, but the 1802 was never known as a speed demon. Instead, it found its way into ultra low power applications.

My business partner and I managed to get a number of jobs designing deep sea probes that monitored temperature, pressure and other parameters, running for months on small batteries. The 1802 was our only option, as it required only microamps when operating at slow clock rates.

We used an assembler which ran on a floppy-based CP/M machine. Happily, the assembler included macro capability which made generated the convoluted call/return sequences much easier.

We wrote a very simple monitor program that permitted us to download hex files, fiddle with memory and registers, and stop/stop program execution. The cost of an emulator - about $20k in 1981 dollars - was far beyond our bankroll.

The 1802's fully static design greatly simplified debugging. We'd run the machine at one hertz or less at times, using a scope or even a voltmeter to probe address and data lines.

Later we found a customer doing research into the aging of fruit. Apparently, during the course of a month-long trans-oceanic voyage fruit emits various gasses as it ripens. Our customer wanted to understand this process, and so desired a system that could sample the air around a fruit container a number of times over the course of the voyage.

The solution was to fill (empty?) 30 aluminum containers (ex-fire extinguishers) with a vacuum. A valve on the top of each opened for a few seconds at programmed intervals to suck in the ambient environment. We quickly concluded that only an 1802 could stay alive for the 30 required days, running from a small lead acid battery.

The CPU ran a simple program that accepted timing parameters, and that then counted time till it made sense to open a valve. As I recall we ran the 1802 at about 100 hertz to conserve power.

The data collection program was a great success... until a dockworker dumped a 50 ton container on the instrument.

In the early embedded days we debugged code using text-based tools. Set a breakpoint and the terminal (or, earlier, the TTY) would print a character when the program stopped. That morphed into IDEs using windows, a huge improvement, but we're still interacting with the project via a mouse and keyboard. Thirty years ago I dreamed of a debugging tool that would let an engineer manipulate firmware via a VR headset. Grab that interrupt vector and reroute it. That never happened.

A new tool for engineers working on PCB hardware, though, is somewhat available (somewhat, as the free, limited, version is out, but it appears the pro versions have not yet been released). It's called inspectAR, and utilizes augmented reality.

The website is a bit light on details, but watch the second (3 minute) video on the home screen to get a sense of what the tool can do. It's pretty cool!

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Elephant: (n) A mouse with an operating system.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.