You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• New Year's Resolutions for Firmware Developers
• Principles for Interfacing With Humans
• Just How Big is a Million Lines of Code?
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here and here.

Engineering changes at an accelerating pace. The boss wants more bug-free code, today. Tomorrow he'll want even more in the same time-frame. Are you on top of the best ways to deliver it?

Some tell me they just don't have time to attend seminars. I'm reminded of this:

It is important to fight today's engineering battle. But the war will be lost if you don't find better ways to win.

This is what my one-day Better Firmware Faster seminar is all about: giving your team the tools they need to operate at a measurably world-class level, producing code with far fewer bugs in less time. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends January 20. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Jack's latest blog: How I Write Code.

A recent article in Electronic Design spotlights crashing DRAM and NAND prices. The latter are off 79% from their mid-2017 numbers. But a stunning graph later in the article shows that 1 GB of DRAM cost around $70k in 1991! It's around $3 today. I don't know how accurate that $70k number is, but even if it's off by an order of magnitude that shows astonishing progress in electronics over the last three decades.

Coding is "90% finished" for half the total coding time. Fred Brooks

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

John Carter likes the D language, and he also included a useful link to Stroustup's and Sutter's C++ guidelines:

Interestingly, D does not support 8 or 16 bit processors. From the overview page: "No consideration is given in D for mixed near/far pointers and all the machinations necessary to generate good 16 bit code. The D language design assumes at least a 32 bit flat memory space and supports 64 bit as well." I assume that means no MMU or MPU, which is a shame, as these resources can be useful in preventing complete system crashes.

This month's giveaway is a Cypress CY8KIT-044 PSoC 4 M-series Pioneer kit.

Enter via this link.

Percepio is giving away one-year licenses for their TraceAlyzer debugging tool. I've played with it and find it a great way to visualize how your code interacts with the RTOS (many RTOSes supported). The giveaway is not exactly free - you have to send a description of what you'll do with it, and follow-up with a story about solving the problem. More info here. I image the story could be pretty short.

Not a freebie for Muse readers, but here's an astonishing giveaway: a business card that runs Linux. Production cost: $3 in low volumes. Linux for three bucks? For all the sturm and drang in the political news, sometimes we need to take a step back and just be amazed at how cool things are. Thanks for Martin Buchanan for sending the link.

Happy New Year!

I thought it would be fun to Google for New Year's resolutions for software people. Most of the results were things like "share more," "work with the community," "contribute to open source," "cut back to one quart of whiskey per day," and the like. Nice thoughts, but what technical resolutions are worthwhile?

First, I don't like aspirations like "work harder at..." That's impossible to measure. Without numbers, we're lost in a fog. Without numbers, we can't understand if we're improving. Without numbers, we don't know how we measure up against others (e.g., am I in the bottom quartile of developers?). And, as Harry Roberts demonstrates in Quality is Personal, "trying harder" rarely works.

So here are some thoughts:

• Aim to elicit x% of a project's final requirements. (Requirements churn is pretty easy to measure, either with a spreadsheet or a tool like Jira.)
• Solicit criticism. Have a trusted mentor review x of your best modules.
• Achieve x% test coverage (more about this coming next issue of the Muse).
• Eliminate x% of defects pre-test. (Testing will never be adequate by itself. Most test schemes only exercise half the code. Tons of data shows how test is just not enough, or not even possible.) Our goal should be to write correct code, not to beat poor code into submission.
• Find x hours per day for uninterrupted work on one project. (Multitasking on three projects consumes 40% in context switching time. Each interruption costs 15 minutes in overhead).
• Achieve an x% comment density. (Plenty of tools measure this. Best-in-class code has 60 to 70% comment density. That is the percentage of non-blank lines that are, or contain, comments. Plenty of data supports this, such as this. Many argue that comments aren't needed, but after reviewing oceans of code I disagree).
• Reduce shipped defects to under x% (measured as a percentage of all defects found in engineering up to 90 days after shipping). Only 0.4% of companies remove more than 99% of all defects pre-shipping.
• Read x technical books per year. (The average developer reads one. For book recommendations see this. I take notes about the most important points when reading tech content.)

John Carter sent an excerpt from Standard Practice for Human Engineering Design for Marine Systems, Equipment and Facilities. These notions should be embraced by all engineers. Humans are flawed, which this standard addresses explicitly.

But first, a story. I read (somewhere recently, maybe in the Smithsonian's Air and Space magazine) that during WWII B-17s crashed on landing far more often than one would expect. Turns out, the gear and flaps levers were located near each other and looked similar. Tired, stressed pilots would mix up the controls. Relocating and changing the appearance eliminated those sorts of accidents. The moral: design with real people in mind.

Here are the practices:

4.2 Principles of Human Behavior:

4.2.1 There are basic principles of human behavior that control or influence how each person performs in their workplace. Some of these behaviors are culturally derived, while others are general and uniform across all cultures and geographical regions of the world. These behaviors influence a person's physical, social, and psychological approach toward the work they do and how safely they do that work. Failure to satisfy these behavioral principles in the design of a ship or maritime structure can encourage, or even coerce, maritime personnel into taking unsafe risks in their everyday activities. It is, therefore, imperative that designers of ships and maritime equipment, systems, and facilities know these principles to provide a safe and efficient workplace for maritime personnel.

4.2.2 These principles include:

4.2.2.1 If the design of the ship or maritime facility is considered to be unsafe or inefficient by the crew, it will be modified by the users, often solving the initial problem but introducing others that may be as bad, or worse, than the original.

4.2.2.2 Equipment design shall be such that it encourages safe use, that is, does not provide hardware and software that can be used in an unsafe manner.

4.2.2.3 If the equipment or system is not designed to operate as the users' cultural and stereotypical expectations lead them to think that it will operate, the chance for human error is significantly increased.

4.2.2.4 If equipment or systems are perceived by operators/maintainers to be too complex or require more effort to operate or maintain than they believe is necessary, they will always look for a "shortcut." Further, this "shortcut" may be perceived as being safe when it is not.

4.2.2.5 No amount of training, company or organizational policy, threats of retaliatory action, warning notes in a technical manual or training guide, or pleading with personnel to be safe on the job can overcome poor design that encourages, leads, or even coerces personnel into unsafe acts on the job. The most efficient way to prevent unsafe design from contributing to an accident is to eliminate the unsafe design.

4.2.2.6 Equipment users tend to be very unimaginative when it comes to identifying unsafe features and they do not visualize the consequences of unsafe acts. Therefore, do not expect that an "obviously dangerous" task will always be recognized as such by every user.

4.2.2.7 Designers shall consider the possibility for human error and design equipment so that incorrect use (deliberate or accidental) will result in little or no harm to the user.

4.2.2.8 Equipment operators and maintainers tend to make guesses as to what a label, instruction, or operational chart states if it is not complete, legible, readable, and positioned correctly.

4.2.2.9 Designers and engineers shall never use themselves as the standard against which a particular design is evaluated. People come in many shapes, sizes, mental capacities, and capabilities. Therefore, design for the full range of potential users, physically, mentally, and socially.

4.2.2.10 People shall be protected against themselves. Designers cannot create an unsafe piece of equipment or system and expect the users to assume full responsibility for its safe use.

4.2.2.11 Ease of equipment maintenance affects the equipment's reliability, that is, the harder it is to be maintained, the less it will be maintained.

4.2.2.12 Equipment designed to require multiple operators working together simultaneously increases the likelihood of operator errors.

4.2.2.13 Operational/maintenance procedures shall be clear, definitive, and comprehensive, otherwise, they will be misinterpreted or ignored.

4.2.2.14 Structural items such as piping, cable trays, or any other item that appears strong enough to be used by a person to hold onto or stand on, and is placed in a convenient location to use for that purpose, will eventually be used for that purpose.

4.2.2.15 Humans expect consistency in the design and arrangement of their workplace. Therefore, if that workplace, or any part thereof, appears in more than one place in their work environment, it is expected to be located and look the same way at every location.

4.2.2.16 When controls and displays associated with particular pieces of equipment are placed on a console or control panel, they shall be located on that console or panel to replicate the actual location of the equipment on the ship or structure as both are viewed by the operator. Therefore, equipment that is to the operator's left as he/she faces the control station shall appear on the left of the control panel or console, and equipment to the right shall appear on the right side of the console or panel. This "spatial relationship" between the real world and the controls and displays that are associated with the equipments and systems of that world is extremely important in the design of ships and maritime structures.

4.2.3 Humans develop behavioral patterns based on their cultural experiences. Designing a ship or structure that ignores or violate those culturally derived behavior patterns will inevitably lead to human error.

A million lines of code. It's a number bandied about more than ever as software sizes develop overactive pituitaries. Some cell phones use upwards of twenty million. Last year Ford announced that the F150 truck has 150 million. Everett Dirksen once may have said: "A billion here, a billion there, pretty soon you're talking real money." Well, a million lines of code here, a million there, pretty soon you're talking about a program that is as mind boggling and incomprehensible as our national debt.

A million lines of code printed out would be 18,000 pages. That's a stack six feet tall (on typical 20 pound paper). Ironically, the listing weighs in at 180 pounds while the actual operating code is mass-free; it'll live in a fraction of a gram of silicon. Like DNA, code's human-readable description requires tremendously more mass than its actual instantiation.

A million lines of code is probably on the order of 20 million instructions, or around half a billion bits. That's not far off of the 3 billions base pairs in human DNA. Unlike DNA, which has redundancies and so-called "junk" sequences, every single bit in the code must be perfect. A single error causes greater or lesser failure.

Since a typical atom is around 0.3 nm in diameter, if one had as many atoms lined up as the number of instructions needed for a million lines of code, they would stretch 10 cm. That many Ebola viruses would stretch 15 meters.

A million lines of code is as long as 14 copies of War And Peace, 25 of Ulysses, 63 copies of The Catcher in the Rye, or 66 copies of K&R's C Programming Language.

A million lines of code is not ten times more than 100,000. It's well-known that schedules grow exponentially faster than the code. Barry Boehm estimates the exponent is around 1.35 for embedded software. So the schedule for developing a million lines of code is 22 times bigger than for 100,000 LOC.

In the March, 1996 issue of IEEE Computer Watts Humphrey published crude rules of thumb for estimating software projects. Though hardly scientific, they do give a sense of scale. Using his estimates:

• A million lines of code requires 40,000 pages of external documentation.
• A million lines of code will typically have 100,000 bugs pre-test. Best-in-class organizations will ship with around 1k bugs still lurking. The rest of us will do worse by an order of magnitude.
• A million lines of code will occupy 67 people (including testers, tech writers, developers, etc) for 40 months, or 223 person-years. Darwin needed just 1.5 person-years to write The Origin of the Species. Scale that to the 26 copies equal in length of a million lines of code, and it appears writing code is some 6 times more time-consuming than writing a revolutionary scientific tome.

A million lines of code costs $20m to $40m. That's the cost of an F-18 jet, a thousand cars (in America),  ten million gallons of gas, seven times the inflation-adjusted cost of the ENIAC, and something like ten million times the cost of the flash chips it lives in.

Think about that last analogy: Ten million times the cost of the flash chips. Accounting screams over each added penny in recurring costs, while chanting the dual mantras "software is free," and "hey, it's only a software change."

Hex-Five Security's MultiZone product is a software layer that uses hardware to enforce high security when using untrusted code. Currently RISC-V versions are available.

It's a small (under 2 KB) bit of code that is formally verifiable. That's a big selling point when pursuing a Common Criteria security standard. As I understand it, the product enforces separation zones to keep bad actors out of others' sandboxes, which sounds like the Common Criteria's separation kernels.

A datasheet is here. It's free for non-commercial use.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of pi change. - FORTRAN manual for Xerox Computers

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.