You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• µC/OS Goes Open Source
• Salary/Coronavirus Survey Results
• Failure of the Week
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

In Muse 395 I ran a series of pictures of temperature displays showing totally absurd readings. A lot of readers responded, so with this issue I'm introducing a new section entitled "Failure of the Week." This will feature amusing, frustrating, and ridiculous embedded failures. I have a huge collection of these. Many have lessons we should learn, though too many of those are the same lesson: Check your goesintas and goesoutas.

Jack's latest blog is a guest post: Problems In Ramping Up Ventilator Production

Eric Smith sent this: A Tony Hoare quote that is perhaps more applicable to embedded system programming: "Later, we asked the customers whether they wanted the option to be able turn off the type checking in production. It's a bit like wearing a life jacket when you are practicing drills, but then taking it off the ship was sinking. The customers decided to not switch off the type checking."

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Rod Philips won last month's micro:bit.

Firia Labs is donating one of their CodeBots (a Python-enabled robot with a host of sensors) for this month's giveaway.

Enter via this link.

Most readers probably know of Micrium, the company whose µC/OS RTOS and related products have been very popular in the industry. I've known Jean Labrosse, the founder, for decades and have long admired his company. And I've long admired the code base, which is incredibly clean. How clean? It is certifiable to numerous standards, including the rigorous avionics DO-178C.

Micrium was acquired by Silicon Labs a few years ago, but very recently that company has pretty much abandoned the Micrium product line. µC/OS-II and µC/OS-III and many of the other parts of their stack are now open source. Dozens of processor families are supported. The source code is here.

One of Micrium's marketing strategies was to sell a number of well-written deeply-technical books about the products. I always liked the fact that some are tailored for particular processors, so a reader can immediately run example code on specific platforms. PDFs of these are now free.

Support for the product line is now available from Weston Embedded Solutions in the USA and Embedded Office in Germany.

Validated Software is one company that provides certification packages for safety-critical applications. In the avionics world that's DO-178C, and the Micrium products are certifiable to level A, the most stringent. From their website: Ninety percent of the µC/OS Validation Suite is "pre-existing" and has been used for "previously certified" software. I presume that as long as users don't monkey with the code this statement will remain valid.

And not monkeying with the code is a feature Weston Embedded seems to embrace. From their website: Cesium RTOS [is] a commercially licensed RTOS that is derived from Micrium's uC/OS products and completely free from modification by open source contributors. An unusual statement in our times when open source has attained religious stature, but if certification to safety-critical standards is important, one doesn't want every Tom, Dick and Jill changing things.

Rolando Ingles of Weston Embedded told me that Cesium is the Micrium RTOS stack. The company will continue to maintain that code and extend it as needed, while maintaining the clean code standards of the predecessor company. Micrium will be honoring support agreements till the end of the year. Customers will come to Weston Embedded or Embedded Office for licenses, support and consulting (e.g., creating new BSPs, ports, device drivers and the like). Interestingly, he says some customers have bought licenses instead of getting the open source code as they feel that approach is cheaper than getting OSS through their legal department.

This is a time of great change in the RTOS business. Amazon acquired FreeRTOS. Microsoft bought Express Logic (ThreadX and related components). The products are easier to use than the old days; SEGGER has a demo that can get a novice running their RTOS in under 5 minutes.

Do you need an RTOS? Not all projects do. But this is an area where careful up-front design is critical. In the very early 1980s I designed an instrument that had quite a few I/Os, figuring it would be easy enough to handle everything via polling and some interrupts. A combination of requirements growth and my poor planning meant that I eventually had to shoehorn an RTOS into the (now extant) code. That was a nightmare that bloated the schedule... and taught me the importance of good design.

I'm often asked how does one know when an RTOS is needed. There's no generic answer, but if the system needs to run multiple independent activities, it's time to consider adding one. An example: if a display needs to be updated at some rate, inputs monitored, communications is going on, an OS might be a good choice. Sure, one can handle these activities with a simple interrupt-driven design, but as the complexity grows that can get out of hand.

Some engineers avoid OSes because they worry about determinism. We know that under all the wrong conditions, like a perfect storm of interrupts, preemptive multitasking can fail to schedule all of the tasks. And multitasking can create problems with locks and priority inversion. These are legitimate concerns, though there are solutions to many of them.

Last month I did a salary survey and here are the results. First, average income (estimated by respondents including benefits) by region:

In the USA here's salary by years of experience:

There wasn't enough data to create meaningful years of experience vs income for other regions.

Worldwide, we're, on average, 42 years old with 19 years of experience.

What's with the dearth of 45-49 year old engineers? The IEEE has long complained that for unknown reasons EEs leave the field mid-career, and I guess we're seeing that effect here. Probably those that remain engineers later in life are culled because the just love the field. In fact, on a scale from 0 to 3 (from "hate it" to "love it") the average happiness with our careers is 2.35, about in line with data from previous years' surveys.

49% of respondents expect a "Strong demand for engineers." 6% expect the demand to diminish and 43% expect the demand to remain unchanged.

54% of consultants report business is about the same as last year; only 14% report it getting better. The rest are seeing business trail off. Consultants' average rate is $104/hour, but the range is huge: $50 to $330 with a median of $82.

What about the effect of the novel coronavirus? (This data is from the first three weeks of April. As this situation is evolving rapidly the numbers may, too). 2% have lost their jobs. Another 2% are not working and not being paid. Only 1% are not working but continue to draw a salary. 13% are still going to the office every day, which leaves a whopping 82% of us working from home. We're fortunate to have a career that allows such flexibility.

Many people left comments. A few of the more interesting ones follows:

• A trend I've noticed in my small corner of the world is moving embedded systems away from custom designs to embedded computers (embedded Linux/Windows IoT/Raspberry PI like boards) with FPGA handling the low-level hardware/firmware interface, and the firmware becoming more software-like.
• Embedded software development does not have the sex appeal of machine learning or data science or mobile/web application development -- the stuff we do exists largely behind the scenes and you only even know it is there if something goes wrong -- which makes it tough to find recent college grads with an understanding of what we do, let alone an interest. Arduino is a blessing and a curse in this sense -- we've had candidates claiming to have expert knowledge based on some course/hobby work with Arduino kits, but no real idea of how a microcontroller actually works. , ,We've modified our approach (I work at a medical device company) to seek out strong _engineers_ that might not meet the skill requirements, but have the desire to learn and grow, and mentor them to build the missing skills. This is higher up-front cost with longer time to payoff, but it seems to be the easier route for us at the moment.
• I actually find myself being more productive working from home. In fact, I start earlier and stop later. The only bad part is not having lab time with my Techs.
• I think this Coronavirus will have a big impact on how things work in the embedded world, I really hope that managers embrace remote working and automation. In my current project all of the tools are manual because nobody expected to work remotely, so I hope this could be used to change that mindset and start working more efficiently by allowing time for automating tools.
• I work for a company that makes medical equipment, so things are currently (for all the wrong reasons) going pretty well. It is very good to see on of my devices being used to save lives.
• I'm the exception to about every rule there is. I graduated college at the age of 45 with the ""wrong"" degree (physics) and have still managed to succeed in this business. I've never had trouble finding a job. I've seen the age discrimination, but there are lots of companies who need engineers and don't care how old you are. They care that you a) know what you are doing and b) intend to keep it that way.
• Working from home during the current coronavirus pandemic, if find that the transition from being in the office, surrounded by my co-workers, has been much easier that I ever thought it would be.
• In this time, I've been building a new product and taking advantage of the downtime. It won't be like this forever. I'm learning new programming languages and furthering my abilities the best I can, prepping for my return to work.
• Looking back on my career as I near retirement, I'm very happy I chose this career. I've seen some cool stuff over the years. However, I'm not sure I'd recommend the same career to younger folks. It seems few companies have any respect for your person time and family life.
  

Thanks to everyone who participated.

Ed Criscuolo sent this: A man went to an ATM to withdraw some money, and found a "glitch" changed his balance from $13 to $8.2 million. The story is here.

I've been following Eta Compute for a while. Like some other vendors, their interest is putting a little machine learning capability at the "edge" - in a microcontroller right at the sensors. What makes them a bit different is their focus on doing this at extremely low power levels.

Eta's new ECM3532 is an MCU with a Cortex M3 and DSP with dual MACs. The latter, of course, is optimized for multiply-accumulate operations, which are at the heart of inferencing. The company claims (I've seen their eval boards running off 1 cm² solar cells) power consumption of under 5 µA/MHz, which is around an order of magnitude better than most ultra-low-power MCUs. Instead of dynamic voltage and frequency scaling, they tout continuous voltage and frequency scaling, with a near-threshold low-end Vdd of 0.54 V.

Near- and sub-threshold operation yields very low power operation at low frequencies. Ambiq is one of the pioneers in this area. FETs can operate in three regions: subthreshold, linear, and saturated. But in the subthreshold region leakage is a problem, and temperature even more so. Transistors are not well characterized at those voltage levels, so vendors resort to clever, often secret, tricks. I'm told digital watches, at least the non-smart versions, operate in the subthreshold region giving them tremendous battery lives. For more on subthreshold voltage operation of FETs see this.

One of those tricks Eta uses is a non-synchronous architecture. Though there is a clock, the logic isn't conventionally clocked. It's self-timed. Synchronous circuits need plenty of margin to ensure all of the timing is properly closed; self-timed versions report back as soon as an operation has completed so the next can begin. Absent this, since FETs operating in or near the subthreshold region can have wildly-varying characteristics depending on temperature, huge setup and hold margins would have to be designed in.

Why would anyone want an inferencing engine at the edge? Potential uses include listening for wake-up words (think "Hey Google"), sensing glass breaking, and even a bit of image recognition. Local processing means less bandwidth to the cloud, and a lot less power consumption as comm protocols can drain batteries quickly.

Samples of the ECM3532 are available today. In production volumes they will be under $5.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

And 1.1.81 is officially BugFree(tm), so if you receive any bug-reports on it, you know they are just evil lies. - Linus Torvalds

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.