Embedded Muse 187 - January 4, 2010

You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go to https://www.ganssle.com/tem-subunsub.html or drop Jack an email at jack@ganssle.com.

EDITOR: Jack Ganssle, jack@ganssle.com

Contents

- Editor's Notes
- Quotes and Thoughts
- Restoring From Backups
- More on Corrosion
- Ah EPROMs
- Tools and Tips
- Jobs!
- Joke for the Week
- About The Embedded Muse

Editor's Notes

Did you know it IS possible to create accurate schedules? Or that most projects consume 50% of the development time in debug and test, and that it's not hard to slash that number drastically? Or that we know how to manage the quantitative relationship between complexity and bugs? Learn this and far more at my Better Firmware Faster class, presented at your facility. See https://www.ganssle.com/onsite.htm .

Happy New Year to all! I wish you peace, prosperity, and happiness in 2010.

Quotes and Thoughts

The real value of tests is not that they detect bugs in the code but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code. C.A.R. Hoare.

Restoring From Backups

Walter Banks conducted an interesting experiment in reloading old backups: "Twenty years ago or so we like a lot of companies we worried about the reliability of using 3 1/2 inch floppy disks as a back up media. We were also worried about the demise of 720K drives and kept a couple around.

"A few days ago I needed to get some files for a customer that were about 10 years old. These archive files were actually duplicated on back up mag tapes and daily floppy disk dumps. Backup tape drives on working systems are rarer than paper tape systems. (But we actually have a mag tape system that has been booted within the last year). I tracked down the daily dump fairly quickly and decided that in these days of terabyte drives we should just restore all of the old dumps and keep them on line.

"I read about 400 3 1/2 inch floppies both 720K and 1.44M all on a Windows 2000 machine with a 1.44M floppy drive. The files on the disks were all created in the 1990's. Drive software has over the years improved as well. Most of the files we were reading were zip files typically 600K to 1.3M long with occasional plain text files. In this process there were 4 hard errors. One was on a 720K disk - no real surprise. Three of the errors were on 1.44M disks all written within a 6 week period. One of the three errors was on a file that was 3K long and a 1.1M zip was readable without errors on the same disk. The underlying cause on the three errors may have been media (there were likely from the same batch) or a bad original drive.

"We lost no data because the back up scheme we used even for the daily dumps had enough redundancy even without the additional tape backup's to recover all the files. Age really has not changed the reliability data on the floppy drives significantly. Reading the same set of disks in 2000 probably would have had similar if not more errors.

"Being able to restore daily files from 15 to 20 years ago, comforting, looking at those primitive files, scary."

More on Corrosion

Phil Ouellette had some thoughts about corrosion in switch contacts: "4. A more hardware-intensive solution: alternate the voltage on each key - no fixed GND on one side. This will minimize the corrosion."

"This comment about how to prevent corrosion in switches has wider applicability than just with carbon switches. One of my customers had a problem with silver migration in a membrane keyboard. The silver whiskers were created by a constant DC bias across the switch membrane and after a year or so of use began to create intermittent problems with keys being stuck on. The solution was to toggle the polarity across the switch in-between each keyboard scan to prevent (and reverse) whisker growth. I'll bet this technique could also be used to reduce tin whisker problems in lead-free PCBs as well.

"This is a good reason to strobe switches instead of just using a simple pull-up resistor with the switch grounding the input. In my case I reconfigured the keyboard inputs to be outputs whenever I wasn't actively scanning the keyboard. After each scan of the keyboard, I reversed the polarity across the switch during the idle time. This software change actually healed flaky keyboards in the field, eliminating the need to replace keyboards. Since the customer had several hundred thousand units in the field, a software fix was a very desirable solution."

Ah, EPROMS

Paul Tiplady has a story that could be in the Jokes section, except that it's true: "The review about the debugging book, and your comment in the middle about diagnosing problems over the phone being especially challenging, brought back a quick story you might enjoy.

"Many years ago (in the eighties) I worked for a small outfit in the UK making battery-powered EPROM programmer/emulators and cheap (but effective) eight-gang programmers. My job at the time involved telephone sales & support, testing, repairing, packaging, shipping... (it was a SMALL outfit!).

"One day a customer rang me up and asked me why the devices he'd just programmed don't work. He explained that they're windowed EPROMs (probably 2764s), and he knows they programmed properly because the little lights inside each one of them came on! It turned out that he'd inserted them all upside down and thoroughly fried the innards, but since he'd done the same with the master, the programmer told him they were all the same (which they were, of course -- all broken). Not all tech support issues are this easy to solve, unfortunately.

"Yes, you could complain that the programmer shouldn't have destroyed them all. But as I said, it was a very low cost device, and very effective when used properly. People didn't tend to make that mistake twice!"

Tools and Tips

Kalpak wrote: "Just tried out the demo/ trial version of Crystal revs from http://www.sgvsarc.com/ a tool for generating flowchart from C/ C++ code. It a great tool, as they advertise, for understanding legacy code. And some versions also have a MISRA checker. It is also great for review or QC, to see if the design has been followed in coding.

"So I was hoping that it would also allow forward flowcharting, that is, I can use the same for creating flowcharts instead of a regular flowcharting only tool . But it cannot be used like that.

"I am wondering if this tool will be used principally by teams that avoid proper design in the first place and try reversing code to flow to figure out what have they actually done? Will it encourage such bad habits?

"Another interesting aspect; now this tool can be "rented", that is any version can be bought for a limited period of time ( in multiples of 1 month). This will be handy for team leads who are unable to convince the management of the value of the product."

Cristina Ionescu like Eclipse: "The Eclipse IDE for C/C++ developers, currently at Galileo version, has been quite useful for the team I am part of. It offers a huge amount of plugins for various uses. Just with the CDT (C/C++ Development Tools) environment you have a great tool. In parallel, installing Cygwin on the computer allows setting the Eclipse C project toolchain to Cygwin (others may be used as well). One other part of our own made tools and scripting is in Python, and the PyDev plug-in is perfect for having both editor and interpreter through the IDE. SVN subclipse plug-in allows direct SVN connection for the projects. Workspaces can be created where several project contents can be compiled for different targets. Make targets are associated to projects. Etc. etc... a simple download and look at the IDE will show a lot of what it can do. Using this and Doxygen has taken a huge strain off developers in my team because before they used to have several tools. The Eclipse IDE has several versions for Java, PHP, C and other types of projects (being written in JAVA itself). http://www.eclipse.org/downloads/ .

"That's about all I can say without getting into details for everything. I have used Dev C++ and Microsoft Visual C++ in the past, but I really like Eclipse. It is indeed more complex and for simple projects it's obviously to complex, but for a large development team, it relieves the number of actions to take to create a target once the whole environment has been installed as desired."

Ken Smith recommends two tools: "My favorite productivity tool of all time is GNU screen. At a minimum, it gives you tabs in your xterm or other terminal emulator so you can have multiple virtual xterms in a single window. You can detach a screen and the processes that are running in it will remain running. For example, you can detach your screen at the end of the day, drive home, and reattach and you'll see the exact same window that you were looking at before. This is great for checking on long running processes. It can also connect to serial ports. Eg `screen /dev/ttyUSB0 115200` will open a screen window on your serial port USB0.

"The second tool I can't live without is tcpdump for debugging network issues. It's like wireshark or ethereal but it is strictly a command line tool. It links with libpcap and can exchange files with other tools that support that format."

Don Peterson had several recommendations: "gvim prints syntax-highlighted code on color printers (i.e., no need to go find another tool). You can also convert the syntax-highlighted file you're viewing into HTML for other uses.

"I used to use the GNU indent tool constantly on C code, but it didn't work well on C++. Then I found artistic style: http://astyle.sourceforge.net/ which works on C/C++/C# and Java.

"I don't have a copy of Adobe Acrobat, but I sometimes need to make simple changes to a PDF file. The stapler.py script from http://github.com/hellerbarde/stapler is capable of concatenating, splitting, selecting, or deleting pages from PDF files. You need python and a PDF python library. Very handy (and free) when you need it.

"Everyone has their favorite editor, so I won't make any recommendations. I do, however, have something to say about your editor's use. I've often been appalled at how poorly some engineers know their tools. I've watched experienced engineers use vim by hunt-and-peck typing and retype very long command lines in a shell without using the shell's history mechanism (it's enormously frustrating to collaborate with or help such people as it's like working in cement). Your job is to do things rapidly and correctly, so any journeyman should know his tools intimately (imagine a carpenter not knowing how to use a circular saw). Here are some suggestions:

"* Devote an hour or two of your time per week to furthering your knowledge about the key tools you use. Force yourself to learn to better use features you know about or features you don't use. Do it on your own time at home if your work won't pay you for it.

"* If you're a Windows weenie, devote some effort to learning UNIX (you can have a UNIX-like environment on your Windows box by getting cygwin). If you're a UNIX weenie, learn more about some of the tools available on Windows/Mac. Ignore the religious arguments of the immature -- find what's good and use it when needed.

"* If you work in a shell-type environment, master your shell's capabilities. There are lots of references to help you do so and you have no excuse but your own laziness. Your effort will be repaid many times. When I'm a release manager, I like to use a script that generates the whole package including a data file (check it in after the release is made) containing MD5 or SHA1 sums of the files used, the file names, revision/branch numbers, config spec, and whatever else is needed to recreate a package later. You'll also learn (through bitter experience) to check in the compiler and tools you're using to build your software products. Your product support people will thank you later for this foresight.

"* Also master your revision control system -- you'll eventually need to do some sophisticated things or recreate/troubleshoot something from the past. You'll be glad you have that knowledge when you need it.

"* Go through Jack Ganssle's tools page https://www.ganssle.com/tools.htm and familiarize yourself with the tools people talk about there. Also search out tools on Freshmeat, Sourceforge, etc.

"You can tell when your knowledge/experience levels with the tools are appropriate -- people start coming to you for help with the tools. Senior engineers can spend 50% or more of their time helping others with problems -- and get paid/recognized well for it.

"My sister turned me on to Zotero, a bibliographic plugin for Firefox. It lets you capture web page content and save it on your computer and later create a bibliography from the things you found. You can choose from a variety of bibliography formats. An important subtlety is that you have the web page's content on the day you saved it -- important for sites that have dynamic content. I used the Scrapbook Firefox plugin for a few years, but found Zotero a better tool in most ways.

"I've been a huge fan of python for over 10 years and it just keeps getting better -- I consider it the best general purpose programming language around. It has a huge standard library and there are thousands of solutions available for free on the web (chances are someone has already solved the problem you're interested in). For numerical work, numpy (an array processing library) is indispensable. For scientific stuff, SciPy provides a lot of different tools. matplotlib lets you create many types of plots. Writing platform independent C/C++ code is a lot harder than you think it is; it's much easier with python. When python isn't fast enough, you can write the slow stuff in C/C++ and use it as a python add-on.

"By the way, one of the python libraries (os) contains a function called walk() that lets you walk a directory tree in either top-down or bottom-up order. You can use this to make all kinds of utilities. Here's one. Since python uses a regular expression syntax similar to perl's, you can write some fancy recursive grep-type tools pretty easily.

"xref.cpp, a token cross-referencing tool (get it at http://www.gdssw.com/tools/xref.zip -- it's packaged with a Windows executable compiled with g++). It prints out or indexes the tokens in files and will also do spell checking of tokens -- useful if (like me) you like to see variable, class, and function names made from complete words. Should compile with any handy C++ compiler. I used it a lot for indexing and spell checking when I was a working programmer, but now I mostly use it with the -t option to grab tokens from a file.

"I read the comments about PolySpace and had a similar experience with another product in the mid-1990's (alas, I don't remember the name, but it was from a company in Oregon). I had big hopes for the product (a code analysis tool) to help some of us new-to-the-division engineers understand the existing code. The code was a million lines of legacy C code and it brought the tool completely to its knees. I spent a fair bit of time working with the company trying to get things to work, but eventually gave up. I can say good things about the company and their efforts, but ultimately they weren't successful and we abandoned the effort. The lesson I learned is: if you're considering adopting a code analysis tool, throw the biggest, baddest chunk of code you can at it and test it thoroughly. The developers of such tools probably don't have access to such large, legacy systems and haven't tested their tools in quite the same way you can. And don't be overly optimistic about the outcome.

"I've used a few other analysis tools and some are pretty useful (e.g., memory allocation checkers for C). But the best static analysis tool, in my opinion, is and will remain a careful code review by dedicated, thorough, motivated, and experienced engineers. It can all be done by just reading the code. Sadly, it's still an underused and undervalued technique. (And I'm _not_ referring to the company-mandated formal process with 10 or 15 people in a room -- you only need two or three other engineers to read your code and give good feedback, as long as you prepare them properly with the things they need to know.)

"In a similar vein, I also recommend two other techniques:

"First, build your code every day with an automated system and have it automatically tested with its regression test suite. Every component/system has a responsible engineer and any regression test that fails automatically emails the responsible engineer (and other interested parties). Fixing a broken chunk of code on the MAIN branch of the code tree is automatically the highest priority task for the responsible engineer. Yes, it takes some work to architect and implement such an environment, but it will pay for itself many times over and you can use it on subsequent projects. Oh, and merging to MAIN _must_ be done only after running the test suite and showing that it doesn't break MAIN. Then, when someone else finds main broken, it's egg on the face of the engineer who checked that code in. Most of us don't like such negative attention and we learn to correct our own processes quickly to avoid such attention in the future. This becomes even more important when you have a codebase that's being worked on by people all over the world and in different cultures.

"Second, your code build should use every checking feature of your compiler (as others have already mentioned). Pay attention to every warning; even better, treat warnings as errors. If you have to ignore a warning, put a pragma in the code for it _and_ document why it is being ignored with comments and who made the decision to do so. I've worked on projects where there were hundreds of warnings and it was impossible to get to such a state because no one wanted to devote the time to fix the problems causing the warnings. Guess what happens: an important warning alerting you to a problem in the code gets lost in the noise. How stupid!

"Debuggers: (This is going to go against the grain of some folks!) Debuggers are evil -- learn how to figure out where your code's problems are without using a debugger. This was the statement an instructor I had -- and many people think it's heresy. But consider... Every debugger I've ever used has failed me at some point for some reason -- and they seem to fail when you need them most. The challenge is to find other techniques for debugging, such as printf's, signals, shared memory, the file system, another thread, etc. -- whatever tools are handy. This knowledge and practice will serve you in good stead one day when you simply don't have a debugger available -- in fact, I've worked on embedded systems where you would have indentured your oldest child to simply have a printf to help you understand what was going wrong (this was in the startup code of a well-known product family most of you have probably used).

"As I've mentioned elsewhere here, often the best tool is simply a careful code review or detailed reading of the code. OK, really, for embedded systems I'm not going to recommend that you never use a debugger -- some things are just too hard to figure out without one. But the instructor's advice has been validated in the wild: don't get too dependent on a debugger.

"Debuggers are also useful is to step through your code and watch how it executes. McConnell's _Code_Complete_ is one reference that recommends this (you've read this book, right?) and there are others. You can also use the debugger to help execute seldom-used code paths, such as exception handling."

Joke for the Week

Tjark van Dijk sent this gem:

In the beginning was the plan,
and then the specification;
And the plan was without form,
and the specification was void.

And darkness
was on the faces of the implementers thereof;
And they spake unto their leader,
saying:
It is a crock of crap,
and smells as of a sewer.

And the project leader
spake unto his section head, saying:
It is a container of excrement,
and it is very strong, such that none may abide it.

The section head then hurried to his department manager,
and informed him thus:
It is a vessel of fertilizer,
and none may abide its strength.

The department manager carried these words
to his general manager,
and spoke unto him
saying:
It containeth that which aideth the growth of plants,
and it is very strong.

And so it was that the general manager rejoiced
and delivered the good news unto the Vice President.
It promoteth growth,
and it is very powerful.

The Vice President rushed to the President's side,
and joyously exclaimed:
This powerful new software product
will promote the growth of the company!

And the President looked upon the product,
and saw that it was very good.

About The Embedded Muse

The Embedded Muse is a newsletter sent via email by Jack Ganssle. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.