Category Archives: Troubleshooting
(Photo from “Oklahoma Jury Finds Toyota Liable For Sudden Acceleration Fault; Awards $3M In Damages,” by Arjun Kashyap, 25 Oct 2013)
Back in 2009 this newsletter started to express skepticism about Toyota’s insistence that unintended acceleration in their vehicles — in some cases resulting in fatalities — was due to a floor mat that could cause the accelerator pedal to stick. (Those earlier posts are listed at the bottom of this post.)
The reasons for the skepticism were (a) the number of reported cases was relatively high compared to other non-Toyota models, (b) there was at least one case where it was clear that a stuck pedal could not be the cause, and (c) the investigation by the National Highway Traffic Safety Administration (NHTSA) appeared to have some major flaws. Also, DACI’s own experience in investigating several low-probability events has convinced us that customers who report such problems tend to be too easily dismissed, rather than receiving the respectful assumption that they are honest and observant and reporting the problems carefully; i.e. the sudden acceleration stories related by some Toyota owners were not consistent with the floor mat hypothesis. Finally, it didn’t help Toyota’s credibility when Toyota employees were caught congratulating themselves on how they had slowed and limited the accident investigation.
Despite the above reservations, the official report by the NHTSA was that the root cause was the floor mat. Well, after several years it turns out that a detailed analysis of the electronics, as brought out during a recent trial, has confirmed that it was not just the floor mat. Key trial results are detailed in “Toyota Case: Vehicle Testing Confirms Fatal Flaws,” by Junko Yoshida in the 31 October 2013 EETimes. Here’s an excerpted summary of the problems identified:
• Software bugs that specifically can cause memory corruption
• Unmaintainable code complexity in Toyota’s software
• A multifunction kitchen-sink Task X designed to execute everything from throttle control to cruise control and many of the fail-safes
• That all Task X functions, including fail-safes, are designed to run on the main CPU in the Camry’s electronic control module
• That the brake override that is supposed to save the day when there is an unintended acceleration is also in Task X
• The use of an operating system in which there is no protection against hardware or software faults
• A number of other problems
The deficiencies in the throttle design are shocking, because good rules exist for the design of safety-critical electronics (e.g., Chapter 4, “Safety Analyses,” in The Design Analysis Handbook).
The Toyota case makes one wonder how many other possibly-catastrophic flaws are lurking within the cars we drive, or in other electronics-guided machinery, due to poorly-designed safety-related systems.
Prior Toyota Unintended Acceleration Posts:
3 Feb 2010
Stop Driving Recalled Toyotas
21 Feb 2010
Toyota Joins The Gallery Of Shame
“Sounds good,” I cautiously replied. “Send me a list of problems you’ve solved and I’ll consider it.”
It may sound like fun to be paid to solve problems, but the client is not at all interested in paying us to have fun. They are interested in paying us to solve problems, quickly. Usually these problems have been expensively festering, and the client’s design team is fatigued and understaffed. Bringing in a qualified outsider at such times makes sense, since a fresh perspective — unclouded by burnout (and sometimes politics) — can do wonders. In my own experience, oftentimes the solution to a stubborn problem is literally within an inch or two on the schematic of where the team has previously tread, missed only because of the disruptive pressures and distractions that major problems generate.
Therefore, if you want to be a consultant that helps quickly solve a challenging technical problem, there are three qualities that you must be able to offer your client:
1. You must be experienced — with a successful track record — in solving technical problems under high-stress conditions.
2. You must be organized and methodical. The client has already had enough of hit-or-miss scrambling, and is looking for a calm disciplined approach. This means that you will have requested all relevant data on the problem and be ready to provide productive input on day one. (If you are a shoot-from-the-hip type of person, your tenure will last about 24 hours or less.)
3. You must be diplomatic and willing to work closely with the client’s team, simply because you need them as much as they need you. Hotshots or other big ego types will get nowhere fast.
MicroViews: Electric Vehicles Are Not Greener and Cleaner / Dreamliner Batteries Still Misbehaving? / Robot Boogie Time
Recommended Reading: “Unclean at Any Speed“
“Electric cars don’t solve the automobile’s environmental problems,” by Ozzie Zehner, 30 June 2013 IEEE Spectrum. A standout example of scientific journalism. Mr. Zehner provides a remarkably thorough and balanced review of the overall relative pollution impact of electric vehicles.
Is The Boeing Dreamliner Lithium Battery Issue Really Solved?
From “Technical glitches delay two Dreamliner flights from Poland,” 4 July 2013, Reuters:
“A flight from Warsaw to Chicago that was scheduled to fly on Wednesday was canceled because the aircraft had “problems with the power supply…” “The spokeswoman would not say if the latest technical problems were related to over-heating batteries which forced the grounding of all Dreamliners for over three months.”
The Robots Are Coming! The Robots Are Coming!
And wow, can they dance!
Michael Sinnett, Boeing’s chief project engineer, said in a recent briefing that “Boeing is redesigning its batteries to ensure a fire isn’t possible. Among the new features will be a fire-resistant stainless steel case that will prevent oxygen from reaching the cells so fire can’t erupt.” (from “NTSB Contradicts Boeing Claim of No Fire in 787 Battery,” by Alan Levin , 15 Mar 2013 Bloomberg).
The problem with that statement is that once a lithium battery is heated sufficiently, it releases its own oxygen to fuel continued burning/explosion. That’s why lithium fires are extremely difficult to extinguish, and why an outer case, although it may keep a fire from spreading, will not prevent a fire from erupting.
In DACI’s 1st Quarter 2012 newsletter I predicted that a catastrophic safety event would eventually occur due to lithium batteries (please see “Li-Ion Battery Pack Hazards and our Psychic Prediction“). The recent fires in the initial flights of the new Boeing Dreamliner have come close to fulfilling that prophecy.
From “Detecting Lithium-Ion Cell Internal Faults In Real Time” (Celina Mikolajczak, John Harmon, Kevin White, Quinn Horn, and Ming Wu, in the Mar 1, 2010 issue of Power Electronics Technology) it is known that internal cell faults in lithium batteries can lead to thermal runaway, subsequently resulting in fires and/or explosions. Therefore the question arises: do the Boeing lithium batteries have an advanced internal construction that prevents cell faults, or mitigates thermal runaway in the event of a fault? If not, the Boeing team or vendor responsible for the battery system design is in big, big, trouble.
Although deficiencies in basic battery chemistry and/or construction appear to offer the best root cause hypothesis for the fires, there are also other possible factors. For example, it has been reported that perhaps the charging system malfunctioned, causing the batteries to overheat. However, a properly designed charger for an aircraft application would have fail-safe protection, preventing an overcharge. Plus, it was also reported that charging sensors did not detect an overvoltage. Although these factors sound reassuring, they are not sufficient to eliminate the charger from consideration. For example, one can hypothesize a charging waveform that contains spurious high frequency oscillations that create high rms charging currents. This would not necessarily result in overvoltage, but could result in overheating.
It is also possible that battery “cell defects” are nothing more than cell imbalances that vary according to production tolerances. In other words, the lithium battery, by its very nature, tends towards thermal runaway unless the internal cells are very tightly matched. This sensitivity would become more pronounced with a higher number of cells and higher mass, which would explain why no explosions have occurred in small button-style batteries, but do occur in the larger batteries.
There are other scenarios, including the thorny possibility that some combination of conditions conspired to create the failure. And, of course, the root cause may be highly intermittent, making detection extremely difficult. Such hypotheses are undoubtedly being examined by the Boing engineers. I wish them well, and hope that they are allowed to perform their work calmly, methodically, and thoroughly.
Note: Because it may take quite a long time to conclusively establish a root cause, I would suggest that Boeing immediately begin planning to retrofit the lithium system with one containing battery types that have not shown the proclivity to explode; e.g. nickel metal-hydride, or sealed lead acid gel. Heavier, yes, but in this case safety and the economic timeline indicate that it would be wise to be prepared with a retrofit design.
(For some brief guidelines on design failure crisis management, please see Scenario #6: “Coping with Design Panic,” in The Design Analysis Handbook, Appendix A, “How to Survive an Engineering Project.”