Author Archives: DACI1
“Sounds good,” I cautiously replied. “Send me a list of problems you’ve solved and I’ll consider it.”
It may sound like fun to be paid to solve problems, but the client is not at all interested in paying us to have fun. They are interested in paying us to solve problems, quickly. Usually these problems have been expensively festering, and the client’s design team is fatigued and understaffed. Bringing in a qualified outsider at such times makes sense, since a fresh perspective — unclouded by burnout (and sometimes politics) — can do wonders. In my own experience, oftentimes the solution to a stubborn problem is literally within an inch or two on the schematic of where the team has previously tread, missed only because of the disruptive pressures and distractions that major problems generate.
Therefore, if you want to be a consultant that helps quickly solve a challenging technical problem, there are three qualities that you must be able to offer your client:
1. You must be experienced — with a successful track record — in solving technical problems under high-stress conditions.
2. You must be organized and methodical. The client has already had enough of hit-or-miss scrambling, and is looking for a calm disciplined approach. This means that you will have requested all relevant data on the problem and be ready to provide productive input on day one. (If you are a shoot-from-the-hip type of person, your tenure will last about 24 hours or less.)
3. You must be diplomatic and willing to work closely with the client’s team, simply because you need them as much as they need you. Hotshots or other big ego types will get nowhere fast.
MicroViews: Electric Vehicles Are Not Greener and Cleaner / Dreamliner Batteries Still Misbehaving? / Robot Boogie Time
Recommended Reading: “Unclean at Any Speed“
“Electric cars don’t solve the automobile’s environmental problems,” by Ozzie Zehner, 30 June 2013 IEEE Spectrum. A standout example of scientific journalism. Mr. Zehner provides a remarkably thorough and balanced review of the overall relative pollution impact of electric vehicles.
Is The Boeing Dreamliner Lithium Battery Issue Really Solved?
From “Technical glitches delay two Dreamliner flights from Poland,” 4 July 2013, Reuters:
“A flight from Warsaw to Chicago that was scheduled to fly on Wednesday was canceled because the aircraft had “problems with the power supply…” “The spokeswoman would not say if the latest technical problems were related to over-heating batteries which forced the grounding of all Dreamliners for over three months.”
The Robots Are Coming! The Robots Are Coming!
And wow, can they dance!
It sounds like something from The Onion: “Boeing on Monday admitted it may never be established why batteries failed on two of its high-tech Dreamliner aircraft, as work began to fix the safety problem that grounded the company’s wide-body passenger jet.”
-from “Boeing in the dark over 787 battery fires,” by Andrew Parker, 22 April 2013 Financial Times
“Boeing’s fix includes more insulation between each of the eight cells in the batteries. The batteries will also be encased in a new steel box designed to contain any fire and vent possible smoke or hazardous gases out of the planes.
“…both the F.A.A. administrator, Michael P. Huerta, and Transportation Secretary Ray LaHood said they were are satisfied that the proposed changes would eliminate concerns that the plane’s two lithium-ion batteries could erupt in smoke or fire.”
-“F.A.A. Endorses Boeing Remedy for 787 Battery” by C. Drew and J. Mouawad, 19 April 2013 New York Times
Conspicuously absent from this pronouncement is a definitive identification of the root cause of the lithium battery fires. Therefore Boeing, the FAA, and the Department of Transportation are all guessing that the stated modifications will fix the problem. I hope they are correct. But if they are it will be a matter of luck, not engineering diligence. The dissembling of the FAA and Department of Transportation are clearly evident in their own words: they say that they are “…satisfied that the proposed changes would eliminate concerns that the plane’s two lithium-ion batteries could erupt in smoke or fire.” If they are so satisfied, then why is it necessary to have a steel box to contain a fire? If they are so satisfied, then why did they not provide the supporting evidence to support their conclusions?
Also, Boeing and these government agencies have touted a few test flights as being of particular significance in proving the safety of the batteries. This is nonsense. The battery fires are low probability events, occurring only once for thousands of hours of operation. This implies that there are subtle variables in the battery construction, chemistry, and/or operation, which when combined worst case will cause the batteries to overheat. This combination may only occur for a small number of manufactured batteries, and fires may occur only when those particular batteries are exposed to a worst case combination of stresses (temperature, charge currents, etc.).
Therefore a handful of test flights, of a few dozen hours or so total, are not nearly sufficient to empirically identify a low-probability event. The identification of such an event would require hundred or even thousands of test flights, which is obviously not practical. Therefore the only alternative is an investigation that drills down and positively identifies the true underlying failure mechanism (as recommended here: “Flying the Flaming Skies: Should You Trust the Boeing Dreamliner?“). It is my opinion that this has not been done, because if it had, this knowledge would be trumpeted by Boeing.
I’m not flying the Boeing Dreamliner until I see the evidence that supports the optimistic conclusions of Boeing, the FAA, and the Department of Transportation.
“The unfortunate reality is that lithium-ion batteries were not ready for prime time four years ago and they’re not ready for prime time today.”
-from “Are EV Dreams Going Up In Smoke?” by John Petersen, 28 Mar 2013, Seeking Alpha
Despite claims that lithium battery technology is improving, lithium batteries continue to catch on fire. See the article above for some recent Mitsubishi examples.
The continued widespread use of a product that poses a serious and thus far unresolvable safety hazard is apparently based on the unfortunate business practice of balancing the costs of jury awards against the competitive advantage of using a dangerous product (lithium batteries have smaller size/weight for energy stored, compared to much safer types). Is there a CEO out there who is willing to forfeit some profits by discontinuing the use of lithium batteries? If so, please speak up.
Michael Sinnett, Boeing’s chief project engineer, said in a recent briefing that “Boeing is redesigning its batteries to ensure a fire isn’t possible. Among the new features will be a fire-resistant stainless steel case that will prevent oxygen from reaching the cells so fire can’t erupt.” (from “NTSB Contradicts Boeing Claim of No Fire in 787 Battery,” by Alan Levin , 15 Mar 2013 Bloomberg).
The problem with that statement is that once a lithium battery is heated sufficiently, it releases its own oxygen to fuel continued burning/explosion. That’s why lithium fires are extremely difficult to extinguish, and why an outer case, although it may keep a fire from spreading, will not prevent a fire from erupting.
“Boeing Co. is confident that proposed changes to the 787 Dreamliner will provide a permanent solution to battery problems that grounded its newest jet, a senior executive said Monday.” –Reuters, 11 March 2013
The reported changes include “adding ceramic insulation between the cells of the battery and a stronger stainless steel box with a venting tube to contain a fire and expel fumes from the aircraft.” –Reuters, Alwyn Scott and Tim Hepher and Peter Henderson, 5 Mar 2013
Why is Boeing confident? This is a mystery because, based on available published data, it does not appear that Boeing has positively determined the root cause of the battery fires. Furthermore, as for all safety-critical applications, the certainty of the cause should be determined beyond a reasonable doubt. This stringent requirement would be certified by a panel of independent experts of unquestioned expertise and integrity, who have no financial interest in the outcome of their review.
Without positive identification of the root cause, Boeing may be indulging in a logical fallacy that I have seen employed before, with very bad results. The fallacy is in trying to fix what is assumed to be the problem (e.g. inadequate thermal insulation between battery cells). But what if the assumption is wrong? If so, the “fix” could be ineffective, or even make things worse. For example, improving cell insulation will trap more heat within the cells, raising the cell temperature. If the true root cause is related to higher cell temperature, the added insulation could make cell failure more likely, not less.
There are many other troubling scenarios that can be hypothesized, and the only way to disprove them is to dig in and find the true root cause, beyond a reasonable doubt (including rigorous validation as discussed here: “Flying the Flaming Skies: Should You Trust the Boeing Dreamliner?“)
P.S. A good review of the genesis of the Boeing battery problem can be found here: “NTSB report shows Boeing’s battery analysis fell short,” Dominic Gates, Seattle Times
In DACI’s 1st Quarter 2012 newsletter I predicted that a catastrophic safety event would eventually occur due to lithium batteries (please see “Li-Ion Battery Pack Hazards and our Psychic Prediction“). The recent fires in the initial flights of the new Boeing Dreamliner have come close to fulfilling that prophecy.
From “Detecting Lithium-Ion Cell Internal Faults In Real Time” (Celina Mikolajczak, John Harmon, Kevin White, Quinn Horn, and Ming Wu, in the Mar 1, 2010 issue of Power Electronics Technology) it is known that internal cell faults in lithium batteries can lead to thermal runaway, subsequently resulting in fires and/or explosions. Therefore the question arises: do the Boeing lithium batteries have an advanced internal construction that prevents cell faults, or mitigates thermal runaway in the event of a fault? If not, the Boeing team or vendor responsible for the battery system design is in big, big, trouble.
Although deficiencies in basic battery chemistry and/or construction appear to offer the best root cause hypothesis for the fires, there are also other possible factors. For example, it has been reported that perhaps the charging system malfunctioned, causing the batteries to overheat. However, a properly designed charger for an aircraft application would have fail-safe protection, preventing an overcharge. Plus, it was also reported that charging sensors did not detect an overvoltage. Although these factors sound reassuring, they are not sufficient to eliminate the charger from consideration. For example, one can hypothesize a charging waveform that contains spurious high frequency oscillations that create high rms charging currents. This would not necessarily result in overvoltage, but could result in overheating.
It is also possible that battery “cell defects” are nothing more than cell imbalances that vary according to production tolerances. In other words, the lithium battery, by its very nature, tends towards thermal runaway unless the internal cells are very tightly matched. This sensitivity would become more pronounced with a higher number of cells and higher mass, which would explain why no explosions have occurred in small button-style batteries, but do occur in the larger batteries.
There are other scenarios, including the thorny possibility that some combination of conditions conspired to create the failure. And, of course, the root cause may be highly intermittent, making detection extremely difficult. Such hypotheses are undoubtedly being examined by the Boing engineers. I wish them well, and hope that they are allowed to perform their work calmly, methodically, and thoroughly.
Note: Because it may take quite a long time to conclusively establish a root cause, I would suggest that Boeing immediately begin planning to retrofit the lithium system with one containing battery types that have not shown the proclivity to explode; e.g. nickel metal-hydride, or sealed lead acid gel. Heavier, yes, but in this case safety and the economic timeline indicate that it would be wise to be prepared with a retrofit design.
(For some brief guidelines on design failure crisis management, please see Scenario #6: “Coping with Design Panic,” in The Design Analysis Handbook, Appendix A, “How to Survive an Engineering Project.”
Note: On hundreds of projects I have found the great majority of customers to be highly professional and a pleasure to work with. This post addresses the few exceptions that are encountered from time to time. -EW
Several years ago I was hired by an electronics firm to determine the root cause of a circuit problem that was holding up production. I spoke to the young engineer who had created the design, analyzed his circuit, reviewed the test data, and concluded that he had made a design error. (For what it’s worth, most of my troubleshooting investigations have determined that the root cause of circuit problems is insufficient design margin, which is why I always recommend that every circuit be validated with a good WCA.) I provided a solution and that was that. Or so I thought.
I later received a tip from a colleague that the young engineer I had worked with had generated a memo that stated that my conclusions were wrong, and that he had found the “true cause” of the problem. Apparently the engineer felt threatened by the fact that he had designed a circuit with a problem that he could not identify, and decided to lie about the facts behind my back. Based on the tip, I provided a follow-up memo that corrected his inaccuracies. This caused the young engineer some serious embarrassment, but I think he earned it.
I felt bad nonetheless, because the first rule of a consultant is, in my opinion, to be sure that the client’s team perceives you as non-threatening. The consultant is not there to act superior, or to gloat, or to point out the perceived faults of the team. (Hint: such consultants create more damage than they’re worth; fire them.) The consultant’s job is simply to lend a hand.
Furthermore, there is no reason for the consultant to feel superior. Yes, the consultant must have design expertise and problem-solving skills, but more valuable is the fact that the consultant provides an outside and objective viewpoint, unpolluted by the daily hassles (sometimes political) that impede the team. In many cases the team is very close to finding the problem, but they are unable to do so because they are behind schedule, overworked, tired, and distracted by the varied and hectic demands of the typical engineering workplace. This is why it makes good sense to hire a consultant: it’s just not possible for a team to be completely objective about their own efforts, particularly when they’re under a lot of pressure.
Yet, despite the tactful and low-key assistance of a modest consultant, there will still be those cases where the defensiveness of some individuals cannot be disarmed. Untruthful memos, passive-aggressive unhelpfulness, “I thought of it before the consultant did” posturing, and other immature behavior will sometimes be encountered. If you want to be a consultant, then you will need to deal with such unpleasantness forthrightly but tactfully. It’s just part of the job.
-from The Onion, “Study Finds Working at Work Improves Productivity“
Let’s face it. A lot of us are highly skilled at finding ways to avoid boring assignments, regardless of how essential they may be. Precious time is grossly wasted on web surfing, daydreaming, coffee breaks, and wanton socializing.
Sure, we make exceptions. We’ll work our ass off into the evening and weekends for the really challenging assignments, the ones worthy of our talents. Okay, so maybe we haven’t developed a lot of talent, not yet, but someday we’ll have it, and we want those assignments now, as a prepaid reward in advance of that fine day.
And if we’ve been assigned a project management role, maybe the reason we can’t nail our butts down at our desks for more than ten minutes to do some real work is simply because we’re lazy and undisciplined, preferring to walk around sipping coffee and barking orders, while we avoid the detailed planning required for proper staffing, the establishment of priorities, and the creation of realistic budgets and schedules.
Because they make us uncomfortable, we also avoid personnel issues and just let them fester. Judging makes us uncomfortable, too. We don’t want anyone to not like us, so every year we give everybody on the team the same generic review and the same average raise, not realizing that this makes everyone not like us, rather than just the underperformers.
So, instead of really managing, we host lots of useless meetings and scurry around out on the floor, butting in and injecting our naive and meager knowledge as we futilely flail at putting out the latest fire, never realizing that those fires are ignited as a result of ignoring staffing and budgets and schedules and priorities and personnel issues.
And then we wonder why our new products are late, costly, unimpressive, and unreliable.