Chris, Thank you for the interesting story. Unfortunately this case is not all that unusual. About a decade or so ago (names have been changed to protect the guilty) I was working in the hardware engineering group at a company that had just instituted an aggressive cost-reduction program.
Thanks to a lucky break, we managed to hear about the latest cost reduction "victory" before any product was shipped. Manufacturing had decided to save money by eliminating those useless decoupling caps - ALL of them! They removed all the decoupling caps from 2 boards and re-ran the ATE tests. The boards passed, so they got an engineer to sign off on the ECO. This (mechanical) engineer had always wondered if those darn caps were really needed so he was more than pleased to sign off. We put an immediate halt to the cost reduction. I still have nightmares about what could have happened to our customers if that one had sneaked through. The product was buggy enough as it was! A couple of things need to be mentioned. One is that there were generous cash bonus awards for cost reductions. This caused a gold-rush mentality to take hold. People had strong incentives to cheat by making endruns around the normal review process. Another is that, due to politics, the Mechanical Engineering group had almost total control over the entire PCB process. A PCB is not just a mechanical object. It is a complex electronic component. I have seen great designs ruined by lousy layouts and mediocre designs rescued by careful layouts. Again, Thank You Chris for taking the time to relate this experience to us. Scott Lacey -----Original Message----- From: owner-emc-p...@ieee.org [mailto:owner-emc-p...@ieee.org]On Behalf Of Chris Maxwell Sent: Friday, January 26, 2001 9:38 AM To: 'EMC-PSTC Internet Forum' Subject: Decoupling Capacitors Just a little personal experience I want to relate. The EMC people can really appreciate this. Sorry about the long, conversational tone, but I think it will help people appreciate how much effort could have been prevented by following simple EMC design rules. One of our products has a motherboard with an ISA bus with 7 standard connectors. Over the past 7 years or so, we have taken advantage of this flexibility to use processor, VGA, ethernet and serial extender cards from numerous third party vendors with quite a bit of success. About a month ago, we started having troubles with an 8 port serial expansion card that we buy from a third party vendor. Some of the units would fail the serial comm test during the final quality test before shipping the units. We started testing samples extensively and set up some overnight tests to exercise the cards. We found that some cards would run indefinitely, others would fail. Sometimes they would fail after 1 minute, sometimes after 10 hours. We considered interrupt servicing (software) problems, bad chips, faulty connectors ... We then did about two weeks' worth of "isolation" testing, where we would shift boards between units, trying different software, firmware, hardware, processors (cookware?) configurations. We found only one common thread. That was, "bad" boards would always fail, although the time to failure was random. "Good" boards were always good. We then started swapping chips between "good" boards and "bad" boards. We swapped RS-232 drivers... no change. Uarts... no change. Processors ... no change. Memory... no change. Firmware... no change. We then swapped 3each 74LS374's and 2each 74LS373's all at once.... Aha! the "bad" board became good and the "good" board became bad. We thought we were narrowing in. So, we put the 373's back so that only the 374's were swapped. Suddenly, neither board would fail after an entire day's testing. Both boards became good? This brought our isolation testing to a screeching halt. We thought we were on the trail of finding a bad chip either by date code, manufacturer ... something. But when both boards went "good" this hypothesis went out the window. (For those who are curious, we did put the chips back so that all 5 were swapped and got our "good" and "bad" boards back.) We then sent one "good" and one "bad" board back to the manufacturer. They couldn't duplicate the problem. Their owner talked to our software guy and suggested changes in interrupt handling, handshaking, initialization ... None of this worked. About three weeks ago, my boss was sitting down with us evaluating the boards, we were looking for differences in manufacturers, date codes, bad solder joints ... He made a casual comment that the board didn't have many decoupling caps. ( You know, those $1 per ton, ubiquitous, little yellow gumdrop, 0.1 uF capacitors). At the time, we all agreed, but thought nothing of it. This product has been "stop-ship" for about a month. We have customers screaming, marketing is asking for daily updates. The guys in manufacturing are renting hotel rooms to store backed up units (that part's just a joke). You get the picture. We were at the end of our rope. My boss suggested to take a "bad" board and solder on 0.1uF cap across each of the 5 chips in question. I had nothing to lose. As I'm writing this email today, the "bad" board with those 5 capacitors (of which we probably have 10,000 in stock) is happily running through test after test. What's the moral of the story. If you EVER think you can save money by skimping on decoupling caps. Think again. If you ever think that good EMC design is only for EMC's sake. Think again. If you ever put EMC problems on the side and don't consider them a possible failure mode. Think again. I've learned my lesson and I'm going to relay this message to our supplier for these boards. My fingers are crossed. I'm now praying to the EMC Gods. (It might help) I'm hoping that a weekend long test will prove that we can end a month-long stop ship on a quarter of a million dollars in product with one dollar's worth of well placed capacitance. I'm hoping that this same $1 worth of capacitance will let us and our board vendor sleep easy for the first time in quite a while. Please don't respond to this via the forum. I feel guilty enough about tying it up, but I thought the moral of the story and the validation of why EMC design is worth doing was worth it. Chris ------------------------------------------- This message is from the IEEE EMC Society Product Safety Technical Committee emc-pstc discussion list. To cancel your subscription, send mail to: majord...@ieee.org with the single line: unsubscribe emc-pstc For help, send mail to the list administrators: Jim Bacher: jim_bac...@mail.monarch.com Michael Garretson: pstc_ad...@garretson.org For policy questions, send mail to: Richard Nute: ri...@ieee.org ------------------------------------------- This message is from the IEEE EMC Society Product Safety Technical Committee emc-pstc discussion list. To cancel your subscription, send mail to: majord...@ieee.org with the single line: unsubscribe emc-pstc For help, send mail to the list administrators: Jim Bacher: jim_bac...@mail.monarch.com Michael Garretson: pstc_ad...@garretson.org For policy questions, send mail to: Richard Nute: ri...@ieee.org
