Too often people, including Consultants, spend time trying to solve the wrong problem due to having incomplete or incorrect information. Once I was investigating a series of performance problems and unplanned outages that were assumed to be two separate problems. As I gathered information several people provided anecdotal stories of anomalous behaviors in a variety of systems, speculation about the “real problem,” and discussions about “chasing ghosts” during previous attempts to resolve the problem.
I remember stating that I was there to solve a real problem having a serious negative impact on production and that it was not my intent to chase ghosts or do anything else that would unnecessarily waste time. Next, I outlined the approach I would use to make a Root Cause determination, and that we would reconvene to discuss the real problem and potential solutions. A few people scoffed and felt that this was a waste of time and money.
The process followed was simple, structured, and logical. It took everything that was known to be true and mapped it out. I looked for patterns, commonalities, and intersections of systems and events. Within two days my team and I had identified a complex root cause involving multiple components, which we demonstrated would reliably reproduce the symptoms that our client was experiencing. From there we worked with their teams to make minor network changes, system configuration changes, and several small application changes.
By the end of the second week, they were no longer experiencing major slowdowns or unplanned outages. Each outage cost this company tens of thousands of dollars in lost sales due to the time-sensitive nature of their product. Within one week they had recovered the cost of hiring me and my team. What stuck with us was how many really smart people “believed in ghosts” and failed to focus on the information that they already had.
A few years later we decided to create a white paper to potentially help others in need of a simple structured approach. Below is a link to that white paper, which was written by one of the top people on my team. We received very positive feedback at the time so it seemed that this could potentially still be useful today. Please take a look and let me know what you think.
My wife has a Nissan Maxima and loves her car. Over the past 9 months there has been a persistent but seemingly random problem where the radio is used for a few minutes while the car is off and then the battery dies when she goes to start the car. This has happened more than a dozen times over the past 3 1/2 years, and it has been seen by two dealerships a for a total of three times recently with no success – the most recent visit being one day before this problem occurred.
Saturday morning I was running errands when my wife called and let me know that this problem happened again (second time this week and she was very frustrated). I was pretty excited because this time the problem occurred at home, not at some parking lot like usual, so I had the luxury of time to try to make a root cause determination. I’m somewhat mechanical but certainly no professional, so I followed my own consulting advice and contacted a professional.
Dave T. is a mechanical guru, and has an uncanny ability to offer sage advice with only a modicum of information. He is an incredibly busy guy, but is always willing to spend a few minutes and give helpful advice. It helps that he is a great guy, but it also helps him generate business (leads and referrals). It is an approach that helps create a constant backlog of work and very loyal clientele, which is good business.
I called Dave, described the problem, mentioned what I had read on various forums (i.e., similar electrical problems observed after some arbitrary mileage). Next, I mentioned that this had just been to the dealership and they did not find anything wrong. Dave laughed, stated that, “There is a 99%+ likelihood that alternator is bad, and possibly both the alternator and battery.” He sounded very confident.
There was a pause, and then he asked, “What’s more likely – that there is some completely random problem that only happens when your wife is out and your son stays in the car and listens to music for a few minutes, which by the way only happens to Maximas after X number of miles, or that there are issues with the alternators where they tend to fail after a certain amount of use, which causes them not to charge the battery properly and leads to a condition where there is not enough of a charge to start the car?”
When Dave explained it like that I felt kind of stupid, consoled only by the fact that other professional mechanics had not resolved the problem before me. He then added, “Anything that would be able to drain a battery within a few minutes would be noticeable. It would start a fire or melt wires, and smoke or smell as a result. You haven’t seen or smelled anything like that, have you?”
I described my plan to troubleshoot the problem, and Dave suggested that in addition I test the alternator and the specific gravity of the individual battery cells. So, in less than five minutes into that call I had a plan and was off and running.
Yesterday afternoon I spent several hours using a very methodical approach to troubleshooting, documenting everything with pictures and videos to help me recall both details and sequence if needed. Sure enough, Dave’s knowledge and intuition led to the correct conclusion.
I gave him a call to thank him, and while talking I wondered aloud why the dealership could not figure this out? Dave replied, “It’s not that they couldn’t have done what you did, but instead they focused on the symptoms you described. The mechanic probably sat there for 10-15 minutes with the lights and radio on while the car was off. After that the car started so they assumed that everything was fine. I listened to what you said, ignored the randomness and speculation, and honed-in on the problem. The fact that this happened again so soon also made sense because now your battery was run down from the testing performed by the dealership.” He added, “In my business I get paid for results, so I can’t get away with taking the easy way out.”
I’m big on lessons learned, wanting to make the most of each and every experience because I have learned that skills and knowledge are often very transferrable. As I thought about this I realized that Dave’s analysis was the perfect practical application of Occam’s razor. It’s a skill that is very helpful as a Consultant, but more importantly, it is something that can help when problem solving in any line of work.