problem solving
Never Panic!
Regardless of your position or role, panic is not a good problem-solving tool. It is especially bad when you are in charge of people, or when you are brought in for your expertise. Panic leads to a myopic view of the problem, and that hinders creativity.
The point in my career when this became readily apparent is when I was working for a small software company. We had a new product (Warehouse Management System) and were launching our third deployment. This one was more complicated than the rest because it was for a pharmaceutical company. In addition to requirements like refrigeration and lot control, there was a mix of FDA-controlled items requiring various forms of auditing and security as well as storage areas that were significantly smaller than previous installations. It was a challenge to be sure.
During this implementation, a critical component, “Location Search,” failed. There were about 10-12 people in the “war room” when my boss, the VP of Development, began to panic. He was an extremely talented person who normally did an excellent job, but his reaction began negatively affecting the others in the room. The mood quickly worsened.
Partly because I did not want to be stuck there all weekend, and mostly because I wanted this implementation to be a success, I jumped in and took over. I asked my boss to go out and get a bunch of pizzas. Next, I organized a short meeting to review what we knew, what was different from our prior tests and asked for speculation about the root cause of this problem. The team came up with two potential causes and one potential workaround. Everyone organized into three teams and we began attacking each item independently.
We ended up identifying the root cause which led to an ideal fix a few days later, as well as a work-around that allowed us to finish the user acceptance testing and go live the following day. A change in mindset fostered the collaboration and problem-solving needed to move forward.
But, this isn’t just limited to groups. I was a consultant working at a large insurance company where I was on a team redesigning their Risk Management system. We were using new software and wanted to be sure that the proper environment variables were set during the Unix login process for this new system. I volunteered to create an external function that was executed as part of the login process. Trying to maintain clean code, I had an “exit” at the end of the function. It worked well during testing but once it was placed into production the function immediately logged people out as they were attempting to log into the system.
As you can imagine, I had a sinking feeling in my gut. How could I have missed this? This was a newer system deployed just for this risk management application so there were no other privileged users logged in at the time. Then, I remembered reading about a Unix “worm” that used FTP to infiltrate systems. The article stated that FTP bypassed the standard login process. This allowed me to FTP into the system and then delete the offending function. In less than 5 minutes everything was back to normal.
A related lesson learned was to make key people aware of what just happened, noting first that the problem had been resolved and that there was no lasting damage. Hiding mistakes kills careers. Then, we created a “Lessons Learned” log, with this as the first entry, to foster the idea of sharing mistakes as a way to avoid them in the future. Understanding that mistakes can happen to anyone turns out to be a good way to get people to plan better and then keep them from panicking when problems occur.
Staying calm and focused on resolving the problem is a much better approach than worrying about blame and the implication of those actions. And, most people appreciate the honesty.
Occam’s razor, our Maxima, and the Sage Mechanic
My wife has a Nissan Maxima and loves her car. Over the past 9 months there has been a persistent but seemingly random problem where the radio is used for a few minutes while the car is off and then the battery dies when she goes to start the car. This has happened more than a dozen times over the past 3 1/2 years, and it has been seen by two dealerships a for a total of three times recently with no success – the most recent visit being one day before this problem occurred.
Saturday morning I was running errands when my wife called and let me know that this problem happened again (second time this week and she was very frustrated). I was pretty excited because this time the problem occurred at home, not at some parking lot like usual, so I had the luxury of time to try to make a root cause determination. I’m somewhat mechanical but certainly no professional, so I followed my own consulting advice and contacted a professional.
Dave T. is a mechanical guru, and has an uncanny ability to offer sage advice with only a modicum of information. He is an incredibly busy guy, but is always willing to spend a few minutes and give helpful advice. It helps that he is a great guy, but it also helps him generate business (leads and referrals). It is an approach that helps create a constant backlog of work and very loyal clientele, which is good business.
I called Dave, described the problem, mentioned what I had read on various forums (i.e., similar electrical problems observed after some arbitrary mileage). Next, I mentioned that this had just been to the dealership and they did not find anything wrong. Dave laughed, stated that, “There is a 99%+ likelihood that alternator is bad, and possibly both the alternator and battery.” He sounded very confident.
There was a pause, and then he asked, “What’s more likely – that there is some completely random problem that only happens when your wife is out and your son stays in the car and listens to music for a few minutes, which by the way only happens to Maximas after X number of miles, or that there are issues with the alternators where they tend to fail after a certain amount of use, which causes them not to charge the battery properly and leads to a condition where there is not enough of a charge to start the car?”
When Dave explained it like that I felt kind of stupid, consoled only by the fact that other professional mechanics had not resolved the problem before me. He then added, “Anything that would be able to drain a battery within a few minutes would be noticeable. It would start a fire or melt wires, and smoke or smell as a result. You haven’t seen or smelled anything like that, have you?”
I described my plan to troubleshoot the problem, and Dave suggested that in addition I test the alternator and the specific gravity of the individual battery cells. So, in less than five minutes into that call I had a plan and was off and running.
Yesterday afternoon I spent several hours using a very methodical approach to troubleshooting, documenting everything with pictures and videos to help me recall both details and sequence if needed. Sure enough, Dave’s knowledge and intuition led to the correct conclusion.
I gave him a call to thank him, and while talking I wondered aloud why the dealership could not figure this out? Dave replied, “It’s not that they couldn’t have done what you did, but instead they focused on the symptoms you described. The mechanic probably sat there for 10-15 minutes with the lights and radio on while the car was off. After that the car started so they assumed that everything was fine. I listened to what you said, ignored the randomness and speculation, and honed-in on the problem. The fact that this happened again so soon also made sense because now your battery was run down from the testing performed by the dealership.” He added, “In my business I get paid for results, so I can’t get away with taking the easy way out.”
I’m big on lessons learned, wanting to make the most of each and every experience because I have learned that skills and knowledge are often very transferrable. As I thought about this I realized that Dave’s analysis was the perfect practical application of Occam’s razor. It’s a skill that is very helpful as a Consultant, but more importantly, it is something that can help when problem solving in any line of work.
Where Ideas, Experiences, and Lessons Learned Intersect 