Where Ideas, Experiences, and Lessons Learned Intersect

Never Panic!

Posted on April 8, 2021 Updated on June 4, 2021

Regardless of your position or role, panic is not a good problem-solving tool. It is especially bad when you are in charge of people, or when you are brought in for your expertise. Panic leads to a myopic view of the problem, and that hinders creativity.

The point in my career when this became readily apparent is when I was working for a small software company. We had a new product (Warehouse Management System) and were launching our third deployment. This one was more complicated than the rest because it was for a pharmaceutical company. In addition to requirements like refrigeration and lot control, there was a mix of FDA-controlled items requiring various forms of auditing and security as well as storage areas that were significantly smaller than previous installations. It was a challenge to be sure.

During this implementation, a critical component, “Location Search,” failed. There were about 10-12 people in the “war room” when my boss, the VP of Development, began to panic. He was an extremely talented person who normally did an excellent job, but his reaction began negatively affecting the others in the room. The mood quickly worsened.

Partly because I did not want to be stuck there all weekend, and mostly because I wanted this implementation to be a success, I jumped in and took over. I asked my boss to go out and get a bunch of pizzas. Next, I organized a short meeting to review what we knew, what was different from our prior tests and asked for speculation about the root cause of this problem. The team came up with two potential causes and one potential workaround. Everyone organized into three teams and we began attacking each item independently. 

We ended up identifying the root cause which led to an ideal fix a few days later, as well as a work-around that allowed us to finish the user acceptance testing and go live the following day. A change in mindset fostered the collaboration and problem-solving needed to move forward.

But, this isn’t just limited to groups. I was a consultant working at a large insurance company where I was on a team redesigning their Risk Management system. We were using new software and wanted to be sure that the proper environment variables were set during the Unix login process for this new system. I volunteered to create an external function that was executed as part of the login process. Trying to maintain clean code, I had an “exit” at the end of the function. It worked well during testing but once it was placed into production the function immediately logged people out as they were attempting to log into the system.

As you can imagine, I had a sinking feeling in my gut. How could I have missed this? This was a newer system deployed just for this risk management application so there were no other privileged users logged in at the time. Then, I remembered reading about a Unix “worm” that used FTP to infiltrate systems. The article stated that FTP bypassed the standard login process. This allowed me to FTP into the system and then delete the offending function. In less than 5 minutes everything was back to normal.

A related lesson learned was to make key people aware of what just happened, noting first that the problem had been resolved and that there was no lasting damage. Hiding mistakes kills careers. Then, we created a “Lessons Learned” log, with this as the first entry, to foster the idea of sharing mistakes as a way to avoid them in the future. Understanding that mistakes can happen to anyone turns out to be a good way to get people to plan better and then keep them from panicking when problems occur. 

Staying calm and focused on resolving the problem is a much better approach than worrying about blame and the implication of those actions. And, most people appreciate the honesty.