Where Ideas, Experiences, and Lessons Learned Intersect

Never Panic!

Posted on April 8, 2021 Updated on December 30, 2024

Panic is not a good problem-solving tool, regardless of your position or role. It is especially bad when you are in charge of people or brought in for your expertise. Panic leads to a myopic view of the problem, hindering creativity.

The point in my career when this became readily apparent was when I was working for a small software company. We had a new product (Warehouse Management System) and were launching our third deployment. This one was more complicated than the rest because it was for a pharmaceutical company. In addition to requirements like refrigeration and lot control, there was a mix of FDA-controlled items requiring various forms of auditing and security and storage areas significantly smaller than previous installations. It was a challenge, to be sure.

A critical component, “Location Search,” failed during this implementation. About 10-12 people were in the “war room” when my boss, the VP of Development, began to panic. He was extremely talented and normally did an excellent job, but his reaction negatively affected the others in the room. The mood quickly worsened.

I jumped in and took over because I did not want to be stuck there all weekend and mostly because I wanted this implementation to succeed. I asked my boss to go out and get a bunch of pizzas. Next, I organized a short meeting to review what we knew and what was different from our prior tests and asked for speculation about the root cause of this problem. The team came up with two potential causes and one potential workaround. Everyone was organized into three teams, and we began attacking each item independently and in parallel. 

We identified the root cause, which led to an ideal fix a few days later and a workaround that allowed us to finish the user acceptance testing and go live the following day. A change in mindset fostered the collaboration and problem-solving needed to move forward.

But this isn’t just limited to groups. I was a consultant at a large insurance company on a team redesigning their Risk Management system. We were using new software and wanted to be sure that the proper environment variables were set during the Unix login process for this new system. I volunteered to create an external function executed as part of the login process. Trying to maintain clean code, I had an “exit” at the end of the function. It worked well during testing, but once it was placed into production, the function immediately logged people out as they attempted to log into the system.

As you can imagine, I had a sinking feeling in my gut. How could I have missed this? This was a newer system deployed just for this risk management application, so no other privileged users were logged in at the time. Then, I remembered reading about a Unix “worm” that used FTP to infiltrate systems. The article stated that FTP bypassed the standard login process. This allowed me to FTP into the system and then delete the offending function. In less than 5 minutes, everything was back to normal.

A related lesson learned was to make key people aware of what happened, noting that the problem had been resolved and that there was no lasting damage. Hiding mistakes kills careers. Then, we created a “Lessons Learned” log, with this as the first entry, to foster the idea of sharing mistakes to avoid them in the future. Understanding that mistakes can happen to anyone is a good way to get people to plan better and keep them from panicking when problems occur. 

Staying calm and focused on resolving the problem is a much better approach than worrying about blame and the implications of those actions. And most people appreciate the honesty.

As the novelist James Lane Allen stated, “Adversity does not build character; it reveals it.”