Uncategorized
Never Panic!
Regardless of your position or role, panic is not a good problem-solving tool. It is especially bad when you are in charge of people, or when you are brought in for your expertise. Panic leads to a myopic view of the problem, and that hinders creativity.
The point in my career when this became readily apparent is when I was working for a small software company. We had a new product (Warehouse Management System) and were launching our third deployment. This one was more complicated than the rest because it was for a pharmaceutical company. In addition to requirements like refrigeration and lot control, there was a mix of FDA-controlled items requiring various forms of auditing and security as well as storage areas that were significantly smaller than previous installations. It was a challenge to be sure.
During this implementation, a critical component, “Location Search,” failed. There were about 10-12 people in the “war room” when my boss, the VP of Development, began to panic. He was an extremely talented person who normally did an excellent job, but his reaction began negatively affecting the others in the room. The mood quickly worsened.
Partly because I did not want to be stuck there all weekend, and mostly because I wanted this implementation to be a success, I jumped in and took over. I asked my boss to go out and get a bunch of pizzas. Next, I organized a short meeting to review what we knew, what was different from our prior tests and asked for speculation about the root cause of this problem. The team came up with two potential causes and one potential workaround. Everyone organized into three teams and we began attacking each item independently.
We ended up identifying the root cause which led to an ideal fix a few days later, as well as a work-around that allowed us to finish the user acceptance testing and go live the following day. A change in mindset fostered the collaboration and problem-solving needed to move forward.
But, this isn’t just limited to groups. I was once working at a large insurance company where I was on a team redesigning their Risk Management system. We were using new software and wanted to be sure that the proper environment variables were set during the Unix login process for this new system. I volunteered to create an external function that was executed as part of the login process. Trying to maintain clean code, I had an “exit” at the end of the function. It worked well during testing but immediately logged people out as they were attempting to log into the system.
As you can imagine, I had a sinking feeling in my gut. How could I have missed this? This was a newer system deployed just for this risk management application so there were no other privileged users logged in at the time. Then, I remembered reading about a Unix “work” that used FTP to infiltrate systems. The article stated that FTP bypassed the standard login process. This allowed me to FTP into the system and then delete the offending function. In less than 5 minutes everything was back to normal.
A related lesson learned was to make key people aware of what just happened, noting first that the problem had been resolved and that there was no lasting damage. Hiding mistakes kills careers. Then, we created a “Lessons Learned” log, with this as the first entry, to foster the idea of sharing mistakes as a way to avoid them in the future. Understanding that mistakes can happen to anyone turns out to be a good way to get people to plan better and then keep them from panicking when problems occur.
Staying calm and focused on resolving the problem is a much better approach than worrying about blame and the implication of those actions. And, most people appreciate the honesty.
Understanding the Real Issue using Root Cause Analysis
Too often people, including Consultants, spend time trying to solve the wrong problem due to having incomplete or incorrect information. Once I was investigating a series of performance problems and unplanned outages that were assumed to be two separate problems. As I gathered information several people provided anecdotal stories of anomalous behaviors in a variety of systems, speculation about the “real problem,” and discussions about “chasing ghosts” during previous attempts to resolve the problem.
I remember stating that I was there to solve a real problem having a serious negative impact on production and that it was not my intent to chase ghosts or do anything else that would unnecessarily waste time. Next, I outlined the approach I would use to make a Root Cause determination, and that we would reconvene to discuss the real problem and potential solutions. A few people scoffed and felt that this was a waste of time and money.
The process followed was simple, structured, and logical. It took everything that was known to be true and mapped it out. I looked for patterns, commonalities, and intersections of systems and events. Within two days my team and I had identified a complex root cause involving multiple components, which we demonstrated would reliably reproduce the symptoms that our client was experiencing. From there we worked with their teams to make minor network changes, system configuration changes, and several small application changes.
By the end of the second week, they were no longer experiencing major slowdowns or unplanned outages. Each outage cost this company tens of thousands of dollars in lost sales due to the time-sensitive nature of their product. Within one week they had recovered the cost of hiring me and my team. What stuck with us was how many really smart people “believed in ghosts” and failed to focus on the information that they already had.
A few years later we decided to create a white paper to potentially help others in need of a simple structured approach. Below is a link to that white paper, which was written by one of the top people on my team. We received very positive feedback at the time so it seemed that this could potentially still be useful today. Please take a look and let me know what you think.
Innovations “Iron Triangle”
The concept of an Iron Triangle is that along each side of the triangle is one item that is constrained by the items from the other two sides. In Project Management this is often referred to as a triple constraint. This identifies the fundamental relationships (such as Time, Cost, and Scope in Project Management) without addressing related aspects such as Risk and Quality. It provides a simple understanding of both requirements and tradeoffs.
Yesterday I spoke with Dave Mosby, an impressive person with an equally impressive background. He related Innovation to Fire, noting that in order to create fire you need fuel, oxygen, and heat. He added that they need to be in the right combination to achieve the desired flame. What a brilliant analogy.
Dave stated that for Corporate Innovation to succeed you need the proper balance of Innovation, Capital, and Entrepreneurship. I found this enlightening because his description substituted “entrepreneurship” for “culture” in my existing mental model. While the difference is subtle, I found it to be important.
As noted above, simplified frameworks do not provide a complete understanding. But, they do help understand and plan around the foundational items required for success. Mapping this to past experiences I was able to gain a better understanding of things that did not move forward as desired, and what I could have done differently in order to be more effective.
One idea was to create a fault-tolerant database using Red Hat’s JBoss middleware. We had a Services partner willing to create a working prototype, tune it for performance, document the system requirements and configuration, and package it for easy deployment. They wanted was $10K to cover their costs.
At the time I did not hold a budget of my own so created a purchase request supported by a logical justification. It modeled potential revenue increases for database subscriptions based on the need for a failover installation, as well as growth from potential expanded use cases. This was a slam dunk!
In my mind, this was simple as it was “only $10K,” and I had funded many similar efforts when I had my own company. But that’s the rub. I viewed these efforts as investments in understanding, lessons learned, and growth. Not every investment had a direct payoff, but nearly each one of them had an indirect payoff for my company. It was an entrepreneurial mindset that accepted risk as something required for rewards and success. I now see, many years later, how reframing my proposal as a way to foster innovation and entrepreneurship could have been far more effective.
It is never too late to gain new insights and lessons learned. And, a slightly different perspective on an important topic provided the understanding that should help with positioning projects for success in the future. And this flowed from a discussion with an interesting person who has, “been there, done that” many times.
Using Themes for Enhanced Problem Solving
Thematic Analysis is a powerful qualitative approach used by many consultants. It involves identifying patterns and themes to better understand how and why something happened, which provides the context for other quantitative analysis. It can also be utilized when developing strategies and tactics due to its “cause and effect” nature.
Typical analysis tends to be event-based. Something happened that was unexpected. Some type of triggering or compelling event is sought to either stop something from happening or to make something happen. With enough of the right data, you may be able to identify patterns, which can help predict what will happen next based on past events. This data-based understanding may be simplistic or incomplete, but often it is sufficient.
But, people are creatures of habit. If you can identify and understand those habits, and place them within the context of a specific environment that includes interactions with others, you may be able to identify patterns within the patterns. Those themes can be much better indicators of what may or may not happen than the data itself. They not only become better predictors of things to come but can also help identify more effective strategies and tactics to achieve your goals.
This approach requires that a person view an event (desired or historical) from various perspectives to help understand:
- Things that are accidental but predictable because of human nature.
- Things that are predictable based on other events and interactions.
- Things that are the logical consequence of a series of events and outcomes.
Aside from the practical implications of this approach I find it fascinating relative to AI and Predictive Analysis.
For example, by understanding the recurring themes and triggers you can monitor data and activities proactively. That is actionable intelligence that can be automated and incorporated into a larger system. Machine Learning and Deep Learning can analyze tremendous volumes of data from a variety of sources in realtime.
Combine that with Semantic Analysis, which is challenging due to the complexity of taxonomies and ontologies, and now that system more accurately understand what is really happening in order to make accurate predictions. Add in spatial and temporal data such as IoT, metadata from photographs, etc. and you should have the ability to view something as though you were very high up – providing the ability to “see” what is on the path ahead. It is obviously not that simple, but it is exciting to think about.
From a practical perspective, keeping these thoughts in the back of your mind will help you see details that other people have missed. That makes for better analysis, better strategies, and better execution.
Who wouldn’t want that?
Interesting Article about ADHD and Creativity
This “pocket” story is from Scientific American, originally published on March 5, 2019. The Creativity of ADHD.
Over the years I have found that some of the most interesting, creative, and effective CEOs have ADHD-like tendencies. The strange thing is that they may not even be aware that they have it. Hyperfocus can be incredibly effective when attached to a driven person.
Below are links to a couple of posts that make these concepts and their benefits tangible:
Just examples of why it is best to look beyond labels, lay your preconceived notions to the side, and explore the potential that each and every individual can contribute. The best managers and leaders tend to have this ability.
And if you want to take that a step further, let it guide you on finding the best approach to teaching, coaching, and motivating the people on your team. It may take a little extra effort but the results are amazing.
