lessons learned
Lessons Learned from Selling Kubernetes
Cloud-native, containerization, microservices, and Kubernetes have become very popular over the past few years. They are as complex as they are powerful, and for a large, complex organization, these technologies can be a game changer. Kubernetes itself is a partial solution – the foundation for something extraordinary. It can take 20-25 additional products to handle all aspects of the computing environment (e.g., ingress, services mesh, storage, networking, security, observability, continuous delivery, policy management, and more).
Consider the case of a major Financial Services company, one of my clients. They operated with 200 Development teams, each comprising 5-10 members, who were frequently tasked with deploying new applications and application changes. Prior to embracing Kubernetes, their approach involved deploying massive monolithic applications, with changes occurring only 2-3 times per year. However, with the introduction of Kubernetes, they were able to transition to a daily deployment model, maintaining control and swiftly rolling back changes if necessary. This shift in their operations not only allowed them to innovate at a faster pace but also enabled them to respond to opportunities and address needs more promptly.
Most platforms utilize Ansible and Terraform for creating playbooks, configuration management, and other purposes. Those configurations could become very long and complex over time and were prone to errors. For more complicated configurations, such as multi-cloud and hybrid environments, the complexity is further amplified. “Configuration Drift,” or runtime configurations that differ from what was expected for various reasons, leads to problems such as increased costs due to resource misconfiguration, potential security issues resulting from incorrectly applied or missing policies, and issues with identity management.
The surprising thing was that when prospects identified those problems, they would look to new platforms that used the same tools to solve them. Sometimes, things would temporarily improve (after much time and expense for a migration), but then fall back into disarray as the underlying process issues still needed to be addressed.
Our platform used a new technology called Cluster API (or CAPI). It provided a central (declarative) configuration repository, making it quick and easy to create new clusters. More importantly, it would perform regular configuration checks and automatically reconcile incorrect or missing policies. It was an immutable and self-healing Kubernetes infrastructure. It simplified overall cluster management and standardized infrastructure implementation.
All great stuff – who would not want that? This technology was new but proven, but it was different, which scared some people. These were a couple of recurring themes:
- The Platform and DevOps teams had a backlog of work due to existing problems, so there was more fear about falling further behind than confidence in a better alternative.
- Teams focused on their existing investment in a platform or on the sunk costs spent over a long period, attempting to solve their problems. The ROI on a new platform was often only 3-4 months, but that was challenging to believe, given their own experiences on an inferior platform.
- Teams would look at outsourcing the problem to a managed service provider. They could not explain how the problems would be specifically resolved, but did not seem concerned about that lack of clarity.
- There was a lack of consistency on the versions of Kubernetes used, the add-ons and their versions, and one-off changes that were never intended to become permanent. Reconciling those issues or migrating to new, clean clusters both involved time and effort. That became an excuse to maintain the status quo.
- Unplanned outages were common and usually expensive. Using the cost of those outages as justification for something new was typically a last resort, as people did not like acknowledging problems that put a spotlight on themselves.
- Architects had a curiosity about new and different things, but often lacked the gravitas within business leadership to effect change. They were usually unwilling or unable to explain how real changes happen within their company, or introduce you to the actual decision-makers and budget holders.
Focusing on outcomes and working with the Executives most affected by them tended to be the best path forward. Those companies and teams were rewarded with a platform that simplified fleet management, improved observability, and helped them avoid the risky, expensive problems that had plagued them in the past. And, working with satisfied customers who appreciated your efforts and became loyal partners made selling this platform that much more rewarding.
“Acting Like a Startup”
Over the years, I have heard comments like, “We operate like a startup,” “We act like a startup,” and “We are an overnight success that was 10 years in the making.” These statements are often euphemisms for “We are small and not growing as quickly as we would like.”
There are numerous estimates of startups in their first few years. One of the best descriptions I have found is from Failory, but Investopedia and LendingTree have similar but differing take on the statistics and root causes. All three articles linked to are worth reading. The net result is that the outcome of failure is much greater than success, especially over time. So, “acting like a startup” is not necessarily good, even when true. You want to act like a successful startup!
Understanding the data and various causes for success and failure are great inputs to business plans. I have been a principal with successful startups, both early employees and founders. Understanding the data and various causes for success and failure are significant inputs to business plans focused on long-term success. As a Founder, there are a few points that I believe to be key to success:
- You have specific expertise that is in demand and would be valuable to an identifiable number of prospective customers. How would those customers use those skills, and how would they quantify the value? That understanding provides focus on what to sell and to whom.
- A detailed understanding of the market and key players is needed to hone in on a niche to succeed.
- Understand your strengths and weaknesses, then hire the most intelligent and ambitious people who complement your weaknesses and strengths.
- Understand how to reach those potential customers and the messaging you believe will compel them. Then, find a way to test and refine those assumptions as necessary. Marketing and Lead Generation are very important.
- Have a plan for delivering on whatever you sell before you get your first sale. A startup needs to develop its track record of success, beginning with its first sale.
- Cash flow is king. It is far too easy to run out of money while looking at an excellent balance sheet because of receivables. Understand what matters and why.
- Founders need to understand the administrative side of a business – especially the financial, legal (especially contract law), insurance, and taxes. Find experts to validate your approach and fill in knowledge gaps.
- Consistency leads to repeatable success. You standardize, optimize, and automate everything possible. Wasted time and effort become wasted opportunities.
- Finally, there needs to be sufficient cash on hand to fund the time it takes to find and close your first deals, deliver and invoice the work, and then receive your first payments. That could easily be a 3-6 month period.
Those are the foundational items that are reasonably tangible. What is not as concrete but equally as important are:
- Having or developing the ability to spot trends and identify gaps that could become opportunities for your business.
- An agile mindset allows you to pivot your offerings or approach to refine your business model and hone in on that successful niche for your business.
- Foster a sense of innovation within your business. Always look for opportunities to deliver a better product or service, improve the efficiency and effectiveness of your business, and create intellectual property (IP) that adds long-term value.
- Focus on being the best and building a brand that helps differentiate you from your competition.
- Become a Leader, Not a Manager. Create your vision of success, set expectations for each person and team, and help eliminate roadblocks to their success. Trust your team to help you grow and replace members quickly if it becomes clear they are not a good fit.
Steve Jobs once said, “It doesn’t make sense to hire smart people and then tell them what to do; we hire smart people so they can tell us what to do.”
Winning is hard, so focus on the journey. Making your customers’ lives easier and allowing your employees to be creative while doing something they are proud of will lead you to your destination. But when things start going well, don’t sit back and convince yourself you are successful. Instead, continue to focus on ways to improve and grow.
Success means different things to different people, but longevity, growth, profitability, and some form of contributing to the greater good should be dimensions of success for any vision.
Never Panic!
Panic is not a good problem-solving tool, regardless of your position or role. It is especially bad when you are in charge of people or brought in for your expertise. Panic leads to a myopic view of the problem, hindering creativity.
The point in my career when this became readily apparent was when I was working for a small software company. We had a new product (Warehouse Management System) and were launching our third deployment. This one was more complicated than the rest because it was for a pharmaceutical company. In addition to requirements like refrigeration and lot control, there was a mix of FDA-controlled items requiring various forms of auditing and security and storage areas significantly smaller than previous installations. It was a challenge, to be sure.
A critical component, “Location Search,” failed during this implementation. About 10-12 people were in the “war room” when my boss, the VP of Development, began to panic. He was extremely talented and normally did an excellent job, but his reaction negatively affected the others in the room. The mood quickly worsened.
I jumped in and took over because I did not want to be stuck there all weekend and mostly because I wanted this implementation to succeed. I asked my boss to go out and get a bunch of pizzas. Next, I organized a short meeting to review what we knew and what was different from our prior tests and asked for speculation about the root cause of this problem. The team came up with two potential causes and one potential workaround. Everyone was organized into three teams, and we began attacking each item independently and in parallel.
We identified the root cause, which led to an ideal fix a few days later and a workaround that allowed us to finish the user acceptance testing and go live the following day. A change in mindset fostered the collaboration and problem-solving needed to move forward.
But this isn’t just limited to groups. I was a consultant at a large insurance company on a team redesigning their Risk Management system. We were using new software and wanted to be sure that the proper environment variables were set during the Unix login process for this new system. I volunteered to create an external function executed as part of the login process. Trying to maintain clean code, I had an “exit” at the end of the function. It worked well during testing, but once it was placed into production, the function immediately logged people out as they attempted to log into the system.
As you can imagine, I had a sinking feeling in my gut. How could I have missed this? This was a newer system deployed just for this risk management application, so no other privileged users were logged in at the time. Then, I remembered reading about a Unix “worm” that used FTP to infiltrate systems. The article stated that FTP bypassed the standard login process. This allowed me to FTP into the system and then delete the offending function. In less than 5 minutes, everything was back to normal.
A related lesson learned was to make key people aware of what happened, noting that the problem had been resolved and that there was no lasting damage. Hiding mistakes kills careers. Then, we created a “Lessons Learned” log, with this as the first entry, to foster the idea of sharing mistakes to avoid them in the future. Understanding that mistakes can happen to anyone is a good way to get people to plan better and keep them from panicking when problems occur.
Staying calm and focused on resolving the problem is a much better approach than worrying about blame and the implications of those actions. And most people appreciate the honesty.
As the novelist James Lane Allen stated, “Adversity does not build character; it reveals it.”
Are you Thinking About Starting a Business?
The last post on Starting a Business was popular, so I thought I would share a key lesson learned and then provide links to previous posts that will provide insights as you launch your own business. If you have any questions, just post them as comments; I would happily reply.
The COVID-19 pandemic has created a great deal of uncertainty and opportunity. For many, now is the ideal time to explore their dream of starting a business and jumping into entrepreneurship. That can be exciting, fun, stressful, financially rewarding, and financially challenging, all within the same short period of time.
Being prepared for that roller coaster ride and having the ability and strength to continue pushing forward is important. Something to understand is that “Things don’t happen to you. They are the Direct Result of your own Actions and Inactions.” That may sound harsh, but here is a prime example:
When I was closing my consulting business down, I trusted my Accountant and Payroll company to handle all of the required Federal, Wisconsin, Ohio, and Colorado filings – something they stated they would handle, and I accepted at face value. Both companies had done a great job before, so why would I expect any less this time?
About nine months later, I started receiving letters from Ohio and Colorado about filings due, so I forwarded them to the Accountant and Payroll company. I thought this was “old business” and was being handled, plus I had moved on. It was probably just a timing error, something easy to explain away.
Skipping forward nearly three years, I had been threatened by the IRS and the Revenue Departments from both Ohio and Colorado. I started with a combined total of nearly $500K in assessments. Slowly that dropped to $50K, and then to $10K. I spent countless hours on the phone and writing letters explaining the misunderstanding. It wasn’t until I finally found a helpful person in each department willing to listen and tell me specifically what needed to be done to resolve that situation. My final cost was around $1,000. I was relieved that this fiasco was finally over.
I blamed both the Accountant and Payroll Service for these problems for the longest time. Ultimately I realized that it was my business and, therefore, my responsibility to understand the shutdown process – regardless of who did the work. I would have saved hundreds of hours of my time and several hundred dollars by gaining that understanding initially.
I was not a victim of anything – this situation directly resulted from my own inaction. It did not seem very important at the time, but my understanding of the situation and its importance was incorrect, and I paid the price. Lesson learned. It was my business, so it was still my responsibility to the very end.
Below are the other links. You don’t have to read them all at once, but it would be worth bookmarking them and reading one per day. Every new perspective, idea, and lesson learned could be the thing that helps you achieve your goal a day, week, or month sooner than expected. Every day and every dollar matters, so make the most of both!
- Comments on and a link to an on Curt Culver about Entrepreneurship.
- Comments on and a link to an HBR article about Start-ups and Entrepreneurship.
- Innovation, Intelligent Failure, and Failing Productively.
- Acting Like an Owner – Good Preparation for Becoming an Owner.
- Profitability Through Operational Efficiency.
- What Are You Really Selling?
- Continuous Improvement and a Growth Mindset.
- The Value Created by a Strong Team.
Presentation about Starting a Business and Entrepreneurship
It is interesting how often you see ads for some franchise offering that touts, “Become your own boss.” While that may not be all bad, it is just the tip of the iceberg. The presentation below is intended to provide insight to people considering starting their first company. This was from a one-hour presentation that glosses over many things, such as the need for registrations and insurance, but it could be helpful for a first-timer.
One of the first and most important lessons I learned when I started my consulting company long ago was that paying attention to cash flow was far more important than focusing on my balance sheet. Once you understand a problem, altering what you do to manage it becomes easy. For example, using fixed pricing based on tasks where we received 50% up-front and the remaining 50% upon acceptance of the deliverable smoothed out cash flow, which was a big help.
So, take a look and post any questions that you may have. If one person has a question, many more will likely do as well! Cheers.
- ← Previous
- 1
- 2
- 3
- 4
- Next →
Where Ideas, Experiences, and Lessons Learned Intersect 