intelligent failure
Good Article on Why AI Projects Fail
Today I came across this very good article focused on lessons learned, which could help anyone interested in these topics. It included a good mix of non-technical problems.
This is the link to the article, along with my commentary on the Top 3 items listed: https://www.cio.com/article/3429177/6-reasons-why-ai-projects-fail.html
Item #1:
The article discusses how the “problem” being evaluated was misstated using technical terms. At least some of these efforts are conducted “in a vacuum.” Given the cost and strategic importance of getting these early-adopter AI projects right, that surprised me.
In Sales and Marketing, you start the question, “What problem are we trying to solve?” and evolve that to, “How would customers or prospects describe this problem in their own words?” Without that understanding, you can neither vet the solution initially nor quickly qualify the need for it when speaking with customers or prospects. That leaves room for error when transitioning from strategy to execution.
More collaboration with Business likely would have helped. This was touched on at the end of the article under “Cultural challenges,” but the importance seemed to be downplayed. Lessons learned are valuable – especially when you are able to learn from the mistakes of others. This should have been called out early as a major lesson learned.
Item #2:
This second area had to do with the perspective of the data, whether that was the angle of the subject in photographs (overhead from a drone vs horizontal from the shoreline) or the type of customer data evaluated (such as from a single source) used to train the ML algorithm.
That was interesting because assumptions may have played a role in overlooking other aspects of the problem, or the teams may have been overly confident they could get the right results with the data available. In the examples cited, those teams identified the problems and took corrective action. A follow-up article describing the process used to determine the root cause in each case would be very interesting.
As an aside, from my perspective, this is why Explainable AI is so important. Sometimes, you just don’t know what you don’t know (the unknown unknowns). Understanding why and on what the AI is basing its decisions should help provide better-quality curated data up front, as well as identify potential drifts in the wrong direction while it is still early enough to make corrections without impacting deadlines or deliverables.
Item #3:
This didn’t surprise me, but it should be a cause for concern as advances are made at faster rates and organizations race to be first to market with an AI-based competitive advantage, potentially with less validation than ideal. The last paragraph under ‘Training data bias’ stated that based on a PWC survey, “only 25 percent of respondents said they would prioritize the ethical implications of an AI solution before implementing it.“
Bonus Item:
The discussion about the value of unstructured data was very interesting, especially when you consider:
- The potential for NLU (natural language understanding) products in conjunction with ML and AI.
- This is a great NLU-pipeline diagram from North Side Inc. in Canada, one of the pioneers in this space.
- The importance of semantic data analysis relative to any ML effort.
- The incredible value that products like MarkLogic’s database or Franz’s AllegroGraph provide over standard Analytics Database products.
- I personally believe the biggest exception to this assertion will be GPU databases (like OmniSci) that easily handle streaming data, can accomplish extreme computational feats well beyond traditional CPU-based products, and have geospatial capabilities that add an extra dimension of insight to the problem being solved.
Update: This is a link to a related article that discusses trends in areas of implementation, important considerations, and the potential ROI of AI projects: https://www.fastcompany.com/90387050/reduce-the-hype-and-find-a-plan-how-to-adopt-an-ai-strategy
This is an exciting space that will grow significantly over the next 3-5 years. The more information, experiences, and lessons learned are shared, the better it will be for everyone.
Failing Productively
As an entrepreneur, you will typically get advice like, “Fail fast and fail often.” I always found this somewhat amusing, similar to the saying, “It takes money to make money” (a lot of bad investments are made using that philosophy). Living this yourself is an amazing experience – especially when things turn out well. But as I have written about before, you learn as much from the good experiences as you do from the bad ones.
Innovating is tough. You need people who always think of different and better ways of doing things or question why something has to be done or made a certain way. It means shifting away from the “how” and “why” and focusing on the “what” (outcomes). It takes confidence to ask questions that many would view as stupid (“Why would you do that? It’s always been done this way.”) But when you have the right mix of people and culture, amazing things can and do happen, and it feels great.
Innovating takes a willingness to lose time and money, hoping to win something big enough later to make it all worthwhile. This is where many companies fall short because they lack the patience, budget, or appetite to fail. I believe this is why innovation often comes from small companies and small teams. For them, the prospect of doing something cool or making a big impact is motivation enough to try something, and the barriers to getting started are often much lower.
It also takes a lot of discipline to follow a plan when a project appears to be failing, but it takes even more discipline to kill a project that has demonstrated real potential but isn’t meeting expectations. That was one of my first and probably most important lessons learned in this area. Let me explain…
In 2000, we looked at franchising our “Consulting System” – processes, procedures, tools, metrics, etc., developed and proven in my business. We believed this approach could help average consultants deliver above-average work products in less time. The idea seemed to have real potential.
Finding an attorney who would even consider this idea took a lot of work. Most believed it would be impossible to proceduralize a somewhat ambiguous task like solving a business or technical problem. We finally found an attorney who, after a 2-hour no-cost interview, agreed to work with us. When asked about his approach, he replied, “I did not want to waste my [his] time or our money on a fool’s errand.”
We estimated it would take 12 months and cost approximately $100,000 to fully develop our consulting system. We met with potential prospects to validate the idea (it would have been illegal to pre-sell the system) and then got to work. Twelve months turned into 18, and the original $100K budget increased nearly 50%. All indications were positive, and we felt very good about the success and business potential of this effort.
Then, the terror attacks occurred on Sept. 11th, and businesses everywhere saw a decline. In early 2002, we reevaluated the project and felt that it could be completed within the next 6-8 months and would cost another $50K+. Our confidence was high.
After a long and emotional debate, we decided to kill the project – not because we felt it would not work, but because there was less of a target market, and now the payback period (time to value) would double or triple. This was one of the most difficult business decisions that I ever made.
A big lesson learned from this experience was that our approach needed to be more analytical.
- From that point forward, we created a budget for “time off” (we bought our own time, rather than waiting for bench time) and other project-related items.
- We developed a simple system to collect and track ideas and feedback. When an idea felt right, we took the next steps and created a plan with a defined budget, milestones, and timeline. If the project failed to meet any defined objectives, it would be killed – No questions asked.
- We documented what we did, why we did it, our goals, and expected outcomes and timelines. Regardless of success or failure, we would conduct postmortem reviews to learn and document as much as possible from every effort and investment.
We still had failures, but with each one, we took less time and spent less money. More importantly, we learned how to do this better, which enhanced our resilience and helped us realize several successes. It gave us both the structure and the freedom to create amazing things. Since failure was an acceptable outcome, we never feared it.
This approach was more than just “failing fast and failing often”; it was “intelligent failure,” and it served us well for nearly a decade.
Where Ideas, Experiences, and Lessons Learned Intersect 