Big Data
Using Themes for Enhanced Problem Solving
Thematic Analysis is a powerful qualitative approach used by many consultants. It involves identifying patterns and themes to better understand how and why something happened, which provides the context for other quantitative analysis. It can also be utilized when developing strategies and tactics due to its “cause and effect” nature.
Typical analysis tends to be event-based. Something happened that was unexpected. Some type of triggering or compelling event is sought to either stop something from happening or to make something happen. With enough of the right data, you may be able to identify patterns, which can help predict what will happen next based on past events. This data-based understanding may be simplistic or incomplete, but often it is sufficient.
But, people are creatures of habit. If you can identify and understand those habits, and place them within the context of a specific environment that includes interactions with others, you may be able to identify patterns within the patterns. Those themes can be much better indicators of what may or may not happen than the data itself. They not only become better predictors of things to come but can also help identify more effective strategies and tactics to achieve your goals.
This approach requires that a person view an event (desired or historical) from various perspectives to help understand:
- Things that are accidental but predictable because of human nature.
- Things that are predictable based on other events and interactions.
- Things that are the logical consequence of a series of events and outcomes.
Aside from the practical implications of this approach I find it fascinating relative to AI and Predictive Analysis.
For example, by understanding the recurring themes and triggers you can monitor data and activities proactively. That is actionable intelligence that can be automated and incorporated into a larger system. Machine Learning and Deep Learning can analyze tremendous volumes of data from a variety of sources in realtime.
Combine that with Semantic Analysis, which is challenging due to the complexity of taxonomies and ontologies, and now that system more accurately understand what is really happening in order to make accurate predictions. Add in spatial and temporal data such as IoT, metadata from photographs, etc. and you should have the ability to view something as though you were very high up – providing the ability to “see” what is on the path ahead. It is obviously not that simple, but it is exciting to think about.
From a practical perspective, keeping these thoughts in the back of your mind will help you see details that other people have missed. That makes for better analysis, better strategies, and better execution.
Who wouldn’t want that?
New Perspectives on Business Ecosystems
One of the many changes resulting from the COVID-19 pandemic has been a sea change in thoughts and goals around Supply Chain Management (SCM). Existing SCM systems were up-ended in mere months as it has become challenging to procure raw materials to components, manufacturing has shifted to meet new unanticipated needs, and logistics challenges have arisen out of health-related staffing issues, safe working distances, and limited shipping options and availability. In short, things are a mess!
Foundational business changes will require modern approaches to Change Management. Change is not easy – especially at scale, so having ongoing support from the top down and providing incentives to motivate the right behaviors, actions, and outcomes will especially critical to the success of those initiatives. And remember, “What gets measured gets managed,” so focusing on the aspects of business and change that really matter will become a greater focus.
Business Intelligence systems will be especially important for Descriptive Analysis. Machine Learning will likely begin to play a larger role as organizations seek a more comprehensive understanding of patterns and work towards accurate Predictive Analysis. And of course, Artificial Intelligence / Deep Learning / Neural Networks use should accelerate as the need for Prescriptive Analysis grows. Technology will provide many of the insights needed for business leaders to make the best decisions in the shortest amount of time that is both possible and prudent.
This is also the right time to consider upgrading to a modern business ecosystem that is collaborative, agile, and has the ability to quickly and cost-effectively expand and adapt to whatever comes next. Click on this link to see more of the benefits of this type of model.
Whether you like it or not, change is coming. So, why not take a proactive posture to help ensure that this change is good and meets the objectives your company or organization needs.
Changes like this are all-encompassing so it is helpful to begin with the mindset of, “Win together, Lose together.” In general, it helps to have all areas of an organization moving in lockstep towards a common goal but at a critical juncture like this that is no longer an option.
Blockchain, Data Governance, and Smart Contracts in a Post-COVID-19 World
The last few months have been very disruptive to nearly everyone across the globe. There are business challenges galore; such has managing large remote workforces – many of whom are new to working remotely, and managing risk while attempting to conduct “business as usual.” Unfortunately for most businesses, their systems, processes, and internal controls were not designed for this “new normal.”
While there have been many predictions around Blockchain for the past few years it is still not widely adopted. We are beginning to see an uptick in adoption with Supply Chain Management Systems for reasons that include traceability of items – especially food and drugs. But large-scale adoption has been elusive to date.
My personal belief is that we will soon begin to see large shifts in mindset, investments, and effort towards modern digital technology driven by Data Governance and Risk Management. I also believe that this will lead to these technologies becoming easier to use via new platforms and integration tools, and that will lead to faster adoption by SMBs and other non-Enterprise organizations, and that will lead to the greater need for DevOps, Monitoring, and Automation solutions as a way to maintain control of a more agile environment.
Here are a few predictions:
- New wearable technology supporting Medical IoT will be developed to help provide an early warning system for disease and future pandemics. That will fuel a number of innovations in various industries including Biotech and Pharma.
- Blockchain can provide the necessary data privacy, data ownership, and data provenance to ensure the veracity of that data.
- New legislation will be created to protect medical providers and other users of that data from being liable for missing information or trends that could have saved lives or avoided some other negative outcome.
- In the meantime, Hospitals, Insurance Providers, and others will do everything possible to mitigate the risk of using the Medical IoT data, which could include Smart Contracts as a way to ensure compliance (which assumes that there is a benefit being provided to the data providers).
- Platforms may be created to offer individuals control over their own data, how it is used and by whom, ownership of that data, and payment for the use of that data. This is something that I wrote about in 2013.
- Data Governance will be taken more seriously by every business. Today companies talk about Data Privacy, Data Security, or Data Consistency, but few have a strategic end-to-end systematic approach to managing and protecting their data and their company.
- Comprehensive Data Governance will become both a driving and gating force as organizations modernize and grow. Even before the pandemic there were growing needs due to new data privacy laws and concerns around areas such as the data used for Machine Learning.
- In a business environment where more systems are distributed there is an increased risk of data breaches and Cybercrime. That will need to be addressed as a foundational component of any new system or platform.
- One or two Data Integration Companies will emerge as undisputed industry leaders due to their capabilities around MDM, Data Provenance & Traceability, and Data Access (an area typically managed by application systems).
- New standardized APIs akin to HL7 FHIR will be created to support a variety of industries as well as interoperability between systems and industries. Frictionless integration of key systems become even more important than it is today.
- Anything that can be maintained and managed in a secure and flexible distributed digital environment will be implemented as a way to allow companies to quickly pivot and adapt to new challenges and opportunities on a global scale.
- Smart Contracts and Digital Currency Payment Processing Systems will likely be core components of those systems.
- This will also foster the growth of next generation Business Ecosystems and collaborations that will be more dynamic in nature.
- Ongoing compliance monitoring, internal and external, will likely become a priority (“trust but verify”).
All in all this is exciting from a business and technology perspective. It will require most companies to review and adjust their strategies and tactics to embrace these concepts and adapt to the coming New Normal.
The steps we take today will shape what we see and do in the coming decade so it is important to quickly get this right, knowing that whatever is implemented today will evolve and improve over time.
Good Article on Why AI Projects Fail
Today I ran across this article that was very good as it focused on lessons learned, which potentially helps everyone interested in these topics. It contained a good mix of problems at a non-technical level.
Below is the link to the article, as well as commentary on the Top 3 items listed from my perspective.
https://www.cio.com/article/3429177/6-reasons-why-ai-projects-fail.html
Item #1:
The article starts by discussing how the “problem” being evaluated was misstated using technical terms. It led me to believe that at least some of these efforts are conducted “in a vacuum.” That was a surprise given the cost and strategic importance of getting these early-adopter AI projects right.
In Sales and Marketing you start the question, “What problem are we trying to solve?” and evolve that to, “How would customers or prospects describe this problem in their own words?” Without that understanding, you can neither initially vet the solution nor quickly qualify the need for your solution when speaking with those customers or prospects. That leaves a lot of room for error when transitioning from strategy to execution.
Increased collaboration with Business would likely have helped. This was touched on at the end of the article under “Cultural challenges,” but the importance seemed to be downplayed. Lessons learned are valuable – especially when you are able to learn from the mistakes of others. To me, this should have been called out early as a major lesson learned.
Item #2:
This second area had to do with the perspective of the data, whether that was the angle of the subject in photographs (overhead from a drone vs horizontal from the shoreline) or the type of customer data evaluated (such as from a single source) used to train the ML algorithm.
That was interesting because it appears that assumptions may have played a part in overlooking other aspects of the problem, or that the teams may have been overly confident about obtaining the correct results using the data available. In the examples cited those teams did figure those problems out and took corrective action. A follow-on article describing the process used to make their root cause determination in each case would be very interesting.
As an aside, from my perspective, this is why Explainable AI is so important. There are times that you just don’t know what you don’t know (the unknown unknowns). Being able to understand why and on what the AI is basing its decisions should help with providing better quality curated data up-front, as well as being able to identify potential drifts in the wrong direction while it is still early enough to make corrections without impacting deadlines or deliverables.
Item #3:
This didn’t surprise me but should be a cause for concern as advances are made at faster rates and potentially less validation is made as organizations race to be first to market with some AI-based competitive advantage. The last paragraph under ‘Training data bias’ stated that based on a PWC survey, “only 25 percent of respondents said they would prioritize the ethical implications of an AI solution before implementing it.”
Bonus Item:
The discussion about the value of unstructured data was very interesting, especially when you consider:
- The potential for NLU (natural language understanding) products in conjunction with ML and AI.
- This is a great NLU-pipeline diagram from North Side Inc. in Canada, one of the pioneers in this space.
- The importance of semantic data analysis relative to any ML effort.
- The incredible value that products like MarkLogic’s database or Franz’s AllegroGraph provide over standard Analytics Database products.
- I personally believe that the biggest exception to assertion this will be from GPU databases (like OmniSci) that easily handle streaming data, can accomplish extreme computational feats well beyond those of traditional CPU based products, and have geospatial capabilities that provide an additional dimension of insight to the problem being solved.
Update: This is a link to a related article that discusses trends in areas of implementation, important considerations, and the potential ROI of AI projects: https://www.fastcompany.com/90387050/reduce-the-hype-and-find-a-plan-how-to-adopt-an-ai-strategy
This is definitely an exciting space that will experience significant growth over the next 3-5 years. The more information, experiences, and lessons learned shared the better it will be for everyone.
The Unsung Hero of Big Data
Earlier this week I was reading a blog post regarding the recent Gartner Hype Cycle for Advanced Analytics and Data Science, 2015. The Gartner chart reminded me of the epigram, “Plus ça change, plus c’est la même chose” (asserting that history repeats itself by stating the more things change, the more they stay the same.)
To some extent that is true, as you could consider today’s Big Data as derivative of yesterday’s VLDBs (very large databases) and Data Warehouses. One of the biggest changes IMO is the shift away from Star Schemas and practices implemented for performance reasons such as aggregation of data sets, use of derived and encoded values, the use of surrogate and foreign keys to establish linkage, etc. Going forward it may not be possible to have that much rigidity and be as responsive as needed from a competitive perspective.
There are many dimensions to big data: Huge sample of data (volume), which becomes your universal set and supports deep analysis as well as temporal and spatial analysis; A variety of data (structured and unstructured) that often does not lend itself to SQL based analytics; and often data streaming in (velocity) from multiple sources – an area that will become even more important in the era of the Internet of Things. These are the “Three V’s” that people have been talking about for the past five years.
Like many people, my interest in Object Database technology initially waned in the late 1990’s. That is, until about four years ago when a project at work led me back in this direction. As I dug into the various products I learned that they were alive and doing very well in several niche areas. That finding led to a better understanding of the real value of object databases.
Some products try to be, “All Vs to all people,” but generally what works best is a complementary, integrated set of tools working together as services within a single platform. It makes a lot of sense. So, back to object databases.
One of the things I like most about my job is the business development aspect. One of the product families I’m responsible for is Versant. With the Versant Object Database (VOD – high performance, high throughput, high concurrency) and Fast Objects (great for embedded applications). I’ve met and worked with some brilliant people who have created amazing products based on this technology. Creative people like these are fun to work with, and helping them grow their business is mutually beneficial. Everyone wins.
An area where VOD excels is with the near real-time processing of streaming data. The reason it is so adept to this task is the way that object map out in the database. They do so in a way that essentially mirrors reality. So, optionality is not a problem – no disjoint queries or missed data, no complex query gyrations to get the correct data set, etc. Things like sparse indexing are no problem with VOD. This means that pattern matching is quick and easy, as well as more traditional rule and look-up validation. Polymorphism allows objects, functions, and even data to have more than one form.
VOD does more by allowing data to be more, which is ideal for environments where change is the norm. Cyber Security, Fraud Detection, Threat Detection, Logistics, and Heuristic Load Optimization. In each case, the key to success is performance, accuracy, and adaptability.
The ubiquity of devices generating data today, combined with the desire for people and companies to leverage that data for commercial and non-commercial benefit, is very different than what we saw 10+ years ago. Products like VOD are working their way up that Slope of Enlightenment because there is a need to connect the dots better and faster – especially as the volume and variety of those dots increases. It is not a, “one size fits all” solution, but it is often the perfect tool for this type of work.
These are indeed exciting times!
Where Ideas, Experiences, and Lessons Learned Intersect 