KDnuggets Home » News » 2016 » May » Opinions, Interviews, Reports » From Insight-as-a-Service to Insightful Applications ( 16:n17 )

From Insight-as-a-Service to Insightful Applications


Applications that combine machine learning, AI, and domain knowledge have strong potential for industry and investors.



By Evangelos Simoudis, Synapse Partners.

Editor's note: This is the third in a three-part series of blog posts by Evangelos Simoudis covering insightful applications (see part one and part two). From an investor's perspective, Evangelos will discuss the changing landscape of data analytics and big data applications, the rise of insightful applications, and key areas in big data that are attracting most VC interest.

We started this series with the premise that insightful applications, essentially the next-generation of big data applications, are the key to effectively addressing important problems, such as autonomous driving. In the first post, we examined how data analytics has evolved over the past 25 years. In the second post, we established how the early success with big data infrastructures has given rise to applications, and we defined a taxonomy of such applications. In this third post, we will discuss insightful applications in more detail and conclude by outlining the future for such applications, along with the sector’s investment potential.

Insight generation leads to insightful applications

A discussion on insightful applications must begin by first defining what is an insight. An insight is a novel, interesting, plausible, and understandable relation, or set of associated relations, that is selected from a larger set of relations derived from a data set. The selected relation, or relations, must lead to the formation of an action plan. This action plan, when applied, results in a change that can be measured through a set of Key Performance Indicators (KPIs). The ability to measure KPIs enables such systems to learn and improve over time. Insights and their associated action plans are generated by complex systems, which were described in an earlier post.

IntelligenceInsightful applications combine big data management with an insight generation system, which incorporates a variety of machine intelligence techniques, (e.g., machine learning, planning and reasoning, natural language processing, vision, and other AI techniques) and appropriate domain knowledge for those techniques to operate effectively. These applications must not only be able to generate an action plan from insights, but they must be able to approximate the necessary resources (e.g., time, money) that will be consumed in the process of executing that plan.

A good example of an insightful application is found in autonomous cars, such as Google’s self-driving car. In order for an autonomous car to function properly, its software must be able to deal with very large volumes of diverse data that are generated in real time from sensors likelidar; make a variety of predictions using numerous sophisticated models; select the most interesting and plausible of these predictions (e.g., within 50 yards, a bicyclist riding ahead of the car will likely enter the car’s lane); and generate an action plan based on these insights (e.g., slow the car by applying the brakes) while constantly evaluating the results of these actions (e.g., make sure the car does not hit the bicyclist). Based on the data that is collected from this incident and the evaluation of the plan that was executed, there is an opportunity for the insightful application to learn.

Insight-as-a-Service: A precursor to insightful applications

Because of their complexity, insight generation systems—and therefore insightful applications—have been hard and expensive to develop, and few insightful applications have been introduced to date. At best, corporations have been utilizing what we have termed “Insight-as-a-Service.” As shown in Figure 1, insights and their associated action plans are generated periodically as part of a service offered by specialized personnel, called “connectors,” in collaboration with data scientists.


Insight-as-a-Service process
Figure 1. Insight-as-a-Service process.
Figure courtesy of Evangelos Simoudis.

At a high level, a connector works with the business user to understand the problem to which insights and actions will be applied. In addition to defining the business problem, the connector gains an understanding of the data that is available and can be used to address the problem. The business problem definition combined with the data description is then “translated” to a data problem, which can be addressed by data scientists. Data scientists, with the help of data stewards, work on the available data to extract patterns and other relations. The connector evaluates the extracted patterns and relations to identify those that constitute insights. The connector then proceeds to associate one or more appropriate action plans with each identified insight, and presents the resulting pairs to the business user. After applying the prescribed actions, the business user communicates to the connector the results of these actions, along with the effectiveness of the actions to address the problem being solved (as determined through the use of a previously defined set of KPIs).

Because of their role in providing Insight-as-a-Service, connectors must have strong knowledge of the processes, technologies, and issues in the industries they work. They must be able to “translate” business problems into data problems, and effectively communicate this translation to the data scientists. Lastly, connectors must be able to fuse the relations extracted by the data scientists and their own domain knowledge to generate insights and action plans, which will be communicated to business users.

Today, Insight-as-a-Service is offered primarily by management consulting firms like IBM, PwC, McKinsey, and Accenture. Because of the skills required and the critical role they play in the insight generation process, connectors tend to be some of the most expensive resources a management consulting firm employs. There are too few such individuals in management consulting firms today. This means that too few corporations are generating insights from their data, focusing instead on areas with potential for a high return on investment (ROI) such as cybersecurity, supply chain optimization, and marketing budget optimization.

An abbreviated example of Insight-as-a-Service

Consider the process a cable company might go through in order to optimize its marketing budget as it tries to determine what percent of its budget to allocate toward reducing customer churn. The marketing department must create a set of actions that are within the limits of the marketing budget. Using his specialized knowledge, the connector must understand the nature of the churn problem and the data that is available to address this problem. He may determine that, as a first step, the company’s subscribers should be scored in terms of their probability to abandon their service, and maybe even their probability to upgrade to a higher level of service. He must then present this as a scoring problem to the data scientist, along with the available data. The data scientist can then proceed to develop the appropriate scoring model(s).

Once the models are developed and the scores have been established, the connector uses his domain knowledge to do the following:

  1. Organize the scored customers into segments.
  2. Select the segments of at-risk customers that are worth paying attention to (this is the insight). For example, at-risk customers may be the segment of heavy cable Internet users that spend 10-20% of their time watching cable programming.
  3. Establishes the actions to take and the percentage of the marketing budget to allocate to each selected segment. For example, offer the at-risk segment a free premium channel upgrade for a period of six months. The cost of this action to the cable company may be $100/customer, but the Lifetime Value (LTV) of each salvaged customer may be $1,000, and the expected target response to the offer may be 30%.

There is increasing interest in developing, and funding, insightful applications. We’re seeing this increased interest because we have a greater understanding about what is involved in providing Insight-as-a-Service and the strong ROI that such solutions already provide. We’re also aware of the early successes with other types of big data applications, as described in the second post of this series, and the ever-increasing availability of big data. Finally, we acknowledge recent advances in the technologies that are used by insight generation systems and, more importantly, the strong demand for more automated, accurate, and faster decision-making.

Areas where we are seeing great opportunity for insightful applications, as well as entrants starting to offer such applications, include:

  1. Cybersecurity in the area of threat intelligence.
  2. Online marketing in the area of multichannel programmatic advertising.
  3. Automotive in the area of autonomous driving.
  4. Health care in the area of personalized medicine.
  5. Financial services in the area of wealth advisory services.
  6. Manufacturing and logistics (across several industries such as retail and consumer packaged goods) in the area of dark warehouses and fully robotic assembly lines. For example, Foxconn is adding 30,000 robots per year in its assembly lines.
  7. Customer experience (across several industries such as retail, financial services, travel, and health care) in the area of automated customer support.
  8. Various industries such as manufacturing, agriculture, and oil exploration, as well as production in areas where IoT data can be used for applications like machine diagnosis, crop yield optimization, oil field optimization, and others.

These areas share the following common characteristics, which make them particularly relevant as insightful applications:

  1. Not enough professionals to provide remedies to problems (e.g., security).
  2. Need for quick response in the execution of actions (e.g., autonomous cars, security, online marketing).
  3. Big data at a scale that makes it impractical or even impossible to sift through for the right insights, but where more data improves the opportunity to identify better insights (e.g., health care, security, marketing).
  4. Need to provide higher quality service that is consistent across all channels at a lower cost (e.g., financial services, manufacturing, logistics).

In general, my thesis is that through insightful applications and a fixed amount of accuracy, we can use lower-cost resources, including less computation, because of the availability of more data.

An interesting early example of an insightful application is IBM’s Watson Oncology Assistant, which IBM is developing in collaboration with various medical centers such as the MD Anderson Cancer Center and the Memorial Sloan Kettering Cancer Center. This insightful application optimizes the decision-making process that determines which patient therapy to recommend based on a) patient genetic data that is generated from DNA sequencing, b) MRI data, c) other documents describing the patient's health history, and d) scientific publications. For example, while DNA sequencing enables us to identify mutations linked to cancers, the data that is generated in the process is voluminous. Many physicians cannot easily make use of all available data because of its volume and variety, and often because they may not be aware of publications describing relevant research results. The Watson Oncology Assistant will be able to address all these problems and make the appropriate therapy recommendations to the attending physician.


Sign Up

By subscribing you accept KDnuggets Privacy Policy