Machine Learning: Separating Hype From Reality

When it comes to business value and ROI, does machine learning live up tot he claims? We’ll explore a pure machine learning approach through the lens of a typical enterprise use case.

By Marco Varone, CTO and Founder, Expert System.

Machine Learning

Today’s push behind machine learning and artificial intelligence is so powerful that it has led to some pretty high expectations for performance and deliverables. When it comes to business value and ROI, can it really live up to the claims? Evaluating the value of a full machine learning project requires moving past the hype and managing the realities of this evolving technology, one example at the time. We’ll look at the realities of a pure machine learning approach through the lens of a typical enterprise use case.

If we’re to believe the past couple of years’ worth of marketing hype, machine learning is a magic box, supported by an evolved approach, strengthened by the latest technology and most importantly, able to effortlessly produce results. The high performance and even greater ROI implied is made possible by computers that learn by themselves, with minimal effort required on the part of the customer and rapid, painless implementation. That’s quite a vision! From 5,000 feet, the workings of such a technology/approach may seem like magic, but a closer look is warranted.

Zooming in, we see that this vision vastly understates what’s going on inside the box, which, as this great article by @jasontanz in Wired highlights (, is part of the shift that machine learning is ushering in. He writes: “With machine learning, the engineer never knows precisely how the computer accomplishes its tasks.” While computers still rely on humans to train them, we haven’t yet been able to reverse engineer the process of how they learn. Basically, we have to trust that the system will provide results—and if it doesn’t, all we can do is keep feeding it data and information.

This highlights another aspect of machine learning that, for the time being, is potentially problematic: reaching the performance required by these machines is not something that just anyone can do. In the article, Google’s own Demis Hassabis, who leads its DeepMind AI team is quoted, saying, “It’s almost like an art form to get the best out of these systems. There’s only a few hundred people in the world that can do that really well.”

This might strike fear in the hearts of anyone considering a machine learning approach for managing information, after all, implementing a black box that only 200 people in the world can train may not be the best investment. However, there is more than one way to take advantage of machine learning while taking these realities into account.

Let’s take a look at a simple and common enterprise use case of machine learning applied to an information repository of internal and external unstructured information, to be used to fuel everything from automatic email management to strategic research and innovation, and operational risk management. Starting with the raw data, the first step will be to classify documents and information based on a customer-developed taxonomy that is specific to the business. In this case, let’s say that the taxonomy contains 200 nodes, typical of an average enterprise taxonomy. What does the workload/investment really look like?

  1. We know that Machine learning uses a black box approach. System performance improves based on the amount of training documents fed to the system. By definition there is a limit to the level of improvement possible, and it is often difficult to understand why the system has improved or how you can improve it further.
  2. A 200-node category project requires identification and manual tagging of at least a few thousand documents per node, i.e. several tens of thousands of documents to be tagged manually.
  3. A machine learning approach requires identification of several documents for EACH node of the taxonomy; this alone can be a complicated and time consuming task. Without these, a machine learning approach would require a basic and primitive list of keywords per node.
  4. Computing time could easily take many hours. If any mistakes are discovered in the assessment phase—which is not uncommon—the whole project may have to be run again, from scratch.
  5. If the trained system needs to be modified for any reason (i.e. changing the taxonomy or adding extraction to categorization),l the entire process must go back to square one.

At this point, the customer is so invested in the project—not only in terms of time and resources, but let’s be honest, reputation is factor here as well—that they “decide” to be happy with whatever results the system will provide. I understand the need to make the best of what you have, but as I mentioned earlier, there is a better way.

Combining machine learning with language understanding and rule based approaches offers a valid alternative that addresses the limitations and risks of a full machine learning approach, and provides the ROI and TOC that you require.

Marco VaroneBio: Marco Varone, founder, president and CTO of Expert System, is one of the leading experts on semantic technology and natural language processing. He created the Cogito platform, which is the basis of all Expert System’s products. He has worked in the field application of semantic technology in every advanced context: search engines, text analytics, natural language interfaces, Q & A systems, automatic categorization and many others.