5 Tribes of Machine Learning – Questions and Answers

On Nov 24 I moderated ACM Webinar with Pedro Domingos.

Pedro, one of the leading researchers in Machine Learning and Data Science, expounded on his excellent book, "The Master Algorithm" where he explained the approaches of 5 tribes of Machine Learning: symbolists, Connectionists, Evolutionaries, Bayesians, and Analogizers, and how those methods can be combined in search of the master algorithm for Machine Learning that will revolutionize our world.



Over 1,500 people attended the webinar.

After introducing Pedro I was prepared to relax and enjoy the presentation, but my role as a moderator turned out unexpectedly intense, when lots of questions - which I had to read and quickly prioritize - started coming in after only a few minutes.

Pedro had time to answer only about 5 from over a hundred submitted, but many questions were very good, so I asked to send me answers to a few more interesting questions.

See also Pedro's answer to my earlier question:

Is Deep Learning the Master Algorithm?
(bottom of the post)

Here are Pedro Domingos answers to additional questions from the webinar that he did not have time to answer live.


 


Question by Juan Alvarado: In order to obtain the Master Algorithm. Have you considered its impossibility because of Goedel incompleteness theorem ?

Pedro Domingos: The Master Algorithm only learns what can be learned; it can't prove statements that are unprovable by Goedel's theorem.

Pramod Anantharam: Do you think interpretable models would win over blackbox based approaches in future?

PD: It depends on the application. Interpretabilty is a very important property, and for many applications it overrides the extra accuracy that can be gained from (say) deep learning.

Mohamed Helmy: Can we get a brief about rules engines and symbolic logic (decision trees) ?

PD: Sure, check out Chapter 3 of "The Master Algorithm" or Weeks 2 and 3 of my MOOC (Machine Learning on Coursera).

Jesus Morales: For a college student that wants to follow this path what would be the recommendations that he or she needed to know before hand?

PD: Take a MOOC, read a textbook, play with some algorithms. See the further readings section in "The Master Algorithm".

Cristal Jones-Harris, Do you have a sample of the text on the 360 degree recommender?

PD See my Wall Street Journal piece "Get Ready for Your Digital Model".

Subhra Mazumdar: Will a 360 degree recommender system affect the decision making capability of human? Will it be a symbiotic relation of man and machine or tend to be more masters-slave type?

PD: The 360-degree recommender is an extension of your brain, so you're in control. All it does is try to make the choices that you would if you had the time, and when it gets it wrong it should learn to do better next time.

Abdelaziz Mahoui, how would serendipity manifest itself in th new 360 recommender?

PD: The recommender should have a random component, just like real life.

Delane Pickel: The problem with a "master algorithm" implies that there is only one. The ultimate learning machine will have to have some kind of understanding for successful learning. Some learning can actually become destructive. Isn't that what cancer is?

PD: There is more than one version of the Master Algorithm, in the same way that there are many computing schemes equivalent to a Turing machine. What it learns depends on the data and evaluation function you give it.

Aude Dufresne: what if evaluation and optimisation change reality

PD: Not sure what you mean, but deploying the results of learning often results in the people who are being modeled changing their behavior. This is an important and under-explored problem, but see my KDD-04 paper "Adversarial Classification"

Michael Valenzuela: The No Free Lunch for Supervised Learning By Wolpert proves no "Universal" machine learning algorithm can exist. How do you reconcile this proof with your goal?

PD: The Master Algorithm only has to work in our world, not in all possible worlds, and the "no free lunch" theorem is about the latter. Also, the Master Algorithm doesn't just have data as input; it also inputs knowledge (e.g., logic formulas).

Jim Talley: First of all, that was a very nice summary. In my view though, the piece that is still missing is perception, or more specifically learning to structure and prepare the inputs to these five paradigms. Any thoughts about this? Do you consider it to already be part of one or more of these paradigms?

PD: Each paradigm has its approach(es) to this problem, and indeed it needs to be solved. For example, the main claim of deep learning is that it learns its own representations from raw data (e.g., pixels).

Joanna Biega: What about traditional data structures for storing large amounts of data ? Are we forgetting that we need not only good machine learning algorithms or asymptotically linear algorithms and efficient data structures for big data ?

PD: Certainly a general-purpose learner must be efficient, and data structures are a big part of that.

Bhaskar Veeraraghavan, Are there any applications for Active reinforcement learning?

PD: Reinforcement learning is active learning in some sense. I hear that Deep Mind is applying it to Web search, which will be a major application if it's deployed in Google's engine.

Habibollah Daneshpajouh, as you have mentioned there is a type of problem that each of this tribes are doing better in, how can we detect the best tribe of ML to be used for a specific problem

PD: Despite much research, no one has a good theoretical answer to this problem, but there are many practical heuristics, and you can also just try all of them.

Latha Krishnaswamy, ML algorithms are usually unsupervised. What in your view allow or suffer from lack of human supervision?

PD: Actually, supervised learning is used more often than unsupervised. Definitely use supervision if it's available (or not too expensive to acquire).

Parlinggoman Hasibuan, do you think machine learning can make decision and replace human ?

PD: Yes, in many areas it already has (e.g., credit scoring, direct marketing).

Raul Sierra: You talk about the master algorithm, but isn't it the case that we also need something like the master tasks to evaluate the master algorithm?

PD: Certainly. We need a sufficiently varied and difficult set of tasks that if a learning algorithm can solve it, we're justified in claiming it's the Master Algorithm.

Theodore Grammatikopoulos: Talking from the possibility of a Master Algorithm building an algo that could probably choose for the difference pieces of data (columns, rows) a different algo to use giving an appropriate weight would not be a great idea?

PD: This is really a type of model ensemble, which as I discussed is better than a single model, but not really the Master Algorithm yet.

Wee Kiat: Do you think it is possible for master algorithm to solve problems that are unsolvable currently? If so, will machine learning be more superior than our brain?

PD: Yes, definitely, like curing cancer. But it's not so much that machine learning will be superior to our brain, but that our brain extended with machine learning will be superior to our brain without it.

Victor: What do you think will be the next near breakthrough in ML?

PD: I don't know, but I'm working on some candidates :)