Nando de Freitas AMA: Bayesian Deep Learning, Justice, and the Future of AI

A Little More Personal

Following are some questions about things closer to Nando.

Question from zhongwenxu:

What are the key differences between your research life at DeepMind and the one at Oxford, except for the great infrastructure and machine resources?

de Freitas:

DeepMind has a vibrant research atmosphere with an amazing concentration of bright people focused on solving problems - every week someone there totally blows my mind. The support is amazing. The collegiality is wonderful. Oxford is also an outstanding place to work. However, at DeepMind, there is more focus on problems and grand challenges than on techniques (both are however important). There's a lot less admin in industry too, and they pay way better than universities!! It's shocking how low the salaries of computer science professors and teachers are, specially in Europe, in comparison to many other jobs that in my view contribute much less. Profs should at least be able to afford rent - they work so bloody hard.

Question from up7up:

Is strong AI possible? What prevents its implementation? Combinatorial explosion? Curse of dimensionality? P versus NP problem? Something else?

(Clarifying link added afterward by the author: https://en.m.wikipedia.org/wiki/Artificial_general_intelligence)

de Freitas:

Thanks. I don't think we have a good grasp on what intelligence is. Also, our understanding of what constitutes intelligence keeps changing.

Building machines that can do what humans do does however seem plausible. Humans do not solve NP hard problems or combinatorial problems with any ease. There appear to be much harder problems than matching human intelligence.

As a pertinent follow up to this, and to demonstrate some of how Nando seems to spend his time contemplating "intelligence," this excerpt from an answer to a different question seems relevant here:

What do you think would happen if you place a human all its live in a dark anechoic chamber with only a drip of food directly into the veins. 10 years later I doubt you would see much intelligence, and this despite the fact that millenia of environmental adaptation through embodiment has been passed through genes.

The following question from lars_ was particularly suited to getting to know what de Freitas spends his time contemplating:

What questions are you asking yourself these days? What question would you most like to find the answer to?

Here is an excerpt from Nando's answer (there were simply too many great questions being pondered to include them all!):

(v) What cool datasets can I harness to learn stuff? I love it when people use data in creative ways. One example is the recent paper of Karl Moritz Hermann and colleagues on teaching machines to read. How can we automate this? This automation is to me what unsupervised learning is about.

(vi) Is intelligence simply a consequence of the environment? Is it deep? Or is it just multi-modal association with memory, perception and action as I allude to above (when talking about waking up hungry)?

(viii) When will we finally fully automate the construction of vanilla recurrent nets and convnets? Surely Bayesian optimization should have done this by now. Writing code for a convnet in Torch is something that could be automated. We need to figure out how to engineer this, or clarify the stumbling blocks.

What Do You Suggest?

A number of professional and academic career advice questions were posed. Here are a few of the more helpful highlights.

Question from datagibus420:

Hello Prof. Nando! - If you had one book on ML to recommend, which one would you pick? - Do you plan to build a MOOC on deep learning, or more generally on machine learning?

BTW thanks for uploading your lectures on YouTube, they are awesome!

de Freitas:

I love Kevin Murphy's textbook. He is currently writing a new version which will have a much better deep learning section. It'll follow more or less what I discussed in my youtube deep learning course.

Thank you for your support and positive feedback.

Question from learnin_no_bully_pls:

I want to learn ALL the math required to properly study machine learning research papers and understand them. What do I need to add to my study to-do list?

de Freitas:

Calculus and linear algebra are the basics. Make sure you know gradients, linear systems of equations, basics of optimisation, eigenvalues, ..., etc. Kreyszig's Advanced Engineering Mathematics provides enough background. Mathematics is useful to the extent with which it enables us to learn new abstractions (e.g. recurrences and functions) and be able to reason with such abstractions. This process of reasoning can lead to new discoveries, faster more succinct arguments or simply more precise communication of ideas.

Question from Pafnouti:

I read a lot of papers from Facebook, Google and co. which seem to be very empirical, as in "We have tried this and that and here are our results", without much theoretical work in them. So I'm wondering if there is a point spending 3-4 years in a PhD doing this stuff just to get a shiny paper at the end, or if getting experience in this field (although in a less intense way than if I was doing a PhD) in my current job would be enough.

de Freitas:

Most industrial labs do require that you have a PhD to work in research. I strongly recommend a PhD in machine learning as you learn a lot. I also don't think that "We have tried this and that and here are our results" is an accurate characterisation of work done at Google, Facebook, Twitter, Microsoft and other labs. There are important advances in methodology and theory coming from industry.

Having said this, Turing didn't have a PhD when he transformed the world of AI and philosophy!

And, importantly, vivanov asks:

TensorFlow vs Torch?

To which de Freitas, both diplomatically and practically, responds:

Ha ha! Both for now.

After sharing some personal stories, some lessons learned, and some resultant core beliefs that he has developed over the years, which mostly come from a seemingly very deep sense of justice, and which (necessarily) focus on the bad in the world in order to explain how particular events have molded him as a human, he shares this quote, otherwise unrelated to machine learning, but a great way to end this article:

Fortunately, the world is a much better place today than it ever was. I hope Yoshua Bengio is right when he tells me that he believes in people and that tolerance and compassion will prevail.

I highly suggest reading the entire AMA if you have the time. It will not disappoint, on a number of levels.

Bio: Matthew Mayo is a computer science graduate student currently working on his thesis parallelizing machine learning algorithms. He is also a student of data mining, a data enthusiast, and an aspiring machine learning scientist.

Related:

• Juergen Schmidhuber AMA: The Principles of Intelligence and Machine Learning
• Geoffrey Hinton talks about Deep Learning, Google and Everything
• Top /r/MachineLearning Posts, Apr 12-18: Andrew Ng AMA, Autoencoders, and Deep Learning Textbooks