Nando de Freitas AMA: Bayesian Deep Learning, Justice, and the Future of AI

Nando de Freitas is a well-known and -respected machine learning professor at the University of Oxford and a senior research scientist at Google DeepMind. After receiving his PhD from Trinity College in 2000, de Freitas was a post-doctoral fellow at UC Berkeley from 1999 to 2001, and a professor at the University of British Columbia from 2001 to 2014. de Freitas is also a fellow at the Canadian Institute For Advanced Research (CIFAR), as well as the recipient of numerous academic awards.

Reading anything originating from Nando's mind, his intellect is instantly visible, as is his genuine curiosity surrounding a whole host of topics.

de Freitas describes his interests succinctly on his website as follows:

I want to understand intelligence and how minds work. My tools are computer science, statistics, mathematics, and plenty of thinking.

On December 26, 2015, Nando de Freitas participated in an AMA (Ask Me Anything) hosted by the /r/MachineLearning subreddit. What follows is some summarization of, commentary on, and excerpts from said AMA.

Before getting to any specific questions posed by Redditors, I will point out some of the main themes prevalent in the discussion. Beyond specific back and forth on the current state of machine learning, learning theory, and deep learning, the following were prominent throughout the AMA:

1 - Nando seems to be concerned with equality of those influencing the future of artificial intelligence (AI) and machine learning research and implementation. He feels that the need to address the under-representation of women, minorities, and other groups beyond white males goes beyond the simple game of representation by numbers; he believes that it has important societal implications. de Freitas mentions at a few points that it is important to have all groups in society shape its collective future. He believes that education and its quality has a role to play here, and that several issues related to inequality need to be addressed in this realm, including the "pernicious correlation of education quality and real estate costs." Says de Freitas:

I'm not saying it's anyone's fault. I am however saying that we need to look at the roots of this and understand it. I find it crazy to talk about the future of humanity and only involve white males in it.

2 - Unlike a few publicized comments from other AI personalities over the past year, de Freitas does not seem to be worried about SkyNet-style deviant intelligences destroying mankind, but does recognize that technology is changing people and their lifestyles, and that we need to be cognizant of such developments and their repercussions. He also addresses some of the fear-mongering surrounding "self-aware AI."

I think worrying about terminator like scenarios and risk is a bit of a distraction - I don't enjoy much of the media on this. However, worrying about the fact that technology is changing people is important. Worrying about the use of technology for war and to exploit others is important. Worrying about the fact that there are not enough people of all races and women in AI is important. Worrying about the fact that there are people scaring others about AI and not focusing on how to harness AI to improve our world is also important.

3 - Professor de Freitas very clearly spends time pondering, discussing, and philosophizing about what exactly "intelligence" is.

Is intelligence simply a consequence of the environment? Is it deep? Or is it just multi-modal association with memory, perception and action as I allude to above (when talking about waking up hungry)?

4 - He also seems to maintain a positive view of the future that AI can provide humanity.

I do think the deep nets are just one step toward more intelligent computers. By extending our intelligence with machines I think we have better hopes of solving incredibly complex problems like wealth distribution, cancer, and so on. But for this to happen we need strong leaders.

On a more technical level, a lot of general questions regarding the future of deep learning were posed, as were a number of questions related to the marriage of Bayesian methods and deep learning, which is reasonable given Dr. de Freitas' background. We will now have a look at some of the best questions posed to, and best responses from, Nando de Freitas. Numerous questions solicited Nando's insight as to where the fields of AI/machine learning/deep learning are headed in the future.

What's in Store for AI?

Redditor dexter89_kp asks:

I had asked a similar question to Prof LeCun: what do you think are the two most important problems in ML that need to be solved in the next five years. Answer this from the perspective of someone who wants to pursue a PhD in ML.

Nando de Freitas responds:

Here's a few topics: Low sample complexity Deep RL and DL. Applications that are useful to people (healthcare, environment, exploring new data). Inducing programs (by programs I mean goals, logical relations, plans, algorithms, ..., etc). Energy efficient machine learning.

Question from rmcantin:

One of the great features of both MC and NN methods is their potential to scale up with the available resources. Do you think there will be a second rebirth of Monte Carlo methods in a near future when we have the computational power to sample a billion (or trillion) particles to estimate the weights of a deep NN and do full-Bayes deep learning? Or do you think Bayesian optimization would have already catch up in that problem? :-)

de Freitas:

I'm waiting for Yee Whye Teh or Arnaud Doucet to lead the new Monte Carlo revolution ;) However, we need to make sure we understand deep learning first. The mathematical principles behind these high-dimensional models and the optimisation processes we use for learning are not well understood.
...
I do think Bayesian optimization is much needed in deep learning. But it must done properly and it will be hard and a lot of work. I'm waiting for people like you to do it ;)

Question from spaceanubis1:

What are your thoughts and ideas on unsupervised learning (or maybe one-shot learning)? How do you think this will be achieved in the coming future?

de Freitas:

For me, learning is never unsupervised. Whether predicting the current data (autoencoders), next frames, other data modalities, etc., there always appears to be a target. The real question is how do we come up with good target signals (labels) automatically for learning? This question is currently being answered by people who spend a lot of time labelling datasets like ImageNet.

Question from maltoss:

Could you elaborate on what the next steps in working with Bayesian methods and deep learning will be according to you?

de Freitas:

Some folks use information theory to learn autoencoders - it's not clear what the value of the prior is in this setting. Some are using Bayesian ideas to obtain confidence intervals - but the bootstrap could have been equally used. Where it becomes interesting is where people use ideas of deep learning to do Bayesian inference. An example of this is Kevin Murphy and colleagues using distillation (aka dark knowledge) for reducing the cost of Bayesian model averaging. I also think deep nets have enough power to implement Bayes rule and sampling rules. This could turn out to be a lot of fun!

Staying with Bayesian methods, here's a question from HillbillyBoy:

Bayesian Optimization seems to be a hot topic nowadays:

1. What results/breakthroughs have changed things since early work in the 90s?

2. Where do you see the field going in the next five years?

de Freitas:

1. There's been a lot of methodological and theoretical progress. Ryan Adams and his gang, Philipp Hennig, Frank Hutter, Matt Hofmann, Ziyu Wang, Bobak Shahriari, Ruben Martinez-Cantin and many many others (see our recent Bayesian optimization review) have been making important innovations.

2. We need an emphatic demonstration: e.g. fully automate Torch or Caffe, so that given a dataset and specification of the problem (e.g. ImageNet site), Bayesian optimisation automatically generates the code (including architecture and algorithm specification) that wins ImageNet.