Deep Learning and Startups: Notes on Rework Conference, San Francisco

The Rework Deep Learning conference came to San Francisco this past January, and showcased both prominent deep learning researchers and startups. Get an overview of the proceedings with notes from an attendee.

By Lisha Li (@lishali88).

Editor's note: This is is the first half of an in-depth set of notes written by an attendee of the Rework Deep Learning Conference, which focus mainly on researchers in the field. The second half of the notes are available in the link to the original at the bottom of the post, and cover deep learning startups which were in attendance, for those interested.

General Thoughts

This past week I went to the Rework Deep Learning conference. It was a good two days of talks by both top researchers in DL and companies applying DL. I made summary notes for the talks and divided them by ‘Research’ and ‘Companies’. Within these two categories they are simply in order of who talked first. There was also a Q&A with Andrew Ng which I stuffed under ‘Research’.

The research talks were an excellent line up. All presented on recent work, though if you keep up with the literature, they should be familiar. I won’t mention any in particular here because I think their summaries are all worth reading if you are not yet familiar.

There is clearly a lot of excitement about this area of research for its applications, the companies at this conference is that they can be generally divided into the following categories. The fairly large ones (Pinterest, Twitter, eBay, Flickr…etc) use DL to be competitive. It is another tool in the bag of various other statistical/machine learning techniques they were already using, but can work well with the deluge of visual data and natural language processing tasks they must do at scale. I.e they already have a product, DL makes it marginally or perhaps much better.

Of the smaller companies, some with medical applications had very clear and tractable problems (see for instance BayLabs below for ultrasound diagnosis) along with a well defined market. These companies mainly take advantage of the advances in computer vision to do classification, which can make diagnosis both more efficient and more accessible. This seems convincing as long as there is a well labelled set for the problem, and in the case of ultrasound diagnosis, partnering with hospitals and research institutions seems to provide just that. Nonetheless, the regulatory tangle that surrounds businesses in healthcare push such businesses (at least in the near future) to assume an assistance role, so that the responsibility of diagnosis/practice still falls with a healthcare practitioner.

There were also two companies focusing on the analysis of satellite images. One can imagine how valuable this data is once obtained (counting world oil supplies is one good example, as well as measuring construction activity in China as a proxy for the Chinese economy). So there is certainly a market (ranging from hedge funds to even non-profits). The price paid for the data is another issue, though as companies like SpaceX lower the cost of space tech, we can expect growth in the availability of private suppliers of satellite data (for better or for worse), so a business founded on this supply is a good start. Finally, there were some traditional consumer facing verticals that try to use DL to improve the personalization aspect of human-computer interaction (e.g in shopping and recommendations, organizing your photo library…etc).

These verticals seem more challenging to me as standalone businesses; An issue to consider in creating a successful business using DL is to find a vertical that is ok with the accuracy level that is achieved by the state of the art, while being willing to pay a lot (profitable amounts) of money for it. In short are there people who care enough given the achieved level of accuracy to pay for your product? Though a lot of the companies look promising, it is not obvious they have succeeded yet. Moving away from specific verticals, it seems easier to be business facing and build a platform to do DL, as many of the companies below do, especially since the companies that can afford it have already built DL teams (large companies mentioned above).

All in all I find this area of entrepreneurial activity very exciting and I plan on posting more analysis on startups in this space on my blog.

Keep in mind the notes are a bit unpolished as I took them in real time for all the talks.


Open AI

Speaker/Company name: Ilya Sutskever/OpenAI

Theme of presentation: Deep Learning Theory and Applications

Technology advancements: Deep Neural Networks (DNN) are very expressive and can represent solutions to very difficult problems we are interested in. They are also trainable from actual data (that is, we can find good weights). Speaker gives a soft argument by analogy with biological neurons to convince ourselves about why neural nets should be able to solve similar problems biological neural nets solve (biological neurons fire slowly a small number of times in a second). Presented on new research coined “Neural GPU” to address the issue of how to make DNN generalize better by studying how they learn algorithms. Results: can train binary number addition of up to 20 bits and generalize to 2000 bits, same for multiplication. Also highly parallel.

Key take-away: DNNs strike a sweet spot between trainability and expressiveness.

Speaker/Company name: Fireside Chat with Andrew Ng.

Main things discussed: we need to work together to educate the community on what is possible and what is not, so that there is no artificial scare. Andrew is very optimistic about the technology, and also about how the community approaches it in an open sourced way. There needs to be more of a role from government, but not in constraining solutions. Rather their role should be focusing on setting safety standards.

More in depth summary (paraphrased): Optimistic about self-driving. Automakers increasing levels of autonomy (but there is a chasm). Start with a bus route (more feasible to make it safe), then grow from that. Passionate about role of government in AD (autonomous driving). Get behind and help grow this system.

Question: Viability, private and public attacks?

Answer: Distinguish between come and stop (vision cannot distinguish at this point). So need improvements. But cars have no blind spots. So there are better and worse of self driving cars (manage our expectations, i.e should be different). Analogy: trains are more dangerous than horses. Society had to understand things not to do in front of trains. Similarly to autonomous cars. Want people to recognize the car and be primed to expect different behaviours (so physically make the car distinctive). Not the prettiest but most easily recognizable. See the world different than people (ACars).

Q: Anticipate accident from AI. If there is a PR backlash, how to respond.

A: Be unapologetically optimistic about AI. Be transparent about what AI can and cannot do. Fears that reckless company that release features that are not ready. Careful, honest, open at explain to regulators and public.

Q: What legislation do you support and not support?

A: Regulating safety, but putting a human in the car to achieve that is unimaginative. Get a standard rather than a specific solution (such as putting a human in the car).

Q: pledge for safe AI. Trusting companies?

A: These decision makers are just people in this room. Optimistic about technology.

Q: Musk fear of AI, where does this stem from?

A: AI so smart in so many verticals, so many data we have from these problems, has risen very rapidly. Gap between AI and what humans do, so we don’t have the appropriately formulated data for supervised learning. So the value created from AI is from supervised learning. Tempting to exponentially extrapolate, but amount of data that needs to be formatted is not growing exponentially so fear is very far.

Q: Projecting, what are the AI ethical issues in the 5 year out in the future?

A: AI create so much value. So much bigger than any company (google, baidu, fb). Eg. Iphone transformed our technology. Less worried about problems. Growing the ecosystem. Have everyone benefit from a piece of it. Encourage the growth.

Q: Narrative is an AI race? You don’t see it as that?

A: Sure we compete for talent. We are at the start of growing an ecosystem, so all need to work to make it bigger. Shifting from software and algorithms to data (company focus shift). Open publishing papers. Role of open source software.

Q: How do you convince businesses to participate in the open approach?

A: Good PR, accrue talent. Yes we need to make profit, but it is aligned with helping silicon valley can lift together. Eg. Coursera. Wouldn’t be this excited about job if I didn’t really believe in it.

Q: Changes in IP regulations to help this vision?

A: Patent system is badly broken. Valuable IP is data. Value captured in short amount of time, so hence broken patent system. In addition, for data, what is the social contract? Complicated conversation.

Q: Affects of AI on jobs. How should businesses think about this? You? Role of government?

A: AI will create massive labour displacement (putting people out of jobs). Truck drivers for instance. Want to encourage public education of what AI is. The more we can educate, the higher chance we can make better decisions. Coursera is a positive impact (so educate people whose jobs we displace). Basic income (he supports). Or negative income tax? More discourse on this. Supports university income for studying so we have high chance of entering work force.