Small Data requires Specialized Deep Learning and Yann LeCun response
For industries that have relatively small data sets (less than a petabyte), a Specialized Deep Learning approach based on unsupervised learning and domain knowledge is needed.
Until all fields have a data repository as big as Google or Facebook (Quadrant B), I think Deep Learning engineers will exist separately (split between Generalized Deep Learning and Specialized Deep Learning engineers) as their different approaches work for their individual objectives.
Best regards,
Shalini Ananda, PhD.
Bio: Shalini Ananda is a Co-Founder of Quantified Skin, Data scientist with a focus on deep learning and computational chemistry. Lover of street music : music should be enjoyed live!
PS: Yann was kind enough to reply to this letter:
"Hi Shalini, I don’t disagree with you.
I have been confronted with many Quadrant C problems in my career. Much of our research on deep learning in the mid-2000s was actually focused on unsupervised learning precisely because most of the datasets we had were squarely in Quadrant C.
Computer vision datasets for object recognition may be large now (though not petabyte-large) but they were rather small until very recently.
In the early 2000s, the “standard” dataset for object recognition was Caltech-101, which had only 30 training samples per category (and 101 categories). Convolutional nets didn’t work very well compared with more conventional methods because the dataset was so small. But we did invent several unsupervised techniques to deal with that, as well as several new operators that are now common in deep learning (such as ReLUs and contrast normalization). It’s only since 2010 with datasets like LabeLMe and ImageNet that computer vision datasets have been large enough to train large convolutional nets on natural images.
I do agree that the future is in unsupervised learning, and I say so in the interview. In fact, we got pretty good results using a combination of unsupervised and supervised learning on tasks like pedestrian detection where the dataset we had access to only had a few thousand pedestrian images in it.
I’m essentially lamenting about the fact that all the nice work we did on unsupervised learning has not really paid off so far, because supervised learning works so well with lots of labelled data and collecting more labelled data is often the most efficient way to get good performance."
Original: https://medium.com/@ShaliniAnanda1/an-open-letter-to-yann-lecun-22b244fc0a5a
Related:
Best regards,
Shalini Ananda, PhD.
Bio: Shalini Ananda is a Co-Founder of Quantified Skin, Data scientist with a focus on deep learning and computational chemistry. Lover of street music : music should be enjoyed live!
PS: Yann was kind enough to reply to this letter:
"Hi Shalini, I don’t disagree with you.
I have been confronted with many Quadrant C problems in my career. Much of our research on deep learning in the mid-2000s was actually focused on unsupervised learning precisely because most of the datasets we had were squarely in Quadrant C.
Computer vision datasets for object recognition may be large now (though not petabyte-large) but they were rather small until very recently.
In the early 2000s, the “standard” dataset for object recognition was Caltech-101, which had only 30 training samples per category (and 101 categories). Convolutional nets didn’t work very well compared with more conventional methods because the dataset was so small. But we did invent several unsupervised techniques to deal with that, as well as several new operators that are now common in deep learning (such as ReLUs and contrast normalization). It’s only since 2010 with datasets like LabeLMe and ImageNet that computer vision datasets have been large enough to train large convolutional nets on natural images.
I do agree that the future is in unsupervised learning, and I say so in the interview. In fact, we got pretty good results using a combination of unsupervised and supervised learning on tasks like pedestrian detection where the dataset we had access to only had a few thousand pedestrian images in it.
I’m essentially lamenting about the fact that all the nice work we did on unsupervised learning has not really paid off so far, because supervised learning works so well with lots of labelled data and collecting more labelled data is often the most efficient way to get good performance."
Original: https://medium.com/@ShaliniAnanda1/an-open-letter-to-yann-lecun-22b244fc0a5a
Related:
- KDnuggets Free Pass to Big Data TechCon How-To Conference, Apr 26-28, Boston
- Top stories for Mar 8-14: 7 common Machine Learning mistakes; Deep Learning for Text Understanding from Scratch
- Deep Learning for Text Understanding from Scratch