The Insiders’ Guide to Generative and Discriminative Machine Learning Models

In this article, we will look at the difference between generative and discriminative models, how they contrast, and one another.



By Saurav Singla, Data Scientist

Figure

Photo by Photos Hobby on Unsplash

 

In this article, we will look at the difference between generative and discriminative models, how they contrast, and one another.

Discriminative machine learning is to recognize the rig output among possible output choices. Given something about the data, and done by learning parameters. That maximizes the joint probability of P(X, Y).

Classification is additionally mentioned as discriminative modeling. This is often on the grounds; the model must separate instances of input variables across classes. It must pick or make a call with regards to what class a given instance belongs.

Unsupervised models summarize the distribution of input variables. Also, able to be accustomed to create or generate new instances within the input distribution. As such, these varieties of models are observed as generative models.

One variable may have a known data distribution like a Gaussian distribution.

A generative model could also be able to summarize the data distribution. This is used to generate new variables that fit into the distribution of the input variable.

A straightforward model within the generative setting would must less information. Then an intricate one within the discriminative setting, and also the other way around.

Along these lines, discriminative models outflank generative models at conditional prediction. Likewise, discriminative models should regularize more than generative models.

Figure

Photo by Tran Mau Tri Tam on Unsplash

 

For instance, there are two kids, Tony and Mark. Both visited the pet store to determine the difference between Cat and Dog. Both pay special attention to color, size, eye color, hair size, a voice, which are feature sets of the pets.

Two photos got one amongst a cat and one among a dog to Mark and asked which one is which. Mark has written down several conditions. If the voice looks like meow and eyes are blue or green.

It has stripes with color brown or black, and then the animal may be a cat.

Due to his simple rules, he detected which one is a cat and which one could be a dog.

Now instead of showing two photos to Tony, two pieces of blank paper and ask him to draw what a cat and a dog look seem like. Tony draws the drawing.

Well now, given any photo, Tony may also tell which one could be a cat. Which one may be a dog supported the drawing he created. Drawing is a time-consuming task for the task of detection, which one could be a cat.

But if there have been only some dogs and cats to seem for Tony and Mark means low training data. In such cases, if a photograph of a brown dog with stripes with blue eyes.

There is an opportunity that Mark would mark it sort of a cat. While Tony has her drawing and he can better detect that this photo is of a dog.

If Tony to listen to more things like features, it will create a better sketch. But, if more examples show employing a dataset of cat and dog, Mark would be better than Tony.

Mark is meticulous in his observations. Suppose you ask him to listen to more features. It will create more complicated rules called overfitting. Thus the chance of finding a cat and a dog will increase, but that will not happen with Tony.

What if before visiting the pet store, they are not informed. There are only two kinds of an animal that means no labeled data.

Mark would fail completely because he won’t know what to look for while Tony would be able to draw the sketch anyway. This is a huge advantage, sometimes called semi-supervised.

This shows Mark is for discriminative, and Tony is for Generative.

Figure

Photo by Morning Brew on Unsplash

 

Another example, classify a speech to a language model.

Discriminative approach determining the difference within the linguistic models. Without learning the languages and so classifying the speech.

The generative approach means learning each language. So classifying it using the knowledge you gained.

Figure

Photo by Antoine Dautry on Unsplash

 

In maths

Discriminative machine learning is actually training a model. To tell apart the right output among possible output choices. This is done by learning model parameters that maximize the conditional probability P(Y|X).

Generative machine learning is training a model to learn parameters maximizing the joint probability of P(X, Y).

Often learned in probabilistic models in its factorized form P(Y), P(X|Y) is simplified in most scenarios. With assumptions of conditional independence P(Y), P(X|Y).

Figure

Photo by Markus Spiske on Unsplash

 

Discriminative models

  • Logistic regression
  • Random forests
  • Support vector machine (SVM)
  • Traditional neural networks
  • Nearest neighbor

Generative Models

  • Hidden Markov model (HMM)
  • Naïve Bayes
  • Bayesian Networks
  • Gaussian mixture model (GMM)
Figure

Photo by Jon Tyson on Unsplash

 

One should ask the following questions before carrying them out.

  • Which model needs less data for training?
  • Which model can generate data?
  • When should one use such kind of models?
  • Which model is more sensitive to extreme values?
  • Which model is more prone to overfitting?
  • Which model can train in less time?
  • Which model learns conditional probability?
  • Which model is better in the case of uncertainty?
  • Which model is better when the features have a relationship?
  • Which model is better when an explanatory model needed?
  • Which model is better when optimizing the classification accuracy needed?
  • Which model is better when labeled data is not available?
  • Which model is better when labeled data is available?
  • Which model is easy and fast to carry out?
Figure

Photo by Photos Hobby on Unsplash

 

In deep learning

Generative adversarial networks (GAN), the generator and discriminator trained together. The generator generates a batch of samples — these, together with the real dataset, given to the discriminator for classification.

Figure

Photo by Nathan Dumlao on Unsplash

 

Disadvantages of Discriminative Classifiers

It lacks the elegance of generative like priors, structure, uncertainty. Feel like black-boxes, relationships between variables don’t seem to be explicit and visualizable.

Figure

Photo by Austin Distel on Unsplash

 

Conclusion

Generative and Discriminative methods are two-broad approaches. The generative involves modeling and discriminative solve classification. The generative models are more elegant, have explanatory power.

The richness of a model isn’t always an upside. Fitting more parameters takes longer, space, and computation. Discriminative models have to be regularized more than generative models.

An easy model within the generative setting would want less data than a posh one within the discriminative setting.

Reference

  • Yogatama, D., Dyer, C., Ling, W., and Blunsom, P., 2017. Generative and discriminative text classification with recurrent neural networks. arXiv preprint arXiv:1703.01898.
  • Lasserre, J.A., Bishop, C.M. and Minka, T.P., 2006, June. Principled hybrids of generative and discriminative models. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) (Vol. 1, pp. 87–94). IEEE.
  • Wang, K., Zong, C., and Su, K.Y., 2012. Integrating generative and discriminative character-based models for Chinese word segmentation. ACM Transactions on Asian Language Information Processing (TALIP), 11(2), pp.1–41.
  • Prasad, A., Niculescu-Mizil, A., and Ravikumar, P.K., 2017. On separability of loss functions, and revisiting discriminative vs. generative models. In Advances in Neural Information Processing Systems (pp. 7050–7059).

 
Bio: Saurav Singla is an accomplished and high performing analytical professional with 15 years of deep expertise in the application of analytics, business intelligence, data mining and statistics in multiple industries and 3 years of consulting experience and 5 years of managing a team in the data science field. He is a creative problem solver with a unique mix of technical, business and research proficiency that lends itself to developing key strategies and solutions with a significant impact on revenue and ROI. He possesses working experience in machine learning, statistics, natural language processing and deep learning with extensive use of Python, R, SQL & Tableau.

Original. Reposted with permission.

Related: