Recently, OpenAI announced a new successor to their language model, GPT-3, that is now the largest model trained so far with 175 billion parameters. Training a language model this large has its merits and limitations, so this article covers some of its most interesting and important aspects.
In some domains, new values appear all the time, so it's crucial to handle them in a good way. Using deep learning, one can learn a special Out-of-Vocabulary embedding for these new values. But how can you train this embedding to generalize well to any unseen value? We explain one of the methods employed at Taboola.
We explain how to implement VAE - including simple to understand tensorflow code using MNIST and a cool trick of how you can generate an image of a digit conditioned on the digit.
We describe how we handle performance issues with our deep learning models, including how to find subgraphs that take a lot of calculation time and how to extract these into a caching mechanism.
We outline why you should care about uncertainty and discuss the different types, including model, data and measurement uncertainty and what different purposes these all serve.
Algorithm Engineer at AI21 Labs, in the mission of building AI systems with the capacity to understand and generate natural language. 
