2020: A Year Full of Amazing AI Papers — A Review

So much happened in the world during 2020 that it may have been easy to miss the great progress in the world of AI. To catch you up quickly, check out this curated list of the latest breakthroughs in AI by release date, along with a video explanation, link to an in-depth article, and code.



By Louis (What's AI) Bouchard, Montrealer, explaining AI stuff on YouTube and Medium.

Photo by Kelly Sikkema on Unsplash.

Even with everything that happened in the world this year, we still had the chance to see a lot of amazing research come out, especially in the field of artificial intelligence. More, many important aspects were highlighted this year, like the ethical aspects, important biases, and much more. Artificial intelligence and our understanding of the human brain and its link to AI is constantly evolving, showing promising applications in the near future.

Here are the most interesting research papers of the year, in case you missed any of them. In short, it is basically a curated list of the latest breakthroughs in AI and Data Science by release date with a clear video explanationlink to a more in-depth article, and code (if applicable).

The complete reference to each paper is listed at the end of this article, and you can access the complete list in a GitHub repository.

Watch a complete 2020 rewind in 15 minutes:

 

1. YOLOv4: Optimal Speed and Accuracy of Object Detection

 

This 4th version has been recently introduced in April 2020 by Alexey Bochkovsky et al. in the paper “YOLOv4: Optimal Speed and Accuracy of Object Detection”. The main goal of this algorithm was to make a super-fast object detector with high quality in terms of accuracy.

 

2. DeepFaceDrawing: Deep Generation of Face Images from Sketches

 

You can now generate high-quality face images from rough or even incomplete sketches with zero drawing skills using this new image-to-image translation technique! If your drawing skills as bad as mine you can even adjust how much the eyes, mouth, and nose will affect the final image! Let’s see if it really works and how they did it.

 

3. Learning to Simulate Dynamic Environments with GameGAN

 

 

4. PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

 

This new algorithm transforms a blurry image into a high-resolution image! It can take a super low-resolution 16x16 image and turn it into a 1080p high definition human face! You don’t believe me? Then you can do just like me and try it on yourself in less than a minute! But first, let’s see how they did that.

 

5. Unsupervised Translation of Programming Languages

 

This new model converts code from a programming language to another without any supervision! It can take a Python function and translate it into a C++ function, and vice-versa, without any prior examples! It understands the syntax of each language and can thus generalize to any programming language! Let’s see how they did that.

 

6. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

 

This AI Generates 3D high-resolution reconstructions of people from 2D images! It only needs a single image of you to generate a 3D avatar that looks just like you, even from the back!

 

7. High-Resolution Neural Face Swapping for Visual Effects

 

Researchers at Disney developed a new High-Resolution Face Swapping algorithm for Visual Effects in the paper of the same name. It is capable of rendering photo-realistic results at megapixel resolution. Working for Disney, they are most certainly the best team for this work. Their goal is to swap the face of a target actor from a source actor while maintaining the actor’s performance. This is incredibly challenging and is useful in many circumstances, such as changing the age of a character, when an actor is not available, or even when it involves a stunt scene that would be too dangerous for the main actor to perform. The current approaches require a lot of frame-by-frame animation and post-processing by professionals.

Disney’s New High-Resolution Face Swapping Algorithm | New 2020 Face Swap Technology Explained

 

8. Swapping Autoencoder for Deep Image Manipulation

 

This new technique can change the texture of any picture while staying realistic using complete unsupervised training! The results look even better than what GANs can achieve while being way faster! It could even be used to create deepfakes!

 

9. GPT-3: Language Models are Few-Shot Learners

 

The current state-of-the-art NLP systems struggle to generalize to work on different tasks. They need to be fine-tuned on datasets of thousands of examples, while humans only need to see a few examples to perform a new language task. This was the goal behind GPT-3 to improve the task-agnostic characteristic of language models.

 

10. Learning Joint Spatial-Temporal Transformations for Video Inpainting

 

This AI can fill the missing pixels behind a removed moving object and reconstruct the whole video with way more accuracy and less blurriness than current state-of-the-art approaches!

 

11. Image GPT — Generative Pretraining from Pixels

 

A good AI, like the one used in Gmail, can generate coherent text and finish your phrase. This one uses the same principles in order to complete an image! All done in an unsupervised training with no labels required at all!

 

12. Learning to Cartoonize Using White-box Cartoon Representations

 

This AI can cartoonize any picture or video you feed it in the cartoon style you want! Let’s see how it does that and some amazing examples. You can even try it yourself on the website they created as I did for myself!

 

13. FreezeG: Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs

 

This face generating model is able to transfer normal face photographs into distinctive styles such as Lee Mal-Nyeon’s cartoon style, the Simpsons, arts, and even dogs! The best thing about this new technique is that it’s super simple and significantly outperforms previous techniques used in GANs.

 

14. Neural Re-Rendering of Humans from a Single Image

 

The algorithm represents body pose and shape as a parametric mesh, which can be reconstructed from a single image and easily reposed. Given an image of a person, they are able to create synthetic images of the person in different poses or with different clothing obtained from another input image.

 

15. I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image

 

Their goal was to propose a new technique for 3D Human Pose and Mesh Estimation from a single RGB image. They called it I2L-MeshNet, where I2L stands for Image-to-Lixel. Just like a voxel, volume + pixel, is a quantized cell in three-dimensional space, they defined lixel, a line, and pixel, as a quantized cell in one-dimensional space. Their method outperforms previous methods, and the code is publicly available!

 

16. Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

 

Language-guided navigation is a widely studied field and a very complex one. Indeed, it may seem simple for a human to just walk through a house to get to your coffee that you left on your nightstand to the left of your bed. But it is a whole other story for an agent, which is an autonomous AI-driven system using deep learning to perform tasks.

 

17. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

 

ECCV 2020 Best Paper Award Goes to Princeton Team. They developed a new end-to-end trainable model for optical flow. Their method beats state-of-the-art architectures’ accuracy across multiple datasets and is way more efficient. They even made the code available for everyone on their Github!

 

18. Crowdsampling the Plenoptic Function

 

Using tourists’ public photos from the internet, they were able to reconstruct multiple viewpoints of a scene, conserving the realistic shadows and lighting! This is a huge advancement of the state-of-the-art techniques for photorealistic scene rendering, and their results are simply amazing.

 

19. Old Photo Restoration via Deep Latent Space Translation

 

Imagine having the old, folded, and even torn pictures of your grandmother when she was 18 years old in high definition with zero artifacts. This is called old photo restoration, and this paper just opened a whole new avenue to address this problem using a deep learning approach.

 

20. Neural circuit policies enabling auditable autonomy

 

Researchers from IST Austria and MIT have successfully trained a self-driving car using a new artificial intelligence system based on the brains of tiny animals, such as threadworms. They achieved that with only a few neurons able to control the self-driving car, compared to the millions of neurons needed by the popular deep neural networks such as Inceptions, Resnets, or VGG. Their network was able to completely control a car using only 75 000 parameters, composed of 19 control neurons, rather than millions!

 

21. Lifespan Age Transformation Synthesis

 

A team of researchers from Adobe Research developed a new technique for age transformation synthesis based on only one picture from the person. It can generate the lifespan pictures from any picture you sent it.

 

22. DeOldify

 

DeOldify is a technique to colorize and restore old black and white images or even film footage. It was developed and is still getting updated by only one person Jason Antic. It is now the state of the art way to colorize black and white images, and everything is open-sourced, but we will get back to this in a bit.

 

23. COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

 

As the name states, it uses transformers to generate accurate text descriptions for each sequence of a video, using both the video and a general description of it as inputs.

 

24. Stylized Neural Painting

 

This Image-to-Painting Translation method simulates a real painter on multiple styles using a novel approach that does not involve any GAN architecture, unlike all the current state-of-the-art approaches!

 

25. Is a Green Screen Really Necessary for Real-Time Portrait Matting?

 

Human matting is an extremely interesting task where the goal is to find any human in a picture and remove the background from it. It is really hard to achieve due to the complexity of the task, having to find the person or people with the perfect contour. In this post, I review the best techniques used over the years and a novel approach published on November 29th, 2020. Many techniques are using basic computer vision algorithms to achieve this task, such as the GrabCut algorithm, which is extremely fast but not very precise.

 

26. ADA: Training Generative Adversarial Networks with Limited Data

 

With this new training method developed by NVIDIA, you can train a powerful generative model with one-tenth of the images! Making possible many applications that do not have access to so many images!

 

27. Improving Data‐Driven Global Weather Prediction Using Deep Convolutional Neural Networks on a Cubed Sphere

 

The current traditional approach for weather forecasting uses what we call “Numerical weather prediction” models. It uses mathematical models of the atmosphere and oceans to predict the weather based on the current conditions. It was first introduced in the 1920s and produced realistic results in the 1950s using computer simulations. These mathematical models work for predicting both short and long-term forecasts. But it’s heavy in computation and cannot base its predictions on as much data as a deep neural network. This is partly why it is so promising. These current numerical weather prediction models already use machine learning to improve the forecasts as a post-processing tool. Weather forecasting is receiving more and more attention from machine learning researchers, already yielding promising results.

 

28. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

 

This new method is able to generate a complete 3-dimensional scene and has the ability to decide the lighting of the scene. All this with very limited computation costs and amazing results compared to previous approaches.

 

Conclusion

 

As you can see, this was an extremely insightful year for artificial intelligence, and I am super excited to see what’s going to happen in 2021! I will be sure to cover the most exciting and interesting papers, and I would love it if you could take part in this adventure! If you like my work and want to stay up-to-date with AI technologies, you should definitely follow me on my social media channels.

 

References

 

[1] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, Yolov4: Optimal speed and accuracy of object detection, 2020. arXiv:2004.10934 [cs.CV].

[2] S.-Y. Chen, W. Su, L. Gao, S. Xia, and H. Fu, “DeepFaceDrawing: Deep generation of face images from sketches,” ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH2020), vol. 39, no. 4, 72:1–72:16, 2020.

[3] S. W. Kim, Y. Zhou, J. Philion, A. Torralba, and S. Fidler, “Learning to Simulate DynamicEnvironments with GameGAN,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2020.

[4] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin, Pulse: Self-supervised photo upsampling via latent space exploration of generative models, 2020. arXiv:2003.03808 [cs.CV].

[5] M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample, Unsupervised translation of programming languages, 2020. arXiv:2006.03511 [cs.CL].

[6] S. Saito, T. Simon, J. Saragih, and H. Joo, Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization, 2020. arXiv:2004.00452 [cs.CV].

[7] J. Naruniec, L. Helminger, C. Schroers, and R. Weber, “High-resolution neural face-swapping for visual effects,” Computer Graphics Forum, vol. 39, pp. 173–184, Jul. 2020.doi:10.1111/cgf.14062.

[8] T. Park, J.-Y. Zhu, O. Wang, J. Lu, E. Shechtman, A. A. Efros, and R. Zhang,Swappingautoencoder for deep image manipulation, 2020. arXiv:2007.00653 [cs.CV].

[9] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P.Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S.Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei,“Language models are few-shot learners,” 2020. arXiv:2005.14165 [cs.CL].

[10] Y. Zeng, J. Fu, and H. Chao, Learning joint spatial-temporal transformations for video in-painting, 2020. arXiv:2007.10247 [cs.CV].

[11] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, “Generative pretraining from pixels,” in Proceedings of the 37th International Conference on Machine Learning, H. D. III and A. Singh, Eds., ser. Proceedings of Machine Learning Research, vol. 119, Virtual: PMLR, 13–18 Jul 2020, pp. 1691–1703. [Online]. Available:http://proceedings.mlr.press/v119/chen20s.html.

[12] Xinrui Wang and Jinze Yu, “Learning to Cartoonize Using White-box Cartoon Representations.”, IEEE Conference on Computer Vision and Pattern Recognition, June 2020.

[13] S. Mo, M. Cho, and J. Shin, Freeze the discriminator: A simple baseline for fine-tuning gans,2020. arXiv:2002.10964 [cs.CV].

[14] K. Sarkar, D. Mehta, W. Xu, V. Golyanik, and C. Theobalt, “Neural re-rendering of humans from a single image,” in European Conference on Computer Vision (ECCV), 2020.

[15] G. Moon and K. M. Lee, “I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image,” in European Conference on ComputerVision (ECCV), 2020

[16] J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environments,” 2020. arXiv:2004.02857 [cs.CV].

[17] Z. Teed and J. Deng, Raft: Recurrent all-pairs field transforms for optical flow, 2020. arXiv:2003.12039 [cs.CV].

[18] Z. Li, W. Xian, A. Davis, and N. Snavely, “Crowdsampling the plenoptic function,” inProc.European Conference on Computer Vision (ECCV), 2020.

[19] Z. Wan, B. Zhang, D. Chen, P. Zhang, D. Chen, J. Liao, and F. Wen, Old photo restoration via deep latent space translation, 2020. arXiv:2009.07047 [cs.CV].

[20] Lechner, M., Hasani, R., Amini, A. et al. Neural circuit policies enabling auditable autonomy. Nat Mach Intell 2, 642–652 (2020). https://doi.org/10.1038/s42256-020-00237-3

[21] R. Or-El, S. Sengupta, O. Fried, E. Shechtman, and I. Kemelmacher-Shlizerman, “Lifespanage transformation synthesis,” in Proceedings of the European Conference on Computer Vision(ECCV), 2020.

[22] Jason Antic, Creator of DeOldify, https://github.com/jantic/DeOldify

[23] S. Ging, M. Zolfaghari, H. Pirsiavash, and T. Brox, “Coot: Cooperative hierarchical trans-former for video-text representation learning,” in Conference on Neural Information ProcessingSystems, 2020.

[24] Z. Zou, T. Shi, S. Qiu, Y. Yuan, and Z. Shi, Stylized neural painting, 2020. arXiv:2011.08114[cs.CV].

[25] Z. Ke, K. Li, Y. Zhou, Q. Wu, X. Mao, Q. Yan, and R. W. Lau, “Is a green screen really necessary for real-time portrait matting?” ArXiv, vol. abs/2011.11961, 2020.

[26] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, Training generative adversarial networks with limited data, 2020. arXiv:2006.06676 [cs.CV].

[27] J. A. Weyn, D. R. Durran, and R. Caruana, “Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere”, Journal of Advances in Modeling Earth Systems, vol. 12, no. 9, Sep. 2020, issn: 1942–2466.doi:10.1029/2020ms002109

[28] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron, “Nerv: Neural reflectance and visibility fields for relighting and view synthesis,” in arXiv, 2020.

 

Original. Reposted with permission.

 

Related: