Recurrent Neural Net describes images like Taylor Swift or Romantic Novel

Deep learning has recently, and famously, taken on painting by imitating artists. We now find recurrent neural networks writing stories corresponding to images, in the style of romance novels or Taylor Swift lyrics.

By Samim.

Stories are a fundamental human tool that we use to communicate thought. Creating a stories about a image is a difficult task that many struggle with. New machine-learning experiments are enabling us to generate stories based on the content of images. This experiment explores how to generate little romantic stories about images (incl. guest star Taylor Swift).

Sumo Romance


neural-storyteller is a recently published experiment by Ryan Kiros (University of Toronto). It combines recurrent neural networks (RNN), skip-thoughts vectors and other techniques to generate little stories about images. Neural-storyteller’s outputs are creative and often comedic. It is open-source.


This experiment started by running 5000 randomly selected web-images through neural-storyteller and experimenting with hyper-parameters. neural-storyteller comes with 2 pre-trained models: One trained on 14 million passages of romance novels, the other trained on Taylor Swift Lyrics. Inputs and outputs were manually filtered and recombined into two videos.

Generating Romance

Generating Taylor Swift

How does it work?

1. Train a recurrent neural network (RNN) decoder on romance novels.

2. Each passage from a novel is mapped to a skip-thought vector.

3. Conditions RNN on skip-thought vector & generate the encoded passage.

4. Train a visual-semantic embedding between COCO images and captions. Captions and images are mapped into a common vector space.

5. After training, embed new images and retrieve captions.

A selection of generated stories

Beatles Romance

Happy Guys Romance

Shark Attack Romance

Final thoughts

neural-storyteller gives us a fascinating glimpse into the future of storytelling. Even though these technologies are not fully mature yet, the art of storytelling is bound to change. In the near future, authors will be training custom models, combining styles across genres and generating text with images & sounds. Exploring this exiting new medium is rewarding!

Bio: SamimSamim is a compassionate alien & thinker. Game, Web, Music & A.I. experimenter. Code magician and Narrative engineer. Tactical storytelling.