Add A New Dimension To Your Photos Using Python

By Dylan Roy, Engineer & Side Project Junkie

Image by Author

Some Backstory

 
After looking through some photos from a recent camping trip that I had taken a couple of weeks ago. I decided to take some of my old vacation photos, and turn them into a slideshow with some Python. Don’t worry I undertook this endeavor with the understanding that Google Photos, and the Apple equivalent do this really well already. This was really just intended to feed my addiction to building out side projects.

When looking through the available options I stumbled across a paper with some code that would take your photos, and add a transition that applies a 3D Ken Burns Effect to your photos. The Ken Burns Effect gets the 3D aspect for free with video obviously, but is less compelling with photos due to that missing dimension. So in the following write up I am hoping to set you up to be able to try, and add this extra dimension to your photos as I was able to.

Concepts

 
Here are some concepts that are worth knowing due to their relevance in what we are looking to achieve. Though we don’t need to be aware of this the AI models are applying the 3D dimension to our Ken Burns effect by adding perspective through correctly scaling the size of the objects in the photo through estimating their depth. As the effect moves through the photo the model is required to reconstruct the missing objects, and textures that don’t appear in the 2D image through inpainting.

Ken Burns Effect — A zooming and panning across photographs gives the feeling of motion. It’s named after an American documentarian who often used this effect. The writers of this paper have added the third dimension to this that would normally be missing from photos.

CC BY-SA 3.0, via Wikimedia Commons

Depth Estimation — Describes the method for obtaining a measure of the distance of the objects in a scene. Doing this effectively will result in a convincing transition that will scale each object with the parallax.

Inpainting — This describes the method for reconstructing the part of the image that doesn’t exist due to the changing perspective of this transition. In images, and videos this technique also refers to repairing damaged, deteriorating, or missing parts of an artwork are filled in to present a complete image.

DIY

 
Luckily the authors of this paper, Simon Niklaus, Long Mai, Jimei Yang, Feng Liu, have provided us with a codebase that allows us to apply this effect to our photos with some pre-trained models. I have noticed that they don’t work well for photos with lots of sharp edges in the foreground, but yield pretty impressive results on a number of the photos that I have tested it with.

I won’t get into the details into how this is achieved as the authors do this in their paper, Github Repo, and Youtube video which all have been provided at the end in the resources section.

I am going to step you through the approach that we take to create these 3D scenes, but if you want to just blindly try it yourself at the end of this section I added all of the code that we will be going through to a Google Colab that you can copy, and use immediately.

Before stepping through this yourself I suggest opening a new Google Colab, because in the code samples I go through I use some Colab specific functionality to help with ease of use.

Setup Environment and Install Requirements

 
The first step to get this working yourself is to import their repository that contains this implementation, and pre-tained models. So what happens below is we clone the repo, navigate to it, make our output directory, and install the required libraries. The Google Colab has a lot of the requirements already preloaded when you start a new runtime so there was just two that were missing.

!git clone https://github.com/sniklaus/3d-ken-burns.git# Move into the downloaded repository
%cd 3d-ken-burns# Make a directory for the videos
!mkdir videos
video_dir = "./videos/"# Install dependencies
!pip install moviepy gevent

Upload Photos

 
At this stage all that you need to do is have an easy way to upload photos to your source/image directory. You don’t really need to include this in your notebook to move forward, as you could just drag your files into their corresponding folders with the Colab interface, but it’s a pretty nice connivence cell.

from google.colab import filesuploads = files.upload()for filename in uploads.keys():
    !mv ./$filename ./images/$filename

Convert Your Photos

 
So if we didn’t want to convert all of our photos that we provide to the two directories, and easy one liner would be as follows which just uses the autozoom.py interface to convert one photo.

python autozoom.py --in {input_image}.jpg --out {output_video}.mp4

Though since we want to automate what we can, we can take advantage of bash, and iterate through the directory then execute on each photo.

!for image in ./images/*; do python autozoom.py --in $image --out ./videos/$(basename $image | cut -f1 -d '.').mp4; done

View Your Video

 
Once your photos are converted you may want to view the output first we use IPython’s functionality to build the function to view the video, and then provide a selector for us to choose which video to view amongst those in the directory.

import os
from base64 import b64encodefrom IPython.display import HTML
import ipywidgets as widgetsdef video(path):
    mp4 = open(path,'rb').read()
    data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
    return HTML(f'<video width=600 controls loop> <source src="{data_url}" type="video/mp4"></video>')files_list = os.listdir(video_dir)
video_list = widgets.Dropdown(
    options=files_list,
    value="" if not files_list else files_list[0],
    description='Video:',
    disabled=False,
)
display(video_list)

Convert Mp4 to Animated GIF

 
I added this step as a result of wanting to upload, and embed the output of this applied effect in this post, so as with all of my efforts, please take advantage of this knowledge, and convert your mp4’s to animated gif’s as you please. As a bonus we have already imported imageio as a dependency so there’s no need to install it. Below we use this function, and iterate through the destination directory looking for mp4’s to convert.

import imageio
import os, sysclass TargetFormat(object):
    GIF = ".gif"
    MP4 = ".mp4"
    AVI = ".avi"def convertFile(inputpath, targetFormat):
    """Reference: https://gist.github.com/michaelosthege/cd3e0c3c556b70a79deba6855deb2cc8"""
    outputpath = os.path.splitext(inputpath)[0] + targetFormat
    print("converting\r\n\t{0}\r\nto\r\n\t{1}".format(inputpath, outputpath))reader = imageio.get_reader(inputpath)
    fps = reader.get_meta_data()['fps']writer = imageio.get_writer(outputpath, fps=fps)
    for i,im in enumerate(reader):
        sys.stdout.write("\rframe {0}".format(i))
        sys.stdout.flush()
        writer.append_data(im)
    print("\r\nFinalizing...")
    writer.close()
    print("Done.")for file in [x for x in os.listdir(video_dir) if x.endswith(".mp4")]:
    convertFile(f"{video_dir}{file}", TargetFormat.GIF)

Download All Videos

 
If you don’t want to use the file explorer to download your output. The last step is to iterate through the destination folder, and download each video. This could also be done as we did previously when using bash to iterate, and apply the effect.

for file in os.listdir(video_dir):
    files.download(f"{video_dir}{file}")

If all has gone well then your photos have been properly converted as one of mine I took here when enjoying Bolonga when times we different.

Image by Author

For those that want to skip putting a Colab notebook together themselves here is the complete implementation for you to make a copy of, and start converting your photo albums.

Key Takeaways

 
Standing on the shoulders of giants we live in an age where we can take advantage of cutting edge AI techniques to help us do things that normally would require someone with specific knowledge. In this instance we can use the AI techniques developed by Simon Niklaus and team instead of needing to learn sophisticated editing skills.

Other Posts By Dylan

 
Create Beautiful Architecture Diagrams with Python
Stop spending time manually tweaking misaligned arrows

 
Deploy To Google Cloud Run Using Github Actions
A CI/CD Solution That Scales To Zero With Actions And Cloud Run

 
Auto-Updating Your Github Profile With Python
Showcase Your Skills Through Automating Your Readme Profile

Resources

• Video Description
• Research Paper
• Github Repo
• Gist for MP4 To GIF Conversion
• Other Google Colabs Leveraging Paper from Arnaldo Gabriel, one from Vlad Alex, and one from Ahmed Harmouche.

 
Bio: Dylan Roy is an Engineer & Side Project Junkie sharing what he is working on so readers like you can benefit from his experiences. Subscribe here for even more (dylanroy.com)

Original. Reposted with permission.

Related:

• Extraction of Objects In Images and Videos Using 5 Lines of Code
• 5 Tasks To Automate With Python
• Which Face is Real? Applying StyleGAN to Create Fake People