Medical Image Analysis with Deep Learning
In this article, I start with basics of image processing, basics of medical image format data and visualize some medical data.
By Taposh Roy, Kaiser Permanente.
Analyzing images and videos, and using them in various applications such as self driven cars, drones etc. with underlying deep learning techniques has been the new research frontier. The recent research papers such as “A Neural Algorithm of Artistic Style”, show how a styles can be transferred from an artist and applied to an image, to create a new image. Other papers such as “Generative Adversarial Networks” (GAN) and “Wasserstein GAN” have paved the path to develop models that can learn to create data that is similar to data that we give them. Thus opening up the world to semi-supervised learning and paving the path to a future of unsupervised learning.
While these research areas are still on the generic images, our goal is to use these research into medical images to help healthcare. We need to start with some basics. In this article, I start with basics of image processing, basics of medical image format data and visualize some medical data. In the next article I will deep dive into some convolutional neural nets and use them with Keras for predicting lung cancer.
Basic Image Processing (using python)
There are a variety of image processing libraries, however OpenCV (open computer vision) has become mainstream due to its large community support and availability in C++, java and python. I prefer using opencv using jupyter notebook.
Install OpenCV using: pip install opencv-python or install directly from the source from opencv.org
Installing opencv.
Now open your Jupyter notebook and confirm you can import cv2. You will also need numpy and matplotlib to view your plots inside the notebook.
Now, lets check if you can open an image and view it on your notebook using the code below.
Example image load through OpenCV.
Basic Face Detection
Lets, do something fun such as detecting a face. To detect face we will use an open source xml stump-based 20x20 gentle adaboost frontal face detector originally created by Rainer Lienhart. A good post with details on Haar-cascade detection is here.
Face detection using OpenCV.
There are a lot of examples for image processing using opencv in the docs section. http://docs.opencv.org/trunk/d6/d00/tutorial_py_root.html. I leave it up to the reader to play with more examples. Now that we know the basics of image processing, lets move to the next level of understanding medical image format.
Medical Image Data Format
Medical images follow Digital Imaging and Communications (DICOM) as a standard solution for storing and exchanging medical image-data. The first version of this standard was released in 1985. Since then there are several changes made. This standard uses a file format and a communications protocol.
- File Format — All patient medical images are saved in the DICOM file format. This format has PHI (protected health information) about the patient such as — name, sex, age in addition to other image related data such as equipment used to capture the image and some context to the medical treatment. Medical Imaging Equipments create DICOM files. Doctors use DICOM Viewers, computer software applications that can display DICOM images, read and to diagnose the findings in the images.
- Communications Protocol — The DICOM communication protocol is used to search for imaging studies in the archive and restore imaging studies to the workstation in order to display it. All medical imaging applications that are connected to the hospital network use the DICOM protocol to exchange information, mainly DICOM images but also patient and procedure information. There are also more advanced network commands that are used to control and follow the treatment, schedule procedures, report statuses and share the workload between doctors and imaging devices.
A very good blog that goes into details of the DICOM standard is here
Analyze DICOM Images
A very good python package used for analyzing DICOM images is pydicom. In this section, we will see how to render a DICOM image on a Jupyter notebook.
Install OpenCV using: pip install pydicom
After you install pydicom package, go back to the jupyter notebook. In the notebook, import the dicom package and other packages as shown below.
We also use other packages such as pandas, scipy, skimage, mpl_toolkit for data processing and analysis.
There’s a wealth of freely available DICOM datasets online but here’s a few that should help you get started:
- Kaggle Competitions and Datasets: This is my personal favorite. Check out the data for lung cancer competition and diabetes retinopathy.
- Dicom Library : DICOM Library is a free online medical DICOM image or video file sharing service for educational and scientific purposes.
- Osirix Datasets: Provides a large range of human datasets acquired through a variety of imaging modalities.
- Visible Human Datasets: Parts of the Visible Human project are somehow freely distributed here which is weird cause getting that data is neither free nor hassle-free.
- The Zubal Phantom: This website offers multiple datasets of two human males in CT and MRI which are freely distributed.
Download the dicom files and load them on your jupyter notebook.
Now, load the DICOM images into a list.
Step 1 : Basic Viewing of DICOM Image in Jupyter
In the first line we load the 1st DICOM file, which we’re gonna use as a reference named RefDs, to extract metadata and whose filename is first in the lstFilesDCM list.
We then calculate the total dimensions of the 3D NumPy array which are equal to (Number of pixel rows in a slice) x (Number of pixel columns in a slice) x (Number of slices) along the x, y, and z cartesian axes. Lastly, we use the PixelSpacing and SliceThickness attributes to calculate the spacing between pixels in the three axes. We store the array dimensions in ConstPixelDims and the spacing in ConstPixelSpacing [1].
Step 2: Looking into details of DICOM format
The unit of measurement in CT scans is the Hounsfield Unit (HU), which is a measure of radiodensity. CT scanners are carefully calibrated to accurately measure this. A detailed understanding on this can be found here.
Each pixel is assigned a numerical value (CT number), which is the average of all the attenuation values contained within the corresponding voxel. This number is compared to the attenuation value of water and displayed on a scale of arbitrary units named Hounsfield units (HU) after Sir Godfrey Hounsfield.
This scale assigns water as an attenuation value (HU) of zero. The range of CT numbers is 2000 HU wide although some modern scanners have a greater range of HU up to 4000. Each number represents a shade of grey with +1000 (white) and –1000 (black) at either end of the spectrum.
Hounsfield Scale [credits: “Introduction to CT physics” (PDF). elsevierhealth.com.]
Some scanners have cylindrical scanning bounds, but the output image is square. The pixels that fall outside of these bounds get the fixed value -2000.
CT Scanner Image [credits : “Introduction to CT physics” (PDF). elsevierhealth.com.]
The first step usually is setting these values to 0. Next, let’s go back to HU units, by multiplying with the rescale slope and adding the intercept (which are conveniently stored in the metadata of the scans!).
In the next part, we will use Kaggle’s lung cancer data-set and Convolution Neural Nets using Keras. We will build upon the information provided by this article to go to the next one.
Acknowledgements:
- https://pyscience.wordpress.com/2014/09/08/dicom-in-python-importing-medical-image-data-into-numpy-with-pydicom-and-vtk/
- http://www.osirix-viewer.com/resources/dicom-image-library/
- http://wearables.cc.gatech.edu/paper_of_week/viola01rapid.pdf
- http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/
- http://dicomiseasy.blogspot.com/
- https://www.kaggle.com/c/data-science-bowl-2017
- http://docs.opencv.org/trunk/d6/d00/tutorial_py_root.html
- Kaggle community for all the different scripts and support
Bio: Taposh Roy leads innovation team in Kaiser Permanente's Decision Support group. He works with research, technology and business leaders to derive insights from data.
Original. Reposted with permission.
Related:
