Python, Selenium & Google for Geocoding Automation: Free and Paid
This tutorial will take you through two options that have automated the geocoding process for the user using Python, Selenium and Google Geocoding API.
Geocoding is the beginning of Spatial Analysis. Whether geocoding a place or residence, an accurate geocoder is the first step to make sure your analysis is on the right track. This tutorial will take you through two options that have automated the geocoding process for the user using Python, Selenium and Google Geocoding API. Automation is something every Data Scientist needs to learn. But automating workflows is unique to every person and not everyone will have the same workflow. Automation is something to strive for when we believe that repeating the same process is too time consuming and your time is better spent doing something else while your automated process will take care of everything.
Free Geocoding with Python and Selenium
The first tutorial was written to automate my geocoding process when a paid service was not an option. From my personal observation, my script has given me a minimum 70% accuracy given the following standards are met:
- The address must not contain the name of a place.
- Minimal spelling errors
- The country is also part of the address
This script assumes you are:
- Using xlxs file extension (Microsoft Excel Workbook)
- The whole address is in one column.
- Using Google Chrome
Note: When using selenium, make sure that you download and keep the Google Chrome Diriver File in the same directory as your script and that the version is compatible with your own Google Chrome Browser.
The final output will be an excel file with the geocoded address and coordinates in separate columns.
Once you have updated
link_col and the name of the output excel file, you can begin running the script.
The above script once run will open a google chrome browser and go to https://www.mapdevelopers.com/geocode_tool.php where you will see the your geocoding taking place. If not for automation, the usual scenario will be keeping a excel and google screen open and a repetitive copy and paste motion. A person with thousands of rows will just be wasting time.
There are many ways to automate and improve on geocoding and this is a method I personally developed for when I do not feel like paying for an API and I am not time constrained. This brings me to some of the drawbacks of using a free geocoding method versus an API:
- It is very slow in comparison but more efficient and fast than manually geocoding
- Accuracy is rarely and almost never beaten
Google Geocoding with Python
Google used to allow a certain amount (about 2,500 API calls) for its geocoding per day for free. This is an annoying limitation but one that allowed for highly accurate geocoding even if your address had some spelling mistakes and even better if you had a place name. However recently google now charges for every API call.
If not mistaken, currently the damage is $5 for every 1,000 API calls and $4 for every API call above 100,000. To put it into perspective, for the first 100,000, you will be spending $0.005 for every API call. Anything above 500,000 may need a call to the their sales team. See here for their Geocoding API billing. Every new sign up makes you eligible for $200 credit.
In order to start using Google Geocoder API, you have to obtain an API key: Google Developers API Key
Once you obtain your API Key for their geocoding service, you can start using their service. The address used can include place names as well and this should even improve the results most of the time. If it can be found on google maps when you manually check, it can be used with google's geocoder to extract the coordinates.
The only parts you need to add are
df, name of the column with the full address and the name of the output file at the end.
The above script will return an excel file with three new columns: 'Latitude', 'Longitude', 'Google Address'. It will be much faster than any custom script built. Google is undoubtedly the most preferred geocoding API as it is reliable for a client side project. In a business setting, if a client wishes to have the coordinates of their customers residence mapped, visualized and analyzed, explaining the use of google to geocode will be simple and understandable to them.
This method uses much less lines of code and is easier to maintain. With no restrictions and a relatively cheap price per API call, it can be feasible to a certain extent.
- Visualising Geospatial data with Python using Folium
- How to build a data science project from scratch
- OpenStreetMap Data to ML Training Labels for Object Detection