Top 12 Essential Command Line Tools for Data Scientists
This post is a short introductory overview of 12 Unix-like operating system command line tools of value to data science tasks, and the data scientists who perform them.
This post is a short overview of a dozen Unix-like operating system command line tools which can be useful for data science tasks. The list does not include any general file management commands (
rm, ...) or remote session management tools (
ssh, ...), but is instead made up of utilities which would be useful from a data science perspective, generally those related to varying degrees of data inspection and processing. They are all included within a typical Unix-like operating system as well.
It is admittedly elementary, but I encourage you to seek out additional command examples where appropriate. Tool names link to Wikipedia entries as opposed to man pages, as the former are generally more friendly to newcomers, in my view.
wget is a file retrieval utility, used for downloading files from remote locations.
In its most basic form,
wget is used as follows to download a remote file:
cat is a tool for outputting file contents to the standard output. The name comes from concatenate.
More complex use cases include combining files together (actual concatenation), appending file(s) to another, numbering file lines, and more.
wc command is used for producing word counts, line counts, byte counts, and related from text files. The default output for
wc, when run without options, is a single line consisting of, left to right, line count, word count (note that the single string without breaks on each line are counted as a single word), character count, and filename(s).
head outputs the first n lines of a file (10, by default) to standard output. The number of lines displayed can be set with the
Any guesses as to what
Working that command line sorcery.
find is a utility for searching the file system for particular files.
The following searches the tree structure starting in the current directory (".") for any file starting with "iris" and ending in any dumber of characters ("-name 'iris*'") of regular file type ("-type f"):
cut is used for slicing out sections of a line of text from a file. While these slices can be made using a variety of criteria,
cut can be useful for extracting columnar data from CSV files.
This outputs the fifth column ("-f 5") of the iris.csv file using the comma as field delimiter ("-d ','"):
uniq modifies the output of text files to standard output by collapsing identical consecutive lines into a single copy. On its own, this may not seem too terribly interesting, but when used to build pipelines at the command line (piping the output of one command into the input of another, and so on), this can become useful.
The following gives us a unique count of the iris dataset class names held in the fifth column, along with their counts:
What the cow say.
awk isn't actually a "command," but is instead a full programming language. It is meant for processing and extracting text, and can be invoked from the command line in single line command form.
awk would take some time, but until then here is a sample of what it can accomplish. Considering that our sample file -- iris.csv -- is rather limited (especially when it comes to diversity of text), this line will invoke
awk, search for the string "setosa" within a given file ("iris.csv"), and print to standard output, one by one, the items which it has encountered (held in the $0 variable):
grep is another text processing tool, this one for string and regular expression matching.
If you spend much time doing text processing at the command line, grep is definitely a tool you will get to know well. See here for some more useful details.
sed is a stream editor, yet another text processing and transformation tool, similar to
awk. Let's use it below to change the occurrences of "setosa" in our iris.csv file to "iris-setosa," using this line:
history is pretty straightforward, but also pretty useful, especially if you're depending on replicating some data preparation you accomplished at the command line.
And there you have a simple introduction to 12 handy command line tools. This is only a taste of what is possible at the command line for data science (or any other goal, for that matter). Free yourself from the mouse and watch your productivity increase.
- Data Science at the Command Line: Exploring Data
- How Docker Can Help You Become A More Effective Data Scientist
- Using Excel with Pandas