Introduction to Named Entity Recognition
Named Entity Recognition is a tool which invariably comes handy when we do Natural Language Processing tasks. Read on to find out how.
NLTK (Natural Language Toolkit) is a Python package that provides a set of natural languages corpora and APIs of wide varieties of NLP algorithms.
To perform Named Entity Recognition using NLTK, it needs to be done in three stages —
- Work Tokenization
- Parts of Speech (POS) tagging
- Named Entity Recognition
Now, let’s perform the first two stages here -
Note, we need to download some standard corpora and API from NLTK to perform parts of speech tagging and named entity recognition. Hence, we downloaded these from nltk in the above Python code.
Snapshot of Output (POS tagging) from the above code
Now to understand what each codes mean, please refer to the below list-
CC coordinating conjunction
CD cardinal digit
EX existential there (like: “there is” … think of it like “there exists”)
FW foreign word
IN preposition/subordinating conjunction
JJ adjective ‘big’
JJR adjective, comparative ‘bigger’
JJS adjective, superlative ‘biggest’
LS list marker 1)
MD modal could, will
NN noun, singular ‘desk’
NNS noun plural ‘desks’
NNP proper noun, singular ‘Harrison’
NNPS proper noun, plural ‘Americans’
PDT predeterminer ‘all the kids’
POS possessive ending parent’s
PRP personal pronoun I, he, she
PRP$ possessive pronoun my, his, hers
RB adverb very, silently,
RBR adverb, comparative better
RBS adverb, superlative best
RP particle give up
TO, to go ‘to’ the store.
VB verb, base form take
VBD verb, past tense took
VBG verb, gerund/present participle taking
VBN verb, past participle taken
VBP verb, sing. present, non-3d take
VBZ verb, 3rd person sing. present takes
WDT wh-determiner which
WP wh-pronoun who, what
WP$ possessive wh-pronoun whose
WRB wh-abverb where, when
Now once we have done the parts-of-speech tagging we will be doing a process called chunking. Text chunking is also called as shallow parsing which typically follows POS tagging to add more structure to the sentence. The result is grouping of words in “chunks”.
So, lets perform chunking to our article which we have already POS tagged.
Our target here would be to NER tag only the Nouns.
The snapshot of the output is as follows-
Snapshot of the output from the above code
The output looks decent but not great. Say we take up a little more complex task.
Say, we want to implement noun phrase chunking to identify named entities.
Our chunk pattern consists of one rule, that a noun phrase, NP, should be formed whenever the chunker finds an optional determiner, DT, followed by any number of adjectives, JJ, and then a noun, NN.
The output of the above chunking is below-
Snapshot from the output from above
The output can be read as a tree with “S” means the sentence as the first level. It can viewed in a more acceptable format called IOB tags (Inside, Outside, Beginning)
The snapshot of the output from the above code
Here, in the output each token is a line with parts-of-speech and named entity tagged. If you want to extract the IOB tags, as it is a tuple you simply do-
The entire code for the NLTK NER process-
NER using NLTK
What’s next ?
So, we have just learnt what is Named Entity Recognition tagging and how to use them to solve generic problems using API’s.
The natural progression from here would be to accomplish three things -
- Build your own NER tagger and also explore languages other than English.
- Build more sophisticated NER models (let’s say using Deep Learning) and also evaluate how better they perform.
- Take a task which you encounter daily which deals with Natural Language, figure out a problem which you want to solve and then use all what you have learnt in NER to solve it.
I will be working on these lines and will try to share my learning in coming posts on NER. You can contribute as well, please drop me how would you like to do that in the comment section.
Happy learning :)
- Parallel Dots blog on Named Entity Recognition
- NER Applications and Use Cases by Shashank Gupta
- Stanford Named Entity Recognizer
- Named Entity Recognition from Sicara’s blog
- spaCy library documentation on NER
- Use of NER in Information Extraction by Edward Ma
- NER & POS using NLTK by Gianpaul Rachiele
- NER using NLTK & spaCy by Susan Li
- Text chunking with NLTK from NLP for Hackers by Bogdan
Bio: Suvro Banerjee is a Machine Learning Engineer @ Juniper Networks.
Original. Reposted with permission.
- Named Entity Recognition and Classification with Scikit-Learn
- Introduction to Apache Spark
- Named Entity Recognition: A Practitioner’s Guide to NLP