In this lesson, we will study some of the concepts related to text analysis with the help of NLTK library in Python. Some of these concepts will involve:<\/p>\n
- \n
- Tokenization, how to break a piece of text into words, sentences<\/li>\n
- Avoiding stop words based on English language<\/li>\n
- Performing stemming and lemmatization on a piece of text<\/li>\n
- Identifying the tokens to be analysed<\/li>\n<\/ul>\n
NLP will be the main area of focus in this lesson as it is applicable to enormous real-life scenarios where it can solve big and crucial problems. If you think this sounds complex, well it does but the concepts are equally easy to understand if you try examples side by side. Let\u2019s jump into installing NLTK on your machine to get started with it.<\/p>\n
Installing NLTK<\/strong><\/h3>\n
Just a note before starting, you can use a\u00a0virtual environment<\/a>\u00a0for this lesson which we can be made with the following command:<\/p>\n
\npython -m virtualenv nltk
\nsource nltk\/bin\/activate<\/div>\n<\/div>\nOnce the virtual environment is active, you can install NLTK library within the virtual env so that examples we create next can be executed:<\/span><\/p>\n
\npip install nltk<\/div>\n<\/div>\nWe will make use of\u00a0Anaconda<\/a>\u00a0and Jupyter in this lesson. If you want to install it on your machine, look at the lesson which describes \u201cHow to Install Anaconda Python on Ubuntu 18.04 LTS<\/a>\u201d and share your feedback if you face any issues. To install NLTK with Anaconda, use the following command in the terminal from Anaconda:<\/span><\/p>\n
\nconda install -c anaconda nltk<\/div>\n<\/div>\nWe see something like this when we execute the above command:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/1-3.png\")
<\/p>\nOnce all of the packages needed are installed and done, we can get started with using the NLTK library with the following import statement:<\/span><\/p>\n
\nimport<\/span>\u00a0nltk<\/div>\n<\/div>\nLet\u2019s get started with basic NLTK examples now that we have the prerequisites packages installed.<\/p>\n
Tokenization<\/strong><\/h3>\n
We will start with Tokenization which is the first step in performing text analysis. A token can be any smaller part of a piece of text which can be analysed. There are two types of Tokenization which can be performed with NLTK:<\/p>\n
- \n
- Sentence Tokenization<\/li>\n
- Word Tokenization<\/li>\n<\/ul>\n
You can guess what happens on each of the Tokenization so let\u2019s dive into code examples.<\/p>\n
Sentence Tokenization<\/strong><\/h3>\n
As the name reflects, Sentence Tokenizers breaks a piece of text into sentences. Let\u2019s try a simple code snippet for the same where we make use of a text we picked from\u00a0Apache Kafka<\/a>\u00a0tutorial<\/span>. We will perform the necessary imports<\/p>\n
\nimport<\/span>\u00a0nltk
\nfrom<\/span>\u00a0nltk.tokenize<\/span>\u00a0import<\/span>\u00a0sent_tokenize<\/div>\n<\/div>\nPlease note that you might face an error due to a missing dependency for nltk called\u00a0punkt<\/strong>. Add the following line right after the imports in the program to avoid any warnings:<\/p>\n
\nnltk.download<\/span>(<\/span>‘punkt’<\/span>)<\/span><\/div>\n<\/div>\nFor me, it gave the following output:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/2-3.png\")
<\/p>\nNext, we make use of the sentence tokenizer we imported:<\/span><\/p>\n
\ntext = “””A Topic in Kafka is something where a message is sent. The consumer
\napplications which are interested in that topic pulls the message inside that
\ntopic and can do anything with that data. Up to a specific time, any number of
\nconsumer applications can pull this message any number of times.”””<\/p>\nsentences = sent_tokenize(text)
\nprint<\/span>(sentences)<\/div>\n<\/div>\nWe see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/3-3.png\")
<\/p>\nAs expected, the text was correctly organised into sentences.<\/p>\n
Word Tokenization<\/strong><\/h3>\n
As the name reflects, Word Tokenizers breaks a piece of text into words. Let\u2019s try a simple code snippet for the same with the same text as the previous example:<\/p>\n
\nfrom<\/span>\u00a0nltk.tokenize<\/span>\u00a0import<\/span>\u00a0word_tokenize<\/p>\nwords\u00a0=<\/span>\u00a0word_tokenize(<\/span>text)<\/span>
\nprint<\/span>(<\/span>words)<\/span><\/div>\n<\/div>\nWe see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/4-3.png\")
<\/p>\nAs expected, the text was correctly organised into words.<\/p>\n
Frequency Distribution<\/strong><\/h3>\n
Now that we have broken the text, we can also calculate frequency of each word in the text we used. It is very simple to do with NLTK, here is the code snippet we use:<\/p>\n
\nfrom<\/span>\u00a0nltk.probability<\/span>\u00a0import<\/span>\u00a0FreqDist<\/p>\ndistribution\u00a0=<\/span>\u00a0FreqDist(<\/span>words)<\/span>
\nprint<\/span>(<\/span>distribution)<\/span><\/div>\n<\/div>\nWe see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/5-3.png\")
<\/p>\nNext, we can find most common words in the text with a simple function which accepts the number of words to show:<\/span><\/p>\n
\n# Most common words<\/span>
\ndistribution.most_common<\/span>(<\/span>2<\/span>)<\/span><\/div>\n<\/div>\nWe see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/6-3.png\")
<\/p>\nFinally, we can make a frequency distribution plot to clear out the words and their count in the given text and clearly understand the distribution of words:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/7-3.png\")
<\/p>\nStopwords<\/strong><\/h3>\n
Just like when we talk to another person via a call, there tends to be some noise over the call which is unwanted information. In the same manner, text from real world also contain noise which is termed as\u00a0Stopwords<\/strong>. Stopwords can vary from language to language but they can be easily identified. Some of the Stopwords in English language can be \u2013 is, are, a, the, an etc.<\/p>\n
We can look at words which are considered as Stopwords by NLTK for English language with the following code snippet:<\/p>\n
\nfrom<\/span>\u00a0nltk.corpus<\/span>\u00a0import<\/span>\u00a0stopwords
\nnltk.download<\/span>(<\/span>‘stopwords’<\/span>)<\/span><\/p>\nlanguage\u00a0=<\/span>\u00a0“english”<\/span>
\nstop_words\u00a0=<\/span>\u00a0set<\/span>(<\/span>stopwords.words<\/span>(<\/span>language)<\/span>)<\/span>
\nprint<\/span>(<\/span>stop_words)<\/span><\/div>\n<\/div>\nAs of course the set of stop words can be big, it is stored as a separate dataset which can be\u00a0downloaded<\/span>\u00a0with NLTK as we shown above. We see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/8-3.png\")
<\/p>\nThese stop words should be removed from the text if you want to perform a precise text analysis for the piece of text provided. Let\u2019s remove the stop words from our textual tokens:<\/span><\/p>\n
\nfiltered_words\u00a0=<\/span>\u00a0[<\/span>]<\/span><\/p>\nfor<\/span>\u00a0word\u00a0in<\/span>\u00a0words:
\nif<\/span>\u00a0word\u00a0not<\/span>\u00a0in<\/span>\u00a0stop_words:
\nfiltered_words.append<\/span>(<\/span>word)<\/span><\/p>\nfiltered_words<\/p><\/div>\n<\/div>\n
We see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/9-2.png\")
<\/p>\nWord Stemming<\/strong><\/h3>\n
A stem of a word is the base of that word. For example:<\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/11-2.png\")
<\/p>\nWe will perform stemming upon the filtered words from which we removed stop words in the last section. Let\u2019s write a simple code snippet where we use NLTK\u2019s stemmer to perform the operation:<\/span><\/p>\n
\nfrom<\/span>\u00a0nltk.stem<\/span>\u00a0import<\/span>\u00a0PorterStemmer
\nps\u00a0=<\/span>\u00a0PorterStemmer(<\/span>)<\/span><\/p>\nstemmed_words\u00a0=<\/span>\u00a0[<\/span>]<\/span>
\nfor<\/span>\u00a0word\u00a0in<\/span>\u00a0filtered_words:
\nstemmed_words.append<\/span>(<\/span>ps.stem<\/span>(<\/span>word)<\/span>)<\/span><\/p>\nprint<\/span>(<\/span>“Stemmed Sentence:”<\/span>,<\/span>\u00a0stemmed_words)<\/span><\/div>\n<\/div>\n
We see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/12-2.png\")
<\/p>\nPOS Tagging<\/strong><\/h3>\n
Next step in textual analysis is after stemming is to identify and group each word in terms of their value, i.e. if each of the word is a noun or a verb or something else. This is termed as Part of Speech tagging. Let\u2019s perform POS tagging now:<\/p>\n
\ntokens=<\/span>nltk.word_tokenize<\/span>(<\/span>sentences[<\/span>0<\/span>]<\/span>)<\/span>
\nprint<\/span>(<\/span>tokens)<\/span><\/div>\n<\/div>\nWe see something like this when we execute the above script:<\/span><\/p>\n
![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/14-1.png\")
<\/p>\nNow, we can perform the tagging, for which we will have to\u00a0download<\/span>\u00a0another dataset to identify the correct tags:<\/span><\/p>\n
\nnltk.download<\/span>(<\/span>‘averaged_perceptron_tagger’<\/span>)<\/span>
\nnltk.pos_tag<\/span>(<\/span>tokens)<\/span><\/div>\n<\/div>\n
\nHere is the output of the tagging:<\/span><\/p>\n![\"\"](\"https:\/\/linuxhint.com\/wp-content\/uploads\/2019\/03\/15-1.png\")
<\/p>\nNow that we have finally identified the tagged words, this is the dataset on which we can perform sentiment analysis to identify the emotions behind a sentence.<\/p>\n
Conclusion<\/strong><\/h4>\n
In this lesson, we looked at an excellent natural language package, NLTK which allows us to work with unstructured textual data to identify any stop words and perform deeper analysis by preparing a sharp data set for text analysis with libraries like sklearn.<\/p>\n
Find all of the source code used in this lesson on\u00a0Github<\/a>.<\/p>\n