(Abstracted with permission from NLP in a Hurry by Pier Lim.)

Here is a collection of different Python libraries for natural language processing (NLP) which can be invaluable for rapid prototyping.

Semantic Similarity

Sentence Transformers – https://github.com/UKPLab/sentence-transformers

BERT/XLNET produces rather bad sentence embeddings out-of-the-box . This library helps you produce your own sentence embeddings tuned for your specific task. This would be useful for anything to do with semantic textual similarity, clustering and semantic search.

Rule-based Text Sentiment for Social Media

VaderSentiment – https://github.com/cjhutto/vaderSentiment

While deep learning models are cool, rule-based models still have their place under the sun. Especially when you don’t have a lot of data and time to tune your model. The library describes itself as specifically attuned to sentiments expressed in social media. This means that emoticons and sentiment intensity markers (e.g. “!!!”) are taken into account.

Named Entity Recognition

SpaCy – https://spacy.io/usage/linguistic-features#named-entities-101

SpaCy is a popular open-source library which can be used for production. Apart from the default entities, spaCy also gives us the liberty to add arbitrary classes to the NER model, by training the model to update it with newer trained examples. This blog post shows how.

Production-Ready BERT Models

BERT-as-a-Service – https://github.com/hanxiao/bert-as-service

BERT-as-a-Service wraps the BERT code and serves it using ZeroMQ, allowing one to serve BERT embeddings with just a few lines of code which is fast (optimised), scalable and reliable.

Related Articles

Analysis of Tweets on the Hong Kong Protest Movement 2019 with Python demonstrates the use of the Vader tool.