Search

Natural Language Processing

to be continued...

# regex for removing punctuation!
import re
# nltk preprocessing magic
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
from nltk.stem import WordNetLemmatizer
# grabbing a part of speech function:
from part_of_speech import get_part_of_speech

text = "So many squids are jumping out of suitcases these days that you can barely go anywhere without seeing one burst forth from a tightly packed valise. I went to the dentist the other day, and sure enough I saw an angry one jump out of my dentist's bag within minutes of arriving. She hardly even noticed."

cleaned = re.sub('\W+', ' ', text)
tokenized = word_tokenize(cleaned)

stemmer = PorterStemmer()
stemmed = [stemmer.stem(token) for token in tokenized]

## -- CHANGE these -- ##
lemmatizer = None
lemmatized = []

print("Stemmed text:")
print(stemmed)
print("\nLemmatized text:")
print(lemmatized)

Recent Posts

See All

Socket Programming in Python

Sockets and the socket API are used to send messages across a network. The network can be a logical, local network to the computer, or one that’s physically connected to an external network like the i

Financial Machine Learning

"The essential tool of econometrics is multivariate linear regression, an 18th century technology that was already mastered by Gauss before 1794...It is hard to believe that something as complex as 21

©2020 by Arturo Devesa.