top of page
Search

Natural Language Processing

  • Writer: Arturo Devesa
    Arturo Devesa
  • Apr 15, 2020
  • 1 min read

to be continued...

# regex for removing punctuation!
import re
# nltk preprocessing magic
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
from nltk.stem import WordNetLemmatizer
# grabbing a part of speech function:
from part_of_speech import get_part_of_speech

text = "So many squids are jumping out of suitcases these days that you can barely go anywhere without seeing one burst forth from a tightly packed valise. I went to the dentist the other day, and sure enough I saw an angry one jump out of my dentist's bag within minutes of arriving. She hardly even noticed."

cleaned = re.sub('\W+', ' ', text)
tokenized = word_tokenize(cleaned)

stemmer = PorterStemmer()
stemmed = [stemmer.stem(token) for token in tokenized]

## -- CHANGE these -- ##
lemmatizer = None
lemmatized = []

print("Stemmed text:")
print(stemmed)
print("\nLemmatized text:")
print(lemmatized)

 
 
 

Recent Posts

See All
Python Parallel AI Agents

example that illustrates how you might evolve the simple iterative planner–critic loop into a parallel agents design. In this example,...

 
 
 

Comments


©2020 by Arturo Devesa.

bottom of page