NLTK(Natural Language Toolkit)是一個用于自然語言處理的Python庫,可以用來實現文本清洗。下面是使用NLTK庫來進行文本清洗的一些常見步驟:
from nltk.tokenize import word_tokenize
text = "Hello, how are you?"
tokens = word_tokenize(text)
print(tokens)
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
filtered_words = [word for word in tokens if word.lower() not in stop_words]
print(filtered_words)
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
stemmed_words = [stemmer.stem(word) for word in filtered_words]
print(stemmed_words)
import re
cleaned_text = re.sub(r'[^\w\s]', '', text)
print(cleaned_text)
通過上述步驟,可以使用NLTK庫實現文本清洗,將文本數據轉換為更易于處理和分析的形式。