python – 用nltk训练我自己的分类器后,如何在textblob中加载它?
textblob中的内置分类器非常愚蠢.它受过电影评论的训练,因此我在我的背景下创建了大量示例(57,000个故事,分类为正面或负面),然后使用nltk进行训练.我尝试使用textblob来训练它,但它总是失败:
with open('train.json', 'r') as fp:
cl = NaiveBayesClassifier(fp, format="json")
这将持续数小时并以内存错误结束.
我查看了源代码,发现它只是使用了nltk并将其包装起来,所以我使用了它,并且它起作用了.
nltk训练集的结构需要是一个元组列表,第一部分是文本中的单词计数器和出现频率.元组的第二部分是“pos”或“neg”的情绪.
>>> train_set = [(Counter(i["text"].split()),i["label"]) for i in data[200:]]
>>> test_set = [(Counter(i["text"].split()),i["label"]) for i in data[:200]] # withholding 200 examples for testing later
>>> cl = nltk.NaiveBayesClassifier.train(train_set) # <-- this is the same thing textblob was using
>>> print("Classifier accuracy percent:",(nltk.classify.accuracy(cl, test_set))*100)
('Classifier accuracy percent:', 66.5)
>>>>cl.show_most_informative_features(75)
然后我腌了它.
with open('storybayes.pickle','wb') as f:
pickle.dump(cl,f)
现在……我拿了这个腌制文件,然后重新打开它以获得nltk.classifier’nltk.classify.naivebayes.NaiveBayesClassifier’> – 并尝试将其提供给textblob.代替
from textblob.classifiers import NaiveBayesClassifier
blob = TextBlob("I love this library", analyzer=NaiveBayesAnalyzer())
我试过了:
blob = TextBlob("I love this library", analyzer=myclassifier)
Traceback (most recent call last):
File "<pyshell#116>", line 1, in <module>
blob = TextBlob("I love this library", analyzer=cl4)
File "C:\python\lib\site-packages\textblob\blob.py", line 369, in __init__
parser, classifier)
File "C:\python\lib\site-packages\textblob\blob.py", line 323, in
_initialize_models
BaseSentimentAnalyzer, BaseBlob.analyzer)
File "C:\python\lib\site-packages\textblob\blob.py", line 305, in
_validated_param
.format(name=name, cls=base_class_name))
ValueError: analyzer must be an instance of BaseSentimentAnalyzer
现在怎么办?我查看了源代码,两者都是类,但不完全一样.