ソフトウェアエンジニアの技術ブログ：Software engineer tech blog

transformersを使います
$ pip3 install transformers==4.3.3 torch==1.8.0 sentencepiece==0.1.91

# -*- coding: utf-8 -*-
import transformers

tokenizer = transformers.AutoTokenizer.from_pretrained("colorfulscoop/gpt2-small-ja")
model = transformers.AutoModelForCausalLM.from_pretrained("colorfulscoop/gpt2-small-ja")

input = tokenizer.encode("昔々あるところに", return_tensors="pt")
output = model.generate(input, do_sample=True, top_p=0.95, top_k=50, num_return_sequences=3)

print(tokenizer.batch_decode(output))

[‘昔々あるところには、お祭りの女神さんが現れ、そこでお姫様の姫様’, ‘昔々あるところに、ある。ある夏の日、彼は旅人と出会い、その目的がどう’, ‘昔々あるところに、一億年も前には人間たちが住んでいた。いまや、それはこの’]

おおお、なんか色々出来そうではある…
途中の処理を考える必要はあるが

$ pip3 install asari

from asari.api import Sonar

sonar = Sonar()
sonar.ping(text="広告が多すぎる♡")

$ python3 asari.py
Traceback (most recent call last):
File “asari.py”, line 1, in
from asari.api import Sonar
File “/home/vagrant/dev/nlp/asari.py”, line 1, in
from asari.api import Sonar
ModuleNotFoundError: No module named ‘asari.api’; ‘asari’ is not a package

—-
Requirement already satisfied: joblib>=0.11 in /home/vagrant/.local/lib/python3.8/site-packages (from scikit-learn>=0.19.1->asari) (1.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/vagrant/.local/lib/python3.8/site-packages (from scikit-learn>=0.19.1->asari) (2.2.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7.3->pandas>=0.22.0->asari) (1.14.0)
Installing collected packages: pytz, pandas, Janome, asari
Successfully installed Janome-0.4.1 asari-0.0.4 pandas-1.3.4 pytz-2021.3

python2系じゃないと動かないのか？

python2系の環境で再度やり直します。
$ pip install scikit-learn==0.20.4
$ pip install Janome==0.3.7

[{‘text’: ‘広’, ‘top_class’: ‘positive’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.28345109397418017}, {‘class_name’: ‘positive’, ‘confidence’: 0.7165489060258198}]}, {‘text’: ‘告’, ‘top_class’: ‘positive’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.3450436370027103}, {‘class_name’: ‘positive’, ‘confidence’: 0.6549563629972897}]}, {‘text’: ‘が’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.9006654377630458}, {‘class_name’: ‘positive’, ‘confidence’: 0.09933456223695437}]}, {‘text’: ‘多’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.9364137330979464}, {‘class_name’: ‘positive’, ‘confidence’: 0.06358626690205357}]}, {‘text’: ‘す’, ‘top_class’: ‘positive’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.2124141523260128}, {‘class_name’: ‘positive’, ‘confidence’: 0.7875858476739873}]}, {‘text’: ‘ぎ’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.5383816766180572}, {‘class_name’: ‘positive’, ‘confidence’: 0.4616183233819428}]}, {‘text’: ‘る’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.6881484923868434}, {‘class_name’: ‘positive’, ‘confidence’: 0.3118515076131566}]}]

# -*- coding: utf-8 -*-
from asari.api import Sonar

sonar = Sonar()
text="広告が多すぎる"
res = sonar.ping(text="広告多すぎる♡")
print(res)

$ python3 app.py
{‘text’: ‘広告多すぎる♡’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.9086981552962491}, {‘class_name’: ‘positive’, ‘confidence’: 0.0913018447037509}]}

なるほど、janomeとscikit-learnはversionを指定すると動く