[自然言語処理] pythonとasariで感情分析を行う

$ pip3 install asari

from asari.api import Sonar

sonar = Sonar()
sonar.ping(text="広告が多すぎる♡")

$ python3 asari.py
Traceback (most recent call last):
File “asari.py”, line 1, in
from asari.api import Sonar
File “/home/vagrant/dev/nlp/asari.py”, line 1, in
from asari.api import Sonar
ModuleNotFoundError: No module named ‘asari.api’; ‘asari’ is not a package

—-
Requirement already satisfied: joblib>=0.11 in /home/vagrant/.local/lib/python3.8/site-packages (from scikit-learn>=0.19.1->asari) (1.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/vagrant/.local/lib/python3.8/site-packages (from scikit-learn>=0.19.1->asari) (2.2.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7.3->pandas>=0.22.0->asari) (1.14.0)
Installing collected packages: pytz, pandas, Janome, asari
Successfully installed Janome-0.4.1 asari-0.0.4 pandas-1.3.4 pytz-2021.3

python2系じゃないと動かないのか?

python2系の環境で再度やり直します。
$ pip install scikit-learn==0.20.4
$ pip install Janome==0.3.7

[{‘text’: ‘広’, ‘top_class’: ‘positive’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.28345109397418017}, {‘class_name’: ‘positive’, ‘confidence’: 0.7165489060258198}]}, {‘text’: ‘告’, ‘top_class’: ‘positive’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.3450436370027103}, {‘class_name’: ‘positive’, ‘confidence’: 0.6549563629972897}]}, {‘text’: ‘が’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.9006654377630458}, {‘class_name’: ‘positive’, ‘confidence’: 0.09933456223695437}]}, {‘text’: ‘多’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.9364137330979464}, {‘class_name’: ‘positive’, ‘confidence’: 0.06358626690205357}]}, {‘text’: ‘す’, ‘top_class’: ‘positive’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.2124141523260128}, {‘class_name’: ‘positive’, ‘confidence’: 0.7875858476739873}]}, {‘text’: ‘ぎ’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.5383816766180572}, {‘class_name’: ‘positive’, ‘confidence’: 0.4616183233819428}]}, {‘text’: ‘る’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.6881484923868434}, {‘class_name’: ‘positive’, ‘confidence’: 0.3118515076131566}]}]

# -*- coding: utf-8 -*-
from asari.api import Sonar

sonar = Sonar()
text="広告が多すぎる"
res = sonar.ping(text="広告多すぎる♡")
print(res)

$ python3 app.py
{‘text’: ‘広告多すぎる♡’, ‘top_class’: ‘negative’, ‘classes’: [{‘class_name’: ‘negative’, ‘confidence’: 0.9086981552962491}, {‘class_name’: ‘positive’, ‘confidence’: 0.0913018447037509}]}

なるほど、janomeとscikit-learnはversionを指定すると動く

[Lazy Load] 画像の遅延ロードの書き方

class=”lazyload”とし、画像はdata-srcで読み込む。
lazyloadはcdnで読み込み、scriptでlazyload();とする。

	<img class="lazyload" data-src="img/1.jpg" width="300" height="200"><br><br><br><br>
	<img class="lazyload" data-src="img/2.jpg" width="300" height="200"><br><br><br><br>
	<img class="lazyload" data-src="img/3.jpg" width="300" height="200">
	<script src="https://cdn.jsdelivr.net/npm/lazyload@2.0.0-rc.2/lazyload.min.js"></script>
	<script>
    lazyload();
	</script>

Googleのdevtoolで確認すると、3枚あるのが2枚しか読み込まれていないことがわかる。

なかなかやりおる

[RaspberryPI4] DeepSpeechをmicで動かす

### ラズパイのUBSマイクの設定
$ sudo vim /etc/modprobe.d/alsa-base.conf

options snd slots=snd_usb_audio,snd_bcm2835
options snd_usb_audio index=0
options snd_bcm2835 index=1
$ sudo vim ~/.profile
一番最後の行に追加
export ALSADEV=”plughw:0,0″

$ sudo apt-get install alsa-utils sox libsox-fmt-all
$ sudo sh -c “echo snd-pcm >> /etc/modules”

ラズパイ再起動

### マイクから音声認識
$ source dev/deepspeech-train-venv/bin/activate
$ cd deepspeech
$ git clone https://github.com/mozilla/DeepSpeech-examples
$ cd DeepSpeech-examples/DeepSpeech-examples
$ pip install -r requirements.txt
$ sudo apt install portaudio19-dev
$ pip3 install halo webrtcvad –upgrade
$ python3 DeepSpeech-examples/mic_vad_streaming/mic_vad_streaming.py -m deepspeech-0.7.1-models.tflite -s deepspeech-0.7.1-models.scorer

こんな感じになる
Listening (ctrl-C to exit)…
Recognized:
Recognized:
Recognized:
Recognized: you
Recognized:
Recognized: night

きたあああああああああああああああああああああ
さあ サーバーサイドやろう
とりあえず満足

https://github.com/mozilla/DeepSpeech-examples/blob/r0.9/mic_vad_streaming/mic_vad_streaming.py
line188行目でtextをrecognized:としているので、ここでテキストとして保存すれば良い

        else:
            if spinner: spinner.stop()
            logging.debug("end utterence")
            if ARGS.savewav:
                vad_audio.write_wav(os.path.join(ARGS.savewav, datetime.now().strftime("savewav_%Y-%m-%d_%H-%M-%S_%f.wav")), wav_data)
                wav_data = bytearray()
            text = stream_context.finishStream()
            print("Recognized: %s" % text)
            stream_context = model.createStream()

[RaspberryPI4]DeepSpeechを動かしたい

まずラズパイ4とモニターを秋葉原で購入して起動
キーボードから日本語入力ができるように設定

$ mkdir dev
$ cd dev
$ python3 -m venv deepspeech-train-venv
$ source dev/deepspeech-train-venv/bin/activate
$ mkdir deepspeech
$ cd deepspeech
$ pip install deepspeech

$ curl -LO https://github.com/mozilla/STT/releases/download/v0.7.1/deepspeech-0.7.1-models.tflite
$ curl -LO https://github.com/mozilla/STT/releases/download/v0.7.1/deepspeech-0.7.1-models.pbmm
$ curl -LO https://github.com/mozilla/STT/releases/download/v0.7.1/deepspeech-0.7.1-models.scorer

$ curl -LO https://github.com/mozilla/STT/releases/download/v0.7.1/audio-0.7.1.tar.gz
$ tar xvf audio-0.7.1.tar.gz

$ deepspeech –model deepspeech-0.7.*-models.tflite –scorer deepspeech-0.7.*-models.scorer –audio audio/2830-3980-0043.wav
Loading model from file deepspeech-0.7.1-models.tflite
TensorFlow: v2.3.0-6-g23ad988
DeepSpeech: v0.9.3-0-gf2e9c85
Loaded model in 0.00285s.
Loading scorer from files deepspeech-0.7.1-models.scorer
Loaded scorer in 0.00296s.
Running inference.
experience proves this
Inference took 1.546s for 1.975s audio file.

venvを使わないと以下のようなエラーになるので注意が必要
The script deepspeech is installed in ‘/home/pi/.local/bin’ which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use –no-warn-script-location.

OK 続いてdeeepspeechをmicでやりたい

[ubunut] apache2でサブドメインをnodeのexpressを動かしたい

### ローカル環境
挙動テスト用に簡単にインストールしてコードを書きます。
$ node –version
v14.17.6

$ npm install express

var express = require('express')
var app = express()
app.get('/', function(req, res){
	res.send('hello world')
})

app.listen(3000)

http://192.168.34.10:3000/

### vps
$ curl -sL https://deb.nodesource.com/setup_14.x -o nodesource_setup.sh
$ sudo bash nodesource_setup.sh
$ sudo apt install nodejs
$ node -v
v14.18.1
$ cd /var/www/app
$ sudo npm install express
$ sudo touch app.js
$ sudo vi app.js // 上記と同じ

$ sudo vi /etc/apache2/sites-available/virtual.host.conf

<VirtualHost *:80>
DocumentRoot /var/www/app
ServerName ****-test.site
ServerAlias www.****-test.site
ProxyPass / http://***.**.***.**:3000/
ProxyPassReverse / http://***.**.***.**:3000/
# Other directives here

</VirtualHost>

Invalid command ‘ProxyPass’, perhaps misspe>

$ sudo a2enmod proxy_http
$ sudo systemctl restart apache2

$ node app.js

var express = require('express')
var app = express()
app.get('/', function(req, res){
	res.send('hello node')
})

app.listen(3000)

OK

$ sudo vi /etc/apache2/sites-available/virtual.host.conf

<VirtualHost *:80>
DocumentRoot /var/www/app
ServerName ****-test.site
ServerAlias www.sample-test.site
ProxyPass / http://***.**.**.***:3000
ProxyPassReverse / http://***.**.**.***:3000
# Other directives here

</VirtualHost>

<VirtualHost *:80>
DocumentRoot /var/www/node
ServerName ***.****-test.site
# Other directives here

</VirtualHost>

こうすることで、一つのディレクトリはnodeで動かし、サブドメインの方はapacheで動かすということもできる

OKKKKKKKKKKK
さ、ラズパイやろう

[Ubuntu] apache2のvirtualhostの反映 (vps)

vps(さくらvps)でapache2のvirtualhostの反映を設定します。

sudo vi /etc/apache2/sites-available/virtual.host.conf

<VirtualHost *:80>
DocumentRoot /var/www/app
ServerName sample-test.site
ServerAlias www.sample-test.site
# Other directives here

</VirtualHost>

$ sudo systemctl reload apache2
$ sudo systemctl restart apache2
-> これだと000-default.confの設定が優先されるので、反映されない

$ sudo a2dissite 000-default.conf
$ sudo a2ensite virtual.host.conf
$ sudo systemctl reload apache2

なるほど
ここから、サブドメインを設定する

[RDS] snapshotから別のDBを起動する

### 1. RDS作成
まず、RDSでpostgreSQLを作成し、EC2から接続してデータを挿入します。

$ psql -h xxxxxx.*.ap-northeast-1.rds.amazonaws.com -U root -d postgres

CREATE TABLE playground (
    equip_id serial PRIMARY KEY,
    type varchar (50) NOT NULL,
    color varchar (25) NOT NULL,
    location varchar(25) check (location in ('north', 'south', 'west', 'east', 'northeast', 'southeast', 'southwest', 'northwest')),
    install_date date
);

INSERT INTO playground (type, color, location, install_date) VALUES ('slide', 'blue', 'south', '2022-04-28');
INSERT INTO playground (type, color, location, install_date) VALUES ('swing', 'yellow', 'northwest', '2022-08-16');

postgres=> SELECT * FROM playground;

### 2. snapshotsからrestore
RDSのsnapshotsでsnapshotsを作成し、少し時間を置いてから、restoreを押下します。

するとDB作成時と同じ様な設定画面が表示される。

create databaseをすると元のDBとは別のDBが作成される

### restoreしたRDSに接続
endpointをコピーして、新しいDBの方に接続
$ psql -h xxxxxx.*.ap-northeast-1.rds.amazonaws.com -U root -d postgres
postgres=> SELECT * FROM playground;
equip_id | type | color | location | install_date
———-+——-+——–+———–+————–
1 | slide | blue | south | 2022-04-28
2 | swing | yellow | northwest | 2022-08-16
(2 行)

うおおおおおおおおおおおおおおお
これは凄い
完全に理解した

[Docker] コンテナのディレクトリ・ファイルをローカル環境(ホスト)に取得

ファイルのコピーは、コンテナ⇆ホスト どちらもdocker cpで行う

以下の様にループで回すこともできるが、

for f in $(sudo docker exec -it vigorous_shannon bash -c "ls /opt/app/*"); do sudo docker cp vigorous_shannon:$f /home/vagrant/dev/test; done

この様なエラーになる
Error: No such container:path: vigorous_shannon:models.py

そのため、ディレクトリを丸ごとコピーした方が早い

sudo docker cp vigorous_shannon:/opt/app /home/vagrant/dev/test

おっけーーーーーーー
さて、作業するか

[Sass] media queryを変数で書いて重複をなくしたい

HTML: チャット画面をBluma CSSでinputとsubmitを書いているが、classのis-expandが横幅maxにならないので、CSSで横幅を指定する。その際に、レスポンシブルでスマホはwidth85%, desktopはwidth65%としたい。

			<div class="field is-grouped" id="newMessage">
			  <p class="control is-expanded" >
				<input type="text" id="message" class="input">
			  </p>
			  <p class="control">
			    <input type="submit" value="send" class="button is-link">
			  </p>
			</div>

画面

### 従来のmedia queryで書く場合
style.scss: これでも動くには動くが、プログラマーとしては重複して書くのはかなり気持ちが悪い

@media screen and (min-width: 768px) {
	#newMessage {
		position: fixed;
		width:65%;
		bottom: 30px;	
	}
}
@media screen and (max-width: 768px) {
    #newMessage {
		position: fixed;
		width:85%;
		bottom: 30px;	
	}
}

### Sassで変数を使って書く場合
media queryで表示を分けて書く箇所が多い場合はかなり重宝できる

$breakpoints: (
	'sm': 'screen and (max-width: 768px)',
	'md': 'screen and (min-width: 768px)',
) !default;

@mixin mq($breakpoint: md) {
	@media #{map-get($breakpoints, $breakpoint)}{
		@content;
	}
}
#newMessage {
	position: fixed;
	@include mq(sm) { 
    	width: 85%
  	}
  	@include mq(md) { 
    	width: 65%
  	}
	bottom: 30px;	
}

webpackでコンパイルされると、以下のように出力されている

あら、 かなり良いですね~
寿司食べたい🍣

[Go言語] revelでchatを作る

conf/routes

GET		/websocket/room		  WebSocket.Room
WS      /websocket/room/socket   WebSocket.RoomSocket 

app/views/WebSocket/room.html

{{set . "title" "Chat room"}}
{{template "header.html" .}}

<h1>WebSocket - You are now chatting as {{.user}} <a href="/">Leave</a></h1>

<div id="thread">
	<script type="text/html" id="message_tmpl">
		{{raw "<%"}} if(event.Type == 'message') { %>
			<div class="message <%= event.User == '{{.user}}' ? 'you' : '' %>">
				<h2>{{raw "<%"}}= event.User %></h2>
				<p>
					{{raw "<%"}}= event.Text %>
				</p>
			</div>
		{{raw "<%"}} } %>
		{{raw "<%"}} if(event.Type == 'join') { %>
			<div class="message notice">
				<h2></h2>
				<p>
					{{raw "<%"}}= event.User %> joined the room
				</p>
			</div>
		{{raw "<%"}} }  %>
		{{raw "<%"}} if(event.Type == 'leave') { %>
			<div class="message notice">
				<h2></h2>
				<p>
					{{raw "<%"}}= event.User %> left the room
				</p>
			</div>
		{{raw "<%"}} }  %>
		{{raw "<%"}} if(event.Type == 'quit') { %>
			<div class="message important">
				<h2></h2>
				<p>
					You are disconnected!
				</p>
			</div>
		{{raw "<%"}} }  %>
	</script>
</div>

<div id="newMessage">
	<input type="text" id="message" autocomplete="off" autofocus>
	<input type="submit" value="send" id="send">
</div>

<script type="text/javascript">

	var socket = new WebSocket('ws://'+window.location.host+'/websocket/room/socket?user={{.user}}')

	var display = function(event){
		$('#thread').append(tmpl('message_tmpl', {event: event}));
		$('#thread').scrollTo('max')
	}

	socket.onmessage = function(event){
		display(JSON.parse(event.data))
	}

	$('#send').click(function(e){
		var message = $('#message').val()
		$('#message').val('')
		socket.send(JSON.stringify(message))
	})

	$('#message').keypress(function(e){
		if(e.charCode == 13 || e.keyCode == 13){
			$('#send').click()
			e.preventDefault()
		}
	})
</script>

app/views/header.html

    <script src="/public/js/jquery.scrollTo-min.js"></script>
    <script src="/public/js/templating.js"></script>

chatroom.go

package chatroom

import (
	"container/list"
	"time"
)

type Event struct {
	Type string
	User string
	Timestamp int
	Text string
}

type Subscription struct {
	Archive []Event
	New <-chan Event
}

func (s Subscription) Cancel(){
	unsubscribe <- s.New
	drain(s.New)
}

func newEvent(typ, user, msg string) Event {
	return Event{typ, user, int(time.Now().Unix()), msg}
}

func Subscribe() Subscription {
	resp := make(chan Subscription)
	subscribe <- resp
	return <- resp
}

func Join(user string){
	publish <- newEvent("join", user, "")
}

func Say(user, message string){
	publish <- newEvent("message", user, message)
}

func Leave(user string){
	publish <- newEvent("leave", user, "")
}

const archiveSize = 10

var (
	subscribe = make(chan (chan<- Subscription), 10)

	unsubscribe = make(chan (<-chan Event), 10)
	publish = make(chan Event, 10)
)

func chatroom(){
	archive := list.New()
	subscribers := list.New()

	for {
		select {
		case ch := <-subscribe:
			var events []Event
			for e := archive.Front(); e != nil; e = e.Next(){
				events = append(events, e.Value.(Event))
			}
			subscriber := make(chan Event, 10)
			subscribers.PushBack(subscriber)
			ch <- Subscription{events, subscriber}
		case event := <-publish:
			for ch := subscribers.Front(); ch != nil; ch = ch.Next(){
				ch.Value.(chan Event) <- event
			}
			if archive.Len() >= archiveSize {
				archive.Remove(archive.Front())
			}
			archive.PushBack(event)
		case unsub := <-unsubscribe:
			for ch := subscribers.Front(); ch != nil; ch = ch.Next(){
				if ch.Value.(chan Event) == unsub {
					subscribers.Remove(ch)
					break
				}
			}
		}
	}
}

func init(){
	go chatroom()
}

func drain(ch <-chan Event){
	for {
		select {
		case _, ok := <-ch:
			if !ok {
				return
			}
		default:
			return
		}
	}
}

websocket.go

package controllers

import (
	"github.com/revel/revel"
	"app/app/chatroom"
)

type WebSocket struct {
	*revel.Controller
}

func (c WebSocket) Room(user string) revel.Result {
	user = "hpscript"
	return c.Render(user)
}

func (c WebSocket) RoomSocket(user string, ws revel.ServerWebSocket) revel.Result {

	if ws == nil {
		return nil
	}

	subscription := chatroom.Subscribe()
	defer subscription.Cancel()

	chatroom.Join(user)
	defer chatroom.Leave(user)

	for _, event := range subscription.Archive {
		if ws.MessageSendJSON(&event) != nil {
			return nil
		}
	}

	newMessages := make(chan string)
	go func(){
		var msg string
		for {
			err := ws.MessageReceiveJSON(&msg)
			if err != nil {
				close(newMessages)
				return
			}
			newMessages <- msg
		}
	}()

	for {
		select {
		case event := <-subscription.New:
			if ws.MessageSendJSON(&event) != nil {
				return nil
			}
		case msg, ok := <-newMessages:
			if !ok {
				return nil
			}
			chatroom.Say(user, msg)
		}
	}
	return nil
}

public/js/templating.js

{{set . "title" "Chat room"}}
{{template "header.html" .}}

<h1>WebSocket - You are now chatting as {{.user}} <a href="/">Leave</a></h1>

<div id="thread">
	<script type="text/html" id="message_tmpl">
		{{raw "<%"}} if(event.Type == 'message') { %>
			<div class="message <%= event.User == '{{.user}}' ? 'you' : '' %>">
				<h2>{{raw "<%"}}= event.User %></h2>
				<p>
					{{raw "<%"}}= event.Text %>
				</p>
			</div>
		{{raw "<%"}} } %>
		{{raw "<%"}} if(event.Type == 'join') { %>
			<div class="message notice">
				<h2></h2>
				<p>
					{{raw "<%"}}= event.User %> joined the room
				</p>
			</div>
		{{raw "<%"}} }  %>
		{{raw "<%"}} if(event.Type == 'leave') { %>
			<div class="message notice">
				<h2></h2>
				<p>
					{{raw "<%"}}= event.User %> left the room
				</p>
			</div>
		{{raw "<%"}} }  %>
		{{raw "<%"}} if(event.Type == 'quit') { %>
			<div class="message important">
				<h2></h2>
				<p>
					You are disconnected!
				</p>
			</div>
		{{raw "<%"}} }  %>
	</script>
</div>

<div id="newMessage">
	<input type="text" id="message" autocomplete="off" autofocus>
	<input type="submit" value="send" id="send">
</div>

<script type="text/javascript">

	var socket = new WebSocket('ws://'+window.location.host+'/websocket/room/socket?user={{.user}}')

	var display = function(event){
		$('#thread').append(tmpl('message_tmpl', {event: event}));
		$('#thread').scrollTo('max')
	}

	socket.onmessage = function(event){
		display(JSON.parse(event.data))
	}

	$('#send').click(function(e){
		var message = $('#message').val()
		$('#message').val('')
		socket.send(JSON.stringify(message))
	})

	$('#message').keypress(function(e){
		if(e.charCode == 13 || e.keyCode == 13){
			$('#send').click()
			e.preventDefault()
		}
	})
</script>

<% ●● %>: 構文
<%= ■■ %>: 出力

これは凄し