Category: Speech Recognition

話者データを学習し、どの人のデータなのか機械学習で予測する
Libraryにmatplotlib, numpy, scikit-learn, libROSA、
識別にSVM、フーリエ変換、MFCCなどを使う

SVM
L 境界線(直線・曲線)を作り、データがどちら側か判定する
　　L 境界線付近をサポートベクトルと呼ぶ

import numpy as np
import matplotlib.pyplot as plt
import random
from sklearn import svm

N = 15
random.seed(11813)
train_X = np.array([[random.randint(0,100), random.randint(0, 100)] for i in range(N)])

for i in range(len(train_X)):
	plt.plot(train_X[i][0], train_X[i][1],'o', color='blue')
	plt.annotate(i, (train_X[i][0], train_X[i][1]), size=20)
plt.savefig('test.jpg',dpi=100)

分類する

train_y = np.array([0 for i in range(N)])
train_y[2] = train_y[3] = train_y[1] = train_y[4] = train_y[5] = train_y[6] = train_y[11] = 1

colors = ['blue','red']
for i in range(len(train_X)):
	plt.plot(train_X[i][0], train_X[i][1],'o', color=colors[train_y[i]])
	plt.annotate(i, (train_X[i][0], train_X[i][1]), size=20)
plt.savefig('test.jpg',dpi=100)

test_X = np.array([[30,60]])

plt.plot(test_X[0][0], test_X[0][1],'x', color="black")
plt.annotate('test', (test_X[0][0], test_X[0][1]), size=20)

plt.savefig('test.jpg',dpi=100)

clf = svm.SVC(gamma=0.0001, C=1)
clf.fit(train_X, train_y)

test_y = clf.predict(test_X)
print(test_y)

なるほど、SVMが何をやってるのかは理解できた。

まずenglishのwave fileを用意します。

app.py

# -*- coding: utf-8 -*-
#! /usr/bin/python3

import librosa
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Tokenizer

audio, rate = librosa.load("sample.wav", sr = 16000)

tokenizer = Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

input_values = tokenizer(audio, return_tensors="pt").input_values

logits = model(input_values).logits

prediction = torch.argmax(logits, dim = -1)

transcription = tokenizer.batch_decode(prediction)[0]

print(transcription)

$ python3 app.py
// 省略
FOUR SCORE AND SEVEN YEARS AGO OUR FATHERS BROUGHT FORTH ON THIS CONTINENT A NEW NATION CONCEIVED IN LIBERTY AND DEDICATED TO THE PROPOSITION THAT ALL MEN ARE CREATED EQUAL

なるほど、これは凄い
tokenizerとmodelを日本語でやりたい

from asrecognition import ASREngine

asr = ASREngine("ja", model_path="jonatasgrosman/wav2vec2-large-xlsr-53-japanese")

audio_paths = ["itou.wav"]
transcriptions = asr.transcribe(audio_paths)

$ python3 app.py
/home/vagrant/.local/lib/python3.8/site-packages/transformers/configuration_utils.py:340: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
warnings.warn(
Killed

うむ、うまくいかないが近づいている様な気がする
vagrantだとうまくいかないので、vpsでやることにした

micによるliveだと、こちらで出来ますね
https://github.com/oliverguhr/wav2vec2-live

最近のトレンドはtransformarを用いた自然言語処理とのことで、wav2vecを使いたい

### ライブラリのinstall
$ pip3 install transformers datasets librosa

main.py

# -*- coding: utf-8 -*-
#! /usr/bin/python3

import librosa
import matplotlib.pyplot as plt
from IPython.display import display, Audio
import librosa.display
import numpy as np 

import torch
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from datasets import load_dataset
import soundfile as sf

processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

def map_to_array(batch):
	speech, sr_db = sf.read(batch["file"])
	batch["speech"] = speech
	batch['sr_db'] = sr_db
	return batch

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy","clean",split="validation")
ds = ds.map(map_to_array)

librosa.display.waveplot(np.array(ds['speech'][0]), sr=ds['sr_db'][0])
plt.savefig('01')
display(Audio(np.array(ds['speech'][0]), rate=ds['sr_db'][0]))

input_values = processor(ds["speech"][0], return_tensors="pt").input_values
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)

transcription = processor.decode(predicted_ids[0])

print(transcription)

$ python3 main.py
// 省略
A MAN SAID TO THE UNIVERSE SIR I EXIST

なんだこれええええええええええええええええええ
音声をvector graphicにしてるのはわかるが、、、

辞書ファイルを作成するには、「語彙」「音素」「構文」が必要

語彙: Juliusに認識させたい単語を定義
音素: 語彙の読みを定義
構文: 認識する文章の構成を定義

### 読みファイルの作成
/julius/dict/greeting.yomi
L 平仮名で作成

おはよう	おはよう
ございます ございます
こんにちは こんにちわ
こんばんは こんばんわ

### 音素ファイルの作成
$ iconv -f utf8 -t eucjp dict/greeting.yomi | gramtools/yomi2voca/yomi2voca.pl | iconv -f eucjp -t utf8 > dict/greeting.phone

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = “ja_JP.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).

どうやらjuliusはeucではなくutf8になってるらしい

$ iconv dict/greeting.yomi | gramtools/yomi2voca/yomi2voca.pl > dict/greeting.phone

/julius/dict/greeting.phone

おはよう	o h a y o u
ございます	g o z a i m a s u
こんにちは	k o N n i ch i w a
こんばんは	k o N b a N w a

### 構文ファイルの作成
greeting.grammar
L NS_Bが文書の開始、NS_Eが文章の終了　2行目以降がGREETで認識される読みの文字列

S : NS_B GREET NS_E
GREET : OHAYOU
GREET : OHAYOU GOZAIMASU
GREET : KONNICHIWA
GREET : KONBANWA

### 語彙ファイルの作成
各音素に対して、構文で割り振った読みと、開始終了を表すNS_B、NS_Eを設定

% OHAYOU
おはよう	o h a y o u
% GOZAIMASU
ございます	g o z a i m a s u
% KONNICHIWA
こんにちは	k o N n i ch i w a
% KONBANWA
こんばんは	k o N b a N w a
% NS_B
s		silB
% NS_E
/s	silE

辞書ファイルを作成する
$ cp -b gramtools/mkdfa/mkfa-1.44-flex/mkfa gramtools/mkdfa/mkfa
$ cp -b gramtools/dfa_minimize/dfa_minimize gramtools/mkdfa/dfa_minimize
$ cp -b gramtools/dfa_determinize/dfa_determinize gramtools/mkdfa/dfa_determinize

$ gramtools/mkdfa/mkdfa.pl dict/greeting
// 省略
now reversing dict/greeting.dfa into NFA “dict/greeting.dfa.forward_nfa”
executing [gramtools/mkdfa/dfa_determinize dict/greeting.dfa.forward_nfa -o dict/greeting.dfa.forward_beforeminimize]
6 categories, 6 nodes, 7 arcs
-> determinized: 5 nodes, 7 arcs
executing [gramtools/mkdfa/dfa_minimize dict/greeting.dfa.forward_beforeminimize -o dict/greeting.dfa.forward]
6 categories, 5 nodes, 7 arcs
-> minimized: 5 nodes, 7 arcs
—
generated: dict/greeting.dfa dict/greeting.term dict/greeting.dict dict/greeting.dfa.forward

### 辞書を指定
$ julius/julius -C ../ja_model/am-gmm.jconf -nostrip -gram dict/greeting -input mic

なるほど、辞書の作り方はOK
次はGoやな

まず中国語の音声ファイルを用意します

続いてDeepspeechの中国語モデルをDLします。
deepspeech-0.9.3-models-zh-CN.pbmm
deepspeech-0.9.3-models-zh-CN.scorer

実行は、Englishと同様
$ source deepspeech-venv/bin/activate
$ deepspeech –model deepspeech-0.9.3-models-zh-CN.pbmm –scorer deepspeech-0.9.3-models-zh-CN.scorer –audio audio/zh_test.wav
Loading model from file deepspeech-0.9.3-models-zh-CN.pbmm
TensorFlow: v2.3.0-6-g23ad988
DeepSpeech: v0.9.3-0-gf2e9c85
2021-09-04 02:47:32.705419: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Loaded model in 0.0436s.
Loading scorer from files deepspeech-0.9.3-models-zh-CN.scorer
Loaded scorer in 0.00114s.
Running inference.
同的祖母是一位佛教徒但他从二没有在未向前年国佛经有一天他前我是菜里杨聪认我在我一八年气派来结果动的管领流泪总合了他说乔丽多难受到这一个密绝大这么起就有机
Inference took 12.015s for 25.003s audio file.

ほう、何でもできるような気がしてきた。
まあ設計次第かな。