vi elasticsearch.yml

[vagrant@localhost ~]$ sudo ls /etc/elasticsearch
elasticsearch.keystore jvm.options role_mapping.yml users
elasticsearch.yml log4j2.properties roles.yml users_roles
[vagrant@localhost ~]$ sudo cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ———————————- Cluster ———————————–
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ———————————— Node ————————————
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ———————————– Paths ————————————
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ———————————– Memory ———————————–
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ———————————- Network ———————————–
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# ——————————— Discovery ———————————-
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is [“127.0.0.1”, “[::1]”]
#
#discovery.seed_hosts: [“host1”, “host2”]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: [“node-1”, “node-2”]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ———————————- Gateway ———————————–
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ———————————- Various ———————————–
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

Set IP where ElasticSearch is installed
#network.host: 192.168.0.1
→ 192.168.35.10

#http.port: 9200
→ http.port: 8000

[vagrant@localhost ~]$ sudo service elasticsearch restart
elasticsearch を停止中: [失敗]
elasticsearch を起動中: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[ OK ]

install selenium

[vagrant@localhost ~]$ python3
Python 3.5.2 (default, Jul 28 2018, 11:25:01)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-23)] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>>

[vagrant@localhost ~]$ pip install selenium
Requirement already satisfied: selenium in ./.pyenv/versions/3.5.2/lib/python3.5/site-packages (3.13.0)
You are using pip version 18.0, however version 19.1.1 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.

1
2
3
4
5
6
7
8
9
10
11
12
from selenium import webdriver
from selenium.webdriver.chrome.options import options
 
options = Options()
 
options.add_argument('--headless')
options.add_argument('--no-sandbox')
 
driver = webdriver.Chrome(executable_path='/path/to/chromedriver',chrome_options=options)
 
driver.get('https://yahoo.co.jp')
print(driver.title)

Canvas

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<!DOCTYPE html>
<html>
    <head>
        <title>canvas tutorial</title>
        <style>
            #canvas {
                background: #666;
            }
        </style>
    </head>
    <body>
        <canvas id="canvas" width="640" height="480"></canvas>
        <script>
            var canvas = document.getElementById("canvas");
        </script>
    </body>
</html>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<!DOCTYPE html>
<html>
    <head>
        <title>canvas tutorial</title>
        <style>
            #canvas {
                background: #666;
            }
        </style>
    </head>
    <body>
        <canvas id="canvas" width="640" height="480"></canvas>
        <script>
            var canvas = document.getElementById("canvas");
            function onDown(e){
                console.log("down");
            }
            function onUp(e){
                console.log("up");
            }
            function onClick(e){
                console.log("click");
            }
            function onOver(e){
                console.log("mouseover");
            }
            function onOut(){
                console.log("mouseout");
            }
 
            canvas.addEventListener('mousedown', onDown, false);
            canvas.addEventListener('mouseup', onUp, false);
            canvas.addEventListener('click', onClick, false);
            canvas.addEventListener('mouseover', onOver, false);
            canvas.addEventListener('mouseout', onOut, false);
        </script>
    </body>
</html>

Install ChromeDriver and Fonts

Prepare for install
yum -y install libX11 GConf2 fontconfig
[root@localhost test]# wget https://chromedriver.storage.googleapis.com/73.0.3683.68/chromedriver_linux64.zip
–2019-05-24 09:20:33– https://chromedriver.storage.googleapis.com/73.0.3683.68/chromedriver_linux64.zip
chromedriver.storage.googleapis.com をDNSに問いあわせています… 216.58.197.208, 2404:6800:4004:80e::2010
chromedriver.storage.googleapis.com|216.58.197.208|:443 に接続しています… 接続しました。
HTTP による接続要求を送信しました、応答を待っています… 200 OK
長さ: 5008482 (4.8M) [application/zip]
`chromedriver_linux64.zip’ に保存中

100%[======================================>] 5,008,482 1.67M/s 時間 2.9s

2019-05-24 09:20:36 (1.67 MB/s) – `chromedriver_linux64.zip’ へ保存完了 [5008482/5008482]

[root@localhost test]# ls
chromeTest.js chromedriver_linux64.zip index.php node_modules selenium-test
[root@localhost test]# unzip chromedriver_linux64.zip
Archive: chromedriver_linux64.zip
inflating: chromedriver
[root@localhost test]# mv chromedriver /usr/local/bin/
[root@localhost test]# chmod 755 /usr/local/bin/chromedriver

要求: libatspi.so.0()(64bit)

[root@localhost test]# sudo yum whatprovides libatspi.so.0
読み込んだプラグイン:fastestmirror
Repository nodesource is listed more than once in the configuration
Repository nodesource-source is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: ftp.nara.wide.ad.jp
* extras: ftp.nara.wide.ad.jp
* remi-safe: ftp.riken.jp
* updates: ftp.nara.wide.ad.jp
警告: yum のバージョン 3.0.x は、誤ってファイル名に対して一致するでしょう。
その振る舞いをするには「*/libatspi.so.0」もしくは「*bin/libatspi.so.0」が使用できます。
No Matches found

SyntaxError: Use of const in strict mode.

JavaScript has introduced const and let in ECMAScript 2015, which is relatively recent. This is enabled by default in Node.js since Node.js 4.x.

$ git clone git://github.com/creationix/nvm.git ~/.nvm
$ source ~/.nvm/nvm.sh
$ nvm help
$ nvm ls-remote
[root@localhost vagrant]# nvm install v8.16.0
Downloading and installing node v8.16.0…
Downloading https://nodejs.org/dist/v8.16.0/node-v8.16.0-linux-x64.tar.xz…
######################################################################## 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v8.16.0 (npm v6.4.1)
[root@localhost vagrant]# node -v
v8.16.0

[root@localhost test]# node chromeTest.js
/home/vagrant/local/app/test/chromeTest.js:3
var By = webdriver.By;
^

ReferenceError: webdriver is not defined
at Object. (/home/vagrant/local/app/test/chromeTest.js:3:10)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Function.Module.runMain (module.js:694:10)
at startup (bootstrap_node.js:204:16)
at bootstrap_node.js:625:3

What…

install selenium

$ npm -g install selenium-webdriver
$ npm install selenium-webdriver

https://sites.google.com/a/chromium.org/chromedriver/downloads

sudo yum install -y google-chrome-unstable libOSMesa google-noto-cjk-fonts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
var webdrive = require('selenium-webdriver');
var driver;
var By = webdriver.By;
 
driver = new webdriver.Builder()
.withCapabilities(webdriver.Capabilities.chrome())
.build();
 
driver.get('https://www.google.co.jp/').then(function(){
    driver.findElement(By.id('hplog')).click()
    .then(function(){
        console.log('clicked logo');
    });
});

[vagrant@localhost test]$ node chromeTest.js

/home/vagrant/local/app/test/node_modules/selenium-webdriver/index.js:25
const _http = require(‘./http’);
^^^^^
SyntaxError: Use of const in strict mode.
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/home/vagrant/local/app/test/chromeTest.js:1:78)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)

Error: EACCES

[vagrant@localhost test]$ sudo chmod 777 /usr/local/lib;

[vagrant@localhost test]$ sudo npm install -g bower
npm http GET https://registry.npmjs.org/bower
npm http 200 https://registry.npmjs.org/bower
npm WARN deprecated bower@1.8.8: We don’t recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/
npm http GET https://registry.npmjs.org/bower/-/bower-1.8.8.tgz
npm http 200 https://registry.npmjs.org/bower/-/bower-1.8.8.tgz
/usr/bin/bower -> /usr/lib/node_modules/bower/bin/bower
npm WARN unmet dependency /usr/lib/node_modules/block-stream requires inherits@’~2.0.0′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /usr/lib/node_modules/fstream requires inherits@’~2.0.0′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /usr/lib/node_modules/fstream-ignore requires inherits@’2′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /usr/lib/node_modules/fstream-npm requires inherits@’2′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /usr/lib/node_modules/glob requires inherits@’2′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /usr/lib/node_modules/npmconf requires inherits@’~2.0.0′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency /usr/lib/node_modules/tar requires inherits@’2′ but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
bower@1.8.8 /usr/lib/node_modules/bower

Node.jp update

[vagrant@localhost test]$ node –version
v0.10.48
[vagrant@localhost test]$ npm –version
1.3.6
[vagrant@localhost test]$ n –latest
-bash: n: コマンドが見つかりません

There is a handy version control tool, n, which lets you check and update versions.

[vagrant@localhost test]$ npm install -g n
npm http GET https://registry.npmjs.org/n
npm http GET https://registry.npmjs.org/n
npm http GET https://registry.npmjs.org/n
npm ERR! Error: CERT_UNTRUSTED
npm ERR! at SecurePair. (tls.js:1430:32)
npm ERR! at SecurePair.emit (events.js:92:17)
npm ERR! at SecurePair.maybeInitFinished (tls.js:1029:10)
npm ERR! at CleartextStream.read [as _read] (tls.js:521:13)
npm ERR! at CleartextStream.Readable.read (_stream_readable.js:341:10)
npm ERR! at EncryptedStream.write [as _write] (tls.js:418:25)
npm ERR! at doWrite (_stream_writable.js:226:10)
npm ERR! at writeOrBuffer (_stream_writable.js:216:5)
npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:183:11)
npm ERR! at write (_stream_readable.js:602:24)
npm ERR! If you need help, you may report this log at:
npm ERR!
npm ERR! or email it to:
npm ERR!

npm ERR! System Linux 2.6.32-754.3.5.el6.x86_64
npm ERR! command “node” “/usr/bin/npm” “install” “-g” “n”
npm ERR! cwd /home/vagrant/local/app/test
npm ERR! node -v v0.10.48
npm ERR! npm -v 1.3.6
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/vagrant/local/app/test/npm-debug.log
npm ERR! not ok code 0

The reason is that SSL key validation is performed on the https registry, so once this setting is set to false, npm install will succeed.

vagrant@localhost test]$ npm config set strict-ssl false

[vagrant@localhost test]$ npm install -g n
npm http GET https://registry.npmjs.org/n
npm http 200 https://registry.npmjs.org/n
npm http GET https://registry.npmjs.org/n/-/n-4.1.0.tgz
npm http 200 https://registry.npmjs.org/n/-/n-4.1.0.tgz
npm ERR! Error: EACCES, mkdir ‘/usr/lib/node_modules/n’
npm ERR! { [Error: EACCES, mkdir ‘/usr/lib/node_modules/n’]
npm ERR! errno: 3,
npm ERR! code: ‘EACCES’,
npm ERR! path: ‘/usr/lib/node_modules/n’,
npm ERR! fstream_type: ‘Directory’,
npm ERR! fstream_path: ‘/usr/lib/node_modules/n’,
npm ERR! fstream_class: ‘DirWriter’,
npm ERR! fstream_stack:
npm ERR! [ ‘/usr/lib/node_modules/fstream/lib/dir-writer.js:36:23’,
npm ERR! ‘/usr/lib/node_modules/mkdirp/index.js:37:53’,
npm ERR! ‘Object.oncomplete (fs.js:108:15)’ ] }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! System Linux 2.6.32-754.3.5.el6.x86_64
npm ERR! command “node” “/usr/bin/npm” “install” “-g” “n”
npm ERR! cwd /home/vagrant/local/app/test
npm ERR! node -v v0.10.48
npm ERR! npm -v 1.3.6
npm ERR! path /usr/lib/node_modules/n
npm ERR! fstream_path /usr/lib/node_modules/n
npm ERR! fstream_type Directory
npm ERR! fstream_class DirWriter
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! stack Error: EACCES, mkdir ‘/usr/lib/node_modules/n’
npm ERR! fstream_stack /usr/lib/node_modules/fstream/lib/dir-writer.js:36:23
npm ERR! fstream_stack /usr/lib/node_modules/mkdirp/index.js:37:53
npm ERR! fstream_stack Object.oncomplete (fs.js:108:15)
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/vagrant/local/app/test/npm-debug.log
npm ERR! not ok code 0

This time with a different error, it says “Error: EACCES”.

About PhantomJS

WebKit based headless browser.
Webkit is a rendering engine mainly used in web browsers. PhantomJS uses a JavaScript engine called JavaScriptCore (which also uses safari) which is built-in Web-kit.

An environment in which a browser can be executed by commands programs.
Because the program can control the movement, PhantomJS is good at
– Automation test
– Scraping

In automated testing, Selenium + PhantomJS often come out as a set.