Log collection

First of all, middleware for log collection can be divided into Push type and Pull type according to the data acquisition method.

Systems that transfer logs to a destination for which middleware is configured are classified as Push. Famously speaking, Apache Flume, Fluent, Scribe, Logstash etc are classified as Push type.

On the other hand, it is Pull type that the client side explicitly specifies and acquires necessary data for middleware that centrally manages logs. Apache Kafka and so-called PubSub models such as Google Cloud Pub/Sub and Kinesis Stream, which are not middleware, are classified as Pull types.

Source => Push type middleware that receives records. Corresponds to input in Fluentd.
Channel => An area to save until transmission of a record is completed in Push type middleware. Equivalent to buffer in Fluentd.
Sink => Push destination of record in Push type middleware. Equivalent to output in Fluentd.
Upstream => The side that sends the record.
Downstream => The side that receives the record.


Highly scalable full-text search engine developed by Elastic
Enables various analysis such as real time data analysis, log analysis, full text search
Often used with log aggregation Logstash and fluentd, and visualization tools kibana
Searching across multiple databases is provided as a common use

– Strong search performance and scalability
– Indexes are often separated by time intervals
– Throw away indexes that are no longer needed
– Store data for analysis and search

DB -> index
table -> mapping type
column -> field
record -> document

Mapping: Setting of field type and analysis method
Analysis: About processing of field values such as language processing and normalization
Query DSL: Assembling search condition in JSON format


wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
tar -xzf elasticsearch-6.4.0.tar.gz


cd elasticsearch-6.4.0

[vagrant@localhost elasticsearch-6.4.0]$ bin/elasticsearch-plugin install analysis-kuromoji
-> Downloading analysis-kuromoji from elastic
[=================================================] 100%  
-> Installed analysis-kuromoji

[vagrant@localhost elasticsearch-6.4.0]$ bin/elasticsearch
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error=’Cannot allocate memory’ (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# logs/hs_err_pid17300.log