awsの統合ログ収集

awsでログ管理に必要なサービス
– CloudWatch Logs
– CloudTrail
– AWS Config

Splunkでログデータ運用を実現?
https://www.splunk.com
あああああああ、ログまわりだと、必ず機械学習が出てくるなー なんだかんだで。。

LogStorage For AWS

Purpose
The purpose of managing logs in the system is largely 1)monitoring, 2)management, and 3)auditing. 1)In monitoring, whether the system on AWS is operating normally, if the rules for network access are set properly, or if there is a security problem in the system on AWS, these will be continuously logged. It is checked by monitoring in real time. In addition, if there is any problem in the system, logs are also used to investigate the cause. 2) In management, the above monitoring points can be analyzed from the viewpoint of historical data, and trends in system usage and traffic can be analyzed and used for system expansion and change planning. 3)In audits, logs are important in terms of leaving a system operation trail of who did what and when.

AWSのけるログ管理
AWS上でのログ管理は、ログを収集・蓄積するAWSサービスが既に用意されており、そうしたサービスを利用してログを管理・モニタリングしていく
– AWSサービスに対する管理アクセスログ: AWS CloudTrail、 AWS Config → CloudWatch Logs, S3, SQS
– AWS上の通信トラフィックのログ: VPC Flow Logs, ELB Logs → CloudWatch Logs
– EC2インスタンス内のアクセスログ、アプリケーションログ、システムロ:AWS CloudWatch

なるほどー、肝心なのはCloudWatchやなー

Logstashを学ぼう

Input
While data is distributed across many systems in different formats, Logstash is a variety of input plug-ins that capture events from different sources simultaneously. You can easily, continuously and smoothly implement data injection from logs, metrics, webapplications, data stores, or various cloud services.

Filter ここは重要か?
Data analysis and transformation
As data is transferred from source to store, Logstash’s filter parses each event and identifies and structures the fields. It further transforms the data into a common format that is best for analysis that delivers business value.

– Extract structures from unstructured data using grok filters
– Get geographical information from IP address
– Anonymize personal information and completely exclude confidential information fields
– Easy processs data from any source, format or schema.
あ、なんかこれは凄そうだ。。

Output
Choose a storage location, and transfer data.
The ideal destination is Elasticsearch, but other destinations are of course also available. The search and analysis possibilities are not impeded.

Logstash allows you to specify various output destinations and transfer data freely. This high degree of flexibility allows you to work with many downstreams.

うん、サンプルをいじりたくなってきました。OKOK!

Laravel log

storate/logs/laravel.log

[2018-11-17 11:22:12] local.ERROR: SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes (SQL: alter table `users` add unique `users_email_unique`(`email`)) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 42000): SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes (SQL: alter table `users` add unique `users_email_unique`(`email`)) at /home/vagrant/local/zeus/vendor/laravel/framework/src/Illuminate/Database/Connection.php:664, PDOException(code: 42000): SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes at /home/vagrant/local/zeus/vendor/laravel/framework/src/Illuminate/Database/Connection.php:458)
[stacktrace]

httpd.conf error log

path: /etc/httpd/conf

ErrorLog logs/error_log

Set the log output destination in error_log of php.ini.
When describing the file name, write it with absolute path.

Centralized log service configuration settingss

Log type
The logs are not all the same, but the purpose are different depending on the type.
– application log
– accessibility log
– security log
– other

Overall view of log processing
– collect: AmazonS3, Amazon Kinesis, Amazon DynamoDB, Amazon RDS(Aurora)
– process: AWS Lambda, KCL Apps
– analyze: Amazon EMR
– save: Amazon Redshift, Amazon Machine Learning

Logs output from AWS services
Log specific to each environment such as OS and application