td agent execution log

$ /var/log/td-agent/td-agent.log
~ omission  ~
013-11-16 14:21:22 +0900 [info]: adding source type="forward"
2013-11-16 14:21:22 +0900 [info]: adding source type="http"
2013-11-16 14:21:22 +0900 [info]: adding source type="debug_agent"
2013-11-16 14:21:22 +0900 [info]: adding source type="tail"
2013-11-16 14:21:22 +0900 [warn]: 'pos_file PATH' parameter is not set to a 'tail' source.
2013-11-16 14:21:22 +0900 [warn]: this parameter is highly recommended to save the position to resume tailing.
2013-11-16 14:21:22 +0900 [info]: adding match pattern="td.*.*" type="tdlog"
2013-11-16 14:21:22 +0900 [info]: adding match pattern="debug.**" type="stdout"
2013-11-16 14:21:22 +0900 [info]: adding match pattern="mongo.**" type="mongo"
2013-11-16 14:21:22 +0900 [info]: adding match pattern="apache.access" type="file"
2013-11-16 14:21:22 +0900 [error]: unexpected error error_class=Mongo::ConnectionFailure error=#
  2013-11-16 14:21:22 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/networking.rb:167:in `receive_header'
~ omission  ~

/var/log/lsyncd/lsyncd.status

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/lsyncd/lsyncd.status
Lsyncd status report at Tue Mar 12 00:44:51 2019

Sync1 source=/tmp/origin/
There are 0 delays
Excluding:
  nothing.


Inotify watching 1 directories
  1: /tmp/origin/

It doesn’t seem to be an lsync execution log.

What is /var/log/messages described?

The various logs output according to the system status are output to the following log files.

Log file name contents
/var/log/messages: general system related messages
/var/log/cron: messages about proccessing results that are executed periodically
/var/log/maillog: messages about mail
/var/log/spooler: printing messasges
/var/log/boot.log: Message about OS boot

Therefore, “messages” output general system related messages.

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/messages
May 29 08:22:59 localhost kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 29 08:22:59 localhost kernel: ata3.00: failed command: FLUSH CACHE
May 29 08:22:59 localhost kernel: ata3.00: cmd e7/00:00:00:00:00/00:00:00:00:00/a0 tag 7
May 29 08:22:59 localhost kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
May 29 08:22:59 localhost kernel: ata3.00: status: { DRDY }
May 29 08:22:59 localhost kernel: ata3: hard resetting link
May 29 08:22:59 localhost kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 29 08:22:59 localhost kernel: ata3.00: configured for UDMA/133
May 29 08:22:59 localhost kernel: ata3.00: device reported invalid CHS sector 0
May 29 08:22:59 localhost kernel: ata3: EH complete
[vagrant@localhost ~]$ sudo tail -n 10 /var/log/messages-20190526
May 25 12:24:02 localhost /etc/init.d/kibana: Attempting ‘restart’ on kibana
May 25 12:24:02 localhost /etc/init.d/kibana: Killing kibana (pid 10114) with SIGTERM
May 25 12:24:02 localhost /etc/init.d/kibana: Waiting kibana (pid 10114) to die…
May 25 12:24:03 localhost /etc/init.d/kibana: Waiting kibana (pid 10114) to die…
May 25 12:24:03 localhost /etc/init.d/kibana: kibana stopped.
May 25 12:24:03 localhost /etc/init.d/kibana: kibana started
May 25 12:36:01 localhost dhclient[963]: DHCPREQUEST on eth0 to 10.0.2.2 port 67 (xid=0x5c1ebdd1)
May 25 12:36:01 localhost dhclient[963]: DHCPACK from 10.0.2.2 (xid=0x5c1ebdd1)
May 25 12:36:03 localhost dhclient[963]: bound to 10.0.2.15 — renewal in 39825 seconds.
May 26 03:27:07 localhost rsyslogd: [origin software=”rsyslogd” swVersion=”5.8.10″ x-pid=”1170″ x-info=”http://www.rsyslog.com”] rsyslogd was HUPed

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/secure
May 27 22:14:51 localhost sshd[12653]: pam_unix(sshd:session): session closed for user vagrant
May 29 08:31:19 localhost sshd[13390]: Accepted password for vagrant from 192.168.35.1 port 64022 ssh2
May 29 08:31:19 localhost sshd[13390]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
May 29 08:31:19 localhost sshd[13392]: subsystem request for sftp
May 29 08:50:13 localhost sshd[13400]: Accepted password for vagrant from 192.168.35.1 port 49488 ssh2
May 29 08:50:13 localhost sshd[13400]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
May 29 08:51:32 localhost sudo: vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/usr/bin/tail -n 10 /var/log/messages
May 29 08:52:25 localhost sudo: vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/usr/bin/tail -n 10 /var/log/messages-20190526
May 29 08:58:59 localhost sudo: vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/usr/bin/tail -n /var/log/secure
May 29 08:59:23 localhost sudo: vagrant : TTY=pts/0 ; PWD=/home/vagrant ; USER=root ; COMMAND=/usr/bin/tail -n 10 /var/log/secure

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/cron
May 29 03:22:02 localhost anacron[13128]: Normal exit (1 job run)
May 29 04:01:01 localhost CROND[13336]: (root) CMD (run-parts /etc/cron.hourly)
May 29 04:01:01 localhost run-parts(/etc/cron.hourly)[13336]: starting 0anacron
May 29 04:01:01 localhost run-parts(/etc/cron.hourly)[13345]: finished 0anacron
May 29 06:01:02 localhost CROND[13347]: (root) CMD (run-parts /etc/cron.hourly)
May 29 06:01:02 localhost run-parts(/etc/cron.hourly)[13347]: starting 0anacron
May 29 06:01:02 localhost run-parts(/etc/cron.hourly)[13356]: finished 0anacron
May 29 07:01:02 localhost CROND[13372]: (root) CMD (run-parts /etc/cron.hourly)
May 29 07:01:03 localhost run-parts(/etc/cron.hourly)[13372]: starting 0anacron
May 29 07:01:03 localhost run-parts(/etc/cron.hourly)[13381]: finished 0anacron

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/maillog

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/spooler

[vagrant@localhost ~]$ sudo tail -n 10 /var/log/boot.log
Starting httpd: httpd: Could not reliably determine the server’s fully qualified domain name, using localhost.localdomain for ServerName
[ OK ]
Starting crond: [ OK ]
Starting Jenkins [ OK ]
May 19 07:16:49 localhost NET[12444]: /sbin/dhclient-script : updated /etc/resolv.conf
May 20 20:09:00 localhost NET[14514]: /sbin/dhclient-script : updated /etc/resolv.conf
May 22 08:31:26 localhost NET[14936]: /sbin/dhclient-script : updated /etc/resolv.conf
May 24 19:43:41 localhost NET[6409]: /sbin/dhclient-script : updated /etc/resolv.conf
May 26 13:02:39 localhost NET[12459]: /sbin/dhclient-script : updated /etc/resolv.conf
May 29 03:04:33 localhost NET[13192]: /sbin/dhclient-script : updated /etc/resolv.conf

Well, I absolutely understood that the output is completely different depending on the log file.

LogLevel’s Error Log

It is possible to specify up to 8 levels in the error log up to which level errors should be recorded. Set to “LogLevel” to specify.

Level: Meaning
emerg: Serious error that the server can not run.
alert: an error more serious than crit
crit: serious error
error: error
warn: warning
notice: notification message
info: server information
debug: information debugging

Is this basically something to do when look at “crit” and “error”?

zabbix monitoring log

When I read the official document of Zabbix, /var/log/syslog,error is described, so it seems not in monitoring log.

I can not find it even if I look at the directory of /var/log/. What is it?

Perl session

#! /usr/local/bin/perl

use strict;
use warnings;

# 環境依存
use lib "$ENV{DOCUMENT_ROOT}/lib/perl5";
use lib "$ENV{DOCUMENT_ROOT}/lib";
use lib "./lib";

use CGI qw|:standard|;
use CGI::Session;
use Password;

my $title = "セッションID管理テスト";
my $expire = "+1h";
my $encode = "UTF-8";

# データベースを使うところを省略
my %User ('hoge'=> {pass => q|$1$hEeN3T%+$CRKHRxko1cWGNjE69mTNw.|});

my $cgi = new CGI;
print $cgi->redirect( -uri=>'http://'.$ENV{SERVER_NAME}.$ENV{SCRIPT_NAME}, -status=>301) and exit if $cgi->param('CGISSID');

my $sid = $cgi->cookie('CGISESSID') || undef;

my $session = CGI::Session->load(undef, $sid, {Directory=>'./data'}) or die CGI:Session->errstr();
Error("Your session time out! Refresh the screen to start new session!") if $session->is_expired;
$session->expire($expire); #有効期限の設定
#$session->expire('+1m');

if($session-> is_empty){
	$session = $session->new(undef, $sid, {Directory=>'./data'}) or die $session->errstr;
} #取得したセッションidが有効ならそのまま、無効なら別のidを発番

my %param = $cgi->Vars();
my @message;

if ($session){
	if (my $action = $param{'action'}){
		push @message, forget() if $action eq 'forget me'; #　セッションの削除依頼
		push @message, loguout() if $action eq 'logout';  #　セッション内のログインステータスを初期化
	}

	$session->save_param($cgi); # 入力値をセッション内に保存
	$session->clear('pass'); #パスワードの平文保存を回避
	push @message, login($session->param('username'), $param{'pass'}) if $param{'action'} and $param{'action'} eq 'login';
	push @message, $session->param('username')? confirm(): ask();
} else {
	push @message, ask();
}

print $session->header ( -charset => $encode),
	start_html( -title => $title, -encoding => $encode, -lang=>'ja'),
	@message,
	a({href=>$ENV{"SCRIPT_NAME"}}, '戻る',),
	end_html(),
;

sub Error {
	my $msg = shift;

	print $session->header( -charset => encode),
	start_html( -title => "エラー / " . $title, -encoding => $encode, -lang => 'ja'),
	h1("エラー"),hr();
	p(strong($msg)),hr(),
	a({href=>$ENV{"SCRIPT_NAME"}}, '戻る',),
	end_html(),
};
exit;
}

sub forget {
	$session->clear(['username','firstname','lastname','like','action','login']);
	$session->close;
	$session->delete;
	return h2("we've forgotten you!");
}

setting virtual host

# sudo cat /etc/httpd/conf.d/hoge.conf
<VirtualHost *:80>
DocumentRoot /var/www/html/tv
DirectoryIndex index.html index.php
ServerName www.hoge
ServerAlias on-tv.fun
CustomLog logs/hoge-access.log common
ErrorLog  logs/hoge-error.log
AddDefaultCharset UTF-8
<Directory "/var/www/html/tv/">
AllowOverride All
</Directory>
</VirtualHost>

# /etc/init.d/httpd graceful

Not reflected unless restart.
Also modify google developer console.

cpanm Net::Amazon::S3@0.86

[vagrant@localhost ~]$ cpan App::cpanminus
[vagrant@localhost ~]$ sudo yum install -y openssl openssl-devel
パッケージ openssl-1.0.1e-57.el6.x86_64 はインストール済みか最新バージョンです
パッケージ openssl-devel-1.0.1e-57.el6.x86_64 はインストール済みか最新バージョンです

[vagrant@localhost ~]$ cpanm LWP::Protocol::https
-bash: cpanm: コマンドが見つかりません
[vagrant@localhost ~]$ /usr/local/bin/cpanm LWP::Protocol::https
-bash: /usr/local/bin/cpanm: そのようなファイルやディレクトリはありません

Log collection

First of all, middleware for log collection can be divided into Push type and Pull type according to the data acquisition method.

Systems that transfer logs to a destination for which middleware is configured are classified as Push. Famously speaking, Apache Flume, Fluent, Scribe, Logstash etc are classified as Push type.

On the other hand, it is Pull type that the client side explicitly specifies and acquires necessary data for middleware that centrally manages logs. Apache Kafka and so-called PubSub models such as Google Cloud Pub/Sub and Kinesis Stream, which are not middleware, are classified as Pull types.

—
Source => Push type middleware that receives records. Corresponds to input in Fluentd.
Channel => An area to save until transmission of a record is completed in Push type middleware. Equivalent to buffer in Fluentd.
Sink => Push destination of record in Push type middleware. Equivalent to output in Fluentd.
Upstream => The side that sends the record.
Downstream => The side that receives the record.