EFK版本信息
- Fluentd: 在线安装
- Elasticsearch: 6.4.1
- Kibana: 6.4.1
Fluentd
在线安装
1.安装
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
2.查看状态
/etc/init.d/td-agent status
3.安装插件
td-agent-gem install fluent-plugin-elasticsearch
td-agent-gem install fluent-plugin-typecast
td-agent-gem install fluent-plugin-secure-forward
离线安装
在下载页面中根据操作系统版本选择相应rpm包
- 安装依赖
- redhat-lsb-core
- redhat-lsb-core会有一堆依赖包,建议还是在线安装
Elasticsearch
处理依赖
安装Java8
从2.1版本后依赖Java8, 下载地址,需要在机器上安装。
- 卸载默认版本Java
rpm -qa | grep -i java | xargs rpm -e --nodeps
- 安装
rpm -ivh jdk-8u181-linux-x64.rpm
修改可以虚拟内存大小
修改配置文件/etc/sysctl.conf
,增加vm.max_map_count=262144
,执行sysctl -p
使之生效
或者执行
sysctl -w vm.max_map_count=262144
非root用户运行
elastaicsearch不允许使用root用户运行,我们新建一个 es
用户来运行。
- 创建用户组
groupadd es
- 创建用户
useradd -g es -m es
CentOS6 需要添加以下两行
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
离线安装
- 1.下载
cd /tmp && curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.1.tar.gz
- 2.设置文件句柄大小
cat >>/etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 16384
* hard nproc 16384
EOF
- 3.解压运行
scp /tmp/elasticsearch-6.4.1.tar.gz /home/es
su - es
tar zxvf elasticsearch-6.4.1.tar.gz
cd elasticsearch-6.4.1/
sed -ie 's/#network.host:.*/network.host: 0.0.0.0/' config/elasticsearch.yml
./bin/elasticsearch
Kibana
离线安装
1.下载
cd /tmp && curl -O https://artifacts.elastic.co/downloads/kibana/kibana-6.4.1-linux-x86_64.tar.gz
2.解压运行
tar zxvf kibana-6.4.1-linux-x86_64.tar.gz
cd kibana-6.4.1-linux-x86_64
sed -ie 's/#server.host:.*/server.host: "0.0.0.0"/' config/kibana.yml
./bin/kibana
端口信息
- 5601: kibana web接口
- 9200: elasticsearch JSON接口
- 24224: fluentd端口
整体测试
- 1.nginx设置日志格式
log_format main ‘$remote_addr $http_host [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"’;
- 2.准备测试文件
touch /tmp/access.log
chmod 666 /tmp/access.log
mkdir -p /var/log/td-agent/access/
touch /var/log/td-agent/access/access.log.pos
chmod 666 /var/log/td-agent/access/access.log.pos
- 3.fluentd配置采集
需要注意path
权限问题
cat >>/etc/td-agent/td-agent.conf <<EOF
<source>
@type tail
path /tmp/access.log
pos_file /var/log/td-agent/access/access.log.pos
tag nginx.access
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<other>[^ ]*)"$/
time_format %d/%b/%Y:%H:%M:%S %z
</source>
<match nginx.access>
@type elasticsearch
host localhost
port 9200
flush_interval 2s
buffer_queue_limit 4096
buffer_chunk_limit 1024m
num_threads 4
logstash_format true
</match>
EOF
- 4.写入日志进行测试
cat >>/tmp/access.log <<EOF
10.10.110.41 - - [11/Sep/2018:14:50:46 +0800] "POST /auth HTTP/1.1" 202 97 "http://10.10.20.98/login.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
10.10.110.41 - - [11/Sep/2018:14:50:58 +0800] "GET /auth/ HTTP/1.1" 200 424 "http://10.10.20.98/login.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
EOF
或者实时从Nginx中读取日志
tail -n 2 /home/sendoh/qdata-cloud/logs/nginx/access.log >> /tmp/access.log
- 5.查询json格式
curl http://10.10.20.98:9200/_search?pretty
- 6.kibana页面访问
Discover
supervisor配置
直接启动的方式程序容易挂掉,建议使用supervisor来对Elasticsearch
, Kibana
服务进行管理
- 安装supervisor
pip install supervisor
- supervisor默认配置
mkdir -p /etc/conf.d
vi /etc/supervisord.conf
注意:supervisor默认使用文件句柄大小是1024,对启动elasticsearch
来说还是太小了,需要进行修改
; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
; - Shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".
; - Comments must have a leading space: "a=b ;comment" not "a=b;comment".
[unix_http_server]
file=/tmp/sendoh_supervisor.sock ; (the path to the socket file)
;chmod=0700 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
;[inet_http_server] ; inet (TCP) server disabled by default
;port=0.0.0.0:9002 ; (ip_address:port specifier, *:port for all iface)
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/sendoh_supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=65536
minprocs=32768
;umask=022 ; (process file creation umask;default 022)
user=root ; (default is current user, required if root)
;identifier=supervisor ; (supervisord identifier, default is 'supervisor')
;directory=/tmp ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
;environment=KEY="value" ; (key value pairs to add to environment)
;strip_ansi=false ; (strip ansi escape codes in logs; def. false)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/sendoh_supervisor.sock ; use a unix:// URL for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available
[include]
files = conf.d/*.conf
- EFK配置
vi /etc/conf.d/supervisor_efk.conf
[program:elasticsearch]
command=/home/es/elasticsearch-6.4.1/bin/elasticsearch
user=es
minfds=65536
minprocs=32768
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/tmp/elasticsearch.log
stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
directory=/home/es/elasticsearch-6.4.1
[program:kibana]
command=/tmp/kibana-6.4.1-linux-x86_64/bin/kibana
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
user=es
numprocs=1 ; number of processes copies to start (def 1)
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/tmp/kibana.log
stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
directory=/tmp/kibana-6.4.1-linux-x86_64