Redhat本地搭建EFK环境

| 分类 efk  | 标签 efk  Fluentd  Elasticsearch  Kibana  浏览次数: -

EFK版本信息

EFK架构图

Fluentd

在线安装

1.安装

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

2.查看状态

/etc/init.d/td-agent status

3.安装插件

td-agent-gem install fluent-plugin-elasticsearch
td-agent-gem install fluent-plugin-typecast
td-agent-gem install fluent-plugin-secure-forward

离线安装

下载页面中根据操作系统版本选择相应rpm包

  • 安装依赖
    • redhat-lsb-core
    • redhat-lsb-core会有一堆依赖包,建议还是在线安装

Elasticsearch

处理依赖

安装Java8

从2.1版本后依赖Java8, 下载地址,需要在机器上安装。

  • 卸载默认版本Java

rpm -qa | grep -i java | xargs rpm -e --nodeps

  • 安装

rpm -ivh jdk-8u181-linux-x64.rpm

修改可以虚拟内存大小

修改配置文件/etc/sysctl.conf,增加vm.max_map_count=262144,执行sysctl -p使之生效

或者执行

sysctl -w vm.max_map_count=262144

非root用户运行

elastaicsearch不允许使用root用户运行,我们新建一个 es 用户来运行。

  • 创建用户组

groupadd es

  • 创建用户

useradd -g es -m es

CentOS6 需要添加以下两行

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

离线安装

  • 1.下载

cd /tmp && curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.1.tar.gz

  • 2.设置文件句柄大小
cat >>/etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 16384
* hard nproc 16384
EOF
  • 3.解压运行
scp /tmp/elasticsearch-6.4.1.tar.gz /home/es
su - es
tar zxvf elasticsearch-6.4.1.tar.gz
cd elasticsearch-6.4.1/
sed -ie 's/#network.host:.*/network.host: 0.0.0.0/' config/elasticsearch.yml
./bin/elasticsearch

Kibana

离线安装

1.下载

cd /tmp && curl -O https://artifacts.elastic.co/downloads/kibana/kibana-6.4.1-linux-x86_64.tar.gz

2.解压运行

tar zxvf kibana-6.4.1-linux-x86_64.tar.gz
cd kibana-6.4.1-linux-x86_64
sed -ie 's/#server.host:.*/server.host: "0.0.0.0"/' config/kibana.yml
./bin/kibana

端口信息

  • 5601: kibana web接口
  • 9200: elasticsearch JSON接口
  • 24224: fluentd端口

整体测试

  • 1.nginx设置日志格式

log_format main ‘$remote_addr $http_host [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"’;

  • 2.准备测试文件
touch /tmp/access.log
chmod 666 /tmp/access.log
mkdir -p /var/log/td-agent/access/
touch /var/log/td-agent/access/access.log.pos
chmod 666 /var/log/td-agent/access/access.log.pos
  • 3.fluentd配置采集

需要注意path权限问题

cat >>/etc/td-agent/td-agent.conf <<EOF
<source>
  @type tail
  path /tmp/access.log
  pos_file /var/log/td-agent/access/access.log.pos

  tag nginx.access
  format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<other>[^ ]*)"$/
  time_format %d/%b/%Y:%H:%M:%S %z
</source>
<match nginx.access>
  @type elasticsearch
  host localhost
  port 9200

  flush_interval 2s
  buffer_queue_limit 4096
  buffer_chunk_limit 1024m
  num_threads 4
  logstash_format true
</match>
EOF
  • 4.写入日志进行测试
cat >>/tmp/access.log <<EOF
10.10.110.41 - - [11/Sep/2018:14:50:46 +0800] "POST /auth HTTP/1.1" 202 97 "http://10.10.20.98/login.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
10.10.110.41 - - [11/Sep/2018:14:50:58 +0800] "GET /auth/ HTTP/1.1" 200 424 "http://10.10.20.98/login.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
EOF

或者实时从Nginx中读取日志

tail -n 2 /home/sendoh/qdata-cloud/logs/nginx/access.log >> /tmp/access.log

  • 5.查询json格式

curl http://10.10.20.98:9200/_search?pretty

  • 6.kibana页面访问Discover

Kibana配置1 Kibana配置2 Kibana日志效果

supervisor配置

直接启动的方式程序容易挂掉,建议使用supervisor来对Elasticsearch, Kibana服务进行管理

  • 安装supervisor

pip install supervisor

  • supervisor默认配置

mkdir -p /etc/conf.d

vi /etc/supervisord.conf

注意:supervisor默认使用文件句柄大小是1024,对启动elasticsearch来说还是太小了,需要进行修改

; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
;  - Shell expansion ("~" or "$HOME") is not supported.  Environment
;    variables can be expanded using this syntax: "%(ENV_HOME)s".
;  - Comments must have a leading space: "a=b ;comment" not "a=b;comment".

[unix_http_server]
file=/tmp/sendoh_supervisor.sock   ; (the path to the socket file)
;chmod=0700                 ; socket file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

;[inet_http_server]         ; inet (TCP) server disabled by default
;port=0.0.0.0:9002        ; (ip_address:port specifier, *:port for all iface)
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/sendoh_supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=65536
minprocs=32768
;umask=022                   ; (process file creation umask;default 022)
user=root                ; (default is current user, required if root)
;identifier=supervisor       ; (supervisord identifier, default is 'supervisor')
;directory=/tmp              ; (default is not to cd during start)
;nocleanup=true              ; (don't clean up tempfiles at start;default false)
;childlogdir=/tmp            ; ('AUTO' child log dir, default $TEMP)
;environment=KEY="value"     ; (key value pairs to add to environment)
;strip_ansi=false            ; (strip ansi escape codes in logs; def. false)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/sendoh_supervisor.sock ; use a unix:// URL  for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris              ; should be same as http_username if set
;password=123                ; should be same as http_password if set
;prompt=mysupervisor         ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history  ; use readline history if available

[include]
files = conf.d/*.conf
  • EFK配置

vi /etc/conf.d/supervisor_efk.conf

[program:elasticsearch]
command=/home/es/elasticsearch-6.4.1/bin/elasticsearch
user=es
minfds=65536
minprocs=32768
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1                    ; number of processes copies to start (def 1)
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/tmp/elasticsearch.log
stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false   ; emit events on stdout writes (default false)
directory=/home/es/elasticsearch-6.4.1

[program:kibana]
command=/tmp/kibana-6.4.1-linux-x86_64/bin/kibana
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
user=es
numprocs=1                    ; number of processes copies to start (def 1)
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/tmp/kibana.log
stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false   ; emit events on stdout writes (default false)
directory=/tmp/kibana-6.4.1-linux-x86_64

上一篇 Mac安装kubernetes     下一篇 SOLID原则
目录导航