Top

NSD ARCHITECTURE DAY02

  1. 案例1：安装Logstash
  2. 案例2：编写logstash配置文件
  3. 案例3：Logstash input插件
  4. 案例4：Web日志解析实验
  5. 案例6：部署beats与filebeat
  6. 案例7：实时分析日志案例（拓展试验）
  7. 参考课上讲解的内容自己完成 web 日志的收集及使用 ELK 进行分析

1 案例1：安装Logstash

1.1 问题

本案例要求：

1.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：安装logstash

1）配置主机名，ip和yum源，配置/etc/hosts

  1. [root@logstash ~]# vim /etc/hosts
  2. 192.168.1.41    es-0001
  3. 192.168.1.42    es-0002
  4. 192.168.1.43    es-0003
  5. 192.168.1.44    es-0004
  6. 192.168.1.45    es-0005
  7. 192.168.1.46    kibana
  8. 192.168.1.47    logstash

2）安装java-1.8.0-openjdk和logstash

  1. [root@logstash ~]# yum -y install java-1.8.0-openjdk logstash
  2. [root@logstash ~]# java -version
  3. openjdk version "1.8.0_161"
  4. OpenJDK Runtime Environment (build 1.8.0_161-b14)
  5. OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
  6.  
  7. [root@logstash ~]# ln -s /etc/logstash /usr/share/logstash/config 
  8. [root@logstash ~]# vim /etc/logstash/conf.d/my.conf
  9. input { 
  10.   stdin {}
  11. }
  12.  
  13. filter{ }
  14.  
  15. output{ 
  16.   stdout{}
  17. }
  18. [root@logstash ~]# /usr/share/logstash/bin/logstash

2 案例2：编写logstash配置文件

2.1 问题

本案例要求：

2.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：codec类插件

1）codec类插件

  1. [root@logstash ~]# vim /etc/logstash/conf.d/my.conf
  2. input { 
  3.   stdin { codec => "json" }
  4. }
  5.  
  6. filter{ }
  7.  
  8. output{ 
  9.   stdout{ codec => "rubydebug" }
  10. }
  11. [root@logstash ~]# /usr/share/logstash/bin/logstash
  12. Settings: Default pipeline workers: 2
  13. Pipeline main started
  14. a
  15. {
  16.        "message" => "a",
  17.            "tags" => [
  18.           [0] "_jsonparsefailure"
  19. ],
  20.       "@version" => "1",
  21.     "@timestamp" => "2020-05-23T12:34:51.250Z",
  22.           "host" => "logstash"
  23. }

3 案例3：Logstash input插件

3.1 问题

本案例要求：

3.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：file模块插件

1）file模块插件

  1. [root@logstash ~]# vim /etc/logstash/conf.d/my.conf
  2. input { 
  3.   file {
  4.     path => ["/tmp/c.log"]
  5.     type => "test"
  6.     start_position => "beginning"
  7.     sincedb_path => "/var/lib/logstash/sincedb"
  8.   }
  9. }
  10. filter{ }
  11. output{ 
  12.   stdout{ codec => "rubydebug" }
  13. }
  14. [root@logstash ~]# rm -rf /var/lib/logstash/plugins/inputs/file/.sincedb_*
  15. [root@logstash ~]# touch /tmp/a.log /tmp/b.log
  16. [root@logstash ~]# /usr/share/logstash/bin/logstash

另开一个终端：写入数据

  1. [root@logstash ~]#  echo a1 >> /tmp/a.log 
  2. [root@logstash ~]#  echo b1 >> /var/tmp/b.log

之前终端查看：

  1. [root@logstash ~]# /usr/share/logstash/bin/logstash
  2. Settings: Default pipeline workers: 2
  3. Pipeline main started
  4. {
  5.        "message" => "a1",
  6.       "@version" => "1",
  7.     "@timestamp" => "2019-03-12T03:40:24.111Z",
  8.           "path" => "/tmp/a.log",
  9.           "host" => "logstash",
  10.           "type" => "testlog"
  11. }
  12. {
  13.        "message" => "b1",
  14.       "@version" => "1",
  15.     "@timestamp" => "2019-03-12T03:40:49.167Z",
  16.           "path" => "/tmp/b.log",
  17.           "host" => "logstash",
  18.           "type" => "testlog"
  19. }
  20.     

4 案例4：Web日志解析实验

4.1 问题

本案例要求：

4.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：filter grok模块插件

grok插件：

解析各种非结构化的日志数据插件

grok使用正则表达式把飞结构化的数据结构化

在分组匹配，正则表达式需要根据具体数据结构编写

虽然编写困难，但适用性极广

解析Apache的日志，之前已经安装过的可以不用安装

浏览器访问网页，在/var/log/httpd/access_log有日志出现

  1. [root@es-0005 ~]# cat /var/log/httpd/access_log
  2. 192.168.1.254 - - [12/Mar/2019:11:51:31 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
  3.  
  4. [root@logstash ~]#  vim /etc/logstash/logstash.conf
  5.  
  6. input{
  7.     file {
  8.       path           => [ "/tmp/a.log", "/tmp/b.log" ]
  9.       sincedb_path   => "/var/lib/logstash/sincedb"
  10.       start_position => "beginning"
  11.       type           => "testlog"
  12.    }
  13. }
  14.  
  15. filter{
  16.     grok{
  17.        match => [ "message",  "(?<key>reg)" ]
  18.     }
  19. }
  20.  
  21. output{
  22.     stdout{ codec => "rubydebug" }
  23. }

复制/var/log/httpd/access_log的日志到logstash下的/tmp/c.log

  1. [root@logstash ~]# echo '192.168.1.252 - - [29/Jul/2020:14:06:57 +0800] "GET /info.html HTTP/1.1" 200 119 "-" "curl/7.29.0"' >/tmp/c.log
  2. [root@logstash ~]# vim /etc/logstash/conf.d/my.conf
  3. input { 
  4.   file {
  5.     path => ["/tmp/c.log"]
  6.     type => "test"
  7.     start_position => "beginning"
  8.     sincedb_path => "/dev/null"
  9.   }
  10. }
  11. filter{ 
  12.   grok {
  13.     match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  14.   }
  15. }
  16. output{ 
  17.   stdout{ codec => "rubydebug" }
  18. }
  19. [root@logstash ~]# /usr/share/logstash/bin/logstash

查找正则宏路径

  1. [root@logstash ~]# cd 
  2. /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns
  3. [root@logstash ~]# cat httpd  //查找COMBINEDAPACHELOG
  4. COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
  5.  
  6. [root@logstash ~]#  vim /etc/logstash/logstash.conf
  7. ...
  8. filter{
  9.    grok{
  10.         match => ["message", "%{ HTTPD_COMBINEDLOG }"]
  11.   }
  12. }
  13. ...

解析出的结果

  1.  [root@logstash ~]#  /opt/logstash/bin/logstash -f  /etc/logstash/logstash.conf  
  2. Settings: Default pipeline workers: 2
  3. Pipeline main started
  4. {
  5.         "message" => "192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET /noindex/css/open-sans.css HTTP/1.1\" 200 5081 \"http://192.168.1.65/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
  6.        "@version" => "1",
  7.      "@timestamp" => "2018-09-15T10:55:57.743Z",
  8.            "path" => "/tmp/a.log",
  9. ZZ           "host" => "logstash",
  10.            "type" => "testlog",
  11.        "clientip" => "192.168.1.254",
  12.           "ident" => "-",
  13.            "auth" => "-",
  14.       "timestamp" => "15/Sep/2019:18:25:46 +0800",
  15.            "verb" => "GET",
  16.         "request" => "/noindex/css/open-sans.css",
  17.     "httpversion" => "1.1",
  18.        "response" => "200",
  19.           "bytes" => "5081",
  20.        "referrer" => "\"http://192.168.1.65/\"",
  21.           "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\""
  22. }
  23. ...

5 案例6：部署beats与filebeat

5.1 问题

本案例要求：

5.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：filter grok模块插件

  1. [root@logstash ~]# vim /etc/logstash/conf.d/my.conf
  2. input { 
  3.   stdin { codec => "json" }
  4.   file{
  5.     path => ["/tmp/c.log"]
  6.     type => "test"
  7.     start_position => "beginning"
  8.     sincedb_path => "/var/lib/logstash/sincedb"
  9.   }
  10.   beats {
  11.     port => 5044
  12.   }
  13. } 
  14.  
  15. filter{ 
  16.   grok {
  17.     match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  18.   }
  19. } 
  20.  
  21. output{ 
  22.   stdout{ codec => "rubydebug" }
  23.   elasticsearch {
  24.     hosts => ["es-0004:9200", "es-0005:9200"]
  25.     index => "weblog-%{+YYYY.MM.dd}"
  26.   }
  27. }
  28. [root@logstash ~]# /usr/share/logstash/bin/logstash

2）在之前安装了Apache的主机上面安装filebeat

  1. [root@web ~]# yum install -y filebeat
  2. [root@web ~]# vim /etc/filebeat/filebeat.yml
  3. 24:  enabled: true
  4. 28:  - /var/log/httpd/access_log
  5. 45:    fields: 
  6. 46:       my_type: apache
  7. 148, 150 注释掉
  8. 161: output.logstash:
  9. 163:   hosts: ["192.168.1.47:5044"]
  10. 180, 181, 182 注释掉
  11. [root@web ~]# grep -Pv "^\s*(#|$)" /etc/filebeat/filebeat.yml
  12. [root@web ~]# systemctl enable --now filebeat

6 案例7：实时分析日志案例（拓展试验）

7 参考课上讲解的内容自己完成 web 日志的收集及使用 ELK 进行分析