ELK学习实验017：filebeat收集java⽇志
收集JAVA格式⽇志
1 查看Java格式⽇志
elasticsearch属于Java⽇志，可以收集elasticsearch作为Java⽇志范本
[root@node3 ~]# tail -f /usr/local/elasticsearch/logs/my-elktest-cluster.log
[2020-01-19T01:57:18,496][INFO ][TransportService  ] [node-3] publish_address {192.168.132.133:9300}, bound_addresses {[::]:9300}
[2020-01-19T01:57:18,506][INFO ][o.e.b.BootstrapChecks    ] [node-3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-01-19T01:57:18,531][INFO ][Coordinator      ] [node-3] cluster UUID [4xt-ZTijTz2oTnlz1gMFjg]
[2020-01-19T01:57:19,195][INFO ][s.ClusterApplierService] [node-3] master node changed {previo
us [], current [{node-2}{9qVjdVSvSAGlZ7lpB9O78g}{m9_mEscHTMKQ0VwFt9YHog}{192.168.132.132}{192.168.132.132:9300}{dilm}{ml.machi 2 直接使⽤filebeat收集
直接配置，使⽤filebeat收集
filebeat.inputs:
>>>>>>>>>>###
## Nginx log
>>>>>>>>>>###
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/access.log
json.key_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/error.log
tags: ["error"]
>>>>>>>>>>###
## tomcat  log
>>>>>>>>>>###
- type: log
enabled: true
paths:
- /var/log/tomcat/localhost_access_log.*.txt
json.key_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
>>>>>>>>>>###
## tomcat  log
>>>>>>>>>>###
- type: log
enabled: true
paths:
- /usr/local/elasticsearch/logs/my-elktest-cluster.log
tags: ["es-java"]
>>>>>>>>>>###
## Output
>>>>>>>>>>###
setup.kibana:
host: "192.168.132.131:5601"
output.elasticsearch:
hosts: ["192.168.132.131:9200","192.168.132.132:9200","192.168.132.133:9200"]
#index: "nginx-%{[agent.version]}-%{+yyyy.MM.dd}"
indices:
- index: "access-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "access"
- index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "error"
- index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "tomcat"
- index: "javaes-access-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "es-java"
abled: false
[root@node3 ~]# systemctl restart filebeat
3 错误⽇志显⽰不清晰
但是错误信息分开显⽰
[2020-01-19T01:55:48,077][WARN ][NodeConnectionsService] [node-3] failed to connect to {node-2}{9qVjdVSvSAGlZ7lpB9O78g}{aFwQRTD4TWKE97npXNwSVg}{192.168.132.132}{192.168.132.132:9300}{dilm}{ml.machine_memory= ansport.ConnectTransportException: [node-2][192.168.132.132:9300] connect_exception
at ansport.Failure(TcpTransport.java:976) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.4.2.jar:7.4.2]
at urrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.4.2.jar:7.4.2]
at urrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at urrent.Fire(CompletableFuture.java:837) ~[?:?]
at urrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at urrent.CompletableFuturepleteExceptionally(CompletableFuture.java:2159) ~[?:?]
at urrent.CompletableContextpleteExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.4.2.jar:7.4.2]
at ansportty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at ifyListener0(DefaultPromise.java:500) ~[?:?]
at ifyListeners0(DefaultPromise.java:493) ~[?:?]
at ifyListenersNow(DefaultPromise.java:472) ~[?:?]
at ifyListeners(DefaultPromise.java:413) ~[?:?]
at urrent.DefaultPromise.setValue0(DefaultPromise.java:538) ~[?:?]
at urrent.DefaultPromise.setFailure0(DefaultPromise.java:531) ~[?:?]
at Failure(DefaultPromise.java:111) ~[?:?]
at ioty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323) ~[?:?]
查看svg
at ioty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[?:?]
at ioty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685) ~[?:?]
at ioty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) ~[?:?]
at ioty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) ~[?:?]
at ioty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[?:?]
Caused by: java.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?]
at ioty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at ioty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336) ~[?:?]
... 7more
[2020-01-19T01:55:49,727][INFO ][Node              ] [node-3] stopping ...
这是⼀段错误⽇志，需要单独收集
4 配置filebat的多⾏匹配
以时间作为标识，再读取⼀个时间标记之后，先缓存，当读到下⼀个时间标记，再整个发送⽇志，相当于多⾏匹配
官⽅⽂档：
multiline.pattern: '^\['
multiline.match: after
配置filebeat
filebeat.inputs:
>>>>>>>>>>###
## Nginx log
>>>>>>>>>>###
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/access.log
json.key_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/error.log
tags: ["error"]
>>>>>>>>>>###
## tomcat  log
>>>>>>>>>>###
- type: log
enabled: true
paths:
- /var/log/tomcat/localhost_access_log.*.txt
json.key_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
>>>>>>>>>>###
## tomcat  log
>>>>>>>>>>###
- type: log
enabled: true
paths:
- /usr/local/elasticsearch/logs/my-elktest-cluster.log
tags: ["es-java"]
multiline.pattern: '^\['
multiline.match: "after"
>>>>>>>>>>###
## Output
>>>>>>>>>>###
setup.kibana:
host: "192.168.132.131:5601"
output.elasticsearch:
hosts: ["192.168.132.131:9200","192.168.132.132:9200","192.168.132.133:9200"]
#index: "nginx-%{[agent.version]}-%{+yyyy.MM.dd}"
indices:
- index: "access-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "access"
- index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "error"
- index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"
tags: "tomcat"
- index: "javaes-access-%{[agent.version]}-%{+yyyy.MM.dd}"      ains:
tags: "es-java"
abled: false
[root@node3 ~]# systemctl restart filebeat
5 ⽣成错误⽇志
然后修改配置⽂件，重启产⽣错误⽇志
到kibana查看，错误⽇志⼀起显⽰
收集Java⽇志，配置完成
dolphinscheduler工作流导出
« 上一篇
各种文件的mime类型
下一篇 »

发表评论

Copyright © 2024 baidu.com
Powered By 688IT编程网 豫ICP备2022007602号 豫公网安备41160202000602