常见的几种Flume日志收集场景实战

常见的⼏种Flume⽇志收集场景实战
这⾥主要介绍⼏种常见的⽇志的source来源，包括监控⽂件型，监控⽂件内容增量，TCP和HTTP。

Spool类型
⽤于监控指定⽬录内数据变更，若有新⽂件，则将新⽂件内数据读取上传
在最后有介绍此案例
Exec
EXEC执⾏⼀个给定的命令获得输出的源,如果要使⽤tail命令，必选使得file⾜够⼤才能看到输出内容
创建agent配置⽂件
# vi /usr/local/flume170/conf/exec_tail.conf
a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels = c1 c2
mand = tail -F /var/log/haproxy.log
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = file
a1.channels.c2.checkpointDir = /usr/local/flume170/checkpoint
a1.channels.c2.dataDirs = /usr/local/flume170/data
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel =c1
a1.sinks.k2.type = FILE_ROLL
a1.sinks.k2.channel = c2
a1.sinks.k2.sink.directory = /usr/local/flume170/files
a1.sinks.k2.sink.rollInterval = 0
启动flume agent a1
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/exec_tail.conf -n a1 -Dflume.root.logger=INFO,console ⽣成⾜够多的内容在⽂件⾥
# for i in {1..100};do echo "exec tail$i" >> /usr/local/flume170/log_exec_tail;echo $i;sleep 0.1;done
在H32的控制台，可以看到以下信息：
Http
JSONHandler型
基于HTTP POST或GET⽅式的数据源，⽀持JSON、BLOB表⽰形式
创建agent配置⽂件
# vi /usr/local/flume170/conf/post_json.conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 5142
a1.sources.r1.channels = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
启动flume agent a1
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/post_json.conf -n a1 -Dflume.root.logger=INFO,console ⽣成JSON 格式的POST request
# curl -X POST -d '[{ "headers" :{"a" : "a1","b" : "b1"},"body" : "_body"}]' http://localhost:8888
在H32的控制台，可以看到以下信息：
Tcp
Syslogtcp监听TCP的端⼝做为数据源
创建agent配置⽂件
# vi /usr/local/flume170/conf/syslog_tcp.conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = H32
a1.sources.r1.channels = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
启动flume agent a1
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/syslog_tcp.conf -n a1 -Dflume.root.logger=INFO,console 测试产⽣syslog
# echo "hello syslog" | nc localhost 5140
在H32的控制台，可以看到以下信息：
Flume Sink Processors和Avro类型
Avro可以发送⼀个给定的⽂件给Flume，Avro 源使⽤AVRO RPC机制。

failover的机器是⼀直发送给其中⼀个sink，当这个sink不可⽤的时候，⾃动发送到下⼀个sink。

channel的transactionCapacity参数不能⼩于sink的batchsiz
在H32创建Flume_Sink_Processors配置⽂件
# vi /usr/local/flume170/conf/Flume_Sink_Processors.conf
a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1 c2
a1.sources.r1.selector.type = replicating
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = H32
a1.sinks.k1.port = 5141
a1.sinks.k2.type = avro
a1.sinks.k2.channel = c2
a1.sinks.k2.hostname = H33
a1.sinks.k2.port = 5141
# 这个是配置failover的关键，需要有⼀个sink group
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
# 处理的类型是failover
a1.sinkgroups.g1.processor.type = failover
# 优先级，数字越⼤优先级越⾼，每个sink的优先级必须不相同
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
# 设置为10秒，当然可以根据你的实际状况更改成更快或者很慢
a1.sinkgroups.g1.processor.maxpenalty = 10000
在H32创建Flume_Sink_Processors_avro配置⽂件
# vi /usr/local/flume170/conf/Flume_Sink_Processors_avro.conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5141
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
将2个配置⽂件复制到H33上⼀份
/usr/local/flume170# scp -r /usr/local/flume170/conf/Flume_Sink_Processors.conf
H33:/usr/local/flume170/conf/Flume_Sink_Processors.conf
/usr/local/flume170# scp -r /usr/local/flume170/conf/Flume_Sink_Processors_avro.conf
H33:/usr/local/flume170/conf/Flume_Sink_Processors_avro.conf
打开4个窗⼝，在H32和H33上同时启动两个flume agent
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/Flume_Sink_Processors_avro.conf -n a1 -Dflume.root.logger=INFO,console
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/Flume_Sink_Processors.conf -n a1 -Dflume.root.logger=INFO,console
然后在H32或H33的任意⼀台机器上，测试产⽣log
# echo " test1 failover" | nc H32 5140
因为H33的优先级⾼，所以在H33的sink窗⼝，可以看到以下信息，⽽H32没有：
这时我们停⽌掉H33机器上的sink(ctrl+c)，再次输出测试数据
# echo " test2 failover" | nc localhost 5140
可以在H32的sink窗⼝，看到读取到了刚才发送的两条测试数据：
我们再在H33的sink窗⼝中，启动sink：
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/Flume_Sink_Processors_avro.conf -n a1 -Dflume.root.logger=INFO,console
输⼊两批测试数据：
# echo " test3 failover" | nc localhost 5140 && echo " test4 failover" | nc localhost 5140
在H33的sink窗⼝，我们可以看到以下信息，因为优先级的关系，log消息会再次落到H33上：
Load balancing Sink Processor
load balance type和failover不同的地⽅是，load balance有两个配置，⼀个是轮询，⼀个是随机。

两种情况下如果被选择的sink不可⽤，就会⾃动尝试发送到下⼀个可⽤的sink上⾯。

在H32创建Load_balancing_Sink_Processors配置⽂件
# vi /usr/local/flume170/conf/Load_balancing_Sink_Processors.conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1 k2
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = H32
a1.sinks.k1.port = 5141
a1.sinks.k2.type = avro
a1.sinks.k2.channel = c1
a1.sinks.k2.hostname = H33
a1.sinks.k2.port = 5141
# 这个是配置failover的关键，需要有⼀个sink group
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
# 处理的类型是load_balance
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.backoff = true
a1.sinkgroups.g1.processor.selector = round_robin
在H32创建Load_balancing_Sink_Processors_avro配置⽂件
# vi /usr/local/flume170/conf/Load_balancing_Sink_Processors_avro.conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5141
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
将2个配置⽂件复制到H33上⼀份
/usr/local/flume170# scp -r /usr/local/flume170/conf/Load_balancing_Sink_Processors.conf
H33:/usr/local/flume170/conf/Load_balancing_Sink_Processors.conf
/usr/local/flume170# scp -r /usr/local/flume170/conf/Load_balancing_Sink_Processors_avro.conf
H33:/usr/local/flume170/conf/Load_balancing_Sink_Processors_avro.conf
打开4个窗⼝，在H32和H33上同时启动两个flume agent
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/Load_balancing_Sink_Processors_avro.conf -n a1 -Dflume.root.logger=INFO,console
# /usr/local/flume170/bin/flume-ng agent -c . -f /usr/local/flume170/conf/Load_balancing_Sink_Processors.conf -n a1 -
Dflume.root.logger=INFO,console
然后在H32或H33的任意⼀台机器上，测试产⽣log，⼀⾏⼀⾏输⼊，输⼊太快，容易落到⼀台机器上
# echo " test1" | nc H32 5140
# echo " test2" | nc H32 5140
# echo " test3" | nc H32 5140
# echo " test4" | nc H32 5140
在H32的sink窗⼝，可以看到以下信息
1. 14/08/10 15:35:29 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61
6C 6C 2E 6F 72 67 20 74 65 73 74 32 test2 }
2. 14/08/10 15:35:33 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61
6C 6C 2E 6F 72 67 20 74 65 73 74 34 test4 }
在H33的sink窗⼝，可以看到以下信息：
1. 14/08/10 15:35:27 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61
6C 6C 2E 6F 72 67 20 74 65 73 74 31 test1 }
2. 14/08/10 15:35:29 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61
6C 6C 2E 6F 72 67 20 74 65 73 74 33 test3 }
说明轮询模式起到了作⽤。

志收集，其实还可以⼴泛的认为“⽇志”可以当作是信息流，不局限于认知的⽇志。