大数据技术-flume的安装使用
合集下载
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced # during Flume startup.
# Enviroment variables can be set here.
export JAVA_HOME=/root/apps/jdk1.7.0_45
Transferring apache-flume-1.6.0-bin.tar.gz...
100% 51318 KB 12829 KB/sec 00:00:04
0 Errors
[root@min1 ~]# ll apache-flume-1.6.0-bin.tar.gz -rw-r--r--. 1 root root 52550402 6 月 10 2017 apache-flume-1.6.0-bin.tar.gz [root@min1 ~]#
# Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444
# Describe the sink a1.sinks.k1.type = logger
2 解压安装包
然后解压 tar -zxvf apache-flume-1.6.0-bin.tar.gz
3 配置 java_home
然后进入 flume 的目录，修改 conf 下的 flume-env.sh，在里面配置 JAVA_HOME [root@min1 conf]# cp flume-env.sh.template flume-env.sh [root@min1 conf]# ll 总用量 20 -rw-r--r--. 1 501 games 1661 5 月 9 2015 flume-conf.properties.template -rw-r--r--. 1 501 games 1110 5 月 9 2015 flume-env.ps1.template -rw-r--r--. 1 root root 1214 12 月 3 02:56 flume-env.sh -rw-r--r--. 1 501 games 1214 5 月 9 2015 flume-env.sh.template -rw-r--r--. 1 501 games 3107 5 月 9 2015 log4j.properties [root@min1 conf]#
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
# Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH=""
# # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
[root@min1 conf]# vi flume-env.sh # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # /licenses/LICENSE-2.0
~ ~ -- INSERT --
4 配置采集方案
根据数据采集的需求配置采集方案，描述在配置文件中(文件名可任意自定义)
vi /root/apps/apache-flume-1.6.0-bin/conf/netcat-logger.conf
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1
Flume 安装使用
1 上传安装包
Flume 的安装，只需要解压即可，当然，前提是已有 hadoop 环境 上传安装包到数据源所在 linux 上
பைடு நூலகம்
[root@min1 ~]# rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.
# Enviroment variables can be set here.
export JAVA_HOME=/root/apps/jdk1.7.0_45
Transferring apache-flume-1.6.0-bin.tar.gz...
100% 51318 KB 12829 KB/sec 00:00:04
0 Errors
[root@min1 ~]# ll apache-flume-1.6.0-bin.tar.gz -rw-r--r--. 1 root root 52550402 6 月 10 2017 apache-flume-1.6.0-bin.tar.gz [root@min1 ~]#
# Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444
# Describe the sink a1.sinks.k1.type = logger
2 解压安装包
然后解压 tar -zxvf apache-flume-1.6.0-bin.tar.gz
3 配置 java_home
然后进入 flume 的目录，修改 conf 下的 flume-env.sh，在里面配置 JAVA_HOME [root@min1 conf]# cp flume-env.sh.template flume-env.sh [root@min1 conf]# ll 总用量 20 -rw-r--r--. 1 501 games 1661 5 月 9 2015 flume-conf.properties.template -rw-r--r--. 1 501 games 1110 5 月 9 2015 flume-env.ps1.template -rw-r--r--. 1 root root 1214 12 月 3 02:56 flume-env.sh -rw-r--r--. 1 501 games 1214 5 月 9 2015 flume-env.sh.template -rw-r--r--. 1 501 games 3107 5 月 9 2015 log4j.properties [root@min1 conf]#
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
# Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH=""
# # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
[root@min1 conf]# vi flume-env.sh # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # /licenses/LICENSE-2.0
~ ~ -- INSERT --
4 配置采集方案
根据数据采集的需求配置采集方案，描述在配置文件中(文件名可任意自定义)
vi /root/apps/apache-flume-1.6.0-bin/conf/netcat-logger.conf
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1
Flume 安装使用
1 上传安装包
Flume 的安装，只需要解压即可，当然，前提是已有 hadoop 环境 上传安装包到数据源所在 linux 上
பைடு நூலகம்
[root@min1 ~]# rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.