Flume初步搭建使用手册

合集下载

大数据、云计算系统高级架构师课程学习路线图

大数据、云计算系统高级架构师课程学习路线图大数据之Linux+大数据开发篇项目部分大数据之阿里云企业级认证篇大数据之Java企业级核心技术篇大数据之PB级别网站性能优化篇项目部分大数据之数据挖掘\分析＆机器学习篇项目部分大数据之运维、云计算平台篇项目部分c:\iknow\docshare\data\cur_work\javascript：open53kf()课程体系北风大数据、云计算系统架构师高级课程课程一、大数据运维之Linux基础本部分是基础课程，帮大家进入大数据领域打好Linux基础，以便更好地学习Hadoop，hbase,NoSQL，Spark，Storm,docker,openstack等众多课程。

因为企业中的项目基本上都是使用Linux环境下搭建或部署的。

1）Linux系统概述2)系统安装及相关配置3）Linux网络基础4）OpenSSH实现网络安全连接5）vi文本编辑器6）用户和用户组管理7）磁盘管理8）Linux文件和目录管理9）Linux终端常用命令10）linux系统监测与维护课程二、大数据开发核心技术- Hadoop 2。

x从入门到精通本课程是整套大数据课程的基石：其一，分布式文件系统HDFS用于存储海量数据,无论是Hive、HBase或者Spark数据存储在其上面；其二是分布式资源管理框架YARN,是Hadoop 云操作系统（也称数据系统），管理集群资源和分布式数据处理框架MapReduce、Spark应用的资源调度与监控；分布式并行计算框架MapReduce目前是海量数据并行处理的一个最常用的框架。

Hadoop 2。

x的编译、环境搭建、HDFS Shell使用，YARN 集群资源管理与任务监控,MapReduce编程，分布式集群的部署管理（包括高可用性HA）必须要掌握的。

1）大数据应用发展、前景2）Hadoop 2。

x概述及生态系统3)Hadoop 2。

x环境搭建与测试1)HDFS文件系统的架构、功能、设计2）HDFS Java API使用3）YARN 架构、集群管理、应用监控4）MapReduce编程模型、Shuffle过程、编程调优1)分布式部署Hadoop2.x2)分布式协作服务框架Zookeeper3）HDFS HA架构、配置、测试4)HDFS 2.x中高级特性5）YARN HA架构、配置6）Hadoop 主要发行版本（CDH、HDP、Apache）1)以【北风网用户浏览日志】数据进行实际的分析 2)原数据采集 3）数据的预处理(ETL） 4)数据的分析处理（MapReduce）课程三、大数据开发核心技术—大数据仓库Hive精讲hive是基于Hadoop的一个数据仓库工具,可以将结构化的数据文件映射为一张数据库表，并提供简单的sql查询功能，可以将sql语句转换为MapReduce任务进行运行。

视频云平台运维管理系统使用说明书

目录
使用说明书
全部服务集群新增集群修改资源升级集群
管理服务用户备份管理网络新增服务多网映射新增接入多网映射新增运维多网映射资源资源纳管
纳管资源修改
单个修改批量修改安装代理资源监控资源管理新增资源修改资源管理磁盘管理本地磁盘挂载网络磁盘查看资源清单检测网络诊断网络
常见图标
图标、
说明刷新图标。单击该图标，刷新当前界面、列表等。下拉框。单击该图标，可显示下拉菜单。文本框。可输入数字、字母等。
图标、、
、
、
使用说明书
说明
搜索框。
输入关键字，单击或者按【】键，可快速搜索到所需的信息。
支持模糊搜索。
折叠图标。
单击该图标，可展示隐藏的信息、配置项等，此时图标变为
使用说明书
第章系统简介
使用说明书
系统概述
视频云平台运维管理系统（以下简称“运维系统”）为满足大华视频云基线产品的统一安装部署、升级、扩容和运行管理需求，提供了层硬件服务、层开源云计算框架服务、层业务服务的状态监控和管理，同时支持半自动修复异常服务。运维系统致力于成为一个智能化运维平台，主要具备以下产品亮点： ● 通过视频云平台运维管理系统，开发人员能够方便地管理基础设备的网络与存储，并对其进
表示已使能。
选择框。
单击该图标，可选择对应的配置项。表示已选中。
关闭图标按钮。单击该图标，可关闭窗口。
确定按钮。单击该图标，保存当前页面所做的修改。
取消按钮。
单击该图标，取消保存当前页面所做的修改，并返回上级菜单。
使用安全须知
使用说明书
下面是关于产品的正确使用方法、为预防危险、防止财产受到损失等内容，使用设备前请仔细阅读本说明书并在使用时严格遵守，阅读后请妥善保存说明书。

Hadoop平台搭建与应用(第2版)(微课版)项目8 Hadoop平台应用综合案例

教学过程
教学提示
项目8 Hadoop平台应用综合案例
任务8.1 本地数据集上传到数据仓库Hive中
下面把test.txt中的数据导入到数据仓库Hive中。为了完成这个操作，需要先把test.txt上传到HDFS中，再在Hive中创建一个外部表，完成数据的导入。
1．启动HDFS
HDFS是Hadoop的核心组件，因此，要想使用HDFS，必须先安装Hadoop。这里已经安装了Hadoop，打开一个终端，执行命令“start-all.sh”，启动Hadoop服务。
（3）在项目名上，右击选择Open Module Settings，在弹出的页面中选择Modules，单击+号，选择Scala，若本地没有scala，则单击Create...，再单击Download...，选择scala版本2.11.0,单击OK。
（4）编辑pom.xml文件。
（5）在scala文件夹下，新建scala文件，选择Object型，命名为WordCount。
578 NodeManager
3154 Jps
1028 QuorumPeerMain
474 DataNode
1102 HRegionServer
进入HBase Shell。
在HBase中创建user_action表。
新建一个终端，导入数据。
再次切换到HBase Shell运行的终端窗口，执行命令“scan'user_action'”，查询插入的数据。
启动Hadoop集群和HBase服务，并查看集群节点进程。
master1节点的进程如下。
[root@master1 bin]# jps
1714 SecondaryNameNode

Sun Fire X2270 M2服务器初始启动和引导指南说明书

This guide describes the minimum steps you need to perform to power on and boot Oracle’s Sun Fire X2270M2server for the first time.Detailed installation information can be found in the Sun Fire X2270M2Server Installation Guide .For a complete list of documentation for the Sun Fire X2270M2Server ,see the section ”Documentation for this Product”.ContentsCarefully unpack all server components from the packing cartons.Some of the following items might be packaged separately.Documentation and Media KitYou can order the optional Documentation and Media Kit for your system.The kit contains the server installation guide,the Sun Fire X2270M2Server Tools and Drivers DVD ,Sun Installation Assistant CD/DVD ,and the SunVTS CD/DVD .When ordering the Documentation and Media Kit,using the following marketing part number:X8365A-X-MEDDOCKITDocumentation and software that are included in the Documentation and Media Kit is also available at:.Safety and Compliance InformationBefore installing your server,see the following documents for safety information:■Important Safety Information for Sun Hardware Systems –Printed document included in the ship kit.■Sun Fire X2270M2Server Safety and Compliance Manual –See the section ”Documentation for this Product”.•Power cord packaged separately with country kit•Hardware applicable to the system,such as adapters (RJ-45serial cable adapter)and network cables •Sun Fire X2270M2Accessory Kit,which includes the following:Sun Fire X2270M2Server Getting StartedGuide and License and safety documentation•(Optional )Rackmount kit containing rack rails and installation instructions •Optional server components•For documentation:/app/docs/prod/server.x64#hic•For software:/us/products/servers-storage/servers/x64/index.htmlSun Fire X2270 M2 Server Getting Started Guide▼Install Optional ComponentsBefore mounting the server in a rack,install any optional components that you ordered for your server (for example,DIMMs and HBA cards).♦See the Sun Fire X2270M2Server Service Manual for the instructions on installing components.▼Mount the System in a Rack♦To safely and properly mount the server in a rack,see the instructions included with the rail kit,the service label on the server,or the Sun Fire X2270M2Server Installation Guide .Front and Back Panel Connections and FeaturesFIGURE 1Back Panel Connections FIGURE 2Front Panel Features1AC Power connector5USB 2.0ports (2)2Serial Management (SER MGT)RS-232RJ-45serial port6Network Management (NET MGT)Ethernet port (available only in systems that contain a service processor module)3Gigabit Ethernet port (NET -0)7HD15video connector (available only in systems that contain a service processor module)4Gigabit Ethernet port (NET -1)1Green Power/OK LED 2Power button 3SerialNumber2635417123▼Connect the Cables1.Connect a serial cable between the server’s SER MGT port (see FIGURE 1)and a terminal device.You can use the supplied RJ-45crossover adapter with a standard serial cable to achieve a null modem configuration.2.(Optional)Connect an Ethernet cable between the server’s NET MGT port and the network to whichfuture connections to the service processor (SP)and host will be made.After the initial configuration of the system using the SER MGT port,communication with the SP and host is usually performed through this Ethernet interface.3.Connect an Ethernet cable between one of the server’s NET ports (NET-0or NET-1)and the network towhich the server will communicate.4.Connect the power cord to the power supply,but do not power on the server.When the server is connected to an AC power source,the SP initializes and the Power/OK LED (see FIGURE 2)blinks.After a few minutes,the SP login prompt appears on the terminal device.▼Log In to ILOM and Conﬁgure the SP IP AddressThis section shows you how to use a serial cable and a terminal device to log in to the SP Integrated Lights out Manager (ILOM),so you can configure the SP IP address.You can also connect to the SP to perform initial setup and configuration using these alternate methods:■Connect to ILOM Using an Ethernet Connection (DHCP)■Connect to ILOM Using an Ethernet Connection (static IP address)Note –Alternate connection methods and instructions on initial SP configuration are available in the Sun FireX2270M2Server Installation Guide .1.Log in to ILOM as root and use the password changme .2.Start the serial console:3.Refer to the ILOM documentation to configure your server’s network settings.4.To log out and exit ILOM,type:->exit▼Power On the SystemCaution –Component thermal damage.Server components require proper cooling.Do not operate theserver with the cover removed or without all fans,component heatsinks,air baffles installed.-> cd /SP/console ->start♦To power on the server to full-power mode,press and release the Power button on the server front panel.When main power is applied to the full server,the Power/OK LED next to the Power button lights and remains lit (does not blink).Note –The first time the server powers on,the power-on self-test (POST)can take up to 1minute to complete.▼Update Firmware1.Oracle highly recommends that you use the Sun Installation Assistant (SIA)to update system BIOS andILOM firmware.SIA is easy to use and available in the Documentation and Media Kit,or online at:/systemmanagement/sia.jspFor more information,refer to the documentation for the version of SIA supported for your server at:/app/docs/prod/install.x64svr#hic2.To update the system BIOS and ILOM firmware without using SIA,refer to the documentation for theversion of ILOM supported for your server.See the Sun Integrated Lights Out Manager 3.0Documentation Collection at:/app/docs/prod/int.lights.mgr30#hic▼Set Up the Platform Operating System,Driver Software,and FirmwareNote –Oracle highly recommends that you use the Sun Installation Assistant (SIA)software when installing asupported Windows or Linux OS.SIA assists in locating and installing all necessary OS components.♦After configuring the ILOM SP with network settings,configure the optional preinstalled operatingsystem (OS)or install a supported Linux,Windows or Solaris OS and drivers.For more information and instructions,see the following documents.Operating SystemRead this document:For additional information,go to:Preinstalled Solaris Sun Fire X2270M2Server Installation Guide/software/preinstall Solaris Sun Fire X2270M2Server Installation Guide for Oracle Solaris OS /servers/x64/x2270m2/os.jspLinuxSun Fire X2270M2Server Installation Guide for Linux OS Sun Installation Assistant 2.3through 2.4User’s Guide for x64Servers/systemmanagement/sia.jspWindowsSun Installation Assistant 2.3through 2.4User’s Guide for x64ServersSun Fire X2270M2Server Installation Guide for Windows OS/servers/x64/x2270/os.jspVirtualization Software Sun Fire X2270M2Server Installation Guide for Virtual Machine SoftwareFor information about supported operating systems,patches,and the latest firmware,refer to the Sun Fire X2270M2Server Product Notes .Information on the latest firmware is also available at:/us/products/servers-storage/servers/x64/index.html .▼Power Off the Server1.To perform a graceful shutdown of the system,quickly press and release the Power button.2.To perform an emergency shutdown,press and hold the Power button for four seconds to force mainpower off and to enter standby power mode.Documentation for this ProductYou can find the most up-to-date versions of all the Oracle’s Sun Fire X2270M2Server documents online by selecting the server from the product list at:/app/docs/prod/server.x64#hicTaskDocument TypePart NumberReview the safety information.Safety and Compliance Manual 821-1376Important Safety Information for Sun Hardware Systems816-7190Review any known issues and workarounds.Server Product Notes 821-1344Install rack hardware.Cable and power on server.Connect to the service processor and determine network settings.Configure the optional preinstalled Solaris OS.Server Installation Guide821-1348Install supported operating systems.Reinstall Solaris,or install Linux or Windows Server.Installation Guide for Oracle Solaris OS 821-1347Installation Guide for Windows OS 821-1346Installation Guide for Linux OS 821-1367Installation Guide for Oracle VM 821-1683Sun Installation Assistant 2.3through 2.4User’s Guide for x64Servers821-0694Install a supported version of virtualization software Installation Guide for ESX821-1369Manage server accounts,monitor alerts,set remote access and redirection,and view component status and event logs.Sun Integrated Lights Out Manager 3.0Getting Started Guide820-5523Sun Integrated Lights Out Manager (ILOM)3.0Supplement for Sun Fire X2270M2Server821-1611Customize BIOS settings and RAID settings.Remove and replace hardware components.Troubleshoot and isolateserver problems.Review system specifications and hardware reference material.Server Service Manual 821-1345Diagnose system issues.Systems Diagnostics Guide 821-1349Part No. 821-1377-10, Rev. A April 2010Translated versions of the documentation for you server might be available.Go to the ,select your language,and navigate to your server’s document collection.Locating the Server Serial NumberThe serial number is located on the front of your sever (see FIGURE 2).You can also use the ILOM web interface or the ILOM command-line interface to retrieve the serial number (for more information,see the Installation Guide,Service Manual,or ILOM documentation for your server).Hardware Warranty and Software Service Plan■For information on hardware warranty,got to:/service/warranty/index.jsp ■For information on a software service plan,go to:/service/serviceplanssoftware/Copyright ©2010Oracle and/or its affiliates.All rights reserved.。

相关主题

fortify安装使用手册

1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。

一、Flume介绍
1、Flume的概述（情报人员）
（1）Flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。

（2）Flume可以采集文件，socket数据包等各种形式源数据，又可以将采集到的数据输出到HDFS、hbase、hive、kafka等众多外部存储系统中
（3）一般的采集需求，通过对flume的简单配置即可实现
（4）Flume针对特殊场景也具备良好的自定义扩展能力，因此，flume可以适用于大部分的日常数据采集场景
2、运行机制
（1）Flume分布式系统中最核心的角色是agent，Flume采集系统就是由一个个agent所连接起来形成
（2）每一个agent相当于一个数据传递员，内部有三个组件：
Source：采集源，用于跟数据源对接，以获取数据
Channel：主要提供一个队列的功能，对source提供中的数据进行简单的缓存并传递到Sink 中
Sink：下沉地，采集数据的传送目的，用于往下一级agent传递数据或者往最终存储系统传递数据
如图所示：
3、采集结构
（1）Flume的核心组件Source的使用：
Flume提供了各种source的实现，包括Avro Source、Exce Source、Spooling Directory Source、NetCat Source、Syslog Source等
对现有系统影响最小的是Exec Source和Spool Source这两种方式：
其中Exec Source是以运行Linux命令的方式，持续的按照文件名输出最新的数据，如tail -F 文件名指令
而Spool Source是监测配置的目录下新增的文件，并将文件中的数据读取出来
三、Flume的配置种类总览
1、多代理流程
2、流合并
3、多路复用流
四、Flume配置种类实例
1、单一代理流定义
你需要通过一个通道将来源和接收器链接。

你需要列出源，接收器和通道，为给定的代理，然后指向源和接收器及通道。

一个源的实例可以指定多个通道，但只能指定一个接收器实例。

格式如下：
1. # list the sources, sinks and channels for the agent
2. <Agent>.sources = <Source>
3. <Agent>.sinks = <Sink>
4. <Agent>.channels = <Channel1> <Channel2>
5. # set channel for source
6. <Agent>.sources.<Source>.channels = <Channel1> <Channel2> ...
7. # set channel for sink
8. <Agent>.sinks.<Sink>.channel = <Channel1>
9.
10. #下面是正式版的例子
11. # list the sources, sinks and channels for the agent
12. agent_foo.sources = avro‐appserver‐src‐1
13. agent_foo.sinks = hdfs‐sink‐1
14. agent_foo.channels = mem‐channel‐1
15.
16. # set channel for source
17. agent_foo.sources.avro‐appserver‐src‐1.channels = mem‐channel‐1
18.
19. # set channel for sink
3、案例--将目标文件夹中的文件发送到kafka
仿照上一个案例，同样，也是两个Flume Agent
这里面的Collection机器中的Flume Agent中要配置sink为kafka，这里开始研究一下文档中的kafka sink
翻译内容：
然后启动两个Flume Agent
1. bin/flume‐ng agent ‐‐conf conf ‐‐conf‐file conf/weblog.conf ‐‐name a1 >logs/flume_log
另一个
1. bin/flume‐ng agent ‐‐conf conf ‐‐conf‐file conf/testkafka.conf ‐‐name a1 ‐Dflume.monitorin
六、实际案例开发
1、案例--采集日志文件的监控
目标采集主机的配置文件
创建一个weblog.conf，并且添加如下内容
1. #定义三大组件的名称
2. a1.sources = r1
3. a1.sinks = k1
4. a1.channels = c1
5. #配置source组件
6. a1.sources.r1.type = spooldir
7. a1.sources.r1.spoolDir = /usr/logs
8. a1.sources.r1.fileHeader = false
9. #配置sink组件
10. a1.sinks.k1.type = avro
11. a1.sinks.k1.hostname = 192.168.163.163
12. a1.sinks.k1.port = 44444
13. #配置channel组件
14. a1.channels.c1.type = file
15. a1.channels.c1.checkpointDir = /usr/flumetem/checkpoint
16. a1.channels.c1.dataDirs = /usr/flumetem/data
17. #给source和sink绑定channel
18. a1.sources.r1.channels = c1
19. a1.sinks.k1.channel = c1
发现有一个新的文件被写入到hdfs中去了
解析这个json字符串
1. {。