集团大数据平台总体设计

集团大数据平台总体设计

目录

1综述 ------------------------------------------------------------------------------------------------ 4

1.1项目背景 --------------------------------------------------------------------------------------------- 4

1.2建设目标 --------------------------------------------------------------------------------------------- 4

1.3需求分析 --------------------------------------------------------------------------------------------- 4

基础平台------------------------------------------------------------------------------------------ 5

企业画像应用 ----------------------------------------------------------------------------------- 6

2总体建设方案 ----------------------------------------------------------------------------------- 7

2.1平台框架设计理念 -------------------------------------------------------------------------------- 7

2.2功能架构 --------------------------------------------------------------------------------------------- 8

2.3技术架构 --------------------------------------------------------------------------------------------- 9

2.4产品覆盖综述------------------------------------------------------------------------------------- 10

2.5数据分布架构------------------------------------------------------------------------------------- 11

2.6关键技术说明------------------------------------------------------------------------------------- 11

云服务平台------------------------------------------------------------------------------------- 11

Hadoop平台------------------------------------------------------------------------------------ 23

元数据管理------------------------------------------------------------------------------------- 24

数据治理---------------------------------------------------------------------------------------- 31

爬虫引擎---------------------------------------------------------------------------------------- 36

数据探索---------------------------------------------------------------------------------------- 40

自助分析---------------------------------------------------------------------------------------- 41

企业画像---------------------------------------------------------------------------------------- 42

3技术方案特性 --------------------------------------------------------------------------------- 45

3.1平台开放性 ---------------------------------------------------------------------------------------- 45

3.2高性能----------------------------------------------------------------------------------------------- 46

应用高性能------------------------------------------------------------------------------------- 46

平台系统资源高性能 ----------------------------------------------------------------------- 46

实时数据抽取 --------------------------------------------------------------------------------- 47

压缩流处理------------------------------------------------------------------------------------- 48

库外处理与计算 ------------------------------------------------------------------------------ 48

分布式处理------------------------------------------------------------------------------------- 51

多网卡支持------------------------------------------------------------------------------------- 52

3.3高可用性 ------------------------------------------------------------------------------------------- 54

Hadoop平台高可用性----------------------------------------------------------------------- 54

ETL高可用性 ---------------------------------------------------------------------------------- 58

应用高可用性 --------------------------------------------------------------------------------- 59

3.4高可靠性 ------------------------------------------------------------------------------------------- 60

3.5开发易用性 ---------------------------------------------------------------------------------------- 61

3.6可维护性 ------------------------------------------------------------------------------------------- 66

3.7弹性扩展能力------------------------------------------------------------------------------------- 70

3.8资源管控能力------------------------------------------------------------------------------------- 71

多租户资源管控 ------------------------------------------------------------------------------ 71