苏宁大数据平台运维实践
合集下载
相关主题
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
sk
h
+ "(
$#
%
MTTR
1
+
,
&
SLA 00
DOING & TO DO
zbDataCloudc
Kafka
CBT
v OLAP
Hive
Sqoop
Spark SQL
Spark Streaming
MLLib
ES
MapReduce
Spark Core
Kylin
YARN
Druid
6GO BINK RK f p
p UURd
g
d
jg
- HGIQ U Z L US % +g s p
d
g
p
d
73
f
d
OSOZ/ G GRRKR73CN KG Y/+g
73COSK OSOZ/ 738KG 6 KK
3M U
:GLQG
3R YZK
AKMOUT
AKMOUT%
AKMOUT
AKMOUT
3R YZK %
Libra Storm
Flume
HBase
Alluxio HDFS
Cassan dra
Zookeeper
Web
app
n
WebServer
Http
LogServer
Flume
FTP
Push
n
Kafka
Flume/DataFlow
HDFS/Hive
DB
Storm/Spark Streaming
MR/Spark
Kafka
poll
n
HDFS/Hive
FTP
DB
p8G UU p
BZU S B b OH Gc j
BZU S
j
j
B GQ
4 O 5Bj j
2013.10
2014.6
2015.5 2014.12
2015.12
2016.7
2017.3
j BZU S :GLQG
8G UU % 0%
j
(
(i
di
( C2
'"vm.zone_reclaim_mode=0
•
System CPU
echo never >/sys/kernel/mm/transparent_hugepage/defrag
-
•
/
fp
• Hive metaserver • yUser/IP
• skipTrash f
p dl
fRPC
d
-
HDFS
• Cache %$ ! • Shuffle %$ !
+-
0-
• Task*. • /Task !
• +Overhead ! • Reduce ! • JDBC !Kafka*
!
• Shuffle • HDFS Commit/ • Executor • Spill ! • 1!Parquet
•
d
: Hive unix_timestamp
•
• YARN • YARN
1 p
• Federation + Alluxio
DOING & TO DO
U QK
NUKTO
82GYK
4.
<A
l
4.
<A
4.
<A
4.
'(
CPG EG JMG
-
NHDI A
Y GQ
-
-
-
-
-
Huatuo-server
-
-
4.
Y GQ
,%
• # • HDFS, !
• Driver/Executor- !
• ,
+) • RDD Size Estimator*) •
4.
Y GQ
4.
Y GQ
3GINK
4.
Y GQ
CGYQ
4.
Y GQ
BN LLRK
4.
> Z>L<KSU t
:GLQG
673 3G GIOZ l
4.
GYNHUG
GRG S
UUQKK K YZU S
ULLYKZ
SUTOZU
AKMOUT
AKMOUT%
AKMOUT
AKMOUT
f F:
d IUTZ URRK
fln源自文库
dl
n
BZU S
%
(
)
%
(
)
3 Dm 3 D 3 D BC
d
l
su
l
4.
<A
SG K IK PUHKT TUZOLOIGZOUT R
<A
<A
<A
x
8OYZU
8OYZU
8OYZU
846B
=UZOL
8OYZU
YK K
h
n
p
s
dph
crontab
r
dl
l
od
1.
ft
d
t
1.
ft
d
t t
1.
ft
d
1.
ft
d
1.
f
d
1.
f
d
d
1.
f
d
1.
f 6R SKd
d
%
.
Yn
q Yn
0
Yn
FUU:KK K
F:
F:%
3R YZK
a F:(
e
<GYZK GIZO K
failover
<GYZK YZGT H
metrics
C2
( d(
8O K-<A-B G Q/ - -
B dp
(
-
-
B
d+ p
KV (HBase)
) d)p ( p=GSKB GIK )
%iB
OLAP
B
p4GZGBU IK
-
-B
(
p9T K
DOING & TO DO
Ø 1. Ø 2. Ø 3. Ø 4.
d d
1.
ud
ud
->
->
d sw p
Flume
metrics
metrics
metrics
Worker1
Worker2
Worker3
1.
f
l
d
d
d
1.
f
l
d
d
d
1.
f
l
d
d
d
2.
2.
ü
/
ü
ü
2.
2.
2.
2.
2.
d
su
wh
l
wh
<A B G Q B G Q BZ KGSOTM
BZU S :GLQG
82GYK
4O
5B
f
E1A=
Y K OYU
U QK P S SKZ OI :GLQG
NUKTO
RUMYO K ULLYKZ
UUQKK K bQGLQGc
4.
4.
4.
4.
DOING & TO DO
-
• Hive metaserver
bonecp&.)2maxConnectionsPerPartition=1
• Spark Streaming & Druid System CPU
h
+ "(
$#
%
MTTR
1
+
,
&
SLA 00
DOING & TO DO
zbDataCloudc
Kafka
CBT
v OLAP
Hive
Sqoop
Spark SQL
Spark Streaming
MLLib
ES
MapReduce
Spark Core
Kylin
YARN
Druid
6GO BINK RK f p
p UURd
g
d
jg
- HGIQ U Z L US % +g s p
d
g
p
d
73
f
d
OSOZ/ G GRRKR73CN KG Y/+g
73COSK OSOZ/ 738KG 6 KK
3M U
:GLQG
3R YZK
AKMOUT
AKMOUT%
AKMOUT
AKMOUT
3R YZK %
Libra Storm
Flume
HBase
Alluxio HDFS
Cassan dra
Zookeeper
Web
app
n
WebServer
Http
LogServer
Flume
FTP
Push
n
Kafka
Flume/DataFlow
HDFS/Hive
DB
Storm/Spark Streaming
MR/Spark
Kafka
poll
n
HDFS/Hive
FTP
DB
p8G UU p
BZU S B b OH Gc j
BZU S
j
j
B GQ
4 O 5Bj j
2013.10
2014.6
2015.5 2014.12
2015.12
2016.7
2017.3
j BZU S :GLQG
8G UU % 0%
j
(
(i
di
( C2
'"vm.zone_reclaim_mode=0
•
System CPU
echo never >/sys/kernel/mm/transparent_hugepage/defrag
-
•
/
fp
• Hive metaserver • yUser/IP
• skipTrash f
p dl
fRPC
d
-
HDFS
• Cache %$ ! • Shuffle %$ !
+-
0-
• Task*. • /Task !
• +Overhead ! • Reduce ! • JDBC !Kafka*
!
• Shuffle • HDFS Commit/ • Executor • Spill ! • 1!Parquet
•
d
: Hive unix_timestamp
•
• YARN • YARN
1 p
• Federation + Alluxio
DOING & TO DO
U QK
NUKTO
82GYK
4.
<A
l
4.
<A
4.
<A
4.
'(
CPG EG JMG
-
NHDI A
Y GQ
-
-
-
-
-
Huatuo-server
-
-
4.
Y GQ
,%
• # • HDFS, !
• Driver/Executor- !
• ,
+) • RDD Size Estimator*) •
4.
Y GQ
4.
Y GQ
3GINK
4.
Y GQ
CGYQ
4.
Y GQ
BN LLRK
4.
> Z>L<KSU t
:GLQG
673 3G GIOZ l
4.
GYNHUG
GRG S
UUQKK K YZU S
ULLYKZ
SUTOZU
AKMOUT
AKMOUT%
AKMOUT
AKMOUT
f F:
d IUTZ URRK
fln源自文库
dl
n
BZU S
%
(
)
%
(
)
3 Dm 3 D 3 D BC
d
l
su
l
4.
<A
SG K IK PUHKT TUZOLOIGZOUT R
<A
<A
<A
x
8OYZU
8OYZU
8OYZU
846B
=UZOL
8OYZU
YK K
h
n
p
s
dph
crontab
r
dl
l
od
1.
ft
d
t
1.
ft
d
t t
1.
ft
d
1.
ft
d
1.
f
d
1.
f
d
d
1.
f
d
1.
f 6R SKd
d
%
.
Yn
q Yn
0
Yn
FUU:KK K
F:
F:%
3R YZK
a F:(
e
<GYZK GIZO K
failover
<GYZK YZGT H
metrics
C2
( d(
8O K-<A-B G Q/ - -
B dp
(
-
-
B
d+ p
KV (HBase)
) d)p ( p=GSKB GIK )
%iB
OLAP
B
p4GZGBU IK
-
-B
(
p9T K
DOING & TO DO
Ø 1. Ø 2. Ø 3. Ø 4.
d d
1.
ud
ud
->
->
d sw p
Flume
metrics
metrics
metrics
Worker1
Worker2
Worker3
1.
f
l
d
d
d
1.
f
l
d
d
d
1.
f
l
d
d
d
2.
2.
ü
/
ü
ü
2.
2.
2.
2.
2.
d
su
wh
l
wh
<A B G Q B G Q BZ KGSOTM
BZU S :GLQG
82GYK
4O
5B
f
E1A=
Y K OYU
U QK P S SKZ OI :GLQG
NUKTO
RUMYO K ULLYKZ
UUQKK K bQGLQGc
4.
4.
4.
4.
DOING & TO DO
-
• Hive metaserver
bonecp&.)2maxConnectionsPerPartition=1
• Spark Streaming & Druid System CPU