海量日志采集、解析实践

海量日志采集、解析实践

01

02

03

04

05

01.自我介绍

邓小刚

奇安信网络安全部

负责内部安全大数据平台架构、实施和维护20年以上IT及安全领域从业经验，2007年起从事SIEM

学习及实践2007年获得ArcSight AEIA/AESA认证

2011年帮助公司首家获得中国ArcSight Service Partner认证并成为第一个认证讲师

主要参与并负责了如下知名企业的SIEM相关项目实施

香港证券交易所培训服务：ArcSight ESM、Logger、Flex

招商银行安全运营平台建设：ArcSight ESM、Logger

世纪互联MSSP平台搭建及运营：ArcSight ESM

中国石化安全管理平台建设：ArcSight ESM

神华集团安全监控平台建设：ArcSight ESM Appliance

厦门国际银行日志管理平台建设：ArcSight Logger

包商银行日志审计平台建设：ArcSight Logger

安利中国专业服务：ArcSight ESM

太平洋保险SIEM平台咨询服务：McAfee ESM

灵活快速适应日志格式变化量大难以横向扩容昂贵按E PS 计费难以承受度量日志有效性监测

02.挑战

02.应对

虚拟化集群支持横向扩容开源产品自我可控灵活

按需解析、

规范化格式富化

增加监测点

03.采集

-Linux imuxsock imfile impstats

omrelp

03.采集

-Windows WinlogBeat FileBeat PacketBeat

Logstash

03.采集

-DB/API JDBC Rest

KAFKA

03.采集

-SFTP Shell Python SFTP

Bat/Powershell

03.采集-过滤/转换/富化

原始日志负载均衡

过滤/转换/富化初级原始日志消息队列

{

"method": "GET","path": "/xxxx ","format": "*/*",

"controller": "Projects::GitHttpController","time": "2020-07-22T04:38:19.394Z","params": [{

"key": "service",

"value": "git-upload-pack "},{

"key": "namespace_id","value": "xxxxx "},{

"key": "project_id","value": "xxxxx.gi t"}],

"remote_ip": "xx.xx.xx.xx ","user_id": xxxx ,"username": "xxxx ",

{

"@timestamp ": "2020-07-22 12:38:19 +0800","@timegenerated ": "2020-07-22 12:38:19 +0800","fromhost-ip ": ”xxx.xxx.xxx.xxx ","myhostname ": "xxx.xxx.xxx.xxx.xxx ","syslog-tag ": "git.production_json","inputname ": "imrelpxxxx","message ":{xxxxxxxxx }}

初级富化

日志检索

富化数据

"xx": "xx网段"

{

"name": "GET",

"sourceAddress ": "xxx.xxx.xxx.xxx ",

"sourceZoneURI ": "xx-xx.xx.xx.xx_xx.xx.xx.xx_xx产品部","requestMethod": "GET",

"requestClientApplication": "git/2.21.0 (Apple Git-122.2)","deviceReceiptTime": "1595392699394","destinationUserName ": "xxxx ","deviceSeverity": "200","requestUrl": "/xxxx",

"deviceCustomString1": "git-upload-pack ","deviceCustomString2": "xxxxx ","deviceCustomString3": "xxxxx",

"deviceEventCategory": "git.production_json","deviceFacility": "Projects::GitHttpController","fileType": "*/*",}

关联分析{

"method": "GET","path": "/xxxx ","format": "*/*",

"controller": "Projects::GitHttpController","time": "2020-07-22T04:38:19.394Z","params": [{

"key": "service",

"value": "git-upload-pack "},{

"key": "namespace_id","value": "xxxxx "},{

"key": "project_id","value": "xxxxx.gi t"}],

"remote_ip": "xx.xx.xx.xx ","user_id": xxxx ,"username": "xxxx ",

关联数据

仓库/部门映射表