Netty直接内存（堆外内存）溢出分析

Netty直接内存（堆外内存）溢出分析
问题描述
通过压测，发现系统最后会停⽌响应，不再接收新的请求。

查看⽇志，发现有如下申请直接内存错误。

ty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2130706439, max: 2147483648)
at ty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:742) ~[netty-all-4.1.47.Final.jar!/:4.1.47.Final]
at ty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:697) ~[netty-all-4.1.47.Final.jar!/:4.1.47.Final]
at ty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.PoolArena.allocate(PoolArena.java:215) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.PoolArena.allocate(PoolArena.java:147) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:356) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139) ~[netty-all-4.1.47.Final.jar!/:?]
at ty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[netty-all-4.1.47.Final.jar!/:?] at ty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147) [netty-all-4.1.47.Final.jar!/:?]
at ty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-all-4.1.47.Final.jar!/:?]
at ty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-all-4.1.47.Final.jar!/:?]
at ty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-all-4.1.47.Final.jar!/:?]
at ty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.47.Final.jar!/:?]
at ty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.47.Final.jar!/:?]
at ty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.47.Final.jar!/:4.1.47.Final]
at ty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.47.Final.jar!/:?]
at ng.Thread.run(Thread.java:748) [?:1.8.0_191]
问题分析
堆内存
⾸先怀疑是堆内存泄露，通过jmap命令查看堆内存使⽤情况，发现使⽤正常。

Xmx设置是8G，只使⽤了954M
如果确认是堆溢出，使⽤命令jmap -dump:format=b,file=heap2.hprof 28624将堆快照dump出来，再使⽤MAT/JProfiler等⼯具分析，同时可以使⽤jConsole和JVisualVM监控堆内存分配情况。

jmap -heap 17667
Attaching to process ID 17667, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.191-b12
using thread-local object allocation.
Garbage-First (G1) GC with 4 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 8589934592 (8192.0MB)
NewSize = 1363144 (1.2999954223632812MB)
MaxNewSize = 5152702464 (4914.0MB)
OldSize = 5452592 (5.1999969482421875MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 268435456 (256.0MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 2097152 (2.0MB)
Heap Usage:
G1 Heap:
regions = 4096
capacity = 8589934592 (8192.0MB)
used = 1001361104 (954.9723663330078MB)
free = 7588573488 (7237.027*********MB)
11.657377518713474% used
G1 Young Generation:
Eden Space:
regions = 54
capacity = 1847590912 (1762.0MB)
used = 113246208 (108.0MB)
free = 1734344704 (1654.0MB)
6.129398410896708% used
Survivor Space:
regions = 0
capacity = 0 (0.0MB)
used = 0 (0.0MB)
free = 0 (0.0MB)
0.0% used
G1 Old Generation:
regions = 429
capacity = 1113587712 (1062.0MB)
used = 888114896 (846.9723663330078MB)
free = 225472816 (215.0276336669922MB)
79.75257686751486% used
43044 interned Strings occupying 5008680 bytes.
堆外内存
分析堆外内存。

通过设置-Xmx参数只能限制java进程中Java Heap的内存⼤⼩。

⽽java进程的内存是由Java Heap、Class、Thread、Code、GC、Internal、Symbol、Native Memory Tracking、unknown这⼏部分组成。

⽐如当系统⽹络请求过⼤时，Internal部分的内存会显著上升，占⽤⼤量内存。

实际场景中，要结合top，jmap，NMT⼯具对java进程的内存进⾏分析。

NMT的全称是Native Memory Tracker ，是⼀个本地内存跟踪⼯具。

常⽤来分析JVM的内存使⽤情况。

NMT功能默认关闭，可以通过以下⽅式开启:
-XX:NativeMemoryTracking=[off | summary | detail]
配置项说明
off 默认配置
summary 只收集汇总信息
detail 收集每次调⽤的信息
注意，根据Java官⽅⽂档，开启NMT会有5%－10%的性能损耗；
如果想JVM退出时打印退出时的内存使⽤情况，可以通过如下配置项:
-XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics
通过jcmd查看堆NMT内存、类内存、内部内存等详细信息
jcmd 17667 VM.native_memory detail scale=MB >temp.txt
17667:
Native Memory Tracking:
Total: reserved=15306MB, committed=11529MB
- Java Heap (reserved=8192MB, committed=5678MB)
(mmap: reserved=8192MB, committed=5678MB)
- Class (reserved=1146MB, committed=138MB)
(classes #23198)
(malloc=4MB #56498)
(mmap: reserved=1142MB, committed=134MB)
- Thread (reserved=461MB, committed=461MB)
(thread #458)
(stack: reserved=459MB, committed=459MB)
(malloc=2MB #2292)
(arena=1MB #912)
- Code (reserved=265MB, committed=118MB)
(malloc=21MB #25211)
(mmap: reserved=244MB, committed=97MB)
- GC (reserved=409MB, committed=315MB)
(malloc=73MB #151121)
(mmap: reserved=336MB, committed=243MB)
- Compiler (reserved=2MB, committed=2MB)
(malloc=2MB #3347)
- Internal (reserved=4778MB, committed=4778MB)
(malloc=4778MB #218361)
- Symbol (reserved=27MB, committed=27MB)
(malloc=23MB #249386)
(arena=4MB #1)
- Native Memory Tracking (reserved=11MB, committed=11MB)
(tracking overhead=11MB)
- Unknown (reserved=16MB, committed=0MB)
(mmap: reserved=16MB, committed=0MB)
Virtual memory map:
[0x00000005c0000000 - 0x00000007c0000000] reserved 8192MB for Java Heap from
[0x00007fce6c249efb] ReservedSpace::initialize(unsigned long, unsigned long, bool, char*, unsigned long, bool)+0xab
[0x00007fce6c24a4e2] ReservedHeapSpace::ReservedHeapSpace(unsigned long, unsigned long, bool, char*)+0x52
[0x00007fce6c216f65] Universe::reserve_heap(unsigned long, unsigned long)+0x65
[0x00007fce6bd6e681] G1CollectedHeap::initialize()+0x151
发现Internal区域很⼤。

可以通过如下⽅式，对⽐内存变化。

⼀般对于堆外内存缓慢增长直到爆炸的情况来说，可以先设⼀个基线jcmd pid VM.native_memory baseline。

然后等放⼀段时间后再去看看内存增长的情况，通过jcmd pid VM.native_memory detail.diff(summary.diff)做⼀下 summary 或者 detail 级别的 diff。

这边堆外内存我们重点关注 Internal 的内存增长，如果增长⼗分明显的话那就是有问题了。

jcmd 17667 VM.native_memory baseline
jcmd 13527 VM.native_memory scale=MB detail.diff --包含具体的内存分配信息
jcmd 13527 VM.native_memory scale=MB summary.diff --只包含内存概要信息
通过如上步骤基本确认是堆外内存泄露了。

为了进⼀步监控直接内存，可以使⽤如下代码，在启动时加载。

package netty.util;
import ng.reflect.Field;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.ReflectionUtils;
import ty.util.internal.PlatformDependent;
/*直接内存监控*/
public class DirectMemoryReporter {
private static final Logger LOGGER = LoggerFactory.getLogger(DirectMemoryReporter.class);
private static final int _1K = 1024;
private static final String BUSINESS_KEY = "netty_direct_memory";
private AtomicLong directMemory;
public void init() {
Field field = ReflectionUtils.findField(PlatformDependent.class, "DIRECT_MEMORY_COUNTER");
field.setAccessible(true);
try {
directMemory = ((AtomicLong) field.get(PlatformDependent.class));
startReport();
} catch (IllegalAccessException e) {
LOGGER.error("netty_direct_memory error", e);
}
}
public void startReport() {
Runnable runnable = () -> {
doReport();
};
ScheduledExecutorService service = Executors.newSingleThreadScheduledExecutor();
// 第⼆个参数为⾸次执⾏的延时时间，第三个参数为定时执⾏的间隔时间
service.scheduleAtFixedRate(runnable, 30, 1, TimeUnit.SECONDS);
}
private void doReport() {
try {
int memoryInKB = (int) (directMemory.get());
("{}:{}KB", BUSINESS_KEY, memoryInKB/1024);
} catch (Exception e) {
LOGGER.error("netty_direct_memory error", e);
}
}
}
可以通过new DirectMemoryReporter().init()启动。

之后每隔⼀秒可以在⽇志中看到直接内存的占⽤情况。

2021-03-02 09:58:59,073|[INFO]|[pool-11-thread-1]|[]|xxx.DirectMemoryReporter|netty_direct_memory:2032MB(2130706439B)
通过监控，发现随着压测，直接内存占⽤会不断上涨，直到涨到Xmx设置的值（如果不设置-XX:MaxDirectMemorySize默认与Xmx相同）
定位
上⾯基本确认了是堆外直接内存泄露问题，下⾯要具体定位是哪⾥泄露的，需要使⽤到pmap(查看内存分配情况)、gdb(根据可能的内存泄露地址解析出实际内容)、strace(分析线程的内存分配情况)、jstack(定位线程的具体信息)
注意，除了pmap出来的信息⼤⼩⽐较⼩，strace和jstack的⽇志可能⽐较⼤，以当前定位为例，最终jstack⽇志6G、strace⽇志1G，传到本地，⽂本编辑器打不开。

因此尽量在linux环境下进⾏操作。

1、启动内存pmap、内存分配strace、进程jstack跟踪
先使⽤pmap -X pid >pmap1.txt 保存当前内存情况，同时启动strace和jstack跟踪
当跑性能跑出溢出后，再使⽤pmap -X pid >pmap2.txt 保存溢出时的内存分配
pmap
pmap -X pid >> pmap1.txt
strace
在压测前执⾏该命令，并保持会话
strace -o ⽇志 -p pid -Ff -Tt -e trace=brk,mmap,munmap
jstack
通过脚本test.sh，每隔⼀段时间打印⼀次线程快照
#!/bin/bash
while [ 1 ];
do
jstack -l 16067 >> /opt/jstack_0301.txt
sleep 10
done
./test.sh &
通过对⽐找出可能的泄露的内存地址
image-20210302112208575
通过分析
7f4674483000 rw-p 00000000 00:00 0 1575296 1574868 1574868 1535980 1574868 0 0
这⼀⾏增加的数据很⼤
2、gdb获取可能泄露的内存
通过偏移计算，使⽤gdb获取可能泄露的内存的内容,此步需要确认泄露的内存地址，如果内容中有时间，将时间计算出来
gdb
结束地址可以直接使⽤10进制设置偏移，如起始地址是0x7f4674483000，偏移1G 即偏移1x1024x1024x1024（1073741824），则结束地址为0x7f4674483000+1073741824，注意地址要以0x开头，pmap出来的地址省略了0x
gdb attach pid
c
dump memory ⽂件名开始地址结束地址
打开⽂件就可以看到解析的内存内容
可以通过vi/hexdump命令查看
下⾯是实例
[root@host-x-x-x-x ~]# gdb attach 16067
GNU gdb (GDB) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later </licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
···省略
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
--Type <RET> for more, q to quit, c to continue without paging--c #此处输⼊c，并回车，则进⼊gdb
(gdb) dump memory out.bin 0x7f4674483000 0x7f4674483000+1073741824 #解析对应内存内容，⼀般需要计算偏移，如64M
(gdb) q #输⼊q，退出
A debugging session is active.
Inferior 1 [process 16067] will be detached.
Quit anyway? (y or n) y #输⼊y，退出
Detaching from program: /root/jdk-8u191/bin/java, process 16067
[Inferior 1 (process 16067) detached]
解析后的⽂件名为out.bin，打开后可以查看解析后的内容
[root@host-x-x-x-x ~]# ls -alh out.bin
-rw------- 1 root root 1.0G Mar 2 11:01 out.bin
[root@host-10-33-144-227 ~]# vi out.bin
^@^@^@^@^@^@^@^@^B^P^@^A^@^@^@^@^@^@^@^A^@^@^@^@^G^@^@ó^@^@^@^@DATA^M
2 xxx@xxx^M
3 To: xxx^M
4 Message-ID: <651317388.574761.1614644538528.JavaMail.root@host-xxx>^M
5 Subject: =?UTF-8?B?5rWL6K+V6YKu5Lu2?=^M
6 MIME-Version: 1.0^M
7 Content-Type: multipart/mixed; ^M
8 boundary="----=_Part_574760_1068795689.1614644538528"^M
9 ^M
10 ------=_Part_574760_1068795689.1614644538528^M
11 Content-Type: text/plain; charset=UTF-8^M
12 Content-Transfer-Encoding: base64^M
13 ^M
14 6L+Z5piv5LiA5p2h5rWL6K+V6YKu5Lu277yBClRoaXMgaXMgYSB0ZXN0IGVtYWlsIQDATA^
[root@host-10-33-144-227 ~]# hexdump -C out.bin #使⽤⼗六进制查看
00219420 41 55 54 48 20 4c 4f 47 49 4e 0d 0a 40 68 6d 61 |AUTH LOGIN..@xx|
00219430 69 6c 2e 68 75 61 77 65 69 2e 63 6f 6d 0d 0a 54 |..T|
00219440 6f 3a 20 6c 68 6e 39 34 31 34 40 68 6d 61 69 6c |o: xxx|
00219450 2e 68 75 61 77 65 69 2e 63 6f 6d 0d 0a 4d 65 73 |..Mes|
00219460 73 61 67 65 2d 49 44 3a 20 3c 31 33 38 38 32 34 |sage-ID: <138824|
00219470 33 34 39 2e 35 38 31 31 31 33 2e 31 36 31 34 36 |349.581113.16146|
00219480 34 34 39 35 33 32 32 30 2e 4a 61 76 61 4d 61 69 |44953220.JavaMai|
00219490 6c 2e 72 6f 6f 74 40 68 6f 73 74 2d 31 30 2d 33 |l.root@host-10-3|
002194a0 33 2d 31 34 34 2d 31 39 30 3e 0d 0a 53 75 62 6a |3-144-190>..Subj|
002194b0 65 63 74 3a 20 3d 3f 55 54 46 2d 38 3f 42 3f 35 |ect: =?UTF-8?B?5|
002194c0 72 57 4c 36 4b 2b 56 36 59 4b 75 35 4c 75 32 3f |rWL6K+V6YKu5Lu2?|
002194d0 3d 0d 0a 4d 49 4d 45 2d 56 65 72 73 69 6f 6e 3a |=..MIME-Version:|
002194e0 20 31 2e 30 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 | 1.0..Content-Ty|
002194f0 70 65 3a 20 6d 75 6c 74 69 70 61 72 74 2f 6d 69 |pe: multipart/mi|
00219500 78 65 64 3b 20 0d 0a 09 62 6f 75 6e 64 61 72 79 |xed; ...boundary|
00219510 3d 22 2d 2d 2d 2d 3d 5f 50 61 72 74 5f 35 38 31 |="----=_Part_581|
00219520 31 31 32 5f 37 38 38 34 33 32 36 31 34 2e 31 36 |112_788432614.16|
00219530 31 34 36 34 34 39 35 33 32 32 30 22 0d 0a 0d 0a |14644953220"....|
00219540 2d 2d 2d 2d 2d 2d 3d 5f 50 61 72 74 5f 35 38 31 |------=_Part_581|
00219550 31 31 32 5f 37 38 38 34 33 32 36 31 34 2e 31 36 |112_788432614.16|
00219560 31 34 36 34 34 39 35 33 32 32 30 0d 0a 43 6f 6e |14644953220..Con|
00219570 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 78 74 2f |tent-Type: text/|
00219580 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 74 3d 55 |plain; charset=U|
00219590 54 46 2d 38 0d 0a 43 6f 6e 74 65 6e 74 2d 54 72 |TF-8..Content-Tr|
002195a0 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e 67 3a |ansfer-Encoding:|
002195b0 20 62 61 73 65 36 34 0d 0a 0d 0a 36 4c 2b 5a 35 | base64....6L+Z5|
002195c0 70 69 76 35 4c 69 41 35 70 32 68 35 72 57 4c 36 |piv5LiA5p2h5rWL6|
002195d0 4b 2b 56 36 59 4b 75 35 4c 75 32 37 37 79 42 43 |K+V6YKu5Lu277yBC|
002195e0 6c 52 6f 61 58 4d 67 61 58 4d 67 59 53 42 30 5a |lRoaXMgaXMgYSB0Z|
002195f0 58 4e 30 49 47 56 74 59 57 6c 73 49 51 3d 3d 0d |XN0IGVtYWlsIQ==.|
00219600 0a 2d 2d 2d 2d 2d 2d 3d 5f 50 61 72 74 5f 35 38 |.------=_Part_58|
00219610 31 31 31 32 5f 37 38 38 34 33 32 36 31 34 2e 31 |1112_788432614.1|
00219620 36 31 34 36 34 34 39 35 33 32 32 30 2d 2d 0d 0a |614644953220--..|
00219630 2e 0d 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00219640 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00219820 45 48 4c 4f 20 68 6f 73 74 2d 31 30 2d 33 33 2d |EHLO host-xxx-|
00219830 31 34 34 2d 31 39 30 0d 0a 00 00 00 00 00 00 00 |xx-xx.........|
00219840 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
通过内容，确认数据时间(注意到秒，后⾯3位毫秒值去掉)，如
[root@host-x-x-x-x ~]# date -d @1614625596
Tue Mar 2 03:06:36 CST 2021
3、strace查看申请内存的pid
通过strace⽇志，查找上⼀步得到的泄露的内存地址是哪个进程申请的，通过搜索内存地址，可以查看是否该地址只被该进⾏mmap没有munmap，如果没有进⾏释放，则怀疑是其导致内存泄露
通过上⾯分析，内存0x7f4674483000确认发⽣泄露,查找泄露内存的线程
[root@host-x-x-x-x opt]# grep "0x7f4674483000" strace_0301.txt
25897 08:22:40.071076 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4674483000 <0.000524>
由查找结果，发现线程25897只使⽤了mmap申请内存（⼤⼩为16M），未进⾏释放munmap，存在泄漏
通过申请内存的⼤⼩16781312，我们可以过滤⼀下
[root@host-10-33-144-227 opt]# grep 16781312 strace_0301.txt |wc -l
96
[root@host-10-33-144-227 opt]# grep 16781312 strace_0301.txt
25900 22:48:40.293407 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46d34e2000 <0.001026>
25898 22:48:40.779514 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46d24e1000 <0.000931>
25895 22:48:42.072311 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25894 22:48:43.126104 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46d04df000 <0.000846>
25896 22:48:43.453893 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 22:48:54.240661 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25893 22:48:55.871836 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 22:48:58.918825 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46cc4db000 <0.000926>
25899 23:40:52.070042 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25900 23:40:52.076028 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46ca4d9000 <0.001435>
25894 23:40:53.138579 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25895 23:40:53.993883 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 23:40:54.021415 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46c74d6000 <0.000961>
25896 23:40:55.046603 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46c64d5000 <0.001035> 25898 23:41:01.209776 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46c54d4000 <0.001205> 25893 23:41:38.545986 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 00:32:43.584853 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 00:32:45.662573 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25900 00:32:46.247013 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25896 00:32:47.807808 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46c04cf000 <0.000738> 25894 00:32:48.105049 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46bf4ce000 <0.000832> 25895 00:32:48.646481 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46be4cd000 <0.000840> 25897 00:32:51.705296 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25893 00:33:31.796557 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 01:24:53.214635 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 01:24:54.268791 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25900 01:24:55.509481 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b94c8000 <0.001084> 25894 01:24:55.975972 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 01:24:56.313316 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b74c6000 <0.000877> 25896 01:24:57.998530 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b64c5000 <0.001148> 25895 01:25:02.973712 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25893 01:25:39.658599 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b44c3000 <0.000751> 25898 02:16:08.550683 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b34c2000 <0.000872> 25894 02:16:09.253646 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b24c1000 <0.000981> 25900 02:16:09.868975 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 02:16:10.299096 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46b04bf000 <0.000774> 25895 02:16:10.898468 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46af4be000 <0.000921> 25896 02:16:11.490967 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46ae4bd000 <0.000816> 25899 02:16:16.064957 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46ad4bc000 <0.001132> 25893 02:16:56.362348 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 03:07:37.304262 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46ab4ba000 <0.000832> 25899 03:07:37.985234 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46aa4b9000 <0.000840> 25895 03:07:40.860608 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46a94b8000 <0.001047> 25900 03:07:41.419694 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25896 03:07:43.400255 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 03:07:43.572092 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46a64b5000 <0.001165> 25894 03:07:49.068627 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46a54b4000 <0.001139> 25893 03:08:22.889127 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 04:00:37.946945 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46a34b2000 <0.001101> 25900 04:00:41.183475 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 04:00:41.470902 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46a14b0000 <0.000641> 25896 04:00:42.676048 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25894 04:00:43.223683 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f469f4ae000 <0.001021> 25897 04:00:45.400673 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f469e4ad000 <0.000564> 25895 04:00:46.895991 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25893 04:01:24.135505 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 04:52:44.120418 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f469b4aa000 <0.000958> 25894 04:52:46.038504 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f469a4a9000 <0.001086> 25900 04:52:46.404094 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46994a8000 <0.001060> 25896 04:52:46.780690 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25895 04:52:47.319693 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 04:52:47.591897 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 04:52:54.528811 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f46954a4000 <0.000924> 25893 04:53:27.404271 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 05:44:25.134003 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 05:44:26.525414 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25900 05:44:27.165630 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25894 05:44:28.995850 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f469049f000 <0.000962> 25896 05:44:31.020882 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 05:44:31.657577 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25895 05:44:33.198010 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f468d49c000 <0.000796> 25893 05:45:08.843821 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25898 06:37:21.368773 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f468b49a000 <0.000512> 25899 06:37:23.990442 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f468a499000 <0.000969> 25894 06:37:24.215510 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4689498000 <0.001021> 25897 06:37:26.174882 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25895 06:37:26.414563 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4687496000 <0.000920> 25900 06:37:26.507630 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25896 06:37:39.847076 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25893 06:37:51.091554 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25894 07:31:27.933343 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25896 07:31:29.163123 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4682491000 <0.000441> 25898 07:31:29.324907 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4681490000 <0.000298> 25900 07:31:31.294559 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25893 07:31:31.443325 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 07:31:39.104532 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f467e48d000 <0.000974> 25895 07:31:43.078283 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25897 07:31:52.607503 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f467c48b000 <0.001193> 25898 08:22:19.669250 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f467b48a000 <0.000921> 25900 08:22:20.093744 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f467a489000 <0.000832> 25896 08:22:22.625889 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4679488000 <0.001093> 25893 08:22:23.076138 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25894 08:22:27.951371 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25895 08:22:38.615218 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
25899 08:22:38.734819 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4675484000 <0.000974> 25897 08:22:40.071076 mmap(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4674483000 <0.000524>申请的总内存为1536.375MB(16781312*96=1611005952)，发现与第2步中分析可能泄露的内存
7f4674483000 rw-p 00000000 00:00 0 1575296 1574868 1574868 1535980 1574868 0 0
中的⼤⼩1575296/1024=1538.375MB，基本相等。

由此推断，这些内存均是泄露的内存。