VMware 透明NUMA--vNUMA设置

VMware 透明NUMA（vNUMA)设置
测试的机器是4路，16核心，加上超线程，总计128个逻辑CPU；
当设置了超过8颗core，会自动打开vNUMA，这里加上超线程，需要设置超过16个vCPU，才会打开vNUMA；
设置操作系统透明NUMA，需要关闭CPU热添加，如果打开CPU热添加，系统里只会认到一个NUMA节点,这个时候如果这个节点vCPU的数量超过物理机一个NUMA节点，就会跨NUMA使用内存，对计算型的系统来说会导致延迟；另外某些软件可以利用NUMA架构，更好的调度CPU和内存，这个时候就需要将物理的NUMA架构透传给虚拟机，使得虚拟机可以正确识别NUMA架构；
测试一、
系统识别到的NUMA节点数量跟虚拟硬件设置的插槽数量无关，这里设置了2插槽，每个插槽32个vCPU，但是虚拟NUMA（vNUMA)有4个；
从下面vmware.log里可以看到，有4个NUMA节点，每个节点16颗CPU，总计64颗CPU
cat vmware.log |grep -i numa
2020-06-11T00:38:13.218Z| vmx| I125: numaHost: NUMA config: consolidation= 1 preferHT= 0 2020-06-11T00:38:13.218Z| vmx| I125: numa: coresPerSocket= 32 maxVcpusPerVPD= 16
2020-06-11T00:38:13.218Z| vmx| A100: ConfigDB: Setting numa.autosize.cookie = "640001" 2020-06-11T00:38:13.218Z| vmx| A100: ConfigDB: Setting numa.autosize.vcpu.maxPerVirtualNode = "16"
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: 64 VCPUs 4 VPDs 4 PPDs
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 0 VPD 0 PPD 0
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 1 VPD 0 PPD 0
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 2 VPD 0 PPD 0
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 3 VPD 0 PPD 0
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 4 VPD 0 PPD 0
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 6 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 7 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 8 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 9 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 10 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 11 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 12 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 13 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 14 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 15 VPD 0 PPD 0 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 16 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 17 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 18 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 19 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 20 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 21 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 22 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 23 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 24 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 25 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 26 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 27 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 28 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 29 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 30 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 31 VPD 1 PPD 1 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 32 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 33 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 34 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 35 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 36 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 37 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 38 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 39 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 40 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 41 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 42 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 43 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 44 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 45 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 46 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 47 VPD 2 PPD 2 2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 48 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 50 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 51 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 52 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 53 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 54 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 55 VPD 3 PPD 3
2020-06-11T00:38:13.219Z| vmx| I125: numaHost: VCPU 56 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 57 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 58 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 59 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 60 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 61 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 62 VPD 3 PPD 3
2020-06-11T00:38:13.220Z| vmx| I125: numaHost: VCPU 63 VPD 3 PPD 3
在客户机系统里可以看到有4个NUMA节点
vNUMA架构在虚拟机第一次启动就生成了，vMotion迁移到不通架构的物理机上并不会导致vNUMA架构发生变化；只有虚拟CPU的数量发生了变化，才会重新生成新的vNUMA 架构，如果需要每次重启都生成vNUMA架构，需要设置numa.autosize.one=FALSE；
测试二、
现在测试设置64核心，4插槽，但是设置两个虚拟NUMA节点，并将NUMA亲和性设置成NODE0和NODE1，可以看到虚拟NUMA和插槽设置无关
cat vmware.log |grep -i numa
2020-06-11T00:57:10.262Z| vmx| I125: DICT numa.autosize.cookie = "640001"
2020-06-11T00:57:10.262Z| vmx| I125: DICT numa.autosize.vcpu.maxPerVirtualNode = "32" 2020-06-11T00:57:10.262Z| vmx| I125: DICT numa.nodeAffinity = "0,1"
2020-06-11T00:57:10.279Z| vmx| I125: numaHost: NUMA config: consolidation= 1 preferHT= 1 2020-06-11T00:57:10.279Z| vmx| I125: numaHost: 64 VCPUs 2 VPDs 2 PPDs
从vmware.log里可以看到64和虚拟CPU，两个虚拟NODE，两个物理NODE
从操作系统里可以看到只有两个NUMA节点，但是因为设置了4个插槽，/proc/cpuinfo里可以看到4个CPU插槽
测试三、
测试生产vNUMA最少CPU，分别测试了16vCPU和18vCPU，可以看到18vCPU正确生产了vNUMA
8C，16vCPU的情况
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: 16 VCPUs 1 VPDs 1 PPDs
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 0 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 1 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 2 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 3 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 4 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 5 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 6 VPD 0 PPD 0
2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 7 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 8 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 9 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 10 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 11 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 12 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 13 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 14 VPD 0 PPD 0 2020-06-11T01:55:46.045Z| vmx| I125: numaHost: VCPU 15 VPD 0 PPD 0 9 CORE，18VCPU的情况
2020-06-11T01:57:14.179Z| vmx| I125: numaHost: 18 VCPUs 2 VPDs 2 PPDs 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 0 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 1 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 2 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 3 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 4 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 5 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 6 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 7 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 8 VPD 0 PPD 0 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 9 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 10 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 11 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 12 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 13 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 14 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 15 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 16 VPD 1 PPD 1 2020-06-11T01:57:14.179Z| vmx| I125: numaHost: VCPU 17 VPD 1 PPD 1。