高性能计算CUDA-opt

Instruction Cache Scheduler Dispatch Register File Core Core Core Core Core Core Core Core Core Core Core Core Scheduler Dispatch
Execution unit
32 CUDA Cores
Warp and SIMT
32 Threads
...
Block
=
32 Threads 32 Threads
Warps
• Blocks divide into groups of 32
Warp Scheduler 0
warp 8 instruction 11 warp 2 instruction 42 warp 14 instruction 95
SFU, LSU
Memory
Registers: 32bit, 32K 32-bit Cache
L1+shared memory (64 KB) Texture (8 KB) Constant (8 KB)
© NVIDIA Corporation 2010
Load/Store Units x 16 Special Func Units x 4 Interconnect Network 64K Configurable Cache / Shared Mem Uniform Cache
Latency bound
<<
Memory optimization
~
Latency optimization
Done!
© NVIDIA Corporation 2010
Latency Optimization
© NVIDIA Corporation 2009
Latency Optimization
. . .
warp 9 instruction 12 warp 3 instruction 34 warp 15 instruction 96
threads called warps Warps are basic scheduling units Warps always perform the same instruction (Single Instruction Multiple Threads, SIMT) • A lot of warps can hide memory latency
Latency=4 Throughput=2 Concurrency=8
output
Little’s Law
Concurrency = Throughput * Latency
Basic units (instruction/cache line) that are active in the pipeline: “on-the-fly” C2050: 150 GB/s * (600 / 1.15 GHz) / 128 B / 14 SM ~ 44 cache line per SM on the fly
Adjust resource usage to change occupancy:
Measure effective memory/instruction throughput
Optimize for peak memory/instruction throughput
Finding out the bottleneck Typically an iterative process
Major techniques: adjust resource usage to increase threads
© NVIDIA Corporation 2010
Understanding Latency Bound
input
Hardware (inst/mem) is pipelined
Latency: “length” of the pipeline Throughput: “width” of the pipeline
Warp Scheduler 1
warp 9 instruction 11 warp 3 instruction 33 warp 15 instruction 95
. . .
warp 8 instruction 12 warp 14 instruction 96 warp 2 instruction 43
Global Memory
© NVIDIA Corporation 2010
General Optimization Strategies: Measurement
Find out the limiting factor in kernel performance
Memory bandwidth bound (memory optimization) Instruction throughput bound (instruction optimization) Latency bound (configuration optimization)
• •
• Context switching is free
Time
© NVIDIA Corporation 2010
Fermi Memory Hierarchy
3 levels, very similar to CPU Register
Spills to local memory
Caches
Shared memory L1 cache L2 cache Constant cache Texture cache
Global memory
© NVIDIA Corporation 2010
Fermi Memory Hierarchy Review
SM-0
Registers
C L1& SMEM TEX C
SM-1
Registers
L1& SMEM TEX C
SM-N
Registers
L1& SMEM TEX
L2
Threads are executed by scalar processors
Multiprocessor
A kernel is launched as a grid of thread blocks
...
Grid
© NVIDIA Corporation 2010
Up to 16 kernels can execute on a device at one time Device
Maximum number: 48 in Fermi
We need the occupancy to be high enough to hide latency
© NVIDIA Corporation 2010
What Limits Occupancy?
Partitioning of SM Resources
© NVIDIA Corporation 2010
Example: Little’s Law on GPU
Achieved bandwidth as a function of requests per SM
© NVIDIA Corporation 2010
Enough Blocks
# of blocks >> # of SM > 100 to scale well to future device Block size should be a multiple of 32 (warp size) Minimum: 64. I generally use 128 or 256. But use whatever is best for your app. Depends on the problem, do experiments!
When the code is latency bound
Both the memory and instruction throughputs are far from the peak
Latency hiding: switching threads Purpose: have enough threads to hide latency
CUDA Optimization
Peng Wang, Ph.D.
HPC Developer Technology, NVIDIA
© NVIDIA Corporation 2010
Optimization Overview
GPU architecture Kernel optimization
Latency optimization Memory optimization Instruction optimization
© NVIDIA Corporation 2010
Occupancy Optimizations
Know the current occupancy
Visual profiler --ptxas-options=-v: output resource usage info; input to Occupancy Calculator
Data transfer optimization
Overlapped execution using streams
© NVIDIA Corporation 2010
GPU Architecture
© NVIDIA Corporation 2009
GPU High Level View
SMEM SMEM SMEM SMEM
© NVIDIA Corporation 2010