[KV Cache][Feature] Support Layerwise KV Pooling (#10077)
Limitations of the Previous Layerwise Design
1. Per-Layer Key Management

Each KV cache block in every layer was assigned a unique key.
The number of keys increased proportionally with the number of layers
and blocks, resulting in significant metadata management overhead.

2. High Metadata Lookup Latency

Both put and get operations required querying the Meta Server to
resolve the memory address associated with a key.
Frequent metadata lookups introduced additional latency on the
critical access path.

3. Inefficient HBM Address Computation

HBM addresses were computed using Python lists.
This approach incurred noticeable CPU overhead when processing large
numbers of blocks.


Optimizations
1. Unified Key Management

Keys are now managed at the request ID level.
KV cache blocks belonging to different layers but the same request
share a single key.
This significantly reduces the number of keys and metadata management
overhead.

2. One-Time Address Resolution

Key-to-address mapping is performed only once during initialization.
Subsequent accesses locate layer-specific blocks using a base address
plus a layer offset.
This eliminates repeated Meta Server lookups during runtime.

3. New Address-Based Memcache Interface

Introduced a new batch_copy API in Memcache.
Memory can be accessed directly through physical addresses without key
resolution.
This reduces access latency and improves batch transfer efficiency.

4. Accelerated HBM Address Computation

Replaced Python list-based address computation with NumPy vectorized
operations.
This reduces CPU overhead and improves address generation efficiency.

5. CPU Affinity Optimization

Applied CPU affinity binding to Memcache worker threads.
This minimizes thread migration, reduces cache misses, and improves
scheduling efficiency.

6. Controlled Overlap Between Data Transfer and Attention
Computation

Limited the degree of concurrency between KV cache transfers and
Attention computation.
This reduces resource contention and minimizes interference with the
inference execution path.

Results
The optimized design delivers the following benefits:

Reduced metadata lookup overhead.
Lower access latency through address-based memory operations.
Improved HBM address computation efficiency.
Reduced CPU overhead through vectorized computation and CPU affinity
optimization.
Lower runtime contention between KV cache transfers and model
execution.
Improved overall Layerwise KV Cache Pool performance and scalability.

DeepSeek V3.2 TP16 hit rate 90%
| kv cache pool | Input tokens | Output tokens | Concurrency | Average
TTFT |
|—-|—-|—-|—-|—-|
| no-layerwise | 64k | 1 | 36.9 | 312519.2 ms |
| layerwise | 64k | 1 | 36.9 | 191323.1 ms |
| no-layerwise | 32k | 1 | 37.1 | 149498.4 ms |
| layerwise | 32k | 1 | 36.8 | 64203.3 ms |
| HBM hit | 32k | 1 | 37.1 | 64276.3 ms |
What this PR does / why we need it?
Does this PR introduce any user-facing change?
How was this patch tested?

vLLM version: v0.22.1
vLLM main:
https://github.com/vllm-project/vllm/commit/967c5c3bc38891f4465d3f4e99917ed837bb3833


Signed-off-by: F.Liu 1661888967@qq.com
Co-authored-by: luokui183 luokui@huawei.com
Co-authored-by: 520xie 1023005685@qq.com
Co-authored-by: wangxiaochao6 wangxiaochao6@hisilicon.com
Co-authored-by: zhangsicheng5 zhangsicheng5@huawei.com
Co-authored-by: tyy0829 1455207791@qq.com
Co-authored-by: Pz1116 zpbzpb123123@gmail.com
Co-authored-by: DreamerLeader 2270923832@qq.com