[BugFix][xlite] Performance improvement in xlite full mode by avoiding redundant memory allocation during profiling runs (#11380)
What this PR does / why we need it?
xlite backend in xlite/xlite_model_runner:XliteModelRunner keep a
copy of NPUModelRunner‘s model for compatibility. In xlite full
mode, xlite handles both the prefill stage and the decode stage.
Before this PR, NPUModelRunner/aclgraph reserves peak memory during
profile runs, which reduced available memory for KV caches and affected
xlite perrformance.
With this PR, aclgraph model reserves less peak runtime memory in
xlite full mode. More memory can be allocated to KV caches and the
concurrency level is increased with some performance improvement. For
example, using TP2DP1EP2 to serve Qwen3-30B-A3B with xlite full mode
(max seq 64, max length 40960, batch tokens 8192):
Before this PR:

(Worker_TP0_EP0 pid=18480) INFO 07-03 15:23:17 [xlite.py:592] xlite
runtime pool size: 519 MB
(Worker_TP0_EP0 pid=18480) INFO 07-03 15:23:35 [worker.py:564]
Available KV cache memory: 25.53 GiB
(EngineCore pid=18453) INFO 07-03 15:23:35 [kv_cache_utils.py:1744]
GPU KV cache size: 557,696 tokens
(EngineCore pid=18453) INFO 07-03 15:23:35 [kv_cache_utils.py:1745]
Maximum concurrency for 40,960 tokens per request: 13.62x
(Worker_TP0_EP0 pid=18480) INFO 07-03 15:23:37 [worker.py:790] Free
memory on device (60.89/61.27 GiB) on startup. Desired GPU memory
utilization is (0.95, 58.21 GiB). Actual usage: 28.48 GiB for weights,
0.64 GiB for peak activation, 3.56 GiB for non-torch memory, 0.0 GiB
for NPU graph memory. Replace gpu_memory_utilization with
--kv-cache-memory=27260496384 (25.39 GiB) to fit into requested
memory, or --kv-cache-memory=30142716416 (28.07 GiB) to fully utilize
NPU free memory. Current KV cache memory: 25.53 GiB.

With this PR:

(Worker_TP0_EP0 pid=19702) INFO 07-03 15:28:30 [xlite.py:603] xlite
runtime pool size: 519 MB
(Worker_TP0_EP0 pid=19702) INFO 07-03 15:28:34 [worker.py:564]
Available KV cache memory: 28.67 GiB
(EngineCore pid=19675) INFO 07-03 15:28:35 [kv_cache_utils.py:1744]
GPU KV cache size: 626,176 tokens
(EngineCore pid=19675) INFO 07-03 15:28:35 [kv_cache_utils.py:1745]
Maximum concurrency for 40,960 tokens per request: 15.29x
(Worker_TP0_EP0 pid=19702) INFO 07-03 15:28:37 [worker.py:790] Free
memory on device (60.89/61.27 GiB) on startup. Desired GPU memory
utilization is (0.95, 58.21 GiB). Actual usage: 28.48 GiB for weights,
0.05 GiB for peak activation, 1.02 GiB for non-torch memory, 0.0 GiB
for NPU graph memory. Replace gpu_memory_utilization with
--kv-cache-memory=30625051136 (28.52 GiB) to fit into requested
memory, or --kv-cache-memory=33509896704 (31.21 GiB) to fully utilize
NPU free memory. Current KV cache memory: 28.67 GiB.


vLLM version: v0.23.0
vLLM main:
https://github.com/vllm-project/vllm/commit/ee0da84ab9e04ac7610e28580af62c365e898389

Signed-off-by: Sijie Fu fusijie@huawei.com
Co-authored-by: LuLina lina.lulina@huawei.com