vllm-plugin-FL

vllm-plugin-FL is a plugin for the vLLM inference/serving framework, built on FlagOS’s unified multi-chip backend — including the unified operator library FlagGems and the unified communication library FlagCX. It extends vLLM’s capabilities and performance across diverse hardware environments. Without changing vLLM’s original interfaces or usage patterns, the same command can run model inference/serving on different chips.

Version Compatibility

vllm-plugin-FL Branch	Community vLLM Version
`release/0.2`	v0.20.2
`main`	v0.24.0

Supported Models and Chips

In theory, vllm-plugin-FL can support all models available in vLLM, as long as no unsupported operators are involved. The tables below summarize the current support status of end-to-end verified models and chips, including both fully supported and in-progress (“Merging”) entries.

Supported Models

Model	Status	Reference
Qwen3.5-397B-A17B	Supported	example
Qwen3-Next-80B-A3B	Supported	example
Qwen3-4B	Supported	example
MiniCPM-o 4.5	Supported	example
GLM-5	Supported	example
Qwen3.5-35B-A3B	Supported	example
BAAI/bge-m3	Supported	implementation
MiniMax-M2.7	Supported	implementation

Supported Chips

Chip Vendor	Status	Reference
NVIDIA	Supported	-
Ascend	Supported	-
MetaX	Supported	-
T-Head	Supported	-
Iluvatar	Supported	-
Tsingmicro	Supported	-
Moore Threads	Supported	-
Hygon	Supported	-
Sunrise	Supported	-
ARM64 CPU	Supported	-

Quick Start

Setup

Install vLLM

For NVIDIA GPUs, install vLLM from the official v0.24.0 release (optional if the correct version is already installed):
```
pip install vllm==0.24.0
```
For non-NVIDIA chips, install vLLM from source with the empty device target:
```
git clone -b v0.24.0 https://github.com/vllm-project/vllm.git
cd vllm
VLLM_TARGET_DEVICE=empty pip install -v --no-build-isolation --no-deps .
```
Install vllm-plugin-FL

2.1 Clone the repository:
```
git clone https://github.com/flagos-ai/vllm-plugin-FL
```
2.2 Install
```
cd vllm-plugin-FL
pip install --no-build-isolation .
# or editable install
pip install --no-build-isolation -e .
```
For CUDA-like devices, including CUDA and HIP/ROCm environments that use PyTorch’s CUDA dispatch key, build the plugin native extension by setting VLLM_VENDOR=cuda during installation:
```
cd vllm-plugin-FL
VLLM_VENDOR=cuda pip install --no-build-isolation .
# or editable install
VLLM_VENDOR=cuda pip install --no-build-isolation -e .
```
This builds and installs vllm_fl._C, which provides native C++ support required by some graph/custom-op paths, especially when vLLM is installed with VLLM_TARGET_DEVICE=empty.

If VLLM_VENDOR is not set, vllm-plugin-FL is installed as a Python-only plugin and the native extension is skipped.

Install FlagGems

3.1 Install Build Dependencies

pip install -U scikit-build-core==0.11 pybind11 ninja cmake

3.2 Install FlagGems

git clone -b v5.3.4 https://github.com/flagos-ai/FlagGems
cd FlagGems
pip install --no-build-isolation .
# or editable install
pip install --no-build-isolation -e .

Runtime compatibility hooks

The plugin installs runtime compatibility hooks through vLLM’s plugin entry points without modifying the installed vLLM package. Model-specific config and model registrations are loaded only for their corresponding architectures.

Operator adapters use the plugin dispatch manager, so backend selection, fallback, per-op policy, operator-list recording, and I/O diagnostics continue to follow the common FlagOS controls.

(Optional) Install FlagCX

4.1 Clone the repository:

git clone -b v0.13.0 https://github.com/flagos-ai/FlagCX.git
cd FlagCX
git submodule update --init --recursive

4.2 Build the library with different flags targeting to different platforms:

make USE_NVIDIA=1

4.3 Set environment

export FLAGCX_PATH="$PWD"

4.4 Installation FlagCX

cd plugin/torch/
FLAGCX_ADAPTOR=[xxx] pip install . --no-build-isolation
# or editable install
FLAGCX_ADAPTOR=[xxx] pip install -e . --no-build-isolation

Note: [xxx] should be selected according to the current platform, e.g., nvidia, ascend, etc.

If there are multiple plugins in the current environment, you can specify use vllm-plugin-fl via VLLM_PLUGINS=’fl’.

Additional Steps for Ascend

Install FlagTree

RES="--index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple --trusted-host=https://resource.flagos.net"
python3 -m pip install flagtree==0.6.1rc1+ascend3.5 $RES

For other chips, please refer to FlagTree for the corresponding version (e.g., flagtree==0.6.1+iluvatar3.6, flagtree==0.6.1+metax3.6, etc.).

Set required environment variable
```
export TRITON_ALL_BLOCKS_PARALLEL=1
```
Enable eager execution

Ascend requires eager execution. Add enforce_eager=True to the LLM constructor or pass --enforce-eager on the command line.

Run a Task

Offline Batched Inference

With vLLM and vLLM-fl installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: offline_inference. Or use blow python script directly.

from vllm import LLM, SamplingParams


if __name__ == "__main__":
    prompts = [
        "Hello, my name is",
    ]
    # Create a sampling params object.
    sampling_params = SamplingParams(max_tokens=10, temperature=0.0)
    # Create an LLM.
    llm = LLM(model="Qwen/Qwen3-4B", max_num_batched_tokens=16384, max_num_seqs=2048)
    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Advanced use

For dispatch environment variable usage, see environment variables usage.

Using Cuda Communication library

If you want to use the original Cuda Communication, you can unset the following environment variables.

unset FLAGCX_PATH

Using native CUDA operators

If you want to use the original CUDA operators, you can set the following environment variables.

export USE_FLAGGEMS=0