Hidden Decoding at Scale

This repository hosts the Hidden Decoding paper, public demonstration checkpoints, and inference patch.

Updates

• 2026-07-07: Added the paper PDF, frontier-scale WeLM-HD4-80B/617B results, and the Stream-Factorized Attention framing.
• 2026-04-01: Released Hidden-Decoding-8B-n8-Instruct, our first instruction-tuned demonstration model.
• 2026-03-10: Initial release of base models (n=2, n=4, n=8) and SGLang inference patch.

Paper

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

• PDF
• Blog post

The paper studies whether an already-strong LLM can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Hidden Decoding expands each token into multiple hidden streams, supervises only the final stream, and retains the intermediate streams’ KV cache so their latent computation remains available to later tokens.

Key Idea

Hidden Decoding scales computation along the sequence-length dimension:

1. Each input token is expanded into n streams with independent embedding tables.
2. The expanded length-nL sequence is processed by the same Transformer backbone.
3. Only the final stream predicts the next token.
4. Intermediate streams act as latent computation states and keep their own KV cache.

This differs from recurrent-depth or looped Transformers: the extra computation is a longer sequence in a single forward pass, which fits the pipeline-parallel training stack used for large MoE models.

To make the expanded sequence trainable at scale, the paper introduces Stream-Factorized Attention. Most layers attend only within each stream, and only a subset of layers mix information across streams. This reduces the attention growth from quadratic in n to roughly linear in n.

Main Results

Frontier-Scale MoE

We train WeLM-HD4-80B and WeLM-HD4-617B with n=4 during continued pretraining. The matched autoregressive and Hidden Decoding models use the same early SFT-only post-training recipe, with no reinforcement learning. Active Transformer parameters per token stay unchanged: 3B for 80B and 23B for 617B.

The 4x expanded sequence costs 5.1x per batch on WeLM-HD4-80B and 4.4x per batch on WeLM-HD4-617B, close to the 4x linear reference and far below the 16x dense-attention baseline.

Expansion-Factor Scaling

Increasing the expansion factor improves language modeling loss and downstream accuracy while keeping the Transformer backbone fixed.

Released Demonstration Models

The released HuggingFace checkpoints are not the main models of the paper. They are Qwen3-8B based demonstration models provided to show that Hidden Decoding can scale with the expansion factor n: as n increases, the same Transformer backbone receives more latent computation per token and generally improves. The main paper results are the WeLM-HD4-80B and WeLM-HD4-617B studies above.

Base Models

All released base models share the same 8B Transformer backbone. They differ by the Hidden Decoding expansion factor n.

Instruct Model

Qwen3-8B Results

These released 8B models are included as a public, reproducible demonstration of expansion-factor scaling. They are not intended to be the paper’s main model release.

Base Model

Evaluated on Qwen3-8B-Base with progressive Hidden Decoding scaling:

Instruct Model

Instruction-tuned from Hidden-Decoding-8B-n8 and compared with Qwen3-8B-Instruct plus a matched Qwen3-8B SFT baseline trained on the same data:

Sampling: temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5, max_tokens=4096. Judge: GPT-4o.

Installation (Inference)

Option 1: Docker Image

docker pull aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126

Option 2: Use the Forked SGLang

git clone https://github.com/exlaw/sglang.git
cd sglang
pip install -e "python[all]"

Option 3: Apply Patch Manually

Apply the patch on top of SGLang:

git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout 4efe844a2
git apply /path/to/hidden_decoding.patch
pip install -e "python[all]"

Serving

Launch the server:

python -m sglang.launch_server \
    --model-path tencent/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128

docker run --gpus all -p 8080:8080 -p 8081:8081 \
  -v /path/to/models:/models \
  aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126 \
  bash -c "python -m sglang.launch_server \
    --model-path /models/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128"

Hidden Decoding models process n times longer sequences internally, so --chunked-prefill-size -1, --attention-backend fa3, and conservative batch sizes are important for stability and performance. Adjust --tp-size for multi-GPU setups.

Usage

Chat Completions

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain hidden decoding in simple terms."},
    ],
    max_tokens=512,
    temperature=0.7,
)
print(response.choices[0].message.content)

Text Completions

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.completions.create(
    model="default",
    prompt="The meaning of life is",
    max_tokens=128,
    temperature=0,
)
print(response.choices[0].text)

Chat UI

We also provide a lightweight, zero-dependency web chat interface for interactive testing:

python chat_ui.py

Then open http://localhost:8081 in your browser. The Chat UI connects to http://localhost:8080/v1 by default.

docker run --gpus all -p 8080:8080 -p 8081:8081 \
  -v /path/to/models:/models \
  -v /path/to/chat_ui.py:/app/chat_ui.py \
  aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126 \
  bash -c "python -m sglang.launch_server \
    --model-path /models/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128 &
  sleep 5 && python /app/chat_ui.py"

Features:

• Streaming responses with real-time token speed display
• Thinking/reasoning block visualization
• Configurable system prompt, temperature, and max tokens
• Pure Python standard library implementation

Patch Contents

The patch adds the qwen3_scale_seq model architecture and modifies the scheduler, batch manager, and CUDA graph runner to handle expanded sequence lengths.

Citation

If you find this work useful, please cite:

@article{hidden_decoding_at_scale_2026,
  title  = {Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models},
  author = {WeChat AI Team},
  year   = {2026},
  url    = {https://github.com/Tencent/Sequential-Hidden-Decoding/blob/main/paper/hidden_decoding_at_scale.pdf}
}

Contact

Sijun Zhang (nepheloturbulence@gmail.com), Aiwei Liu (liuaiwei20@gmail.com)

License

This project is released under the License Terms of Hidden-Decoding. The dependent open-source models and software components remain licensed under their respective original licenses. See LICENSE for details.