PARFusion Scheduler

PARFusion turns a dense NVIDIA GPU node into a cross-layer Resource/Parallelism Intent control plane for human-machine-object fusion workloads. One ResourceIntent flows down through Kubernetes-style orchestration, the application runtime (worker pools), the OS layer (CPU affinity / nice / cgroup v2), and the GPU layer (lane / share / stream-priority); telemetry flows back up and closes the loop.

Its core is the D-MUG algorithm (Deadline-aware Marginal-Utility Greedy, parfusion/dmug.py, see docs/algorithm.md). The default sample configuration maps four GPU lanes across eight GPU indexes, which matches an 8×H200 profile, but the runtime uses whatever GPUs nvidia-smi reports. It includes a safe modeled dashboard run, a measured CPU mixed-parallel co-run, and an opt-in GPU co-run harness that runs only on an idle GPU.

Documentation

• docs/architecture.md — system architecture and the two control loops
• docs/algorithm.md — the D-MUG algorithm (math + code map)
• docs/intent-spec.md — the ResourceIntent fields
• docs/real-benchmark.md — real CPU/GPU benchmark methodology and safety
• docs/experiments.md — the policy/ablation matrix and how to run everything

Safety Defaults

• Runtime files, logs, and experiment data stay under this directory: run/, logs/, results/.
• GPU load is disabled by default. GPU telemetry is read from nvidia-smi when available; no placeholder GPUs are reported when it is unavailable.
• GPU MPS/MIG/stream-priority control is dry-run by default. It records plans without changing GPU state.
• CPU synthetic load is capped by configs/demo.yaml.
• cgroup v2 integration is dry-run by default. It records safe execution plans without writing /sys/fs/cgroup.
• The real CPU benchmark confines itself to a small isolated arena of the most-idle cores.
• The real GPU benchmark is opt-in and only uses an idle GPU after checking nvidia-smi.
• Enable low-intensity GPU workers only when the machine is idle enough:

PARFUSION_ENABLE_GPU_LOAD=1 scripts/start_demo.sh

Quick Start

scripts/check_env.sh
scripts/run_tests.sh      # unit + integration test suite
scripts/start_demo.sh

Open the printed dashboard URL and click Guided Demo (or append ?story=guided to auto-play it). The walkthrough runs baseline, PARFusion, urgent-event, disturbance, and recovery stages in about 70 seconds. The top cockpit shows the live SLO (online p99 vs deadline, miss %, contention), the annotated latency chart marks every mode change and event, and the D-MUG Allocator table shows the per-task risk / slack / parallelism / GPU share / NMU decision. The What-if Preview sliders ask the allocator what it would do under a hypothetical disturbance without applying it. The manual Baseline / PARFusion / ablation / reference buttons remain for free exploration.

Prometheus metrics are available at:

http://127.0.0.1:<printed-port>/metrics

Stop it:

scripts/stop_demo.sh

Run a headless baseline-vs-PARFusion experiment:

scripts/run_experiment.sh 24

The JSON result is written to results/, figures are written to results/figures/, and a Markdown report is written to results/reports/.

Run the dynamic disturbance experiment:

scripts/run_dynamic_experiment.sh 18

This triggers an urgent robot event and a safe modeled CPU/GPU/network disturbance. It does not start real stress processes; the controller sees the disturbance, changes intents, and the report records normal/fp16/int8 online degradation decisions.

Run the full safe ablation experiment:

scripts/run_ablation_experiment.sh 12

This compares baseline, static, edf-only, no-crosslayer, no-gpuintent, no-predictor, no-elastic, no-degrade, no-preemption, and parfusion without real stress load. It produces a report under results/reports/.

Real Benchmarks (measured CPU; opt-in GPU)

These produce real p50/p95/p99 and throughput, not modeled latency. See docs/real-benchmark.md for methodology and safety.

# REAL CPU mixed-parallel co-run, confined to an isolated arena of idle cores.
scripts/run_cpu_corun.sh 25

# REAL GPU co-run, OPT-IN, only runs on an idle GPU (default candidates 6,7).
scripts/run_gpu_corun.sh 12

Each writes a JSON to results/, figures to results/figures/, and a report to results/reports/.

Run the Kubernetes scheduler simulator:

scripts/run_k8s_scheduler_sim.sh

Run the local Kubernetes scheduler extender API:

scripts/start_k8s_extender.sh
scripts/smoke_k8s_extender.sh
scripts/stop_k8s_extender.sh

This exposes /filter, /prioritize, /bind, /state, and /metrics on 127.0.0.1:18082 without starting a real cluster or pulling images.

Run the local Robonix/ROS2 bridge API:

scripts/start_robot_bridge.sh
scripts/smoke_robot_bridge.sh
scripts/stop_robot_bridge.sh

This exposes /api/mission, /api/urgent, /api/tick, /api/state, and /metrics on 127.0.0.1:18083. It uses rclpy if present and otherwise runs in local-compatible mode without installing ROS2.

What Runs Today

• ResourceIntent data model and CRD/proto draft.
• Real D-MUG scheduler (parfusion/dmug.py): Amdahl speedup curves, slack/deadline-risk, marginal-utility (NMU) allocation, degradation — driving parfusion and the ablations.
• Reference policies: baseline (K8s-default), fifo, static, loadbalance-only, no-crosslayer.
• D-MUG ablations: edf-only, no-gpuintent, no-predictor, no-elastic, no-degrade, no-preemption.
• EWMA+p95/p99 predictor feeding a deadline-risk contention signal.
• Local Fusion Agent loop that applies intents to safe user-space worker pools.
• Real CPU mixed-parallel co-run benchmark with measured p50/p95/p99 + batch throughput, driven by the real D-MUG and enforced via runtime pool resize + OS affinity/nice.
• Real GPU co-run benchmark (opt-in, idle-GPU only) measuring online inference p99 vs batch GEMM under GPU Intent (stream priority + concurrency throttle).
• Safe dynamic disturbance model for CPU/GPU/network pressure and degradation strategy.
• Safe OS/runtime enforcement for PARFusion-owned workers using CPU affinity and nice values.
• Safe cgroup v2 detector/planner with optional delegated attach mode.
• GPU lane manager for configured GPU routing, share, concurrency, and stream-priority intent planning.
• Safe GPU MPS/MIG/stream-priority control planner with dry-run command previews.
• Real GPU telemetry from nvidia-smi when available; empty GPU telemetry otherwise.
• Robot state machine RUNNING -> CHECKPOINTING -> PREEMPTED -> URGENT_TASK_RUNNING -> RESUMING -> COMPENSATING -> RUNNING with safe-point checkpoints and resume/replan metrics.
• Robonix/ROS2 bridge service with local-compatible fallback.
• Dashboard with a live D-MUG Allocator panel, GPU lanes, intents, online p99, node pressure, robot preemption, and events.
• Prometheus /metrics endpoint (including parfusion_dmug_* series).
• Report generators (modeled experiments + the measured CPU co-run; GPU co-run when run) .
• Kubernetes scheduler simulator + local scheduler extender API. The extender consumes scheduler-provided node lists by default; packaged sample nodes are only used when explicitly enabled for standalone local runs.
• Pytest unit + integration suite (scripts/run_tests.sh), Apache-2.0 license, pyproject.toml, Makefile.