fix(scheduler): fail closed on invalid HAMi device requests instead of binding GPU-less pods (#2994)
- fix(scheduler): fail closed on invalid HAMi device requests instead of binding GPU-less pods
An invalid HAMi device request (for example an mthreads core limit outside 0-100) was silently dropped by GenerateResourceRequests, so the pod looked device-less, was bound to a node, and ran with no GPU at all.
- Devices.GenerateResourceRequests now returns (request, error), so each backend can distinguish an invalid request from no request.
- All 16 backends return device.ErrInvalidDeviceRequest on invalid input.
- Resourcereqs propagates the error; Filter rejects the pod with a FilteringFailed event instead of returning all nodes.
- The admission webhook returns the real error instead of a generic quota message.
- mthreads and iluvatar: the MutateAdmission count*cores total is divided back to a per card value, and uneven totals are rejected.
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
- fix(device): address review feedback on invalid device request validation
- enflame GCU and kunlun vdevice: reject device counts outside the int32 range before narrowing to int32, instead of silently wrapping.
- mthreads: divide the admission-generated core total back to a per card value whenever more than one device is requested, so totals at or below 100 (count*16 for two to six cards) are normalized too, and keep the per card limit check after the division.
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
- test(cambricon): assert the error return in Test_GenerateResourceRequests
The error-path cases (zero/negative/overflowing counts, malformed memory) only compared the empty request and discarded the returned error, so they no longer proved the request is rejected. Capture the error and assert its message, and require a nil error for the accepted cases.
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
- fix(device): treat an explicit zero device count as device-less, not invalid
The fail-closed change rejected any count <= 0, but zero is not a malformed request: it is how a workload says it wants none of this vendor’s devices. PR #2740 established that reading, returning an empty request so the pod is admitted and scheduled without a device, and charts commonly render a disabled GPU count as 0.
fitResourceQuota runs GenerateResourceRequests on every pod the webhook sees, not only on pods carrying device resources, so an error for zero turned ordinary CPU pods with “nvidia.com/gpu: 0” into admission denials.
Split the guard: zero returns an empty request with a nil error, while a negative or overflowing count still fails closed. A backend for which zero really is malformed keeps rejecting it in MutateAdmission, which only rejects containers that actually carry its resources – awsneuron’s shared validator is left untouched for exactly that reason.
Also rewrite the mthreads Coresreq comment. The divide is unconditional on purpose and cannot adopt iluvatar’s “corenums > 100 && n > 1” gate: iluvatar writes count100 so every multi-card total exceeds 100, while mthreads writes count16, whose 2-to-6 card totals are 32 to 96. Gating on > 100 there would leave those totals undivided and report 16x too many cores per card. A test pins the reasoning.
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
- fix(device): reject a device count that does not fit in an int64
Quantity.AsInt64 fails for a count that is integral but larger than an int64, such as 1Ei or 1e19. The apiserver accepts those for an extended resource, so they reach the backends, where the failed conversion fell through to an empty request with a nil error. That is the silently device-less pod this change set exists to remove, so report it as an invalid request instead.
Assert both this and the zero-is-device-less contract across every registered backend at once, since either is easy to regress one backend at a time.
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
- fix(device-plugin): fail closed on config load error instead of Fatalf
LoadNvidiaDevicePluginConfig aborted the process with klog.Fatalf when the device config file failed to load. After master’s strict-YAML config parsing (#2941), that path became reachable from TestLoadNvidiaDevicePluginConfigFailsWhenTheNodeCannotBeRead, which fed a minimal config and expected a returned error; the Fatalf killed the whole test binary instead.
Return the error like the node-read path already does, so a bad config surfaces to the caller (factory.go already handles it) rather than taking the plugin down.
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
Signed-off-by: Eshiv Pandey eshivpandey18@gmail.com
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802047560号
English version | 中文版 | 日本語版
HAMi
Kubernetes GPU 虚拟化与异构加速器调度,面向 AI 基础设施。
HAMi 全称异构 AI 计算虚拟化中间件(Heterogeneous AI Computing Virtualization Middleware)。前身为
k8s-vGPU-scheduler,HAMi 帮助平台团队在 Kubernetes 工作负载间共享昂贵的 GPU 及其他 AI 加速器,隔离设备显存和算力,并通过设备感知的调度策略调度 Pod,无需修改应用代码。HAMi 是 CNCF Incubating 和 CNCF Landscape 项目,同时也被列入 CNAI Landscape。
为什么选择 HAMi?
AI 基础设施团队常常面临同样的 Kubernetes 加速器问题:整张 GPU 被分配给小型任务,团队争抢稀缺设备,不同加速器厂商暴露不同的操作模型,调度器缺乏足够的设备上下文来高效放置工作负载。
HAMi 提供了一个 Kubernetes 原生层:
使用场景
工作原理
HAMi 由 Mutating Webhook、调度器扩展器、设备插件和设备特定的容器内虚拟化组件组成。
设备虚拟化
HAMi 让工作负载只申请所需的加速器资源。例如,以下 Pod 请求一张具有 3 GiB 显存的物理 NVIDIA GPU:
工作负载在容器内看到已分配的设备资源,HAMi 负责协调调度、分配和隔离。
支持的设备
HAMi 支持多种异构加速器后端,包括 GPU、NPU、HCU、MLU、GCU、XPU 等。设备能力因厂商、型号、驱动和硬件代次而异。
请参阅 HAMi 支持的设备页面获取最新的支持矩阵。
快速开始
前置条件
使用 NVIDIA 设备插件路径,需准备:
nvidia-docker版本 > 2.0使用 Helm 安装
标记 GPU 节点以便 HAMi 管理:
添加 HAMi Helm 仓库:
安装 HAMi:
验证调度器和设备插件是否正在运行:
当
hami-device-plugin和hami-scheduler都为Running状态时,提交示例工作负载:完整的安装指南和配置选项,请参阅 HAMi 文档。
调度策略
HAMi 支持多种 AI 工作负载调度模式:
HAMi 兼容默认 Kubernetes 调度路径,也可与 Volcano 配合用于批量 AI 工作负载。请参阅 HAMi 网站获取当前的调度器集成指南。
可观测性与 WebUI
HAMi 暴露用于监控集群加速器使用情况的指标。安装后,指标可通过调度器监控端点获取:
默认监控端口为
31993。可通过 Helm 参数修改,如--set scheduler.service.monitorPort=<port>。HAMi 还提供:
生态系统集成
路线图、治理与贡献
HAMi 由维护者和贡献者共同治理。治理规则详见 HAMi 社区仓库。
如需贡献代码、文档、测试或设备后端改进,请阅读 CONTRIBUTING.md。
社区
HAMi 社区欢迎用户、贡献者、硬件厂商和构建 Kubernetes AI 基础设施的平台团队。
演讲和参考资料
许可证
HAMi 采用 Apache License 2.0 许可证。详见 LICENSE。
Copyright Contributors to HAMi, established as HAMi a Series of LF Projects, LLC.