[Feature] Add offline training operator profiling (#1310)
Summary
- decouple operator report generation from the training profiling callback so training only exports rank-scoped Chrome timelines
- add a streaming offline extractor with External ID attribution, explicit mapping status, communication accounting, training operator hierarchy, multi-rank batch extraction, and cross-rank aggregation
- align per-rank outputs with the inference profiler:
operator_list.csv,kernel_time.csv,kernel_shape_dtype.csv, andsummary.json, using arank-N_prefix- retain unattributed kernels and communication kernels, use all GPU kernel duration as the timing denominator, and validate event/time/CSV/ID conservation
- add unit coverage and a training operator profiling Quick Start
Output compatibility
The old synchronous report names are replaced by:
rank-N_operator_list.csvrank-N_kernel_time.csvrank-N_kernel_shape_dtype.csvrank-N_summary.jsonCommunication is included in the common reports instead of a separate communication CSV. Training-specific hierarchy fields are appended after the inference-compatible columns.
Testing
python -m pytest -q tests/unit_tests/train/test_train_config.py tests/unit_tests/train/utils/test_optim_setup.py tests/unit_tests/train/megatron/test_profiler_reports.py(86 passed)python -m compileall -q flagscale/train/megatron/training/profiler_reports.py flagscale/train/megatron/training/training.pygit diff --check- validated both fusion-on and fusion-off real training timelines; all conservation checks passed
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802047560号
[中文版|English]
介绍
FlagScale 是 FlagOS 的核心组件。FlagOS 是一个统一的开源 AI 系统软件栈,通过无缝集成各类模型、系统与芯片,构建开放的技术生态。秉承”一次开发,多芯迁移”的理念,FlagOS 旨在充分释放硬件算力潜能,打破不同芯片软件栈之间的壁垒,有效降低迁移成本。
作为该生态的核心工具包,FlagScale 提供统一的接口,覆盖大语言模型、多模态模型及具身智能模型的完整生命周期。它在统一的配置项和命令行界面下集成了多个开源后端引擎,支持模型训练、强化学习和推理等关键工作流,并在多种芯片厂商间保持一致的运行体验。快速上手请参阅 快速入门指南。
在 FlagOS 生态中,FlagScale 与以下组件协同工作:
FlagOS 插件项目基于广泛使用的上游开源框架构建,并对其进行扩展以支持多种 AI 芯片,为训练、强化学习和推理提供硬件兼容性和运行时集成。
下表列出了 FlagOS 插件与对应上游项目的映射关系:
TransformerEngine-FL
TransformerEngine
资源
支持列表
模型训练
服务、推理
参与贡献
请加入我们的微信群
授权许可
FlagScale 采用 Apache License (Version 2.0) 授权许可。 本项目中也包含一些使用其他开源授权许可的第三方组件。