[Fix][Bench] Record binary ops by canonical identity (#1788)

Problem

Several legacy binary benchmarks pass a short string such as maximum, minimum, or cmp_eq to BenchmarkReport.record() instead of passing the Op instance that was actually profiled.

String records contain neither the canonical class identity nor op_module. The nightly report therefore treats a legacy alias and its manifest-driven benchmark as different operators. For example, the same MaximumFwdOp implementation can appear once as MaximumFwdOp and again as maximum. Downstream documentation then has to classify the alias from its name alone, which can also place maximum under Reduction because it contains max.

This affects the standard binary arithmetic, comparison, logical, and bitwise benchmark groups. The measurements themselves are valid; the recorded operator identity is not canonical.

Solution

Pass the real Op instance to BenchmarkReport.record() for both the TileOPs measurement and its baseline:

op = op_cls(...)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag='tileops')

result_bl = bm.profile(baseline_fn, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag='torch')

BenchmarkReport then derives the identity consistently from:

op.__class__.__name__
op.__class__.__module__

This consolidates legacy aliases under canonical identities such as MaximumFwdOp, MinimumFwdOp, and EqFwdOp, while preserving the benchmark parameters, measurements, and baseline tags.

The change is intentionally limited to identity recording. It does not add display-category metadata or move taxonomy into TileOPs; benchmark page classification remains owned by TileOPs.github.io.

Changes

record standard binary arithmetic benchmarks with their real Op instances
record comparison benchmarks with their real Op instances instead of cmp_* aliases
record logical and bitwise benchmarks with their real Op instances
preserve canonical class names and op_module metadata in nightly JUnit output

Validation

python -m py_compile benchmarks/ops/bench_binary_elementwise.py
git diff --check

GPU benchmark execution was not available in the local worktree.

Related documentation taxonomy fix: tile-ai/TileOPs.github.io#13

TileOPs

Spec-driven GPU operator library for LLMs — designed for AI agents to build, evaluate, and optimize

Built on TileLang

Installation | Quick Start | Docs

Status: TileOPs is under active development. APIs may change.

Overview

TileOPs is a GPU operator library for LLM training and inference, built on TileLang. Beyond providing a growing collection of production-quality operators, TileOPs explores a spec-driven development model where AI agents can read declarative operator specifications, generate kernel implementations, and evaluate them against hardware-theoretical performance bounds — with minimal human scaffolding.

Architecture

Every operator is split into two layers with a strict boundary:

Op (L2) — stateless Python entry point. Handles validation, dtype casting, and memory layout. CUDA-Graph compatible; torch.compile(fullgraph=True) support is declared per op in the manifest.
Kernel (L1) — TileLang GPU implementation with hardware-specific optimizations (Hopper).

This separation keeps user-facing behavior independent of GPU strategy, allowing agents and developers to modify either layer without side effects on the other.

Key Properties

Spec-driven — each operator is declared in a machine-readable manifest (tileops/manifest/) that specifies signatures, workloads, and roofline formulas, serving as the entry point for both agent code generation and automated validation
Roofline-evaluated — kernel performance is measured against Speed-of-Light hardware bounds, not relative baselines
Auto-tuning — built-in search over tile sizes, pipelines, and scheduling parameters
Lightweight — depends only on TileLang, PyTorch, and einops

Installation

TileOPs is under active development and is installed from source; PyPI releases will begin with the first stable release. A CUDA-capable GPU is required.

Prerequisites

Python >= 3.10
PyTorch >= 2.1, < 2.11 (CI validates 2.10)
CUDA Toolkit 12.x
NVIDIA GPU: Hopper (SM_90)
TileLang >= 0.1.9, < 0.2.0 (CI validates 0.1.11)

From source

git clone https://github.com/tile-ai/TileOPs
cd TileOPs
make install    # dev dependencies + pre-commit hooks

[!NOTE] If CUDA and TileLang are already installed system-wide and you encounter build issues: PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev]' -v && pre-commit install

Verify:

python -m pytest tests/ -q    # requires a CUDA GPU

Quick Start

import torch
from tileops.ops import GemmOp

M, N, K = 1024, 1024, 512
dtype = torch.float16

gemm = GemmOp()  # shapes and dtype are inferred at call time

a = torch.randn(M, K, device="cuda", dtype=dtype)
b = torch.randn(N, K, device="cuda", dtype=dtype)  # trans_b=True by default

d = gemm(a, b)  # equals a @ b.T

Documentation

Design docs and development guides are in docs/. The full API reference and performance tables are published at TileOPs.github.io.

Contributing

See docs/ for design docs. Branch and commit conventions are in .claude/conventions/types.sh.

License

TileOPs is released under the MIT License.