markitdown-mb

markitdown-mb is a MoonBit document-to-Markdown tool for document ingestion, RAG, and automation pipelines.

The repository is currently on the unreleased 0.7.0 development line. Local archives and validation evidence are development artifacts, not published releases.

The project is inspired by Microsoft MarkItDown, but it is not a port. It is designed around stable routing, traceable provenance, explicit failure boundaries, and reproducible behavior under complex formats and engineering-scale workloads.

Before first use, read:

• Environment setup and install commands: docs/environment-dependencies.md
• Core architecture overview: docs/architecture/mb-markitdown-architecture.md
• Optional enhancement paths for OCR / accurate PDF / audio: docs/architecture/optional-enhancement-architecture.md

Prepare runtime dependencies before first run. Otherwise OCR, accurate PDF, audio, and benchmark examples will not work as expected.

Core native document readers need no external runtime. Optional profiles use one managed entrypoint:

./tools/env/optional_deps.sh install balance  # top-level/ZIP image OCR
./tools/env/optional_deps.sh install audio
./tools/env/optional_deps.sh install accurate # direct image/PDF accurate
./tools/env/optional_deps.sh install bench    # development comparison only

Performance Snapshot

Official performance claims come from bench only. To reproduce the current formal benchmark:

moon build --target native --release --package ZSeanYves/markitdown/cli
moon build --target native --release --package ZSeanYves/markitdown/bench/runner
./tools/env/optional_deps.sh install bench
RUNNER="_build/native/release/build/bench/runner/runner.exe"
"$RUNNER" doctor
"$RUNNER" run --preset official-external-compare

The runner report is the source of truth for performance numbers. This README keeps one reproducible snapshot and its environment rather than presenting machine-specific results as universal constants.

Current audited official-external-compare snapshot on macOS 15.3 arm64, recorded on 2026-07-17 against the repo-locked Microsoft MarkItDown 0.1.6 baseline:

• 25 selected rows, 25 semantically comparable rows
• 75/75 trusted tool cases; no route, fidelity, provenance, or density failure
• MoonBit CLI aggregate median: 53,036 us; baseline: 507,465 us
• Microsoft baseline: 24 successful rows plus one comparable XLSX huge row censored after all five 60-second samples reached the timeout
• performance gate: pass, with every case >=2x and every format geometric mean >=3x

Format	MoonBit CLI geometric-mean speedup
TXT	9.71x
CSV	9.22x
Markdown	10.78x
HTML	13.08x
ZIP	9.79x
EPUB	7.70x
PDF	4.17x
DOCX	54.15x
PPTX	32.39x
XLSX	61.45x
IPYNB	102.63x

The RSS gate measures the MoonBit CLI process for each row. The maxima below are MoonBit-only peaks from the selected rows; they do not include or combine the external baseline process RSS.

Format	Maximum MoonBit CLI RSS
HTML	154,912 KiB (151.28 MiB)
PDF	252,672 KiB (246.75 MiB)
XLSX	248,608 KiB (242.78 MiB)

ODF and optional dependency-backed balance cases use reviewed self baselines instead of invalid external comparisons. Dependency versions are locked under tools/env/config/. Approved macOS arm64 and Linux x64 baselines, each covering 106 CLI/engine cases, live under markitdown-quality-lab/performance_baselines/; inspect a generated results/summary.json for the complete measurement evidence.

For full benchmark usage, see bench/README.md. For benchmark architecture, see docs/architecture/benchmark-architecture.md.

Input Coverage

A brief input surface:

• Core plain text: txt/csv/tsv/srt/vtt
• Core structured text: json/jsonl/ndjson/ipynb
• Core markup and config text: xml/yaml/toml/html/markdown
• Core technical text: tex/rst/asciidoc
• Mail: eml
• Mail alias: msg is accepted only as a mail parsing alias. It does not imply native Outlook .msg binary parsing
• Containers: zip/epub
• Office: docx/xlsx/pptx odt/ods/odp
• PDF and direct image OCR: pdf/png/jpg/jpeg/bmp/webp/tif/tiff
• Audio (optional): wav/mp3/m4a

For the full capability matrix, see docs/capabilities-and-limitations.md.

Quick Start

moon build --target native --release --package ZSeanYves/markitdown/cli
./_build/native/release/build/cli/cli.exe --help
./_build/native/release/build/cli/cli.exe balance samples/fixtures/contracts/txt/txt_plain.txt .tmp/manual/out.md

If you need provenance output:

./_build/native/release/build/cli/cli.exe balance --provenance-out .tmp/manual/out.provenance.json samples/fixtures/contracts/html/html_simple.html .tmp/manual/out.md

For more CLI options, batch usage, and OCR / PDF / audio examples, see docs/cli-usage-guide.md.

Modes and Boundaries

• The default mode is balance
• Core support means the built-in balanced reader path. Direct image OCR, audio transcription, and PDF accurate are optional enhancements backed by local external runtimes; they are not part of the core reader commitment.
• Formal benchmarks measure balance mode only. ODT/ODS/ODP remain core native formats but use reviewed self baselines because the external baseline does not support them. OCR and audio also use self baselines.
• ocr, accurate PDF / image paths, and audio are optional capabilities. They are unavailable in a bare environment and require runtime dependencies
• accurate currently has two supported cases: pdf and direct image OCR use a high-fidelity path; docx/xlsx/pptx/odt/ods/odp use a semantic enhancement path. Other formats reject the unsupported mode with a non-zero exit status
• stream currently supports: txt/csv/tsv/srt/vtt/json/jsonl/ndjson/ipynb/xml/yaml/html/markdown/eml/epub/xlsx/odt/ods/odp
• docx/pptx/pdf/audio/direct image OCR do not expose a standalone stream path and reject stream with a non-zero exit status
• PDF OCR is only entered through the accurate PdfOcr route
• If accurate image OCR is missing Paddle runtime, it falls back to balanced image OCR
• If accurate PDF OCR is missing Paddle dependencies, it reports the missing dependency directly
• audio is only available when a local transcription backend is installed
• Requests outside the supported boundary fail closed

Development and Verification

Daily commands:

moon fmt --check
moon info && git diff --exit-code
moon check --target all --warn-list +73 --deny-warn
moon test --target all
moon build --target all

The quality-lab repository is only required for tools/regression/check*.sh and formal benchmarks, including internal baseline enforcement. If it is not present locally:

git clone https://github.com/ZSeanYves/markitdown-quality-lab.git markitdown-quality-lab

Common regression commands:

moon build --target native --release --package ZSeanYves/markitdown/cli
bash tools/regression/check_balance.sh
bash tools/regression/check_balance_quality.sh
bash tools/regression/check_accurate.sh
./tools/regression/check_coverage.sh --enforce

The formal benchmark reproduction commands are shown above. When run from the repo root, the runner prefers ./env/.venv-markitdown-bench/bin/markitdown.

To point to a specific baseline:

RUNNER="_build/native/release/build/bench/runner/runner.exe"
"$RUNNER" run --preset official-external-compare --markitdown-path /absolute/path/to/markitdown