目录
luoxiaojian

feat: add complete Carquet writer through QueryExportWriter (#1048)

Add a private CarquetExportWriter that consumes the QueryResponse produced by the existing QueryExportWriter and writes through the existing Carquet output adapter. This completes the scalar and nested writer implementation while production COPY TO continues to use the Arrow writer.

The implementation supports scalar values, NULLs, LIST/fixed ARRAY and STRUCT, plus the existing string representations of INTERVAL and graph values. Both scalar and nested timestamps use NeuG’s millisecond unit. It retains compression, dictionary encoding and explicit row group options, validates response shape before opening output, and aborts partial output on write or finalization failure. The private C Data bridge owns its temporary buffers and needs no Arrow C++ SDK.

This PR uses only dependencies already merged into main. It has no dependency on the Carquet reader PR, changes no public IO/query interfaces or production registrations, and adds no Carquet submodule or adjacent-patch changes. The existing Arrow reader is used only as an independent test oracle. Tests generate files in memory; no new binary fixtures are checked in. Feature documentation is under doc/source/extensions/carquet_writer.md.

Validation

  • Debug and Release: all 6 CI-selected Parquet CTest targets pass, including 16 writer tests, 10 adapter tests, 46 existing Parquet tests and 3 Carquet C probes.
  • Debug and Release Python backend/export regression: 88 passed in each build; 6 cases skipped for the missing comprehensive_graph dataset, and 6 remote credential-dependent cases deselected.
  • ASan/UBSan: all 16 writer tests pass with 48 Carquet C sources and 4 private C++ implementation sources instrumented. The remaining NeuG core/dependencies are not instrumented.
  • The 4 private C++ sources compile after removing Arrow include directories. Clean pinned Carquet source plus the unchanged main patch builds with pkg-config unavailable.
  • clang-format 10.0.1, git diff whitespace checks, documentation links and navigation pass.

Linux/system-Arrow CI and the full Sphinx documentation build were not available locally.

Local delivery

  • Branch: codex/carquet-r07-complete-writer
  • Commit: 48a4e1dce0e68d331e978bbc13395b4892ce39e0
  • Base: ef76f38ff4af60d4681bdf9cc1a76e0a4476940d
  • Size: 10 files, +2543/-1 (implementation 1314 lines; tests 1120 lines; build/docs +109/-1).
  • Validation logs: /Users/luo/workspace/neug/build/review-r07-expanded-dmfpp0um
2天前433次提交

NeuG
The one data index for your agentic applications

NeuG Test (Linux) NeuG Wheel Packaging NeuG Documentation Coverage Follow NeuG


NeuG (pronounced “new-gee”) is a high-performance, graph-native transactional database that runs embedded in your application or behind a service. It provides durable storage, explicit transactions, Cypher-native querying, and in-place graph analytics.

Built on this data foundation, NeuG is the one data index for your agentic applications—indexing structure, semantics, and exact keywords over the same managed data. For more information, see the NeuG documentation.

News

Previous news - **2026-06** — NeuG v0.1.3: [GDS extensions](https://neug.io/docs/extensions/load_gds/), [`COPY TEMP`](https://neug.io/docs/data_io/import_data/), [Node.js client](https://neug.io/docs/reference/nodejs_api/) - **2026-05** — NeuG v0.1.2: [`LOAD FROM`](https://neug.io/docs/data_io/load_data/), [Parquet](https://neug.io/docs/extensions/load_parquet/) & [HTTPFS](https://neug.io/docs/extensions/load_httpfs/) extensions - **2026-03** — NeuG v0.1 released - **2025-06** — GraphScope Flex, the engine foundation behind NeuG, set an [LDBC SNB Interactive Benchmark record](https://graphscope.io/blog/tech/2025/06/12/graphscope-flex-achieved-record-breaking-on-ldbc-snb-interactive-workload-declarative) with 80,000+ QPS

Installation

The packages support Linux and macOS on x86_64 and ARM64. Windows users can run NeuG through WSL2; native Windows support is on the roadmap. For more detailed instructions (including C++ from source), see the installation guide.

Python  ·  requires Python 3.8+
pip install neug
Node.js  ·  requires Node.js 20+  (since v0.1.3)
npm install @graphscope-neug/neug

Quick Example

The same data can be queried by graph structure, vector similarity, or exact keywords. With the extensions installed and Service and Runbook data already loaded:

import neug

db = neug.Database("agent.db")
conn = db.connect()
conn.execute("LOAD vector_search;")
conn.execute("LOAD fts;")
conn.execute("CREATE INDEX runbook_vec ON Runbook USING HNSW (embedding) WITH (metric = 'l2');")
conn.execute("CREATE INDEX runbook_text ON Runbook USING FTS (content);")
query_embedding = [0.1, 0.2, 0.3, 0.4]

# Structure
conn.execute("""
    MATCH (:Service {name: 'PaymentService'})-[:HAS_RUNBOOK]->(r:Runbook)
    RETURN r.title
""")

# Semantics — accelerated by an HNSW index on Runbook.embedding
conn.execute("""
    MATCH (r:Runbook)
    RETURN r.title, vector_distance_l2(r.embedding, $embedding) AS distance
    ORDER BY distance ASC LIMIT 5
""", parameters={"embedding": query_embedding})

# Keywords — ranked by an FTS index on Runbook.content
conn.execute("""
    MATCH (r:Runbook)
    RETURN r.title, bm25(r.content, 'retry timeout') AS score
    ORDER BY score ASC LIMIT 5
""")

Create an HNSW index · Create a full-text index

One Data, Indexed Three Ways

NeuG provides complementary ways to retrieve and analyze the same entities and properties:

What NeuG indexes What it enables
Structure Entities, relationships, and graph topology Cypher traversal, pattern matching, PageRank, Leiden, shortest paths, and more
Semantics Dense vector properties with HNSW Similarity search using cosine, L2, or inner-product distance
Keywords Text properties with full-text indexes BM25-ranked word, phrase, prefix, Boolean, and exclusion search

Structure is native to NeuG’s graph storage. Vector and full-text indexes are maintained with the same underlying graph properties: graph changes and index changes commit atomically, and committed indexes recover with the graph through checkpoints and the write-ahead log.

Embedded or Service

Run NeuG in-process for local agent workflows and low-overhead analytics. When concurrent applications need network access, expose the same runtime as a service with db.serve().

See the reproducible dual-mode benchmark for complete results and methodology.

Development & Contributing

For building NeuG from source, see the Development Guide. We welcome contributions — please read the Contributing Guide before submitting issues or pull requests.

AI-Assisted Workflow

We apply an AI-assisted Spec-Driven workflow inspired by GitHub Spec-Kit:

  • 🐛 Bug Reports: Use /create-issue command in your IDE, or submit an issue manually
  • 💻 Pull Requests: Use /create-pr command in your IDE, or submit a PR manually

For more details, see the AI-Assisted Development Guide.

Acknowledgements

NeuG builds upon the excellent work of the open-source community. We would like to acknowledge:

  • Kùzu: Our C++ Cypher compiler is adapted from Kùzu’s implementation
  • DuckDB: Our runtime value system and extension framework are inspired by DuckDB’s architecture
  • zvec: Its in-process vector indexing engine provides the HNSW foundation for NeuG’s vector search extension

License

NeuG is distributed under the Apache License 2.0.

邀请码
    Gitlink(确实开源)
  • 加入我们
  • 官网邮箱:gitlink@ccf.org.cn
  • QQ群
  • QQ群
  • 公众号
  • 公众号

版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9 京公网安备 11010802047560号