目录
Xiaoli Zhou

fix: handle unavailable HNSW vectors during concurrent reads and writes (#1170)

What do these changes do?

Fix HNSW searches that encounter newly appended index IDs through shared graph topology while retaining an older vector-column snapshot. HNSW loads neighbor vectors before applying visibility filters, so the captured index-ID count is not a safe physical-access boundary.

  • Bound vector reads by the snapshot column’s allocated buffer capacity and remove the stale captured index-ID count.
  • Log a warning and return nullptr when an ID exceeds the retained buffer, allowing ZVec to skip unavailable vectors after a writer resizes the column.
  • Pin ZVec to 960ecbc23f566df729429bdff655ece6dbb2261c, which includes alibaba/zvec#861 for alibaba/zvec#858. Adapt the renamed APIs, link the new RaBitQ libraries, and ensure incremental builds recheck updated ZVec sources.
  • Add regression coverage for new IDs within the retained buffer and IDs beyond it after resize. Cover both 16-node flat scans and 1001-node HNSW searches, with fetch_vector disabled and enabled. Verify the runtime scan threshold, actual unavailable-vector accesses, returned IDs, distances, and fetched vectors. For the 1001-node case, querying [1000, 0] returns IDs 1000, 999 with the new buffer and 999, 998 with the old snapshot.

Validation

  • Built NeuG and the vector-search extension against the pinned ZVec revision on macOS ARM64 with Python 3.9.
  • All 4 C++ snapshot/vector-source regression tests passed.
  • 38 Python HNSW tests passed; the Parquet import case was excluded because the Parquet extension was not built.
  • All 3 Python HNSW TP tests passed in a separate process.
  • C++ formatting and whitespace checks passed.

The new tests reproduce the relevant snapshot interleavings deterministically; they are not a rerun of the full concurrent workload from the issue. The upstream HNSW implementation can fall back to scanning available vectors when graph traversal encounters a missing external vector.

Fixes #1169

15小时前498次提交

NeuG
The one data index for your agentic applications

NeuG Test (Linux) NeuG Wheel Packaging NeuG Documentation Coverage Follow NeuG


NeuG (pronounced “new-gee”) is a high-performance, graph-native transactional database that runs embedded in your application or behind a service. It provides durable storage, explicit transactions, Cypher-native querying, and in-place graph analytics.

Built on this data foundation, NeuG is the one data index for your agentic applications—indexing structure, semantics, and exact keywords over the same managed data. For more information, see the NeuG documentation.

News

Previous news - **2026-06** — NeuG v0.1.3: [GDS extensions](https://neug.io/docs/extensions/load_gds/), [`COPY TEMP`](https://neug.io/docs/data_io/import_data/), [Node.js client](https://neug.io/docs/reference/nodejs_api/) - **2026-05** — NeuG v0.1.2: [`LOAD FROM`](https://neug.io/docs/data_io/load_data/), [Parquet](https://neug.io/docs/extensions/load_parquet/) & [HTTPFS](https://neug.io/docs/extensions/load_httpfs/) extensions - **2026-03** — NeuG v0.1 released - **2025-06** — GraphScope Flex, the engine foundation behind NeuG, set an [LDBC SNB Interactive Benchmark record](https://graphscope.io/blog/tech/2025/06/12/graphscope-flex-achieved-record-breaking-on-ldbc-snb-interactive-workload-declarative) with 80,000+ QPS

Installation

The packages support Linux, macOS, and Windows on x86_64 and ARM64. For more detailed instructions (including C++ from source), see the installation guide.

Python  ·  requires Python 3.8+
pip install neug
Node.js  ·  requires Node.js 20+  (since v0.1.3)
npm install @neug-ai/neug

Quick Example

The same data can be queried by graph structure, vector similarity, or exact keywords. With the extensions installed and Service and Runbook data already loaded:

import neug

db = neug.Database("agent.db")
conn = db.connect()
conn.execute("LOAD vector_search;")
conn.execute("LOAD fts;")
conn.execute("CREATE INDEX runbook_vec ON Runbook USING HNSW (embedding) WITH (metric = 'l2');")
conn.execute("CREATE INDEX runbook_text ON Runbook USING FTS (content);")
query_embedding = [0.1, 0.2, 0.3, 0.4]

# Structure
conn.execute("""
    MATCH (:Service {name: 'PaymentService'})-[:HAS_RUNBOOK]->(r:Runbook)
    RETURN r.title
""")

# Semantics — accelerated by an HNSW index on Runbook.embedding
conn.execute("""
    MATCH (r:Runbook)
    RETURN r.title, vector_distance_l2(r.embedding, $embedding) AS distance
    ORDER BY distance ASC LIMIT 5
""", parameters={"embedding": query_embedding})

# Keywords — ranked by an FTS index on Runbook.content
conn.execute("""
    MATCH (r:Runbook)
    RETURN r.title, bm25(r.content, 'retry timeout') AS score
    ORDER BY score ASC LIMIT 5
""")

Create an HNSW index · Create a full-text index

One Data, Indexed Three Ways

NeuG provides complementary ways to retrieve and analyze the same entities and properties:

What NeuG indexes What it enables
Structure Entities, relationships, and graph topology Cypher traversal, pattern matching, PageRank, Leiden, shortest paths, and more
Semantics Dense vector properties with HNSW Similarity search using cosine, L2, or inner-product distance
Keywords Text properties with full-text indexes BM25-ranked word, phrase, prefix, Boolean, and exclusion search

Structure is native to NeuG’s graph storage. Vector and full-text indexes are maintained with the same underlying graph properties: graph changes and index changes commit atomically, and committed indexes recover with the graph through checkpoints and the write-ahead log.

Embedded or Service

Run NeuG in-process for local agent workflows and low-overhead analytics. When concurrent applications need network access, expose the same runtime as a service with db.serve().

See the reproducible dual-mode benchmark for complete results and methodology.

Development & Contributing

For building NeuG from source, see the Development Guide. We welcome contributions — please read the Contributing Guide before submitting issues or pull requests.

AI-Assisted Workflow

We apply an AI-assisted Spec-Driven workflow inspired by GitHub Spec-Kit:

  • 🐛 Bug Reports: Use /create-issue command in your IDE, or submit an issue manually
  • 💻 Pull Requests: Use /create-pr command in your IDE, or submit a PR manually

For more details, see the AI-Assisted Development Guide.

Acknowledgements

NeuG builds upon the excellent work of the open-source community. We would like to acknowledge:

  • Kùzu: Our C++ Cypher compiler is adapted from Kùzu’s implementation
  • DuckDB: Our runtime value system and extension framework are inspired by DuckDB’s architecture
  • zvec: Its in-process vector indexing engine provides the HNSW foundation for NeuG’s vector search extension

License

NeuG is distributed under the Apache License 2.0.

    Gitlink(确实开源)
  • 加入我们
  • 官网邮箱:gitlink@ccf.org.cn
  • QQ群
  • QQ群
  • 公众号
  • 公众号

版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9 京公网安备 11010802047560号