fix: handle unavailable HNSW vectors during concurrent reads and writes (#1170)
What do these changes do?
Fix HNSW searches that encounter newly appended index IDs through shared graph topology while retaining an older vector-column snapshot. HNSW loads neighbor vectors before applying visibility filters, so the captured index-ID count is not a safe physical-access boundary.
- Bound vector reads by the snapshot column’s allocated buffer capacity and remove the stale captured index-ID count.
- Log a warning and return
nullptrwhen an ID exceeds the retained buffer, allowing ZVec to skip unavailable vectors after a writer resizes the column.- Pin ZVec to
960ecbc23f566df729429bdff655ece6dbb2261c, which includes alibaba/zvec#861 for alibaba/zvec#858. Adapt the renamed APIs, link the new RaBitQ libraries, and ensure incremental builds recheck updated ZVec sources.- Add regression coverage for new IDs within the retained buffer and IDs beyond it after resize. Cover both 16-node flat scans and 1001-node HNSW searches, with
fetch_vectordisabled and enabled. Verify the runtime scan threshold, actual unavailable-vector accesses, returned IDs, distances, and fetched vectors. For the 1001-node case, querying[1000, 0]returns IDs1000, 999with the new buffer and999, 998with the old snapshot.Validation
- Built NeuG and the vector-search extension against the pinned ZVec revision on macOS ARM64 with Python 3.9.
- All 4 C++ snapshot/vector-source regression tests passed.
- 38 Python HNSW tests passed; the Parquet import case was excluded because the Parquet extension was not built.
- All 3 Python HNSW TP tests passed in a separate process.
- C++ formatting and whitespace checks passed.
The new tests reproduce the relevant snapshot interleavings deterministically; they are not a rerun of the full concurrent workload from the issue. The upstream HNSW implementation can fall back to scanning available vectors when graph traversal encounters a missing external vector.
Related issue number
Fixes #1169
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802047560号
The one data index for your agentic applications
NeuG (pronounced “new-gee”) is a high-performance, graph-native transactional database that runs embedded in your application or behind a service. It provides durable storage, explicit transactions, Cypher-native querying, and in-place graph analytics.
Built on this data foundation, NeuG is the one data index for your agentic applications—indexing structure, semantics, and exact keywords over the same managed data. For more information, see the NeuG documentation.
News
Previous news
- **2026-06** — NeuG v0.1.3: [GDS extensions](https://neug.io/docs/extensions/load_gds/), [`COPY TEMP`](https://neug.io/docs/data_io/import_data/), [Node.js client](https://neug.io/docs/reference/nodejs_api/) - **2026-05** — NeuG v0.1.2: [`LOAD FROM`](https://neug.io/docs/data_io/load_data/), [Parquet](https://neug.io/docs/extensions/load_parquet/) & [HTTPFS](https://neug.io/docs/extensions/load_httpfs/) extensions - **2026-03** — NeuG v0.1 released - **2025-06** — GraphScope Flex, the engine foundation behind NeuG, set an [LDBC SNB Interactive Benchmark record](https://graphscope.io/blog/tech/2025/06/12/graphscope-flex-achieved-record-breaking-on-ldbc-snb-interactive-workload-declarative) with 80,000+ QPSInstallation
The packages support Linux, macOS, and Windows on x86_64 and ARM64. For more detailed instructions (including C++ from source), see the installation guide.
Python · requires Python 3.8+
Node.js · requires Node.js 20+ (since v0.1.3)
Quick Example
The same data can be queried by graph structure, vector similarity, or exact keywords. With the extensions installed and
ServiceandRunbookdata already loaded:Create an HNSW index · Create a full-text index
One Data, Indexed Three Ways
NeuG provides complementary ways to retrieve and analyze the same entities and properties:
Structure is native to NeuG’s graph storage. Vector and full-text indexes are maintained with the same underlying graph properties: graph changes and index changes commit atomically, and committed indexes recover with the graph through checkpoints and the write-ahead log.
Embedded or Service
Run NeuG in-process for local agent workflows and low-overhead analytics. When concurrent applications need network access, expose the same runtime as a service with
db.serve().See the reproducible dual-mode benchmark for complete results and methodology.
Development & Contributing
For building NeuG from source, see the Development Guide. We welcome contributions — please read the Contributing Guide before submitting issues or pull requests.
AI-Assisted Workflow
We apply an AI-assisted Spec-Driven workflow inspired by GitHub Spec-Kit:
/create-issuecommand in your IDE, or submit an issue manually/create-prcommand in your IDE, or submit a PR manuallyFor more details, see the AI-Assisted Development Guide.
Acknowledgements
NeuG builds upon the excellent work of the open-source community. We would like to acknowledge:
License
NeuG is distributed under the Apache License 2.0.