refactor(parquet): split
arrow_reader/selectioninto smaller modules (#10434)Which issue does this PR close?
- Closes #10424.
Rationale for this change
Follow up to #10141.
selection/mod.rshad grown past 2200 lines and mixed theRowSelectiontype and its public API, the set algebra, the page range mapping, the cursor machinery and a large test module. This is a reorganization to make the code easier to navigate; there is no functional change.What changes are included in this PR?
The module is split by concern, and
mod.rsgoes from 2219 to 947 lines:
module contents mod.rsRowSelection,RowSelectionInnerand the public APIselector.rsrun length backing: RowSelector,RowSelectionIter,and the split_off/offset/limitprimitivesboolean.rsbitmap backing: MaskSelection,MaskRunIterand themask primitives algebra.rsand_then,intersectionandunionfor bothbackings ranges.rspage byte ranges and batch boundary expansion cursor.rsRowSelectionCursor,MaskCursor,SelectorsCursor,LoadedRowRanges,RowSelectionPolicy/RowSelectionStrategyAre these changes tested?
Yes, by the existing tests, moved next to the code they now cover. No test was added, removed or renamed: the set of test names under
arrow::arrow_reader::selectionis identical before and after this PR, and thesplit_off/trim/offset/limittests still go through theRowSelectionmethods rather than the extracted helpers.Are there any user-facing changes?
No. The public API (
parquet::arrow::arrow_reader::{RowSelection, RowSelector, RowSelectionCursor, RowSelectionPolicy, RowSelectionIter, MaskRunIter}) is unchanged, and no call site outsideselection/needed modification.One internal note:
SelectorsCursorandMaskChunkare no longer nameable outsideselection/, as they moved into the privatecursormodule and are not re-exported. Neither was reachable from outside the crate before, sinceselectionispub(crate).
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802047560号
Native Rust implementation of Apache Arrow and Apache Parquet
Welcome to the Rust implementation of Apache Arrow, a popular in-memory columnar format and Apache Parquet, a popular columnar file format.
Community
We welcome participation from everyone and encourage you to join us, ask questions, help others, and get involved. All participation in the Apache Arrow project is governed by the Apache Software Foundation’s code of conduct.
We use GitHub issues and pull requests for all technical discussions, reviews, new features, bug fixes and release coordination. This ensures that all communication is public and archived for future reference.
The
dev@arrow.apache.orgmailing list is the communication channel for the overall Apache Arrow community. Instructions for signing up and links to the archives can be found on the Arrow Community page.Some community members also use the Arrow Rust Discord Server and the official ASF Slack server for informal discussions and coordination. This is a great place to meet other contributors and get guidance on where to contribute. However, all technical designs should also be recorded and formalized in GitHub issues, so that they are accessible to everyone. In Slack, find us in the
#arrow-rustchannel and feel free to ask for an invite via Discord, GitHub issues, or other means.There is more information in the contributing guide and the security policy.
Repository Structure
This repository contains the following crates:
arrowarrow-flightparquetparquet_deriveThe current development version the API documentation can be found here.
Note: previously the
object_storecrate was also part of this repository, but it has been moved to the arrow-rs-object-store repositoryRelease Versioning and Schedule
The Arrow Rust project releases approximately monthly and follows Semantic Versioning.
Due to available maintainer and testing bandwidth,
arrowcrates (arrow,arrow-flight, etc.) are released on the same schedule with the same versions as theparquetandparquet_derivecrates.This crate releases every month. We release new major versions (with potentially breaking API changes) at most once a quarter, and release incremental minor versions in the intervening months. See ticket #5368 for more details.
To keep our maintenance burden down, we do regularly scheduled releases (major and minor) from the
mainbranch. How we handle PRs with breaking API changes is described in the contributing guide.Planned Release Schedule
59.1.059.2.060.0.0Rust Version Compatibility Policy
arrow-rs and parquet are built and tested with stable Rust, and will keep a rolling MSRV (minimum supported Rust version) that can only be updated in major releases on an as needed basis (e.g. project dependencies bump their MSRV or a particular Rust feature is useful for us etc.). The new MSRV if selected will be at least 6 months old. The minor releases are guaranteed to have the same MSRV.
Note: If a Rust hotfix is released for the current MSRV, the MSRV will be updated to the specific minor version that includes all applicable hotfixes preceding other policies.
Guidelines for
panicvsResultIn general, use panics for bad states that are unreachable, unrecoverable or harmful. For those caused by invalid user input, however, we prefer to report that invalidity gracefully as an error result instead of panicking. In general, invalid input should result in an
Erroras soon as possible. It is ok for code paths after validation to assume validation has already occurred and panic if not. See ticket #6737 for more nuances.Deprecation Guidelines
Minor releases may deprecate, but not remove APIs. Deprecating APIs allows downstream Rust programs to still compile, but generate compiler warnings. This gives downstream crates time to migrate prior to API removal.
To deprecate an API:
#[deprecated]and specify the exact arrow-rs version in which it was deprecatedThe deprecated version is the next version which will be released (please consult the list above). To mark the API as deprecated, use the
#[deprecated(since = "...", note = "...")]attribute.For example
In general, deprecated APIs will remain in the codebase for at least two major releases after they were deprecated (typically between 6 - 9 months later). For example, an API deprecated in
51.3.0can be removed in54.0.0(or later). Deprecated APIs may be removed earlier or later than these guidelines at the discretion of the maintainers.Related Projects
There are several related crates in different repositories
object_storedatafusionballistaparquet_opendalopendal] forparquetArrow IOCollectively, these crates support a wider array of functionality for analytic computations in Rust.
For example, you can write SQL queries or a
DataFrame(using thedatafusioncrate) to read a parquet file (using theparquetcrate), evaluate it in-memory using Arrow’s columnar format (using thearrowcrate), and send to another process (using thearrow-flightcrate).Generally speaking, the
arrowcrate offers functionality for using Arrow arrays, anddatafusionoffers most operations typically found in SQL, includingjoins and window functions.You can find more details about each crate in their respective READMEs.