JingsongLi opened a new pull request, #1138:
URL: https://github.com/apache/paimon-rust/pull/1138
## Purpose
PyPaimon already exposes filesystem `$partitions` metadata, including
deletion-vector cardinalities, but `Table.new_system_read("partitions")` was
unsupported. Add this missing core implementation so Python can reuse the
existing thin `SystemRead` binding.
## Changes
- Read the 12 Java `PartitionsTable` columns from the resolved manifest
snapshot. Files and deletion-vector metadata share one pinned snapshot,
including fixed-empty and historical views.
- Share typed partition aggregation with filesystem catalog listing and
`Table.partition_stats()`. Preserve declared `total_buckets` rather than
counting occupied buckets, and resolve file creation times as instants before
aggregation.
- Sum deletion-vector cardinalities across buckets, preserve unknown
cardinalities as NULL, and skip index-manifest reads when neither projection
nor predicates need the count.
- Render typed partition specs with Java casts, default partition names,
Java whitespace and UTF-8 rules. Reuse the existing Java floating formatter.
- Format BinaryRow timestamp values directly from
millisecond/sub-millisecond storage, sharing formatting with schema evolution.
This avoids Arrow nanosecond overflow for high-precision timestamps outside
1677–2262.
This covers the filesystem metadata behavior already implemented by Python.
Catalog-owned partition audit metadata and the DataFusion catalog-backed
provider retain their existing paths. No new public builder or Python binding
API is introduced.
## Validation
- Core system-table tests under `TZ=America/New_York`: 35 passed.
- Explicit New York DST-gap aggregation regression: passed.
- Partition-statistics unit regressions: passed.
- New core end-to-end partition metadata tests: 2 passed.
- DataFusion system-table integration suite: 28 passed.
- Workspace Clippy, Rust 1.98.0, `--all-targets --features fulltext,vortex
-- -D warnings`: passed.
- Companion Python tests with Native plan/read enabled: 204 passed in UTC
and 204 passed in America/New_York. These exercise the compiled binding,
Java-specific partition representations, deletion metadata, captured views, and
Python/Native parity.
Independent core and Python review completed; actionable findings were fixed
with regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
