ChaomingZhangCN opened a new pull request, #453: URL: https://github.com/apache/paimon-cpp/pull/453
### Purpose Follow-up to #438. Independently rolled VECTOR files split a main file into several read ranges, reopening the same main file for each range. In the existing Java rolling fixture, a mixed projection opens the Parquet main file four times for four Vortex VECTOR files. This change reduces that to one main-file open while preserving the projected values. Add a sequential path for one main file and one VECTOR column whose files have compatible schema IDs, formats, and write columns and cover the main row-ID range continuously without overlaps. Keep the main reader across VECTOR boundaries and lazily open VECTOR readers one at a time, reusing the existing field-union reader. Unsupported layouts, historical inline VECTOR providers, and explicit `_SEQUENCE_NUMBER` projections retain the existing range path. The latter preserves the system column's provider precedence when it changes across VECTOR boundaries. Original file metadata and deletion-vector offsets are retained. ### Tests - Add a physical file-open regression with prefetch disabled and manifest scanning excluded from the count. The upstream reader opens the main file four times; the optimized reader opens it once and returns identical projected column values. - Cover sequential planning, unsupported layout fallback, schema error propagation, lazy opening, early close, deferred open failure, and reader lifetime after `TableRead` destruction. - Extend integration coverage for deletion vectors across main/VECTOR boundaries, fully deleted VECTOR files, disjoint row selections, and per-range sequence-number precedence. Existing NULL-vector and batch-boundary checks also exercise the sequential path. Validated in the existing macOS arm64 Debug build: - Built `paimon-data-evolution-table-test`, `paimon-core-test`, and `paimon-blob-table-inte-test`. - `paimon-data-evolution-table-test`: 302 passed, 12 skipped by existing format conditions. - `paimon-core-test --gtest_filter='DataEvolutionSplitReadTest.*:LazyConcatBatchReaderTest.*:AbstractSplitReadTest.*'`: 48 passed. - `paimon-blob-table-inte-test`: 176 passed, 4 skipped by existing format conditions. - `pre-commit run --files` on all four changed files and `git diff --check`: passed. Local VECTOR format coverage includes Parquet and Vortex. Lance is disabled in this build and was not tested. The full unit suite and throughput benchmarks were not run; the measured performance result is the reduction in main-file opens. ### API and Format No public API, storage format, or protocol changes. ### Documentation Internal reader optimization with no new user-facing configuration. No documentation changes are required. ### Generative AI tooling Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
