lszskye opened a new pull request, #423: URL: https://github.com/apache/paimon-cpp/pull/423
### Purpose Implement Java-compatible bitmap64 deletion vectors for existing Paimon C++ read and primary-key lookup compaction paths. The change includes: - Bitmap64 deletion vector operations, serialization and deserialization, including Java-compatible framing and metadata length semantics. - Creation of bitmap64 deletion vectors when `deletion-vectors.bitmap64=true`. Existing vectors retain their format, and mixed bitmap32/bitmap64 index files remain readable. - Bitmap64 filtering during ordinary reads and data-evolution reads, including position-shifted views. - Java-aligned merge validation for both bitmap32 and bitmap64 deletion vectors. - Reading each deletion vector at its metadata offset in the index file. Iceberg V3 integration, append-table DV writing, and data-evolution DV writing are outside this change. Bitmap64 support does not extend the bitmap interfaces used by File Index or File Format: - **File Index:** selection results remain bitmap32. Ordinary reads skip File Index evaluation when the file has a bitmap64 deletion vector. File Index evaluation also returns `Remain` when the file's row count exceeds the supported bitmap32 range. - **File Format:** selection bitmap pushdown remains bitmap32. Bitmap64 deletion vectors are not narrowed or pushed into format readers; `ApplyDeletionVectorBatchReader` applies them using the file-level row positions reported by the reader. ### Tests - Deletion vector tests for boundary positions, merging, conversion, serialization, malformed input, and mixed-format persistence. - Parameterize DV scenarios in `PkCompactionInteTest` and `DataEvolutionTableTest` for both bitmap32 and bitmap64. - Add `DeduplicateWithBitmap64HighPositions`: a test-only format backed by a small real Parquet file exposes large row positions. The test exercises actual lookup index construction, compaction, DV persistence, scan filtering, and subsequent full compaction. It verifies that positions with identical low 32 bits remain distinct. ### Documentation Update the read guide to document bitmap64 and mixed-format DV support, configuration behavior, and the existing data-evolution write limitations. ### Generative AI tooling Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
