zjw1111 opened a new pull request, #446: URL: https://github.com/apache/paimon-cpp/pull/446
### Purpose Closes #445. V1 uses 32-bit payload offsets and lengths, and the data-file index write path buffers the complete container before deciding whether to embed it or create an external file. Add Java-compatible V2 containers with 64-bit payload ranges and streamed payload writes, while keeping V1 as the default. Split the format implementations into V1/V2 writers, add `FileIndexWriter::WriteTo()` with a `SerializedBytes()` fallback, and implement direct streaming for Lumina. The data-file index writer now spills to an external file when the container crosses the inline threshold. ### Tests - Final submission checks: `pre-commit run --files` for all 40 changed files, `git diff --cached --check`, and `git diff --check` passed. - Earlier focused validation passed for format reading/writing, Java V2 golden bytes, large ranges, plugin limits, data-file index writing, spill thresholds and cleanup, and Lumina round trips. The latest Lumina validation passed all four tests. - The golden bytes were independently generated by running the unmodified Java Paimon format sources at commit `5a6739c6c10e96052bac678eb245742ad92bc556`; all 86 bytes match the C++ fixture. - No local compilation or test rerun was performed for this final submission, as requested by the author. ### API and Format - Add `file-index.format-version=1|2` (default: `1`), with automatic reader dispatch based on the stored version. - V2 stores payloads before a footer and trailer, with int64 offsets and lengths. V1 layout remains compatible. - Widen `FileIndexer::CreateReader()` offsets and lengths to int64. Individual plugins retain their own payload limits; bitmap and range bitmap explicitly reject lengths above `INT32_MAX`. - Add `Result<bool> FileIndexWriter::WriteTo(const std::shared_ptr<OutputStream>&) const`. False means no payload; true means a non-empty payload was written. Null or zero-length `SerializedBytes()` results produce no index. - Replace container `WriteColumnIndexes()` with `WriteIndex()` and `Finish()`, and add a version argument to `CreateWriter()` with a V1 default. Downstream API implementations and callers must adapt to these interface changes. ### Documentation Document version selection and streaming behavior in the public format API comments. Describe each serialized layout in `file_index_format_v1.h` and `file_index_format_v2.h`. ### Generative AI tooling Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
