zjw1111 opened a new issue, #445: URL: https://github.com/apache/paimon-cpp/issues/445
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found no existing issue specifically tracking File Index V2. ### Motivation The V1 File Index container uses 32-bit payload offsets and lengths. The existing write path also materializes serialized index payloads and the complete container in memory before deciding whether to embed the index or write an external file. These limitations make it difficult to support large index artifacts, especially vector indexes. Java Paimon introduced a V2 container with 64-bit payload ranges and a streaming write interface. Paimon C++ should support the same format while retaining V1 compatibility. ### Solution - Add V2 container reading and writing, with int64 payload offsets and lengths and a footer/trailer layout aligned with Java Paimon. - Support `file-index.format-version` values `1` and `2`, keeping V1 as the default. - Add `FileIndexWriter::WriteTo()` so plugins can write payloads directly to a caller-owned stream. Keep a default implementation based on `SerializedBytes()` and implement direct streaming output for Lumina. - Widen plugin reader offsets and lengths to int64, with plugin-specific size limits checked inside the affected plugins. - Integrate a spillable output stream into the data-file index write path, embedding small containers and spilling larger ones to external files without buffering the complete V2 container. - Add focused coverage for V1 compatibility, Java-generated V2 golden bytes, large payload ranges, empty indexes, and spill thresholds. The V2 container's 64-bit ranges do not imply that every index plugin supports payloads larger than 2 GiB; individual plugin serialization formats may retain their own limits. ### Anything else? Java reference: https://github.com/apache/paimon/pull/10101 Related to File Index writing in #173. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
