ChaomingZhangCN commented on PR #438: URL: https://github.com/apache/paimon-cpp/pull/438#issuecomment-6258595500
> Java rolls VECTOR files independently, so one main file can correspond to multiple VECTOR files with different row ranges—for example, a main file covering `[0,9999]` and two VECTOR files covering `[0,999]` and `[1000,9999]`. > > However, `DataEvolutionSplitRead::MergeRangesAndSort` still treats VECTOR files as ordinary data files and requires their row ranges to be identical. This rejects valid Java-written tables with `Data files should be all row id ranges same.`, even when projecting only ordinary columns. > > Could we add the corresponding VECTOR grouping/concatenation logic and an interoperability fixture with multiple VECTOR files per main file? The current fixture only covers one-to-one pairs with identical ranges, so it does not exercise this case. Thanks for catching this. Fixed in 5e9fb80: the reader now aligns row-id ranges to support independently rolled VECTOR files. Added a Java-generated fixture with 1 main file and 4 VECTOR files (3,001 rows), covering scalar/vector projections, NULLs, and row-range selection. Relevant tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
