vustef commented on issue #10381: URL: https://github.com/apache/arrow-rs/issues/10381#issuecomment-5043827101
Thanks all, the plan sounds good to me as well. `RowNumberReader` will error out in case missing ordinals, which will include mixed row-groups, so erroring out in `OrdinalAssigner` is not needed. The only caveat, and I forgot how things actually work, but if it works by reusing deserialized data, then this might be a problem: 1. Use a reader without row number. It's a file without rowgroup ordinal, on all row groups. OrdinalAssigner is skipped. 2. Now reuse the same metadata, but with row number virtual column. This fails, because there are no row group ordinals. It would've failed if this was the first read that deserialized metadata. So perhaps this constraint should be lifted: ``` Ordinals are assigned only when needed for row numbering (or already set in metadata) ``` It's enough that it doesn't error out, but we could keep ordinal assigning the best effort regardless of whether row numbering is used? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
