adamreeve commented on issue #10129: URL: https://github.com/apache/arrow-rs/issues/10129#issuecomment-4715372585
I think it would be a reasonable decision to continue to error if writing too many row groups but allow reading files where the row group ordinal overflows i16. It seems like the ordinal in the metadata is there for convenience but is not strictly necessary, as readers should be able to keep track of this internally like the encryption implementation does. Maybe the row group index and row number virtual column implementations could be refactored to handle when the ordinal isn't set, or just error if they can't access the ordinal. It seems too strict to require that the ordinal is available when reading if it isn't needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
