alamb commented on issue #10381: URL: https://github.com/apache/arrow-rs/issues/10381#issuecomment-5024019303
It looks like this is the relevant conversation on https://github.com/apache/arrow-rs/pull/8715: https://github.com/apache/arrow-rs/pull/8715/changes#r2499219469 I don't think there is a reason we have to error if some row groups have ordinal assigned and some do not. From my reading of the spec, there is no requirement that the ordinal field is either all set or all not set: https://github.com/apache/parquet-format/blob/21ecc2b0f5786070561ae0d34152230853c83e2b/src/main/thrift/parquet.thrift#L1054-L1055 > Alternative: downgrade the hard error to a tracing::warn! unless the caller enables row-number virtual columns. This flips the default back to tolerant but keeps the invariant for the code path that actually depends on it. I think this is better (without the warn!). So specifically I suggest 1. assigning ordinal to any row group that doesn't have it 2. I am not quite sure how to handle metadata where the ordinal is not correctly set (e.g. `ordinal=10` even when there is only 1 row groups0. We could simply overwrite the oridinal value in the Metadata to be consistent, on the theory that we should be as lenient as posisble on reads and strict on writes Maybe @vustef remembers more context Maybe @etseidl has an opinion on this matter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
