DuanRuixiao commented on issue #17501: URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5258873781
@amitpoorab Thanks for the brainstorm. I believe we could probably do following things: 1. Version-aware reader (must ship first). Before the corrected writer is enabled by default, the reader needs to handle both encodings. A table property (e.g., `write.avro.fixed-encoding`) or manifest-level flag can serve as the discriminator. 2. Tie the corrected writer to new format version. This ensures the discriminator is always present and inspectable by the reader. Old writers produce old manifests; new writers produce new manifests. Both coexist safely during a rolling upgrade. 3. Provide a manifest rewrite migration procedure. Long-lived tables accumulate manifests that won't naturally be replaced. A `rewriteManifests()`-style procedure lets users actively migrate existing data to the corrected encoding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
