sandip-db commented on PR #59435: URL: https://github.com/apache/spark/pull/59435#issuecomment-6164210101
> Instead of adding a new config, can we re-use the `compression` option and add a SQLConf (one of the `.legacy`) ones that will ignore the config during reads if it's enabled? Then we should update documentation accordingly As mentioned in the PR description, `compression` is persisted in the catalog by `saveAsTable` and `CREATE TABLE ... OPTIONS (compression ...)`, and those storage properties are merged into every read of the table. So after this change, such tables would force that codec on every file in their location. Files written with that codec still read fine, but locations with uncompressed or differently compressed files (written earlier or by other tools) would start failing unless the legacy conf is set. That's why I originally went with a separate read-only option. Are you OK with the new behavior being the default? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
