xndai commented on PR #16094: URL: https://github.com/apache/iceberg/pull/16094#issuecomment-5669994161
> > > > > > I don’t think we should use column name in table property. Column name is not unique. > > > > > > > > > > > > > > > I follow the pattern of current table property. https://github.com/Gerrrr/iceberg/blob/main/docs/docs/configuration.md. For example, bloom filter config is also based on column name. > > > > > > > > > > > > I don't know about the history of bloom filter column. But this doesn't look like the right thing to do. We shouldn't take this approach just because there's precedence. > > > > > > > > > Could you explain more on what do you mean by column name is not unique and what your concern is? This property is a write-time hint only — the codec is stored per column chunk in the Parquet file, and readers always decompress from that, never from the table property. So name ambiguity (rename, drop/re-create) can't cause incorrect reads. > > > > > > Yes, it won't not be a correctness issue. But you could have: > > ALTER TABLE .. DROP COLUMN foo; > > then > > ALTER TABLE ... ADD COLUMN foo; > > Now the new column shares the same column name but it's different than the one that was dropped. Even the data type could be different. So the hint can be wrongly applied to a different column. Iceberg disambiguates this through field id. In the example above, Iceberg spec would have two different field id for the foo column. > > > > > > > I don’t think we should use column name in table property. Column name is not unique. > > > > > > > > > > > > > > > I follow the pattern of current table property. https://github.com/Gerrrr/iceberg/blob/main/docs/docs/configuration.md. For example, bloom filter config is also based on column name. > > > > > > > > > > > > I don't know about the history of bloom filter column. But this doesn't look like the right thing to do. We shouldn't take this approach just because there's precedence. > > > > > > > > > Could you explain more on what do you mean by column name is not unique and what your concern is? This property is a write-time hint only — the codec is stored per column chunk in the Parquet file, and readers always decompress from that, never from the table property. So name ambiguity (rename, drop/re-create) can't cause incorrect reads. > > > > > > Yes, it won't not be a correctness issue. But you could have: > > ALTER TABLE .. DROP COLUMN foo; > > then > > ALTER TABLE ... ADD COLUMN foo; > > Now the new column shares the same column name but it's different than the one that was dropped. Even the data type could be different. So the hint can be wrongly applied to a different column. Iceberg disambiguates this through field id. In the example above, Iceberg spec would have two different field id for the foo column. > > For schema changes like drop/add/rename, keeping the per-column properties in sync with the new schema would be up to the user — which is already the case for the existing name-keyed properties (dictionary encoding, bloom filters), so this wouldn't introduce a new expectation. Not sure what you mean by "be up to the user". In my example, it's very hard for the user to detect the existence of such column compression setting on `foo` and drop it after the `foo` column is changed. Note that the user who add column compression property can be a different than the one who runs `ALTER`. This can be easily addressed by storing the field id instead internally. The SQL interface can still use column name. The Iceberg library can perform such translation easily. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
