mengna-lin commented on PR #16094: URL: https://github.com/apache/iceberg/pull/16094#issuecomment-5273233937
> > > > > I don’t think we should use column name in table property. Column name is not unique. > > > > > > > > > > > > I follow the pattern of current table property. https://github.com/Gerrrr/iceberg/blob/main/docs/docs/configuration.md. For example, bloom filter config is also based on column name. > > > > > > > > > I don't know about the history of bloom filter column. But this doesn't look like the right thing to do. We shouldn't take this approach just because there's precedence. > > > > > > Could you explain more on what do you mean by column name is not unique and what your concern is? This property is a write-time hint only — the codec is stored per column chunk in the Parquet file, and readers always decompress from that, never from the table property. So name ambiguity (rename, drop/re-create) can't cause incorrect reads. > > Yes, it won't not be a correctness issue. But you could have: > > ALTER TABLE .. DROP COLUMN foo; > > then > > ALTER TABLE ... ADD COLUMN foo; > > Now the new column shares the same column name but it's different than the one that was dropped. Even the data type could be different. So the hint can be wrongly applied to a different column. Iceberg disambiguates this through field id. In the example above, Iceberg spec would have two different field id for the foo column. > > > > > I don’t think we should use column name in table property. Column name is not unique. > > > > > > > > > > > > I follow the pattern of current table property. https://github.com/Gerrrr/iceberg/blob/main/docs/docs/configuration.md. For example, bloom filter config is also based on column name. > > > > > > > > > I don't know about the history of bloom filter column. But this doesn't look like the right thing to do. We shouldn't take this approach just because there's precedence. > > > > > > Could you explain more on what do you mean by column name is not unique and what your concern is? This property is a write-time hint only — the codec is stored per column chunk in the Parquet file, and readers always decompress from that, never from the table property. So name ambiguity (rename, drop/re-create) can't cause incorrect reads. > > Yes, it won't not be a correctness issue. But you could have: > > ALTER TABLE .. DROP COLUMN foo; > > then > > ALTER TABLE ... ADD COLUMN foo; > > Now the new column shares the same column name but it's different than the one that was dropped. Even the data type could be different. So the hint can be wrongly applied to a different column. Iceberg disambiguates this through field id. In the example above, Iceberg spec would have two different field id for the foo column. For schema changes like drop/add/rename, keeping the per-column properties in sync with the new schema would be up to the user — which is already the case for the existing name-keyed properties (dictionary encoding, bloom filters), so this wouldn't introduce a new expectation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
