Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16851 )
Change subject: IMPALA-10358: Correct Iceberg type mappings ...................................................................... Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16851/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16851/1//COMMIT_MSG@13 PS1, Line 13: * annotates strings with UTF8 in Parquet metadata > These changes make sense to me. I did add a TYPE_FIXED_UDA_INTERMEDIATE typ Thanks for the review. Seems like in SQL the BINARY type is used for fixed-length binary data, and VARBINARY for variable-length binary data. However, Hive added the BINARY type for variable-length binary data, but at least it's consistent with Iceberg which also has a BINARY type with the same semantics. Maybe FIXED_BINARY would be a good name in HiveQL for fixed-length data. Thanks for bringing up TIMESTAMPTZ. Yeah, on the read-side we use Parquet metadata to decide if we need to convert the data to the local timezone or just read it as it is. But once we've read the values as TimestampValue objects in memory, we loose all the timezone-related information, and at the end we just write/output them as they are. Until now we could write Iceberg tables with TIMESTAMPTZ columns which was clearly wrong, because in the Parquet files Impala didn't convert the timestamps to UTC. In the new patch set I raise an error when Impala tries to write TIMESTAMPTZ. -- To view, visit http://gerrit.cloudera.org:8080/16851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I652565f82708824f5cf7497139153b06f116ccd3 Gerrit-Change-Number: 16851 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: wangsheng <sky...@163.com> Gerrit-Comment-Date: Fri, 11 Dec 2020 14:21:59 +0000 Gerrit-HasComments: Yes
