[jira] [Commented] (HIVE-22062) WriteId is not updated for a partitioned ACID table when schema changes
[ https://issues.apache.org/jira/browse/HIVE-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170290#comment-17170290 ] Vihang Karajgaonkar commented on HIVE-22062: DDLs advance the table level writeId after HIVE-23573. Can you recheck if this still is a problem? cc [~gaborkaszab] > WriteId is not updated for a partitioned ACID table when schema changes > --- > > Key: HIVE-22062 > URL: https://issues.apache.org/jira/browse/HIVE-22062 > Project: Hive > Issue Type: Bug >Reporter: Gabor Kaszab >Assignee: Laszlo Kovari >Priority: Major > Labels: ACID > > Changing the schema (e.g. adding a new column) of a non-partitioned ACID > table results in the table-level writeId being incremented. This is as > expected. > However, if you do the same on a partitioned ACID table then neither the > table-level nor the partition-level writeIds are updated. I would expect in > this case to increment the table-level writeId to reflect that the table has > been changed. > Note, that get_valid_write_ids() shows that the high watermark is incremented > even though the writeId isn't. > Update: I'd extend the scope of this Jira further a bit. There are a number > of use cases in Hive that doesn't result in a writeId change on ACID tables > and as a result there is no way from other systems (like Impala) to judge if > a refresh should be run on a table or not. The only option is to every time > update all the data for a table that is expensive. E.g. Additionally to the > above use-case compaction is something that is not noticeable outside from > Hive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22062) WriteId is not updated for a partitioned ACID table when schema changes
[ https://issues.apache.org/jira/browse/HIVE-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985113#comment-16985113 ] Csaba Ringhofer commented on HIVE-22062: Linked HIVE-22565 as it also deals with changing table level writeId. > WriteId is not updated for a partitioned ACID table when schema changes > --- > > Key: HIVE-22062 > URL: https://issues.apache.org/jira/browse/HIVE-22062 > Project: Hive > Issue Type: Bug >Reporter: Gabor Kaszab >Assignee: Laszlo Kovari >Priority: Major > Labels: ACID > > Changing the schema (e.g. adding a new column) of a non-partitioned ACID > table results in the table-level writeId being incremented. This is as > expected. > However, if you do the same on a partitioned ACID table then neither the > table-level nor the partition-level writeIds are updated. I would expect in > this case to increment the table-level writeId to reflect that the table has > been changed. > Note, that get_valid_write_ids() shows that the high watermark is incremented > even though the writeId isn't. > Update: I'd extend the scope of this Jira further a bit. There are a number > of use cases in Hive that doesn't result in a writeId change on ACID tables > and as a result there is no way from other systems (like Impala) to judge if > a refresh should be run on a table or not. The only option is to every time > update all the data for a table that is expensive. E.g. Additionally to the > above use-case compaction is something that is not noticeable outside from > Hive. -- This message was sent by Atlassian Jira (v8.3.4#803005)