[ https://issues.apache.org/jira/browse/IMPALA-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616720#comment-17616720 ]
ASF subversion and git services commented on IMPALA-10976: ---------------------------------------------------------- Commit b92a8ad745e2499103a55aeb5e58aa090aee05e6 in impala's branch refs/heads/branch-4.1.1 from stiga-huang [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b92a8ad74 ] IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables When applying ALTER_PARTITION events on transactional tables, we refresh the partition using the metadata in events if hms_event_incremental_refresh_transactional_table is enabled (which is the default). This could be wrong if the ALTER_PARTITION event is stale. The partition metadata will be rolled back to a stale state. This patch compares the eventId with the createEventId of the table and ignores those ALTER_PARTITION events that have older (smaller) event ids. Note that we already do this for many other event types, ALTER_PARTITION is somehow missing the checks. Eventually we should depend on the lastSyncedEventId and replace createEventId with it. The self-event detection can also be replaced since self-events are also stale events. These will be addressed in IMPALA-10976. Tests - Verified locally with local-catalog mode and event-processor enabled and iterated test_acid_compute_stats for 1400 times. Without the fix, the test would fail in tens of runs. Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Reviewed-on: http://gerrit.cloudera.org:8080/19020 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-on: http://gerrit.cloudera.org:8080/19126 Reviewed-by: Csaba Ringhofer <csringho...@cloudera.com> Tested-by: Quanlong Huang <huangquanl...@gmail.com> > Sync db/table in catalogd to latest HMS event id for all DDLs from Impala > shell > ------------------------------------------------------------------------------- > > Key: IMPALA-10976 > URL: https://issues.apache.org/jira/browse/IMPALA-10976 > Project: IMPALA > Issue Type: Task > Components: Catalog, Frontend > Reporter: Sourabh Goyal > Assignee: Sourabh Goyal > Priority: Major > > This is a follow up from IMPALA-10926. The idea is that when any DDL > operation is performed from Impala shell, it also syncs the db/table to its > latest event ID as per HMS. This way updates to a db/table's are applied in > the same order as they appear in the Notification log in HMS which ensures > consistency. Currently catalogD applies any updates received from Impala > shell in place. Instead it should perform an HMS operation first and then > replay all the HMS events since the last synced event. > However there are subtle differences in how Impala processes DDLs via shell > vs how it processes HMS events These are: > * When processing an alter table event, currently catalogD does a full table > reload. This has a performance impact as table reload is time consuming. > Whereas in place alter table DDL operation in catalogOpExecutor (via Impala > shell) is faster since detects when to reload table schema or file metadata > or both. Need some improvements in Alter table event processing logic to > detect whether to reload the file metadata or not. > * Similar improvement is required in processing alter partition event. As of > now, when processing AlterPartition HMS event, catalogd always reloads file > metadata but when doing the same from shell, it reloads metadata only when it > is required. > * Impala shell already caches hive fns in catalog db’s object. But catalogD > does *not* process CREATE/DROP Fns HMS event > * When creating a db/table from Impala shell, if the operation fails because > the db/table already exists, then there is no reliable way in catalogd to > determine create event id for that db/table. The create event is required so > that for any subsequent ddl operations, catalogd can process HMS events > starting from createEvent Id. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org