[
https://issues.apache.org/jira/browse/IMPALA-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Goyal updated IMPALA-10925:
-----------------------------------
Description:
h3. Problem Statement
Impala catalogd has Events processor which polls metastore events at regular
intervals to automatically apply changes to the metadata in the catalogd.
However, the current design to detect the self-generated events (DDL/DMLs
coming from the same catalogd) have consistency problems which can cause query
failures under certain circumstances.
h3. Current Design
The current design of self-event detection is based on adding markers to the
HMS objects which are detected when the event is received later to determine if
the event is self-generated or not. These markers constitute a serviceID which
is unique to the catalogd instance and a catalog version number which is unique
for each catalog object. When a DDL is executed, catalogd adds these as object
parameters. When the event is received, Events processor checks the serviceID
and if the catalog version of the current object with the same name in the
catalogd cache and makes a decision of whether to ignore the event or not.
h3. Problems with the current design
The approach is problematic under some circumstances where there are
conflicting DDLs repeated at a faster interval . For example, consider the
following sequence of alter table operations on a table t1 as per HMS
1. alter table t1 from source s1 say another Impala cluster
2. alter table t1 from source s2 say another Hive cluster
3. alter table t1 from local
The #3 alter table ddl operation would get reflected in the local cache
immediately. However, event processor later on would process events from #1 and
#2 above and try to alter the table. In an ideal scenario, these alters should
have been applied before #3 (i.e in the same order as that in HMS). This
leaves table t1 in an inconsistent state.
h3. Proposed Solution
The main idea of the solution is to keep track of the last event id for a given
table as eventId which the catalogd has synced to in the Table object. The
events processor ignores any event whose EVENT_ID is less than or equal to the
eventId stored in the table. Once the events processor successfully processes a
given event, it updates the value of eventId in the table before releasing the
table lock. Also, any DDL or refresh operation on the catalogd (from both
catalog metastore server and Impala shell) will follow the following steps to
update the event id for the table:
1. Acquire write lock on the table
2. Perform ddl operation in HMS
3. Sync table till the latest event id (as per HMS) since its last synced
event id
The above steps ensure that any concurrent updates applied on a same db/table
from multiple sources like Hive, Impala or say multiple Impala clusters, get
reflected in the local catalogd cache (in the same order as they appear in HMS)
thus removing any inconsistencies.
Also the solution relies on the existing locking mechanism in the catalogd to
prevent any other concurrent updates to the table (even via EventsProcessor).
In case of database objects, we will also have a similar eventId which
represents the events on the database object (CREATE, DROP, ALTER database) and
to which the catalogd as synced to. Since there is no refresh database command,
catalogOpExecutor will only update the database eventId when there are DDLs at
the database level (e.g CREATE, DROP, ALTER database)
cc - [~vihangk1] [~kishendas]
was:
h3. Problem Statement
Impala catalogd has Events processor which polls metastore events at regular
intervals to automatically apply changes to the metadata in the catalogd.
However, the current design to detect the self-generated events (DDL/DMLs
coming from the same catalogd) have consistency problems which can cause query
failures under certain circumstances.
h3. Current Design
The current design of self-event detection is based on adding markers to the
HMS objects which are detected when the event is received later to determine if
the event is self-generated or not. These markers constitute a serviceID which
is unique to the catalogd instance and a catalog version number which is unique
for each catalog object. When a DDL is executed, catalogd adds these as object
parameters. When the event is received, Events processor checks the serviceID
and if the catalog version of the current object with the same name in the
catalogd cache and makes a decision of whether to ignore the event or not.
h3. Problems with the current design
The approach is problematic under some circumstances where there are
conflicting DDLs repeated at a faster interval. For example, a sequence of
create/drop table DDLs will generate CREATE_TABLE and DROP_TABLE events. When
the events are received, it is possible that the CREATE_TABLE event is
processed because the catalogd doesn’t have the table in the catalogd cache.
h3. Proposed Solution
The main idea of the solution is to keep track of the last event id for a given
table as eventId which the catalogd has synced to in the Table object. The
events processor ignores any event whose EVENT_ID is less than or equal to the
eventId stored in the table. Once the events processor successfully processes a
given event, it updates the value of eventId in the table before releasing the
table lock. Also, any DDL or refresh operation on the catalogd will follow the
steps given below to update the event id for the table. The solution relies on
the existing locking mechanism in the catalogd to prevent any other concurrent
updates to the table (even via EventsProcessor).
In case of database objects, we will also have a similar eventId which
represents the events on the database object (CREATE, DROP, ALTER database) and
to which the catalogd as synced to. Since there is no refresh database command,
catalogOpExecutor will only update the database eventId when there are DDLs at
the database level (e.g CREATE, DROP, ALTER database)
cc - [~vihangk1] [~kishendas]
> Improved self event detection for event processor in catalogd
> --------------------------------------------------------------
>
> Key: IMPALA-10925
> URL: https://issues.apache.org/jira/browse/IMPALA-10925
> Project: IMPALA
> Issue Type: Epic
> Components: Catalog
> Reporter: Sourabh Goyal
> Assignee: Sourabh Goyal
> Priority: Major
>
> h3. Problem Statement
> Impala catalogd has Events processor which polls metastore events at regular
> intervals to automatically apply changes to the metadata in the catalogd.
> However, the current design to detect the self-generated events (DDL/DMLs
> coming from the same catalogd) have consistency problems which can cause
> query failures under certain circumstances.
>
> h3. Current Design
> The current design of self-event detection is based on adding markers to the
> HMS objects which are detected when the event is received later to determine
> if the event is self-generated or not. These markers constitute a serviceID
> which is unique to the catalogd instance and a catalog version number which
> is unique for each catalog object. When a DDL is executed, catalogd adds
> these as object parameters. When the event is received, Events processor
> checks the serviceID and if the catalog version of the current object with
> the same name in the catalogd cache and makes a decision of whether to ignore
> the event or not.
>
> h3. Problems with the current design
> The approach is problematic under some circumstances where there are
> conflicting DDLs repeated at a faster interval . For example, consider the
> following sequence of alter table operations on a table t1 as per HMS
> 1. alter table t1 from source s1 say another Impala cluster
> 2. alter table t1 from source s2 say another Hive cluster
> 3. alter table t1 from local
> The #3 alter table ddl operation would get reflected in the local cache
> immediately. However, event processor later on would process events from #1
> and #2 above and try to alter the table. In an ideal scenario, these alters
> should have been applied before #3 (i.e in the same order as that in HMS).
> This leaves table t1 in an inconsistent state.
> h3. Proposed Solution
> The main idea of the solution is to keep track of the last event id for a
> given table as eventId which the catalogd has synced to in the Table object.
> The events processor ignores any event whose EVENT_ID is less than or equal
> to the eventId stored in the table. Once the events processor successfully
> processes a given event, it updates the value of eventId in the table before
> releasing the table lock. Also, any DDL or refresh operation on the catalogd
> (from both catalog metastore server and Impala shell) will follow the
> following steps to update the event id for the table:
> 1. Acquire write lock on the table
> 2. Perform ddl operation in HMS
> 3. Sync table till the latest event id (as per HMS) since its last synced
> event id
> The above steps ensure that any concurrent updates applied on a same db/table
> from multiple sources like Hive, Impala or say multiple Impala clusters, get
> reflected in the local catalogd cache (in the same order as they appear in
> HMS) thus removing any inconsistencies.
> Also the solution relies on the existing locking mechanism in the catalogd to
> prevent any other concurrent updates to the table (even via EventsProcessor).
> In case of database objects, we will also have a similar eventId which
> represents the events on the database object (CREATE, DROP, ALTER database)
> and to which the catalogd as synced to. Since there is no refresh database
> command, catalogOpExecutor will only update the database eventId when there
> are DDLs at the database level (e.g CREATE, DROP, ALTER database)
>
> cc - [~vihangk1] [~kishendas]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
