[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-16785: - Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to master. Thanks for the patch and good test coverage [~sankarh]! Thanks for the review [~anishek]! > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: HIVE-16785.05.patch > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: (was: HIVE-16785.05.patch) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Open (was: Patch Available) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: HIVE-16785.05.patch Added 05.patch with below updates. - Reverted changes in WarehouseInstance to fix the multi-instance metastore problem. This fix wasn't proper. - Cleaned-up ReplicationSpec class for too many duplicate methods. - Fixed Abishek's comments. Request [~anishek] to review this patch! > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch, HIVE-16785.05.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Open (was: Patch Available) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: HIVE-16785.04.patch Attached 04.patch with fixes for Anishek's accepted comments. > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch, HIVE-16785.04.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Open (was: Patch Available) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: HIVE-16785.03.patch Added 03.patch with below updates. - Bug fix in CreateFunctionHandler where replicationSpec was set incorrectly. - Fixed a framework issue in WarehouseInstance where metastore should be instantiated with same hive configuration all the time else it fails due to HIVE-16844. - Added few debug logs in case of repeated events applied on replica warehouse. > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch, > HIVE-16785.03.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Open (was: Patch Available) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Open (was: Patch Available) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: HIVE-16785.02.patch Added 02.patch with below updates. - Added idempotent behaviour for CREATE FUNCTION and DROP FUNCTION operations. > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch, HIVE-16785.02.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Fix Version/s: 3.0.0 > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16785.01.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Open (was: Patch Available) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16785.01.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Attachment: HIVE-16785.01.patch Added 01.patch with below updates. - Added repl.last.id validation for RENAME and TRUNCATE events against table/partition objects. If the operating object doesn't exist, then the event is noop because it means, the table/partition is already dropped/renamed and hence no need to apply this event. - Updated createReplImportTasks method to validate the eventId against repl.last.id of table object if exists and if not exists, then validate against database object. This ensures, table is created only if not exist and is not dropped or renamed already. - Removed unnecessary handling of drop+create in-case of partition policy different from the new table. - Update table or database object's repl.last.id only if the last applied eventId is newer than the object itself. In base code, it was always updated. Request [~anishek]/[~sushanth] to review the patch! cc [~thejas] > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16785.01.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16785) Ensure replication actions are idempotent if any series of events are applied again.
[ https://issues.apache.org/jira/browse/HIVE-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16785: Status: Patch Available (was: Open) > Ensure replication actions are idempotent if any series of events are applied > again. > > > Key: HIVE-16785 > URL: https://issues.apache.org/jira/browse/HIVE-16785 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16785.01.patch > > > Some of the events(ALTER, RENAME, TRUNCATE) are not idempotent and hence > leads to failure of REPL LOAD if applied twice or applied on an object which > is latest than current event. For example, if TRUNCATE is applied on a table > which is already dropped will fail instead of noop. > Also, need to consider the scenario where the object is missing while > applying an event. For example, if RENAME_TABLE event is applied on target > where the old table is missing should validate if table should be recreated > or should treat the event as noop. This can be done by verifying the DB level > last repl ID against the current event ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
