[jira] [Commented] (HIVE-26341) Replication logs & metrics shows last repl id as 0 when number of events is zero
[ https://issues.apache.org/jira/browse/HIVE-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556655#comment-17556655 ] Haymant Mangla commented on HIVE-26341: --- Committed to master. > Replication logs & metrics shows last repl id as 0 when number of events is > zero > > > Key: HIVE-26341 > URL: https://issues.apache.org/jira/browse/HIVE-26341 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-26341) Replication logs & metrics shows last repl id as 0 when number of events is zero
[ https://issues.apache.org/jira/browse/HIVE-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-26341. --- Resolution: Fixed > Replication logs & metrics shows last repl id as 0 when number of events is > zero > > > Key: HIVE-26341 > URL: https://issues.apache.org/jira/browse/HIVE-26341 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26341) Replication logs & metrics shows last repl id as 0 when number of events is zero
[ https://issues.apache.org/jira/browse/HIVE-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-26341: -- Labels: pull-request-available (was: ) > Replication logs & metrics shows last repl id as 0 when number of events is > zero > > > Key: HIVE-26341 > URL: https://issues.apache.org/jira/browse/HIVE-26341 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26341) Replication logs & metrics shows last repl id as 0 when number of events is zero
[ https://issues.apache.org/jira/browse/HIVE-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26341: - > Replication logs & metrics shows last repl id as 0 when number of events is > zero > > > Key: HIVE-26341 > URL: https://issues.apache.org/jira/browse/HIVE-26341 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HIVE-26326) Support enabling background threads when failover is in progress.
[
https://issues.apache.org/jira/browse/HIVE-26326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555901#comment-17555901
]
Haymant Mangla commented on HIVE-26326:
---
Committed to master.
> Support enabling background threads when failover is in progress.
> -
>
> Key: HIVE-26326
> URL: https://issues.apache.org/jira/browse/HIVE-26326
> Project: Hive
> Issue Type: Sub-task
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> repl.target.for doesn't allow background threads, expose a
> {*}repl.backgroundthread{*}=enable: To force enable the background threads,
> irrespective of repl.target.for, once B takes over as primary.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HIVE-26326) Support enabling background threads when failover is in progress.
[
https://issues.apache.org/jira/browse/HIVE-26326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla resolved HIVE-26326.
---
Resolution: Fixed
> Support enabling background threads when failover is in progress.
> -
>
> Key: HIVE-26326
> URL: https://issues.apache.org/jira/browse/HIVE-26326
> Project: Hive
> Issue Type: Sub-task
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> repl.target.for doesn't allow background threads, expose a
> {*}repl.backgroundthread{*}=enable: To force enable the background threads,
> irrespective of repl.target.for, once B takes over as primary.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HIVE-26338) Repl Dump should fail if source database does not exist.
[ https://issues.apache.org/jira/browse/HIVE-26338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555902#comment-17555902 ] Haymant Mangla commented on HIVE-26338: --- Merged to master. > Repl Dump should fail if source database does not exist. > > > Key: HIVE-26338 > URL: https://issues.apache.org/jira/browse/HIVE-26338 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-26338) Repl Dump should fail if source database does not exist.
[ https://issues.apache.org/jira/browse/HIVE-26338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-26338. --- Resolution: Fixed > Repl Dump should fail if source database does not exist. > > > Key: HIVE-26338 > URL: https://issues.apache.org/jira/browse/HIVE-26338 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-24884) Move top level dump metadata content to be in JSON format
[
https://issues.apache.org/jira/browse/HIVE-24884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla updated HIVE-24884:
--
Assignee: Haymant Mangla (was: Pravin Sinha)
Status: Open (was: Patch Available)
> Move top level dump metadata content to be in JSON format
> -
>
> Key: HIVE-24884
> URL: https://issues.apache.org/jira/browse/HIVE-24884
> Project: Hive
> Issue Type: Task
>Reporter: Pravin Sinha
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {color:#172b4d}The current content for _dumpmetadata file is TAB separated.
> This is not very flexible for extension. A more flexible format like JSON
> based content would be helpful for extending the content.{color}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HIVE-24884) Move top level dump metadata content to be in JSON format
[
https://issues.apache.org/jira/browse/HIVE-24884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla resolved HIVE-24884.
---
Resolution: Won't Fix
> Move top level dump metadata content to be in JSON format
> -
>
> Key: HIVE-24884
> URL: https://issues.apache.org/jira/browse/HIVE-24884
> Project: Hive
> Issue Type: Task
>Reporter: Pravin Sinha
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {color:#172b4d}The current content for _dumpmetadata file is TAB separated.
> This is not very flexible for extension. A more flexible format like JSON
> based content would be helpful for extending the content.{color}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26338) Repl Dump should fail if source database does not exist.
[ https://issues.apache.org/jira/browse/HIVE-26338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26338: - > Repl Dump should fail if source database does not exist. > > > Key: HIVE-26338 > URL: https://issues.apache.org/jira/browse/HIVE-26338 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26326) Support enabling background threads when failover is in progress.
[
https://issues.apache.org/jira/browse/HIVE-26326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla reassigned HIVE-26326:
-
> Support enabling background threads when failover is in progress.
> -
>
> Key: HIVE-26326
> URL: https://issues.apache.org/jira/browse/HIVE-26326
> Project: Hive
> Issue Type: Sub-task
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
>
> repl.target.for doesn't allow background threads, expose a
> {*}repl.backgroundthread{*}=enable: To force enable the background threads,
> irrespective of repl.target.for, once B takes over as primary.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26316) Handle dangling open txns on both src & tgt in unplanned failover.
[ https://issues.apache.org/jira/browse/HIVE-26316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26316: - > Handle dangling open txns on both src & tgt in unplanned failover. > -- > > Key: HIVE-26316 > URL: https://issues.apache.org/jira/browse/HIVE-26316 > Project: Hive > Issue Type: Sub-task >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26301) Fix ACID tables bootstrap during reverse replication in unplanned failover.
[ https://issues.apache.org/jira/browse/HIVE-26301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-26301: -- Summary: Fix ACID tables bootstrap during reverse replication in unplanned failover. (was: Handle ACID tables bootstrap during reverse replication in unplanned failover.) > Fix ACID tables bootstrap during reverse replication in unplanned failover. > --- > > Key: HIVE-26301 > URL: https://issues.apache.org/jira/browse/HIVE-26301 > Project: Hive > Issue Type: Sub-task >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26301) Handle ACID tables bootstrap during reverse replication in unplanned failover.
[ https://issues.apache.org/jira/browse/HIVE-26301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26301: - Assignee: Haymant Mangla > Handle ACID tables bootstrap during reverse replication in unplanned failover. > -- > > Key: HIVE-26301 > URL: https://issues.apache.org/jira/browse/HIVE-26301 > Project: Hive > Issue Type: Sub-task >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-26225) Delete operations in ObjectStore.cleanWriteNotificationEvents should be performed in different transactions
[ https://issues.apache.org/jira/browse/HIVE-26225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-26225. --- Resolution: Fixed > Delete operations in ObjectStore.cleanWriteNotificationEvents should be > performed in different transactions > --- > > Key: HIVE-26225 > URL: https://issues.apache.org/jira/browse/HIVE-26225 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > We need to improve the ObjectStore.cleanWriteNotificationEvents in the same > way as it was done for notification log table in: > https://issues.apache.org/jira/browse/HIVE-24432 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26285) Overwrite database metadata on original source in optimised failover.
[ https://issues.apache.org/jira/browse/HIVE-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-26285: -- Parent: HIVE-25699 Issue Type: Sub-task (was: Bug) > Overwrite database metadata on original source in optimised failover. > - > > Key: HIVE-26285 > URL: https://issues.apache.org/jira/browse/HIVE-26285 > Project: Hive > Issue Type: Sub-task >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26285) Overwrite database metadata on original source in optimised failover.
[ https://issues.apache.org/jira/browse/HIVE-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26285: - > Overwrite database metadata on original source in optimised failover. > - > > Key: HIVE-26285 > URL: https://issues.apache.org/jira/browse/HIVE-26285 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-26237) Check if replication cause metastore connection leakage.
[ https://issues.apache.org/jira/browse/HIVE-26237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-26237. --- Resolution: Fixed Thanks for review @[deniskuzZ!|https://github.com/deniskuzZ] > Check if replication cause metastore connection leakage. > > > Key: HIVE-26237 > URL: https://issues.apache.org/jira/browse/HIVE-26237 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > It is observed that the after running replication unit tests, in some cases, > the final number of metastore connections is not logged as 0. > Sample test : TestReplicationScenarios.testBasic > The last entry of hive.log which records connection count is as follows: > INFO [main] metastore.HiveMetaStoreClient: Closed a connection to metastore, > current connections: 3 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26237) Check if replication cause metastore connection leakage.
[ https://issues.apache.org/jira/browse/HIVE-26237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26237: - > Check if replication cause metastore connection leakage. > > > Key: HIVE-26237 > URL: https://issues.apache.org/jira/browse/HIVE-26237 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > It is observed that the after running replication unit tests, in some cases, > the final number of metastore connections is not logged as 0. > Sample test : TestReplicationScenarios.testBasic > The last entry of hive.log which records connection count is as follows: > INFO [main] metastore.HiveMetaStoreClient: Closed a connection to metastore, > current connections: 3 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26225) Delete operations in ObjectStore.cleanWriteNotificationEvents should be performed in different transactions
[ https://issues.apache.org/jira/browse/HIVE-26225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-26225: - > Delete operations in ObjectStore.cleanWriteNotificationEvents should be > performed in different transactions > --- > > Key: HIVE-26225 > URL: https://issues.apache.org/jira/browse/HIVE-26225 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26225) Delete operations in ObjectStore.cleanWriteNotificationEvents should be performed in different transactions
[ https://issues.apache.org/jira/browse/HIVE-26225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-26225: -- Description: We need to improve the ObjectStore.cleanWriteNotificationEvents in the same way as it was done for notification log table in: https://issues.apache.org/jira/browse/HIVE-24432 > Delete operations in ObjectStore.cleanWriteNotificationEvents should be > performed in different transactions > --- > > Key: HIVE-26225 > URL: https://issues.apache.org/jira/browse/HIVE-26225 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > We need to improve the ObjectStore.cleanWriteNotificationEvents in the same > way as it was done for notification log table in: > https://issues.apache.org/jira/browse/HIVE-24432 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-24697) DbNotificationListener Cleaner thread dies on NoSuchMethodError
[
https://issues.apache.org/jira/browse/HIVE-24697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla resolved HIVE-24697.
---
Resolution: Not A Problem
> DbNotificationListener Cleaner thread dies on NoSuchMethodError
> ---
>
> Key: HIVE-24697
> URL: https://issues.apache.org/jira/browse/HIVE-24697
> Project: Hive
> Issue Type: Bug
>Reporter: Pravin Sinha
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NoSuchMethodError: javax.jdo.Query.close()V
> at
> org.apache.hadoop.hive.metastore.ObjectStore.doCleanNotificationEvents(ObjectStore.java:11025)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:10965)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HIVE-24697) DbNotificationListener Cleaner thread dies on NoSuchMethodError
[
https://issues.apache.org/jira/browse/HIVE-24697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla updated HIVE-24697:
--
Assignee: Haymant Mangla (was: Pravin Sinha)
Status: Open (was: Patch Available)
> DbNotificationListener Cleaner thread dies on NoSuchMethodError
> ---
>
> Key: HIVE-24697
> URL: https://issues.apache.org/jira/browse/HIVE-24697
> Project: Hive
> Issue Type: Bug
>Reporter: Pravin Sinha
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NoSuchMethodError: javax.jdo.Query.close()V
> at
> org.apache.hadoop.hive.metastore.ObjectStore.doCleanNotificationEvents(ObjectStore.java:11025)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:10965)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-25787) Prevent duplicate paths in the fileList while adding an entry to NotifcationLog
[ https://issues.apache.org/jira/browse/HIVE-25787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25787: - Assignee: Haymant Mangla > Prevent duplicate paths in the fileList while adding an entry to > NotifcationLog > --- > > Key: HIVE-25787 > URL: https://issues.apache.org/jira/browse/HIVE-25787 > Project: Hive > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Haymant Mangla >Priority: Major > > As of now, while adding entries to notification logs, in case of retries, > sometimes the same path gets added to the notification log entry, which > during replication leads to failures during copy. > Avoid having same path more than once for single transaction. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25872) Skip tracking of alterDatabase events for replication specific properties.
[ https://issues.apache.org/jira/browse/HIVE-25872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-25872: -- Summary: Skip tracking of alterDatabase events for replication specific properties. (was: Skip tracking of alterDatabase events for replication specific property (repl.last.id).) > Skip tracking of alterDatabase events for replication specific properties. > -- > > Key: HIVE-25872 > URL: https://issues.apache.org/jira/browse/HIVE-25872 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25872) Skip tracking of alterDatabase events for replication specific property (repl.last.id).
[ https://issues.apache.org/jira/browse/HIVE-25872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25872: - > Skip tracking of alterDatabase events for replication specific property > (repl.last.id). > --- > > Key: HIVE-25872 > URL: https://issues.apache.org/jira/browse/HIVE-25872 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25719) Fix flaky test TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest]
[ https://issues.apache.org/jira/browse/HIVE-25719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-25719. --- Resolution: Fixed > Fix flaky test > TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest] > - > > Key: HIVE-25719 > URL: https://issues.apache.org/jira/browse/HIVE-25719 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Haymant Mangla >Priority: Major > > flaky checker failed after 3 attempts with a q.out difference > there seems to be some ID difference - maybe 2 events happened in a different > order? > http://ci.hive.apache.org/job/hive-flaky-check/465/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_replication_metrics_ingest_/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25814) Add entry in replication_metrics table for skipped replication iterations.
[ https://issues.apache.org/jira/browse/HIVE-25814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25814: - > Add entry in replication_metrics table for skipped replication iterations. > -- > > Key: HIVE-25814 > URL: https://issues.apache.org/jira/browse/HIVE-25814 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25790) Make managed table copies handle updates (FileUtils)
[ https://issues.apache.org/jira/browse/HIVE-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25790: - > Make managed table copies handle updates (FileUtils) > > > Key: HIVE-25790 > URL: https://issues.apache.org/jira/browse/HIVE-25790 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25719) Fix flaky test TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest]
[ https://issues.apache.org/jira/browse/HIVE-25719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25719: - Assignee: Haymant Mangla > Fix flaky test > TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest] > - > > Key: HIVE-25719 > URL: https://issues.apache.org/jira/browse/HIVE-25719 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Haymant Mangla >Priority: Major > > flaky checker failed after 3 attempts with a q.out difference > there seems to be some ID difference - maybe 2 events happened in a different > order? > http://ci.hive.apache.org/job/hive-flaky-check/465/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_replication_metrics_ingest_/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25719) Fix flaky test TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest]
[ https://issues.apache.org/jira/browse/HIVE-25719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452282#comment-17452282 ] Haymant Mangla commented on HIVE-25719: --- Flaky test check green. http://ci.hive.apache.org/job/hive-flaky-check/484/ > Fix flaky test > TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest] > - > > Key: HIVE-25719 > URL: https://issues.apache.org/jira/browse/HIVE-25719 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Haymant Mangla >Priority: Major > > flaky checker failed after 3 attempts with a q.out difference > there seems to be some ID difference - maybe 2 events happened in a different > order? > http://ci.hive.apache.org/job/hive-flaky-check/465/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_replication_metrics_ingest_/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25719) Fix flaky test TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest]
[ https://issues.apache.org/jira/browse/HIVE-25719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25719 started by Haymant Mangla. - > Fix flaky test > TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest] > - > > Key: HIVE-25719 > URL: https://issues.apache.org/jira/browse/HIVE-25719 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Haymant Mangla >Priority: Major > > flaky checker failed after 3 attempts with a q.out difference > there seems to be some ID difference - maybe 2 events happened in a different > order? > http://ci.hive.apache.org/job/hive-flaky-check/465/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_replication_metrics_ingest_/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25719) Fix flaky test TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest]
[ https://issues.apache.org/jira/browse/HIVE-25719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452281#comment-17452281 ] Haymant Mangla commented on HIVE-25719: --- Fixed as part of[ HIVE-25756|http://issues.apache.org/jira/browse/HIVE-25756] > Fix flaky test > TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest] > - > > Key: HIVE-25719 > URL: https://issues.apache.org/jira/browse/HIVE-25719 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > flaky checker failed after 3 attempts with a q.out difference > there seems to be some ID difference - maybe 2 events happened in a different > order? > http://ci.hive.apache.org/job/hive-flaky-check/465/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_replication_metrics_ingest_/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (HIVE-25719) Fix flaky test TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest]
[ https://issues.apache.org/jira/browse/HIVE-25719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452281#comment-17452281 ] Haymant Mangla edited comment on HIVE-25719 at 12/2/21, 10:02 AM: -- Fixed as part of HIVE-25756 was (Author: haymant): Fixed as part of[ HIVE-25756|http://issues.apache.org/jira/browse/HIVE-25756] > Fix flaky test > TestMiniLlapLocalCliDriver#testCliDriver[replication_metrics_ingest] > - > > Key: HIVE-25719 > URL: https://issues.apache.org/jira/browse/HIVE-25719 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > flaky checker failed after 3 attempts with a q.out difference > there seems to be some ID difference - maybe 2 events happened in a different > order? > http://ci.hive.apache.org/job/hive-flaky-check/465/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_replication_metrics_ingest_/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25756) Fix replication metrics backward compatibility issue.
[ https://issues.apache.org/jira/browse/HIVE-25756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25756: - > Fix replication metrics backward compatibility issue. > - > > Key: HIVE-25756 > URL: https://issues.apache.org/jira/browse/HIVE-25756 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25731) Differentiate between failover revert and complete
[ https://issues.apache.org/jira/browse/HIVE-25731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-25731. --- Resolution: Won't Do > Differentiate between failover revert and complete > -- > > Key: HIVE-25731 > URL: https://issues.apache.org/jira/browse/HIVE-25731 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25731) Differentiate between failover revert and complete
[ https://issues.apache.org/jira/browse/HIVE-25731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25731: - > Differentiate between failover revert and complete > -- > > Key: HIVE-25731 > URL: https://issues.apache.org/jira/browse/HIVE-25731 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25596) Compress Hive Replication Metrics while storing
[ https://issues.apache.org/jira/browse/HIVE-25596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437796#comment-17437796 ] Haymant Mangla commented on HIVE-25596: --- Attached two sample RM_Progress field pre and post compression. # For k=5, Original String:14276 Compressed String: 5564 # For k=10, Original String:16559 Compressed String: 7176 > Compress Hive Replication Metrics while storing > --- > > Key: HIVE-25596 > URL: https://issues.apache.org/jira/browse/HIVE-25596 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Attachments: CompressedRM_Progress(k=10), CompressedRM_Progress(k=5), > PlainTextRM_Progress(k=10), PlainTextRM_Progress(k=5) > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Compress the json fields of sys.replication_metrics table to optimise RDBMS > space usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25596) Compress Hive Replication Metrics while storing
[ https://issues.apache.org/jira/browse/HIVE-25596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-25596: -- Attachment: PlainTextRM_Progress(k=10) PlainTextRM_Progress(k=5) CompressedRM_Progress(k=10) CompressedRM_Progress(k=5) > Compress Hive Replication Metrics while storing > --- > > Key: HIVE-25596 > URL: https://issues.apache.org/jira/browse/HIVE-25596 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Attachments: CompressedRM_Progress(k=10), CompressedRM_Progress(k=5), > PlainTextRM_Progress(k=10), PlainTextRM_Progress(k=5) > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Compress the json fields of sys.replication_metrics table to optimise RDBMS > space usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25602) Fix failover metadata file path in repl load execution.
[
https://issues.apache.org/jira/browse/HIVE-25602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla resolved HIVE-25602.
---
Resolution: Resolved
> Fix failover metadata file path in repl load execution.
> ---
>
> Key: HIVE-25602
> URL: https://issues.apache.org/jira/browse/HIVE-25602
> Project: Hive
> Issue Type: Bug
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When executed through scheduled queries, repl load fails with following error:
>
> {code:java}
> Reading failover metadata from file:
> 2021-10-08 02:02:51,824 ERROR org.apache.hadoop.hive.ql.Driver: [Scheduled
> Query Executor(schedule:repl_load_p1, execution_id:43)]: FAILED:
> SemanticException java.io.FileNotFoundException: File does not exist:
> /user/hive/repl/c291cmNl/36d04dfd-ee5d-4faf-bc0a-ae8d665f95f9/_failovermetadata
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2035)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:737)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25609) Preserve XAttrs in normal file copy case.
[ https://issues.apache.org/jira/browse/HIVE-25609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25609: - > Preserve XAttrs in normal file copy case. > - > > Key: HIVE-25609 > URL: https://issues.apache.org/jira/browse/HIVE-25609 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25602) Fix failover metadata file path in repl load execution.
[
https://issues.apache.org/jira/browse/HIVE-25602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla updated HIVE-25602:
--
Issue Type: Bug (was: Improvement)
> Fix failover metadata file path in repl load execution.
> ---
>
> Key: HIVE-25602
> URL: https://issues.apache.org/jira/browse/HIVE-25602
> Project: Hive
> Issue Type: Bug
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
>
> When executed through scheduled queries, repl load fails with following error:
>
> {code:java}
> Reading failover metadata from file:
> 2021-10-08 02:02:51,824 ERROR org.apache.hadoop.hive.ql.Driver: [Scheduled
> Query Executor(schedule:repl_load_p1, execution_id:43)]: FAILED:
> SemanticException java.io.FileNotFoundException: File does not exist:
> /user/hive/repl/c291cmNl/36d04dfd-ee5d-4faf-bc0a-ae8d665f95f9/_failovermetadata
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2035)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:737)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25602) Fix failover metadata file path in repl load execution.
[
https://issues.apache.org/jira/browse/HIVE-25602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla reassigned HIVE-25602:
-
> Fix failover metadata file path in repl load execution.
> ---
>
> Key: HIVE-25602
> URL: https://issues.apache.org/jira/browse/HIVE-25602
> Project: Hive
> Issue Type: Improvement
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
>
> When executed through scheduled queries, repl load fails with following error:
>
> {code:java}
> Reading failover metadata from file:
> 2021-10-08 02:02:51,824 ERROR org.apache.hadoop.hive.ql.Driver: [Scheduled
> Query Executor(schedule:repl_load_p1, execution_id:43)]: FAILED:
> SemanticException java.io.FileNotFoundException: File does not exist:
> /user/hive/repl/c291cmNl/36d04dfd-ee5d-4faf-bc0a-ae8d665f95f9/_failovermetadata
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2035)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:737)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25596) Compress Hive Replication Metrics while storing
[ https://issues.apache.org/jira/browse/HIVE-25596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25596: - > Compress Hive Replication Metrics while storing > --- > > Key: HIVE-25596 > URL: https://issues.apache.org/jira/browse/HIVE-25596 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > Compress the json fields of sys.replication_metrics table to optimise RDBMS > space usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25534) Don't preserve FileAttribute.XATTR to initialise distcp.
[
https://issues.apache.org/jira/browse/HIVE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla updated HIVE-25534:
--
Description:
Remove the preserve xattr while calling distcp.
{code:java}
2021-08-23 10:06:18,485 ERROR org.apache.hadoop.tools.DistCp:
[HiveServer2-Background-Pool: Thread-73]: XAttrs not supported on at least one
file system:
org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not
supported for file system: s3a://hmangla1-dev
at
org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:337)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:304)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:214)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]{code}
was:
Remove the preserve xattr while calling distcp.
2021-08-23 10:06:18,485 ERROR org.apache.hadoop.tools.DistCp:
[HiveServer2-Background-Pool: Thread-73]: XAttrs not supported on at least one
file system:
org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not
supported for file system: s3a://hmangla1-dev
at
org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at
org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:337)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:304)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:214)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> Don't preserve FileAttribute.XATTR to initialise distcp.
>
>
> Key: HIVE-25534
> URL: https://issues.apache.org/jira/browse/HIVE-25534
> Project: Hive
> Issue Type: Bug
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
>
> Remove the preserve xattr while calling distcp.
> {code:java}
> 2021-08-23 10:06:18,485 ERROR org.apache.hadoop.tools.DistCp:
> [HiveServer2-Background-Pool: Thread-73]: XAttrs not supported on at least
> one file system:
> org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not
> supported for file system: s3a://hmangla1-dev
> at
> org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:337)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:304)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:214)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25534) Don't preserve FileAttribute.XATTR to initialise distcp.
[ https://issues.apache.org/jira/browse/HIVE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25534: - > Don't preserve FileAttribute.XATTR to initialise distcp. > > > Key: HIVE-25534 > URL: https://issues.apache.org/jira/browse/HIVE-25534 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > Remove the preserve xattr while calling distcp. > 2021-08-23 10:06:18,485 ERROR org.apache.hadoop.tools.DistCp: > [HiveServer2-Background-Pool: Thread-73]: XAttrs not supported on at least > one file system: > org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not > supported for file system: s3a://hmangla1-dev > at > org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513) > ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?] > at > org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:337) > ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?] > at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:304) > ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?] > at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:214) > ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?] > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193) > ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25475) TestStatsReplicationScenarios.testForParallelBootstrapLoad is unstable
[
https://issues.apache.org/jira/browse/HIVE-25475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405300#comment-17405300
]
Haymant Mangla commented on HIVE-25475:
---
Committed to master.
Thanks for the review, [~maheshk114]
> TestStatsReplicationScenarios.testForParallelBootstrapLoad is unstable
> --
>
> Key: HIVE-25475
> URL: https://issues.apache.org/jira/browse/HIVE-25475
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> http://ci.hive.apache.org/job/hive-flaky-check/389/
> {code}
> 16:19:18 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 141.73 s <<< FAILURE! - in
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios
> 16:19:18 [ERROR]
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testForParallelBootstrapLoad
> Time elapsed: 122.979 s <<< ERROR!
> 16:19:18 org.apache.hadoop.hive.ql.metadata.HiveException
> 16:19:18 at
> org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5032)
> 16:19:18 at
> org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3348)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:429)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> 16:19:18 at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> 16:19:18 at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
> 16:19:18 at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:230)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:235)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.load(WarehouseInstance.java:309)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.dumpLoadVerify(TestStatsReplicationScenarios.java:359)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testStatsReplicationCommon(TestStatsReplicationScenarios.java:663)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testForParallelBootstrapLoad(TestStatsReplicationScenarios.java:688)
> 16:19:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 16:19:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 16:19:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 16:19:18 at java.lang.reflect.Method.invoke(Method.java:498)
> 16:19:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 16:19:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 16:19:18 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 16:19:18 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 16:19:18 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 16:19:18 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 16:19:18 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 16:19:18 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 16:19:18 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 16:19:18 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 16:19:18 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 16:19:18 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
[jira] [Resolved] (HIVE-25475) TestStatsReplicationScenarios.testForParallelBootstrapLoad is unstable
[
https://issues.apache.org/jira/browse/HIVE-25475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla resolved HIVE-25475.
---
Resolution: Fixed
> TestStatsReplicationScenarios.testForParallelBootstrapLoad is unstable
> --
>
> Key: HIVE-25475
> URL: https://issues.apache.org/jira/browse/HIVE-25475
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> http://ci.hive.apache.org/job/hive-flaky-check/389/
> {code}
> 16:19:18 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 141.73 s <<< FAILURE! - in
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios
> 16:19:18 [ERROR]
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testForParallelBootstrapLoad
> Time elapsed: 122.979 s <<< ERROR!
> 16:19:18 org.apache.hadoop.hive.ql.metadata.HiveException
> 16:19:18 at
> org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5032)
> 16:19:18 at
> org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3348)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:429)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> 16:19:18 at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> 16:19:18 at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
> 16:19:18 at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:230)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:235)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.load(WarehouseInstance.java:309)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.dumpLoadVerify(TestStatsReplicationScenarios.java:359)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testStatsReplicationCommon(TestStatsReplicationScenarios.java:663)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testForParallelBootstrapLoad(TestStatsReplicationScenarios.java:688)
> 16:19:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 16:19:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 16:19:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 16:19:18 at java.lang.reflect.Method.invoke(Method.java:498)
> 16:19:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 16:19:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 16:19:18 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 16:19:18 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 16:19:18 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 16:19:18 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 16:19:18 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 16:19:18 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 16:19:18 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 16:19:18 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 16:19:18 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 16:19:18 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 16:19:18 at
>
[jira] [Assigned] (HIVE-25475) TestStatsReplicationScenarios.testForParallelBootstrapLoad is unstable
[
https://issues.apache.org/jira/browse/HIVE-25475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haymant Mangla reassigned HIVE-25475:
-
Assignee: Haymant Mangla
> TestStatsReplicationScenarios.testForParallelBootstrapLoad is unstable
> --
>
> Key: HIVE-25475
> URL: https://issues.apache.org/jira/browse/HIVE-25475
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Haymant Mangla
>Priority: Major
>
> http://ci.hive.apache.org/job/hive-flaky-check/389/
> {code}
> 16:19:18 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 141.73 s <<< FAILURE! - in
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios
> 16:19:18 [ERROR]
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testForParallelBootstrapLoad
> Time elapsed: 122.979 s <<< ERROR!
> 16:19:18 org.apache.hadoop.hive.ql.metadata.HiveException
> 16:19:18 at
> org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5032)
> 16:19:18 at
> org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3348)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:429)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> 16:19:18 at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> 16:19:18 at
> org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> 16:19:18 at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> 16:19:18 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> 16:19:18 at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
> 16:19:18 at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:230)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:235)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.load(WarehouseInstance.java:309)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.dumpLoadVerify(TestStatsReplicationScenarios.java:359)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testStatsReplicationCommon(TestStatsReplicationScenarios.java:663)
> 16:19:18 at
> org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenarios.testForParallelBootstrapLoad(TestStatsReplicationScenarios.java:688)
> 16:19:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 16:19:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 16:19:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 16:19:18 at java.lang.reflect.Method.invoke(Method.java:498)
> 16:19:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 16:19:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 16:19:18 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 16:19:18 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 16:19:18 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 16:19:18 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 16:19:18 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 16:19:18 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 16:19:18 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 16:19:18 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 16:19:18 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 16:19:18 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 16:19:18 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 16:19:18 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 16:19:18 at
>
[jira] [Assigned] (HIVE-25455) Include root cause of error in CopyUtils exception handling usecase.
[ https://issues.apache.org/jira/browse/HIVE-25455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25455: - > Include root cause of error in CopyUtils exception handling usecase. > > > Key: HIVE-25455 > URL: https://issues.apache.org/jira/browse/HIVE-25455 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25433) DistCp job fails with yarn map compatibility mode issue.
[ https://issues.apache.org/jira/browse/HIVE-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25433: - > DistCp job fails with yarn map compatibility mode issue. > > > Key: HIVE-25433 > URL: https://issues.apache.org/jira/browse/HIVE-25433 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24946) Handle failover case during Repl Load
[ https://issues.apache.org/jira/browse/HIVE-24946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-24946: -- Description: To handle: # Introduce two states of failover db property to denote nature of database at the time failover was initiated. # If failover start config is enabled and dump directory contains failover marker file, then in incremental load as a preAckTask, we should ## Remove repl.target.for from target db. ## Set repl.failover.endpoint = "TARGET" ## Updated the replication metrics saying that target cluster is failover ready # In the first dump operation in reverse direction, presence of failover ready marker and repl.failover.endpoint = "TARGET" will be used as indicator for bootstrap iteration. # In any dump operation except the first dump operation in reverse dxn, if repl.failover.endpoint is set for db and failover start config is set to false, then remove this property. # In incremental load, if the failover start config is disabled, then add repl.target.for and remove repl.failover.endpoint if present. was: * Update metric during load to capture the readiness for failover * Remove repl.target.for property on target cluster * Prepare the dump directory to be used during failover first dump operation > Handle failover case during Repl Load > - > > Key: HIVE-24946 > URL: https://issues.apache.org/jira/browse/HIVE-24946 > Project: Hive > Issue Type: New Feature >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > To handle: > # Introduce two states of failover db property to denote nature of database > at the time failover was initiated. > # If failover start config is enabled and dump directory contains failover > marker file, then in incremental load as a preAckTask, we should > ## Remove repl.target.for from target db. > ## Set repl.failover.endpoint = "TARGET" > ## Updated the replication metrics saying that target cluster is failover > ready > # In the first dump operation in reverse direction, presence of failover > ready marker and repl.failover.endpoint = "TARGET" will be used as indicator > for bootstrap iteration. > # In any dump operation except the first dump operation in reverse dxn, if > repl.failover.endpoint is set for db and failover start config is set to > false, then remove this property. > # In incremental load, if the failover start config is disabled, then add > repl.target.for and remove repl.failover.endpoint if present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24918) Handle failover case during Repl Dump
[ https://issues.apache.org/jira/browse/HIVE-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-24918: -- Description: To handle: a) If failover start config is enabled, dump operation will perform following tasks: 1. Enable repl.failover.enabled prop for srcDb. 2. Abort all the open transactions related to db under replication. 3. Capture current Notification Id as cursor point. 4. Fetch list of all open transactions. If still there are any txns for srcDb, abort them. Keep track of open txns, aborted txns and txnsWithoutLocks. 5. Capture current notification ID as failover eventId and dump all the events till this eventId. 6. Store all the info captured in Step 2, 3, 4, 5 in __failover_metadata_ in dump dir. 7. Dump all the events till failover eventId. 8. Pass location of __failover_metadata, failover eventId_ and failover ready status in the replication_metrics. 9. Just before creation of DUMP ACK file, place failover_ready marker file in dump dir. b) Skip subsequent repl dump instances once we have the failover_ready marker file placed. c) If failover start config is disabled + dump dir contains _failover__ready marker file + repl.failover.enabled is unset for srcDb, then it should be treated as a bootstrap dump as this is the first dump operation in reverse direction. e) If failover start config is disabled + dump dir contains _failover__ready marker file + repl.failover.enabled is set for srcDb, then dump has to rollback the failover process. was: To handle: a) Whenever user wants to go ahead with failover, during the next or subsequent repl dump operation upon confirming that there are no pending open transaction events, It should create a _failover_ready marker file in the dump dir. This marker file would contain scheduled query name that has generated this dump. b) Skip next repl dump instances once we have the marker file placed. > Handle failover case during Repl Dump > - > > Key: HIVE-24918 > URL: https://issues.apache.org/jira/browse/HIVE-24918 > Project: Hive > Issue Type: New Feature >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 7h 50m > Remaining Estimate: 0h > > To handle: > a) If failover start config is enabled, dump operation will perform following > tasks: > 1. Enable repl.failover.enabled prop for srcDb. > 2. Abort all the open transactions related to db under replication. > 3. Capture current Notification Id as cursor point. > 4. Fetch list of all open transactions. If still there are any txns > for srcDb, abort them. Keep track of open txns, aborted txns and > txnsWithoutLocks. > 5. Capture current notification ID as failover eventId and dump all > the events till this eventId. > 6. Store all the info captured in Step 2, 3, 4, 5 in > __failover_metadata_ in dump dir. > 7. Dump all the events till failover eventId. > 8. Pass location of __failover_metadata, failover eventId_ and > failover ready status in the replication_metrics. > 9. Just before creation of DUMP ACK file, place failover_ready marker > file in dump dir. > b) Skip subsequent repl dump instances once we have the failover_ready marker > file placed. > c) If failover start config is disabled + dump dir contains _failover__ready > marker file + repl.failover.enabled is unset for srcDb, then it should be > treated as a bootstrap dump as this is the first dump operation in reverse > direction. > e) If failover start config is disabled + dump dir contains _failover__ready > marker file + repl.failover.enabled is set for srcDb, then dump has to > rollback the failover process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25304) Fix test org.apache.hive.spark.client.rpc.TestRpc.testServerPort
[ https://issues.apache.org/jira/browse/HIVE-25304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla resolved HIVE-25304. --- Resolution: Fixed > Fix test org.apache.hive.spark.client.rpc.TestRpc.testServerPort > > > Key: HIVE-25304 > URL: https://issues.apache.org/jira/browse/HIVE-25304 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25246) Fix the clean up of open repl created transactions
[ https://issues.apache.org/jira/browse/HIVE-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25246: - > Fix the clean up of open repl created transactions > -- > > Key: HIVE-25246 > URL: https://issues.apache.org/jira/browse/HIVE-25246 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25154) Disable StatsUpdaterThread and PartitionManagementTask for db that is being failoved over.
[ https://issues.apache.org/jira/browse/HIVE-25154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-25154: -- Attachment: HIVE-25154.patch > Disable StatsUpdaterThread and PartitionManagementTask for db that is being > failoved over. > -- > > Key: HIVE-25154 > URL: https://issues.apache.org/jira/browse/HIVE-25154 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Attachments: HIVE-25154.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25154) Disable StatsUpdaterThread and PartitionManagementTask for db that is being failoved over.
[ https://issues.apache.org/jira/browse/HIVE-25154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25154: - > Disable StatsUpdaterThread and PartitionManagementTask for db that is being > failoved over. > -- > > Key: HIVE-25154 > URL: https://issues.apache.org/jira/browse/HIVE-25154 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25086) Create Ranger Deny Policy for replication db in all cases if hive.repl.ranger.target.deny.policy is set to true.
[ https://issues.apache.org/jira/browse/HIVE-25086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25086: - > Create Ranger Deny Policy for replication db in all cases if > hive.repl.ranger.target.deny.policy is set to true. > > > Key: HIVE-25086 > URL: https://issues.apache.org/jira/browse/HIVE-25086 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25002) modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
[ https://issues.apache.org/jira/browse/HIVE-25002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25002: - > modify condition for target of replication in statsUpdaterThread and > PartitionManagementTask > > > Key: HIVE-25002 > URL: https://issues.apache.org/jira/browse/HIVE-25002 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24918) Handle failover case during Repl Dump
[ https://issues.apache.org/jira/browse/HIVE-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-24918: -- Description: To handle: a) Whenever user wants to go ahead with failover, during the next or subsequent repl dump operation upon confirming that there are no pending open transaction events, It should create a _failover_ready marker file in the dump dir. This marker file would contain scheduled query name that has generated this dump. b) Skip next repl dump instances once we have the marker file placed. was: To handle: a) Whenever user wants to go ahead with failover, during the next or subsequent repl dump operation upon confirming that there are no pending open transaction events, in should create a _failover_ready marker file in the dump dir. b) Skip next repl dump instances once we have the marker file placed. > Handle failover case during Repl Dump > - > > Key: HIVE-24918 > URL: https://issues.apache.org/jira/browse/HIVE-24918 > Project: Hive > Issue Type: New Feature >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > To handle: > a) Whenever user wants to go ahead with failover, during the next or > subsequent repl dump operation upon confirming that there are no pending open > transaction events, It should create a _failover_ready marker file in the > dump dir. This marker file would contain scheduled query name > that has generated this dump. > b) Skip next repl dump instances once we have the marker file placed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24953) Set repl.source.for property in the db if db is under replication in incremental dump.
[ https://issues.apache.org/jira/browse/HIVE-24953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-24953: - > Set repl.source.for property in the db if db is under replication in > incremental dump. > -- > > Key: HIVE-24953 > URL: https://issues.apache.org/jira/browse/HIVE-24953 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24946) Handle failover case during Repl Load
[ https://issues.apache.org/jira/browse/HIVE-24946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-24946: - > Handle failover case during Repl Load > - > > Key: HIVE-24946 > URL: https://issues.apache.org/jira/browse/HIVE-24946 > Project: Hive > Issue Type: New Feature >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > * Update metric during load to capture the readiness for failover > * Handle repl.source.for and repl.target.for properties on target cluster > (Enable repl.source.for and disable repl.target.for) > * Prepare the dump directory to be used during failover first dump operation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24918) Handle failover case during Repl Dump
[ https://issues.apache.org/jira/browse/HIVE-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-24918: - > Handle failover case during Repl Dump > - > > Key: HIVE-24918 > URL: https://issues.apache.org/jira/browse/HIVE-24918 > Project: Hive > Issue Type: New Feature >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > To handle: > a) Whenever user wants to go ahead with failover, during the next or > subsequent repl dump operation upon confirming that there are no pending open > transaction events, in should create a _failover_ready marker file in the > dump dir. > b) Skip next repl dump instances once we have the marker file placed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24912) Support to add repl.target.for property during incremental run
[ https://issues.apache.org/jira/browse/HIVE-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-24912: - > Support to add repl.target.for property during incremental run > -- > > Key: HIVE-24912 > URL: https://issues.apache.org/jira/browse/HIVE-24912 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24909) Skip the repl events from getting logged in notification log
[ https://issues.apache.org/jira/browse/HIVE-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-24909: - > Skip the repl events from getting logged in notification log > > > Key: HIVE-24909 > URL: https://issues.apache.org/jira/browse/HIVE-24909 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > > Currently REPL dump events are logged and replicated as a part of replication > policy. Whenever one replication cycle completed, we always have one > transaction left open on the target corresponding to repl dump operation. > This will never be caught up without manually dealing with the transaction on > target cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24783) Store currentNotificationID on target during repl load operation
[ https://issues.apache.org/jira/browse/HIVE-24783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294033#comment-17294033 ] Haymant Mangla commented on HIVE-24783: --- My Pleasure. > Store currentNotificationID on target during repl load operation > > > Key: HIVE-24783 > URL: https://issues.apache.org/jira/browse/HIVE-24783 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24783) Store currentNotificationID on target during repl load operation
[ https://issues.apache.org/jira/browse/HIVE-24783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24783 started by Haymant Mangla. - > Store currentNotificationID on target during repl load operation > > > Key: HIVE-24783 > URL: https://issues.apache.org/jira/browse/HIVE-24783 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24783) Store currentNotificationID on target during repl load operation
[ https://issues.apache.org/jira/browse/HIVE-24783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-24783: - > Store currentNotificationID on target during repl load operation > > > Key: HIVE-24783 > URL: https://issues.apache.org/jira/browse/HIVE-24783 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
