[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021370#comment-16021370 ] ASF GitHub Bot commented on HIVE-16727: --- GitHub user sankarh opened a pull request: https://github.com/apache/hive/pull/186 HIVE-16727: REPL DUMP for insert event should't fail if the table is already dropped. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sankarh/hive HIVE-16727 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/186.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #186 commit c06a9df4c2c1a735dc8c0139ebdf6451cc6c9017 Author: Sankar Hariappan Date: 2017-05-22T08:15:54Z HIVE-16727: REPL DUMP for insert event should't fail if the table is already dropped. > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021488#comment-16021488 ] Hive QA commented on HIVE-16727: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12869480/HIVE-16727.01.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10750 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=236) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30] (batchId=149) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5400/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5400/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5400/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12869480 - PreCommit-HIVE-Build > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022326#comment-16022326 ] Sankar Hariappan commented on HIVE-16727: - The test failures are irrelevant to the patch! > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025967#comment-16025967 ] anishek commented on HIVE-16727: [~sankarh] * TestReplicationScenarios : The test case seems to test both partitioned and unpartitioned cases in the same test case, It would be easier to spot errors if they were split if into two tests cases one for partition and another for unpartitioned tables, will also additionally make it more readable. Might be possible to abstract most of the commands using the WarehouseInstance representation. * InsertEvent: get_table_core could be used since this is for internal hive classes. * There are unused Imports in most of the classes above. > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026037#comment-16026037 ] Sankar Hariappan commented on HIVE-16727: - [~anishek], Please find my responses here and let me know if any concerns. {quote} TestReplicationScenarios : The test case seems to test both partitioned and unpartitioned cases in the same test case, It would be easier to spot errors if they were split if into two tests cases one for partition and another for unpartitioned tables, will also additionally make it more readable. Might be possible to abstract most of the commands using the WarehouseInstance {quote} - As the InsertEvent handling is common for both partitioned and unpartitioned table, I intended to keep one test case for this scenario. Also, REPL dump will process these insert events one by one which helps to spot the errors easily if one fails.Using WarehouseInstance, I shall able to abstract the dump commands but it is not too many in this test case and this change can be more, so, think of doing it for future patches. :) {quote} InsertEvent: get_table_core could be used since this is for internal hive classes. {quote} - In the base code for Repl dump was using get_table_req after setting this req.setCapabilities(HiveMetaStoreClient.TEST_VERSION); Not sure, why is it there. However, even get_table_req internally invokes get_table_core. So, shouldn't be a problem I guess. {quote} There are unused Imports in most of the classes above. {quote} - Yes, agreed. Will share a patch removing all these unused imports from files modified. > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026044#comment-16026044 ] anishek commented on HIVE-16727: [~sankarh] You are correct in that there is common code for partitioned and unpartitioned code but that can may be abstracted out as a common code, its just that the test becomes difficult to read, And the idea of having two test cases as incrementalInsertDropPartitionTable and incrementalInsertDropUnpartitionTable vs having one method doing both, on failures the test name will tell you what failed rather than looking at the line in the test where it failed and trying to figure out which one of them is failing. I just wanted to bring out the point that readability might be better. I was just going by the comments on the method get_table_core, as you said there is no difference in execution path for anyone of them just additional request object setup and as such in get_table_req method. > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026153#comment-16026153 ] Hive QA commented on HIVE-16727: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12870033/HIVE-16727.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10790 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] (batchId=237) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=145) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5440/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5440/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5440/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12870033 - PreCommit-HIVE-Build > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch, HIVE-16727.02.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026180#comment-16026180 ] anishek commented on HIVE-16727: +1 Looks good . over to you [~thejas]/[~sushanth] > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: HIVE-16727.01.patch, HIVE-16727.02.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.
[ https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040548#comment-16040548 ] ASF GitHub Bot commented on HIVE-16727: --- Github user sankarh closed the pull request at: https://github.com/apache/hive/pull/186 > REPL DUMP for insert event should't fail if the table is already dropped. > - > > Key: HIVE-16727 > URL: https://issues.apache.org/jira/browse/HIVE-16727 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-16727.01.patch, HIVE-16727.02.patch > > > Currently, insert event doesn't log the table object as part of event > notification message. During dump, the table object is obtained from > metastore which can be null if the table is already dropped and hence REPL > DUMP fails. > Steps: > 1. Bootstrap dump/load with a table. > 2. Insert into the table. > 3. Drop the table. > 4. REPL DUMP (incremental). -- This message was sent by Atlassian JIRA (v6.3.15#6346)