[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654666#comment-16654666 ] Vihang Karajgaonkar commented on HIVE-20697: yeah, nothing to be done with regards to batching of these tests. > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654633#comment-16654633 ] anishek commented on HIVE-20697: Thanks [~vihangk1], does this mean skipbatching is available for all commits running / submitted ? There is no patch that needs to be uploaded here for now for this correct ? > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[
https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654106#comment-16654106
]
Vihang Karajgaonkar commented on HIVE-20697:
{noformat}
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=243_TestReplicationScenarios, id=243,
moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenarios]]
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=245_TestReplicationScenariosAcrossInstances, id=245,
moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenariosAcrossInstances]]
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=246_TestReplicationScenariosAcidTables, id=246,
moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenariosAcidTables]]
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=247_TestReplicationScenariosIncrementalLoadAcidTables,
id=247, moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenariosIncrementalLoadAcidTables]]
{noformat}
These tests are not batched anymore. They run in batches of 1 test each.
> Some replication tests are super slow and cause batch timeouts
> --
>
> Key: HIVE-20697
> URL: https://issues.apache.org/jira/browse/HIVE-20697
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Some of these tests are taking a long time and can cause test batch timeouts
> given that we only give 40 min for a batch to complete. We should speed these
> tests up.
> TestReplicationScenarios 20 min
> TestReplicationScenariosAcidTables11 min
> TestReplicationScenariosAcrossInstances 5 min 14 sec
> TestReplicationScenariosIncrementalLoadAcidTables 20 min
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[
https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653965#comment-16653965
]
Vihang Karajgaonkar commented on HIVE-20697:
this is what I had added to master-mr2.properties file on ptest server
{noformat}
ut.ql.skipBatching=TestDbTxnManager2
TestTxnCommands2WithSplitUpdateAndVectorization TestTxnCommands2WithSplitUpdate
TestReplicationScenarios TestReplicationScenariosAcidTables
TestReplicationScenariosAcrossInstances
TestReplicationScenariosIncrementalLoadAcidTables
{noformat}
But it should have been in the itests/hive-unit since these tests are in
itests/hive-unit module. I have updated the configs now. Hopefully the next
precommit job should skip batching for these tests.
> Some replication tests are super slow and cause batch timeouts
> --
>
> Key: HIVE-20697
> URL: https://issues.apache.org/jira/browse/HIVE-20697
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Some of these tests are taking a long time and can cause test batch timeouts
> given that we only give 40 min for a batch to complete. We should speed these
> tests up.
> TestReplicationScenarios 20 min
> TestReplicationScenariosAcidTables11 min
> TestReplicationScenariosAcrossInstances 5 min 14 sec
> TestReplicationScenariosIncrementalLoadAcidTables 20 min
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653920#comment-16653920 ] Vihang Karajgaonkar commented on HIVE-20697: Hi [~anishek] Sorry couldn't take a look at this. In fact I had tried adding them to the skipBatching list on the ptest server and I thought it should work. But apparently not. Let me take a look again. > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649994#comment-16649994 ] anishek commented on HIVE-20697: i tried to skip batching of these tests via the master-mr2.porperties but that didnot help, All f this is in HIVE-20679 runs however. run times. The reason the tests take a longer time is because there are various uses cases that are tried with test trying to do it as a end user, there are commands that needs processing in hive like insert / delete / create followed by repl dump / load commands which effectively does the same thing twice with some metadata work in between, hence the longer run times. > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645314#comment-16645314 ] Vihang Karajgaonkar commented on HIVE-20697: I think we can control that to reduce batch size or isolate them. Let me see how can we do that. It may cause a increase in the overall test run time though. Is it not possible to reduce the time taken for these tests? 20 min for a unit test seems very high. > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643028#comment-16643028 ] anishek commented on HIVE-20697: [~vihangk1] do you know how the batch is decided, is there a way to force batches with only couple of tests in batch above ? > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640184#comment-16640184 ] Vihang Karajgaonkar commented on HIVE-20697: cc [~sankarh] [~anishek] [~thejas] Can you please take a look since you are more familiar with this code? Can we disable the tests until this is fixed? > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640173#comment-16640173 ] Vihang Karajgaonkar commented on HIVE-20697: More runtime history https://builds.apache.org/job/PreCommit-HIVE-Build/14249/testReport/org.apache.hadoop.hive.ql.parse/TestReplicationScenarios/history/ https://builds.apache.org/job/PreCommit-HIVE-Build/14249/testReport/org.apache.hadoop.hive.ql.parse/TestReplicationScenariosAcidTables/history/ https://builds.apache.org/job/PreCommit-HIVE-Build/14249/testReport/org.apache.hadoop.hive.ql.parse/TestReplicationScenariosAcrossInstances/history/ https://builds.apache.org/job/PreCommit-HIVE-Build/14249/testReport/org.apache.hadoop.hive.ql.parse/TestReplicationScenariosIncrementalLoadAcidTables/history/ > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
