[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[ https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036625#comment-16036625 ] Lefty Leverenz commented on HIVE-16503: --- Doc note: This adds two configs (*hive.llap.mapjoin.memory.oversubscribe.factor* and *hive.llap.memory.oversubscription.max.executors.per.query*) to HiveConf.java, so they need to be documented in the wiki. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC3.0 label. > LLAP: Oversubscribe memory for noconditional task size > -- > > Key: HIVE-16503 > URL: https://issues.apache.org/jira/browse/HIVE-16503 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch, > HIVE-16503.3.patch, HIVE-16503.4.patch > > > When running map joins in llap, it can potentially use more memory for hash > table loading (assuming other executors in the daemons have some memory to > spare). This map join conversion decision has to be made during compilation > that can provide some more room for LLAP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[ https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985210#comment-15985210 ] Prasanth Jayachandran commented on HIVE-16503: -- dynamic_semijoin_user_level.q is failing already in master (tracked in HIVE-16540) and not related to this patch. > LLAP: Oversubscribe memory for noconditional task size > -- > > Key: HIVE-16503 > URL: https://issues.apache.org/jira/browse/HIVE-16503 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch, > HIVE-16503.3.patch, HIVE-16503.4.patch > > > When running map joins in llap, it can potentially use more memory for hash > table loading (assuming other executors in the daemons have some memory to > spare). This map join conversion decision has to be made during compilation > that can provide some more room for LLAP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[
https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982435#comment-15982435
]
Hive QA commented on HIVE-16503:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12864877/HIVE-16503.3.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10629 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join45] (batchId=19)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=140)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4862/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4862/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4862/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12864877 - PreCommit-HIVE-Build
> LLAP: Oversubscribe memory for noconditional task size
> --
>
> Key: HIVE-16503
> URL: https://issues.apache.org/jira/browse/HIVE-16503
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch,
> HIVE-16503.3.patch
>
>
> When running map joins in llap, it can potentially use more memory for hash
> table loading (assuming other executors in the daemons have some memory to
> spare). This map join conversion decision has to be made during compilation
> that can provide some more room for LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[
https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982262#comment-15982262
]
Prasanth Jayachandran commented on HIVE-16503:
--
bq. Is SESSIONS_PER_DEFAULT_QUEUE guaranteed to be >= 1?
The default value is 1 but it doesn't guard against values set by user. In .3
patch added guard against <=0 values and a unit test for the same.
bq. Does it make sense to add range validators for the new settings (0, 1.0)?
.1 patch had RangeValidator but then I removed it as >1.0 values are also
valid. Something like 120% oversubscription is a valid scenario. However under
subscription is not allowed and is guarded by the return value
{code}
return Math.max(maxSize, llapMaxSize);
{code}
where maxSize is the initial no conditional task size.
> LLAP: Oversubscribe memory for noconditional task size
> --
>
> Key: HIVE-16503
> URL: https://issues.apache.org/jira/browse/HIVE-16503
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch,
> HIVE-16503.3.patch
>
>
> When running map joins in llap, it can potentially use more memory for hash
> table loading (assuming other executors in the daemons have some memory to
> spare). This map join conversion decision has to be made during compilation
> that can provide some more room for LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[ https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982242#comment-15982242 ] Gunther Hagleitner commented on HIVE-16503: --- LGTM +1. Some smaller questions: - Is SESSIONS_PER_DEFAULT_QUEUE guaranteed to be >= 1? - Does it make sense to add range validators for the new settings (0, 1.0)? > LLAP: Oversubscribe memory for noconditional task size > -- > > Key: HIVE-16503 > URL: https://issues.apache.org/jira/browse/HIVE-16503 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch > > > When running map joins in llap, it can potentially use more memory for hash > table loading (assuming other executors in the daemons have some memory to > spare). This map join conversion decision has to be made during compilation > that can provide some more room for LLAP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[ https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982193#comment-15982193 ] Prasanth Jayachandran commented on HIVE-16503: -- test failures are unrelated to the patch. accumulo_index.q and accumulo_index.q are already failing in master. skewjoinopt1.q passes locally for me which is probably flaky. I will trigger another test run to make sure anyways. > LLAP: Oversubscribe memory for noconditional task size > -- > > Key: HIVE-16503 > URL: https://issues.apache.org/jira/browse/HIVE-16503 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch > > > When running map joins in llap, it can potentially use more memory for hash > table loading (assuming other executors in the daemons have some memory to > spare). This map join conversion decision has to be made during compilation > that can provide some more room for LLAP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[
https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980680#comment-15980680
]
Hive QA commented on HIVE-16503:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12864689/HIVE-16503.2.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10629 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt1]
(batchId=74)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4854/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4854/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4854/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12864689 - PreCommit-HIVE-Build
> LLAP: Oversubscribe memory for noconditional task size
> --
>
> Key: HIVE-16503
> URL: https://issues.apache.org/jira/browse/HIVE-16503
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16503.1.patch, HIVE-16503.2.patch
>
>
> When running map joins in llap, it can potentially use more memory for hash
> table loading (assuming other executors in the daemons have some memory to
> spare). This map join conversion decision has to be made during compilation
> that can provide some more room for LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[
https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979937#comment-15979937
]
Hive QA commented on HIVE-16503:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12864587/HIVE-16503.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10626 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4841/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4841/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4841/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12864587 - PreCommit-HIVE-Build
> LLAP: Oversubscribe memory for noconditional task size
> --
>
> Key: HIVE-16503
> URL: https://issues.apache.org/jira/browse/HIVE-16503
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16503.1.patch
>
>
> When running map joins in llap, it can potentially use more memory for hash
> table loading (assuming other executors in the daemons have some memory to
> spare). This map join conversion decision has to be made during compilation
> that can provide some more room for LLAP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16503) LLAP: Oversubscribe memory for noconditional task size
[ https://issues.apache.org/jira/browse/HIVE-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979552#comment-15979552 ] Prasanth Jayachandran commented on HIVE-16503: -- [~sseth] [~hagleitn] could someone please take a look? > LLAP: Oversubscribe memory for noconditional task size > -- > > Key: HIVE-16503 > URL: https://issues.apache.org/jira/browse/HIVE-16503 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16503.1.patch > > > When running map joins in llap, it can potentially use more memory for hash > table loading (assuming other executors in the daemons have some memory to > spare). This map join conversion decision has to be made during compilation > that can provide some more room for LLAP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
