[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[
https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842371#comment-15842371
]
Hive QA commented on HIVE-15669:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849060/HIVE-15669.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10988 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=114)
[bucketmapjoin3.q,load_dyn_part5.q,union_date.q,cbo_gby.q,auto_join31.q,auto_sortmerge_join_1.q,join_cond_pushdown_unqual1.q,ppd_outer_join3.q,bucket_map_join_spark3.q,union28.q,statsfs.q,escape_sortby1.q,leftsemijoin.q,union_remove_6.q,join29.q]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=93)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3209/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3209/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3209/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12849060 - PreCommit-HIVE-Build
> LLAP: Improve aging in shortest job first scheduler
> ---
>
> Key: HIVE-15669
> URL: https://issues.apache.org/jira/browse/HIVE-15669
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15669.1.patch, HIVE-15669.1.patch,
> HIVE-15669.2.patch, HIVE-15669.3.patch
>
>
> Under high concurrency, some jobs can gets starved for longer time when
> hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for
> locality).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840584#comment-15840584 ] Rajesh Balamohan commented on HIVE-15669: - Thanks for the patch [~prasanth_j]. LGTM. +1. > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch, HIVE-15669.1.patch, > HIVE-15669.2.patch, HIVE-15669.3.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833797#comment-15833797 ] Prasanth Jayachandran commented on HIVE-15669: -- I am not sure what would be correct delta here. Should it be wrt to dag start time or attempt start time? Don't starve the dag for long time as opposed to don't starve the attempt. My thinking was not starving the dag for a long time. > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833777#comment-15833777 ] Rajesh Balamohan commented on HIVE-15669: - [~prasanth_j] - Should the comparison be based on "getFirstAttemptStartTime" as opposed to "getDagStartTime" ? > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[
https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832822#comment-15832822
]
Hive QA commented on HIVE-15669:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12848458/HIVE-15669.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3086/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3086/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3086/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-01-21 05:33:23.425
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3086/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-01-21 05:33:23.427
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at d9343f6 HIVE-15544 : Support scalar subqueries (Vineet Garg via
Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at d9343f6 HIVE-15544 : Support scalar subqueries (Vineet Garg via
Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-01-21 05:33:24.446
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java
patching file
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
patching file
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestShortestJobFirstComparator.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
DataNucleus Enhancer (version 4.1.6) for API "JDO"
DataNucleus Enhancer : Classpath
>> /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder
ENHANCED (Persistable) :
org.apache.hadoop.hive.metastore.model.MColumnDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList
ENHANCED (Persistable) :
org.apache.hadoop.hive.metastore.model.MStorageDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartition
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MIndex
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRole
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRoleMap
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MGlobalPrivilege
ENHANCED (Persistable) :
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830937#comment-15830937 ] Prasanth Jayachandran commented on HIVE-15669: -- cc [~rajesh.balamohan] > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
