[
https://issues.apache.org/jira/browse/HIVE-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052380#comment-17052380
]
Gopal Vijayaraghavan commented on HIVE-22966:
---------------------------------------------
bq. even thought this patch takes into account task aging we do not cure the
long-tail task issue and we need to properly take care of it.
This entire patch is hiding in the shadow of YARN FIFO assumptions in long tail
task scheduling order code inside Tez.
https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java#L591
There's also a somewhat equivalent version for the splits as well
https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/MRInputHelpers.java#L501
So Tez explicitly picks the biggest splits and the heaviest skewed reducers to
start first, which is mostly relevant for query latency when we have a large
number of tasks and a low number of executors.
That is why this patch makes a difference, because at the same priority, we get
FIFO back.
> LLAP: Consider including waitTime for comparing attempts in same vertex
> -----------------------------------------------------------------------
>
> Key: HIVE-22966
> URL: https://issues.apache.org/jira/browse/HIVE-22966
> Project: Hive
> Issue Type: Improvement
> Components: llap
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22966.3.patch, HIVE-22966.4.patch
>
>
> When attempts are compared within same vertex, it should pick up the attempt
> with longest wait time to avoid starvation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
