[
https://issues.apache.org/jira/browse/HIVE-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879954#comment-15879954
]
Siddharth Seth commented on HIVE-16013:
---------------------------------------
{code}
+ if (nodeInfo != null && nodeInfo.canAcceptTask()) {
{code}
This gets in the way of determining the next host to use. The requested host
will have canAcceptTask = false (since we already tried and failed a local
allocation). The requestedHostIdx will stay at -1, and we'll the first
available in the reduce allNodes assigned. (The loop to determine the next host
isn't serving any purpose after the patch - since the canAcceptTask check is
removed).
Should've been caught by a unit test :(
I'd say create a separate list for the randomAllocation vs the consistent
locality based allocation. random filtered by canAccept. Locality based would
be the complete list.
Unrelated to this specific jira: what happens in cases where a node is not
found in 'allNodes' during a consistent allocation (node went down between
split generation and actual execution of the query). requestedHostIdx will stay
at -1. Is this handled?
{code}
// no locality-requested, iterate the available hosts in consistent order from
the beginning
{code}
This comment needs to be fixed. We're not iterating the available hosts any
longer.
StackServlet.java - unrelated change?
Nit: Unused import in the Test
> Fragments without locality can stack up on nodes
> ------------------------------------------------
>
> Key: HIVE-16013
> URL: https://issues.apache.org/jira/browse/HIVE-16013
> Project: Hive
> Issue Type: Bug
> Components: llap
> Affects Versions: 2.2.0
> Reporter: Siddharth Seth
> Assignee: Prasanth Jayachandran
> Attachments: HIVE-16013.1.patch
>
>
> When no locality information is provide, task requests can stack up on a node
> because of consistent no selection. When locality information is not provided
> we should fallback to random selection for better work distribution.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
