Aditya Kishore created DRILL-855:
------------------------------------
Summary: Improve work assignment parallelization
Key: DRILL-855
URL: https://issues.apache.org/jira/browse/DRILL-855
Project: Apache Drill
Issue Type: Improvement
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Currently if only a small number of endpoints have data affinity, all the work
units get assigned to this small subset leaving other endpoints in the cluster
underutilized.
It will lead to poor query performance since the level of parallelization is
reduced below what is otherwise possible.
This JIRA intend to modify the assignment to do a slightly more equitable
distribution of work fragments among all available nodes while slightly
favoring the nodes with data affinity.
This \*favor\* is controlled by the setting
{{"drill.exec.work.affinity.factor"}} (defaults to 1.2f).
As an example, let's say that there are 2 endpoints of which only one is with
affinity and 10 work fragments. In the current scheme of things, all of these
work fragments will get assigned to the single node with affinity. With the
modified scheme and with default affinity factor of 1.2, it will get 20% more
assignment that it would get under equal distribution (i.e. 6:4 instead of 5:5).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
