Hello Michael Ho, Lars Volker, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14026
to look at the new patch set (#6).
Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small
clusters
......................................................................
IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
In the original change for consistent scheduling, if a cluster has
fewer nodes than the number of remote executor candidates, then
the scheduler falls back to using the old SelectRemoteExecutor().
SelectRemoteExecutor() considers all backends and picks the backend
with the least assigned bytes; to break ties, it uses randomness.
This means that clusters with fewer backends than
num_remote_executor_candidates do not have consistent placement.
For the file handle cache (the original user of consistent
placement), this is not a major problem. However, for data caching,
it can result in slower warm up of the data cache and greater
duplication of the same data across different nodes.
This changes the algorithm to use consistent placement even for
small clusters (num nodes <= num_remote_executor_candidates).
To make this more predictable, it increases the maximum number
of iterations.
This also changes GetRemoteExecutorCandidates() to return the
candidates in the order that they were selected. While still
using a set for detecting duplicate backends, the vector of
distinct backends is constructed directly rather than by
iterating over the set.
Testing:
- Modify the scheduler-test backend test to verify that small
clusters use consistent scheduling.
Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741
---
M be/src/scheduling/scheduler-test.cc
M be/src/scheduling/scheduler.cc
2 files changed, 84 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/14026/6
--
To view, visit http://gerrit.cloudera.org:8080/14026
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741
Gerrit-Change-Number: 14026
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
