Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 )
Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters ...................................................................... Patch Set 3: (3 comments) Working on a new upload http://gerrit.cloudera.org:8080/#/c/14026/3/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14026/3/be/src/scheduling/scheduler.cc@796 PS3, Line 796: P((1/3)^(n-1)) > Is there a typo here ? Not sure what (P(1/3^(n-1)) means ? Definitely a typo http://gerrit.cloudera.org:8080/#/c/14026/3/be/src/scheduling/scheduler.cc@811 PS3, Line 811: if (candidates_it == remote_executor_candidates->end()) { > This is now O(n^2), right? Is there a bound on num_executors and if so, sho Yes, this is O(n^2). We limit the num_remote_executor_candidates to be at most 16 via the query option setting code. We also limit it to be the number of nodes if that is smaller. The default is 3 and some systems are going to use 2. I doubt we are going to set it higher than 3, so we could cut the maximum allowed value to 8 without any real problem. I haven't benchmarked this. http://gerrit.cloudera.org:8080/#/c/14026/3/be/src/scheduling/scheduler.cc@811 PS3, Line 811: if (candidates_it == remote_executor_candidates->end()) { > Or we can consider using an unordered_set to track the candidates found so My thinking was that when n is small, maintaining one structure (even though it is O(n^2)) might still be better than maintaining two. It is easy to go back to using the set. I would just put the IpAddrs directly in the vector rather than iterating over the set at the end. -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Comment-Date: Mon, 12 Aug 2019 21:27:42 +0000 Gerrit-HasComments: Yes
