Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14756
Change subject: IMPALA-9151: Maintain cluster size in ExecutorMembershipSnapshot ...................................................................... IMPALA-9151: Maintain cluster size in ExecutorMembershipSnapshot This change improves the cluster membership snapshot we maintain in the frontend in cases where all executors have been shut down or none have started yet. Prior to this change when configuring Impala with executor groups, the planner might see a ExecutorMembershipSnapshot that has no executors in it. This could happen if the first executor group had not started up yet, or if all executor groups had been shutdown. If this happened, the planner would make sub-optimal decisions, e.g. decide on a broadcast join vs a PHJ. With this change if no executors have been registered so far, the planner will use a configurable default which should be set to the expected number of executors. After executors come online, the planner will use the size of the largest healthy executor group, and it will hold on to the group's size even if it shuts down or becomes unhealthy. This allows the planner to work on the assumption that a healthy executor group of the same size will eventually come online to execute the query. Change-Id: Ib6b05326c82fb3ca625c015cfcdc38f891f5d4f9 --- M be/src/runtime/exec-env.cc M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java A fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py 8 files changed, 251 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/14756/1 -- To view, visit http://gerrit.cloudera.org:8080/14756 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib6b05326c82fb3ca625c015cfcdc38f891f5d4f9 Gerrit-Change-Number: 14756 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker <l...@cloudera.com>
