Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/18093 )
Change subject: IMPALA-11033: Add support for specifying multiple executor group sets ...................................................................... Patch Set 5: (10 comments) http://gerrit.cloudera.org:8080/#/c/18093/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18093/5//COMMIT_MSG@11 PS5, Line 11: size(the number of executors in each group). Every executor group > Nit: need space after size Done http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr-test.cc File be/src/scheduling/cluster-membership-mgr-test.cc: http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr-test.cc@573 PS5, Line 573: first > nit. Can we add 'group_set_name' and 'expected_size' to ExpectedExecGroupSe just to clarify, the ExpectedExecGroupSets is actually a typedef of vector<pair<string, int>> which is where the 'first' and 'second' accessors come from (std::pair<>). (https://gerrit.cloudera.org/#/c/18093/5/be/src/scheduling/cluster-membership-mgr.h) To be able to name the member variables I would have to create a new struct definition. Let me know if you would strongly prefer to do the alternative. http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr-test.cc@588 PS5, Line 588: // increasing order > nit. of expected group size. Done http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr-test.cc@615 PS5, Line 615: 4 > nit. duplicated case number (@587). Done http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr-test.cc@631 PS5, Line 631: Invalid input with no colon separator > nit title is not right. Done http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr-test.cc@659 PS5, Line 659: EXPECT_EQ(update_req.exec_group_sets.size(), 1); : EXPECT_EQ(update_req.exec_group_sets[0].curr_num_executors, 1); : EXPECT_EQ(update_req.exec_group_sets[0].expected_num_executors, 20); : EXPECT_EQ(update_req.exec_group_sets[0].exec_group_name_prefix, ""); > Can we add some assertions on methods numExecutors() and contains() for all just to clarify, do you want me to add separate JUnit tests that mimic these cases? http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr.cc File be/src/scheduling/cluster-membership-mgr.cc: http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr.cc@622 PS5, Line 622: DCHECK(!prefix.empty() || exec_group_sets.size() == 1) > Should this be exec_group_sets.size() == 1 is there to ensure that if prefix is empty then it should only contain a single exec_group_set that is added in L613 http://gerrit.cloudera.org:8080/#/c/18093/5/be/src/scheduling/cluster-membership-mgr.cc@629 PS5, Line 629: if (!it.second.IsHealthy()) continue; > Is this interesting enough to warrant a log message? if a group is not healthy it, we have visibility over it as it will show up on the metrics so I felt logging a warning here might be an overkill. this is why the way matching_exec_groups_found is incremented, it always accounts for a group regardless of whether it is healthy or not, so an unhealthy group will not be logged below http://gerrit.cloudera.org:8080/#/c/18093/5/fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java File fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java: http://gerrit.cloudera.org:8080/#/c/18093/5/fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java@97 PS5, Line 97: String host = address.getHostname(); > Return false when exec_group_sets.size() > 0? exec_group_sets will always be non empty for any kind of update. If using 'default' exec group, the addresses will be populated and we should return those, but if using non-default exec groups this 'address' will be empty and false will be returned http://gerrit.cloudera.org:8080/#/c/18093/5/fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java File fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java: http://gerrit.cloudera.org:8080/#/c/18093/5/fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java@61 PS5, Line 61: int num > nit. Probably it is better to use two arguments (int expected_num_executors Done -- To view, visit http://gerrit.cloudera.org:8080/18093 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e0a3a5fe2b1f0b7507b7c096b7a3c373bc2e684 Gerrit-Change-Number: 18093 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Comment-Date: Wed, 22 Dec 2021 05:27:34 +0000 Gerrit-HasComments: Yes
