18770832...@163.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/19532 )
Change subject: IMPALA-11904: Data cache support dumping for reloading ...................................................................... Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/19532/5/be/src/scheduling/executor-group.cc File be/src/scheduling/executor-group.cc: http://gerrit.cloudera.org:8080/#/c/19532/5/be/src/scheduling/executor-group.cc@108 PS5, Line 108: auto cmp = [](const BackendDescriptorPB& a, const BackendDescriptorPB& b) { : return a.address().port() < b.address().port(); : }; : std::sort(be_descs.begin(), be_descs.end(), cmp); > This seems fine for providing consistent scheduling across multiple backend Thank you for the reminder! - Regarding the changes in the number of executors in the group, based on the characteristics of consistent hashing, the addition or removal of nodes will only affect neighboring nodes, meaning that the addition or removal of individual nodes will not have a significant impact on the scheduling results. I have already conducted relevant tests before, and the test results have also confirmed this. Shutting down a node will only cause the scan range that should have been assigned to it to be assigned to another node. After restarting the node, we can obtain the same scheduling results as before shutdown. Therefore, this does not require additional processing, and normal dumping and reloading is ok. - If the IP of the executor changes, it will indeed have a relatively large impact, as consistent hashing is based on node IP addresses. However, if only a small number of node IPs change, the situation is similar to changes in the number of executors, with limited impact. But if the IPs of a large number of nodes change, it will lead to completely different scheduling results, and the cache hit rate in the period after restart will indeed be greatly reduced. If IMPALA-11979 is resolved, it will be very helpful in such cases. -- To view, visit http://gerrit.cloudera.org:8080/19532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id867f4fc7343898e4906332c3caa40eb57a03101 Gerrit-Change-Number: 19532 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Comment-Date: Tue, 21 Mar 2023 03:27:58 +0000 Gerrit-HasComments: Yes
