Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 )
Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls ...................................................................... Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/8523/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8523/11//COMMIT_MSG@34 PS11, Line 34: - all core tests pass when configured with s3 > Do the tests validate that an "even" number of bytes is assigned to each ba That block size is used in the specs here, HdfsScanNode.java L755. The division into scan ranges should be the same as before. As before, each of these scan ranges is "random", e.g., no process affinity, so as long as assignment of remote reads balances bytes per backend, then the result should be balanced. There's a scheduler unit test that tests remote reads and checks that the assignment is balanced. I did some manual queries and saw roughly even assignment. That said, I don't have a test that looks at bytes-per-host and checks that they're even. Will look at what other tests we have for this and what can be added. -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 11 Gerrit-Owner: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Comment-Date: Thu, 29 Mar 2018 01:49:33 +0000 Gerrit-HasComments: Yes
