Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13545 )
Change subject: [WIP] IMPALA-8630: Hash the full path when calculating consistent remote placement ...................................................................... Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13545/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13545/2//COMMIT_MSG@27 PS2, Line 27: The alternative is to construct the DescriptorTbl in the scheduler the advantage of the alternative is saving some extra memory per file descriptor. The FileDescs are persistent per-file so this can add up to many MB of increased consumption on a big catalogd. That said, with the recent work to reduce consumption and add eviction on catalogd, maybe it's not such a big deal as it used to be (eg IMPALA-7406 saved 100 bytes per FileFbDesc and here we're only losing a little bit of that gain) http://gerrit.cloudera.org:8080/#/c/13545/2/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/13545/2/be/src/scheduling/scheduler.cc@801 PS2, Line 801: uint32_t hash = HashUtil::Hash(&hdfs_file_split->full_path_hash, : sizeof(hdfs_file_split->full_path_hash), 0); why hash the hash instead of just making the hash here be a uint32_t directly? http://gerrit.cloudera.org:8080/#/c/13545/2/common/fbs/CatalogObjects.fbs File common/fbs/CatalogObjects.fbs: http://gerrit.cloudera.org:8080/#/c/13545/2/common/fbs/CatalogObjects.fbs@82 PS2, Line 82: int should be a uint? -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Comment-Date: Fri, 14 Jun 2019 23:20:50 +0000 Gerrit-HasComments: Yes
