Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 )
Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls ...................................................................... Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/8523/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8523/12//COMMIT_MSG@13 PS12, Line 13: includes S3, ADLS, and local fs. > Does this change the dependency on fs.s3a.block.size? This change does not attempt to change the scan ranges that are produced for these file systems. Currently, we rely on the filestatus blocksize (see L177 of HdfsPartition) when synthesizing blocks to store in the catalog. This patch shifts that synthesis to the scheduler, so that its generated for each use instead of stored in memory. The block parameter used for this synthesis is set in this change on L784 of HdfsScanNode. Instead of filestatus blocksize, it uses the filesystem's default block size, which I think is the same thing for these file-systems (at least, from what I could tell from hadoop.fs.FileSystem). -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 12 Gerrit-Owner: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Comment-Date: Wed, 16 May 2018 06:04:52 +0000 Gerrit-HasComments: Yes
