Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24636 )
Change subject: IMPALA-8523: Migrate hdfsOpen to builder-based openFile API ...................................................................... Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/24636/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24636/8//COMMIT_MSG@36 PS8, Line 36: current runtimes (which recognize only random/sequential/ : adaptive Hadoop 3.4.2 already recognizes the parquet/orc/avro/json policy names http://gerrit.cloudera.org:8080/#/c/24636/8//COMMIT_MSG@58 PS8, Line 58: correctness improvement performance hint? http://gerrit.cloudera.org:8080/#/c/24636/8/be/src/util/open-file-options-util.cc File be/src/util/open-file-options-util.cc: http://gerrit.cloudera.org:8080/#/c/24636/8/be/src/util/open-file-options-util.cc@44 PS8, Line 44: // Row-oriented formats read front-to-back within a split, i.e. sequential access. : // Text and SequenceFile have no dedicated token, so they use plain 'sequential'. : static const char* AVRO_READ_POLICY = "avro, sequential"; : static const char* JSON_READ_POLICY = "json, sequential"; : static const char* SEQUENTIAL_READ_POLICY = "sequential"; Opus 5.5 said the following, please validate: 'sequential' slows text/SequenceFile/Avro/JSON reads on S3. * S3 reads use the file handle cache by default, and ~Accessor calls hdfsUnbufferFile() after every read of up to 8MB. * S3A's unbuffer() aborts the current request if more than 64K of it is unread. It also switches an adaptive stream to random, but it never switches a sequential one. * So with 'sequential', every 8MB read requests the rest of the file and then gets aborted, which costs a new connection each time. | Policy | Result | |-------------------|-----------------------------| | parent (adaptive) | 1 open-ended GET, 1 abort | | 'sequential' | 8 open-ended GETs, 7 aborts | | 'random' | 8 bounded GETs, 0 aborts | * Suggested fix: leave the row formats unset (or use 'random'), and add a text or Avro scan to the S3 validation. -- To view, visit http://gerrit.cloudera.org:8080/24636 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46d810b19fe7d4859e3c2bcd7568b61fe73408c1 Gerrit-Change-Number: 24636 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Thu, 01 Oct 2026 16:10:26 +0000 Gerrit-HasComments: Yes
