Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24636
to look at the new patch set (#6).
Change subject: IMPALA-8523: Migrate hdfsOpen to builder-based openFile API
......................................................................
IMPALA-8523: Migrate hdfsOpen to builder-based openFile API
Replace the deprecated hdfsOpenFile() on the scan read path with the
libhdfs builder-based openFile API (HADOOP-15229, exposed via libhdfs in
HDFS-14478). The builder lets Impala declare per-file configuration when
opening a file, which is the basis for the S3A read optimizations below.
OpenHdfsFileOp::Execute() now uses hdfsOpenFileBuilderAlloc/Opt/Build
and blocks on the returned future via hdfsOpenFileFutureGet() inside the
SynchronousThreadPool worker, so the existing open timeout (IMPALA-7738)
is preserved. Options are threaded from the scan node down to the open
call: HdfsFileDesc::GetFileInfo() computes them once per file so that
all scan ranges for the file (including footer/column ranges) inherit
them via ScanRange::GetFileInfo().
Unlike hdfsOpenFile(), the builder future (hdfsOpenFileFutureGet()) does
not translate the underlying Java exception to errno, so a missing file
would report "Unknown error" instead of ENOENT.
OpenHdfsFileOp::Execute() detects a file-not-found root cause and
restores errno to ENOENT before building the message, preserving the
"No such file or directory" surface that scanners and tests rely on.
The options are set with the standard, filesystem-agnostic openFile keys
and are all soft (hdfsOpenFileBuilderOpt), so filesystems that do not
understand a key ignore it and non-S3A reads are unaffected:
- fs.option.openfile.read.policy, chosen per file format. Each value is
an ordered "first recognized policy wins" list: the precise format
token first, for newer Hadoop runtimes, then a generic fallback that
is what current runtimes (which recognize only random/sequential/
adaptive) actually select. The axis is the access pattern, not
splittability:
Parquet, HUDI_PARQUET -> "parquet, random"
ORC -> "orc, random"
RC_FILE -> "columnar, random"
Avro -> "avro, sequential"
JSON -> "json, sequential"
text, SequenceFile -> "sequential"
Columnar formats read a footer then seek to selected column chunks
(random); row-oriented formats read front-to-back (sequential), so
forcing "random" on Avro/SequenceFile would hurt them. Kudu and JDBC
are not read through this path and get no policy. The format tokens
(parquet/orc/columnar) may enable S3A's analytics/prefetching stream
on a future Hadoop upgrade; that should be benchmarked against plain
"random" at that point rather than assumed to be a win.
- fs.option.openfile.length when the file length is known (> 0), so
S3A can skip the HEAD request on open. Guarded against a stale zero
length that would truncate reads.
No new flags are added: the read policy is a correctness improvement,
and soft options make the change safe across filesystems.
Manual S3A validation on both local MinIO (server-side request trace)
and real AWS S3 (client-side S3A audit logging), comparing this commit
against its parent on an identical cold 24-file Parquet scan:
SELECT count(*), sum(int_col), max(string_col) FROM s3_alltypes
HEAD requests GET requests
parent (baseline) 24 72
this commit 0 72
The known-length hint eliminates the per-file HEAD on open (24 -> 0)
with no change to the read GET pattern (72 = 24 files x 3 ranges),
confirming the S3A runtime honors fs.option.openfile.length. The read
policy was verified at the HTTP layer: on a 69 MiB / 16-column file, a
single-column scan under "parquet, random" issues a tight ranged GET
for just that column (range[4-3497116], ~3.3 MiB) versus an open-ended
GET to EOF (range[4-72589876], ~69 MiB) under "sequential".
Testing:
- new be test: OpenFileOptionsUtilTest
- e2e tests passed
Change-Id: I46d810b19fe7d4859e3c2bcd7568b61fe73408c1
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M be/src/exec/CMakeLists.txt
M be/src/exec/hdfs-scan-node-base.h
A be/src/exec/open-file-options-util-test.cc
A be/src/exec/open-file-options-util.cc
A be/src/exec/open-file-options-util.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/runtime/io/handle-cache.h
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/hdfs-file-reader.cc
M be/src/runtime/io/hdfs-monitored-ops.cc
M be/src/runtime/io/hdfs-monitored-ops.h
A be/src/runtime/io/open-file-options.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_timeout.py
17 files changed, 451 insertions(+), 59 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/24636/6
--
To view, visit http://gerrit.cloudera.org:8080/24636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I46d810b19fe7d4859e3c2bcd7568b61fe73408c1
Gerrit-Change-Number: 24636
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
