sollhui opened a new pull request, #69385:
URL: https://github.com/apache/doris/pull/69385
### What problem does this PR solve?
Related PR: #64009
S3 TVFs and catalog reads still list objects for deterministic paths after
the filesystem SPI refactor. Credentials that can HEAD/GetObject but cannot
ListBucket therefore fail even when all requested object keys are known.
This reapplies #64009 on current master
(`cca2231716264a15cbc884fbeac96eef4eafd66b`) and adapts it to the
StorageAdapter API and the split S3 modules. Exact paths and bounded
brace/bracket/numeric patterns use HEAD; missing objects are skipped, while
wildcards, directory paths, disabled controls and over-limit expansions keep
the LIST path.
- Reuse master's bounded glob expansion and numeric-range validation,
preserving UTF-8 key ordering and incremental cursors.
- Snapshot mutable controls into both FE filesystem caches and construct the
filesystem from the same snapshot, preserving the originally bound provider and
cross-scheme translation.
- Pass controls through S3/COS/OSS/OBS and the GCS/MinIO/Ozone delegation
path.
- Retain the original regression suite and its expected output; add coverage
for UTF-8 pagination, a HEAD limit above the LIST-prefix cap, provider routing,
and delegated properties.
Network behavior: HEAD targets the existing configured object-store endpoint
under the same TVF/catalog access path and storage credentials as LIST. This
changes the request method used by existing external-storage access and adds no
new endpoint input.
### Release note
Deterministic S3-compatible object paths can be read using bounded HEAD
requests without requiring ListBucket permission. Mutable
`s3_skip_list_for_deterministic_path` and `s3_head_request_max_paths` controls
are honored by cached filesystems.
### Check List (For Author)
- Test:
- [x] Unit Test: 511 tests passed across filesystem API, SPI, S3 base,
GCS, MinIO, Ozone, OSS, OBS and COS (0 failures/errors/skips). Includes 104
S3FileSystem tests.
- [ ] Regression test: carried forward from #64009; not rerun locally
because the external S3 fixture/cluster is unavailable.
- Checkstyle passed for all 11 touched modules; `git diff --check`
passed.
- Local filesystem tests used `run-fe-ut.sh` with a temporary Maven
launcher selecting the filesystem modules and the installed JDK 20. The
configured JDK 17 is incomplete. FE cache/factory tests were added/ported but
could not run: the standard reactor stops at the Thrift compiler version gate
(installed 0.16.0, required 0.24.0). CI with the required toolchain remains
necessary.
- Behavior changed:
- [x] Yes. Bounded deterministic paths use HEAD when enabled; other
patterns continue to LIST.
- Does this need documentation?
- [x] No. Existing configuration controls are restored in the filesystem
SPI implementation.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
