sollhui opened a new pull request, #69383:
URL: https://github.com/apache/doris/pull/69383
### What problem does this PR solve?
Related PR: #64941, #64684
Azure glob listing scans only the longest static prefix. For example,
`date=2025-{0[3-9],1[0-2]}-01/*` scans everything under `date=2025-`, including
unrelated dates.
Extract the object-storage glob helpers into `ObjectStorageGlob` and use
them for Azure and S3-compatible listing. Bounded alternatives become sorted,
non-overlapping remote list prefixes; unsafe or excessive expansion falls back
to a conservative prefix and still applies the complete matcher. Numeric ranges
retain zero padding and can match large ranges without enumerating every value.
Azure limits and resume cursors continue across expanded prefixes, and
object-storage cursors consistently use unsigned UTF-8 order. S3 directory
buckets retain slash-terminated prefixes and reject unsupported key-based
pagination.
This reapplies the final changes from #64941 to current master
(`cca2231716264a15cbc884fbeac96eef4eafd66b`). It adapts the S3 changes to
`fe-filesystem-s3-base`, retains the existing SPI hooks and notification
matcher, merges the offset test into master's existing tests, and preserves
master's escaped/nested glob and suffix-sensitive prefix behavior. Hadoop's
`GlobPattern` expansion remains unchanged.
### Release note
Reduce Azure glob listing scans for bounded patterns. Correct object-storage
numeric-range and UTF-8 cursor handling. Reject limited/resumed S3
directory-bucket glob listing because it cannot support key-based pagination.
### Check List (For Author)
- Test:
- [x] Unit Test: ported and extended coverage for prefix pushdown,
fallback, numeric ranges, padding, escaping, nesting, limits, UTF-8 cursors,
and directory buckets.
- Local validation: Checkstyle passed for `fe-filesystem-spi`,
`fe-filesystem-azure`, `fe-filesystem-s3-base`, and `fe-core`; `git diff
--check` passed.
- Unit execution is pending CI. Attempted `bash run-fe-ut.sh --run
ObjectStorageGlobTest,AzureFileSystemTest,S3FileSystemTest,S3CompatibleFileSystemTest,S3SourceOffsetProviderTest`,
but the standard script stopped at the JDK check: this host has no configured
JDK 17. No unit-test success is claimed.
- Behavior changed:
- [x] Yes. Azure pushes down expanded prefixes; object-storage ranges
preserve padding and large-range semantics; cursor comparisons use UTF-8 order;
directory-bucket key pagination fails explicitly.
- Does this need documentation?
- [x] No. No new configuration or syntax.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
