deepakpanda93 opened a new pull request, #19569:
URL: https://github.com/apache/hudi/pull/19569
### Describe the issue this Pull Request addresses
Closes #15174.
#14060 removed glob path support and deprecated
`hoodie.datasource.read.paths` in 1.2.0, replacing
the globbing logic with an unconditional throw. The three rejections in
`DefaultSource`'s 3-arg
`createRelation` ended up sharing one condition and one message, so the
message rarely describes
what the caller actually did.
**A non-glob value is reported as an unsupported glob.** The guard is:
```scala
if (path.exists(_.contains("*")) || readPaths.nonEmpty) {
throw new HoodieException("Glob paths are not supported for read paths as
of Hudi 1.2.0")
}
```
`readPaths.nonEmpty` has nothing to do with wildcards, so a caller passing
plain partition paths:
```scala
spark.read.format("hudi")
.option("hoodie.datasource.read.paths", "prefix/part1,prefix/part2")
.load(basePath)
```
is told their non-glob paths are unsupported glob paths. There is no
wildcard anywhere in that call
and nothing in the message to act on.
**The missing-path message advertises an option that throws.** The guard
above it says:
```
'path' or 'hoodie.datasource.read.paths' or both must be specified.
```
A caller who follows that and sets `hoodie.datasource.read.paths` reaches
the throw quoted above.
**The config documentation still describes the option as usable.**
`READ_PATHS` carries
`@Deprecated` but no `deprecatedAfter(...)`, and reads "Comma separated list
of file paths to read
within a Hudi table", with nothing to say that setting it now fails the read.
### Summary and Changelog
The rejections are unchanged. What changes is that each one now says what
happened and what to do
instead.
- Split the conflated guard into three independent checks: `read.paths` is
set, no `path` was given,
`path` contains a glob. Each throws its own message.
- Every message names the supported replacement, so no rejection says only
what does not work: load
the table base path and filter on the partition columns. Those predicates
are pushed down and
prune partitions before any file is listed, via
`HoodieFileIndex#prunePartitionsAndGetFileSlices`,
which is what selecting paths by hand was for.
- The missing-path message no longer offers `read.paths` as the alternative,
since setting it throws.
- All three checks run before `HoodieStorageUtils.getStorage(...)`, so a
call that cannot succeed no
longer opens a storage handle first. The now-dead `readPaths` and
`allPaths` locals are gone, and
`allPaths.head` becomes `path.get`, which is safe because `path.isEmpty`
is rejected above it.
- `READ_PATHS` gains `deprecatedAfter("1.2.0")`, and its documentation
states that the option now
fails the read and what to use instead.
Two cases that needed a deliberate choice rather than falling out of the
code:
**When both `read.paths` is set and `path` contains a glob, `read.paths` is
reported.** It is the
more fundamental of the two: the option is gone outright, and the
replacement it points at resolves
the glob case as well. Pinned by `testReadPathsWinsOverGlobInPath`.
**An explicitly empty `read.paths=""` is still rejected.** `optParams.get`
yields `Some("")`, so the
key counts as set. 1.2.0 already rejected it, since `Some("")` made the old
`readPaths.nonEmpty`
check true, and treating it as unset here would turn a throw into a
successful read. Pinned by
`testEmptyReadPathsIsStillRejected`.
### Impact
None on behaviour. `read.paths` and glob paths are rejected before this
change and after it, an
empty `read.paths` is rejected in both, and every read that worked before
still works. Only the
wording of the three exceptions and the `READ_PATHS` documentation change.
Users who hit these errors now get a message that matches their input and
names the replacement.
### Risk Level
low
Confined to three error paths in one method plus a config doc string. No
change to relation
construction, planning or reads. Covered by a new suite of 13 tests, and
reverting the change fails
exactly the 7 of them that assert the messages while the 6 no-regression
tests keep passing.
### Documentation Update
The `hoodie.datasource.read.paths` config description is updated in this PR
to say the option now
fails the read and to point at loading the base path with partition
predicates instead. It also
gains `deprecatedAfter("1.2.0")`, so it is rendered as deprecated since that
release. No Hudi website
change is needed.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
