rangareddy commented on issue #5211:
URL: https://github.com/apache/hudi/issues/5211#issuecomment-5101227054
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: The reported `NullPointerException` is resolved as of Hudi 1.2.0.
Glob-path reads have been removed from the Spark datasource and replaced with
an explicit error, and the underlying use case -- reading a subset of
partitions -- is served by loading the base path and filtering on the partition
column.
Root cause was exactly as @jonashartwig described on 2022-11-14: `globPath`
walked *up* from the supplied path looking for the table base path, but a
component like `cluster=*` never resolves to a real directory, so `getParent`
was called repeatedly until it reached `/` and returned `null`.
Verified by running a two-level hive-style partitioned COW table (`cluster`
x `dt`) on Spark 3.4.4 across three Hudi versions:
| Hudi | `load(base)` | `base/cluster=*` | `base/cluster=*/dt=*` |
`base/cluster=abc` | `load(base).filter(cluster='abc')` |
| --- | --- | --- | --- | --- | --- |
| 0.14.0 | OK | OK | **`NullPointerException`** | OK | OK |
| 1.1.1 | OK | OK | `IllegalStateException: Cannot get parent path of a root
path` | OK | OK |
| 1.2.0 | OK | `HoodieException: Glob paths are not supported...` | same
`HoodieException` | OK | OK |
The fix is PR #14060 (`refactor(spark): Remove glob paths and deprecate read
paths support`, commit `2ebc6775dbb6`, merged 2025-11-25, released in 1.2.0),
which adds an explicit guard in
`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:107-109`:
```scala
if (path.exists(_.contains("*")) || readPaths.nonEmpty) {
throw new HoodieException("Glob paths are not supported for read paths as
of Hudi 1.2.0")
}
```
To be explicit about what "fixed" means here: globs were removed rather than
made to work, so on 1.2.0+ you get a clear error instead of a crash. Note also
the 1.1.1 row above -- through 1.1.x the same walk-to-root defect is still
present, just with a better message than the bare NPE.
For @INRIX-Trang-Nguyen's and @haggy's question about reading a subset of
partitions without scanning the table, the supported form is:
```python
spark.read.format("hudi").load(base_path).filter("cluster = 'abc'")
```
which returned the correct pruned result on every version tested above. If
you previously found that this listed the whole table, that is worth a fresh
issue against a current version -- partition pruning goes through
`HoodieFileIndex` now, and enabling the metadata table
(`hoodie.metadata.enable=true`) removes the full file listing.
Closing as fixed. If you hit a partition-pruning or listing problem on
1.2.0, please reopen or file a new issue with the version, table layout and the
query."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
