hulincup opened a new pull request, #28913:
URL: https://github.com/apache/flink/pull/28913
## What
`PartitionPathUtils.listStatusRecursively` descended into every subdirectory
returned by `fs.listStatus`, including hidden ones such as `_temporary`. A
non-hidden child of a hidden dir (e.g. `_temporary/job-123`) sitting at
partition depth was then collected with an empty partition spec, which later
surfaces as a `TableException: incomplete partition spec` in the filesystem
connector source.
Reported in FLINK-38774 (supersedes the older FLINK-31975).
## Fix
Reuse the existing `listStatusWithoutHidden` helper to filter hidden
children at **every** recursion level — not only at the leaf, where
`searchPartSpecAndPaths` already filtered via `isHiddenFile`. Add a null guard
consistent with `listStatusWithoutHidden`'s contract (returns `null` when
`fs.listStatus` returns `null`), which also removes a latent NPE in the old
for-each over a null list.
```java
if (fileStatus.isDir()) {
FileStatus[] children = listStatusWithoutHidden(fs,
fileStatus.getPath());
if (children == null) {
return;
}
for (FileStatus stat : children) {
listStatusRecursively(fs, stat, level + 1, expectLevel, results);
}
}
```
## Test
Adds a `@TempDir` regression test that creates a real partition
(`date=2019-8-30/country=China`) alongside a hidden `_temporary/job-123`
subtree and asserts `searchPartSpecAndPaths` returns only the real partition.
Without the fix the hidden `job-123` leaf leaks in with an empty spec (2
entries); with the fix only the real partition is returned (1 entry).
## Notes
- Supersedes the abandoned #27314 (stale-closed after 120 days of
inactivity, no technical objection — reviewer only asked for a unit test, which
is included here).
- This is a `flink-table/flink-table-common` utility, not literally under
`flink-connectors/`, but the bug manifests via the filesystem connector (tagged
`Connectors / FileSystem`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
