rangareddy opened a new pull request, #19974:
URL: https://github.com/apache/hudi/pull/19974
### Describe the issue this Pull Request addresses
Closes #19862.
Every `show_*` procedure that accepts both `limit` and `filter` truncated
its rows to `limit` first and evaluated `filter` on the truncated list, so
`limit => 10, filter => ...` returned the matching subset of the first 10 rows
rather than the first 10 matching rows. With a selective filter the result is
empty whenever the matches sit past the cut-off, and since every one of these
procedures applies a default `limit` (10, 20 or 100), a plain `filter => ...`
call with no explicit limit was affected too.
Introduced alongside the generic filter option in #13736 / #13790.
### Summary and Changelog
`BaseProcedure` gains `applyFilterAndLimit(results, filter, schema, limit)`,
which filters and then truncates, plus `hasFilter` and `resolveLimit` helpers
(the latter turning an optional `limit` argument into a bound, where "unset"
means no bound rather than zero). The ten affected procedures now call it. They
are not all the same shape, and the difference is the substance of the change:
**Truncating an already-materialised list.** `ShowTablePropertiesProcedure`,
`ShowFileSystemViewProcedure`, `ShowHoodieLogFileMetadataProcedure` and
`ShowMetadataTableFilesProcedure` just dropped the early `stream().limit(...)`;
the rows were already in memory, so nothing about the work done changes.
**Bounding at the source.** `ShowBootstrapMappingProcedure` and
`ShowFsPathDetailProcedure` used `df.orderBy(...).limit(n).collect()`, and
`ShowInvalidParquetProcedure` used `rdd.take(n)`. The bound now moves after the
filter. To keep the unfiltered path exactly as it was, the early bound is only
skipped when a filter is actually present.
**Bounding real work.** In `ShowCleansProcedure`, `ShowTimelineProcedure`
and `ShowHoodieLogFileRecordsProcedure`, `limit` is not display truncation: it
caps how many clean instants get their metadata read and how many records are
pulled out of log files. Filtering first without care would make those
unbounded, turning a correctness fix into a performance regression.
`ShowCleansProcedure` and `ShowTimelineProcedure` use `scanLimit = if
(hasFilter(filter)) Int.MaxValue else limit`, so a call with no filter does
exactly the work it did before and only the filtered path reads further.
`limit` is never used to size an allocation on those paths, so `Int.MaxValue`
is only ever a `take` bound or a loop comparison.
`ShowHoodieLogFileRecordsProcedure` needed more than that, because lifting
its bound would buffer every record of every matched log file into an on-heap
list and could OOM the driver on a large file group. It instead filters
**incrementally**: rows are collected into a small batch, the filter is
evaluated per batch, only the matches are retained, and the scan stops as soon
as `limit` matches have been found. Heap stays bounded by the batch plus the
matches no matter how much the log files hold. With no filter `applyFilter` is
a passthrough, so a batch of one reproduces the previous bound exactly.
`ShowTimelineProcedure` needs one extra condition. `getTimelineEntries`
deliberately ignores `limit` when both `startTime` and `endTime` are given
("Apply limit only if time range is not fully specified"), so reapplying the
bound unconditionally would have truncated a fully specified range to the
default of 20. The reapplied bound honours that case.
The six other procedures that call `applyFilter`
(`ShowBootstrapPartitionsProcedure`, `ShowColumnStatsOverlapProcedure`,
`ShowFileStatusProcedure`, `ShowMetadataTableColumnStatsProcedure`,
`ShowMetadataTablePartitionsProcedure`, `ShowMetadataTableStatsProcedure`) take
no `limit` at all and are untouched, so the list of ten in the issue is
complete.
### Verification
Three regression tests, each confirmed to fail without the corresponding fix.
`TestShowCleansProcedures`: three cleans are created, then the **oldest** is
requested with `limit => 1`. Cleans come back newest first, so the match sits
past a limit of 1.
```
Test show_cleans applies the filter before the limit *** FAILED ***
Array() had length 0 instead of expected length 1
limit must bound the rows the filter matched, not the rows the filter was
shown;
got 0 rows for clean_time = 20260916090635697
```
`TestHoodieLogFileProcedure`: with two records in the log files, each is
requested with `limit => 1` and a filter naming it. Asserting both keeps this
independent of scan order, since whichever record is not first is invisible to
a filter applied after a limit of 1. Bounding the scan by rows seen rather than
rows matched fails it:
```
Test Call show_logfile_records Procedure with merge and filter *** FAILED ***
Expected 1, but got 0
limit => 1 must return the row matching b2, wherever it sits in the scan
```
`TestShowTimelineTableProcedure` Test Case 13, which runs across all four
existing variants (V1/V2 x COW/MOR): a fully specified time range with `limit
=> 1` must not be truncated. Reintroducing the unconditional bound fails it
everywhere:
```
Test show_timeline with various parameters - V2 MOR *** FAILED ***
had length 1 instead of expected length 39
Test 13: a fully specified range must not be truncated by limit, expected
39 got 1
Test show_timeline with various parameters - V2 COW *** FAILED ***
had length 1 instead of expected length 34
```
Full procedure package on Spark 3.5 / Scala 2.12:
```
mvn test -Punit-tests -Dspark3.5 -Dscala-2.12 -pl
hudi-spark-datasource/hudi-spark \
-DwildcardSuites=org.apache.spark.sql.hudi.procedure
-> Suites: completed 50, aborted 0
-> Tests: succeeded 263, failed 0, canceled 0, ignored 2, pending 0
mvn scalastyle:check checkstyle:check -pl hudi-spark-datasource/hudi-spark
-> Found 0 errors
```
### Impact
`limit => n, filter => ...` now returns the first n matching rows instead of
the matches among the first n rows. Calls that pass no filter are unaffected,
including the work they do: the early bound is only lifted when a filter is
present.
One behaviour change worth calling out: with a filter, `show_cleans` reads
the metadata of every clean rather than only the newest `limit`, and
`show_timeline` reads the whole timeline rather than the first `limit`
instants. That is required for the filter to see the rows it is supposed to
match, it is bounded by the size of the timeline, and the unfiltered path,
which is the common one, is unchanged.
`show_logfile_records` is deliberately *not* in that list. An earlier
revision of this PR did lift its bound the same way, and this description said
so; that was wrong, because the records it reads are unbounded by anything the
table caps, so a plain `filter => ...` on a large file group could have
exhausted the driver heap where it previously returned quickly. It now filters
incrementally and stops at `limit` matches, so it stays bounded.
### Risk Level
low: the change is confined to how `limit` and `filter` compose inside these
procedures. No API, config, or table format change, and no engine or table-type
specific behaviour is involved.
### Documentation Update
none: this restores the documented meaning of `limit` combined with `filter`
rather than changing it.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
- [x] CI passes on my PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]