voonhous opened a new issue, #19862: URL: https://github.com/apache/hudi/issues/19862
**Describe the problem** Every `show_*` procedure that takes both `limit` and `filter` truncates its rows to `limit` first and evaluates `filter` on the truncated list (`BaseProcedure.applyFilter`), so `limit => 10, filter => ...` returns the matching subset of the first 10 rows rather than the first 10 matching rows. With a selective filter the result is empty even when matching rows exist past the cutoff, and every procedure applies a default `limit` (10, 20 or 100), so a plain `filter => ...` call is affected too. Introduced with the generic filter option in #13736 / #13790 (HUDI-9726 / HUDI-9746). Affected on master `9903b6d83fbb` (limit -> applyFilter): - `ShowFileSystemViewProcedure.scala:263 -> :264` (`rows.stream().limit(limit)`, default 10) - `ShowTablePropertiesProcedure.scala:60 -> :61` (default 10) - `ShowHoodieLogFileMetadataProcedure.scala:136 -> :137` (default 10) - `ShowMetadataTableFilesProcedure.scala:86 -> :90` (default 100) - `ShowInvalidParquetProcedure.scala:110 -> :114` (`parquetRdd.take(limit)`, default 100) - `ShowBootstrapMappingProcedure.scala:100/:102 -> :104` (`df.orderBy(...).limit(limit).collect()`, default 10) - `ShowFsPathDetailProcedure.scala:82/:88 -> :93` (`df.orderBy(...).limit(limit).collect()`, default 100) - `ShowCleansProcedure.scala:175 -> :179` (`.take(limit)` after the limited timeline read, default 10) - `ShowTimelineProcedure.scala:140 -> :142` (limit applied inside `getTimelineEntries`, default 20) - `ShowHoodieLogFileRecordsProcedure.scala:104 -> :120` (limit bounds record collection, default 10) **To reproduce** On a table with more than 10 file slices, where only a slice past the 10th exceeds `<size>`: ```sql call show_fsview_all(table => 't', filter => 'data_file_size > <size>'); ``` returns 0 rows, and `limit => 1000` makes the row appear. `TestFsViewProcedure` does not catch this because its tables have 2 files, below the default limit. **Suggested fix** For the seven procedures whose rows are materialized before the cut (`stream().limit`, `take`, `df.limit`), apply the filter first: `applyFilter(rows, filter, outputType).take(limit)`. For `show_logfile_records`, `show_timeline` and `show_cleans`, where `limit` bounds how much is read, either document that `filter` only sees the first `limit` entries or push the predicate into the read loop. Add one procedure-level test per procedure with more rows than `limit` and a filter that matches only rows past the cutoff. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
