voonhous opened a new issue, #19638: URL: https://github.com/apache/hudi/issues/19638
**Describe the problem** `HoodieProcedureFilterUtils.evaluateFilter` resolves function calls against a hardcoded resolution table; any function not in the table falls through as `case _ => unresolvedFunc` (`HoodieProcedureFilterUtils.scala:353`), stays unresolved, throws on eval inside the per-row `Try`, and the row is treated as a non-match. `validateFilterExpression` (`:461`) only validates column references, so no error ever surfaces: a typo'd or unsupported function yields an empty result set that is indistinguishable from "no matching rows". Probe (Spark 4.1.1): - `concat(name, 'x') = 'a1x'` keeps 0/1 rows - `instr(name, 'a') = 1` keeps 0/1 - `if(name = 'a1', true, false)` keeps 0/1 (while `case when name = 'a1' then true else false end` keeps 1/1, because the parser lowers it without an `UnresolvedFunction`) - control: `upper(name) = 'A1'` keeps 1/1 Reachable from every procedure that exposes a `filter` argument. **To reproduce** Call any filter-supporting procedure with a function outside the table, e.g. `show_cleans(table => 't', filter => "concat(action, 'x') = 'cleanx'")`: returns zero rows instead of an error. A pinned test documenting the silent drop exists in `TestHoodieProcedureFilterUtils` since #19161. **Suggested fix** Make `validateFilterExpression` reject function names that are not in the resolution table (fail fast, listing the supported functions), or resolve functions through Spark's `FunctionRegistry` so any built-in works. Flip the pinned assertions when fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
