anshulsingh-py opened a new issue, #19852: URL: https://github.com/apache/hudi/issues/19852
### Context Follow-up to #19638 / #19850, as discussed with @voonhous [here](https://github.com/apache/hudi/issues/19638#issuecomment-5559550232). #19850 fixes procedure filters so unsupported functions are rejected with a clear error at validation time, instead of silently returning zero rows. That's the correct safety fix. This issue proposes a complementary enhancement: expand the set of functions `HoodieProcedureFilterUtils` actually supports, by falling back to Spark's own `FunctionRegistry` to resolve any function not covered by the hand-maintained resolution table, instead of rejecting it. This would let filters use any Spark builtin (e.g. `concat`, `instr`) rather than only the ones explicitly enumerated. ### Proposed approach - In the shared bind/resolve pipeline, when a function isn't found in the existing hardcoded table, attempt `sparkSession.sessionState.functionRegistry.lookupFunction(FunctionIdentifier(name), args)`. - If that also fails to resolve (e.g. a genuine typo), fall through to the existing validation-rejection behavior from #19850 — no change to safety guarantees for truly unknown names. - Needs compatibility tests per component/Spark version, since the resolvable function set now depends on Spark's registry contents rather than a fixed list. ### Blocked on - #19638 (needs #19850 merged first, since the fallback builds on the shared resolver introduced there) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
