voonhous commented on code in PR #19850:
URL: https://github.com/apache/hudi/pull/19850#discussion_r3944128281
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedureFilterUtils.scala:
##########
@@ -468,9 +474,22 @@ object HoodieProcedureFilterUtils {
val columnNames = schema.fieldNames.toSet
val referencedColumns = extractColumnReferences(parsedExpr)
val invalidColumns = referencedColumns -- columnNames
+ val resolvedExpr = bindAndResolveExpression(parsedExpr, schema)
+ val unsupportedFunctions = extractFunctionReferences(resolvedExpr)
+ val unsupportedExpressions = resolvedExpr.collect {
+ case expression: Unevaluable
+ if !expression.isInstanceOf[UnresolvedAttribute]
+ && !expression.isInstanceOf[UnresolvedFunction] =>
expression.prettyName
+ }.toSet
if (invalidColumns.nonEmpty) {
Left(s"Invalid column references: ${invalidColumns.mkString(", ")}.
Available columns: ${columnNames.mkString(", ")}")
+ } else if (unsupportedFunctions.nonEmpty) {
+ Left(s"Unsupported functions:
${unsupportedFunctions.toSeq.sorted.mkString(", ")}")
+ } else if (!resolvedExpr.resolved || unsupportedExpressions.nonEmpty) {
Review Comment:
**blocker:** Validation rejects valid Long-column filters before numeric
coercion. For `time_taken_in_millis > 60000`, the compiled PR source on Spark
3.5.5 reports `resolved=false`, while its evaluator matches `Row(60001L)`;
existing `TestShowCleansProcedures.scala:464,467` predicates are also affected.
Could we share the complete preparation, including coercion, between validation
and evaluation, and add a positive Long-filter validation assertion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
