Wan Kun created SPARK-44773: ------------------------------- Summary: Code-gen CodegenFallback expression in WholeStageCodegen if possible Key: SPARK-44773 URL: https://issues.apache.org/jira/browse/SPARK-44773 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Wan Kun
Now both WholeStageCodegen framework and SubExpressionElimination framework does not support CodegenFallback expression, but the CodegenFallback expression which contains nullSafeEval method could gen-code just like common expressions, now they are always be executed in a new SpecificUnsafeProjection class, and we can not eliminate the sub expressions. For example: SQL: {code:sql} SELECT from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').x, from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').b FROM values('{"a":1, "b":0.8}') t(s) {code} plan: {code:java} *(1) Project [from_json(StructField(x,IntegerType,true), regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).x AS from_json(regexp_replace(s, a, x, 1)).x#219, from_json(StructField(b,DoubleType,true), regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).b AS from_json(regexp_replace(s, a, x, 1)).b#220] +- *(1) LocalTableScan [s#218] {code} Due to expression org.apache.spark.sql.catalyst.expressions.JsonToStructs is CodegenFallback expression, so we can not reuse the result of {*}regexp_replace(s, 'a', 'x'){*}. We can support expression org.apache.spark.sql.catalyst.expressions.JsonToStructs code-gen in WholeStageCodegen framework, and then reuse the result of {*}regexp_replace(s, 'a', 'x'){*}. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org