LuciferYang commented on code in PR #55719:
URL: https://github.com/apache/spark/pull/55719#discussion_r3199188769
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/UnionCodegenSuite.scala:
##########
@@ -602,6 +602,45 @@ class UnionCodegenSuite extends QueryTest with
SharedSparkSession {
"numOutputRows should be 0 for all-empty union")
}
}
+
Review Comment:
```scala
test("SPARK-56482: union with sample children fuses (or falls back) without
crashing") {
val a = rangeDF(20).sample(false, 0.5, 1L)
val b = rangeDF(20).sample(false, 0.5, 1L)
val df = a.union(b).filter(col("id") > 0)
df.collect()
assertFlagParity(() => a.union(b).orderBy("id"))
}
```
After this PR, the test above appears to fail
```
13:47:26.119 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
13:47:27.985 ERROR
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: Failed to
compile the generated Java code.
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
168, Column 44: Unknown variable or type "childLocalIdx"
at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:13014)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7199)
at
org.codehaus.janino.UnitCompiler.access$14300(UnitCompiler.java:236)
at
org.codehaus.janino.UnitCompiler$23.visitPackage(UnitCompiler.java:6684)
at
org.codehaus.janino.UnitCompiler$23.visitPackage(UnitCompiler.java:6681)
at org.codehaus.janino.Java$Package.accept(Java.java:4627)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6681)
13:47:27.992 ERROR
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends
org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private boolean range_initRange_0;
/* 010 */ private long range_nextIndex_0;
/* 011 */ private TaskCont...
[info] - SPARK-56482: union with sample children fuses (or falls back)
without crashing *** FAILED *** (1 second, 214 milliseconds)
[info] java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
168, Column 44: Failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
168, Column 44: Unknown variable or type "childLocalIdx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
