[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Aerts updated SPARK-44132: --------------------------------- Summary: nesting full outer joins confuses code generator (was: nesting full outer joins confuses confuses code generator) > nesting full outer joins confuses code generator > ------------------------------------------------ > > Key: SPARK-44132 > URL: https://issues.apache.org/jira/browse/SPARK-44132 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.3.0, 3.4.0, 3.5.0 > Environment: We verified the existence of this bug from spark 3.3 > until spark 3.5. > Reporter: Steven Aerts > Priority: Major > > We are seeing issues with the code generator when querying java bean encoded > data with 2 nested joins. > {code:java} > dsA.join(dsB, seq("id"), "full_outer").join(dsC, seq("id"), "full_outer"); > {code} > will generate invalid code in the code generator. And can depending on the > data used generate stack traces like: > {code:java} > Caused by: java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.smj_consumeFullOuterJoinRow_0$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.wholestagecodegen_findNextJoinRows_0$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown > Source) > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > {code} > Or: > {code:java} > Caused by: java.lang.AssertionError: index (2) should < 2 > at > org.apache.spark.sql.catalyst.expressions.UnsafeRow.assertIndexIsValid(UnsafeRow.java:118) > at > org.apache.spark.sql.catalyst.expressions.UnsafeRow.isNullAt(UnsafeRow.java:315) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.smj_consumeFullOuterJoinRow_0$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown > Source) > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > {code} > When we look at the generated code we see that the code generator seems to be > mixing up parameters. For example: > {code:java} > if (smj_leftOutputRow_0 != null) { //<==== null > check for wrong/left parameter > boolean smj_isNull_12 = smj_rightOutputRow_0.isNullAt(1); //<==== causes > NPE on right parameter here{code} > It is as if the the nesting of 2 full outer joins is confusing the code > generator and as such generating invalid code. > There is one other strange thing. We found this issue when using data sets > which were using the java bean encoder. We tried to reproduce this in the > spark shell or using scala case classes but were unable to do so. > We made a reproduction scenario as unit tests (one for each of the stacktrace > above) on the spark code base and will make it available as a pull request to > this case. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org