[ https://issues.apache.org/jira/browse/SPARK-31403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083576#comment-17083576 ]
Carl Sverre commented on SPARK-31403: ------------------------------------- Thanks for checking this on master [~hyukjin.kwon]! If you happen to have Spark master running, can you send me `plan.asCode` and/or the output of your repro? I am very curious to understand what Spark is generating in this case. It may be possible to repro this in master with something as simple as `select null from t`. If you don't have time I will see if I can get spark master running and try to repro this myself. Thanks! > TreeNode asCode function incorrectly handles null literals > ---------------------------------------------------------- > > Key: SPARK-31403 > URL: https://issues.apache.org/jira/browse/SPARK-31403 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.4 > Reporter: Carl Sverre > Priority: Minor > > In the TreeNode code in Catalyst the asCode function incorrectly handles null > literals. When it tries to render a null literal it will match {{null}} > using the third case expression and try to call {{null.toString}} which will > raise a NullPointerException. > I verified this bug exists in Spark 2.4.4 and the same code appears to be in > master: > [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L707] > The fix seems trivial - add an explicit case for null. > One way to reproduce this is via: > {code:java} > val plan = > spark > .sql("select if(isnull(id), null, 2) from testdb_jdbc.users") > .queryExecution > .optimizedPlan > println(plan.asInstanceOf[Project].projectList.head.asCode) {code} > However any other way which generates a Literal with the value null will > cause the issue. > In this case the above SparkSQL will generate the literal: {{Literal(null, > IntegerType)}} for the "trueValue" of the if statement. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org