[ https://issues.apache.org/jira/browse/SPARK-13337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172881#comment-15172881 ]
Zhong Wang edited comment on SPARK-13337 at 2/29/16 11:40 PM: -------------------------------------------------------------- It doesn't help in my case, because it doesn't support null-safe joins. It would be great if there is an interface like: {code} def join(right: DataFrame, usingColumns: Seq[String], joinType: String, nullSafe:Boolean): DataFrame {code} The current join-using-column interface works great if the joining tables doesn't contain null values: it can eliminate the null columns generated from outer joins automatically. The general joining methods in your example support null-safe joins perfectly, but it cannot automatically eliminate the null columns, which are generated from outer joins. Sorry that it is a little bit complicated here. Please let me know if you need a concrete example. was (Author: zwang): It doesn't help in my case, because it doesn't support null-safe joins. It would be great if there is an interface like: {code} def join(right: DataFrame, usingColumns: Seq[String], joinType: String, nullSafe:Boolean): DataFrame {code} It works great if the joining tables doesn't contain null values: it can eliminate the null columns generated from outer joins automatically. The general joining methods in your example support null-safe joins perfectly, but it cannot automatically eliminate the null columns, which are generated from outer joins. Sorry that it is a little bit complicated here. Please let me know if you need a concrete example. > DataFrame join-on-columns function should support null-safe equal > ----------------------------------------------------------------- > > Key: SPARK-13337 > URL: https://issues.apache.org/jira/browse/SPARK-13337 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 1.6.0 > Reporter: Zhong Wang > Priority: Minor > > Currently, the join-on-columns function: > {code} > def join(right: DataFrame, usingColumns: Seq[String], joinType: String): > DataFrame > {code} > performs a null-insafe join. It would be great if there is an option for > null-safe join. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org