[
https://issues.apache.org/jira/browse/SPARK-13337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172881#comment-15172881
]
Zhong Wang edited comment on SPARK-13337 at 2/29/16 11:40 PM:
--------------------------------------------------------------
It doesn't help in my case, because it doesn't support null-safe joins. It
would be great if there is an interface like:
{code}
def join(right: DataFrame, usingColumns: Seq[String], joinType: String,
nullSafe:Boolean): DataFrame
{code}
The current join-using-column interface works great if the joining tables
doesn't contain null values: it can eliminate the null columns generated from
outer joins automatically. The general joining methods in your example support
null-safe joins perfectly, but it cannot automatically eliminate the null
columns, which are generated from outer joins.
Sorry that it is a little bit complicated here. Please let me know if you need
a concrete example.
was (Author: zwang):
It doesn't help in my case, because it doesn't support null-safe joins. It
would be great if there is an interface like:
{code}
def join(right: DataFrame, usingColumns: Seq[String], joinType: String,
nullSafe:Boolean): DataFrame
{code}
It works great if the joining tables doesn't contain null values: it can
eliminate the null columns generated from outer joins automatically. The
general joining methods in your example support null-safe joins perfectly, but
it cannot automatically eliminate the null columns, which are generated from
outer joins.
Sorry that it is a little bit complicated here. Please let me know if you need
a concrete example.
> DataFrame join-on-columns function should support null-safe equal
> -----------------------------------------------------------------
>
> Key: SPARK-13337
> URL: https://issues.apache.org/jira/browse/SPARK-13337
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Zhong Wang
> Priority: Minor
>
> Currently, the join-on-columns function:
> {code}
> def join(right: DataFrame, usingColumns: Seq[String], joinType: String):
> DataFrame
> {code}
> performs a null-insafe join. It would be great if there is an option for
> null-safe join.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
