[
https://issues.apache.org/jira/browse/SPARK-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154935#comment-15154935
]
Apache Spark commented on SPARK-13410:
--------------------------------------
User 'damnMeddlingKid' has created a pull request for this issue:
https://github.com/apache/spark/pull/11279
> unionAll throws error with DataFrames containing UDT columns.
> -------------------------------------------------------------
>
> Key: SPARK-13410
> URL: https://issues.apache.org/jira/browse/SPARK-13410
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0, 1.6.0
> Reporter: Franklyn Dsouza
> Labels: patch
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Unioning two DataFrames that contain UDTs fails with
> {quote}
> AnalysisException: u"unresolved operator 'Union;"
> {quote}
> I tracked this down to this line
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala#L202
> Which compares datatypes between the output attributes of both logical plans.
> However for UDTs this will be a new instance of the UserDefinedType or
> PythonUserDefinedType
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L158
>
> So this equality check will check if the two instances are the same and since
> they aren't references to a singleton this check fails. Note: this will work
> fine if you are unioning the dataframe with itself.
> I have a patch for this which overrides the equality operator on the two
> classes here: https://github.com/damnMeddlingKid/spark/pull/2
> Reproduction steps
> {code}
> from pyspark.sql.tests import PythonOnlyPoint, PythonOnlyUDT
> from pyspark.sql import types
> schema = types.StructType([types.StructField("point", PythonOnlyUDT(), True)])
> #note they need to be two separate dataframes
> a = sqlCtx.createDataFrame([[PythonOnlyPoint(1.0, 2.0)]], schema)
> b = sqlCtx.createDataFrame([[PythonOnlyPoint(3.0, 4.0)]], schema)
> c = a.unionAll(b)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
