You need to add a select clause to at least one dataframe to give them the
same schema before you can union them (much like in SQL).
On Tue, May 5, 2015 at 3:24 AM, Wilhelm <niznik.pa...@gmail.com> wrote:
> Hey there,
>
> 1.) I'm loading 2 avro files with that have slightly different schema
>
> df1 = sqlc.load(file1, "com.databricks.spark.avro")
> df2 = sqlc.load(file2, "com.databricks.spark.avro")
>
> 2.) I want to unionAll them
>
> nfd = dfs1.unionAll(dfs2)
>
> 3.) Getting the following error
>
> ---------------------------------------------------------------------------
> Py4JJavaError Traceback (most recent call last)
> <ipython-input-190-a86d9adbea83> in <module>()
> 17
> 18
> ---> 19 nfd = dfs1.unionAll(dfs2)
> 20
> 21
>
> /home/hadoop/spark/python/pyspark/sql/dataframe.pyc in unionAll(self,
> other)
> 669 This is equivalent to `UNION ALL` in SQL.
> 670 """
> --> 671 return DataFrame(self._jdf.unionAll(other._jdf),
> self.sql_ctx)
> 672
> 673 def intersect(self, other):
>
> /home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in
> __call__(self, *args)
> 536 answer = self.gateway_client.send_command(command)
> 537 return_value = get_return_value(answer,
> self.gateway_client,
> --> 538 self.target_id, self.name)
> 539
> 540 for temp_arg in temp_args:
>
> /home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in
> get_return_value(answer, gateway_client, target_id, name)
> 298 raise Py4JJavaError(
> 299 'An error occurred while calling {0}{1}{2}.\n'.
> --> 300 format(target_id, '.', name), value)
> 301 else:
> 302 raise Py4JError(
>
> Py4JJavaError: An error occurred while calling o76196.unionAll.
> : org.apache.spark.sql.AnalysisException: unresolved operator 'Union ;
> at
>
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.failAnalysis(CheckAnalysis.scala:37)
> at
>
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$apply$3.apply(CheckAnalysis.scala:97)
> at
>
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$apply$3.apply(CheckAnalysis.scala:43)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:88)
> at
>
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.apply(CheckAnalysis.scala:43)
> at
>
> org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:1069)
> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
> at
> org.apache.spark.sql.DataFrame.logicalPlanToDataFrame(DataFrame.scala:157)
> at org.apache.spark.sql.DataFrame.unionAll(DataFrame.scala:641)
> at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
> at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> at py4j.Gateway.invoke(Gateway.java:259)
> at
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:207)
> at java.lang.Thread.run(Thread.java:745)
> ---------------------------------------------------------------------------
>
> 4.) Is it possible to automatically merge 2 DFs with different schemas like
> that? Am I doing sth. wrong?
>
> Much appreciated!
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Two-DataFrames-with-different-schema-unionAll-issue-tp22765.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
