Top of my head, I can think of the zip operation that RDD provides. So for
example, if you have two DataFrames df1 and df2, you could do something like
this:
val newDF = df1.rdd.zip(df2.rdd).map { case(rowFromDf1, rowFromDf2) =>
(....)}.toDF(...)
Couple of things to keep in mind:
1) Both df1 and df2 should have the same number of rows.
2) You are assuming that row N from df1 is related to row N from df2.
Mohammed
Author: Big Data Analytics with
Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: spR [mailto:data.smar...@gmail.com]
Sent: Wednesday, June 15, 2016 4:08 PM
To: Mohammed Guller
Cc: Natu Lauchande; user
Subject: Re: concat spark dataframes
Hey,
There are quite a lot of fields. But, there are no common fields between the 2
dataframes. Can I not concatenate the 2 frames like we can do in pandas such
that the resulting dataframe has columns from both the dataframes?
Thank you.
Regards,
Misha
On Wed, Jun 15, 2016 at 3:44 PM, Mohammed Guller
<moham...@glassbeam.com<mailto:moham...@glassbeam.com>> wrote:
Hi Misha,
What is the schema for both the DataFrames? And what is the expected schema of
the resulting DataFrame?
Mohammed
Author: Big Data Analytics with
Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: Natu Lauchande [mailto:nlaucha...@gmail.com<mailto:nlaucha...@gmail.com>]
Sent: Wednesday, June 15, 2016 2:07 PM
To: spR
Cc: user
Subject: Re: concat spark dataframes
Hi,
You can select the common collumns and use DataFrame.union all .
Regards,
Natu
On Wed, Jun 15, 2016 at 8:57 PM, spR
<data.smar...@gmail.com<mailto:data.smar...@gmail.com>> wrote:
hi,
how to concatenate spark dataframes? I have 2 frames with certain columns. I
want to get a dataframe with columns from both the other frames.
Regards,
Misha
