Can you try caching the individual dataframes and then union them?
It may save you time.
Thanks
Deepak
On Wed, Nov 16, 2016 at 12:35 PM, Devi P.V wrote:
> Hi all,
>
> I have 4 data frames with three columns,
>
> client_id,product_id,interest
>
> I want to combine these 4 dataframes into one dat
016 11:06 PM
To: user @spark<mailto:user@spark.apache.org>
Subject: what is the optimized way to combine multiple dataframes into one
dataframe ?
Hi all,
I have 4 data frames with three columns,
client_id,product_id,interest
I want to combine these 4 dataframes into one dataframe.I use
Hi all,
I have 4 data frames with three columns,
client_id,product_id,interest
I want to combine these 4 dataframes into one dataframe.I used union like
following
df1.union(df2).union(df3).union(df4)
But it is time consuming for bigdata.what is the optimized way for doing
this using spark 2.0