Minack
Sent: Thursday, February 27, 2020 8:51 PM
To: Manjunath Shetty H ; user@spark.apache.org
Subject: Re: Convert each partition of RDD to Dataframe
Manjunath,
You can define your DataFrame in parallel in a multi-threaded driver.
Enrico
Am 27.02.20 um 15:50 schrieb Manjunath Shetty H:
Hi
What will be the performance implications ?
Regards
Manjunath
From: Enrico Minack
Sent: Thursday, February 27, 2020 7:57 PM
To:user@spark.apache.org
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables
*Sent:* Thursday, February 27, 2020 7:57 PM
*To:* user@spark.apache.org
*Subject:* Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the
first place?
Enrico
Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
Hi Vinodh
will be the performance implications ?
Regards
Manjunath
From: Enrico Minack
Sent: Thursday, February 27, 2020 7:57 PM
To: user@spark.apache.org
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading
*Subject:* Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective
dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H
mailto:manjunathshe...@live.com>>
vinodh
Sent: Thursday, February 27, 2020 7:04 PM
To: manjunathshe...@live.com
Cc: user
Subject: Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective dataframe
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective
dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H
wrote:
>
> Hello All,
>
> In spark i am creating the custom partitions with Custom RDD, each
>
Hello All,
In spark i am creating the custom partitions with Custom RDD, each partition
will have different schema. Now in the transformation step we need to get the
schema and run some Dataframe SQL queries per partition, because each partition
data has different schema.
How to get the
