Re: unsubscribe

[santhosh Gandhe]

To remove your address from the list, send a message to:
   

On Mon, Apr 24, 2023 at 10:41 PM  wrote:

> unsubscribe

unsubscribe

[yxj1141]

unsubscribe

unsubscribe

[Daniel Tavares de Santana]

unsubscribe

unsubscribe

[ypl]

unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Jatinder Assi]

unsubscribe

unsubscribe

[Deepthi Sathia Raj]

> unsubscribe
>

unsubscribe

[William R]

unsubscribe

Re: Unsubscribe

[Atheeth SH]

please send an empty email to:
user-unsubscr...@spark.apache.org
to unsubscribe yourself from the list.

Thanks

On Thu, 23 Feb 2023 at 07:07, Tang Jinxin  wrote:

> Unsubscribe
>

Re: unsubscribe

[Atheeth SH]

please send an empty email to:
user-unsubscr...@spark.apache.org
to unsubscribe yourself from the list.

Thanks,
Atheeth

On Fri, 24 Feb 2023 at 03:58, Roberto Jr  wrote:

> please unsubscribe from that email list.
> thank you in advance.
> roberto.
>

unsubscribe

[Roberto Jr]

please unsubscribe from that email list.
thank you in advance.
roberto.

Unsubscribe

[Tang Jinxin]

Unsubscribe

Unsubscribe

[Qijia Liu]

Unsubscribe

Re: Unsubscribe

[winnie hw]

Please send an email to user-unsubscr...@spark.apache.org rather than this
one.


On Sun, Feb 19, 2023 at 12:06 PM Sendil Chidambaram 
wrote:

> Unsubscribe
>

Unsubscribe

[Sendil Chidambaram]

Unsubscribe

Unsubscribe

[Patrik Medvedev]

Unsubscribe

Re: Unsubscribe

[LinuxGuy]

please send an empty email to:
user-unsubscr...@spark.apache.org
to unsubscribe yourself from the list.


On Thu, Feb 9, 2023 at 12:38 PM fuwei...@163.com  wrote:

> Unsubscribe
>

Unsubscribe

[fuwei901]

Unsubscribe

[fuwei901]

Unsubscribe

[Tushar Machavolu]

Unsubscribe

Unsubscribe

[Spyros Gasteratos]

Unsubscribe

Unsubscribe

[Calum]

Unsubscribe

unsubscribe

[peng]

unsubscribe

unsubscribe

[김병찬]

제목unsubscribe

unsubscribe

[Sebastian Schere]

[UNSUBSCRIBE]

[Sebastian Schere]

UNSUBSCRIBE

UNSUBSCRIBE

[prashanth t]

UNSUBSCRIBE

UNSUBSCRIBE

[Agostino Calamita]

UNSUBSCRIBE

UNSUBSCRIBE

[yixu2...@163.com]

UNSUBSCRIBE

UNSUBSCRIBE

[Joji V J]

UNSUBSCRIBE

UNSUBSCRIBE

[Ricardo Sardenberg]

Unsubscribe

[benson fang]

Thank you all

Unsubscribe

[Peter Kovgan]

Shay Elbaz  schrieb am So. 14. Aug. 2022 um 09:22:

> Hi,
>
> I have a simple ETL application, where the data source schama needs to be
> sanitized. Column names might include special characters that need to be
> removed. For example, from "some{column}" to "some_column".
> Normally I'd just alias the columns, but in this case the schema can have
> thousands of deeply nested columns. Creating a new StructType feels more
> intuitive and simpler, but the only way I know of to apply the new schema
> is to create a new dataframe -
> *spark.createDataFrame(old_df.rdd, new_schema)*. This makes the
> deserialization and re-serialization of the dataframe the most expensive
> operation in that "simple" ETL app.
>
> *To make things worse,* since it's a pyspark application, the RDD is
> treated as Python RDD and all the data is moving from the JVM to Python and
> back, without any real transformation.
> This is resolved by creating the new DF on the JVM only:
>
> jschema = 
> spark._sc._jvm.org.apache.spark.sql.types.DataType.fromJson(sanitized_schema.json())
> sanitized_df = DataFrame(spark._jsparkSession.createDataFrame(df._jdf.rdd(), 
> jschema), spark)
>
> Is there another way to do a bulk rename operation? I'd like to avoid
> creating some uber "select" statement with aliases, or multiple
> withColumnRenamed operations, as much as possible, mainly for maintenance
> reasons.
>
> Thanks
>

Unsubscribe

[Pedro Tuero]

Unsubscribe

Unsubscribe

[sanjeev shrestha]

Unsubscribe

unsubscribe

[Huajian Mao]

unsubscribe

-- 
Best Regards,
- Huajian

Unsubscribe

[Raghunadh Madamanchi]

On Tue, Sep 13, 2022, 6:30 PM Hari Kunapareddy 
wrote:

>
>

Unsubscribe

[Hari Kunapareddy]

Unsubscribe

[Mohd Shukri Hasan]

Unsubscribe

[Nadeem Lalani]

Unsubscribe

[Pascal Taddei]

unsubscribe

[Sivakumar Ganesan]

unsubscribe

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Alexey Milogradov]

Re: Unsubscribe

[pengyh]

to unsubscribe: user-unsubscr...@spark.apache.org


Shrikar archak wrote:



unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

[Shrikar archak]

unsubscribe

Re: unsubscribe

[pengyh]

you could be able to unsubscribe yourself by using the signature below.



To unsubscribe e-mail: user-unsubscr...@spark.apache.org


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Martin Soch]

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

[Ashish]

Unsubscribe
Sent from my iPhone

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

[Karthik Jayaraman]

unsubscribe

[randy clinton]

-- 
I appreciate your time,

~Randy

Unsubscribe

[Daan Stroep]

Unsubscribe

unsubscribe

[Ahmed Kamal Abdelfatah]

-- 










*This email, including any information it contains and any 
attachments to it, is confidential and may be privileged. This email is 
intended only for the use of the named recipient(s). If you are not a named 
recipient, please notify the sender immediately by replying to this message 
and delete the original message. You should not disclose or copy this 
email, any of its contents or any attachments to it. This email may have 
been transmitted over an unsecure public network and, therefore, Careem 
does not accept responsibility for its contents or for any damage sustained 
as a result of viewing its contents or in connection with its transmission. 
Careem reserves the right to monitor all communications from or to this 
account.*

unsubscribe

[Ray Qiu]

Unsubscribe

[Sahil Bali]

Unsubscribe

Unsubscribe

[Sahil Bali]

Unsubscribe

Re: Unsubscribe

[wilson]

please send the message to user-unsubscr...@spark.apache.org
to unsubscribe.


Ajay Thompson wrote:

Unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

[Ajay Thompson]

Unsubscribe

unsubscribe

[Deepak Gajare]

unsubscribe

unsubscribe

[Basavaraj]

unsubscribe

smime.p7s
Description: S/MIME cryptographic signature

Unsubscribe

[van wilson]

> On Mar 16, 2022, at 7:38 AM,   wrote:
> 
> Thanks, Jayesh and all. I finally get the correlation data frame using agg 
> with list of functions.
> I think the list of functions which generate a column should be more detailed 
> description.
> 
> Liang
> 
> - 原始邮件 -
> 发件人："Lalwani, Jayesh" 
> 收件人："ckgppl_...@sina.cn" , Enrico Minack 
> , Sean Owen 
> 抄送人：user 
> 主题：Re: 回复：Re: 回复：Re: calculate 
> correlation_between_multiple_columns_and_one_specific_column_after_groupby_the_spark_data_frame
> 日期：2022年03月16日 20点49分
> 
> No, You don’t need 30 dataframes and self joins. Convert a list of columns to 
> a list of functions, and then pass the list of functions to the agg function
> 
>  
> 
>  
> 
> From: "ckgppl_...@sina.cn" 
> Reply-To: "ckgppl_...@sina.cn" 
> Date: Wednesday, March 16, 2022 at 8:16 AM
> To: Enrico Minack , Sean Owen 
> Cc: user 
> Subject: [EXTERNAL] 回复：Re: 回复：Re: calculate correlation 
> between_multiple_columns_and_one_specific_column_after_groupby_the_spark_data_frame
> 
>  
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
>  
> 
> Thanks, Enrico.
> 
> I just found that I need to group the data frame then calculate the 
> correlation. So I will get a list of dataframe, not columns. 
> 
> So I used following solution:
> 
> 1.   use following codes to create a mutable data frame df_all. I used 
> the first datacol to calculate correlation.  
> df.groupby("groupid").agg(functions.corr("datacol1","corr_col")
> 
> 2.   iterate all remaining datacol columns, create a temp data frame for 
> this iteration. In this iteration, use df_all to join the temp data frame on 
> the groupid column, then drop duplicated groupid column.
> 
> 3.   after the iteration, I will get the dataframe which contains all 
> correlation data.
> 
> 
> 
> 
> I need to verify the data to make sure it is valid.
> 
> 
> 
> 
> Liang
> 
> - 原始邮件 -
> 发件人：Enrico Minack 
> 收件人：ckgppl_...@sina.cn, Sean Owen 
> 抄送人：user 
> 主题：Re: 回复：Re: calculate correlation 
> between_multiple_columns_and_one_specific_column_after_groupby_the_spark_data_frame
> 日期：2022年03月16日 19点53分
> 
>  
> 
> If you have a list of Columns called `columns`, you can pass them to the 
> `agg` method as:
> 
>  
> 
>   agg(columns.head, columns.tail: _*)
> 
>  
> 
> Enrico
> 
>  
> 
>  
> 
> Am 16.03.22 um 08:02 schrieb ckgppl_...@sina.cn :
> 
> Thanks, Sean. I modified the codes and have generated a list of columns.
> 
> I am working on convert a list of columns to a new data frame. It seems that 
> there is no direct  API to do this.
> 
>  
> 
> - 原始邮件 -
> 发件人：Sean Owen  
> 收件人：ckgppl_...@sina.cn 
> 抄送人：user  
> 主题：Re: calculate correlation between multiple columns and one specific column 
> after groupby the spark data frame
> 日期：2022年03月16日 11点55分
> 
>  
> 
> Are you just trying to avoid writing the function call 30 times? Just put 
> this in a loop over all the columns instead, which adds a new corr col every 
> time to a list. 
> 
> On Tue, Mar 15, 2022, 10:30 PM  > wrote:
> 
> Hi all,
> 
>  
> 
> I am stuck at  a correlation calculation problem. I have a dataframe like 
> below:
> 
> groupid
> 
> datacol1
> 
> datacol2
> 
> datacol3
> 
> datacol*
> 
> corr_co
> 
> 1
> 
> 1
> 
> 2
> 
> 3
> 
> 4
> 
> 5
> 
> 1
> 
> 2
> 
> 3
> 
> 4
> 
> 6
> 
> 5
> 
> 2
> 
> 4
> 
> 2
> 
> 1
> 
> 7
> 
> 5
> 
> 2
> 
> 8
> 
> 9
> 
> 3
> 
> 2
> 
> 5
> 
> 3
> 
> 7
> 
> 1
> 
> 2
> 
> 3
> 
> 5
> 
> 3
> 
> 3
> 
> 5
> 
> 3
> 
> 1
> 
> 5
> 
> I want to calculate the correlation between all datacol columns and corr_col 
> column by each groupid.
> 
> So I used the following spark scala-api codes:
> 
> df.groupby("groupid").agg(functions.corr("datacol1","corr_col"),functions.corr("datacol2","corr_col"),functions.corr("datacol3","corr_col"),functions.corr("datacol*","corr_col"))
> 
>  
> 
> This is very inefficient. If I have 30 data_col columns, I need to input 30 
> times functions.corr to calculate correlation.
> 
> I have searched, it seems that functions.corr doesn't accept a List/Array 
> parameter, and df.agg doesn't accept a function to be parameter.
> 
> So any  spark scala API codes can do this job efficiently?
> 
>  
> 
> Thanks
> 
>  
> 
> Liang
> 
>  
> 

Re: Unsubscribe

[Bitfox]

please send an empty email to:
user-unsubscr...@spark.apache.org
to unsubscribe yourself from the list.


On Sat, Mar 12, 2022 at 2:42 PM Aziret Satybaldiev <
satybaldiev.azi...@gmail.com> wrote:

>

Unsubscribe

[Aziret Satybaldiev]

unsubscribe

[Basavaraj]

smime.p7s
Description: S/MIME cryptographic signature

unsubscribe

[Ramaswamy, Muthuraman]

unsubscribe

unsubscribe

[ashmeet kandhari]

Unsubscribe

On Thu, Mar 3, 2022, 05:45 Basavaraj  wrote:

> unsubscribe

Re: unsubscribe

[Ghousia]

unsubscribe

On Thu, Mar 3, 2022 at 5:44 AM Basavaraj  wrote:

> unsubscribe

unsubscribe

[Basavaraj]

unsubscribe

smime.p7s
Description: S/MIME cryptographic signature

Unsubscribe

[ashmeet kandhari]

Unsubscribe

Unsubscribe

[ashmeet kandhari]

Unsubscribe

Unsubscribe

[William R]

Unsubscribe

unsubscribe

[Basavaraj]

unsubscribe

smime.p7s
Description: S/MIME cryptographic signature

Re: Unsubscribe

[Bitfox]

Please send an e-mail: user-unsubscr...@spark.apache.org
to unsubscribe yourself from the mailing list.

On Thu, Feb 10, 2022 at 1:38 AM Yogitha Ramanathan 
wrote:

>

Unsubscribe

[Yogitha Ramanathan]

Unsubscribe

[Yogitha Ramanathan]

Unsubscribe

Re: Unsubscribe

[Bitfox]

Please send an e-mail: user-unsubscr...@spark.apache.org
to unsubscribe yourself from the mailing list.

On Sun, Feb 6, 2022 at 2:21 PM Rishi Raj Tandon 
wrote:

> Unsubscribe
>

Unsubscribe

[Rishi Raj Tandon]

Unsubscribe

Unsubscribe

[chen...@birdiexx.com]

Unsubscribe

Re: unsubscribe

[Bitfox]

The signature in your messages has showed how to unsubscribe.

To unsubscribe e-mail: user-unsubscr...@spark.apache.org

On Mon, Jan 31, 2022 at 7:53 PM Lucas Schroeder Rossi 
wrote:

> unsubscribe
>
> -----
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

unsubscribe

[Lucas Schroeder Rossi]

unsubscribe

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Lucas Schroeder Rossi]

unsubscribe

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Rajeev]

Unsubscribe

[Yogitha Ramanathan]

Re: unsubscribe

[Bitfox]

The signature in your mail has showed the info:

To unsubscribe e-mail: user-unsubscr...@spark.apache.org



On Sun, Jan 30, 2022 at 8:50 PM Lucas Schroeder Rossi 
wrote:

> unsubscribe
>
> -----
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

unsubscribe

[Lucas Schroeder Rossi]

unsubscribe

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Lucas Schroeder Rossi]

unsubscribe

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: unsubscribe

[capitnfrakass]

On 22/01/2022 11:07, Renan F. Souza wrote:

unsubscribe


You could be able to unsubscribe yourself from the list by sending an 
email to:

user-unsubscr...@spark.apache.org

thanks.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[Renan F. Souza]

unsubscribe

Unsubscribe

[Aniket Khandelwal]

unsubscribe

Thanks

Re: unsubscribe

[frakass]

please send an empty message to: user-unsubscr...@spark.apache.org to 
unsubscribe yourself from the list.


Thanks

On 2022/1/15 7:04, ALOK KUMAR SINGH wrote:

unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

unsubscribe

[ALOK KUMAR SINGH]

unsubscribe

Re: Unsubscribe

[Piper H]

please send an empty email to:
user-unsubscr...@spark.apache.org
to unsubscribe yourself from the list.

On Fri, Dec 17, 2021 at 11:14 AM Ankit Maloo 
wrote:

> Please do unsubscribe me from your mailing list.
>

Unsubscribe

[Ankit Maloo]

Please do unsubscribe me from your mailing list.

Unsubscribe

[马殿军]

Unsubscribe


Ma Dianjun

Unsubscribe

[Joji V J]

unsubscribe

Unsubscribe

[HARSH TAKKAR]

Unsubscribe

Unsubscribe

[Edwin Barahona]

Unsubscribe

unsubscribe

[Shafaq]

unsubscribe

-- 
Kind Regards,
Shafaq

unsubscribe

[Anshul Gupta]

unsubscribe

[Sainath Palla]

unsubscribe

unsubscribe

[Sainath Palla]

unsubscribe

[William R]

unsubscribe