Re: Spark Parquet write OOM

Hi Anil, before jumping to the quick symptomatic fix, can we try to understand the issues? 1. What is the version of SPARK you are using? 2. Are you doing a lot of in-memory transformations like adding columns, or running joins, or UDFs thus increasing the size of the data before writing out? 3.

Re: {EXT} Re: Spark Parquet write OOM

Hi Gourav and Yang Thanks for the response. Please find the answers below. 1. What is the version of SPARK you are using? [AD] : Spark 2.4.7 (EMR 5.33.1) 2. Are you doing a lot of in-memory transformations like adding columns, or running joins, or UDFs thus increasing the size of the data

Re: {EXT} Re: Spark Parquet write OOM

2nd attempt.. Any suggestions to troubleshoot and fix the problem ? thanks in advance. Regards, Anil From: Anil Dasari Date: Wednesday, March 2, 2022 at 7:00 AM To: Gourav Sengupta , Yang,Jie(INF) Cc: user@spark.apache.org Subject: Re: {EXT} Re: Spark Parquet write OOM Hi Gourav and Yang

Re: unsubscribe

unsubscribe On Thu, Mar 3, 2022 at 5:44 AM Basavaraj wrote: > unsubscribe

unsubscribe

unsubscribe smime.p7s Description: S/MIME cryptographic signature