Thanks Hemant, underlying data volume increased from 550GB to 690GB and now the same job doesn't succeed. I tried incrementing executor memory to 20G as well, still fails. I am running this in Databricks and start cluster with 20G assigned to spark.executor.memory property.
Also some more information on the job, I have about 4 window functions on this dataset before it gets written out. Any other ideas? Thanks, -Shraddha On Sun, Jan 5, 2020 at 11:06 PM hemant singh <hemant2...@gmail.com> wrote: > You can try increasing the executor memory, generally this error comes > when there is not enough memory in individual executors. > Job is getting completed may be because when tasks are re-scheduled it > would be going through. > > Thanks. > > On Mon, 6 Jan 2020 at 5:47 AM, Rishi Shah <rishishah.s...@gmail.com> > wrote: > >> Hello All, >> >> One of my jobs, keep getting into this situation where 100s of tasks keep >> failing with below error but job eventually completes. >> >> org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 16384 >> bytes of memory >> >> Could someone advice? >> >> -- >> Regards, >> >> Rishi Shah >> > -- Regards, Rishi Shah
