Re: Out of memory Error when using Collection Accumulator Spark 2.2

what is your driver memory? Thanks, Naresh www.linkedin.com/in/naresh-dulam http://hadoopandspark.blogspot.com/ On Mon, Feb 26, 2018 at 3:45 AM, Patrick wrote: > Hi, > > We were getting OOM error when we are accumulating the results of each > worker. We were trying to

Out of memory Error when using Collection Accumulator Spark 2.2

Hi, We were getting OOM error when we are accumulating the results of each worker. We were trying to avoid collecting data to driver node instead used accumulator as per below code snippet, Is there any spark config to set the accumulator settings Or am i doing the wrong way to collect the huge