On Fri, Nov 20, 2020, 8:25 AM Amit Sharma <resolve...@gmail.com> wrote:
> please help. > > > Thanks > Amit > > On Mon, Nov 9, 2020 at 4:18 PM Amit Sharma <resolve...@gmail.com> wrote: > >> Please find below the exact exception >> >> Exception in thread "streaming-job-executor-3" >> java.lang.OutOfMemoryError: Java heap space >> at java.util.Arrays.copyOf(Arrays.java:3332) >> at >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) >> at >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) >> at java.lang.StringBuilder.append(StringBuilder.java:136) >> at >> scala.StringContext.standardInterpolator(StringContext.scala:126) >> at scala.StringContext.s(StringContext.scala:95) >> at sparkStreaming.TRReview.getTRReviews(TRReview.scala:307) >> at >> sparkStreaming.KafkaListener$$anonfun$1$$anonfun$apply$1$$anonfun$3.apply(KafkaListener.scala:154) >> at >> sparkStreaming.KafkaListener$$anonfun$1$$anonfun$apply$1$$anonfun$3.apply(KafkaListener.scala:138) >> at scala.util.Success$$anonfun$map$1.apply(Try.scala:237) >> at scala.util.Try$.apply(Try.scala:192) >> at scala.util.Success.map(Try.scala:237) >> >> On Sun, Nov 8, 2020 at 1:35 PM Amit Sharma <resolve...@gmail.com> wrote: >> >>> Hi , I am using 16 nodes spark cluster with below config >>> 1. Executor memory 8 GB >>> 2. 5 cores per executor >>> 3. Driver memory 12 GB. >>> >>> >>> We have streaming job. We do not see problem but sometimes we get >>> exception executor-1 heap memory issue. I am not understanding if data size >>> is same and this job receive a request and process it but suddenly it’s >>> start giving out of memory error . It will throw exception for 1 executor >>> then throw for other executor also and it stop processing the request. >>> >>> Thanks >>> Amit >>> >>
