You should paste some code. ConcurrentModificationException normally happens when you modify a list or any non-thread safe data structure while you are iterating over it.
On Sun, Sep 17, 2017 at 10:25 PM, HARSH TAKKAR <takkarha...@gmail.com> wrote: > Hi, > > No we are not creating any thread for kafka DStream > however, we have a single thread for refreshing a resource cache on > driver, but that is totally separate to this connection. > > On Mon, Sep 18, 2017 at 12:29 AM kant kodali <kanth...@gmail.com> wrote: > >> Are you creating threads in your application? >> >> On Sun, Sep 17, 2017 at 7:48 AM, HARSH TAKKAR <takkarha...@gmail.com> >> wrote: >> >>> >>> Hi >>> >>> I am using spark 2.1.0 with scala 2.11.8, and while iterating over the >>> partitions of each rdd in a dStream formed using KafkaUtils, i am getting >>> the below exception, please suggest a fix. >>> >>> I have following config >>> >>> kafka : >>> enable.auto.commit:"true", >>> auto.commit.interval.ms:"1000", >>> session.timeout.ms:"30000", >>> >>> Spark: >>> >>> spark.streaming.backpressure.enabled=true >>> >>> spark.streaming.kafka.maxRatePerPartition=200 >>> >>> >>> Exception in task 0.2 in stage 3236.0 (TID 77795) >>> java.util.ConcurrentModificationException: KafkaConsumer is not safe >>> for multi-threaded access >>> >>> -- >>> Kind Regards >>> Harsh >>> >> >>
