Hi, No we are not creating any thread for kafka DStream however, we have a single thread for refreshing a resource cache on driver, but that is totally separate to this connection.
On Mon, Sep 18, 2017 at 12:29 AM kant kodali <kanth...@gmail.com> wrote: > Are you creating threads in your application? > > On Sun, Sep 17, 2017 at 7:48 AM, HARSH TAKKAR <takkarha...@gmail.com> > wrote: > >> >> Hi >> >> I am using spark 2.1.0 with scala 2.11.8, and while iterating over the >> partitions of each rdd in a dStream formed using KafkaUtils, i am getting >> the below exception, please suggest a fix. >> >> I have following config >> >> kafka : >> enable.auto.commit:"true", >> auto.commit.interval.ms:"1000", >> session.timeout.ms:"30000", >> >> Spark: >> >> spark.streaming.backpressure.enabled=true >> >> spark.streaming.kafka.maxRatePerPartition=200 >> >> >> Exception in task 0.2 in stage 3236.0 (TID 77795) >> java.util.ConcurrentModificationException: KafkaConsumer is not safe for >> multi-threaded access >> >> -- >> Kind Regards >> Harsh >> > >
