Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

I don't see anything obvious. If the slowness is correlated with the errors you're seeing, I'd start looking at what's going on with kafka or your network. On Mon, Aug 28, 2017 at 7:06 PM, swetha kasireddy wrote: > Hi Cody, > > Following is the way that I am

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

Hi Cody, Following is the way that I am consuming data for a 60 second batch. Do you see anything that is wrong with the way the data is getting consumed that can cause slowness in performance? val kafkaParams = Map[String, Object]( "bootstrap.servers" -> kafkaBrokers,

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

There is no difference in performance even with Cache being enabled. On Mon, Aug 28, 2017 at 11:27 AM, swetha kasireddy < swethakasire...@gmail.com> wrote: > There is no difference in performance even with Cache being disabled. > > On Mon, Aug 28, 2017 at 7:43 AM, Cody Koeninger

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

ueOf(6), > >> > "request.timeout.ms" -> Integer.valueOf(9), > >> > "enable.auto.commit" -> (false: java.lang.Boolean), > >> >

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

g.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled" -> "false", >> > "group.id" -> "test1" >> > ) >> > >> > val hubbleStream = KafkaUtils.createDirectStream[String, String]( >>

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

), >> > "session.timeout.ms" -> Integer.valueOf(6), >> > "request.timeout.ms" -> Integer.valueOf(9), >> > "enable.auto.commit" -> (false: java.lang.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled" -> &q

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

k.streaming.kafka.consumer.cache.enabled" -> "false", > > "group.id" -> "test1" > > ) > > > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > > ssc, > > LocationStrategies.Pref

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

-> "test1" > ) > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > ssc, > LocationStrategies.PreferConsistent, > ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams) > ) > > > &

Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

d" -> "false", "group.id" -> "test1" ) val hubbleStream = KafkaUtils.createDirectStream[String, String]( ssc, LocationStrategies.PreferConsistent, ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams)