Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-29 Thread Cody Koeninger
I don't see anything obvious. If the slowness is correlated with the errors you're seeing, I'd start looking at what's going on with kafka or your network. On Mon, Aug 28, 2017 at 7:06 PM, swetha kasireddy wrote: > Hi Cody, > > Following is the way that I am consuming data for a 60 second batch

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
Hi Cody, Following is the way that I am consuming data for a 60 second batch. Do you see anything that is wrong with the way the data is getting consumed that can cause slowness in performance? val kafkaParams = Map[String, Object]( "bootstrap.servers" -> kafkaBrokers, "key.deseriali

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
There is no difference in performance even with Cache being enabled. On Mon, Aug 28, 2017 at 11:27 AM, swetha kasireddy < swethakasire...@gmail.com> wrote: > There is no difference in performance even with Cache being disabled. > > On Mon, Aug 28, 2017 at 7:43 AM, Cody Koeninger > wrote: > >> So

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
auto.offset.reset" -> "latest", > >> > "heartbeat.interval.ms" -> Integer.valueOf(2), > >> > "session.timeout.ms" -> Integer.valueOf(6), > >> > "request.timeout.ms" -> Integer.valueOf(9), > >&

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread Cody Koeninger
teger.valueOf(2), >> > "session.timeout.ms" -> Integer.valueOf(6), >> > "request.timeout.ms" -> Integer.valueOf(9), >> > "enable.auto.commit" -> (false: java.lang.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled&quo

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread swetha kasireddy
gt; "request.timeout.ms" -> Integer.valueOf(9), >> > "enable.auto.commit" -> (false: java.lang.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled" -> "false", >> > "group.id" -> "test1

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread swetha kasireddy
roup.id" -> "test1" > > ) > > > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > > ssc, > > LocationStrategies.PreferConsistent, > > ConsumerStrategies.Subscribe[String, String](topicsSet, >

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread Cody Koeninger
est1" > ) > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > ssc, > LocationStrategies.PreferConsistent, > ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams) > ) > > > &g

Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread SRK
nabled" -> "false", "group.id" -> "test1" ) val hubbleStream = KafkaUtils.createDirectStream[String, String]( ssc, LocationStrategies.PreferConsistent, ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams)