Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-29 Thread Cody Koeninger
onsumer strategy for creating the stream below. Any suggestions >>>> on how >>>> >> > to >>>> >> > run this with better performance would be of great help. >>>> >> > >>>> >> > java.lang.AssertionError: assertion fai

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
324027964 after polling for 12 >>> >> > >>> >> > val kafkaParams = Map[String, Object]( >>> >> > "bootstrap.servers" -> kafkaBrokers, >>> >> > "key.deserializer" ->

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
t;> > "heartbeat.interval.ms" -> Integer.valueOf(2), >> >> > "session.timeout.ms" -> Integer.valueOf(6), >> >> > "request.timeout.ms" -> Integer.valueOf(9), >> >> > "enab

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread swetha kasireddy
ueOf(6), > >> > "request.timeout.ms" -> Integer.valueOf(9), > >> > "enable.auto.commit" -> (false: java.lang.Boolean), > >> >

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-28 Thread Cody Koeninger
g.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled" -> "false", >> > "group.id" -> "test1" >> > ) >> > >> > val hubbleStream = KafkaUtils.createDirectStream[String, String]( >>

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread swetha kasireddy
), >> > "session.timeout.ms" -> Integer.valueOf(6), >> > "request.timeout.ms" -> Integer.valueOf(9), >> > "enable.auto.commit" -> (false: java.lang.Boolean), >> > "spark.streaming.kafka.consumer.cache.enabled" -> &q

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread swetha kasireddy
k.streaming.kafka.consumer.cache.enabled" -> "false", > > "group.id" -> "test1" > > ) > > > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > > ssc, > > LocationStrategies.Pref

Re: Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread Cody Koeninger
-> "test1" > ) > > val hubbleStream = KafkaUtils.createDirectStream[String, String]( > ssc, > LocationStrategies.PreferConsistent, > ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams) > ) > > > &

Slower performance while running Spark Kafka Direct Streaming with Kafka 10 cluster

2017-08-25 Thread SRK
d" -> "false", "group.id" -> "test1" ) val hubbleStream = KafkaUtils.createDirectStream[String, String]( ssc, LocationStrategies.PreferConsistent, ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams)

How to make sure that Spark Kafka Direct Streaming job maintains the state upon code deployment?

2017-06-27 Thread SRK
-to-make-sure-that-Spark-Kafka-Direct-Streaming-job-maintains-the-state-upon-code-deployment-tp28799.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr

Re: Spark Kafka Direct Streaming

2015-07-07 Thread Tathagata Das
checkpointing is required. Can I achieve this, i.e. enabling meta checkpointing and not the data checkpointing? Thanks Abhishek Patel -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Kafka-Direct-Streaming-tp23685.html Sent from the Apache

Spark Kafka Direct Streaming

2015-07-07 Thread abi_pat
: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Kafka-Direct-Streaming-tp23685.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org