This pattern works. One note, thought: Use 'union' only if you need to group the data from all RDDs into one RDD for processing (like count distinct or need a groupby). If your process can be parallelized over every stream of incoming data, I suggest you just apply the required transformations on every dstream and avoid 'union' altogether.
-kr, Gerard. On Wed, Sep 10, 2014 at 8:17 PM, Tim Smith <secs...@gmail.com> wrote: > How are you creating your kafka streams in Spark? > > If you have 10 partitions for a topic, you can call "createStream" ten > times to create 10 parallel receivers/executors and then use "union" to > combine all the dStreams. > > > > On Wed, Sep 10, 2014 at 7:16 AM, richiesgr <richie...@gmail.com> wrote: > >> Hi (my previous post as been used by someone else) >> >> I'm building a application the read from kafka stream event. In production >> we've 5 consumers that share 10 partitions. >> But on spark streaming kafka only 1 worker act as a consumer then >> distribute >> the tasks to workers so I can have only 1 machine acting as consumer but I >> need more because only 1 consumer means Lags. >> >> Do you've any idea what I can do ? Another point is interresting the >> master >> is not loaded at all I can get up more than 10 % CPU >> >> I've tried to increase the queued.max.message.chunks on the kafka client >> to >> read more records thinking it'll speed up the read but I only get >> >> ERROR consumer.ConsumerFetcherThread: >> >> [ConsumerFetcherThread-SparkEC2_ip-10-138-59-194.ec2.internal-1410182950783-5c49c8e8-0-174167372], >> Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 73; >> ClientId: >> >> SparkEC2-ConsumerFetcherThread-SparkEC2_ip-10-138-59-194.ec2.internal-1410182950783-5c49c8e8-0-174167372; >> ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [IA2,7] -> >> PartitionFetchInfo(929838589,1048576),[IA2,6] -> >> PartitionFetchInfo(929515796,1048576),[IA2,9] -> >> PartitionFetchInfo(929577946,1048576),[IA2,8] -> >> PartitionFetchInfo(930751599,1048576),[IA2,2] -> >> PartitionFetchInfo(926457704,1048576),[IA2,5] -> >> PartitionFetchInfo(930774385,1048576),[IA2,0] -> >> PartitionFetchInfo(929913213,1048576),[IA2,3] -> >> PartitionFetchInfo(929268891,1048576),[IA2,4] -> >> PartitionFetchInfo(929949877,1048576),[IA2,1] -> >> PartitionFetchInfo(930063114,1048576) >> java.lang.OutOfMemoryError: Java heap space >> >> Is someone have ideas ? >> Thanks >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-scale-more-consumer-to-Kafka-stream-tp13883.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
