Hi Gwen, I have changed the java code kafkawordcount to use reducebykeyandwindow in spark.
----- Messaggio originale ----- Da: "Gwen Shapira" <gshap...@cloudera.com> Inviato: 03/11/2014 21:08 A: "us...@kafka.apache.org" <us...@kafka.apache.org> Cc: "u...@spark.incubator.apache.org" <u...@spark.incubator.apache.org> Oggetto: Re: Spark Kafka Performance Not sure about the throughput, but: "I mean that the words counted in spark should grow up" - The spark word-count example doesn't accumulate. It gets an RDD every n seconds and counts the words in that RDD. So we don't expect the count to go up. On Mon, Nov 3, 2014 at 6:57 AM, Eduardo Costa Alfaia <e.costaalf...@unibs.it > wrote: > Hi Guys, > Anyone could explain me how to work Kafka with Spark, I am using the > JavaKafkaWordCount.java like a test and the line command is: > > ./run-example org.apache.spark.streaming.examples.JavaKafkaWordCount > spark://192.168.0.13:7077 computer49:2181 test-consumer-group unibs.it 3 > > and like a producer I am using this command: > > rdkafka_cachesender -t unibs.nec -p 1 -b 192.168.0.46:9092 -f output.txt > -l 100 -n 10 > > > rdkafka_cachesender is a program that was developed by me which send to > kafka the output.txt’s content where -l is the length of each send(upper > bound) and -n is the lines to send in a row. Bellow is the throughput > calculated by the program: > > File is 2235755 bytes > throughput (b/s) = 699751388 > throughput (b/s) = 723542382 > throughput (b/s) = 662989745 > throughput (b/s) = 505028200 > throughput (b/s) = 471263416 > throughput (b/s) = 446837266 > throughput (b/s) = 409856716 > throughput (b/s) = 373994467 > throughput (b/s) = 366343097 > throughput (b/s) = 373240017 > throughput (b/s) = 386139016 > throughput (b/s) = 373802209 > throughput (b/s) = 369308515 > throughput (b/s) = 366935820 > throughput (b/s) = 365175388 > throughput (b/s) = 362175419 > throughput (b/s) = 358356633 > throughput (b/s) = 357219124 > throughput (b/s) = 352174125 > throughput (b/s) = 348313093 > throughput (b/s) = 355099099 > throughput (b/s) = 348069777 > throughput (b/s) = 348478302 > throughput (b/s) = 340404276 > throughput (b/s) = 339876031 > throughput (b/s) = 339175102 > throughput (b/s) = 327555252 > throughput (b/s) = 324272374 > throughput (b/s) = 322479222 > throughput (b/s) = 319544906 > throughput (b/s) = 317201853 > throughput (b/s) = 317351399 > throughput (b/s) = 315027978 > throughput (b/s) = 313831014 > throughput (b/s) = 310050384 > throughput (b/s) = 307654601 > throughput (b/s) = 305707061 > throughput (b/s) = 307961102 > throughput (b/s) = 296898200 > throughput (b/s) = 296409904 > throughput (b/s) = 294609332 > throughput (b/s) = 293397843 > throughput (b/s) = 293194876 > throughput (b/s) = 291724886 > throughput (b/s) = 290031314 > throughput (b/s) = 289747022 > throughput (b/s) = 289299632 > > The throughput goes down after some seconds and it does not maintain the > performance like the initial values: > > throughput (b/s) = 699751388 > throughput (b/s) = 723542382 > throughput (b/s) = 662989745 > > Another question is about spark, after I have started the spark line > command after 15 sec spark continue to repeat the words counted, but my > program continue to send words to kafka, so I mean that the words counted > in spark should grow up. I have attached the log from spark. > > My Case is: > > ComputerA(Kafka_cachsesender) -> ComputerB(Kakfa-Brokers-Zookeeper) -> > ComputerC (Spark) > > If I don’t explain very well send a reply to me. > > Thanks Guys > -- > Informativa sulla Privacy: http://www.unibs.it/node/8155 > -- Informativa sulla Privacy: http://www.unibs.it/node/8155
