Re: Can we delete topic in kafka

[Eduardo Costa Alfaia]

Hi,
It’s better creating a script that delete the kafka folder where exist the 
kafka topic and after create it again if need.

BR


Eduardo Costa Alfaia
Ph.D. Student in Telecommunications Engineering
Università degli Studi di Brescia
Tel: +39 3209333018








On 5/11/16, 09:48, "Snehalata Nagaje" <snehalata.nag...@harbingergroup.com> 
wrote:

>
>
>Hi , 
>
>Can we delete certain topic in kafka? 
>
>I have deleted using command 
>
>./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181 
>
>It says topic marked as deletion, but it does not actually delete topic. 
>
>Thanks, 
>Snehalata 


-- 

Informativa sulla Privacy: http://www.unibs.it/node/8155

Queue Full

[Eduardo Costa Alfaia]

Hi Guys,

How could I solving this problem?

% Failed to produce message: Local: Queue full
% Failed to produce message: Local: Queue full

Thanks

-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Queue Full

[Eduardo Costa Alfaia]

Hi Magnus
I think this answer
c) producing messages at a higher rate than the network or broker can
handle
How could I manager this?


> On 26 Oct 2015, at 17:45, Magnus Edenhill  wrote:
> 
> c) producing messages at a higher rate than the network or broker can
> handle


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Doubts Kafka

[Eduardo Costa Alfaia]

Hi Guys,

I have some doubts about the Kafka, the first is Why sometimes the applications 
prefer to connect to zookeeper instead brokers? Connecting to zookeeper could 
create an overhead, because we are inserting other element between producer and 
consumer. Another question is about the information sent by producer, in my 
tests the producer send the messages to brokers and a few minutes my HardDisk 
is full (my harddisk has 250GB), is there something to do in the configuration 
to minimize this?

Thanks 
-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

KafkaWordCount

[Eduardo Costa Alfaia]

Hi Guys,

I would like to put in the kafkawordcount scala code the kafka parameter:  val 
kafkaParams = Map(“fetch.message.max.bytes” - “400”). I’ve put this 
variable like this

val KafkaDStreams = (1 to numStreams) map {_ = 

  
KafkaUtils.createStream(ssc, kafkaParams, zkQuorum, group, 
topicpMap).map(_._2)


However I’ve gotten these erros:

 (jssc: org.apache.spark.streaming.api.java.JavaStreamingContext,zkQuorum: 
String,groupId: String,topics: jav  
   a.util.Map[String,Integer],storageLevel: 
org.apache.spark.storage.StorageLevel)org.apache.spark.streaming.api.java.Jav   
 
aPairReceiverInputDStream[String,String] and  

   
[error]   (ssc: org.apache.spark.streaming.StreamingContext,zkQuorum: 
String,groupId: String,topics: scala.collection.
 
immutable.Map[String,Int],storageLevel: 
org.apache.spark.storage.StorageLevel)org.apache.spark.streaming.dstream.Recei  
   
verInputDStream[(String, String)]

Thanks
-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Issue size message

[Eduardo Costa Alfaia]

Hi All,
I am having an issue when using kafka with librdkafka. I've changed the 
message.max.bytes to 2MB in my server.properties config file, that is the size 
of my message, when I run the command line ./rdkafka_performance -C -t test -p 
0 -b computer49:9092, after consume some messages the consumer remain waiting 
something that don't arrive. My producer continues sending messages. Some idea?

% Using random seed 1421685059, verbosity level 1
% 214 messages and 1042835 bytes consumed in 20ms: 10518 msgs/s and 51.26 Mb/s, 
no compression
% 21788 messages and 106128192 bytes consumed in 1029ms: 21154 msgs/s and 
103.04 Mb/s, no compression
% 43151 messages and 210185259 bytes consumed in 2030ms: 21252 msgs/s and 
103.52 Mb/s, no compression
% 64512 messages and 314233575 bytes consumed in 3031ms: 21280 msgs/s and 
103.66 Mb/s, no compression
% 86088 messages and 419328692 bytes consumed in 4039ms: 21313 msgs/s and 
103.82 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 5719ms: 17571 msgs/s and 
85.67 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 6720ms: 14955 msgs/s and 
72.92 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 7720ms: 13018 msgs/s and 
63.47 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 8720ms: 11524 msgs/s and 
56.19 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 9720ms: 10339 msgs/s and 
50.41 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 10721ms: 9374 msgs/s and 
45.71 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 11721ms: 8574 msgs/s and 
41.81 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 12721ms: 7900 msgs/s and 
38.52 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 13721ms: 7324 msgs/s and 
35.71 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 14721ms: 6826 msgs/s and 
33.29 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 15722ms: 6392 msgs/s and 
31.17 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 16722ms: 6010 msgs/s and 
29.30 Mb/s, no 



The software when consume all offset send me the message:

% Consumer reached end of unibs.nec [0] message queue at offset 229790
RD_KAFKA_RESP_ERR__PARTITION_EOF: [-191]

However changed de message.max.bytes to 2MB I don’t receive the code from Kafka.

Anyone has some idea?

Thanks guys.
-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Issue size message

[Eduardo Costa Alfaia]

Hi guys,

Ok, I’ve proved this and it was fine.

Thanks

 On Jan 19, 2015, at 19:10, Joe Stein joe.st...@stealth.ly wrote:
 
 If you increase the size of the messages for producing then you **MUST** also
 change *replica.fetch.max.bytes i*n the broker* server.properties *otherwise
 none of your replicas will be able to fetch from the leader and they will
 all fall out of the ISR. You also then need to change your consumers
 *fetch.message.max.bytes* in your consumers properties (whoever that might
 be configured for your specific consumer being used) so that they can read
 that data otherwise you won't see messages downstream.
 
 /***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /
 
 On Mon, Jan 19, 2015 at 1:03 PM, Magnus Edenhill mag...@edenhill.se wrote:
 
 (duplicating the github answer for reference)
 
 Hi Eduardo,
 
 the default maximum fetch size is 1 Meg which means your 2 Meg messages
 will not fit the fetch request.
 Try increasing it by appending -X fetch.message.max.bytes=400 to your
 command line.
 
 Regards,
 Magnus
 
 
 2015-01-19 17:52 GMT+01:00 Eduardo Costa Alfaia e.costaalf...@unibs.it:
 
 Hi All,
 I am having an issue when using kafka with librdkafka. I've changed the
 message.max.bytes to 2MB in my server.properties config file, that is the
 size of my message, when I run the command line ./rdkafka_performance -C
 -t
 test -p 0 -b computer49:9092, after consume some messages the consumer
 remain waiting something that don't arrive. My producer continues sending
 messages. Some idea?
 
 % Using random seed 1421685059, verbosity level 1
 % 214 messages and 1042835 bytes consumed in 20ms: 10518 msgs/s and 51.26
 Mb/s, no compression
 % 21788 messages and 106128192 bytes consumed in 1029ms: 21154 msgs/s and
 103.04 Mb/s, no compression
 % 43151 messages and 210185259 bytes consumed in 2030ms: 21252 msgs/s and
 103.52 Mb/s, no compression
 % 64512 messages and 314233575 bytes consumed in 3031ms: 21280 msgs/s and
 103.66 Mb/s, no compression
 % 86088 messages and 419328692 bytes consumed in 4039ms: 21313 msgs/s and
 103.82 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 5719ms: 17571 msgs/s
 and
 85.67 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 6720ms: 14955 msgs/s
 and
 72.92 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 7720ms: 13018 msgs/s
 and
 63.47 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 8720ms: 11524 msgs/s
 and
 56.19 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 9720ms: 10339 msgs/s
 and
 50.41 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 10721ms: 9374 msgs/s
 and
 45.71 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 11721ms: 8574 msgs/s
 and
 41.81 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 12721ms: 7900 msgs/s
 and
 38.52 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 13721ms: 7324 msgs/s
 and
 35.71 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 14721ms: 6826 msgs/s
 and
 33.29 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 15722ms: 6392 msgs/s
 and
 31.17 Mb/s, no compression
 % 100504 messages and 490022646 bytes consumed in 16722ms: 6010 msgs/s
 and
 29.30 Mb/s, no
 
 
 
 The software when consume all offset send me the message:
 
 % Consumer reached end of unibs.nec [0] message queue at offset 229790
 RD_KAFKA_RESP_ERR__PARTITION_EOF: [-191]
 
 However changed de message.max.bytes to 2MB I don’t receive the code from
 Kafka.
 
 Anyone has some idea?
 
 Thanks guys.
 --
 Informativa sulla Privacy: http://www.unibs.it/node/8155
 
 


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Information

[Eduardo Costa Alfaia]

Hi Guys

Anyone could explain me this information?

208K), 0.0086120 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 
2014-11-06T12:20:55.673+0100: 1256.382: [GC2014-11-06T12:20:55.674+0100: 
1256.382: [ParNew: 551115K-2816K(613440K), 0.0204130 secs] 
560218K-13933K(4126208K), 0.0205130 secs] [Times: user=0.09 sys=0.01, 
real=0.02 secs] 
2014-11-06T12:21:03.372+0100: 1264.080: [GC2014-11-06T12:21:03.372+0100: 
1264.080: [ParNew: 547827K-1047K(613440K), 0.0073880 secs] 
558944K-12473K(4126208K), 0.0074770 secs] [Times: user=0.06 sys=0.00, 
real=0.00 secs] 
2014-11-06T12:21:10.416+0100: 1271.124: [GC2014-11-06T12:21:10.416+0100: 
1271.124: [ParNew: 545782K-2266K(613440K), 0.0069530 secs] 
557208K-13836K(4126208K), 0.0070420 secs] [Times: user=0.05 sys=0.00, 
real=0.01 secs] 
2014-11-06T12:21:18.307+0100: 1279.015: [GC2014-11-06T12:21:18.307+0100: 
1279.015: [ParNew: 546921K-2156K(613440K), 0.0071050 secs] 
558491K-13855K(4126208K), 0.0071900 secs] [Times: user=0.06 sys=0.00, 
real=0.01 secs] 
2014-11-06T12:21:26.394+0100: 1287.102: [GC2014-11-06T12:21:26.394+0100: 
1287.102: [ParNew: 546237K-3125K(613440K), 0.0071260 secs] 
557936K-14940K(4126208K), 0.0072170 secs] [Times: user=0.05 sys=0.00, 
real=0.00 secs] 
2014-11-06T12:21:33.913+0100: 1294.621: [GC2014-11-06T12:21:33.913+0100: 
1294.621: [ParNew: 547726K-2452K(613440K), 0.0070220 secs] 559541K-14367K(412

Thanks
-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Consumer and Producer configs

[Eduardo Costa Alfaia]

Hi Guys,

How could I use the Consumer and Producer configs in my Kafka environment?

Thanks 

-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Spark and Kafka

[Eduardo Costa Alfaia]

Hi Guys,

I am doing some tests with Spark Streaming and Kafka, but I have seen something 
strange, I have modified the JavaKafkaWordCount to use ReducebyKeyandWindow and 
to print in the screen the accumulated numbers of the words, in the beginning 
spark works very well in each interaction the numbers of the words increase but 
after 12 a 13 sec the results repeats continually. 

My program producer remain sending the words toward the kafka.

Does anyone have any idea about this?


---
Time: 1415272266000 ms
---
(accompanied
them,6)
(merrier,5)
(it
possessed,5)
(the
treacherous,5)
(Quite,12)
(offer,273)
(rabble,58)
(exchanging,16)
(Genoa,18)
(merchant,41)
...
---
Time: 1415272267000 ms
---
(accompanied
them,12)
(merrier,12)
(it
possessed,12)
(the
treacherous,11)
(Quite,24)
(offer,602)
(rabble,132)
(exchanging,35)
(Genoa,36)
(merchant,84)
...
---
Time: 1415272268000 ms
---
(accompanied
them,17)
(merrier,18)
(it
possessed,17)
(the
treacherous,17)
(Quite,35)
(offer,889)
(rabble,192)
(the
bed,1)
(exchanging,51)
(Genoa,54)
...
---
Time: 1415272269000 ms
---
(accompanied
them,17)
(merrier,18)
(it
possessed,17)
(the
treacherous,17)
(Quite,35)
(offer,889)
(rabble,192)
(the
bed,1)
(exchanging,51)
(Genoa,54)
...

---
Time: 141527227 ms
---
(accompanied
them,17)
(merrier,18)
(it
possessed,17)
(the
treacherous,17)
(Quite,35)
(offer,889)
(rabble,192)
(the
bed,1)
(exchanging,51)
(Genoa,54)
...


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Spark and Kafka

[Eduardo Costa Alfaia]

This is my window:

reduceByKeyAndWindow(
   new Function2Integer, Integer, Integer() {
@Override
 public Integer call(Integer i1, Integer i2) { return i1 + i2; }
   },
   new Function2Integer, Integer, Integer() {
 public Integer call(Integer i1, Integer i2) { return i1 - i2; }
   },
   new Duration(60 * 5 * 1000),
   new Duration(1 * 1000)
 );

 On Nov 6, 2014, at 18:37, Gwen Shapira gshap...@cloudera.com wrote:
 
 What's the window size? If the window is around 10 seconds and you are
 sending data at very stable rate, this is expected.
 
 
 
 On Thu, Nov 6, 2014 at 9:32 AM, Eduardo Costa Alfaia e.costaalf...@unibs.it
 wrote:
 
 Hi Guys,
 
 I am doing some tests with Spark Streaming and Kafka, but I have seen
 something strange, I have modified the JavaKafkaWordCount to use
 ReducebyKeyandWindow and to print in the screen the accumulated numbers of
 the words, in the beginning spark works very well in each interaction the
 numbers of the words increase but after 12 a 13 sec the results repeats
 continually.
 
 My program producer remain sending the words toward the kafka.
 
 Does anyone have any idea about this?
 
 
 ---
 Time: 1415272266000 ms
 ---
 (accompanied
 them,6)
 (merrier,5)
 (it
 possessed,5)
 (the
 treacherous,5)
 (Quite,12)
 (offer,273)
 (rabble,58)
 (exchanging,16)
 (Genoa,18)
 (merchant,41)
 ...
 ---
 Time: 1415272267000 ms
 ---
 (accompanied
 them,12)
 (merrier,12)
 (it
 possessed,12)
 (the
 treacherous,11)
 (Quite,24)
 (offer,602)
 (rabble,132)
 (exchanging,35)
 (Genoa,36)
 (merchant,84)
 ...
 ---
 Time: 1415272268000 ms
 ---
 (accompanied
 them,17)
 (merrier,18)
 (it
 possessed,17)
 (the
 treacherous,17)
 (Quite,35)
 (offer,889)
 (rabble,192)
 (the
 bed,1)
 (exchanging,51)
 (Genoa,54)
 ...
 ---
 Time: 1415272269000 ms
 ---
 (accompanied
 them,17)
 (merrier,18)
 (it
 possessed,17)
 (the
 treacherous,17)
 (Quite,35)
 (offer,889)
 (rabble,192)
 (the
 bed,1)
 (exchanging,51)
 (Genoa,54)
 ...
 
 ---
 Time: 141527227 ms
 ---
 (accompanied
 them,17)
 (merrier,18)
 (it
 possessed,17)
 (the
 treacherous,17)
 (Quite,35)
 (offer,889)
 (rabble,192)
 (the
 bed,1)
 (exchanging,51)
 (Genoa,54)
 ...
 
 
 --
 Informativa sulla Privacy: http://www.unibs.it/node/8155
 


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Spark Kafka Performance

[Eduardo Costa Alfaia]

Hi Bhavesh

I will collect the  dump and I will send for you.

I am using a program that I have caught here  
https://github.com/edenhill/librdkafka/tree/master/examples 
https://github.com/edenhill/librdkafka/tree/master/examples and I have 
changed to meet my tests. I have attached the files.






 On Nov 5, 2014, at 04:45, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote:
 
 Hi Eduardo,
 
 Can you please take thread dump and see if there are blocking issues on
 producer side ?  Do you have single instance of Producers and Multiple
 treads ?
 
 Are you using Scala Producer or New Java Producer ?  Also, what is your
 producer property ?
 
 
 Thanks,
 
 Bhavesh
 
 On Tue, Nov 4, 2014 at 12:40 AM, Eduardo Alfaia e.costaalf...@unibs.it
 wrote:
 
 Hi Gwen,
 I have changed the java code kafkawordcount to use reducebykeyandwindow in
 spark.
 
 - Messaggio originale -
 Da: Gwen Shapira gshap...@cloudera.com
 Inviato: ‎03/‎11/‎2014 21:08
 A: users@kafka.apache.org users@kafka.apache.org
 Cc: u...@spark.incubator.apache.org u...@spark.incubator.apache.org
 Oggetto: Re: Spark Kafka Performance
 
 Not sure about the throughput, but:
 
 I mean that the words counted in spark should grow up - The spark
 word-count example doesn't accumulate.
 It gets an RDD every n seconds and counts the words in that RDD. So we
 don't expect the count to go up.
 
 
 
 On Mon, Nov 3, 2014 at 6:57 AM, Eduardo Costa Alfaia 
 e.costaalf...@unibs.it
 wrote:
 
 Hi Guys,
 Anyone could explain me how to work Kafka with Spark, I am using the
 JavaKafkaWordCount.java like a test and the line command is:
 
 ./run-example org.apache.spark.streaming.examples.JavaKafkaWordCount
 spark://192.168.0.13:7077 computer49:2181 test-consumer-group unibs.it 3
 
 and like a producer I am using this command:
 
 rdkafka_cachesender -t unibs.nec -p 1 -b 192.168.0.46:9092 -f output.txt
 -l 100 -n 10
 
 
 rdkafka_cachesender is a program that was developed by me which send to
 kafka the output.txt’s content where -l is the length of each send(upper
 bound) and -n is the lines to send in a row. Bellow is the throughput
 calculated by the program:
 
 File is 2235755 bytes
 throughput (b/s) = 699751388
 throughput (b/s) = 723542382
 throughput (b/s) = 662989745
 throughput (b/s) = 505028200
 throughput (b/s) = 471263416
 throughput (b/s) = 446837266
 throughput (b/s) = 409856716
 throughput (b/s) = 373994467
 throughput (b/s) = 366343097
 throughput (b/s) = 373240017
 throughput (b/s) = 386139016
 throughput (b/s) = 373802209
 throughput (b/s) = 369308515
 throughput (b/s) = 366935820
 throughput (b/s) = 365175388
 throughput (b/s) = 362175419
 throughput (b/s) = 358356633
 throughput (b/s) = 357219124
 throughput (b/s) = 352174125
 throughput (b/s) = 348313093
 throughput (b/s) = 355099099
 throughput (b/s) = 348069777
 throughput (b/s) = 348478302
 throughput (b/s) = 340404276
 throughput (b/s) = 339876031
 throughput (b/s) = 339175102
 throughput (b/s) = 327555252
 throughput (b/s) = 324272374
 throughput (b/s) = 322479222
 throughput (b/s) = 319544906
 throughput (b/s) = 317201853
 throughput (b/s) = 317351399
 throughput (b/s) = 315027978
 throughput (b/s) = 313831014
 throughput (b/s) = 310050384
 throughput (b/s) = 307654601
 throughput (b/s) = 305707061
 throughput (b/s) = 307961102
 throughput (b/s) = 296898200
 throughput (b/s) = 296409904
 throughput (b/s) = 294609332
 throughput (b/s) = 293397843
 throughput (b/s) = 293194876
 throughput (b/s) = 291724886
 throughput (b/s) = 290031314
 throughput (b/s) = 289747022
 throughput (b/s) = 289299632
 
 The throughput goes down after some seconds and it does not maintain the
 performance like the initial values:
 
 throughput (b/s) = 699751388
 throughput (b/s) = 723542382
 throughput (b/s) = 662989745
 
 Another question is about spark, after I have started the spark line
 command after 15 sec spark continue to repeat the words counted, but my
 program continue to send words to kafka, so I mean that the words counted
 in spark should grow up. I have attached the log from spark.
 
 My Case is:
 
 ComputerA(Kafka_cachsesender) - ComputerB(Kakfa-Brokers-Zookeeper) -
 ComputerC (Spark)
 
 If I don’t explain very well send a reply to me.
 
 Thanks Guys
 --
 Informativa sulla Privacy: http://www.unibs.it/node/8155
 
 
 --
 Informativa sulla Privacy: http://www.unibs.it/node/8155
 


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Producer and Consumer properties

[Eduardo Costa Alfaia]

Hi Dudes,

I would like to know if the producer and consumer’s properties files into the 
config folder should be configured. I have configured only the 
server.properties, is it enough? I am doing some tests about the performance, 
for example network throughput my scenario is:

Like producer I am using this program in c:



Like consumer this:




1 Server (zookeeper + 3 Brokers (8 partitions and Replication factor 3))
24GB RAM
5.0TB Hard Disc
eth0: Broadcom NetXtreme II BCM5709 1000Base-T 


Exist a great difference of throughput between the producer and consumer, does 
someone have any ideia?

Results:

ProducerConsumer
throughput (b/s) = 301393419received = 4083875, throughput (b/s) = 5571423
throughput (b/s) = 424807283received = 7146741, throughput (b/s) = 8061556
throughput (b/s) = 445245606received = 13270522, throughput (b/s) = 12925199
throughput (b/s) = 466454739received = 16333527, throughput (b/s) = 13890292
throughput (b/s) = 442368081received = 18375214, throughput (b/s) = 13967440
throughput (b/s) = 436540119received = 20416859, throughput (b/s) = 14127520
throughput (b/s) = 427105440received = 24500066, throughput (b/s) = 15594622
throughput (b/s) = 426395933received = 27563023, throughput (b/s) = 16177493
throughput (b/s) = 409344029received = 34708625, throughput (b/s) = 18740726
throughput (b/s) = 403371185received = 37771189, throughput (b/s) = 17961816
throughput (b/s) = 403325568received = 39813038, throughput (b/s) = 17654058
throughput (b/s) = 397938415received = 47979107, throughput (b/s) = 19686322
throughput (b/s) = 393364006received = 53083307, throughput (b/s) = 20623441
throughput (b/s) = 387393832received = 57166558, throughput (b/s) = 21050531
throughput (b/s) = 380266372received = 59207558, throughput (b/s) = 20654404
throughput (b/s) = 376436729received = 62269998, throughput (b/s) = 20740363
throughput (b/s) = 377043675received = 65332901, throughput (b/s) = 20888135
throughput (b/s) = 368613683received = 67374558, throughput (b/s) = 20467503
throughput (b/s) = 370020865received = 71457763, throughput (b/s) = 20727773
throughput (b/s) = 373827848received = 73499480, throughput (b/s) = 20171583
throughput (b/s) = 369647040received = 75541289, throughput (b/s) = 19599155
throughput (b/s) = 363395680received = 80645776, throughput (b/s) = 20033582


Thanks Guys



-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Spark Kafka Performance

[Eduardo Costa Alfaia]

Hi Guys,
Anyone could explain me how to work Kafka with Spark, I am using the 
JavaKafkaWordCount.java like a test and the line command is:

./run-example org.apache.spark.streaming.examples.JavaKafkaWordCount 
spark://192.168.0.13:7077 computer49:2181 test-consumer-group unibs.it 3

and like a producer I am using this command:

rdkafka_cachesender -t unibs.nec -p 1 -b 192.168.0.46:9092 -f output.txt -l 100 
-n 10


rdkafka_cachesender is a program that was developed by me which send to kafka 
the output.txt’s content where -l is the length of each send(upper bound) and 
-n is the lines to send in a row. Bellow is the throughput calculated by the 
program:

File is 2235755 bytes
throughput (b/s) = 699751388
throughput (b/s) = 723542382
throughput (b/s) = 662989745
throughput (b/s) = 505028200
throughput (b/s) = 471263416
throughput (b/s) = 446837266
throughput (b/s) = 409856716
throughput (b/s) = 373994467
throughput (b/s) = 366343097
throughput (b/s) = 373240017
throughput (b/s) = 386139016
throughput (b/s) = 373802209
throughput (b/s) = 369308515
throughput (b/s) = 366935820
throughput (b/s) = 365175388
throughput (b/s) = 362175419
throughput (b/s) = 358356633
throughput (b/s) = 357219124
throughput (b/s) = 352174125
throughput (b/s) = 348313093
throughput (b/s) = 355099099
throughput (b/s) = 348069777
throughput (b/s) = 348478302
throughput (b/s) = 340404276
throughput (b/s) = 339876031
throughput (b/s) = 339175102
throughput (b/s) = 327555252
throughput (b/s) = 324272374
throughput (b/s) = 322479222
throughput (b/s) = 319544906
throughput (b/s) = 317201853
throughput (b/s) = 317351399
throughput (b/s) = 315027978
throughput (b/s) = 313831014
throughput (b/s) = 310050384
throughput (b/s) = 307654601
throughput (b/s) = 305707061
throughput (b/s) = 307961102
throughput (b/s) = 296898200
throughput (b/s) = 296409904
throughput (b/s) = 294609332
throughput (b/s) = 293397843
throughput (b/s) = 293194876
throughput (b/s) = 291724886
throughput (b/s) = 290031314
throughput (b/s) = 289747022
throughput (b/s) = 289299632

The throughput goes down after some seconds and it does not maintain the 
performance like the initial values:

throughput (b/s) = 699751388
throughput (b/s) = 723542382
throughput (b/s) = 662989745

Another question is about spark, after I have started the spark line command 
after 15 sec spark continue to repeat the words counted, but my program 
continue to send words to kafka, so I mean that the words counted in spark 
should grow up. I have attached the log from spark.
  
My Case is:

ComputerA(Kafka_cachsesender) - ComputerB(Kakfa-Brokers-Zookeeper) - 
ComputerC (Spark)
 
If I don’t explain very well send a reply to me.

Thanks Guys
-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Clean Kafka Queue

[Eduardo Costa Alfaia]

Hi Guys,

Is there a manner of cleaning  a kafka queue after that the consumer consume 
the messages?

Thanks 
-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Clean Kafka Queue

[Eduardo Costa Alfaia]

Ok guys,

Thanks by the help.

Regards
 On Oct 21, 2014, at 18:30, Joe Stein joe.st...@stealth.ly wrote:
 
 The concept of truncate topic comes up a lot.  I will add it as an item
 to https://issues.apache.org/jira/browse/KAFKA-1694
 
 It is a scary feature though, it might be best to wait until authorizations
 are in place before we release it.
 
 With 0.8.2 you can delete topics so at least you can start fresh easier.
 That should work in the mean time.  0.8.2-beta should be out this week :)
 
 /***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /
 
 On Tue, Oct 21, 2014 at 12:03 PM, Harsha ka...@harsha.io wrote:
 
 you can use log.retention.hours or log.retention.bytes to prune the log
 more info on that config here
 https://kafka.apache.org/08/configuration.html
 if you want to delete a message after the consumer processed a message
 there is no api for it.
 -Harsha
 
 
 On Tue, Oct 21, 2014, at 08:00 AM, Eduardo Costa Alfaia wrote:
 Hi Guys,
 
 Is there a manner of cleaning  a kafka queue after that the consumer
 consume the messages?
 
 Thanks
 --
 Informativa sulla Privacy: http://www.unibs.it/node/8155
 


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155