Hi Luca, As far as I am informed, the broker list is provided to retrieve the metadata information, and either broker can provide this information. The brokerList will be shuffled and it will go through each one of the hosts and makes a TopicMetadataRequest. If it succeed, it will return the broker. If not, the next broker is contacted.
So I can imagine that this part of the nprobe kafka exporter plugin could be implemented as round robin and if the contacted broker is failing the other ones provided should be contaced one after the other. What do you think? Thanks! Von: <[email protected]<mailto:[email protected]>> im Auftrag von Luca Deri Antworten an: "[email protected]<mailto:[email protected]>" Datum: Mittwoch, 30. September 2015 17:39 An: "[email protected]<mailto:[email protected]>" Betreff: Re: [Ntop-misc] nprobe and kafka? On 30 Sep 2015, at 17:10, Florian Pieper <[email protected]<mailto:[email protected]>> wrote: Ok, got it. Thanks! So we currently run flume with a HA based kafka sink e.g.: a1.sinks.k1.brokerList = host1:9092,host2:9092,host3:9092 This means that I will speak with the first broker, then if down move to the second etc, send data in round robin, on in fan-out? Luca https://flume.apache.org/FlumeUserGuide.html#kafka-sink Thanks, Florian Von: <[email protected]<mailto:[email protected]>> im Auftrag von Luca Deri Antworten an: "[email protected]<mailto:[email protected]>" Datum: Mittwoch, 30. September 2015 17:01 An: "[email protected]<mailto:[email protected]>" Betreff: Re: [Ntop-misc] nprobe and kafka? Hi Florian On 30 Sep 2015, at 16:34, Florian Pieper <[email protected]<mailto:[email protected]>> wrote: Thanks Luca! Just tried the latest nightly snapshot. This is really looking promising, we have 16800 flows/s now instead of 5xx /s (for whatever reason). Great Looking forward to production release of the kafka integration. The kafka part is stable: we’re working at the Lua part and as soon as this is completed we’ll make a new stable. Don’t be afraid of the name: nprobe-dev is stable PS: Any plans to be able to use more than 1 kafka broker and also a port ? Please explain what you would like Cheers Luca Von: <[email protected]<mailto:[email protected]>> im Auftrag von Luca Deri Antworten an: "[email protected]<mailto:[email protected]>" Datum: Mittwoch, 30. September 2015 15:00 An: "[email protected]<mailto:[email protected]>" Betreff: Re: [Ntop-misc] nprobe and kafka? Florian the kafka integration is already implemented in the development package. deri@ubuntu14 202> ./nprobe -h|grep kafka --kafka <brokerIP>:<topic>:<partition> | Send flows to the specified Apache Kafka broker. Example --kafka localhost:test:1 On 09/30/2015 02:21 PM, Florian Pieper wrote: Hi Luca, I am currently trying to use nprobe in collector mode and forward JSON formatted flows to flume (netcat) Which in return forwards the data to kafka. The export queue in nprobe is filling up (already increased -w) and my guess is that there is some kind of Bottleneck between nprobe and flume. if it increases it is because the collector is slow and -w won't help You are saying that there is some development going on for a direct kafka integration in nprobe? Can you let me know where I can download this special version and when we can expect it to be in the Production branch? It is production ready but it is in the dev packages. Please test it and report us bugs. Regards Luca Thanks, Florian From: <<mailto:[email protected]>[email protected]<mailto:[email protected]>> on behalf of Luca Deri <[email protected]<mailto:[email protected]>> Reply-To: "<mailto:[email protected]>[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, September 1, 2015 at 10:52 PM To: "<mailto:[email protected]>[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [Ntop-misc] nprobe and kafka? Manny we have added kafka support on one of our development prototypes so movign to the official nprobe should not be too difficult. The performance is similar to the ZMQ or elasticsearch implementation, so considered the JSON conversion is at least 1k flows/sec Luca On 01 Sep 2015, at 23:20, Manny Veloso <[email protected]<mailto:[email protected]>> wrote: Hi! I’m looking to use nprobe as a bridge into kafka. In the splunk app nprobe just sends data into splunk. Is that basically the same configuration as a kafka install? Also, what kind of throughput can I expect out of nprobe? -- Manny Veloso Sr. Solutions Engineer Smartrg.com<http://smartrg.com/> _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]>http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
