> On 30 Sep 2015, at 17:10, Florian Pieper <[email protected]> wrote: > > Ok, got it. Thanks! > > So we currently run flume with a HA based kafka sink > > e.g.: > > a1.sinks.k1.brokerList = host1:9092,host2:9092,host3:9092
This means that I will speak with the first broker, then if down move to the second etc, send data in round robin, on in fan-out? Luca > > https://flume.apache.org/FlumeUserGuide.html#kafka-sink > <https://flume.apache.org/FlumeUserGuide.html#kafka-sink> > > Thanks, > > Florian > > > Von: <[email protected] > <mailto:[email protected]>> im Auftrag von Luca Deri > Antworten an: "[email protected] > <mailto:[email protected]>" > Datum: Mittwoch, 30. September 2015 17:01 > An: "[email protected] <mailto:[email protected]>" > Betreff: Re: [Ntop-misc] nprobe and kafka? > > Hi Florian >> On 30 Sep 2015, at 16:34, Florian Pieper <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thanks Luca! >> >> Just tried the latest nightly snapshot. >> >> This is really looking promising, we have 16800 flows/s now instead of 5xx >> /s (for whatever reason). >> > Great > >> Looking forward to production release of the kafka integration. > The kafka part is stable: we’re working at the Lua part and as soon as this > is completed we’ll make a new stable. Don’t be afraid of the name: nprobe-dev > is stable > > >> >> PS: Any plans to be able to use more than 1 kafka broker and also a port ? > Please explain what you would like > > Cheers Luca > >> >> >> Von: <[email protected] >> <mailto:[email protected]>> im Auftrag von Luca Deri >> Antworten an: "[email protected] >> <mailto:[email protected]>" >> Datum: Mittwoch, 30. September 2015 15:00 >> An: "[email protected] <mailto:[email protected]>" >> Betreff: Re: [Ntop-misc] nprobe and kafka? >> >> Florian >> the kafka integration is already implemented in the development package. >> >> deri@ubuntu14 202> ./nprobe -h|grep kafka >> --kafka <brokerIP>:<topic>:<partition> | Send flows to the >> specified Apache Kafka broker. Example --kafka localhost:test:1 >> >> >> On 09/30/2015 02:21 PM, Florian Pieper wrote: >>> Hi Luca, >>> >>> I am currently trying to use nprobe in collector mode and forward JSON >>> formatted flows to flume (netcat) >>> Which in return forwards the data to kafka. >>> >>> The export queue in nprobe is filling up (already increased -w) and my >>> guess is that there is some kind of >>> Bottleneck between nprobe and flume. >> >> if it increases it is because the collector is slow and -w won't help >> >>> >>> You are saying that there is some development going on for a direct kafka >>> integration in nprobe? >>> Can you let me know where I can download this special version and when we >>> can expect it to be in the >>> Production branch? >> It is production ready but it is in the dev packages. Please test it and >> report us bugs. >> >> Regards Luca >> >>> >>> Thanks, >>> >>> Florian >>> >>> From: < >>> <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>> on behalf of Luca Deri >>> <[email protected] <mailto:[email protected]>> >>> Reply-To: " >>> <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>" <[email protected] >>> <mailto:[email protected]>> >>> Date: Tuesday, September 1, 2015 at 10:52 PM >>> To: " <mailto:[email protected]>[email protected] >>> <mailto:[email protected]>" <[email protected] >>> <mailto:[email protected]>> >>> Subject: Re: [Ntop-misc] nprobe and kafka? >>> >>> Manny >>> we have added kafka support on one of our development prototypes so movign >>> to the official nprobe should not be too difficult. The performance is >>> similar to the ZMQ or elasticsearch implementation, so considered the JSON >>> conversion is at least 1k flows/sec >>> >>> Luca >>> >>>> On 01 Sep 2015, at 23:20, Manny Veloso <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi! >>>> >>>> I’m looking to use nprobe as a bridge into kafka. In the splunk app nprobe >>>> just sends data into splunk. Is that basically the same configuration as a >>>> kafka install? >>>> >>>> Also, what kind of throughput can I expect out of nprobe? >>>> -- >>>> Manny Veloso >>>> Sr. Solutions Engineer >>>> Smartrg.com >>>> <http://smartrg.com/>_______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> <mailto:[email protected]>http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
