Andris, I would like to understand where is -- and if it exists -- a bottleneck. Could you please navigate to the ntopng monitored interface page (if_stats.lua)? Is there any packet drop? Are you using pf_ring? Please, enclose a screenshot of the interface page.
Simone On Tue, Jul 5, 2016 at 8:12 PM, Andris Bjornson <and...@everylayer.com> wrote: > Hi Simone, > > Thanks for your response. > > Here is /etc/ntopng/ntopng.conf: > > --pid=/var/run/ntopng.pid > --community > --daemon > --dns-mode=3 > --user=root > --interface=eth1 > --local-networks="10.20.0.0/14, 10.40.0.0/14, 10.60.0.0/14, 10.80.0.0/13, > 10.120.0.0/14" > --dump-flows="es;flows;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;"; > > > I agree that this feels like a capacity issue somewhere, but I'm having a > hard time figuring out where and what to do about it. The machine that > this is running on is: > > > - RAM: 16 GB > - CPU: Intel Xeon L5520 Quad-Core 2.26GHz > - Storage: 2x 7200 RPM, 1TB hard drives in RAID1 > > > > The same machine is running the elasticsearch node that ntopng is trying > to write to. > > However, what seems puzzling, is that none of the metrics of the machine > indicate the machine is over-taxed. > > Thanks again! > > > >> ------------------------------ >> >> Message: 2 >> Date: Tue, 5 Jul 2016 10:25:03 +0200 >> From: Simone Mainardi <maina...@ntop.org> >> To: n...@unipi.it >> Cc: ntop@listgateway.unipi.it >> Subject: Re: [Ntop] ntopng -> elasticsearch - dropped flows >> Message-ID: >> < >> cajcxkcbcdtrgneupxhvycehmn-hnrgauz0jqxa9qzdndbol...@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi, >> >> Could you please share ntopng configuration used? I think your setup >> doesn't allow ntopng to be quick enough. Remember that there is one thread >> per monitored interface and that thread has to >> 1. capture packets / receive flows >> 2. handle them >> 3. export to ES >> >> >> Simone >> >> On Wed, Jun 29, 2016 at 12:33 AM, Andris Bjornson <and...@everylayer.com> >> wrote: >> >> > Hello, >> > >> > Hoping to find a little help here after unsuccessfully googling quite a >> > bit. >> > >> > I've successfully setup the latest dev version of ntopng to dump flows >> > using --dump-flows into elasticsearch. Elasticsearch is on the same >> > machine as ntopng. My total traffic volume to process is about ~130Mbps >> > peak. >> > >> > It works very well, but i think i am losing a lot of flows in the export >> > process. >> > >> > My ntopng log file is rapidly filled (at the rate of ~600 per second) >> like >> > the following: >> > >> > 29/Jun/2016 01:26:02 [ElasticSearch.cpp:64] WARNING: [ES] Message >> dropped. >> > Total messages dropped: 2799026 >> > >> > However, I don't think this is an elasticsearch capacity problem, >> because >> > I am not seeing the errors in elasticsearch.log that would normally >> > accompany elasticsearch running out of capacity. I'm monitoring iostat, >> > system load, and elasticsearch performance via marvel - and those all >> look >> > good. >> > >> > I'm not sure where to look next for more information about what might be >> > causing the "message dropped" logs. >> > >> > Any help much appreciated! >> > >> > Andris >> > >> > >> > >> > --- >> > Andris Bjornson | EveryLayer <http://www.everylayer.com/> >> > skype: andris.bjornson >> > > > > _______________________________________________ > Ntop mailing list > Ntop@listgateway.unipi.it > http://listgateway.unipi.it/mailman/listinfo/ntop >
_______________________________________________ Ntop mailing list Ntop@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop
