That’s a much better cardinality (AS based) but it’s not the general case. Even if you want per-prefix information I’d argue that Influx would still not handle the load (~700k ^ 2 cardinality). For limited tag-sets it would do the trick.
I never did attempt to push it to Influx with some foresight that it’d be suboptimal for my ultimate use cases. I wanted a solution that could handle a wide range of use cases without having to worry about limits on tag-sets. I found Clickhouse able to do what I wanted in a performant way. - Tim > On 2 Jan 2019, at 20:37, H I Baysal <hibay...@gmail.com> wrote: > > Hi Tim, > > That absolutely depends on the amount of TAGs you use, and how you aggregate, > etc. > I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even > sweating a single drop.... > > We have a 4 Tbps of traffic during peak, and as well as pmacct and influxdb > or running very very smooth. > > (With the mentioned aggregations I can see what a single customer costs with > Transit, Peering and IX (per IP even if needed) ) > And dst AS per port/description/ethernet name > > From your mail i derive that you just pushed everything to influx from flows, > you have to be a bit smarter with the layout, aggregations and continuous > queries. > (collect what you need) > > > >> On 02-01-19 13:08, Tim Raphael wrote: >> I would advise against InfluxDB in this case - flow data has a very high >> (and open) tag cardinality which is not suited to Influx (although their >> recently new index format has improved this). >> >> I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse (columnar >> store) with a summing merge tree database engine. >> Clickhouse is very fast for queries across columns as well as aggregating >> down them (e.g. summing number of bytes). >> >> For example this is the results of a query of nearly a year’s worth of >> MAC-to-MAC flows (7-tuple) queried for the last 7 days between two given >> sets of MACs: >> >> 2016 rows in set. Elapsed: 0.208 sec. Processed 17.56 million rows, 1.03 GB >> (84.51 million rows/s., 4.97 GB/s.) >> >> There is also a Grafana datasource plugin for Clickhouse :) >> >> - Tim >> >> >>> On 2 Jan 2019, at 7:50 pm, H I Baysal <hibay...@gmail.com> wrote: >>> >>> PMACCT (Works Awesome) >>> push to influxdb ( Works awesome) >>> >>> With some custom scripts to add/match interface descriptions. And you can >>> query whatever you want in grafana :D >>> And grafana has a nice API for rendering a dashboardgraph >>> to a PNG and you can send this png to whatever chat/bot or mail you want. >>> >>> And all for free with 99% of accuracy. >>> >>> (Mucho gracias to Paulo :D ) >>> >>> >>>> On 01-01-19 05:56, Avi Freedman wrote: >>>> We do have a minimum for commercial service that's more like $1500/mo but >>>> we are coming out with a free tier in Q1 with lower retention (among other >>>> deltas, but including fully slice and dice flow analytics +BGP that it >>>> sounded like Erik might be looking for). >>>> >>>> Feel free to ping me if anyone would like to help us test the free tier in >>>> January. >>>> >>>> Thanks, >>>> >>>> Avi Freedman >>>> CEO, Kentik >>>> >>>>> Doesn't Kentik cost like $2000 a month minimum? >>>>> >>>>> >>>>> On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker >>>>> <matt...@corp.crocker.com> >>>>> wrote: >>>>> >>>>>> +1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to >>>>>> worry about it. >>>>>> >>>>>> On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" < >>>>>> nanog-boun...@nanog.org on behalf of br...@shout.net> wrote: >>>>>> >>>>>> +1 Kentik ... >>>>>> >>>>>> We've been using their DDoS/RTBH mitigation with good success. >>>>>> >>>>>> >>>>>> On 12/31/18 3:52 AM, Eric Lindsjö wrote: >>>>>> > Hi, >>>>>> > >>>>>> > We use kentik and we're very happy. Works great, tons of new >>>>>> features >>>>>> > coming along all the time. Going to start looking into ddos >>>>>> detection >>>>>> > and mitigation soon. >>>>>> > >>>>>> > Would recommend. >>>>>> > >>>>>> > Kind regards, >>>>>> > Eric Lindsjö >>>>>> > >>>>>> > >>>>>> > On 12/31/2018 04:29 AM, Erik Sundberg wrote: >>>>>> >> >>>>>> >> Hi Nanog…. >>>>>> >> >>>>>> >> We are looking at replacing our Netflow collector. I am wonder >>>>>> what >>>>>> >> other service providers are using to collect netflow data off >>>>>> their >>>>>> >> Core and Edge Routers. Pros/Cons… What to watch out for any info >>>>>> would >>>>>> >> help. >>>>>> >> >>>>>> >> We are mainly looking to analyze the netflow data. Bonus if it >>>>>> does >>>>>> >> ddos detection and mitigation. >>>>>> >> >>>>>> >> We are looking at >>>>>> >> >>>>>> >> ManageEngine Netflow Analyzer >>>>>> >> >>>>>> >> PRTG >>>>>> >> >>>>>> >> Plixer – Scrutinizer >>>>>> >> >>>>>> >> PeakFlow >>>>>> >> >>>>>> >> Kentik >>>>>> >> >>>>>> >> Solarwinds NTA >>>>>> >> >>>>>> >> Thanks in advance… >>>>>> >> >>>>>> >> Erik >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> ------------------------------------------------------------------------ >>>>>> >> >>>>>> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any >>>>>> documents, >>>>>> >> files or previous e-mail messages attached to it may contain >>>>>> >> confidential information that is legally privileged. If you are >>>>>> not >>>>>> >> the intended recipient, or a person >>>>>> responsible for delivering it >>>>>> to >>>>>> >> the intended recipient, you are hereby notified that any >>>>>> disclosure, >>>>>> >> copying, distribution or use of any of the information contained >>>>>> in >>>>>> or >>>>>> >> attached to this transmission is STRICTLY PROHIBITED. If you have >>>>>> >> received this transmission in error please notify the sender >>>>>> >> immediately by replying to this e-mail. You must destroy the >>>>>> original >>>>>> >> transmission and its attachments without reading or saving in any >>>>>> >> manner. Thank you. >>>>>> > >>>>>> >>>>>> >>>>>> >>
