Hi Paolo and Daniel, (please allow me to jump in as I may be able to help here, despite currently being in country working on a project.)
On Fri, 19 Feb 2010, Paolo Lucente wrote: > I also wonder: how does the primary key of the 1 min table look like? Is > it any different from the 1 hour table? With the sql_don_try_update > turned on and the default indexing, duplicates are not possible. I deleted the primary key from that table because it should not be necessary (there should not be any duplicates if everything is configured correctly) and it makes inserts extremely slow (by a factor of 10-100) when the table gets large. > Also at a closer look to the configuration you posted i see no > aggregate_filter are specified (see EXAMPLES): it means each plugin > collects and tries to write to the same table both inbound and outbound > traffic. So either you can remove one set of plugins or craft a proper > aggregate_filter so that each does only its bit of the job. The inbound and outbound traffic are supposed to go into the same table, but you're right that the aggregate_filter appears to be missing and this is almost certainly the cause of the duplicate records in the short table. Daniel, could you please add something like this: aggregate_filter[inbound1]: dst net 10.0.156.0/24 aggregate_filter[outbound1]: src net 10.0.156.0/24 aggregate_filter[inbound2]: dst net 10.0.156.0/24 aggregate_filter[outbound2]: src net 10.0.156.0/24 However, I'm surprised that this doesn't also happen in the long table? > With regards to the missing tuples, from the few checks i've done, it is > always the case that something is in the 1 hour table but can be missing > in the 1 minute one. This can very well be the result of a shared > 'sql_preprocess: minb = 1000' directive: a flow can accumulate more than > 1000 bytes in 1 hour but not in 1 minute - and hence it's accounted in > one table and stripped off in the other. Yes, I would expect the long table totals to be slightly more than the short table ones for this reason. However, the problem that we're seeing is the opposite: the totals calculated from the long table are less than those from the short table, even though the long table includes flows that the short table doesn't. And, while this might be accounted for by the duplicate flows in the short table, the same should apply to the long table, so I think it should have balanced out. > Given the sql_preprocess you should never expect counters to match for > the same reason as above. To have a comparison more apples to apples, > you should consider removing it and when confident everything is > allright put it back again. Unfortunately we cannot do this in the production environment, as the number of rows of tiny flows (which are effectively noise) completely dwarfs the real data, overloads our firewall's CPU and disk space, and makes querying so slow that the data is useless. This is where a test lab environment would be useful. Thanks for your help with this :) Cheers, Chris. -- Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
