[pmacct-discussion] mysql - stamp_inserted in the past
I'm doing mysql rollups and seeing odd timestamps. I'm seeing occasional stamp_inserted times in the past. (some even as far back as several weeks) This is using pmacct-1.5.0rc2 snippet of relevant config: !# Tag Map to set agent_id pre_tag_map: pretag.map plugins: mysql[asn] sql_optimize_clauses: true sql_refresh_time: 300 sql_history: 5m sql_history_roundoff: m !sql_dont_try_update: true !# ASN Aggregation aggregate[asn]: tag,dst_as,as_path,peer_dst_as sql_table[asn]: traffic_by_asn sql_table_type[asn]: bgp My mysql schema. CREATE TABLE `traffic_by_asn` ( `agent_id` int(4) unsigned NOT NULL, `as_dst` int(4) unsigned NOT NULL, `as_path` char(21) NOT NULL, `peer_as_dst` int(4) unsigned NOT NULL, `packets` int(10) unsigned NOT NULL, `bytes` bigint(20) unsigned NOT NULL, `stamp_inserted` datetime NOT NULL, `stamp_updated` datetime DEFAULT NULL, PRIMARY KEY (`agent_id`,`as_dst`,`as_path`,`peer_as_dst`,`stamp_inserted`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; I assume the timestamp might be coming from the flow data? Possibly a really old flow just now sending FINs? Any advice on what this might be? Any way to override the flow timestamps with some actual time window data? Cheers, --Joel ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] mysql - stamp_inserted in the past
Hi Joel, Are you using Juniper by any chance? I am working on an issue with Juniper IPFIX data, where the start time of the flows coming from the routers is reported weeks in the past. Is there any way you could capture the flow data to verify the timestamp in there? Best regards Elisa On Feb 24, 2014, at 7:30 PM, Joel Krauska wrote: > I'm doing mysql rollups and seeing odd timestamps. > > I'm seeing occasional stamp_inserted times in the past. > (some even as far back as several weeks) > > This is using pmacct-1.5.0rc2 > > snippet of relevant config: > > > !# Tag Map to set agent_id > pre_tag_map: pretag.map > > plugins: mysql[asn] > > sql_optimize_clauses: true > sql_refresh_time: 300 > sql_history: 5m > sql_history_roundoff: m > !sql_dont_try_update: true > > !# ASN Aggregation > aggregate[asn]: tag,dst_as,as_path,peer_dst_as > sql_table[asn]: traffic_by_asn > sql_table_type[asn]: bgp > > > My mysql schema. > CREATE TABLE `traffic_by_asn` ( > `agent_id` int(4) unsigned NOT NULL, > `as_dst` int(4) unsigned NOT NULL, > `as_path` char(21) NOT NULL, > `peer_as_dst` int(4) unsigned NOT NULL, > `packets` int(10) unsigned NOT NULL, > `bytes` bigint(20) unsigned NOT NULL, > `stamp_inserted` datetime NOT NULL, > `stamp_updated` datetime DEFAULT NULL, > PRIMARY KEY (`agent_id`,`as_dst`,`as_path`,`peer_as_dst`,`stamp_inserted`) > ) ENGINE=InnoDB DEFAULT CHARSET=latin1; > > > I assume the timestamp might be coming from the flow data? > Possibly a really old flow just now sending FINs? > > > Any advice on what this might be? > > Any way to override the flow timestamps with some actual time window data? > > Cheers, > > --Joel > > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] mysql - stamp_inserted in the past
Hi Joel, On top of Elisa's answer, inline: On Mon, Feb 24, 2014 at 07:30:38PM -0800, Joel Krauska wrote: > > Any way to override the flow timestamps with some actual time window data? Yes, nfacctd_time_new set to true. Feel free to read docs and give it a try. Unless: a) timers are set short (ie. less than one minute imho) on the router, also for active flows!, and b) such timers are honoured for true (ie. these timestamps well in the past are only result of a timestamp reporting bug and flows are actual instead) - going this way might not give you the expected solution. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] mysql - stamp_inserted in the past
Yes -- this is on Juniper IPFIX. Elisa: I can try to capture an 'errant' flow packet. Paolo: That seems like a good work around. Thanks. I've fought a lot with the docs, maybe they don't match my learning style. (would it be possible to try to add some internal linking/anchors in the doc?) A FAQ might be in order too based on historical mailing list Qs? Want some help trying to put this together? --Joel On Tue, Feb 25, 2014 at 9:51 AM, Paolo Lucente wrote: > Hi Joel, > > On top of Elisa's answer, inline: > > On Mon, Feb 24, 2014 at 07:30:38PM -0800, Joel Krauska wrote: > > > > Any way to override the flow timestamps with some actual time window > data? > > Yes, nfacctd_time_new set to true. Feel free to read docs and > give it a try. Unless: a) timers are set short (ie. less than > one minute imho) on the router, also for active flows!, and b) > such timers are honoured for true (ie. these timestamps well > in the past are only result of a timestamp reporting bug and > flows are actual instead) - going this way might not give you > the expected solution. > > Cheers, > Paolo > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] mysql - stamp_inserted in the past
Hi Joel, Help with documentation would be indeed much appreciated (along with tips, methodology, etc.). Maybe creating an account for you on the wiki is a good first step? If it sounds a workable approach we can follow-up privately to exchange login details. (Needless to say invite to get an account is warmly extended to anybody feels to contribute any bit of documentation for the benefit of the community). Cheers, Paolo On Tue, Feb 25, 2014 at 10:33:12AM -0800, Joel Krauska wrote: > Yes -- this is on Juniper IPFIX. > > Elisa: I can try to capture an 'errant' flow packet. > > Paolo: That seems like a good work around. Thanks. > I've fought a lot with the docs, maybe they don't match my learning style. > (would it be possible to try to add some internal linking/anchors in the > doc?) > > A FAQ might be in order too based on historical mailing list Qs? > > Want some help trying to put this together? > > --Joel > > > > > > On Tue, Feb 25, 2014 at 9:51 AM, Paolo Lucente wrote: > > > Hi Joel, > > > > On top of Elisa's answer, inline: > > > > On Mon, Feb 24, 2014 at 07:30:38PM -0800, Joel Krauska wrote: > > > > > > Any way to override the flow timestamps with some actual time window > > data? > > > > Yes, nfacctd_time_new set to true. Feel free to read docs and > > give it a try. Unless: a) timers are set short (ie. less than > > one minute imho) on the router, also for active flows!, and b) > > such timers are honoured for true (ie. these timestamps well > > in the past are only result of a timestamp reporting bug and > > flows are actual instead) - going this way might not give you > > the expected solution. > > > > Cheers, > > Paolo > > > > > > ___ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
