Re: [pmacct-discussion] Auto-reconnect to DB
I got a backtrace, though not from GDB. The problem is that I don't know which of the two plugins will crash, and as far as I know I can only attach to one plugin... Anyway, what I have is: *** glibc detected *** nfacctd: PostgreSQL Plugin [out]: break adjusted to free malloc space: 0x01e2be60 *** === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f151cfadd76] /lib/x86_64-linux-gnu/libc.so.6(+0x7a443)[0x7f151cfb1443] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7f151cfb2b90] nfacctd: PostgreSQL Plugin [out](sql_cache_insert+0x822)[0x4503c2] nfacctd: PostgreSQL Plugin [out](pgsql_plugin+0xc60)[0x44b160] nfacctd: PostgreSQL Plugin [out](load_plugins+0x314)[0x4228a4] nfacctd: PostgreSQL Plugin [out](main+0xf53)[0x41b123] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f151cf55ead] nfacctd: PostgreSQL Plugin [out][0x41baf5] The config I have is: sql_optimize_clauses: true sql_db: acct sql_table_version: 7 sql_table: acct sql_refresh_time: 1 sql_history: 1s sql_user: cloudsigma sql_dont_try_update: true sql_use_copy: true plugin_pipe_size: 57344 plugin_buffer_size: 2 aggregate: src_host,dst_host,proto,src_port,dst_port plugins: pgsql[in], pgsql[out] Also, I have to point out that I modified the code a bit to allow sql_history of 1s, but dunno if it's related. On Fri, Jun 21, 2013 at 2:39 PM, George-Cristian Bîrzan g...@birzan.orgwrote: I'll try to, but I'm not so sure it'll be trivial to reproduce. On Thu, Jun 20, 2013 at 8:09 PM, Paolo Lucente pa...@pmacct.net wrote: Hi George-Cristian, One or more plugins that bail out and consequently core process that closes up after all plugins are gone (essentially, the message you posted) could be symptom of plugins crashing for some reason. It can help if you run the daemon under gdb with follow-fork-mode set to child and post the backtrace. Please follow this up with gdb ouptuts, etc. privately. Cheers, Paolo On Wed, Jun 19, 2013 at 11:01:30AM +0300, George-Cristian Bîrzan wrote: Is it possible to auto-reconnect to the DB when the connection is lost? For reasons that pass understanding, sometimes, pmacct decides it lost the connection to the DB, at which point it just dies: Jun 19 04:01:53 host nfacctd[24062]: INFO: connection lost to 'out-pgsql'; closing connection. Jun 19 04:01:53 host nfacctd[24062]: INFO: no more plugins active. Shutting down. At that time, our PostgreSQL server didn't log anything: 2013-06-17 16:51:46 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. 2013-06-17 16:51:48 UTC LOG: checkpoints are occurring too frequently (2 seconds apart) 2013-06-17 16:51:48 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. 2013-06-17 17:53:52 UTC WARNING: pgstat wait timeout 2013-06-17 23:58:02 UTC WARNING: pgstat wait timeout 2013-06-19 07:52:18 UTC LOG: checkpoints are occurring too frequently (25 seconds apart) 2013-06-19 07:52:18 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. (The 7:52 one is when I restarted it now. And, yeah, gonna fix the psql stuff, but so far it's been not a problem, as the load on the machine is literally 0 as long as we don't do stupid stuff like try to read the hundreds of GB of data) -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- George-Cristian Bîrzan -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Crash in pmacct
I think I reported that bug, and it was crashing instantly on start, not within minutes. Also, I think that never ended up in a release afair, it was just in trunk. On 8 Jul 2013 13:30, Joan aseq...@gmail.com wrote: BTW, just found in the changelog for 0.14.1 this: ! fix, net_aggr.c: defining a networks_file configuration directive in conjunction with --enable-ipv6 was causing a SEGVs. This is now solved. That could be the cause for my issue (unless debian backported the fixes) 2013/7/8 Joan aseq...@gmail.com I have tried the version in wheezy with the same results as with squeeze, now, I am trying to reproduce the crash with the 0.14.3 downloaded from the site. So far it hasn't crashed, but so far there's only minimal traffic via this router. I'll be back with more info... 2013/7/6 Karl O. Pinc k...@meme.com As an alternative you should consider upgrading to debian wheezy as squeeze will go out of support about 2013-11-04, in 4 months. You'll have to upgrade anyway and this might fix your problem. Wheezy has pmacct 0.14.0. You can get help with any of this for debian using irc chat on the #debian channel of irc.freenode.net. On 07/05/2013 05:39:41 PM, Paolo Lucente wrote: Hi Joan, I can verify the backtrace you provided does not apply to the current (and 0.14.3 release to that matter) code. Also, the issue is related to querying the content of a networks_file - which is a part of the code that got some changes meanwhile. I propose you download/compile 0.14.3 release or CVS code and try again. If these still give troubles please send me privately a new backtrace to inspect. Let me know. Cheers, Paolo On Fri, Jul 05, 2013 at 06:46:21PM +0200, Joan wrote: Hi again, I am experiencing crashes only after a couple of minutes of starting-04 pmacctd. I am on the current squeeze version, but I recompiled from the sources to get non-stripped binaries. After running the process for some minutes the program crashes as usually leaving a nice backtrace. Could you have a look into this and tell me if it's something that was fixed in a newer version? Regards, Joan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists Karl k...@meme.com Free Software: You don't pay back, you pay forward. -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Crash in pmacct
Fair enough. It did sound similar so I assumed it was that. Sorry On 8 Jul 2013 14:41, Joan aseq...@gmail.com wrote: @george, the issue is not the one you reported (that was against 0.14.3cvs) but with an older version. revision 1.16 date: 2012-04-12 14:44:30 +0200; author: paolo; state: Exp; lines: +3 -3; * nfacctd: etype primitive can now be populated from IP_PROTOCOL_VERSION, ie. Field Type #60, in addition to ETHERTYPE, ie. Field Type #256. Should both be present the latter has priority over the former. * fix, net_aggr.c: if --enable-ipv6 is specified, defining a networks_file can cause SEGVs. This is now solved. 2013/7/8 Joan aseq...@gmail.com The wheezy defautl was crashing for me a bit after loading the networks_file (that take about a couple of minutes to load) I was trying to isolate this to open a bug in debian, so at least others are warned. After unsetting the --enable-ipv6 flag and recompile again with debian settings/patches, it seems that it doens't crash anymore. Still I will recompile the 0.14.3 version because I was planning to use the extended format of networks_file for the nexthop feature. 2013/7/8 George-Cristian Bîrzan g...@birzan.org I think I reported that bug, and it was crashing instantly on start, not within minutes. Also, I think that never ended up in a release afair, it was just in trunk. On 8 Jul 2013 13:30, Joan aseq...@gmail.com wrote: BTW, just found in the changelog for 0.14.1 this: ! fix, net_aggr.c: defining a networks_file configuration directive in conjunction with --enable-ipv6 was causing a SEGVs. This is now solved. That could be the cause for my issue (unless debian backported the fixes) 2013/7/8 Joan aseq...@gmail.com I have tried the version in wheezy with the same results as with squeeze, now, I am trying to reproduce the crash with the 0.14.3 downloaded from the site. So far it hasn't crashed, but so far there's only minimal traffic via this router. I'll be back with more info... 2013/7/6 Karl O. Pinc k...@meme.com As an alternative you should consider upgrading to debian wheezy as squeeze will go out of support about 2013-11-04, in 4 months. You'll have to upgrade anyway and this might fix your problem. Wheezy has pmacct 0.14.0. You can get help with any of this for debian using irc chat on the #debian channel of irc.freenode.net. On 07/05/2013 05:39:41 PM, Paolo Lucente wrote: Hi Joan, I can verify the backtrace you provided does not apply to the current (and 0.14.3 release to that matter) code. Also, the issue is related to querying the content of a networks_file - which is a part of the code that got some changes meanwhile. I propose you download/compile 0.14.3 release or CVS code and try again. If these still give troubles please send me privately a new backtrace to inspect. Let me know. Cheers, Paolo On Fri, Jul 05, 2013 at 06:46:21PM +0200, Joan wrote: Hi again, I am experiencing crashes only after a couple of minutes of starting-04 pmacctd. I am on the current squeeze version, but I recompiled from the sources to get non-stripped binaries. After running the process for some minutes the program crashes as usually leaving a nice backtrace. Could you have a look into this and tell me if it's something that was fixed in a newer version? Regards, Joan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists Karl k...@meme.com Free Software: You don't pay back, you pay forward. -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Auto-reconnect to DB
I'll try to, but I'm not so sure it'll be trivial to reproduce. On Thu, Jun 20, 2013 at 8:09 PM, Paolo Lucente pa...@pmacct.net wrote: Hi George-Cristian, One or more plugins that bail out and consequently core process that closes up after all plugins are gone (essentially, the message you posted) could be symptom of plugins crashing for some reason. It can help if you run the daemon under gdb with follow-fork-mode set to child and post the backtrace. Please follow this up with gdb ouptuts, etc. privately. Cheers, Paolo On Wed, Jun 19, 2013 at 11:01:30AM +0300, George-Cristian Bîrzan wrote: Is it possible to auto-reconnect to the DB when the connection is lost? For reasons that pass understanding, sometimes, pmacct decides it lost the connection to the DB, at which point it just dies: Jun 19 04:01:53 host nfacctd[24062]: INFO: connection lost to 'out-pgsql'; closing connection. Jun 19 04:01:53 host nfacctd[24062]: INFO: no more plugins active. Shutting down. At that time, our PostgreSQL server didn't log anything: 2013-06-17 16:51:46 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. 2013-06-17 16:51:48 UTC LOG: checkpoints are occurring too frequently (2 seconds apart) 2013-06-17 16:51:48 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. 2013-06-17 17:53:52 UTC WARNING: pgstat wait timeout 2013-06-17 23:58:02 UTC WARNING: pgstat wait timeout 2013-06-19 07:52:18 UTC LOG: checkpoints are occurring too frequently (25 seconds apart) 2013-06-19 07:52:18 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. (The 7:52 one is when I restarted it now. And, yeah, gonna fix the psql stuff, but so far it's been not a problem, as the load on the machine is literally 0 as long as we don't do stupid stuff like try to read the hundreds of GB of data) -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Auto-reconnect to DB
Is it possible to auto-reconnect to the DB when the connection is lost? For reasons that pass understanding, sometimes, pmacct decides it lost the connection to the DB, at which point it just dies: Jun 19 04:01:53 host nfacctd[24062]: INFO: connection lost to 'out-pgsql'; closing connection. Jun 19 04:01:53 host nfacctd[24062]: INFO: no more plugins active. Shutting down. At that time, our PostgreSQL server didn't log anything: 2013-06-17 16:51:46 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. 2013-06-17 16:51:48 UTC LOG: checkpoints are occurring too frequently (2 seconds apart) 2013-06-17 16:51:48 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. 2013-06-17 17:53:52 UTC WARNING: pgstat wait timeout 2013-06-17 23:58:02 UTC WARNING: pgstat wait timeout 2013-06-19 07:52:18 UTC LOG: checkpoints are occurring too frequently (25 seconds apart) 2013-06-19 07:52:18 UTC HINT: Consider increasing the configuration parameter checkpoint_segments. (The 7:52 one is when I restarted it now. And, yeah, gonna fix the psql stuff, but so far it's been not a problem, as the load on the machine is literally 0 as long as we don't do stupid stuff like try to read the hundreds of GB of data) -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct-contribs now on GitHub
Out of curiosity, is there any plan to move the main repo to git/mercurial? On Sat, May 11, 2013 at 11:50 AM, Paolo Lucente pa...@pmacct.net wrote: Dears, A brief announcement to say pmacct-contribs, the effort to put together 3rd party contributions to the pmacct project (scripts, tools, frontends, etc.), is now published on GitHub. This is in order to facilitate and encourage sharing of new contributions as well as to try reducing scattering of the existing ones. The URL of the project at GitHub is: https://github.com/paololucente/pmacct-contrib Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] SQL history
On Sun, Apr 28, 2013 at 3:53 PM, Paolo Lucente pa...@pmacct.net wrote: If this is all correct you can get latest code from the CVS where you have availability of newly introduced timestamp_start and timestamp_end primitives precisely for this purpose. I've compiled that and it segfaults when I use a network list file (doesn't seem to matter what's inside): # nfacctd -n /etc/pmacct/networks.lst WARN ( cmdline ): No plugin has been activated; defaulting to in-memory table. WARN ( default/memory ): defaulting to SRC HOST aggregation. Segmentation fault # cat /etc/pmacct/networks.lst 123,10.0.0.0/8 This is the (most) relevant part of catchsegv: Backtrace: nfacctd: IMT Plugin [default](load_networks6+0x852)[0x438132] nfacctd: IMT Plugin [default](imt_plugin+0x190)[0x426930] nfacctd: IMT Plugin [default](load_plugins+0x306)[0x422796] nfacctd: IMT Plugin [default](main+0xee8)[0x41d928] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff)[0x7ffd17b8aeff] nfacctd: IMT Plugin [default][0x418679] -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] SQL history
On Sun, Apr 28, 2013 at 3:53 PM, Paolo Lucente pa...@pmacct.net wrote: Hi George-Cristian, You do not specify what is input of network traffic - netflow, sflow or packet capture. Since you speak of one minute temporal aggregation being too coarse-grained, i believe your input is netflow. Is then your purpose to log down micro-flows to the database, ie. ideally disable any sort of time aggregation? That's not entirely true. While having flows in the DB won't hurt, our idea was to just aggregate over small time intervals (say 15 seconds) so we can then look in more detail at some traffic patterns. Granted, we can probably aggregate with the DB's help, so that'd probably work too. If this is all correct you can get latest code from the CVS where you have availability of newly introduced timestamp_start and timestamp_end primitives precisely for this purpose. I will, thanks! Your question could be why introducing such basic time-related primitives so recently. Q6 in the new FAQS document addresses this point: pmacct was not originally meant to log network traffic at packet/micro-flow level: it allows to get an aggregated view of the traffic -- both in space and in time. On top of that, there are layers of filtering, sampling and tagging. These are the keys to scale. As these features are fully configurable, data granularity and resolution can be traded off in favour of increased scalability or less resources consumption. More recently, logging has been introduced, by means of two new primitives timestamp_start and timestamp_end, fostered by the development of NetFlow/IPFIX as generic transport protocol, ie. as a replacement of syslog; it was then intuitive to generalize the logging support to the more traditional traffic accounting part. My question is more like any particularly convincing reason why there's a 1m limit (which to me looks quite arbitrary and useless) for dumping things into SQL. Hope the above helps, let me know. It probably will, and while I won't really look into it until Monday, I will let you know! -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] SQL history
Is it possible to set the SQL history value to lower than 1m? From the docs, it seems not, same with the source, and my attempts at doing it failed: 's' is not a recognized and putting in something without a suffix is the same as leaving it false. I've been, also, playing with 'sql_dont_use_update', which does give me a better resolution, but it makes it impossible to tell for how long a period the rows are for (probably, this only applies for a sql_refresh_time smaller than sql_history, but yeah). Assuming it's not possible, is there a particular reason for this? We don't have that much traffic, I think we peek at 500Mbps during normal times, I'm pretty sure we can scrounge up a server good enough to handle way more than that, especially since we really don't need ACID on this stuff... -- George-Cristian Bîrzan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
