To wrap this thread up for the time being (ie. until Brent can do some further testing or somebody else hits the same issue).
The issue - with the IMT plugin, under a recent Debian - manifested like if pmacct was trying to free() twice some memory allocated previously - done as part of serving the "pmacct -e" request. Now, freeing for real the same piece of memory twice or multiple times due to an algorithmic flaw within the code would have made the issue more evident than an isolated complain. Bottom line, the workaround was to resort setting the MALLOC_CHECK_ environment variable - which is honoured by recent glibc and documented here: http://www.gnu.org/s/libc/manual/html_node/Heap-Consistency-Checking.html So, in practice sfacctd is being launched from within a shell script as: MALLOC_CHECK_=0 export MALLOC_CHECK_ /path/to/sfacctd -f /path/to/sfacctd.conf Cheers, Paolo On Mon, Mar 15, 2010 at 12:22:24PM -0700, Brent Van Dussen wrote: > It's definitely pmacct -e -p ... that kills it, pmacct -s -p ... works > like a champ every time. > > Sorry I'm not at all familliar with GDB, should I compile up a version > of pmacct with --debug enabled so you can log in and take a look? > > We are running CentOS 5.4 64-bit on a quadcore xeon box. > > bash-3.2# uname -a > Linux <snip> 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2008 x86_64 > x86_64 x86_64 GNU/Linux > > E-mail me privately with a network or host you'll be logging in from and > I can update my hosts.allow for you and setup an acct. > > Thanks! > -Brent > > On Mar 13, 2010, at 3:26 AM, Paolo Lucente wrote: > >> Hi Brent, >> >> Thanks for reporting the issue. I know of at least one >> other people making that large use of the memory table >> (in conjunction with BGP primitives) and nothing like >> that could be seen. >> >> I would start de-coupling the two commands to see which >> one of the two causes the issue (so, "pmacct -s -p ..." >> and "pmacct -e -p ..." after that). My suspects fall on >> the second one. >> >> If you have any gdb skills, any input in that sense is >> greatly appreciated. Otherwise i would be glad to have >> a look myself on the box - if that's an option. Either >> case, if applicable, we can follow it up privately and >> then post outcome here on the list. >> >> Can you also confirm which OS and architecture are you >> running? >> >> Cheers, >> Paolo >> >> >> >> On Fri, Mar 12, 2010 at 09:56:34PM -0800, Brent Van Dussen wrote: >>> Paolo - >>> >>> when I try to run "pmacct -s -e" I get a disconnected memory pipe and >>> have to restart the daemon to get it back. >>> >>> bash-3.2$ pmacct -s -e -p /tmp/sfacctd_prefixes.pipe >>> <...sniiiiiiiip...> >>> >>> For a total of: 1391302 entries <---- !!!! :) >>> >>> >>> bash-3.2$ pmacct -s -e -p /tmp/sfacctd_prefixes.pipe >>> INFO: Connection refused while trying to connect to '/tmp/ >>> sfacctd_prefixes.pipe' >>> >>> In my logs I get: >>> >>> INFO: connection lost to 'prefixes-memory'; closing connection. >>> >>> >>> Thanks, >>> -Brent >>> >>> >>> On Mar 12, 2010, at 4:04 AM, Paolo Lucente wrote: >>> >>>> Hi Brent, >>>> >>>> Good to see progress. >>>> >>>> The entries stay "forever", there is not an aging-out mechanism. >>>> Reason >>>> being you are supposed to do it yourself, at regular intervals, to >>>> build >>>> a time reference for the counters. >>>> >>>> For example a simplistic scenario is a cronjob entry, set up every 5 >>>> minutes, that launches a "pmacct -s -e > counters-<time>.txt". Such >>>> commandline a) fetches the whole content of the table, b) writes it >>>> to a file and c) cleans the full table up. Should be all you need to >>>> feed such data into 3rd party applications for reporting, graphing, >>>> etc. >>>> >>>> Cheers, >>>> Paolo >>>> >>>> >>>> On Thu, Mar 11, 2010 at 02:00:27PM -0800, Brent Van Dussen wrote: >>>>> Good news, no longer getting the errors and my system hasn't fallen >>>>> on >>>>> its face yet. >>>>> >>>>> Also have 2-3x more entries in the memory tables so that's >>>>> excellent >>>>> as >>>>> well. >>>>> >>>>> My next question is regarding aging of entries in the memory table, >>>>> how >>>>> is this done? At first I checked one of the tables and there were >>>>> over >>>>> 100k entries, 3 or 4 seconds later when I checked the table it was >>>>> down >>>>> to 50k entries and seems to fluctuate +/- 15k entries. >>>>> >>>>> I've read the INTERNALS document pertaining to memory tables but it >>>>> wasn't quite clear in this regard. >>>>> >>>>> Thanks! >>>>> -Brent >>> > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
