Hi Paolo, You are wise my friend.
1) Yes just upgraded to 1.5.2. (Was having problems in 1.5.1 where the input ifindex was getting mangled.) 2) Correct - was not seeing memory leak before upgrade. 3) If I leave running, eventually it consumes all available memory on the server. From syslog prior to service restart, Jan 7 15:05:05 packet1 nfacctd[16988]: WARN ( default/print ): Unable to fork writer: Cannot allocate memory 4) config: aggregate: peer_src_ip,in_iface,dst_host,dst_mask plugins: print nfacctd_time_new: true plugin_buffer_size: 10240 imt_buckets: 157 imt_mem_pools_number: 256 imt_mem_pools_size: 32768 syslog: daemon daemonize: true ! print_refresh_time: 64 print_history: 1m print_output: csv print_output_file: /opt/pmacct/data/nfacct-%Y%m%d-%H%M.csv ! print_trigger_exec: /opt/gravitizer/bin/monitor.pl print_output_file_append: true ! nfacctd_port: 2055 Thanks, Ed On Thu, Jan 7, 2016 at 4:20 PM, Paolo Lucente <pa...@pmacct.net> wrote: > Hi Ed, > > You mean you kind of just upgraded to 1.5.2 and you were not seeing > the same before the upgrade? If you leave it running, does it grow > undefinitely or at some stage it looks it reaches a stable memory > footprint? Especially if using malloc()ed structures, ie. derived > by BGP primitives, this could be a valid scenario. Finally, can you > share your config? > > Cheers, > Paolo > > On Thu, Jan 07, 2016 at 04:04:09PM -0600, Edward Henigin wrote: > > Hi Paolo, > > > > I'm seeing a significant memory leak in the print plugin in 1.5.2. > > > > # service nfacctd restart > > Shutting down sflow collection services (nfacctd): [ OK ] > > Starting sflow collection services (nfacctd): INFO ( default/core ): > > Daemonizing. Hmm, bye bye screen. > > [ OK ] > > # pgrep nfacctd > > 874 > > 876 > > # while [ 1 ]; do sleep 60; date; ps uwp 876; done > > Thu Jan 7 15:53:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.1 0.6 115276 104432 ? S 15:52 0:01 nfacctd: > > Print Plugin [default] > > Thu Jan 7 15:54:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.0 0.6 120884 109980 ? S 15:52 0:01 nfacctd: > > Print Plugin [default] > > Thu Jan 7 15:55:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.0 0.7 128408 117608 ? S 15:52 0:02 nfacctd: > > Print Plugin [default] > > Thu Jan 7 15:56:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.1 0.8 147680 136856 ? S 15:52 0:03 nfacctd: > > Print Plugin [default] > > Thu Jan 7 15:57:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.1 0.9 160748 149944 ? S 15:52 0:03 nfacctd: > > Print Plugin [default] > > Thu Jan 7 15:58:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.1 1.0 180548 169668 ? S 15:52 0:04 nfacctd: > > Print Plugin [default] > > Thu Jan 7 15:59:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.1 1.1 203384 192604 ? S 15:52 0:05 nfacctd: > > Print Plugin [default] > > Thu Jan 7 16:00:56 CST 2016 > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 876 1.3 1.5 272024 261120 ? S 15:52 0:07 nfacctd: > > Print Plugin [default] > > ^C > > # strings - /usr/sbin/nfacctd | fgrep 1.5 > > NetFlow Accounting Daemon, nfacctd 1.5.2 > > /opt/pmacct/pmacct-1.5.2/src > > /opt/pmacct/pmacct-1.5.2/src/nfprobe_plugin > > /opt/pmacct/pmacct-1.5.2/src/sfprobe_plugin > > /opt/pmacct/pmacct-1.5.2/src/bgp > > /opt/pmacct/pmacct-1.5.2/src/tee_plugin > > /opt/pmacct/pmacct-1.5.2/src/isis > > /opt/pmacct/pmacct-1.5.2/src/bmp > > > > Any thoughts? Suggestions? > > > > Thanks, > > > > Ed > > > > On Mon, Sep 7, 2015 at 1:01 PM, Paolo Lucente <pa...@pmacct.net> wrote: > > > > > VERSION. > > > 1.5.2 > > > > > > > > > DESCRIPTION. > > > pmacct is a small set of passive network monitoring tools to account, > > > classify, > > > aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable > > > architecture > > > allows to store collected data into memory tables, RDBMS (MySQL, > > > PostgreSQL, > > > SQLite), noSQL databases (MongoDB, BerkeleyDB) AMQP message exchanges > > > (RabbitMQ) and flat-files and export through NetFlow or sFlow > protocols to > > > remote collectors. pmacct offers customizable historical data > breakdown, > > > BGP > > > and IGP correlation, BMP stats and events, GeoIP lookups, sampling and > > > renormalization, filtering, tagging and triggers. Libpcap, Linux > > > Netlink/ULOG, > > > sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are supported. Replication > of > > > incoming NetFlow and sFlow datagrams is also available. Statistics can > be > > > easily > > > exported to tools like RRDtool, Net-SNMP, MRTG, GNUPlot and Cacti. > > > > > > > > > HOMEPAGE. > > > http://www.pmacct.net/ > > > > > > > > > DOWNLOAD. > > > http://www.pmacct.net/pmacct-1.5.2.tar.gz > > > > > > > > > CHANGELOG. > > > + Introduced support for a RabbitMQ broker to be used for queueing and > > > data exchange between Core Process and plugins. This is in > alternative to > > > the home-grown circular queue implementation. plugin_pipe_amqp > directive, > > > along with all other plugin_pipe_amqp_* directives, can be set > globally > > > or apply on a per plugin basis (ie. it is a valid scenario, if > multiple > > > plugins are instantiated, that some make use of home-grown queueing, > > > while others use RabbitMQ based queueing). > > > + Introducting support for Maximind GeoIP v2 (libmaxminddb) library: if > > > pmacct is compiled with --enable-geoipv2, this defines full pathname > to > > > a Maxmind GeoIP database v2 (libmaxminddb) Only the binary database > > > format is supported (ie. it is not possible to load distinct CSVs for > > > IPv4 and IPv6 addresses). > > > + Introduced infrastructure for sFlow counters and support > specifically for > > > generic, ethernet and vlan counters. Counters are exported in JSON > format > > > to files, specified via sfacctd_counter_file. The supplied filename > can > > > contain as variable the sFlow agent IP address. > > > + Introduced a new thread_stack config directive to allow to modify the > > > thread stack size. Natanael Copa reported that some libc > implementations, > > > ie. musl libc, may set a stack size that is too small by default. > > > + Introduced networks_file_no_lpm feature: it applies when aggregation > > > method includes src_net and/or dst_net and nfacctd_net (or > equivalents) > > > and/or nfacctd_as_new (or equivalents) are set to longest (or > fallback): > > > an IP prefix defined as part of the supplied networks_file wins > always, > > > even if it's not longest. > > > + tee plugin: added support for (non-)transparent IPv6 replication > [further > > > QA required] > > > + plugin_common.c, sql_common.c: added log message to estimate base > cache > > > memory usage. > > > + print, AMQP, MongoDB plugins; sfacctd, BGP, BMP daemons: introducing > > > timestamps_since_epoch to write timestamps in 'since Epoch' format. > > > + nfacctd: flow bytes counter can now be sourced via element ID #352 > > > (layer2OctetDeltaCount) in addition to element ID's already > supported. > > > Thanks to Jonathan Thorpe for his support. > > > + Introducing proc_priority: redefines the process scheduling priority, > > > equivalent to using the 'nice' tool. Each daemon process, ie. core, > > > plugins, etc., can define a different priority. > > > ! fix, BMP daemon: improved preliminar checks in bmp_log_msg() and > added > > > missing SIGHUP signal handling to reload bmp_daemon_msglog_file > files. > > > ! fix, bgp_logdump.c: under certain configuration conditions call to > both > > > write_and_free_json() and write_and_free_json_amqp() was leading to > SEGV. > > > Thanks to Yuriy Lachin for reporting the issue. > > > ! fix, BGP daemon: improved BGP dump output: more accurate > timestamping of > > > dump_init, dump_close events. dump_close now mentions amount of > entries > > > and tables dumped. Thanks to Yuriy Lachin for brainstorming around > this. > > > ! fix, cfg.c: raised amount of allowed config lines from 256 to 8K. > > > ! fix, print/AMQP/MongoDB plugins: SEGV observed when IPFIX vlen > variables > > > were stored in the pending_queries_queue structure (ie. as a result > of a > > > time mismatch among the IPFIX exporter and the collector box). > > > ! fix, vlen primitives: when 'raw' semantics was selected, print_hex() > was > > > returning wrong hex string length (one char short). As a consequence > > > occasionally some extra dirty chars were seen at the end of the > converted > > > string. > > > ! fix, vlen primitives: memory leak verified in print/AMQP/MongoDB > plugins. > > > ! fix, print, MongoDB & AMQP plugins: dirty values printed as part of > the > > > 'proto' under certain conditions. Thanks to Rene Stoutjesdijk for his > > > support resolving the issue. > > > ! fix, amqp_common.c: amqp_exchange_declare() call changed so to > address > > > the change of rabbitmq-c API for support of auto_delete & internal > for > > > exchange.declare. Backward compatibility with rabbitmq-c <= 0.5.2 is > > > also taken care of. Thanks to Brent Van Dussen for reporting the > issue. > > > ! fix, compiling on recent FreeBSD: solved some errors catched by the > -Wall > > > compiler flag. Thanks to Stephen Fulton for reporting the issue. > Most of > > > the patch is courtesy by Mike Bowie. > > > ! fix, print/AMQP/MongoDB plugins: enforcing cleanup of malloc()ed > structs > > > part of entries added to the pending queue, ie. because seen as > future > > > entries due to a mismatch of the collector clock with the one of > NetFlow/ > > > IPFIX exporter(s). This may have lead to data inconsistencies. > > > ! fix, amqp_common.c: Content type was only specified for messages > > > published > > > when the amqp_persistent_msg configuration option is specified. This > info > > > should always be applied to describe the payload of the message. > Patch is > > > courtesy by Will Dowling. > > > ! fix, amqp_plugin.c: generate an error on compile if > --enable-rabbitmq is > > > specified without --enable-jansson. It's clear in the documentation > that > > > both are required for AMQP support, but if built without jansson it > will > > > silently not publish messages to AMQP. Patch is courtesy by Will > Dowling. > > > ! fix, amqp_common.c: modified the content type to "application/json" > in > > > line with RFC4627. Patch is courtesy by Will Dowling. > > > ! fix, setsockopt(): u_int64_t pipe_size vars changed to int, in line > with > > > typical OS buffer limits (Linux, Solaris). Introduced check supplied > pipe > > > size values are not bigger than INT_MAX. Many thanks to Markus Weber > for > > > reporting the issue. > > > ! fix, nl.c: removed pretag_free_label() from pcap_cb() and ensuring > init > > > of pptrs. Under certain conditions SEGVs could be noticed. > > > ! fix, flow stitching: when print/AMQP/MongoDB plugins were making use > of > > > the pending queries queue, ie. to compensate for time offsets/flows > in > > > the future, the stitching feature could potentially lead to SEGV due > to > > > unsettled pointers. > > > ! fix, pgsql plugin: SEGV were noticed when insert/update queries to > the > > > PostgreSQL database were returning different than PGRES_COMMAND_OK, > hence > > > triggering the reprocess mechanism. Thanks very much to Alan Turower > for > > > his support. > > > ! fix, improved logging of elements received/sent at buffering point > > > between > > > core process and plugins. Also added explicit start/end purge log > message > > > for cases in which there is no data to purge. > > > ! fix, signals.c: ignore_falling_child() now logs if a child process > exited > > > with abnormal conditions; this is useful to track writer processes > > > (created > > > by plugins) are terminated by a signal, ie. SEGV. This is already the > > > case > > > for plugins themselves, with the Core Process reporting a simlar log > > > message in case of abnormal exit. Thanks very much to Rene > Stoutjesdijk > > > for his support. > > > ! fix, preprocess-data.h: added supported functions minf, minb, minbpp > and > > > minppf to non SQL plugins. Thanks to Jared Deyo for reporting the > issue. > > > ! fix, nfprobe_plugin.c: IP protocol was not set up correctly for IPv6 > > > traffic in NetFlow v9/IPFIX. Thanks to Gabriel Vermeulen his support > > > solving the issue. > > > > > > > > > NOTES. > > > See UPGRADE file. > > > > > > > > > Cheers, > > > Paolo > > > > > > _______________________________________________ > > > pmacct-discussion mailing list > > > http://www.pmacct.net/#mailinglists > > > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
