Congratulations, Paolo, these are really great updates! Cheers, and thanks again for all of your hard work for the community.
Aaron On Sun, May 6, 2018, 6:45 AM Paolo Lucente <pa...@pmacct.net> wrote: > VERSION. > 1.7.1 > > > DESCRIPTION. > pmacct is a small set of multi-purpose passive network monitoring tools. It > can account, classify, aggregate, replicate and export forwarding-plane > data, > ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP > and BMP; collect infrastructure data via Streaming Telemetry. Each > component > works both as a standalone daemon and as a thread of execution for > correlation > purposes (ie. enrich NetFlow with BGP data). > > A pluggable architecture allows to store collected forwarding-plane data > into > memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB, > BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files. > pmacct offers customizable historical data breakdown, data enrichments like > BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers. > Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX > are > all supported as inputs for forwarding-plane data. Replication of incoming > NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be > easily exported to time-series databases like ElasticSearch and InfluxDB > and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. > > Control-plane and infrastructure data, collected via BGP, BMP and Streaming > Telemetry, can be all logged real-time or dumped at regular time intervals > to AMQP (RabbitMQ) and Kafka message exchanges and flat-files. > > > HOMEPAGE. > http://www.pmacct.net/ > > > DOWNLOAD. > http://www.pmacct.net/pmacct-1.7.1.tar.gz > > > CHANGELOG. > + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers > (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a > standalone BGP daemon (pmbgpd). The aim is to facilitate operations > when re-sizing/re-balancing the collection infrastructure without > impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map > expects full pathname to a file where cross-connects are defined; > mapping works only against the IP source address and not the BGP > Router ID, only 1:1 relationships can be formed (ie. this is about > cross-connecting, not replication) and only one session per BGP > peer is supported (ie. multiple BGP agents are running on the same > IP address or NAT traversal scenarios are not supported [yet]). > A sample map is provided in 'examples/bgp_xconnects.map.example'. > + pmbgpd: introduced a BGP Looking Glass server allowing to perform > queries, ie. lookup of IP addresses/prefixes or get the list of BGP > peers, against available BGP RIBs. The server is asyncronous and > uses ZeroMQ as transport layer to serve incoming queries. Sample > C/Python LG clients are available in 'examples/lg'. A sample LG > server config is available in QUICKSTART. Request/Reply Looking > Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'. > + pmacctd: a single daemon can now listen for traffic on multiple > interfaces via a polling mechanism. This can be configured via a > pcap_interfaces_map feature (interface/pcap_interface can still be > used for backward compatiblity to listen on a single interface). The > map allows to define also ifindex mapping and capturing direction on > a per-interface basis. The map can be reloaded at runtime via a USR2 > signal and a sample map is in examples/pcap_interfaces.map.example. > + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and > kafka_partition_key knobs is introduced. The Kafka topic can contain > variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which > are all computed when data is purged to the backend. This feature is > in addition to the existing kafka_partition feature which allows to > rely on the built-in Kafka partitioning to assign data statically to > one partition or rely dynamically on the default partitioner. The > feature is courtesy by Corentin Neau / Codethink ( @weyfonk ). > + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone > represented as yyyy-MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives > the timestamps_rfc3339 knob can be used to enable this feature (left > disabled by default for backward compatibility). > + timestamps_utc: new knob to decode timestamps to UTC timezone even > if the Operating System is set to a different timezone. On the goods > of running a system set to UTC please read Q18 of FAQS. > + sfacctd: implemented mpls_label_top, mpls_label_bottom and > mpls_stack_depth primitives decoded from sFlow flow sample headers. > Thanks to David Barroso ( @dbarrosop ) for his support. > + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131 > (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX > option packets (these IEs were already supported when passed in flow > data). Also added support for IE 351 (dataLinkFrameSection) which > carries the initial portion of a sampled raw packet headers (a-la > sFlow). This was tested working against a Cisco NCS 5k platform. > + nfprobe plugin: added a new nfprobe_dont_cache knob allowing to > disable caching and summarisation of flows (essentially letting the > NetFlow/IPFIX probe behave like a sFlow probe). > + nfprobe plugin: added support for MPLS_LABEL_1, NetFlow v9/IPFIX IE > 70; improved support for BGP next-hop IE 18 and 63. Also support for > IE 130/131 vi NetFlow v9/IPFIX Options was added. > + sfprobe plugin: added sfprobe_source_ip knob to define the local IP > address from which sFlow datagrams are exported; improved support > for BGP next-hop. > + nfacctd, sfacctd, BGP, BMP, Streaming Telemetry daemons: on Linux, > if supported, use SO_REUSEPORT for the listening socket (added to > existing SO_REUSEADDR option). > + nfacctd, sfacctd: introduced new 'export_proto_sysid' primitive to > give visibility to NetFlow v5/v8 engine_id / NetFlow v9 source ID / > IPFIX Obs Domain ID / sFlow agentSubID. > + nfacctd, sfacctd: extended nDPI support to NetFlow v9/IPFIX packets > with IE 315 (dataLinkFrameSection) and sFlow v5 packets with header > section. > + nfacctd, sfacctd: extended custom primitives definition framework, > aggregate_primitives, to NetFlow v9/IPFIX packets with IE 315 > (dataLinkFrameSection) and sFlow v5 sampled headers section. > + nfacctd, sfacctd: added per-collector packets and bytes counts to > stats emitted via SIGUSR1. Also the output was made more formal (so > to be more easily parsed) and is documented in the UPGRADE notes. > + nfacctd, pmacctd, sfacctd: pcap_savefile_delay feature introduced > to sleep for the supplied amount of seconds before playing a given > pcap_savefile. Useful, for example, to let BGP/BMP sessions come up > so that routing data is available for correlation when processing > data in the trace. > + Kafka plugin: configuring statistics.interval.ms to a positive value > in a kafka_config_file makes now librdkafka log plenty of internal > metrics. > + BGP daemon: added support for Extended BGP Administrative Shutdown > Communication (draft-snijders-idr-rfc8203bis-00). > + BMP daemon: added support for draft-ietf-grow-bmp-adj-rib-out-01 and > draft-ietf-grow-bmp-loc-rib-01. As a result of that, Route Monitor > log messages now contain indication of is_out and is_filtered. > + BMP daemon: added support for stats reports 9, 10, 11, 12 and 13 and > descriptions for the different Peer Types and and Peer Down reasons. > Finally, indication of is_post is now making to Route Monitor log > messages. > + plugin_pipe_zmq: introduced plugin_pipe_zmq_hwm (high water mark) > knob to control the maximum amount of messages than can be stored in > the ZeroMQ queue. > + [ns]facctd_allow_file: the map is now made reloadable at runtime via > SIGUSR2 and accepts IPv4/IPv6 prefixes increasing its scale (before > it was only accepting individual IP addresses). > + pmacctd: added support for IPv6, MPLS for DLT_LINUX_SLL captures. > Thanks to David Barroso ( @dbarrosop ) for his support. > + uacctd: added a global 'direction' knob to give visibility of data > capturing direction, ie. in/out. Useful for pre_tag_map use. > + MySQL plugin: added sql_port knob in order to specify non-default > ports for connecting to the database. Patch is courtesy by Vadim > Tkachenko ( @vadimtk ). > ! fix, plugins: getppid() parent process health check improved so > to work in Docker environments not assuming anymore parent PID is > 1. Patch is courtesy by Hidde van der Heide ( @hvanderheide ). > ! fix, plugins: imposing a budget for received messages (100) so to > preserve fairness of other operations (ie. time keeping, bucketing, > reloading maps, etc.) and prevent starvations. > ! fix, plugins: retry when zmq_getsockopt() for ZMQ_EVENTS returns > EINTR. Thanks to Wouter de Jong for his support solving the issue. > ! fix, plugins: when executing triggers, the first argument passed to > execv() should be the path to the invoked executable to prevent > execv(3) to fail and return EFAULT on OpenBSD. Patch is courtesy > by @higgsd. > ! fix, BGP daemon: improved support of multiple capabilities per > optional parameter in the OPEN message. Also add-path capability is > now advertised if neighbor supports send/receive (previously it was > sent back on send only) of such capability. Thanks to Radu Anghel > ( @cozonac ) for his support. > ! fix, BGP daemon: upon route lookup, don't perform ADD-PATH logics if > no PATH-ID (even if ADD-PATH capability is announced by the peer). > Thanks to Camilo Cardona ( @jccardonar ) for his support solving the > issue. > ! fix, BGP daemon: wrong type 2 32-bit ASN Route Distinguisher was > defined in network.h. Thanks to Thomas Graf for reporting the issue. > ! fix, BGP, BMP daemons: lookup of BGP-LU entries is now performed > against the correct RIB. > ! fix, BMP daemon: the BMP thread is now made mutually exclusive with > the BGP one (until an use-case needs to run them both). This is to > potentially prevent BGP and BMP information to interfere with each > other when correlated. Also the 'bmp' keyword was added for *_as and > *_net config directives (ie. nfacctd_as, nfacctd_net). Thanks to > Juan Camilo Cardona ( @jccardonar ) for his support. > ! fix, BMP daemon: improved correlation of BMP data with traffic data > by supporting a replication use-case (the BMP exporter is a route > -server rather than an actual Edge Router) upon lookup. Thanks to > Juan Camilo Cardona ( @jccardonar ) for his support. > ! fix, BMP daemon: in bgp_peer_cmp() and bgp_peer_host_addr_cmp() the > comparison function has been changed from generic memcmp() to a more > specific host_addr_cmp() as paddings were giving issues. Thanks to > Juan Camilo Cardona ( @jccardonar ) for reporting the issue. > ! fix, BMP daemon: a pm_tdestroy call in bmp_peer_close() was leading > to SEGV under certain conditions by not NULL'ing all pointers. Thanks > to Juan Camilo Cardona ( @jccardonar ) for reporting the issue. > ! fix, nfacctd: prevent time calculations to underflow in cases in > which sysUptime < first or last flow switched timestamps in NetFlow > v5. Patch is courtesy by David Steinn Geirsson ( @dsgwork ). > ! fix, nfacctd: in the context of aggregate_primitives, now enforcing > terminating the zero when decoding variable-length IEs when applying > string semantics. > ! fix, nfprobe: changed ifIndex fields from u_int16_t to u_int32_t in > order to prevent overflows and aligning to the rest of structs. > ! fix, MySQL plugin: minor code revisions to restore compiling against > MariaDB 10.2. > ! fix, sql_common.c: increased read_SQLquery_from_file() buffer size > so that sql_table_schema can be fed with longer CREATE TABLE > statements. > ! fix, print, SQL plugins: post_tag, post_tag2 support was added to > sql_table and print_output_file. Also for Kafka, RabbitMQ plugins > kafka_topic and amqp_routing_key variables support was harmonized > with print and SQL plugins (ie. $pre_tag renamed to $tag), see > UPGRADE notes. > ! fix, SQL plugins: sql_startup_delay was not being honored when > sql_trigger_exec was defined without a sql_trigger_time resulting > in empty environment variables being passed to the triggered script. > Thanks to Johannes Maybaum for his support resolving the issue. > ! fix, pkt_handlers.c: tmp_asa_bi_flow value was ignored when applied > to a specific plugin. > ! fix, util.c: when data timestamp is not available, dynamic file and > table names variables were populated with a 1-Jan-1970 date. Now the > current timestamp is used instead as last resort. Patch is courtesy > by Ivan F. Martinez ( @ivanfmartinez ). > ! fix, addr.c: host_addr_mask_sa_cmp() and str_to_addr_mask() network > mask computation for IPv6 addresses was wrong. allow_file feature > was affected. > ! fix, build system: several patches committed to the build system to > simplify libraries probing, make sure to bail out upon error. Also > now a minimum required version is imposed to almost all libraries. > - --enable-threads / --disable-threads: removed the configure switch > that was allowing to compile pmacct even when no pthreads library was > available on a system. From now on support for threads is mandatory. > - BGP daemon: offline code, ie. bgp_daemon_offline_* config directives, > has been deprecated in favor of other approaches, ie. BGP Looking > Glass and BGP Xconnects. > - pkt_len_distrib: the primitive, which was meant to bucket packet / > flow / sample lengths in a distribution has been obsoleted. > > > NOTES. > See UPGRADE file. > > > Cheers, > Paolo > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
