[pmacct-discussion] sql_dont_try_update causes key collisions?

[Edward Henigin]

Hi Paolo,

When enabling sql_dont_try_update: true, I get these errors fairly
continuously:

Mar  3 11:33:30 server nfacctd[10661]: ERROR ( ip_dst/sqlite3 ): columns
peer_ip_src, iface_in, ip_dst, stamp_inserted are not unique#012
Mar  3 11:33:33 server nfacctd[10662]: ERROR ( ip_dst/sqlite3 ): columns
peer_ip_src, iface_in, ip_dst, stamp_inserted are not unique#012
Mar  3 11:33:37 server nfacctd[10663]: ERROR ( ip_dst/sqlite3 ): columns
peer_ip_src, iface_in, ip_dst, stamp_inserted are not unique#012
Mar  3 11:33:44 server nfacctd[10667]: ERROR ( ip_dst/sqlite3 ): columns
peer_ip_src, iface_in, ip_dst, stamp_inserted are not unique#012
Mar  3 11:33:47 server nfacctd[10668]: ERROR ( ip_dst/sqlite3 ): columns
peer_ip_src, iface_in, ip_dst, stamp_inserted are not unique#012

Any suggestions? Per the instructions, we do have these configured:

sql_refresh_time: 60
sql_history: 1m
sql_history_roundoff: m
nfacctd_time_new: true

After commenting out sql_dont_try_update: true there are no errors
operationally.
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] nfacctd - log missed sequence numbers?

[Edward Henigin]

Thank you Paolo!

But would it simply be safe to say that we are not missing any netflow data
if I never see the log line?

# fgrep syslog /etc/pmacct/nfacctd.conf
syslog: daemon
# fgrep -i expecting /var/log/daemon
#

That would be my working assumption.

Thanks,

Ed

On Sat, Feb 25, 2017 at 7:19 AM, Paolo Lucente <pa...@pmacct.net> wrote:

>
> Hi Ed,
>
> The log message produced is actually very simple:
>
> Log([..] expecting flow '%u' but received '%u' collector=%s:%u agent=%s:%u
> [..]);
>
> It's a start for some basic analysis but you can get false positives,
> for example due to out of order arrival of packets. In recent pmacct
> releases you have a new primitive, export_proto_seqno, precisely to
> report on sequence numbers. As it can be read in CONFIG-KEYS:
>
> export_proto_seqno reports about export protocol (NetFlow, sFlow, IPFIX)
> sequence number; due to its potential de-aggregation effect, two main
> use-cases are seen as use of this primitive:
>
> 1) if using a log type (de-)aggregation method, ie. for security,
>forensics, etc., in addition to existing primitives;
>
> 2) if using a reporting type aggregation method, it is recommended to
>split this primitive in a separate plugin instance instead for
>sequencing analysis.
>
> You fall in the use-case #2. You may instantiate a memory or print
> plugins setting the aggregate to 'peer_src_ip, export_proto_seqno'. This
> way you can perform a more contextual analysis over periods of time (ie.
> 1 min).
>
> Cheers,
> Paolo
>
> On Thu, Feb 23, 2017 at 11:09:19AM -0600, Edward Henigin wrote:
> > I see in the config keys for nfacctd that by default it checks sequence
> > numbers and will log an error if any are missing.
> >
> > [ nfacctd_disable_checks | sfacctd_disable_checks ] [GLOBAL, NO_PMACCTD]
> > Values
> >
> > [true|false]
> > Desc
> >
> > both nfacctd and sfacctd check health of incoming NetFlow/sFlow
> datagrams -
> > actually this is limited to just verifying sequence numbers progression.
> > You may want to disable such feature because of non-standard
> > implementations. By default checks are enabled
> >
> > (default: false)
> >
> >
> > My question: what does that log message look like? I suspect I'm losing
> > flows and I want to check the logs for evidence. I looked in
> src/nfacctd.c
> > to see if I could tell what the syslog message would look like but I
> can't
> > figure out where it's checking the sequence numbers for continuity and
> > logging an error on lost data.
> >
> > Thanks,
> >
> > Ed
>
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] nfacctd - log missed sequence numbers?

[Edward Henigin]

I see in the config keys for nfacctd that by default it checks sequence
numbers and will log an error if any are missing.

[ nfacctd_disable_checks | sfacctd_disable_checks ] [GLOBAL, NO_PMACCTD]
Values

[true|false]
Desc

both nfacctd and sfacctd check health of incoming NetFlow/sFlow datagrams -
actually this is limited to just verifying sequence numbers progression.
You may want to disable such feature because of non-standard
implementations. By default checks are enabled

(default: false)


My question: what does that log message look like? I suspect I'm losing
flows and I want to check the logs for evidence. I looked in src/nfacctd.c
to see if I could tell what the syslog message would look like but I can't
figure out where it's checking the sequence numbers for continuity and
logging an error on lost data.

Thanks,

Ed
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] SQLite3 - issues, advice?

[Edward Henigin]

I guess the big question is, is SQLite3 really appropriate to use in a
"light weight" but general purpose way?

Is it fast enough for real-world use? Are there bugs or limitations that
prevent it from really working well?

My particular issues with it, given this configuration:

DROP TABLE host_srcif;
CREATE TABLE host_srcif (
peer_ip_src CHAR(15) NOT NULL DEFAULT '0.0.0.0',
iface_in INT UNSIGNED NOT NULL DEFAULT 0,
ip_dst CHAR(15) NOT NULL DEFAULT '0.0.0.0',
packets INT UNSIGNED NOT NULL,
bytes BIGINT UNSIGNED NOT NULL,
stamp_inserted DATETIME NOT NULL,
stamp_updated DATETIME,
PRIMARY KEY (peer_ip_src, iface_in, ip_dst, stamp_inserted)
);

! nfacctd configuration file
plugins: sqlite3
!
aggregate: peer_src_ip,in_iface,dst_host
sql_optimize_clauses: true
sql_db: /var/opt/pmacct/g2.db
sql_table: host_srcif
!
sql_refresh_time: 120
sql_history: 2m
sql_history_roundoff: m
sql_dont_try_update: true
nfacctd_time_new: true
!

Produced "columns are not unique" errors at a rate of 31 per second:

ERROR ( default/sqlite3 ): columns peer_ip_src, iface_in, ip_dst,
stamp_inserted are not unique#012

combined with "database is locked" errors at around 16 per second:

ERROR ( default/sqlite3 ): database is locked#012

so I commented out "sql_dont_try_update: true", and the "columns are not
unique" errors are gone, but the "database is locked" errors continue.

BUT, even despite those errors, there are lots of records in the database:

sqlite> select stamp_inserted, count(*) from host_srcif group by
stamp_inserted;
2017-02-09 13:42:00|143277
2017-02-09 13:50:00|135071
2017-02-09 15:50:00|122825
2017-02-09 15:52:00|118091

The 13:42:00 and 13:50:00 periods were when nfacctd produced the "columns
are not unique" error, and the 15:50:00 and 15:52:00 periods were when
nfacctd produced the "database is locked" error. So despite the errors,
records are certainly making it in.

That said, the IO load on the server is massively higher in the 2nd
scenario.

So what do I do? Apparently data can go in despite errors, but data must be
getting lost. How do I get all the data to go in without losing any?

And finally my last question. I have little experience with SQLite, I
understand it's a file-based database. I noticed in the 2nd period
(15:50:00+, above) that if I went into the SQLite CLI to run queries (just
SELECTs, just read-only), that I often could not perform my selects because
it said the database was locked. Is it true that an SQLite database cannot
be read while a write is in progress? That would seem to limit the
usefulness of the system. Is there a config parameter in pmacct/nfacctd
that might make it "better" somehow?

Thanks,

Ed
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pmacct 1.5.2 released !

[Edward Henigin]

Hi Paolo,

I'm seeing a significant memory leak in the print plugin in 1.5.2.

# service nfacctd restart
Shutting down sflow collection services (nfacctd): [  OK  ]
Starting sflow collection services (nfacctd): INFO ( default/core ):
Daemonizing. Hmm, bye bye screen.
   [  OK  ]
# pgrep nfacctd
874
876
# while [ 1 ]; do sleep 60; date; ps uwp 876; done
Thu Jan  7 15:53:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.1  0.6 115276 104432 ?   S15:52   0:01 nfacctd:
Print Plugin [default]
Thu Jan  7 15:54:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.0  0.6 120884 109980 ?   S15:52   0:01 nfacctd:
Print Plugin [default]
Thu Jan  7 15:55:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.0  0.7 128408 117608 ?   S15:52   0:02 nfacctd:
Print Plugin [default]
Thu Jan  7 15:56:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.1  0.8 147680 136856 ?   S15:52   0:03 nfacctd:
Print Plugin [default]
Thu Jan  7 15:57:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.1  0.9 160748 149944 ?   S15:52   0:03 nfacctd:
Print Plugin [default]
Thu Jan  7 15:58:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.1  1.0 180548 169668 ?   S15:52   0:04 nfacctd:
Print Plugin [default]
Thu Jan  7 15:59:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.1  1.1 203384 192604 ?   S15:52   0:05 nfacctd:
Print Plugin [default]
Thu Jan  7 16:00:56 CST 2016
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root   876  1.3  1.5 272024 261120 ?   S15:52   0:07 nfacctd:
Print Plugin [default]
^C
# strings - /usr/sbin/nfacctd | fgrep 1.5
NetFlow Accounting Daemon, nfacctd 1.5.2
/opt/pmacct/pmacct-1.5.2/src
/opt/pmacct/pmacct-1.5.2/src/nfprobe_plugin
/opt/pmacct/pmacct-1.5.2/src/sfprobe_plugin
/opt/pmacct/pmacct-1.5.2/src/bgp
/opt/pmacct/pmacct-1.5.2/src/tee_plugin
/opt/pmacct/pmacct-1.5.2/src/isis
/opt/pmacct/pmacct-1.5.2/src/bmp

Any thoughts? Suggestions?

Thanks,

Ed

On Mon, Sep 7, 2015 at 1:01 PM, Paolo Lucente  wrote:

> VERSION.
> 1.5.2
>
>
> DESCRIPTION.
> pmacct is a small set of passive network monitoring tools to account,
> classify,
> aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable
> architecture
> allows to store collected data into memory tables, RDBMS (MySQL,
> PostgreSQL,
> SQLite), noSQL databases (MongoDB, BerkeleyDB) AMQP message exchanges
> (RabbitMQ) and flat-files and export through NetFlow or sFlow protocols to
> remote collectors. pmacct offers customizable historical data breakdown,
> BGP
> and IGP correlation, BMP stats and events, GeoIP lookups, sampling and
> renormalization, filtering, tagging and triggers. Libpcap, Linux
> Netlink/ULOG,
> sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are supported. Replication of
> incoming NetFlow and sFlow datagrams is also available. Statistics can be
> easily
> exported to tools like RRDtool, Net-SNMP, MRTG, GNUPlot and Cacti.
>
>
> HOMEPAGE.
> http://www.pmacct.net/
>
>
> DOWNLOAD.
> http://www.pmacct.net/pmacct-1.5.2.tar.gz
>
>
> CHANGELOG.
> + Introduced support for a RabbitMQ broker to be used for queueing and
>   data exchange between Core Process and plugins. This is in alternative to
>   the home-grown circular queue implementation. plugin_pipe_amqp directive,
>   along with all other plugin_pipe_amqp_* directives, can be set globally
>   or apply on a per plugin basis (ie. it is a valid scenario, if multiple
>   plugins are instantiated, that some make use of home-grown queueing,
>   while others use RabbitMQ based queueing).
> + Introducting support for Maximind GeoIP v2 (libmaxminddb) library: if
>   pmacct is compiled with --enable-geoipv2, this defines full pathname to
>   a Maxmind GeoIP database v2 (libmaxminddb) Only the binary database
>   format is supported (ie. it is not possible to load distinct CSVs for
>   IPv4 and IPv6 addresses).
> + Introduced infrastructure for sFlow counters and support specifically for
>   generic, ethernet and vlan counters. Counters are exported in JSON format
>   to files, specified via sfacctd_counter_file. The supplied filename can
>   contain as variable the sFlow agent IP address.
> + Introduced a new thread_stack config directive to allow to modify the
>   thread stack size. Natanael Copa reported that some libc implementations,
>   ie. musl libc, may set a stack size that is too small by default.
> + Introduced networks_file_no_lpm feature: it applies when aggregation
>   method includes src_net and/or dst_net and nfacctd_net (or equivalents)
>   and/or nfacctd_as_new 

Re: [pmacct-discussion] pmacct 1.5.2 released !

[Edward Henigin]

Hi Paolo,

You are wise my friend.

1) Yes just upgraded to 1.5.2. (Was having problems in 1.5.1 where the
input ifindex was getting mangled.)
2) Correct - was not seeing memory leak before upgrade.
3) If I leave running, eventually it consumes all available memory on the
server. From syslog prior to service restart,
Jan  7 15:05:05 packet1 nfacctd[16988]: WARN ( default/print ): Unable to
fork writer: Cannot allocate memory
4) config:

aggregate: peer_src_ip,in_iface,dst_host,dst_mask
plugins: print
nfacctd_time_new: true
plugin_buffer_size: 10240
imt_buckets: 157
imt_mem_pools_number: 256
imt_mem_pools_size: 32768
syslog: daemon
daemonize: true
!
print_refresh_time: 64
print_history: 1m
print_output: csv
print_output_file: /opt/pmacct/data/nfacct-%Y%m%d-%H%M.csv
! print_trigger_exec: /opt/gravitizer/bin/monitor.pl
print_output_file_append: true
!
nfacctd_port: 2055

Thanks,

Ed

On Thu, Jan 7, 2016 at 4:20 PM, Paolo Lucente <pa...@pmacct.net> wrote:

> Hi Ed,
>
> You mean you kind of just upgraded to 1.5.2 and you were not seeing
> the same before the upgrade? If you leave it running, does it grow
> undefinitely or at some stage it looks it reaches a stable memory
> footprint? Especially if using malloc()ed structures, ie. derived
> by BGP primitives, this could be a valid scenario. Finally, can you
> share your config?
>
> Cheers,
> Paolo
>
> On Thu, Jan 07, 2016 at 04:04:09PM -0600, Edward Henigin wrote:
> > Hi Paolo,
> >
> > I'm seeing a significant memory leak in the print plugin in 1.5.2.
> >
> > # service nfacctd restart
> > Shutting down sflow collection services (nfacctd): [  OK  ]
> > Starting sflow collection services (nfacctd): INFO ( default/core ):
> > Daemonizing. Hmm, bye bye screen.
> >[  OK  ]
> > # pgrep nfacctd
> > 874
> > 876
> > # while [ 1 ]; do sleep 60; date; ps uwp 876; done
> > Thu Jan  7 15:53:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.1  0.6 115276 104432 ?   S15:52   0:01 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 15:54:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.0  0.6 120884 109980 ?   S15:52   0:01 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 15:55:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.0  0.7 128408 117608 ?   S15:52   0:02 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 15:56:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.1  0.8 147680 136856 ?   S15:52   0:03 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 15:57:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.1  0.9 160748 149944 ?   S15:52   0:03 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 15:58:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.1  1.0 180548 169668 ?   S15:52   0:04 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 15:59:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.1  1.1 203384 192604 ?   S15:52   0:05 nfacctd:
> > Print Plugin [default]
> > Thu Jan  7 16:00:56 CST 2016
> > USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> > root   876  1.3  1.5 272024 261120 ?   S15:52   0:07 nfacctd:
> > Print Plugin [default]
> > ^C
> > # strings - /usr/sbin/nfacctd | fgrep 1.5
> > NetFlow Accounting Daemon, nfacctd 1.5.2
> > /opt/pmacct/pmacct-1.5.2/src
> > /opt/pmacct/pmacct-1.5.2/src/nfprobe_plugin
> > /opt/pmacct/pmacct-1.5.2/src/sfprobe_plugin
> > /opt/pmacct/pmacct-1.5.2/src/bgp
> > /opt/pmacct/pmacct-1.5.2/src/tee_plugin
> > /opt/pmacct/pmacct-1.5.2/src/isis
> > /opt/pmacct/pmacct-1.5.2/src/bmp
> >
> > Any thoughts? Suggestions?
> >
> > Thanks,
> >
> > Ed
> >
> > On Mon, Sep 7, 2015 at 1:01 PM, Paolo Lucente <pa...@pmacct.net> wrote:
> >
> > > VERSION.
> > > 1.5.2
> > >
> > >
> > > DESCRIPTION.
> > > pmacct is a small set of passive network monitoring tools to account,
> > > classify,
> > > aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable
> > > architecture
> > > allows to store collected data into memory tables, RDBMS (MySQL,
> &

[pmacct-discussion] multiple nfacctd files being written

[Edward Henigin]

Hi Paolo,

Running pmacct 1.5.2, simply using the print plugin, I'm getting multiple
files coming out at the same time with filenames suggesting they should be
coming out at different times, and sometimes very strange filenames like
the data is very old.

ls output:

(root) packet1:/opt/pmacct/data# ls -lt | head
total 6649800
-rw---   1 root root 1785873 Oct 21 15:20 nfacct-20151021-1450.csv
-rw---   1 root root 2740509 Oct 21 15:20 nfacct-20151021-1518.csv
-rw---   1 root root 2597403 Oct 21 15:20 nfacct-20151021-1519.csv
-rw---   1 root root 2778987 Oct 21 15:19 nfacct-20151021-1517.csv
-rw---   1 root root 3017902 Oct 21 15:18 nfacct-20151021-1516.csv
-rw---   1 root root 2860626 Oct 21 15:17 nfacct-20151021-1515.csv
-rw---   1 root root 3013418 Oct 21 15:16 nfacct-20151021-1514.csv
-rw---   1 root root 3433555 Oct 21 15:15 nfacct-20151021-1513.csv
-rw---   1 root root 2752513 Oct 21 15:14 nfacct-20151021-1512.csv

and timestamps to the second:

(root) packet1:/opt/pmacct/data# perl -le 'foreach (qw{1512 1513 1514 1515
1516 1517 1518 1450}){print("$_: ".
localtime((stat("nfacct-20151021-$_.csv"))[9]))}'
1512: Wed Oct 21 15:14:18 2015
1513: Wed Oct 21 15:15:22 2015
1514: Wed Oct 21 15:16:26 2015
1515: Wed Oct 21 15:17:30 2015
1516: Wed Oct 21 15:18:34 2015
1517: Wed Oct 21 15:19:38 2015
1518: Wed Oct 21 15:20:42 2015
1450: Wed Oct 21 15:20:42 2015

Where is filename "...-1450" coming from, and why is it coming out at the
same time as -1518?

*Configuration:*

! nfacctd configuration file
aggregate: peer_src_ip,in_iface,dst_host,dst_mask
plugins: print
plugin_buffer_size: 10240
imt_buckets: 157
imt_mem_pools_number: 256
imt_mem_pools_size: 32768
syslog: daemon
daemonize: true
!
print_refresh_time: 64
print_history: 1m
print_output: csv
print_output_file: /opt/pmacct/data/nfacct-%Y%m%d-%H%M.csv
print_output_file_append: true
!
nfacctd_port: 2055

*Syslog:*

Oct 21 15:15:21 packet1 nfacctd[558]: INFO ( default/print ): *** Purging
cache - START (PID: 558) ***
Oct 21 15:15:22 packet1 nfacctd[558]: INFO ( default/print ): *** Purging
cache - END (PID: 558, QN: 80643/80847, ET: 1) ***
Oct 21 15:16:25 packet1 nfacctd[593]: INFO ( default/print ): *** Purging
cache - START (PID: 593) ***
Oct 21 15:16:26 packet1 nfacctd[593]: INFO ( default/print ): *** Purging
cache - END (PID: 593, QN: 74092/74503, ET: 1) ***
Oct 21 15:17:29 packet1 nfacctd[651]: INFO ( default/print ): *** Purging
cache - START (PID: 651) ***
Oct 21 15:17:30 packet1 nfacctd[651]: INFO ( default/print ): *** Purging
cache - END (PID: 651, QN: 74211/74621, ET: 1) ***
Oct 21 15:18:33 packet1 nfacctd[690]: INFO ( default/print ): *** Purging
cache - START (PID: 690) ***
Oct 21 15:18:34 packet1 nfacctd[690]: INFO ( default/print ): *** Purging
cache - END (PID: 690, QN: 71470/72233, ET: 1) ***
Oct 21 15:19:37 packet1 nfacctd[739]: INFO ( default/print ): *** Purging
cache - START (PID: 739) ***
Oct 21 15:19:38 packet1 nfacctd[739]: INFO ( default/print ): *** Purging
cache - END (PID: 739, QN: 69195/73637, ET: 1) ***
Oct 21 15:20:41 packet1 nfacctd[779]: INFO ( default/print ): *** Purging
cache - START (PID: 779) ***
Oct 21 15:20:42 packet1 nfacctd[779]: INFO ( default/print ): *** Purging
cache - END (PID: 779, QN: 67848/83867, ET: 1) ***


Ed
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] multiple nfacctd files being written

[Edward Henigin]

And sometimes the filenames look correct but again they all come out at the
same time:

(root) packet1:/opt/pmacct/data# ls -lt | head
total 6670424
-rw---   1 root root 3005237 Oct 21 15:28 nfacct-20151021-1525.csv
-rw---   1 root root 1461133 Oct 21 15:28 nfacct-20151021-1527.csv
-rw---   1 root root 2292406 Oct 21 15:28 nfacct-20151021-1526.csv
-rw---   1 root root 3505033 Oct 21 15:27 nfacct-20151021-1524.csv
-rw---   1 root root 2178942 Oct 21 15:26 nfacct-20151021-1523.csv
-rw---   1 root root 2551924 Oct 21 15:24 nfacct-20151021-1522.csv
-rw---   1 root root 3633218 Oct 21 15:23 nfacct-20151021-1521.csv
-rw---   1 root root 2407956 Oct 21 15:22 nfacct-20151021-1520.csv
-rw---   1 root root 2669403 Oct 21 15:21 nfacct-20151021-1519.csv
(root) packet1:/opt/pmacct/data# perl -le 'foreach (qw{1519 1520 1521 1522
1523 1524 1525 1526 1527}){print("$_: ".
localtime((stat("nfacct-20151021-$_.csv"))[9]))}'
1519: Wed Oct 21 15:21:46 2015
1520: Wed Oct 21 15:22:50 2015
1521: Wed Oct 21 15:23:53 2015
1522: Wed Oct 21 15:24:58 2015
1523: Wed Oct 21 15:26:02 2015
1524: Wed Oct 21 15:27:06 2015
1525: Wed Oct 21 15:28:10 2015
1526: Wed Oct 21 15:28:10 2015
1527: Wed Oct 21 15:28:10 2015

Seems related?


On Wed, Oct 21, 2015 at 3:28 PM, Edward Henigin <e...@eaohana.com> wrote:

> Hi Paolo,
>
> Running pmacct 1.5.2, simply using the print plugin, I'm getting multiple
> files coming out at the same time with filenames suggesting they should be
> coming out at different times, and sometimes very strange filenames like
> the data is very old.
>
> ls output:
>
> (root) packet1:/opt/pmacct/data# ls -lt | head
> total 6649800
> -rw---   1 root root 1785873 Oct 21 15:20 nfacct-20151021-1450.csv
> -rw---   1 root root 2740509 Oct 21 15:20 nfacct-20151021-1518.csv
> -rw---   1 root root 2597403 Oct 21 15:20 nfacct-20151021-1519.csv
> -rw---   1 root root 2778987 Oct 21 15:19 nfacct-20151021-1517.csv
> -rw---   1 root root 3017902 Oct 21 15:18 nfacct-20151021-1516.csv
> -rw---   1 root root 2860626 Oct 21 15:17 nfacct-20151021-1515.csv
> -rw---   1 root root 3013418 Oct 21 15:16 nfacct-20151021-1514.csv
> -rw---   1 root root 3433555 Oct 21 15:15 nfacct-20151021-1513.csv
> -rw---   1 root root 2752513 Oct 21 15:14 nfacct-20151021-1512.csv
>
> and timestamps to the second:
>
> (root) packet1:/opt/pmacct/data# perl -le 'foreach (qw{1512 1513 1514 1515
> 1516 1517 1518 1450}){print("$_: ".
> localtime((stat("nfacct-20151021-$_.csv"))[9]))}'
> 1512: Wed Oct 21 15:14:18 2015
> 1513: Wed Oct 21 15:15:22 2015
> 1514: Wed Oct 21 15:16:26 2015
> 1515: Wed Oct 21 15:17:30 2015
> 1516: Wed Oct 21 15:18:34 2015
> 1517: Wed Oct 21 15:19:38 2015
> 1518: Wed Oct 21 15:20:42 2015
> 1450: Wed Oct 21 15:20:42 2015
>
> Where is filename "...-1450" coming from, and why is it coming out at the
> same time as -1518?
>
> *Configuration:*
>
> ! nfacctd configuration file
> aggregate: peer_src_ip,in_iface,dst_host,dst_mask
> plugins: print
> plugin_buffer_size: 10240
> imt_buckets: 157
> imt_mem_pools_number: 256
> imt_mem_pools_size: 32768
> syslog: daemon
> daemonize: true
> !
> print_refresh_time: 64
> print_history: 1m
> print_output: csv
> print_output_file: /opt/pmacct/data/nfacct-%Y%m%d-%H%M.csv
> print_output_file_append: true
> !
> nfacctd_port: 2055
>
> *Syslog:*
>
> Oct 21 15:15:21 packet1 nfacctd[558]: INFO ( default/print ): *** Purging
> cache - START (PID: 558) ***
> Oct 21 15:15:22 packet1 nfacctd[558]: INFO ( default/print ): *** Purging
> cache - END (PID: 558, QN: 80643/80847, ET: 1) ***
> Oct 21 15:16:25 packet1 nfacctd[593]: INFO ( default/print ): *** Purging
> cache - START (PID: 593) ***
> Oct 21 15:16:26 packet1 nfacctd[593]: INFO ( default/print ): *** Purging
> cache - END (PID: 593, QN: 74092/74503, ET: 1) ***
> Oct 21 15:17:29 packet1 nfacctd[651]: INFO ( default/print ): *** Purging
> cache - START (PID: 651) ***
> Oct 21 15:17:30 packet1 nfacctd[651]: INFO ( default/print ): *** Purging
> cache - END (PID: 651, QN: 74211/74621, ET: 1) ***
> Oct 21 15:18:33 packet1 nfacctd[690]: INFO ( default/print ): *** Purging
> cache - START (PID: 690) ***
> Oct 21 15:18:34 packet1 nfacctd[690]: INFO ( default/print ): *** Purging
> cache - END (PID: 690, QN: 71470/72233, ET: 1) ***
> Oct 21 15:19:37 packet1 nfacctd[739]: INFO ( default/print ): *** Purging
> cache - START (PID: 739) ***
> Oct 21 15:19:38 packet1 nfacctd[739]: INFO ( default/print ): *** Purging
> cache - END (PID: 739, QN: 69195/73637, ET: 1) ***
> Oct 21 15:20:41 packet1 nfacctd[779]: INFO ( default/print ): *** Purging
> cache - START (PID: 779) ***
> Oct 21 15:20:42 packet1 nfacctd[779]: INFO ( default/print ): *** Purging
> cache - END (PID: 779, QN: 67848/83867, ET: 1) ***
>
>
> Ed
>
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Tips on dealing with overflowing 32-bit fields?

[Edward Henigin]

Hi y'all,

I'm hoping that someone has some experience that might help.

I'm using nfacctd to collect flows from a Cisco RSP720. After banging my
head against the keyboard for a few days, I realized I should have
configured pmacct with --enable-64bit. After re-building with that,
accuracy is dramatically improved, but I'm still finding bytes being
under-reported in numerous intervals.

I believe the problem I'm running into is that the RSP720 is collecting its
data in a 32-bit field and that field is wrapping -- or, the netflow v5
packet uses 32 bits for bytes, and it's wrapping on export. In any case, I
think the byte count is lost before it leaves the Cisco.

I'm using the interface-destination flow mask. I've tried using
interface-destination-source but that causes some CPU load on the Cisco
and flow creation failures.

To have the highest resolution and to minimize the risk of netflow creation
failures, all the timers are set to the lowest:

 enable timeout  packet threshold
 -- ---  
normal aging true   32 N/A
fast aging   true   32 7
long aging   true   64 N/A

2**32 bytes in 64 seconds is only 537 Mbps. Doesn't work very well for
multi-gigabit traffic servers.

Does anyone have any ideas on how to reduce or eliminate counter wrap on
the Cisco side for the bytes counter?

Many thanks,

Ed
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Tips on dealing with overflowing 32-bit fields?

[Edward Henigin]

Hi Brent,

From what I've read on the SUP720 and RSP720, they do not support netflow
sampling for creation/update, they only support sampling on export. I
think the way that works is they randomly skip exporting some of the
records. So I don't think that would actually impact the overflow issue...
but I guess I should double-check that as an option...

Thanks for the idea,

Ed


On Fri, May 24, 2013 at 3:47 PM, Brent Van Dussen b...@microsoft.com wrote:

  Hi Ed,

 ** **

 Does your RSP720 support sampled netflow by chance?

 ** **

 -Brent

 ** **

 *From:* pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] *On
 Behalf Of *Edward Henigin
 *Sent:* Friday, May 24, 2013 1:40 PM
 *To:* pmacct-discussion@pmacct.net
 *Subject:* [pmacct-discussion] Tips on dealing with overflowing 32-bit
 fields?

 ** **

 Hi y'all,

 ** **

 I'm hoping that someone has some experience that might help.

 ** **

 I'm using nfacctd to collect flows from a Cisco RSP720. After banging my
 head against the keyboard for a few days, I realized I should have
 configured pmacct with --enable-64bit. After re-building with that,
 accuracy is dramatically improved, but I'm still finding bytes being
 under-reported in numerous intervals.

 ** **

 I believe the problem I'm running into is that the RSP720 is collecting
 its data in a 32-bit field and that field is wrapping -- or, the netflow v5
 packet uses 32 bits for bytes, and it's wrapping on export. In any case, I
 think the byte count is lost before it leaves the Cisco.

 ** **

 I'm using the interface-destination flow mask. I've tried using
 interface-destination-source but that causes some CPU load on the Cisco
 and flow creation failures.

 ** **

 To have the highest resolution and to minimize the risk of netflow
 creation failures, all the timers are set to the lowest:

 ** **

  enable timeout  packet threshold

  -- ---  

 normal aging true   32 N/A

 fast aging   true   32 7

 long aging   true   64 N/A

 ** **

 2**32 bytes in 64 seconds is only 537 Mbps. Doesn't work very well for
 multi-gigabit traffic servers.

 ** **

 Does anyone have any ideas on how to reduce or eliminate counter wrap on
 the Cisco side for the bytes counter?

 ** **

 Many thanks,

 ** **

 Ed

 ___
 pmacct-discussion mailing list
 http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists