I have found that every NetFlow packet arriving to nprobe elicits this message:
[collect.c:326] Received flow with invalid count [sentPkts: 0][sentOctets: 0]: discarded This starts after the templates are received. I'm looking for information online, and finding next to none. When I look at the counters in Wireshark, the numbers do make sense (in the interpreted packets), so I am starting to think it is a something with the way nprobe decodes the packets based on the ASA 5510 templates. Have you seen this before? Any ideas where to look? Thank you, Pablo On Fri, Jan 3, 2014 at 5:02 PM, Pablo Destéfanis <pdestefa...@gmail.com>wrote: > Hello guys, > > I have done a bit more research to check if ASA was sending the template > info. I have set the template timeout at 5 minutes and also directed the > output to Wireshark. > I see first the uninterpreted flows, and then the interpreted NetFlow > packets, including source and destination address, pre/post NAT, etc. While > I did not search for the packets with templates, I'm assuming they were > detected by Wireshark, and then the output was reinterpreted. > > Tried with nrprobe, and flow collection shows processed flows after a > while (initially only shows collected packets) > Flow collection: [collected pkts: 889][processed flows: 12697] > > Still, when connecting from ntopng I get no information. The commands I'm > using are: > > nprobe --zmq "tcp://*:5556" -i none -n none --collector-port 2055 > --city-list /usr/local/nprobe/GeoLiteCity.dat > ntopng -i tcp://127.0.0.1:5556 -m 10.1.0.0/16 > > I see the ports open, and I see ntopng connected to nprobe: > > tcp 0 0 0.0.0.0:5556 0.0.0.0:* > LISTEN 14596/nprobe > tcp 0 0 0.0.0.0:3000 0.0.0.0:* > LISTEN 14602/ntopng > tcp 0 0 127.0.0.1:35085 127.0.0.1:5556 > ESTABLISHED 14602/ntopng > tcp 0 0 127.0.0.1:40789 127.0.0.1:6379 > ESTABLISHED 14602/ntopng > tcp 0 0 127.0.0.1:5556 127.0.0.1:35085 > ESTABLISHED 14596/nprobe > > Upon cancelling the nprobe I see this: > > 03/Jan/2014 14:41:31 [nprobe.c:369] Received shutdown request... > 03/Jan/2014 14:41:31 [cache.c:850] Redis Cache [0 total/0.0 get/sec][0 > total/0.0 set/sec] > 03/Jan/2014 14:41:31 [nprobe.c:2045] Processed packets: 0 (max bucket > search: 0) > 03/Jan/2014 14:41:31 [nprobe.c:2028] Fragment queue length: 0 > 03/Jan/2014 14:41:31 [nprobe.c:2054] Flow export stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 03/Jan/2014 14:41:31 [nprobe.c:2061] Flow collection: [collected pkts: > 3963][processed flows: 67041] > 03/Jan/2014 14:41:31 [nprobe.c:2064] Flow drop stats: [0 bytes/0 pkts][0 > flows] > 03/Jan/2014 14:41:31 [nprobe.c:2069] Total flow stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > > Upon cancelling ntopng I see this: > > ^C03/Jan/2014 14:41:37 [main.cpp:37] Shutting down... > 03/Jan/2014 14:41:39 [ProtoStats.cpp:35] [IPv4] 0 B/0.00 Packets > 03/Jan/2014 14:41:39 [ProtoStats.cpp:35] [IPv6] 0 B/0.00 Packets > 03/Jan/2014 14:41:39 [ProtoStats.cpp:35] [ARP] 0 B/0.00 Packets > 03/Jan/2014 14:41:39 [ProtoStats.cpp:35] [MPLS] 0 B/0.00 Packets > 03/Jan/2014 14:41:39 [ProtoStats.cpp:35] [Other] 0 B/0.00 Packets > 03/Jan/2014 14:41:39 [Ntop.cpp:590] Interface > collector@127.0.0.1:5556[running: 0] > 03/Jan/2014 14:41:39 [main.cpp:55] Deleted PID /var/tmp/ntopng.pid [rc: 0] > 03/Jan/2014 14:41:40 [HTTPserver.cpp:374] HTTP server terminated > 03/Jan/2014 14:41:40 [AddressResolution.cpp:187] Address resolution stats > [0 resolved][0 failures] > > I am starting to believe that the issue is still with nprobe. Just in > case, I'm attaching a capture file before the templates (NF100-200.pcapng) > and after the ASA sent the templates (NF200-300.pcapng) > > Thank you for any ideas, > > Pablo > > > > > > > > > > > On Sat, Dec 14, 2013 at 1:07 PM, Luca Deri <d...@ntop.org> wrote: > >> Hi Pablo >> >> On 13 Dec 2013, at 22:59, Pablo Destéfanis <pdestefa...@gmail.com> wrote: >> >> Hello, >> >> I have ntopng running fine capturing packets redirected from my ASA via a >> SNAP port. Since I cannot use this setup in a VM (now it is running in real >> hardware), I would like to use either ntopng or nprobe as a Netflow >> collector. >> >> I have confirmed that Netflow flows are coming from the ASA unit on >> UDP:2055 with tcpdump >> >> I was unable to find a way/command to start ntopng as a pure Netflow >> collector (listening to UDP:2055), tried the following options with >> different errors: >> >> ntopng -i "tcp://127.0.0.1:2055" [no info displayed in the web >> interface] >> ntopng -i "udp://127.0.0.1:2055" [ERROR: could not open pcap file: udp:// >> 127.0.0.1:2055: No such file or directory] >> >> For what I read at >> http://www.ntop.org/nprobe/why-nprobejsonzmq-instead-of-native-sflownetflow-support-in-ntopng/it >> seems that ntopng will not consume NetFlow flows, but rather ZQM flows >> from nprobe. >> >> yes >> >> >> I have tried that setup with nprobe v.6.15.131213, and while I get it to >> listen (shows "Flow collector listening on port 2055 (IPv4/v6)"), and I see >> port TCP:5556 open, ntopng does not seem to consume/connect to nprobe, in >> spite of stating "Collecting flows..." after initialization. >> >> The web interface for ntopng reads "No packet has been received yet on >> interface collector@127.0.0.1:5556", which is confusing, as I thought >> that actually ntopng would poll packets from nprobe as stated in the >> document referenced above >> >> >> I believe your ASA device is not sending flow templates but just flow >> data. >> >> Please check as this is a common problem with these devices >> >> Luca >> >> >> Command line used: >> ntopng -i "nprobe-collector.lua@tcp://127.0.0.1:5556" >> nprobe --zmq "tcp://*:5556" -i none -n none --collector-port 2055 >> --city-list /usr/local/nprobe/GeoLiteCity.dat >> >> Note: just ran nprobe with -b 1 and apparently is not getting any data >> from the Netflow flows (checked after a few minutes). It collecting >> packets, but not processing any: >> >> 13/Dec/2013 16:39:25 [nprobe.c:2178] Average traffic: [0.00 pps][0 b/sec] >> 13/Dec/2013 16:39:25 [nprobe.c:2185] Current traffic: [0.00 pps][0 b/sec] >> 13/Dec/2013 16:39:25 [nprobe.c:2191] Current flow export rate: [0.0 >> flows/sec] >> 13/Dec/2013 16:39:25 [nprobe.c:2194] Flow drops: [export queue too >> long=0][too many flows=0] >> 13/Dec/2013 16:39:25 [nprobe.c:2198] Export Queue: 0/512000 [0.0 %] >> 13/Dec/2013 16:39:25 [nprobe.c:2203] Flow Buckets: >> [active=0][allocated=0][toBeExported=0] >> 13/Dec/2013 16:39:25 [cache.c:850] Redis Cache [0 total/0.0 get/sec][0 >> total/0.0 set/sec] >> 13/Dec/2013 16:39:25 [nprobe.c:2221] Collector Threads: [644 pkts@0] >> 13/Dec/2013 16:39:25 [nprobe.c:2045] Processed packets: 0 (max bucket >> search: 0) >> 13/Dec/2013 16:39:25 [nprobe.c:2028] Fragment queue length: 0 >> 13/Dec/2013 16:39:25 [nprobe.c:2054] Flow export stats: [0 bytes/0 >> pkts][0 flows/0 pkts sent] >> 13/Dec/2013 16:39:25 [nprobe.c:2061] Flow collection: [collected pkts: >> 644][processed flows: 0] <------- See here >> 13/Dec/2013 16:39:25 [nprobe.c:2064] Flow drop stats: [0 bytes/0 >> pkts][0 flows] >> 13/Dec/2013 16:39:25 [nprobe.c:2069] Total flow stats: [0 bytes/0 >> pkts][0 flows/0 pkts sent] >> >> Versions: >> nProbe.x86_64 6.15.131213-3810 >> @ntop >> ntopng.x86_64 1.1.1_7107-7107 >> @ntop >> ntopng-data.x86_64 1.1.1_7107-7107 >> @ntop >> pfring.x86_64 5.6.2-7113 >> @ntop >> ntopng-data.x86_64 _7113-7113 >> ntop >> >> Based on this: >> >> Is it possible to use ntopng to receive Netflow flows from an ASA unit? >> Can you provide an example command line? >> Is the nprobe/ntopng setup correct? Why would ntopng not collect any data? >> >> Thank you for your help, >> >> Pablo Destefanis >> >> ------ >> >> Startup screens follow: >> >> NPROBE >> >> [root@hqsys2 ntopng]# nprobe --zmq "tcp://*:5556" -i none -n none >> --collector-port 2055 --city-list /usr/local/nprobe/GeoLiteCity.dat >> 13/Dec/2013 16:41:19 [nprobe.c:5673] ERROR: Invalid or missing nProbe >> license (/etc/nprobe.license) >> 13/Dec/2013 16:41:19 [nprobe.c:5683] ERROR: for 6F087510910461D2 >> 13/Dec/2013 16:41:19 [nprobe.c:5688] ERROR: >> *************************************************** >> 13/Dec/2013 16:41:19 [nprobe.c:5689] ERROR: >> ** ** >> 13/Dec/2013 16:41:19 [nprobe.c:5690] ERROR: ** Switching to DEMO MODE >> due to license error ** >> 13/Dec/2013 16:41:19 [nprobe.c:5691] ERROR: >> ** ** >> 13/Dec/2013 16:41:19 [nprobe.c:5692] ERROR: ** Create your nProbe >> license at ** >> 13/Dec/2013 16:41:19 [nprobe.c:5693] ERROR: ** >> http://www.nmon.net/mklicense/ ** >> 13/Dec/2013 16:41:19 [nprobe.c:5694] ERROR: >> ** ** >> 13/Dec/2013 16:41:19 [nprobe.c:5695] ERROR: >> *************************************************** >> 13/Dec/2013 16:41:19 [nprobe.c:5715] ERROR: >> *************************************************************** >> 13/Dec/2013 16:41:19 [nprobe.c:5716] ERROR: * NOTE: This is a DEMO >> version limited to 25000 flows export. * >> 13/Dec/2013 16:41:19 [nprobe.c:5717] ERROR: >> *************************************************************** >> 13/Dec/2013 16:41:19 [plugin.c:161] No plugins found in ./plugins >> 13/Dec/2013 16:41:19 [nprobe.c:3628] Succesfully created ZMQ endpoint >> tcp://*:5556 >> 13/Dec/2013 16:41:19 [util.c:344] GeoIP: loaded cities config file >> /usr/local/nprobe/GeoLiteCity.dat >> 13/Dec/2013 16:41:19 [util.c:353] GeoIP: loaded IPv6 cities config file >> /usr/local/nprobe/GeoLiteCityv6.dat >> 13/Dec/2013 16:41:19 [nprobe.c:3795] WARNING: The output interfaceId is >> set to 0: did you forget to use -Q perhaps ? >> 13/Dec/2013 16:41:19 [nprobe.c:3798] WARNING: The input interfaceId is >> set to 0: did you forget to use -u perhaps ? >> 13/Dec/2013 16:41:19 [nprobe.c:3802] WARNING: You have specified --zmq >> and not specified -n. >> 13/Dec/2013 16:41:19 [nprobe.c:3803] WARNING: We believe you want to use >> just ZMQ and no netflow export >> 13/Dec/2013 16:41:19 [nprobe.c:3804] WARNING: Setting flow export to -n >> none >> 13/Dec/2013 16:41:19 [nprobe.c:3858] Welcome to nprobe v.6.15.131213 >> ($Revision: 3810 $) for x86_64-unknown-linux-gnu with native PF_RING >> acceleration >> 13/Dec/2013 16:41:19 [nprobe.c:3871] nProbe SystemId: 6F087510910461D2 >> 13/Dec/2013 16:41:19 [dbPlugin.c:78] Initializing DB plugin >> 13/Dec/2013 16:41:19 [nprobe.c:5733] Welcome to nprobe v.6.15.131213 for >> x86_64-unknown-linux-gnu >> 13/Dec/2013 16:41:19 [nprobe.c:4943] GEO-533LITE 20090701 Build 1 >> Copyright (c) 2007 MaxMind LLC All Rights Reserved >> 13/Dec/2013 16:41:19 [plugin.c:872] 0 plugin(s) enabled >> 13/Dec/2013 16:41:19 [util.c:308] GeoIP: loaded AS config file >> /usr/local/nprobe/GeoIPASNum.dat >> 13/Dec/2013 16:41:19 [util.c:317] GeoIP: loaded AS IPv6 config file >> /usr/local/nprobe/GeoIPASNumv6.dat >> 13/Dec/2013 16:41:19 [nprobe.c:4379] Using packet capture length 128 >> 13/Dec/2013 16:41:19 [nprobe.c:5909] IPv6 traffic will NOT be >> exported/accounted by this probe >> 13/Dec/2013 16:41:19 [nprobe.c:5910] due to configuration options (e.g. >> use NetFlow v9) >> 13/Dec/2013 16:41:19 [nprobe.c:6038] Not capturing packet from interface >> (collector mode) >> 13/Dec/2013 16:41:19 [collect.c:156] Flow collector listening on port >> 2055 (IPv4/v6) >> >> NTOPNG: >> [root@hqsys2 ~]# ntopng -i "tcp://127.0.0.1:5556" >> 13/Dec/2013 16:59:46 [Ntop.cpp:457] Setting local networks to >> 192.168.1.0/24,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8 >> 13/Dec/2013 16:59:46 [Ntop.cpp:564] Registered interface >> collector@127.0.0.1:5556 [id: 0] >> 13/Dec/2013 16:59:46 [Utils.cpp:238] User changed to nobody >> 13/Dec/2013 16:59:46 [main.cpp:147] PID stored in file /var/tmp/ntopng.pid >> 13/Dec/2013 16:59:46 [HTTPserver.cpp:363] HTTP server listening on port >> 3000 [/usr/local/share/ntopng/httpdocs][/usr/local/share/ntopng/scripts] >> 13/Dec/2013 16:59:46 [main.cpp:179] Using RRD version 1.4.7 >> 13/Dec/2013 16:59:46 [main.cpp:188] Working directory: /var/tmp/ntopng >> 13/Dec/2013 16:59:46 [main.cpp:190] Scripts/HTML pages directory: >> /usr/local/share/ntopng >> 13/Dec/2013 16:59:46 [Ntop.cpp:161] Welcome to ntopng x86_64 v.1.1.1 (r) >> - (C) 1998-13 ntop.org >> 13/Dec/2013 16:59:46 [Redis.cpp:47] Successfully connected to Redis >> 127.0.0.1:6379 >> 13/Dec/2013 16:59:46 [PeriodicActivities.cpp:53] Started periodic >> activities loop... >> 13/Dec/2013 16:59:46 [NetworkInterface.cpp:634] Started packet polling on >> interface collector@127.0.0.1:5556... >> 13/Dec/2013 16:59:46 [CollectorInterface.cpp:100] Collecting flows... >> >> >> >> _______________________________________________ >> Ntop-misc mailing list >> Ntop-misc@listgateway.unipi.it >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> >> >> >> _______________________________________________ >> Ntop-misc mailing list >> Ntop-misc@listgateway.unipi.it >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> >> >
_______________________________________________ Ntop-misc mailing list Ntop-misc@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop-misc
