-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Burton Strauss wrote on 7-6-2005 18:37: > What about ifconfig stats? That should show the # of packets arriving at > the interface. If you look, the counts are consistent, but just 1/2 of what > you expect.
With just looking at the counters of ifconfig, you also see the http traffic to/from the Ntop webserver. When I count the packets with 'tcpdump udp port 2055' I see that the netflow packtes arrive at the interface. But Ntop-netflow-plugin sees about 1/2. In about 1 hour and 5 min the router sends +/- 76,505 pkts to a 'flowtools host' which sends it to the Ntop-host. The Ntop host receives according tho tcpdump +/- 76,513 pkts. In this period the interface (ifconfig) receives +/- 128,012 pkts. The Ntop-Netflow-Plugin receives +/- 28,827 pkts! > It could be that ntop isn't pulling them fast enough from the I think that is the problem....... > interface, but that's just a pretty simple select() recvfrom() call pair. Is that good/wrong? Could that be an perfomance issue? > > -----Burton > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf > Of R.H.Hoek > Sent: Monday, June 06, 2005 3:07 AM > To: [email protected] > Subject: Re: [Ntop-dev] netflow-plugin misses packets > > Burton Strauss wrote on 4-6-2005 14:25: > >>>How about the stats from the plugin itself? > > > The packetcounts from Ntop netflow are taken from the stats from the plugin > itself: > > Flow Statistics > Received Flows > Flow Senders 130.89.244.12 [2,251,319 pkts] > > Number of Packets Received 2,251,319 > Number of Packets with Bad Version 0 > Number of Packets Processed 2,251,319 > Number of Valid Flows Received 65,850,914 > Average Number of Flows per Packet 29.2 > Number of V1 Flows Received 0 > Number of V5 Flows Received 65,850,914 > Number of V7 Flows Received 0 > Number of V9 Flows Received 0 > > Discarded Flows > Number of Flows with Zero Packet Count 0 > Number of Flows with Zero Byte Count 0 > Number of Flows with Bad Data 0 > Number of Flows with Unknown Template 0 > Total Number of Flows Processed 65,850,914 > > Flowtools reports: > ------------------- > Jun 6 09:41:00 localhost flow-capture[27759]: STAT: now=1118043660 > startup=1117802440 src_ip=127.0.0.1 dst_ip=127.0.0.1 d_ver=5 > pkts=4590376 flows=133781439 lost=0 reset=0 filter_drops=0 Jun 6 09:42:00 > localhost flow-capture[27759]: STAT: now=1118043720 startup=1117802440 > src_ip=127.0.0.1 dst_ip=127.0.0.1 d_ver=5 > pkts=4592106 flows=133831723 lost=0 reset=0 filter_drops=0 Jun 6 09:43:00 > localhost flow-capture[27759]: STAT: now=1118043780 startup=1117802440 > src_ip=127.0.0.1 dst_ip=127.0.0.1 d_ver=5 > pkts=4595919 flows=133942507 lost=0 reset=0 filter_drops=0 > -------------------- > > Cisco Netflow reports: > -------------------- > UTWENTE-router>sh ip flow export > Flow export is enabled > Exporting flows to x.x.x.x (2055) > Exporting using source interface Vlan102 > Version 5 flow records > *21104941* flows exported in *703498* udp datagrams > 0 flows failed due to lack of export packet > 0 export packets were sent up to process level > 0 export packets were dropped due to no fib > 0 export packets were dropped due to adjacency issues > 0 export packets were dropped due to fragmentation failures > 0 export packets were dropped due to encapsulation fixup failures > 0 export packets were dropped enqueuing for the RP > 0 export packets were dropped due to IPC rate limiting > UTWENTE-router>sh mls nde > Netflow Data Export enabled > Exporting flows to x.x.x.x (2055) > Exporting flows from y.y.y.y (49744) > Version: 5 > Include Filter not configured > Exclude Filter not configured > Total Netflow Data Export Packets are: > *3892409* packets, 0 no packets, *112837207* records Total Netflow Data > Export Send Errors: > IPWRITE_NO_FIB = 0 > IPWRITE_ADJ_FAILED = 0 > IPWRITE_PROCESS = 0 > IPWRITE_ENQUEUE_FAILED = 0 > IPWRITE_IPC_FAILED = 0 > IPWRITE_MTU_FAILED = 0 > IPWRITE_ENCAPFIX_FAILED = 0 > UTWENTE-router>sho clo > 09:42:52.569 MET-DST Mon Jun 6 2005 > -------------------- > > This measurement runs from Fri 3jun > Cisco and Flowtools reports both the same packets/flows: > > Cisco: 21104941+112837207= 133,942,148 flowtools: 133,942,507 (flows) > Cisco: 703498+3892409= 4,595,907 flowtools: 4,595,919 (packets) > > Netflowplugin: 65,850,914 flows > Netflowplugin: 2,251,319 packets > > > N.B. counters are reset and read manualy on 'about' the same time. > (within 30 sec) > > >>>-----Burton >>> >>>-----Original Message----- >>>From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On >>>Behalf Of R.H.Hoek >>>Sent: Friday, June 03, 2005 9:15 AM >>>To: [email protected] >>>Subject: [Ntop-dev] netflow-plugin misses packets >>> >>>Dear Ntoppers, >>> >>>Some weeks ago I already mentioned that I experience some problems >>>with Ntop in combination with Netflow from a Cisco6509 with supervisor2. >>>Some of you gave me some hints, but nothing couldn't solve my problem >>>(see below). >>> >>>The problem is that the Netflow-plugin misses about 40-60% of the >>>netflowpackets send by the Cisco6509. Some figures: >>> >>>Last night, in a period of about 17 hours the Cisco has sendout >>>31,274,087 flows in 1,073,210 packets. >>>The Netflow-plugin reports it received 16,035,229 valid flows in >>>548,345 packets. The plugin reports no discarded flow. >>> >>>My first though is that it is not the hardware. I (can) run Ntop on >>>two different systems, but experience on both the same problem: >>>System1: >>>Dual-Xeon 3.2GHz, 5GB RAM, with 4 disks in two hardware SCSI raidsets >>>(RAID1)(18GB,73GB) and 100M/1G Ethernet interface with a 100Mbps >>>networkconecction. On this system the OS is SuSE-UnitedLinux 8. (2.4 >>>kernel) System2(originally meant for probe): >>>Dual-Xeon 3.2GHz, 3GB RAM, with 2 disks in a hardware SCSI raidsets >>>(RAID1)(18GB) and 100M/1G Ethernet interface with a 100Mbps >>>networkconecction. On this system the OS is Debian Sarge 2.6 kernel. >>> >>>I have done testing with Ntop 3.1 and the latest CVS versions -> same >>>results. With tcpdump I did some counting. The conclusion is that the >>>packets send by Cisco6509 are received on the system Ntop is running >>>on, but are discarded/missed by the Netflow-plugin >>> >>>Answers on some questions: >>>1) >>>Ntop is running with --interface-none, --track-local-host and the >>>Netflow pseudo-nic is selected. >>>2) >>>Even, with RRD-plugin is set to low-detail, and data to dump is 'none' >>>does not solve the problem >>>3) >>>for testing I have incremented MAX_SUBNET_HOSTS to 8192/16384 in >>>globals-defines.h. >>>Load avg 0.14 (98% idle) >>>The memory usage is about 1.8GB (3.3 GB free) after 1 hour Ntop is > > started. > >>>4) >>>The load graph and (traffic)protocol-distribution should display the >>>total network load of the Cisco6509 when the pseudo-nic is selected. I >>>have seen this on a Ntop system with low netflowtraffic. >>>5) >>>In a test config the Cisco-Netflow is first send to system2 and >>>redistributed with flow-fanout to system1 running Ntop. On system2 I >>>do flow-capture and flow-stat. When I compare this output with >>>Ntop-output, the differences are very large as of 5min net workload >>>and protocol distribution. >>>6) >>>The average netflowstream is about 20 packets/s. But I have seen >>>bursts of 900 packets/s >>> >>> >>> >>>I think that the last point is possibly the problem. With these bursts >>>the netflowplugin inputbuffer is overloaded ? >>>Is there a way to tune this buffer? Or does anybody else has some hints? >>> >>> >>>-- >>> >>>Groeten, >>> >>>Roel H.Hoek, SeniorNetworkmanager >>>Dienst Informatietechnologie, Bibliotheek en Educatie (ITBE) >>>Universiteit Twente, Postbus 217, 7500 AE Enschede kmr SP 422, >>>telefoon: 053 - 489 4598, fax: 053 - 489 2383 >>>e-mail: [EMAIL PROTECTED] http://www.utwente.nl/itbe >>> > > _______________________________________________ > Ntop-dev mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-dev > > _______________________________________________ > Ntop-dev mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-dev > > > -- > > Groeten, > > Roel H.Hoek, SeniorNetworkmanager > Dienst Informatietechnologie, Bibliotheek en Educatie (ITBE) Universiteit > Twente, Postbus 217, 7500 AE Enschede kmr SP 422, telefoon: 053 - 489 > 4598, fax: 053 - 489 2383 > e-mail: [EMAIL PROTECTED] http://www.utwente.nl/itbe > _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev - -- Groeten, Roel H.Hoek -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCqC/OJwlRSGnYBcYRAsfnAJsHG/xZ5DIz9Y44O/oc4ONt+vikKQCfaU2/ u8SFiRt8V298zjIBStKryD8= =5GD8 -----END PGP SIGNATURE----- _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
