Hi Peter, it seems that you are _the_ expert in this project - are you? Thank you for your patience with us/me. I am very new to netflowing and need some advice and knowledge input.
If you can point me to a good FAQ or something else I would be pleased but maybe I am out of "frequently". I don't know if it is relevant for my question but I use standard flow expire/export of five minutes at the probes and standard five minute rotation interval at nfsen. So defaults everywhere. If my data file contains flows over several days, this was done by nfcapd. You mentioned the risk of 32bit overflow at the exporter. What is the "exporter"? Is nfdump the exporter? How can I take care of counter overflows? If a flow lasts some days - well the tools (nfsen) must handle this long lasting flows - if this is the problem with counter overflow at all. ---> I would like to understand the whole netflow process. From the probe to the sensor to the exporter. May I draft my understanding of flow processing? 1. netflow probe handles flow counting If a flow expires (either by timer/max traffic/max flow/FIN/RST or what ever), netflow information is send to the sensor and the probe forgets all about this flow. The probe accumulates only a five minute stateful flow information. After expiration every packet newly arriving creates a new entry in the flow table of the probe. 2. nfsen processes the netflow information and tries to provide something like a stateful flow information from the real beginning to the end of a flow. To accomplish this, nfcapd seems to look into the historical files in the profile-data directory to see if any of the "new" flows were active in the past. Nfcapd retrieves the flow starttime from historical data and puts it into the profile-data file of the current timeslot so that flow starttime is carried over into the timeslot data file. Am I right? What other historical data is carried over into the new data file of the current timeslot? The flags, as you mentioned. I hope that the number of bytes and packets is not accumulated with the past but only flow information gathered by the probe and arriving during the 5 minute interval is stored into the data file of current timeslot. I understand that because of two different 5 minute intervals at the probe and at the sensor and because the flow expiring can take place at any time during this intervals it is possible that traffic information is sometimes stored a little bit to late. So beside this inaccuracy is it true that the "Packets" and "Bytes" printed be nfdump with "-s record/packets" option are only the packets and bytes that had been seen by the probe during this interval? When I do a "-o extended" dump, are the ratio values of pps and bps calculated at a timeslot base or at the flow-duration base? I would expect that if I select a single timeslot to dump that calculation is done for that single timeslot only but as you explained that all time values are taken from the data file maybe pps und bps are calculated since flow starttime. Joerg -----Original Message----- From: Peter Haag [mailto:[EMAIL PROTECTED] Sent: Monday, February 18, 2008 3:29 PM To: Pichel, Jörg; [email protected] Subject: Re: [Nfsen-discuss] How to interpret nfdump output -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Joerg, A flow has a dedicated start a and end time. When aggregating flows ( as it's done in -s ) then more records could be summed up. In any case the resulted flow has a start and end time resulting in a duration, regardless, whether the flow is not yet finished. All rate values are calculated for each flow, which its specific duration, to get an average value for bps, pps etc. for each flow. The summary lines at the bottom give an overall average value, in the time window seen total packets/bytes divided by duration. The fact if a flow is completed or not is not relevant. So time values are taken from the flows found in the file. The file name, or other issues are not relevant. So your file contains flows over several days. Be also aware, that the exporter must be able to deal with the 32bit overflow from sysup time otherwise bizare flows could result - maybe yours below. As for the flags - yes they are all cumulated OR - Peter - --On February 18, 2008 14:02:01 +0100 [EMAIL PROTECTED] wrote: | | | I tried to analyze a very high pks/s peak. For this I selected one | single timeslot "Feb 18 2008 - 10:45", called nfdump and got this output: | | | ** nfdump -M /sw/pkg/nfsen/profiles-data/live/mucwan -T -r | 2008/02/18/nfcapd.200802181045 -n 20 -s record/packets -o long nfdump filter: | any | Aggregated flows 22705 | Top 20 flows ordered by packets: | Date flow start Duration Proto Src IP Addr:Port Dst IP Addr:Port Flags Tos Packets Bytes Flows | 2008-02-12 12:56:04.110 510524.741 TCP 10.40.232.5:445 -> 10.40.200.5:1814 .APR.. 0 5.1 M 1.5 G 1 | 2008-02-12 12:56:04.110 510524.741 TCP 10.40.200.5:1814 -> 10.40.232.5:445 .AP... 0 4.9 M 659.2 M 1 | 2008-02-12 12:56:30.231 510498.626 TCP 10.40.232.5:445 -> 10.40.208.5:1388 .APR.. 0 4.6 M 1.4 G 1 | 2008-02-12 12:56:30.231 510498.626 TCP 10.40.208.5:1388 -> 10.40.232.5:445 .AP... 0 4.4 M 603.7 M 1 | 2008-02-12 12:55:58.184 510530.659 TCP 10.40.232.5:445 -> 10.40.216.5:3228 .APR.. 0 1.9 M 648.2 M 1 | 2008-02-12 12:55:58.184 510530.659 TCP 10.40.216.5:3228 -> 10.40.232.5:445 .AP... 0 1.8 M 238.9 M 1 | 2008-02-12 12:56:32.902 510495.941 TCP 10.40.232.5:445 -> 10.40.224.5:1779 .APR.. 0 1.7 M 624.4 M 1 | 2008-02-12 12:55:58.184 510530.674 TCP 10.40.232.5:445 -> 10.40.204.5:2848 .APR.. 0 1.7 M 620.6 M 1 | 2008-02-12 12:56:32.902 510495.941 TCP 10.40.224.5:1779 -> 10.40.232.5:445 .AP... 0 1.5 M 209.8 M 1 | 2008-02-12 12:55:58.184 510530.674 TCP 10.40.204.5:2848 -> 10.40.232.5:445 .AP... 0 1.5 M 208.3 M 1 | 2008-02-16 07:31:50.615 184378.288 TCP 10.40.232.5:445 -> 10.40.240.32:1084 .APRS. 0 871746 262.1 M 1 | 2008-02-16 07:31:50.615 184378.288 TCP 10.40.240.32:1084 -> 10.40.232.5:445 .AP.S. 0 827661 110.1 M 1 | 2008-02-17 18:10:15.823 59673.029 TCP 10.40.232.5:445 -> 10.40.220.5:3577 .APRS. 0 452797 159.3 M 1 | 2008-02-17 18:10:15.823 59673.029 TCP 10.40.220.5:3577 -> 10.40.232.5:445 .AP.S. 0 422606 55.0 M 1 | 2008-02-18 10:35:23.499 565.379 TCP 10.40.212.5:1083 -> 10.40.232.5:445 .AP.S. 0 26698 4.0 M 1 | 2008-02-18 10:35:23.499 565.379 TCP 10.40.232.5:445 -> 10.40.212.5:1083 .APRS. 0 26084 4.0 M 1 | 2008-02-18 10:22:39.148 1108.766 TCP 80.90.99.41:3505 -> 10.40.224.6:1152 .AP.SF 0 15114 16.6 M 1 | 2008-02-18 10:45:40.760 45.540 TCP 10.40.232.7:8080 -> 10.40.208.7:33746 .AP.S. 0 13181 18.4 M 1 | | Summary: total flows: 22707, total bytes: 7.4 G, total packets: 32.0 | M, avg bps: 124932, avg pps: 65, avg bpp: 237 Time window: 2008-02-12 | 12:55:58 - 2008-02-18 10:46:44 Total flows processed: 22707, Records skipped: 0, Bytes read: 1180788 | Sys: 0.032s flows/second: 709571.6 Wall: 0.030s flows/second: 744174.6 | | | Are the numbers of pkts/s, bytes/s that are printed in the table the | number of pakets and bytes that flew during the 5-minute interval from | 10:40 - 10:45 or are these the packets and bytes that flew since | flowstart, which are mostly some days in the past. The "Time window" given in the "Summary" information could lead to this opinion - but this is not the intention if I select a single timeslot to inspect. | | How can nfdump know when the flow started when I give it only one | single timeslot to inspect? Does nfdump read all the old data as well to find the flow start or is this information collected by nfcapd and stored in the five minutes dump files? | | And what about the "Flags" - Is this a cummulation of TCP-Flags during | this inspection interval (given as single timeslot | 10:45) or since flow start? And can I conclude that if there have been | "R"eset Flags seen in the flow that this flow has ended during the inspection interval and the others are potentially still active? | ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
