Trying to remove duplicates based just the flows is not going to work
well, especially in an environment where sampled NetFlow is used. You
really need to filter based on interfaces. A technique we use with
Abilene is the routers have a key in the ifAlias SNMP variable
"backbone". The ifAlias MIB tree is walked every night for all the
routers then the results are used to build a flow-nfilter config file.
For example
% snmpwalk -O qs Atlang.abilene.ucaid.edu <community> ifAlias
(truncated a little)
ifAlias.44 BACKBONE: oc192 to IPLSng
ifAlias.55 SOX 10GE
ifAlias.56 BACKBONE: OC-48 to IPLSng - OLD
ifAlias.57 Florida LambdaRail
ifAlias.70 ATLAng NOC vLAN
ifAlias.71 ATLAng Measurement vLAN
ifAlias.72 GIGE to ATLAng-M5
ifAlias.73 GIGE to NMS1-ATLA
ifAlias.74 BACKBONE: OC192 to WASHng
ifAlias.77 South Florida GigaPoP/AMPATH
ifAlias.79 BACKBONE: oc192 to HSTNng
Which gets turned into.
filter-primitive ATLAng-bb
type ifindex
permit 44,56,74,79
--
mark
On May 19, 2005, at 10:55 AM, McGlinchy, Alistair wrote:
Michael,
What do you do with duplicate flows?
Example: Traffic destined for client xxx.xxx.xxx.1 comes in via Router
B (Internet Feed), which is then routed to client who is connected to
Router A - Both Router A + Router B will have a flow for this traffic,
so there is a chance of double billing?
I have experienced the problem too, and have read the replies with some
dismay. Not all of us have such control over where we can get our flow
feed from. Ideally I want to be able to add any routers' data into my
collection system without having to worry about the correct filter
required to ensure I don't get duplicates. [Deduping was one of the
best features of 3Com's Traffix application which I still long for].
As a work-around I have written a perl script to hack the output of
"flow-export -f2 -m0x383069". [Attached]. Its slow, CPU intensive,
doesn't output in flow-file format and worst of all is bugged. One of
the main gotchas with this relates to the variable export frequencies.
Our MSFCs export flows at 30s and 7200s at 300sec. So If 1.2.3.4 sends
to 5.6.7.8 300KB over from 270 secs for 90 secs you could receive these
records:
MSFC says:
At 270secs 1.2.3.4 to 5.6.7.8 sent 100KB
At 300secs 1.2.3.4 to 5.6.7.8 sent 100KB
At 330secs 1.2.3.4 to 5.6.7.8 sent 100KB
While the 7200 says
At 270secs 1.2.3.4 to 5.6.7.8 sent 300KB
My code will say that
At 0-300secs 1.2.3.4 to 5.6.7.8 sent 300KB = max(100,300)
witness = 7200
At 300-600secs 1.2.3.4 to 5.6.7.8 sent 200KB = max(0,200)
witness = MSFC
I've contemplated many tweaks to this (eg pro rata the traffic across
interval boundaries etc), but it's all fudging and relies too heavily
on
the syncing of our routers' time stamps.
Hopefully writing this will inspire me to write something better....
Cheers,
Alistair
**********************************************************************
Registered Office:
Marks and Spencer plc
Waterside House
35 North Wharf Road
London
W2 1NW
Registered No. 214436 in England and Wales.
Telephone (020) 7935 4422
Facsimile (020) 7487 2670
<<www.marksandspencer.com>>
Please note that electronic mail may be monitored.
This e-mail is confidential. If you received it by mistake, please let
us know and then delete it from your system; you should not copy,
disclose, or distribute its contents to anyone nor act in reliance on
this e-mail, as this is prohibited and may be unlawful.
<fxdedup.pl>_______________________________________________
Flow-tools mailing list
[EMAIL PROTECTED]
http://mailman.splintered.net/mailman/listinfo/flow-tools
_______________________________________________
Flow-tools mailing list
[EMAIL PROTECTED]
http://mailman.splintered.net/mailman/listinfo/flow-tools
