Re: Finding asymmetric path

[Arie Vayner]

Actually, this can be achieved easily using reflexive ACLs on any Cisco
router, so no real need to change the topology or add new devices in the
path:
http://www.cisco.com/en/US/products/sw/secursw/ps1018/products_tech_note09186a00800a5b9a.shtml#reflexacl

Arie

On Sat, Nov 28, 2009 at 10:26 PM, Duane Waddle wrote:

> On Sat, Nov 28, 2009 at 1:41 PM, Brielle Bruns  wrote:
>
> > My partner Tammy says a PIX could probably accomplish the same task (we
> have some here for the corp lan stuff, including spares).
>
> Yes, a PIX/ASA would stop this cold.  The TCP state tracking would not
> allow traffic to pass unless the whole 3-way handshake was observed by
> the box.  Only recently did Cisco add features to make tracking the
> TCP connection state optional.
> (
> http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpstatebypass.pdf
> )
>  The larger ASA-5580 machines can be virtualized into dozens (or more)
> security contexts as needed.  I imagine it would take some effort to
> figure out how to cleanly integrate such a configuration into a POP.
>
> --D
>
>

Re: Finding asymmetric path

[Jorge Amodio]

> Brielle is correct.  The customer in question is spamming networks and we
> are having trouble filtering them because another provider allows them to
> source traffic however they please.

If they are spamming just pull the plug, whatever revenue you get from them
is not worth your reputation and caring for other good customers, and the
rest of us will speak highly of you for taking another spamcrock out
of the net.

Cheers
Jorge

Re: Finding asymmetric path

[Suresh Ramasubramanian]

Yes - term the account would be my recommendation

And if you filter port 25 traffic do it both ways

Read these old nanog threads ..
http://www.irbs.net/internet/nanog/0408/0465.html and
http://www.mail-archive.com/na...@merit.edu/msg28863.html

On Sun, Nov 29, 2009 at 3:58 AM, William Herrin
 wrote:
> On Sat, Nov 28, 2009 at 2:14 PM, ML  wrote:
>> Brielle is correct.  The customer in question is spamming networks and we
>> are having trouble filtering them because another provider allows them to
>> source traffic however they please.
>
> What trouble? SMTP requires two-way traffic with a static port number
> that nothing else uses. If for some reason you don't want to simply
> terminate their account altogether, block packets outbound to your
> customer sourced from TCP port 25 but not from your SMTP smarthosts.
>
> Seriously though, if you can prove they're spamming (regardless of
> whether the packets pass through your network) save yourself some
> grief and just terminate the account.
>
> Regards,
> Bill Herrin
>
>
> --
> William D. Herrin  her...@dirtside.com  b...@herrin.us
> 3005 Crane Dr. .. Web: 
> Falls Church, VA 22042-3004
>
>



-- 
Suresh Ramasubramanian (ops.li...@gmail.com)

Re: Finding asymmetric path

[Randy Bush]

> Brielle is correct.  The customer in question is spamming networks and
> we are having trouble filtering them because another provider allows
> them to source traffic however they please.

then perhaps the issue is a bit larger than their traffic incoming to
you.  disconnect the schmucks.

randy

Re: Finding asymmetric path

[William Herrin]

On Sat, Nov 28, 2009 at 2:14 PM, ML  wrote:
> Brielle is correct.  The customer in question is spamming networks and we
> are having trouble filtering them because another provider allows them to
> source traffic however they please.

What trouble? SMTP requires two-way traffic with a static port number
that nothing else uses. If for some reason you don't want to simply
terminate their account altogether, block packets outbound to your
customer sourced from TCP port 25 but not from your SMTP smarthosts.

Seriously though, if you can prove they're spamming (regardless of
whether the packets pass through your network) save yourself some
grief and just terminate the account.

Regards,
Bill Herrin


-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Finding asymmetric path

[Brielle Bruns]

(Forgive the top posting, stupid blackberry can't do inline)


If the PoP is connected to a central location, reroute the affected netblock 
there through the appropriate equipment.  If you snag it going both ways before 
it hits the PoP, you should be good.


--Original Message--
From: Duane Waddle
To: nanog@nanog.org
Subject: Re: Finding asymmetric path
Sent: Nov 28, 2009 1:26 PM

On Sat, Nov 28, 2009 at 1:41 PM, Brielle Bruns  wrote:

> My partner Tammy says a PIX could probably accomplish the same task (we have 
> some here for the corp lan stuff, including spares).

Yes, a PIX/ASA would stop this cold.  The TCP state tracking would not
allow traffic to pass unless the whole 3-way handshake was observed by
the box.  Only recently did Cisco add features to make tracking the
TCP connection state optional.
(http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpstatebypass.pdf)
 The larger ASA-5580 machines can be virtualized into dozens (or more)
security contexts as needed.  I imagine it would take some effort to
figure out how to cleanly integrate such a configuration into a POP.

--D



-- 
Brielle Bruns
http://www.sosdg.org  /  http://www.ahbl.org

Re: Finding asymmetric path

[Duane Waddle]

On Sat, Nov 28, 2009 at 1:41 PM, Brielle Bruns  wrote:

> My partner Tammy says a PIX could probably accomplish the same task (we have 
> some here for the corp lan stuff, including spares).

Yes, a PIX/ASA would stop this cold.  The TCP state tracking would not
allow traffic to pass unless the whole 3-way handshake was observed by
the box.  Only recently did Cisco add features to make tracking the
TCP connection state optional.
(http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpstatebypass.pdf)
 The larger ASA-5580 machines can be virtualized into dozens (or more)
security contexts as needed.  I imagine it would take some effort to
figure out how to cleanly integrate such a configuration into a POP.

--D

Re: Finding asymmetric path

[Brielle Bruns]

(Forgive the top posting, stupid blackberry can't do inline)

A creative idea that I did in a test lab one time - stateful connection 
tracking, its not just for NAT you know.

Would require a bit of moving stuff around and reengineering of your connection 
to them, but it would cripple their connection unless it originated through you.

IE: 

You <-> UNIX/Linux firewall <-> T1/eth/dsl/etc

If stuff went out the other way, it would come in, firewall says WTF, and drops 
it because it didn't see the initial SYN exchange.

My partner Tammy says a PIX could probably accomplish the same task (we have 
some here for the corp lan stuff, including spares).


Brielle
-- 
Brielle Bruns
http://www.sosdg.org  /  http://www.ahbl.org

-Original Message-
From: ML 
Date: Sat, 28 Nov 2009 14:14:07 
To: nanog@nanog.org
Subject: Re: Finding asymmetric path

Brielle Bruns wrote:
> On 11/27/09 8:43 PM, ML wrote:
>> I'm reasonable certain a customer of ours who is using one of our
>> netblocks is using a different reverse path to reach us. How might I
>> figure out who is allowing them to source traffic from IPs that belong
>> to us?
>>
>>
>>
> 
> I've had two customers pull this stunt in the past with me - one, a 
> spammer, tried to do this with an ADSL modem from me, the other (a 
> non-spammer with a clueless 'consultant') had a T1 from me and a T1 from 
> UUNet.
> 
> It started with the T1 customer.  I believe they had a smaller block of 
> IPs (less then /24, more like a /25 or /26), and their 'computer 
> consultant' with his infinite wisdom decided to send all outbound 
> traffic through the UUNet T1 rather then source routing which we highly 
> recommended to them.  Of course, we had ingress filters in place to 
> block IP ranges we have from coming into us from the WAN links, so when 
> they tried to contact servers on the other half of the netblock on our 
> end, the connections mysteriously failed.  After lying up and down that 
> it was our fault, that their computer 'consultant' was regarded as best 
> in the country, blah blah blah, we flipped on logging on the ingress 
> filters out of sheer curiosity and discovered exactly what was going on.
> 
> The ADSL customer was a bit more tricky - we were getting spam reports 
> about his single IP address sending spam, but we had his outbound port 
> 25 blocked.  Ended up sniffing the port off the router he sat off of, 
> and discovering that it was all one sided, wasn't even tickling the 
> ingress filters.
> 
> Hey, at least your customer didn't convince AT&T to allow them to 
> announce out one of your /24s when all they had was a /29.
> 
> Your in a tricky bind, I'd approach them under the guise of ingress 
> filtering issues.
> 

Brielle is correct.  The customer in question is spamming networks and 
we are having trouble filtering them because another provider allows 
them to source traffic however they please.

Of course the other provider seems to be a spam friendly upstream. 
Hopefully you can understand why I'm not hopeful of getting anywhere 
with them either.

Re: Finding asymmetric path

[ML]

Brielle Bruns wrote:

On 11/27/09 8:43 PM, ML wrote:

I'm reasonable certain a customer of ours who is using one of our
netblocks is using a different reverse path to reach us. How might I
figure out who is allowing them to source traffic from IPs that belong
to us?





I've had two customers pull this stunt in the past with me - one, a 
spammer, tried to do this with an ADSL modem from me, the other (a 
non-spammer with a clueless 'consultant') had a T1 from me and a T1 from 
UUNet.


It started with the T1 customer.  I believe they had a smaller block of 
IPs (less then /24, more like a /25 or /26), and their 'computer 
consultant' with his infinite wisdom decided to send all outbound 
traffic through the UUNet T1 rather then source routing which we highly 
recommended to them.  Of course, we had ingress filters in place to 
block IP ranges we have from coming into us from the WAN links, so when 
they tried to contact servers on the other half of the netblock on our 
end, the connections mysteriously failed.  After lying up and down that 
it was our fault, that their computer 'consultant' was regarded as best 
in the country, blah blah blah, we flipped on logging on the ingress 
filters out of sheer curiosity and discovered exactly what was going on.


The ADSL customer was a bit more tricky - we were getting spam reports 
about his single IP address sending spam, but we had his outbound port 
25 blocked.  Ended up sniffing the port off the router he sat off of, 
and discovering that it was all one sided, wasn't even tickling the 
ingress filters.


Hey, at least your customer didn't convince AT&T to allow them to 
announce out one of your /24s when all they had was a /29.


Your in a tricky bind, I'd approach them under the guise of ingress 
filtering issues.




Brielle is correct.  The customer in question is spamming networks and 
we are having trouble filtering them because another provider allows 
them to source traffic however they please.


Of course the other provider seems to be a spam friendly upstream. 
Hopefully you can understand why I'm not hopeful of getting anywhere 
with them either.

Re: Finding asymmetric path

[Joe Greco]

> 
> On Sat, Nov 28, 2009 at 09:41:09AM -0600, Joe Greco wrote:
> [attributions lost]
> > > >>> I'm reasonable certain a customer of ours who is using one of our 
> > > >>> netblocks is using a different reverse path to reach us.  How might I 
> > > >>> figure out who is allowing them to source traffic from IPs that 
> > > >>> belong 
> > > >>> to us?
> > > >> you are implying that they are not allowed to multi-home using the ip
> > > >> space you have assigned to them.  good way to lose a customer.
> > > > Does it count as multihoming when we are the only ones announcing the
> > > > space?
> > > 
> > > almost an interesting question.  but i think it is playing with words.
> > > if i understand your original statement, they are clearly attached to at
> > > least two providers.
> > > 
> > > perhaps it is fear of what they, possibly mistakenly, perceive to be
> > > your policy regarding announcement of space that keeps them from
> > > announcing normally to both, or more, links?
> 
> It wasn't clear that the customer was a BGP downstream though by saying 
> 'We are the only ones announcing the space', I think not.  Non-BGP 
> multihoming is broken* and when not done out of ignorance generally is
> the smoke pointing to the fire of someone trying to hide something.
> Was very common for spammers to abuse no-uRPF networks in the early
> days of broadband.

This is still rather common for people to do, at least where there's a
significant cost differential.  There are enough networks that can accept
arbitrary traffic that BGP doesn't really play into it at all.

> > It could also be something simple like pricing.  For example, in a large
> > colo facility, you might easily find that a number of providers offer
> > low cost transit, but not IP space.  For a customer who is heavy on the
> > outbound traffic, they might find it more affordable to buy their inbound
> > plus IP space from you, and then dump onto Cogent or something like that
> > for outbound.  Unless your contract specifically prohibits this, you're
> > probably not going to be able to prevent it.
> 
> I wonder if there is a drift of baseline assumptions between the current
> wave of operators and previous ones?  To me (and BCP38) it is beyond bad 
> practice to allow -and if allowed, to make use of- such sloppy edges.  

Of course it is.

> If the other network truly is practicing bad forwarding hygiene then 
> they are a security problem for everyone else and IMO would be good for 
> naming and shaming.

Sure.  But it's easy enough to go to, for example, $BACKBONE_OF_CHOICE
and say "I'm delegated 1.2.3.4/24 from $SMALLISP, I would like you to
accept traffic from that prefix, here's my SWIP'd WHOIS to prove that"
and there are lots of providers for whom that won't be a problem; it is
not the same sort of security problem that a complete lack of filtering
is.  Generally speaking, what's needed is a control over what's being
cast into the network.  

> * for the majority of the cases. I know there are purposeful Non-BGP 
>   MOAS/anycast purposefully run by those who understand the implications.
>   It is unfortunate that their use of lack of inherent BGP path security
>   contribute to fuzzing what would otherwise have been a clear indicator
>   of 'bad' behavior.  But same could be said for the deaggregators 
>   using longest-match to have everyone else do their TE; water under
>   the bridge pushing work onto everyone else. 

:-)

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Finding asymmetric path

[Brielle Bruns]

On 11/27/09 8:43 PM, ML wrote:

I'm reasonable certain a customer of ours who is using one of our
netblocks is using a different reverse path to reach us. How might I
figure out who is allowing them to source traffic from IPs that belong
to us?





I've had two customers pull this stunt in the past with me - one, a 
spammer, tried to do this with an ADSL modem from me, the other (a 
non-spammer with a clueless 'consultant') had a T1 from me and a T1 from 
UUNet.


It started with the T1 customer.  I believe they had a smaller block of 
IPs (less then /24, more like a /25 or /26), and their 'computer 
consultant' with his infinite wisdom decided to send all outbound 
traffic through the UUNet T1 rather then source routing which we highly 
recommended to them.  Of course, we had ingress filters in place to 
block IP ranges we have from coming into us from the WAN links, so when 
they tried to contact servers on the other half of the netblock on our 
end, the connections mysteriously failed.  After lying up and down that 
it was our fault, that their computer 'consultant' was regarded as best 
in the country, blah blah blah, we flipped on logging on the ingress 
filters out of sheer curiosity and discovered exactly what was going on.


The ADSL customer was a bit more tricky - we were getting spam reports 
about his single IP address sending spam, but we had his outbound port 
25 blocked.  Ended up sniffing the port off the router he sat off of, 
and discovering that it was all one sided, wasn't even tickling the 
ingress filters.


Hey, at least your customer didn't convince AT&T to allow them to 
announce out one of your /24s when all they had was a /29.


Your in a tricky bind, I'd approach them under the guise of ingress 
filtering issues.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Finding asymmetric path

[Joe Provo]

On Sat, Nov 28, 2009 at 09:41:09AM -0600, Joe Greco wrote:
[attributions lost]
> > >>> I'm reasonable certain a customer of ours who is using one of our 
> > >>> netblocks is using a different reverse path to reach us.  How might I 
> > >>> figure out who is allowing them to source traffic from IPs that belong 
> > >>> to us?
> > >> you are implying that they are not allowed to multi-home using the ip
> > >> space you have assigned to them.  good way to lose a customer.
> > > Does it count as multihoming when we are the only ones announcing the
> > > space?
> > 
> > almost an interesting question.  but i think it is playing with words.
> > if i understand your original statement, they are clearly attached to at
> > least two providers.
> > 
> > perhaps it is fear of what they, possibly mistakenly, perceive to be
> > your policy regarding announcement of space that keeps them from
> > announcing normally to both, or more, links?

It wasn't clear that the customer was a BGP downstream though by saying 
'We are the only ones announcing the space', I think not.  Non-BGP 
multihoming is broken* and when not done out of ignorance generally is
the smoke pointing to the fire of someone trying to hide something.
Was very common for spammers to abuse no-uRPF networks in the early
days of broadband.

> It could also be something simple like pricing.  For example, in a large
> colo facility, you might easily find that a number of providers offer
> low cost transit, but not IP space.  For a customer who is heavy on the
> outbound traffic, they might find it more affordable to buy their inbound
> plus IP space from you, and then dump onto Cogent or something like that
> for outbound.  Unless your contract specifically prohibits this, you're
> probably not going to be able to prevent it.

I wonder if there is a drift of baseline assumptions between the current
wave of operators and previous ones?  To me (and BCP38) it is beyond bad 
practice to allow -and if allowed, to make use of- such sloppy edges.  
If the other network truly is practicing bad forwarding hygiene then 
they are a security problem for everyone else and IMO would be good for 
naming and shaming.

Cheers,

Joe

* for the majority of the cases. I know there are purposeful Non-BGP 
  MOAS/anycast purposefully run by those who understand the implications.
  It is unfortunate that their use of lack of inherent BGP path security
  contribute to fuzzing what would otherwise have been a clear indicator
  of 'bad' behavior.  But same could be said for the deaggregators 
  using longest-match to have everyone else do their TE; water under
  the bridge pushing work onto everyone else. 

-- 
 RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Re: Finding asymmetric path

[William Herrin]

On Fri, Nov 27, 2009 at 10:43 PM, ML  wrote:
> I'm reasonable certain a customer of ours who is using one of our netblocks
> is using a different reverse path to reach us.  How might I figure out who
> is allowing them to source traffic from IPs that belong to us?

Hi,

Are they complaining about somehting? If so, ask for a traceroute.

You might also try calling and asking. "We're seeing some strange
traffic purporting to be from your addresses but not coming from your
circuit. We're concerned that someone might be attacking your network.
Before taking action to protect you, we want to eliminate the
possibility that you have a second ISP through which you're
accidentally sourcing the packets."

Beyond that, what's your game plan once you know the answer? Threaten
to cut them off? That's a great way to lose a customer who you know
*already* has a second ISP.  Maybe you'll call their second ISP and
complain about their filtering practices? I'd love to get that call
from you. Tells me exactly which name to pass to my upsell specialist.
Yes Mr. Customer, your other ISP is trying to cut you off. They even
asked us to block your packets. But for just a little more we'll give
you IP addresses that you can use with any ISP.

Regards,
Bill Herrin



-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Finding asymmetric path

[Joe Greco]

> >>> I'm reasonable certain a customer of ours who is using one of our 
> >>> netblocks is using a different reverse path to reach us.  How might I 
> >>> figure out who is allowing them to source traffic from IPs that belong 
> >>> to us?
> >> you are implying that they are not allowed to multi-home using the ip
> >> space you have assigned to them.  good way to lose a customer.
> > Does it count as multihoming when we are the only ones announcing the
> > space?
> 
> almost an interesting question.  but i think it is playing with words.
> if i understand your original statement, they are clearly attached to at
> least two providers.
> 
> perhaps it is fear of what they, possibly mistakenly, perceive to be
> your policy regarding announcement of space that keeps them from
> announcing normally to both, or more, links?

It could also be something simple like pricing.  For example, in a large
colo facility, you might easily find that a number of providers offer
low cost transit, but not IP space.  For a customer who is heavy on the
outbound traffic, they might find it more affordable to buy their inbound
plus IP space from you, and then dump onto Cogent or something like that
for outbound.  Unless your contract specifically prohibits this, you're
probably not going to be able to prevent it.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Finding asymmetric path

[Christopher Morrow]

On Fri, Nov 27, 2009 at 11:23 PM, Randy Bush  wrote:

> perhaps it is fear of what they, possibly mistakenly, perceive to be
> your policy regarding announcement of space that keeps them from
> announcing normally to both, or more, links?

or maybe just better pricing on the other provider, and that provider
offers something akin to 'no-export' (or maybe they just used
no-export).

-chris

Re: Finding asymmetric path

[Randy Bush]

>>> I'm reasonable certain a customer of ours who is using one of our 
>>> netblocks is using a different reverse path to reach us.  How might I 
>>> figure out who is allowing them to source traffic from IPs that belong 
>>> to us?
>> you are implying that they are not allowed to multi-home using the ip
>> space you have assigned to them.  good way to lose a customer.
> Does it count as multihoming when we are the only ones announcing the
> space?

almost an interesting question.  but i think it is playing with words.
if i understand your original statement, they are clearly attached to at
least two providers.

perhaps it is fear of what they, possibly mistakenly, perceive to be
your policy regarding announcement of space that keeps them from
announcing normally to both, or more, links?

randy

RE: Finding asymmetric path

[Stefan Fouant]

> -Original Message-
> From: ML [mailto:m...@kenweb.org]
> Sent: Friday, November 27, 2009 10:44 PM
> 
> I'm reasonable certain a customer of ours who is using one of our
> netblocks is using a different reverse path to reach us.  How might I
> figure out who is allowing them to source traffic from IPs that belong
> to us?

It's been my experience that many providers do not care what address is used
for sourcing traffic.  This is why it is not uncommon to see traffic sourced
from RFC1918 space coming across various providers networks.  If more
providers adopted BCP 38 this wouldn't be a problem, but that doesn't seem
to be happening anytime soon...

I'd try to identify which providers the customer is connected to and take it
from there...

Stefan Fouant
www.shortestpathfirst.net
GPG Key ID: 0xB5E3803D

Re: Finding asymmetric path

[ML]

Randy Bush wrote:
I'm reasonable certain a customer of ours who is using one of our 
netblocks is using a different reverse path to reach us.  How might I 
figure out who is allowing them to source traffic from IPs that belong 
to us?


you are implying that they are not allowed to multi-home using the ip
space you have assigned to them.  good way to lose a customer.

randy


Does it count as multihoming when we are the only ones announcing the space?

Re: Finding asymmetric path

[Randy Bush]

> I'm reasonable certain a customer of ours who is using one of our 
> netblocks is using a different reverse path to reach us.  How might I 
> figure out who is allowing them to source traffic from IPs that belong 
> to us?

you are implying that they are not allowed to multi-home using the ip
space you have assigned to them.  good way to lose a customer.

randy

Finding asymmetric path

[ML]

I'm reasonable certain a customer of ours who is using one of our 
netblocks is using a different reverse path to reach us.  How might I 
figure out who is allowing them to source traffic from IPs that belong 
to us?