Re: Thanks & Let's Prevent this in the Future.

[Mark Tinka]

On Thursday, February 02, 2012 01:00:43 AM George Bonser 
wrote:

> One problem is the number of routing registries and the
> requirements differ for them.  The nefarious operator
> can enter routes in an IRR just as easily as a
> legitimate operator.  There was a time when some
> significant networks used the IRRs for their filtration
> policy.  I'm not sure how many still do.

I've dealt with AfriNIC and APNIC WHOIS databases, and they 
normally control the 'inetnum' and inet6num' entries that go 
into the WHOIS databases. So there is some degree of 
certainty that what is in there is generally true.

You're right, anyone can create an IRR record, and it's 
quite terrible how easy it is to create false information 
that could break another person's network. This is why we 
don't generally trust IRR or PeeringDB data when verifying 
downstream prefixes which we should permit through our 
filters. We rely on the RIR 'inetnum' and 'inet6num' records 
for that.

My memory fails me on what ARIN do, but before AfriNIC was 
established and the majority of Africa's prefixes were 
allocated by RIPE and ARIN, I recall the ARIN policy (SWIP 
templates, et al) being a hassle-rich experience that 
anything else is long forgotten :-).

Mark.


signature.asc
Description: This is a digitally signed message part.

RE: Thanks & Let's Prevent this in the Future.

[Murphy, Sandra]

Thanks for the reminder, Richard.

Yes, as I announced earlier (see
http://mailman.nanog.org/pipermail/nanog/2012-January/044493.html
- the message with the corrected date), there is an interim sidr meeting on 
Thu *9* Feb in San Diego.

Registration is free.  
Registration is easy (email).  
Registration is open to all.
Registration is open ended.
Registration is encouraged (so we know how many to expect).  

As the message says, see the sidr wiki 
http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209 for agenda 
and a list of attendees.

--Sandy Murphy


From: Richard Barnes [richard.bar...@gmail.com]
Sent: Friday, February 03, 2012 8:09 AM
To: Arturo Servin
Cc: nanog
Subject: Re: Thanks & Let's Prevent this in the Future.

In related news, the IETF working group that is writing standards for
the RPKI is having an interim meeting in San Diego just after NANOG.
They deliberately chose that place/time to make it easy for NANOG
attendees to contribute, so comments from this community are
definitely welcome.
<http://www.ietf.org/mail-archive/web/sidr/current/msg03923.html>
<http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209>

Re: Thanks & Let's Prevent this in the Future.

[Richard Barnes]

In related news, the IETF working group that is writing standards for
the RPKI is having an interim meeting in San Diego just after NANOG.
They deliberately chose that place/time to make it easy for NANOG
attendees to contribute, so comments from this community are
definitely welcome.





On Fri, Feb 3, 2012 at 7:16 AM, Arturo Servin  wrote:
>
>        One option is to use RPKI and origin validation. But it won't help 
> much unless prefix holders create their certificates and ROAs and networks 
> operators use those to validate origins. It won't solve all the issues but at 
> least some fat fingers/un-expierience errors.
>
>        We are running an experiment to detect route-hijacks/missconf using 
> RPKI. So far, not many routes are "signed" but at least we can periodically 
> check our own prefix (or any other with ROAs) to detect some inconsistencies:
>
> http://www.labs.lacnic.net/rpkitools/looking_glass/rest/all/pfx/200.7.84.0/
>
> http://www.labs.lacnic.net/rpkitools/looking_glass/
>
>
> Regards,
> -as
>
>
> On 1 Feb 2012, at 06:58, Kelvin Williams wrote:
>
>> First off, I'd like to thank everyone on this list who have reached out
>> today and offered us help with our hijacked network space.  It's so
>> refreshing to see that there are still so many who refuse to leave a
>> man/woman down.
>>
>> I'm not going to place any blame, its useless.  There were lies, there were
>> incompetencies, and there was negligence but that is now water under the
>> bridge.
>>
>> However, I think that we as network operators have a duty to each other to
>> make sure we don't allow a downstream customer wreck the operations of
>> another entity who has been rightfully allocated resources.
>>
>> A few months ago, when establishing a new peering relationship I was
>> encouraged (actually required) to utilize one of the IRRs.  I took the time
>> to register all of my routes, ASNs, etc.  However, as I learned today, this
>> was probably done in vain.  Too many people won't spend the extra
>> 30-seconds to verify the information listed there or in ARINs WHOIS.
>>
>> I don't care what a customer tells me, too many times I've found they
>> aren't 100% honest either for malicious/fraudulent reasons or they are
>> unknowing.  So, for our networks or the networks we manage, we want to
>> verify what a customer is saying to prevent what happened to us today.
>>
>> I'd like to get a conversation going and possibly some support of an
>> initiative to spend that extra 30-seconds to verify ownership and
>> authorization of network space to be advertised.  Additionally, if someone
>> rings your NOC's line an industry-standard process of verifying "ownership"
>> and immediately responding by filtering out announcements. There's no sense
>> in allowing a service provider to be impaired because a spammer doesn't
>> want to give up clean IP space.  Do you protect a bad customer or the
>> Internet as a whole?  I pick the Internet as a whole.
>>
>> How can we prevent anyone else from ever enduring this again?  While we may
>> never stop it from ever happening, spammers (that's what we got hit by
>> today) are a dime a dozen and will do everything possible to hit an Inbox,
>> so how can we establish a protocol to immediate mitigate the effects of an
>> traffic-stopping advertisement?
>>
>> I thought registering with IRRs and up-to-date information in ARINs WHOIS
>> was sufficient, apparently I was wrong.  Not everyone respects them, but
>> then again, they aren't very well managed (I've got several networks with
>> antiquated information I've been unable to remove, it doesn't impair us
>> normally, but its still there).
>>
>> What can we do?  Better yet, how do we as a whole respond when we encounter
>> upstream providers who refuse to look at the facts and allow another to
>> stay down?
>>
>> kw
>>
>> --
>> Kelvin Williams
>> Sr. Service Delivery Engineer
>> Broadband & Carrier Services
>> Altus Communications Group, Inc.
>>
>>
>> "If you only have a hammer, you tend to see every problem as a nail." --
>> Abraham Maslow
>

Re: Thanks & Let's Prevent this in the Future.

[Arturo Servin]

One option is to use RPKI and origin validation. But it won't help much 
unless prefix holders create their certificates and ROAs and networks operators 
use those to validate origins. It won't solve all the issues but at least some 
fat fingers/un-expierience errors.

We are running an experiment to detect route-hijacks/missconf using 
RPKI. So far, not many routes are "signed" but at least we can periodically 
check our own prefix (or any other with ROAs) to detect some inconsistencies:

http://www.labs.lacnic.net/rpkitools/looking_glass/rest/all/pfx/200.7.84.0/

http://www.labs.lacnic.net/rpkitools/looking_glass/


Regards,
-as


On 1 Feb 2012, at 06:58, Kelvin Williams wrote:

> First off, I'd like to thank everyone on this list who have reached out
> today and offered us help with our hijacked network space.  It's so
> refreshing to see that there are still so many who refuse to leave a
> man/woman down.
> 
> I'm not going to place any blame, its useless.  There were lies, there were
> incompetencies, and there was negligence but that is now water under the
> bridge.
> 
> However, I think that we as network operators have a duty to each other to
> make sure we don't allow a downstream customer wreck the operations of
> another entity who has been rightfully allocated resources.
> 
> A few months ago, when establishing a new peering relationship I was
> encouraged (actually required) to utilize one of the IRRs.  I took the time
> to register all of my routes, ASNs, etc.  However, as I learned today, this
> was probably done in vain.  Too many people won't spend the extra
> 30-seconds to verify the information listed there or in ARINs WHOIS.
> 
> I don't care what a customer tells me, too many times I've found they
> aren't 100% honest either for malicious/fraudulent reasons or they are
> unknowing.  So, for our networks or the networks we manage, we want to
> verify what a customer is saying to prevent what happened to us today.
> 
> I'd like to get a conversation going and possibly some support of an
> initiative to spend that extra 30-seconds to verify ownership and
> authorization of network space to be advertised.  Additionally, if someone
> rings your NOC's line an industry-standard process of verifying "ownership"
> and immediately responding by filtering out announcements. There's no sense
> in allowing a service provider to be impaired because a spammer doesn't
> want to give up clean IP space.  Do you protect a bad customer or the
> Internet as a whole?  I pick the Internet as a whole.
> 
> How can we prevent anyone else from ever enduring this again?  While we may
> never stop it from ever happening, spammers (that's what we got hit by
> today) are a dime a dozen and will do everything possible to hit an Inbox,
> so how can we establish a protocol to immediate mitigate the effects of an
> traffic-stopping advertisement?
> 
> I thought registering with IRRs and up-to-date information in ARINs WHOIS
> was sufficient, apparently I was wrong.  Not everyone respects them, but
> then again, they aren't very well managed (I've got several networks with
> antiquated information I've been unable to remove, it doesn't impair us
> normally, but its still there).
> 
> What can we do?  Better yet, how do we as a whole respond when we encounter
> upstream providers who refuse to look at the facts and allow another to
> stay down?
> 
> kw
> 
> -- 
> Kelvin Williams
> Sr. Service Delivery Engineer
> Broadband & Carrier Services
> Altus Communications Group, Inc.
> 
> 
> "If you only have a hammer, you tend to see every problem as a nail." --
> Abraham Maslow

RE: Thanks & Let's Prevent this in the Future.

[Jon Lewis]

On Wed, 1 Feb 2012, George Bonser wrote:

One problem is the number of routing registries and the requirements 
differ for them.  The nefarious operator can enter routes in an IRR just 
as easily as a legitimate operator.  There was a time when some 
significant networks used the IRRs for their filtration policy.  I'm not 
sure how many still do.


A few do, but IRR data really can't be trusted as a means of 
authenticating authority to advertise IP space.  It's a nice system for 
automating route filter updates (between the customer and 
provider...i.e. I add data to an IRR, and Level3 auto-updates my prefix 
filter), but when anyone can put anything into the IRR dbs, obviously 
everyone using the IRR dbs isn't going to stop someone from hijacking 
someone else's space.


As a regional ISP / hosting provider, my policy for accepting BGP routes 
from customers has been to check RIR whois data, make sure the space 
appears to be owned by the customer, and if there's any question, ask for 
clarification and/or an LOA from the customer showing that they are 
authorized to use the space.  Every customer gets a prefix filter that 
allows them to announce only what we've verified they're authorized to 
announce.


Level3, I suppose, just trusts customers won't announce space they 
shouldn't.  The alternative, which I've dealt with with some other "Tier 
1's" is that they require customers to provide an LOA every time they want 
to add a CIDR to their prefix filter.  That's more paper work for me and 
for them, and tends to be slower, as someone has to manually approve 
(maybe manually apply) the change request.


Given what we have available today, there's security or convenience...pick 
one.


It seems like the IRR thing could reasonably easily be solved if the RIRs 
got more deeply involved.


Suppose, as an ARIN member, I used ARIN's IRR.  I'd start out by 
registering my maintainer object, my ASN, my routes, and an as-set. 
Then, when I wanted to add customer ASNs to my as-set, the system wouldn't 
allow me to do so unless the customer had already registered their ASN 
with my ASN as an approved provider/path.  The customer, if they had no 
ASN, would be responsible for updating their route (PI or PA) in the IRR 
db, stating my ASN is a valid origin for their route. This would solve the 
problem of larger providers like Level3 blindly accepting my as-set 
because all the authentication/authorization would already have been done 
before the data went into the IRR db.  Things get a little more 
complicated when I pick up a customer who has "out of region" objects, 
i.e. RIPE IPs/ASN, but if all the RIRs worked together and standardized 
how the information is maintained, it could work.  i.e. When I try to add 
the RIPE ASN to my as-set, ARIN's system would check RIPE's system to see 
if that ASN's owner had set my ASN as a valid provider/path for their ASN.


This seems so simple, either it should have been implemented a decade or 
more ago, or perhaps I'm overlooking something.  It wouldn't stop route 
advertisements from providers who don't care / don't filter, but it would 
make systems like Level3's more secure...and if all the "Tier 1's" adopted 
similar automated filter generators based on the RIR IRR dbs, it would 
stop unauthorized announcements from getting far.


The problem of email spam is an interesting one that has been battled 
for a very long time and is probably better discussed on a list 
dedicated to that topic.


Definitely...but the issue here is ownership/authorization to use IP 
space, and how stop unauthorized BGP announcements when providers don't or 
won't filter their BGP customers.


--
 Jon Lewis, MCP :)   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net|
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

RE: Thanks & Let's Prevent this in the Future.

[George Bonser]

> I'd like to get a conversation going and possibly some support of an
> initiative to spend that extra 30-seconds to verify ownership and
> authorization of network space to be advertised.  Additionally, if
> someone rings your NOC's line an industry-standard process of verifying
> "ownership"
> and immediately responding by filtering out announcements. There's no
> sense in allowing a service provider to be impaired because a spammer
> doesn't want to give up clean IP space.  Do you protect a bad customer
> or the Internet as a whole?  I pick the Internet as a whole.
> 
> How can we prevent anyone else from ever enduring this again?  While we
> may never stop it from ever happening, spammers (that's what we got hit
> by
> today) are a dime a dozen and will do everything possible to hit an
> Inbox, so how can we establish a protocol to immediate mitigate the
> effects of an traffic-stopping advertisement?

One problem is the number of routing registries and the requirements differ for 
them.  The nefarious operator can enter routes in an IRR just as easily as a 
legitimate operator.  There was a time when some significant networks used the 
IRRs for their filtration policy.  I'm not sure how many still do.

But generally speaking, if someone calls me and I can verify that they really 
are a POC for the entity that is assigned an address allocation (generally some 
verification method beyond email if the subnet their MX record points to is 
part of the hijacking!) then I am going to do whatever I can to help them out 
provided what they are asking for is reasonable and within my capabilities.  It 
shouldn't be too hard to verify.  If someone claims to be with a commercial 
entity of Foo.COM then the entity is likely listed in the phone book and a 
phone call can take care of my personal verification requirement.  

Back in the days of Cyberpromo and Sanford Wallace, what I did was used TCP 
wrappers on my mail server so that when I received a connection from a 
Cyberpromo net block, I hairpinned the connection back to his MX server using 
netcat so when he connected to me, the HELO he received was from his own mail 
server, which gladly accepted mail from Cyberpromo.  He could pump mail to me 
all day long if he wanted to, but his mailq wasn't going to get any smaller.

The problem of email spam is an interesting one that has been battled for a 
very long time and is probably better discussed on a list dedicated to that 
topic.

Re: Thanks & Let's Prevent this in the Future.

[Hank Nussbacher]

At 03:58 01/02/2012 -0500, Kelvin Williams wrote:

Those ISPs that are good network citizens have done it already.  Those who 
don't care and who haven't done it yet - won't do it in the future.  The 
only recourse you have is exactly what you have done.


-Hank


How can we prevent anyone else from ever enduring this again?  While we may
never stop it from ever happening, spammers (that's what we got hit by
today) are a dime a dozen and will do everything possible to hit an Inbox,
so how can we establish a protocol to immediate mitigate the effects of an
traffic-stopping advertisement?

I thought registering with IRRs and up-to-date information in ARINs WHOIS
was sufficient, apparently I was wrong.  Not everyone respects them, but
then again, they aren't very well managed (I've got several networks with
antiquated information I've been unable to remove, it doesn't impair us
normally, but its still there).

What can we do?  Better yet, how do we as a whole respond when we encounter
upstream providers who refuse to look at the facts and allow another to
stay down?

kw

--
Kelvin Williams
Sr. Service Delivery Engineer
Broadband & Carrier Services
Altus Communications Group, Inc.


"If you only have a hammer, you tend to see every problem as a nail." --
Abraham Maslow

Re: Thanks & Let's Prevent this in the Future.

[Leigh Porter]

On 1 Feb 2012, at 09:01, "Kelvin Williams"  wrote:

> 
> A few months ago, when establishing a new peering relationship I was
> encouraged (actually required) to utilize one of the IRRs.  I took the time
> to register all of my routes, ASNs, etc.  However, as I learned today, this
> was probably done in vain.  Too many people won't spend the extra
> 30-seconds to verify the information listed there or in ARINs WHOIS.

It's amazing isn't it. It isn't only fraud and maliciousness that this 
prevents. 

A number of times I have been asked to advertise space or open filters for 
space based on typos and people not understanding address notation and CIDR. So 
it's with doing if only to prevent this.

-- 
Leigh


__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__

Thanks & Let's Prevent this in the Future.

[Kelvin Williams]

First off, I'd like to thank everyone on this list who have reached out
today and offered us help with our hijacked network space.  It's so
refreshing to see that there are still so many who refuse to leave a
man/woman down.

I'm not going to place any blame, its useless.  There were lies, there were
incompetencies, and there was negligence but that is now water under the
bridge.

However, I think that we as network operators have a duty to each other to
make sure we don't allow a downstream customer wreck the operations of
another entity who has been rightfully allocated resources.

A few months ago, when establishing a new peering relationship I was
encouraged (actually required) to utilize one of the IRRs.  I took the time
to register all of my routes, ASNs, etc.  However, as I learned today, this
was probably done in vain.  Too many people won't spend the extra
30-seconds to verify the information listed there or in ARINs WHOIS.

I don't care what a customer tells me, too many times I've found they
aren't 100% honest either for malicious/fraudulent reasons or they are
unknowing.  So, for our networks or the networks we manage, we want to
verify what a customer is saying to prevent what happened to us today.

I'd like to get a conversation going and possibly some support of an
initiative to spend that extra 30-seconds to verify ownership and
authorization of network space to be advertised.  Additionally, if someone
rings your NOC's line an industry-standard process of verifying "ownership"
and immediately responding by filtering out announcements. There's no sense
in allowing a service provider to be impaired because a spammer doesn't
want to give up clean IP space.  Do you protect a bad customer or the
Internet as a whole?  I pick the Internet as a whole.

How can we prevent anyone else from ever enduring this again?  While we may
never stop it from ever happening, spammers (that's what we got hit by
today) are a dime a dozen and will do everything possible to hit an Inbox,
so how can we establish a protocol to immediate mitigate the effects of an
traffic-stopping advertisement?

I thought registering with IRRs and up-to-date information in ARINs WHOIS
was sufficient, apparently I was wrong.  Not everyone respects them, but
then again, they aren't very well managed (I've got several networks with
antiquated information I've been unable to remove, it doesn't impair us
normally, but its still there).

What can we do?  Better yet, how do we as a whole respond when we encounter
upstream providers who refuse to look at the facts and allow another to
stay down?

kw

-- 
Kelvin Williams
Sr. Service Delivery Engineer
Broadband & Carrier Services
Altus Communications Group, Inc.


"If you only have a hammer, you tend to see every problem as a nail." --
Abraham Maslow