Re: estimation of number of DFZ IPv4 routes at peak in the future

[William Herrin]

On Sun, Mar 13, 2011 at 5:40 PM, Jeff Wheeler  wrote:
> On Sun, Mar 13, 2011 at 3:42 PM, Christopher Morrow
>  wrote:
>> not need that info, but the edge likely does, yes? Have 100g customers
>> today? planning on having them in the next ~8/12/18 months?
>
> If you did your purchasing the way Bill Herrin suggests, you'd buy a
> box with 100GbE ports for a POP or branch that is not projected to
> have 100GbE customers, just because it's the biggest box.

Jeff,

No, Chris wouldn't, because that misrepresents my suggestion. What I
suggested is that you spend your efforts making solid projections and
then buy a box that satisfies the targeted function for the
foreseeable future. That way you don't spend manpower replacing it
until something materially different than the projections occurs.
Which avoids some mistaken-driven and defect-driven outages and has a
myriad of similar secondary effects.

Circuit outages are minimized when the CWA is on strike. Why? Because
nobody's futzing with the equipment. There's a lesson there: maximize
reliability by minimizing change.

For your information, the ISP where I was the operations director
survived the burst of the bubble. While revenues shrunk significantly
it was still in the black in 2004 when I left. To the best of my
knowledge it remained in the black until it was sold a few years
later. There were a number of causes, but one of them was that in the
key time frames we were able to crunch the capital budget to almost
nothing, there being sufficient excess capacity in most of the
equipment we already owned.

> His
> position is that man-power to do an upgrade is always more costly than
> capital dollars for the actual equipment, and ignores the fact that
> the biggest box is by no means guaranteed to offer new *features*
> which may be required.

My position is that the terminal size of the IPv4 table is visible on
the horizon. Now that it's part of the foreseeable future, I'd like to
be able to buy boxes that support it.

Regards,
Bill Herrin


-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Why does abuse handling take so long ?

[Suresh Ramasubramanian]

Depends on what you're yelling at them about and what you tell them.

I've picked up the phone and had a NOC guy at a russian SP (can't
remember which, Caravan I think) kill off a syn flood that was hitting
us promptly, at like 1 AM their time.

On Mon, Mar 14, 2011 at 7:05 AM,   wrote:
>
> In my experience, most phone calls cause the ISP to become immediately
> hostile. They find abuse report phone calls extremely threatening / scary /
> etc. and go into full shields-up mode. 9 out of 10 times the very first
> words out of their mouth is "talk to our lawyers". the remaining 1 out of 10
> is "block it on your end".
>
> Email tends to be non threatening. As useless as it tends to be, it is still
> generally better than calling.
>
> the real cesspool is POC registries. i wish arin would start revoking
> allocations for entities with invalid POCs.



-- 
Suresh Ramasubramanian (ops.li...@gmail.com)

Re: Why does abuse handling take so long ?

[goemon]

On Sun, 13 Mar 2011, Leo Bicknell wrote:

Quite frankly, most ISP's aren't going to take your DDOS report
seriously via e-mail.  If it's not bad enough to you that it is
worth your time and money to make a phone call and help them track
it down it is not worth their time and money to track it down and
make it stop.

In short, try picking up the phone.  You'll bypass the entire e-mail
reporting cesspool I just described, and show the ISP you actually
care.  9 out of 10 times they will respond by showing they care as
well.


In my experience, most phone calls cause the ISP to become immediately 
hostile. They find abuse report phone calls extremely threatening / scary 
/ etc. and go into full shields-up mode. 9 out of 10 times the very first 
words out of their mouth is "talk to our lawyers". the remaining 1 out of 
10 is "block it on your end".


Email tends to be non threatening. As useless as it tends to be, it is 
still generally better than calling.


the real cesspool is POC registries. i wish arin would start revoking 
allocations for entities with invalid POCs.

Re: Why does abuse handling take so long ?

[Alexander Maassen]

Op 14-3-2011 0:21, Leo Bicknell schreef:
>
> Quite frankly, most ISP's aren't going to take your DDOS report
> seriously via e-mail.  If it's not bad enough to you that it is
> worth your time and money to make a phone call and help them track
> it down it is not worth their time and money to track it down and
> make it stop.
>
> In short, try picking up the phone.  You'll bypass the entire e-mail
> reporting cesspool I just described, and show the ISP you actually
> care.  9 out of 10 times they will respond by showing they care as
> well.
>
Quite frankly, been there, done that, got the t-shirt. And the answer I
get most of the time there is:
[loop]
- Sorry, email abuse and wait for a reply
- Sorry, I can't help you, wait for a reply on your abuse email
- Sorry, there is nothing I can do, my hands are bound, wait for a reply
from the abuse department
[/loop]

So much regarding the 9 out of 10. It's the 1 remaining that actually
cares and tries something.



signature.asc
Description: OpenPGP digital signature

Re: Why does abuse handling take so long ?

[Leo Bicknell]

In a message written on Sun, Mar 13, 2011 at 12:45:04PM +0100, Alexander 
Maassen wrote:
> Why o why are isp's and hosters so ignorant in dealing with such issues
> and act like they do not care?

One of the things you have to remember is that ISP's get a ton of
reports, and most of them are of very low quality.  Abuse queues
are full of people who sign up for a properly run mailing list and
then a year or two later mail abuse to get taken off saying its now
spam.  Or folks who misconfigure their firewall / IDS and send in
reports of being DDOSed, by a nameserver, to which they are sending
queries and then flagging the responses as an "attack".  There are
a lot of reports that don't include either the source or destination
IP, or leave out any time information.

Worst of all, there are the automated reports where someone has a
different opinion than the law, or even reality.  They create systems
to basically DDOS abuse@, by reporting every case they can find
individually when in fact the "spammer" is doing things legally and
properly.

Of course it varies greatly ISP to ISP, depends on customer mix,
time of the day, time of the year and all sorts of other factors.
Still, there are times when I would say less than 1 in 50 e-mails
received to abuse@ is something that is a complete report and
actionable  Keep that in mind, along with what others have pointed
out, that there is generally no "profit" in handling abuse.

Quite frankly, most ISP's aren't going to take your DDOS report
seriously via e-mail.  If it's not bad enough to you that it is
worth your time and money to make a phone call and help them track
it down it is not worth their time and money to track it down and
make it stop.

In short, try picking up the phone.  You'll bypass the entire e-mail
reporting cesspool I just described, and show the ISP you actually
care.  9 out of 10 times they will respond by showing they care as
well.

-- 
   Leo Bicknell - bickn...@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/


pgpvjeuh7yw89.pgp
Description: PGP signature

Re: Why does abuse handling take so long ?

[Larry Brower]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 03/13/2011 05:34 PM, goe...@anime.net wrote:
> On Sun, 13 Mar 2011, Alexander Maassen wrote:
>> On 13-3-2011 18:31, William Allen Simpson wrote:
>>> On 3/13/11 7:45 AM, Alexander Maassen wrote:
 Why o why are isp's and hosters so ignorant in dealing with such issues
 and act like they do not care?
>>> So, part of the problem is *your* upstream.  Why didn't your upstream
>>> actively remove the entire abusive netblock?  Why didn't your upstream
>>> contact other providers with your evidence, and together remove the
>>> abusive network from the global routing tables?
>> My hoster did mail, his upstream is EGI, however, EGI does not want to
>> block/filter since it would pollute their routers they say.
>> I asked through my hoster if they would be willing to place a simple UDP
>> filter, blocking all of it. They refuse.
> 
> again make it a question of economics.
> 
> vote with your wallet, vote with your feet.
> 
> if they won't block, leave.
> 

leaving is not always as easy as you imply. There are some areas with
only one real provider.


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCgAGBQJNfUvcAAoJEPXCUD/44PWqje8QALqCB5SmUamIvItQSsJZ+B0t
bKSDxmUszgAkFMxc9y7n1TuymOKx2lsCQmO8aQmE95NXOUloH1R89aA51DMaxxsX
vt9GspDi2zuzoGngMUhl7Xuho5lxekg0nw8zEqa14MdZK/iMQw1e9D+pfl2GF43X
4KyFBqL85DrpnJhaNpQ3BB/EsM4+hMxxZYm5CAqZYKa2ywuR9LGVjQ5i0zoqc3e9
wFUAk2C/0ATf+eUhBaw6OHtpj7E2JgGfkP8K8npr27U9WVMpjBjd1ERZVy7FZ+7n
rEVj+bUrW57VVvdv1UzE4rHa49Y0YXALnq05rGxuE0iCkIcth8pDI5YVYwTwsvDx
gZR5H0Kmm9bQvOpvUR8TmW7BXlamVOHC1beCYHhI6Oig1bTx1DfCP+CniMnz5l3o
G+sweZA589nJxonawl5qGhySCBg6a9Z4gtoYRaFsTVe8sI59JtzsbyvjdVvUwSkR
UOPdPpHUuG72i3dB+/bdTKWDeGjghn6oTheVY/03oYvOLAg+8zZPupb3Ql9FARnw
qb2Qc5ebzEG6wuaXlC/iHzTuc4DHp4rWvwm9tE1sQ379ntZJiKpfm0eMcdeo+xJ9
RxDg5fksOaihhM8MVRi4recdHyySzHZw9JPrx1VfhPWv3umPYt/csT0/L/EeyAJu
Ybo7ahfzII26suzrAr43
=M/H6
-END PGP SIGNATURE-

Re: Why does abuse handling take so long ?

[goemon]

On Sun, 13 Mar 2011, Alexander Maassen wrote:

On 13-3-2011 18:31, William Allen Simpson wrote:

On 3/13/11 7:45 AM, Alexander Maassen wrote:

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?

So, part of the problem is *your* upstream.  Why didn't your upstream
actively remove the entire abusive netblock?  Why didn't your upstream
contact other providers with your evidence, and together remove the
abusive network from the global routing tables?

My hoster did mail, his upstream is EGI, however, EGI does not want to
block/filter since it would pollute their routers they say.
I asked through my hoster if they would be willing to place a simple UDP
filter, blocking all of it. They refuse.


again make it a question of economics.

vote with your wallet, vote with your feet.

if they won't block, leave.

Re: Why does abuse handling take so long ?

[Alexander Maassen]

On 13-3-2011 18:31, William Allen Simpson wrote:
> On 3/13/11 7:45 AM, Alexander Maassen wrote:
>> Why o why are isp's and hosters so ignorant in dealing with such issues
>> and act like they do not care?
>>
>
> So, part of the problem is *your* upstream.  Why didn't your upstream
> actively remove the entire abusive netblock?  Why didn't your upstream
> contact other providers with your evidence, and together remove the
> abusive network from the global routing tables?
>
My hoster did mail, his upstream is EGI, however, EGI does not want to
block/filter since it would pollute their routers they say.
I asked through my hoster if they would be willing to place a simple UDP
filter, blocking all of it. They refuse.



signature.asc
Description: OpenPGP digital signature

Re: Why does abuse handling take so long ?

[goemon]

On Sun, 13 Mar 2011, Jeff Wheeler wrote:

So ultimately, there is already a good framework in place to
substantially "fix" this problem.  No one uses it.  That is unlikely
to change until there is an economic incentive, such as a lawsuit by
someone targeted by DoS which can be proven to be originated from a
negligent network, causing calculable damages.  Until some network has
to pay out a million bucks because they sat on their hands, I don't
see anything changing.


Exactly.

Make this a question of economics and the problem will solve itself.

It has to become more expensive to ignore abuse than it is to deal with 
it.


Until that changes, the abuse will continue.

Re: Why does abuse handling take so long ?

[Jeff Wheeler]

On Sun, Mar 13, 2011 at 5:33 PM, Florian Weimer  wrote:
> Not that the IRTs are often not the party you want to talk to anyway.

This is why my post highlights the underlying mechanism/system.  It
can and should be used to streamline DDoS mitigation.  It is
unfortunately not in practical use, since the cost of ignoring DoS
originating from one's network is generally low or zero.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: estimation of number of DFZ IPv4 routes at peak in the future

[Jeff Wheeler]

On Sun, Mar 13, 2011 at 3:42 PM, Christopher Morrow
 wrote:
> not everyone drinks the mpls koolaide... so it's not always 'just a
> label switch' and depending upon how large your PE mesh is, there are

If it isn't just a label switch, then features can (and sometimes do)
drive upgrades (therefore costs.)

> not need that info, but the edge likely does, yes? Have 100g customers
> today? planning on having them in the next ~8/12/18 months?

If you did your purchasing the way Bill Herrin suggests, you'd buy a
box with 100GbE ports for a POP or branch that is not projected to
have 100GbE customers, just because it's the biggest box.  His
position is that man-power to do an upgrade is always more costly than
capital dollars for the actual equipment, and ignores the fact that
the biggest box is by no means guaranteed to offer new *features*
which may be required.

I think most of your post is responding to a mis-read of my post, so
I'll skip back to the FIB size question at hand:

> sometimes... sometimes it's just business. I suppose the point here is
> that a box doesn't live ~12 months or even 24, it lives longer.
> Planning that horizon today is problematic when a box today (even the
> largest box) tops out just north of 2m routes (v4, I forget the mix
> v4/6 numbers). your network design may permit you to side step that
> issue in places, but planning for that number is painful today.

I'm not comfortable making the generalization that buying the box with
the largest available FIB is always the most cost-effective choice.
In some "box roles," traffic growth drives upgrades, and increased FIB
size in future boxes will be one advantage of a future upgrade that
also increases port speed or density.  In other "box roles," features
drive upgrades, and again, FIB size may increase in future boxes which
will be bought anyway to gain desired features.

It's foolish and overly-simplistic to assume that every box upgrade
will be driven by an eventual exhaustion of FIB capacity.

Currently, FIB capacity is being driven by the needs of service
providers' VPN PE boxes.  This is great for networks that do not have
that need, because it is driving FIB capacity up (or cost down) and
further reducing the chance that FIB exhaustion will trigger an
upgrade before other factors, such as port speed/density/features.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Why does abuse handling take so long ?

[Florian Weimer]

* Jeff Wheeler:

> On Sun, Mar 13, 2011 at 7:45 AM, Alexander Maassen
>  wrote:
>> In most cases the only thing the abuse@ contacts do as hoster, is relay
>> the mail to the client but do not dare to do anything themself, even if
>
> The RIPE IRR database contains a systemic means for operators,
> responsible for IP address blocks, to exchange PGP-signed messages
> amongst each-other in relation to security incidents.  It
> unfortunately does not see much use: under 1% of allocations in RIPE's
> database include any reference to one of only 235 "incident response
> teams," which are conceptually similar to a POC.

Not that the IRTs are often not the party you want to talk to anyway.
They don't run the box, and in many cases, they don't even run the
network, so they can put in filters (even if they wanted).  In many
cases, the IRT object routes complaints *away* from the party who is
capable of taking action.

Re: Why does abuse handling take so long ?

[Brielle Bruns]

On 3/13/11 7:41 AM, William Pitcock wrote:

well, they should care.  if a customer is compromised and ddosing, it
costs the provider money (additional traffic being pushed bringing your
95% closer to your commit levels or possibly causing an overage to be
incurred.)

by doing nothing it may wind up costing them something - even if they
can make the money back by passing the overage onto the customer, there
is a high likelyhood that the customer will just jump ship and not pay
the invoice and go elsewhere.

william




In the case of a DoS, a call to the legal dept of the ISP might do the 
trick.  One successful lawsuit against a provider for knowingly allowing 
their customers to DoS/DDoS would certainly change alot of attitudes 
about the value of an abuse desk.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Why does abuse handling take so long ?

[Andrew Kirch]

On 3/13/2011 1:24 PM, Joel Jaeggli wrote:
> On 3/13/11 8:36 AM, Andrew Kirch wrote:=
>> Is it time for another "notion of self-defense" in responding
>> to/retaliating against a DDoS attack of sufficient strength to hold down
>> a large network, or resource?
> Because there just aren't enough internet vigilantes already...
>
The problem does seem to persist.  10 years later and DDoS, it's
mitigation, and asleep at the switch abuse departments are still a problem.

Re: Why does abuse handling take so long ?

[Brielle Bruns]

On 3/13/11 7:02 AM, sth...@nethelp.no wrote:

Well now, I'd say this varies considerably. There are definitely ISPs
that care and*do*  work hard at reducing abuse. But even so - assuming
I'm an ISP that cares,

- You're presenting me with evidence of abuse. OK, I don't know you.
Why should I believe your evidence? At best I'm going to take it as a
*hint*.
- If I take your evidence as a hint, I'm going to want to correlate it
with my own logs. This takes time.



This also applies in reverse when your asking to get out of a DNSbl. 
FWIW, when you deal with me on getting out of the AHBL, how well you 
handle my abuse report affects how well I handle your request to be 
delisted.  :)


effort in == effort out



- I probably have customer contracts in place that specify under what
circumstances I can actually take the customer off net. My tolerance
of abuse may not be the same as your. Also, "due process" means that
these things take time.


You aren't by chance related to Andrew Stevens?  He's been going on 
recently about "due process" (quotes and all) to the point where certain 
newsgroups are flooded with socks.



If not, then you have my apology :)



Steinar Haug, Nethelp consulting,sth...@nethelp.no



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: estimation of number of DFZ IPv4 routes at peak in the future

[Christopher Morrow]

On Sun, Mar 13, 2011 at 2:11 PM, Jeff Wheeler  wrote:
> On Sun, Mar 13, 2011 at 1:27 PM, Christopher Morrow
>  wrote:
>> there's probably a different need in TOR and BO/SOHO locations than
>> core devices, eh?
>
> In today's backbone, this is certainly true.  Feature-driven upgrades
> shouldn't be much of a factor for "P boxes" today, because modern
> networks have the option of simply label-switching in the core (just
> like 1990s networks could ATM/Frame-switch) without doing much of
> anything else.  Feature-driven upgrades should be largely confined to
> "PE boxes."
>

not everyone drinks the mpls koolaide... so it's not always 'just a
label switch' and depending upon how large your PE mesh is, there are
still some challenges in scaling this. MPLS also only shifts the
burden to another place, if you provide ip-transit and you need a full
table you'll have to put those routes somewhere. Sure the 'core' may
not need that info, but the edge likely does, yes? Have 100g customers
today? planning on having them in the next ~8/12/18 months?

> For the same reason, upgrading a P box should be easy, not hard.
> After all, it's just label-switching.  In today's backbones, it should

upgrades aren't hard, unless you get yourself into a SPOF situation
with the 'P' router(s)... mechanically the upgrades aren't hard.
Cost-wise though it could be, it depends upon your particular cost
structure I imagine.

> be more practical than ever to buy the most cost-effective box needed
> for now and the predictable near-term.  Cost per gigabit continues to
> fall.  Buying dramatically more capacity than is planned to be
> necessary sinks capital dollars into a box that does nothing but
> depreciate.

The discussion at the RAWS meeting, and which seems to hold true for
larger networks, is that a box lives in the network for ~5-7 years.
First, for the core-class device today, in the core, then
progressively further to the edge. Some thought goes into 'today I
have X requirements, I can project based on some set of metrics I'll
have X+Y tomorrow.'

> I realize that organizationally-painful budgeting and purchasing
> processes often drive networks to buy the biggest thing available.
> Vendors understand this, too: they love to sell you a much bigger box
> than you need just because upgrading is hard to get approved so you
> don't want to do it any more frequently than necessary, even when that
> behavior is detrimental to cash-flow and bottom line.  The more broken
> your organization, the more you need to spend extra money on "too big"
> boxes.  Sounds pretty self-defeating, doesn't it?

sometimes... sometimes it's just business. I suppose the point here is
that a box doesn't live ~12 months or even 24, it lives longer.
Planning that horizon today is problematic when a box today (even the
largest box) tops out just north of 2m routes (v4, I forget the mix
v4/6 numbers). your network design may permit you to side step that
issue in places, but planning for that number is painful today.

-Chris

Re: estimation of number of DFZ IPv4 routes at peak in the future

[Jeff Wheeler]

On Sun, Mar 13, 2011 at 1:27 PM, Christopher Morrow
 wrote:
> there's probably a different need in TOR and BO/SOHO locations than
> core devices, eh?

In today's backbone, this is certainly true.  Feature-driven upgrades
shouldn't be much of a factor for "P boxes" today, because modern
networks have the option of simply label-switching in the core (just
like 1990s networks could ATM/Frame-switch) without doing much of
anything else.  Feature-driven upgrades should be largely confined to
"PE boxes."

For the same reason, upgrading a P box should be easy, not hard.
After all, it's just label-switching.  In today's backbones, it should
be more practical than ever to buy the most cost-effective box needed
for now and the predictable near-term.  Cost per gigabit continues to
fall.  Buying dramatically more capacity than is planned to be
necessary sinks capital dollars into a box that does nothing but
depreciate.

I realize that organizationally-painful budgeting and purchasing
processes often drive networks to buy the biggest thing available.
Vendors understand this, too: they love to sell you a much bigger box
than you need just because upgrading is hard to get approved so you
don't want to do it any more frequently than necessary, even when that
behavior is detrimental to cash-flow and bottom line.  The more broken
your organization, the more you need to spend extra money on "too big"
boxes.  Sounds pretty self-defeating, doesn't it?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Why does abuse handling take so long ?

[William Allen Simpson]

On 3/13/11 7:45 AM, Alexander Maassen wrote:

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?


Because network operators rarely get together and turn off routing to
abusive hosting.  On the few occasions that has happened, it took years
of consensus building.

So, part of the problem is *your* upstream.  Why didn't your upstream
actively remove the entire abusive netblock?  Why didn't your upstream
contact other providers with your evidence, and together remove the
abusive network from the global routing tables?

As we get more experience with global "cyberwar", we're going to need
faster response mechanisms.

What will we do as some major government coordinates an attack on another?

What will we do as some major North American government coordinates an
attack on another region or facility?

In need of an att person at the slo cls

[Chris McDonald]

Please ping me off list.  I'm in urgent  need of escalation of a xcon.

Thx
Chris
cmcdon...@pccwglobal.com

-- 
Sent from my mobile device

Re: estimation of number of DFZ IPv4 routes at peak in the future

[Christopher Morrow]

On Sat, Mar 12, 2011 at 8:44 PM, Jeff Wheeler  wrote:
> On Sat, Mar 12, 2011 at 7:27 PM, William Herrin  wrote:
>> That must be my mistake then, because I thought the exercise was
>> building it in a way that it stays built for the maximum practical
>> number of years. When it has to be touched again (or tweaked if it
>
> So when you upgrade a device, you always buy the suitable device which
> has the highest capabilities?  You put in a top-of-rack switch with
> 10GbE for servers with no 10GbE ports and no plans of needing 10GbE
> connectivity to the next round of servers?  You buy a modular router
> for branch offices that have only a few workstations and no
> predictable need for upgraded connectivity?

there's probably a different need in TOR and BO/SOHO locations than
core devices, eh?

Re: Why does abuse handling take so long ?

[Joel Jaeggli]

On 3/13/11 8:36 AM, Andrew Kirch wrote:
> On 3/13/2011 8:39 AM, goe...@anime.net wrote:
>> On Sun, 13 Mar 2011, Alexander Maassen wrote:
>>> Why o why are isp's and hosters so ignorant in dealing with such issues
>>> and act like they do not care?
>>
>> they don't act like they do not care. they really *don't* care. no
>> acting.
>>
>> 1) you're not a direct customer, why should they do anything? by doing
>> nothing it cost them nothing.
>> 2) why should they do anything to shut down paying customers? shutting
>> down abusive customers is shutting off revenue sources.
>> 3) lifting a finger is too much like work. it costs the money and
>> gains them nothing.
>>
>> the only way to correct this behavior is to make it more expensive for
>> providers to retain abusive customers than it is to keep them.
>>
> Is it time for another "notion of self-defense" in responding
> to/retaliating against a DDoS attack of sufficient strength to hold down
> a large network, or resource?

Because there just aren't enough internet vigilantes already...

> Andrew
> 

Re: Why does abuse handling take so long ?

[Jeff Wheeler]

On Sun, Mar 13, 2011 at 7:45 AM, Alexander Maassen
 wrote:
> In most cases the only thing the abuse@ contacts do as hoster, is relay
> the mail to the client but do not dare to do anything themself, even if

The RIPE IRR database contains a systemic means for operators,
responsible for IP address blocks, to exchange PGP-signed messages
amongst each-other in relation to security incidents.  It
unfortunately does not see much use: under 1% of allocations in RIPE's
database include any reference to one of only 235 "incident response
teams," which are conceptually similar to a POC.

Other things have been tried but haven't reached "critical mass" also,
such as dial-by-ASN VOIP connectivity.

The real problem with handling serious network abuse is it's pretty
hard to get through the "bozo filter" and actually reach anyone who
might understand your request or complaint (DDoS), let alone have the
power to act.  The anti-spam folks have honestly made this problem
far, far worse, by slamming every role mailbox they can find for every
network operator, regardless of whether or not a specific mailbox for
email-related abuse exists or how good (or bad) a network may be at
keeping spam off its network.  I hope this remark doesn't steer the
thread far off-topic, but I wish the anti-spam folks would realize how
counter-productive it is to intentionally send the same complaints to
a multitude of different abuse mailboxes.

For this reason, it really is necessary to have an automatic filtering
mechanism in place just to make sure the network abuse people don't
have to sift through messages which are mostly related to email abuse.

If operators would decide to use a system like IRT, supported in RIPE
IRR, then we would not only be able to filter out a lot of the B.S.,
we would also know that signed messages complaining of DDoS coming in
were actually from the security folks at the complaining organization,
people who have authority to make requests on behalf of the org that
"owns" related netblocks.

This pretty much eliminates the "why should I believe your evidence?"
argument, because we shouldn't have to believe anyone's evidence to at
least block traffic towards the netblocks they operate.

For example: if I am an end-user with address 192.0.2.80 and my web
site is being subject to DDoS which I believe is originating from
203.0.113.66, I would contact my ISP, who registers themselves as the
IRT for 192.0.2.0/24.  My ISP would probably do a sanity check on my
claim, examine their netflow, etc. and then agree that 203.0.113.66 is
a source of the DDoS.  They'd see that an IRT is registered for
203.0.113.0/24 and send over a PGP-signed message to the counter-party
IRT.  That IRT would verify the PGP signature and association with the
target of the DoS, 192.0.2.80, and at that point, they would have
absolutely zero excuse for not immediately dropping all traffic from
203.0.113.66 towards me at 192.0.2.80.

It doesn't matter if there are any logs or "evidence," it matters that
the proven security/abuse contact for 192.0.2.0/24 requested that the
counter-party stop sending traffic to 192.0.2.0/24.  Whether or not
the ISP for 203.0.113.66 decides to investigate any further is up to
them; maybe they log some traffic, find a compromised host, and shut
it down.  Maybe they really don't care.

Now that you know people are capable of doing all that based on data
in RIPE's trusted IRR database, you may also realize that this process
could be streamlined to any point between "human reads email, checks
relationships, and configures network" all the way to "script reads
email, checks relationships, and configures network."  Implementing
this could save NOCs time (if they really cared about outgoing DDoS
from their networks) and improve response to network abuse.

So ultimately, there is already a good framework in place to
substantially "fix" this problem.  No one uses it.  That is unlikely
to change until there is an economic incentive, such as a lawsuit by
someone targeted by DoS which can be proven to be originated from a
negligent network, causing calculable damages.  Until some network has
to pay out a million bucks because they sat on their hands, I don't
see anything changing.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Why does abuse handling take so long ?

[Andrew Kirch]

On 3/13/2011 8:39 AM, goe...@anime.net wrote:
> On Sun, 13 Mar 2011, Alexander Maassen wrote:
>> Why o why are isp's and hosters so ignorant in dealing with such issues
>> and act like they do not care?
>
> they don't act like they do not care. they really *don't* care. no
> acting.
>
> 1) you're not a direct customer, why should they do anything? by doing
> nothing it cost them nothing.
> 2) why should they do anything to shut down paying customers? shutting
> down abusive customers is shutting off revenue sources.
> 3) lifting a finger is too much like work. it costs the money and
> gains them nothing.
>
> the only way to correct this behavior is to make it more expensive for
> providers to retain abusive customers than it is to keep them.
>
Is it time for another "notion of self-defense" in responding
to/retaliating against a DDoS attack of sufficient strength to hold down
a large network, or resource?

Andrew

Re: Why does abuse handling take so long ?

[Florian Weimer]

* Alexander Maassen:

> In most cases the only thing the abuse@ contacts do as hoster, is relay
> the mail to the client but do not dare to do anything themself, even if
> you provide them with a shitload of logs, even if you call them and say
> that the attack from their source is still continueing, they refuse to
> look into it and shutdown the source. And that pisses me off badly.

There is a relatively nice way of putting this.

If you can't contact the customer and don't know what they are doing,
it is difficult to estimate the risk from terminating the customer's
connectivity.  Therefore, giving them some time to react---4 business
hours or perhaps even a business day---seems reasonable, and this can
be a very long time span for many types of network abuse, especially
when time zones are taken into account.

> Why o why are isp's and hosters so ignorant in dealing with such issues
> and act like they do not care?

The less nice way is that many hosters attract customers who don't
care if they are compromised.  These customers do not perceive abuse
notifications as valuable, so the hoster gains nothing from forwarding
them: the abuse won't stop, and the customer is likely less happy than
before.

Re: Why does abuse handling take so long ?

[William Pitcock]

On Sun, 13 Mar 2011 05:39:02 -0700 (PDT)
goe...@anime.net wrote:

> On Sun, 13 Mar 2011, Alexander Maassen wrote:
> > Why o why are isp's and hosters so ignorant in dealing with such
> > issues and act like they do not care?
> 
> they don't act like they do not care. they really *don't* care. no
> acting.

well, they should care.  if a customer is compromised and ddosing, it
costs the provider money (additional traffic being pushed bringing your
95% closer to your commit levels or possibly causing an overage to be
incurred.)

by doing nothing it may wind up costing them something - even if they
can make the money back by passing the overage onto the customer, there
is a high likelyhood that the customer will just jump ship and not pay
the invoice and go elsewhere.

william

Re: Why does abuse handling take so long ?

[sthaug]

> > Why o why are isp's and hosters so ignorant in dealing with such issues
> > and act like they do not care?
> 
> they don't act like they do not care. they really *don't* care. no acting.

Well now, I'd say this varies considerably. There are definitely ISPs
that care and *do* work hard at reducing abuse. But even so - assuming
I'm an ISP that cares,

- You're presenting me with evidence of abuse. OK, I don't know you.
Why should I believe your evidence? At best I'm going to take it as a
*hint*.
- If I take your evidence as a hint, I'm going to want to correlate it
with my own logs. This takes time.
- I probably have customer contracts in place that specify under what
circumstances I can actually take the customer off net. My tolerance
of abuse may not be the same as your. Also, "due process" means that
these things take time.

Steinar Haug, Nethelp consulting, sth...@nethelp.no

Re: Why does abuse handling take so long ?

[goemon]

On Sun, 13 Mar 2011, Alexander Maassen wrote:

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?


they don't act like they do not care. they really *don't* care. no acting.

1) you're not a direct customer, why should they do anything? by doing nothing 
it cost them nothing.
2) why should they do anything to shut down paying customers? shutting down 
abusive customers is shutting off revenue sources.
3) lifting a finger is too much like work. it costs the money and gains them 
nothing.

the only way to correct this behavior is to make it more expensive for
providers to retain abusive customers than it is to keep them.

Why does abuse handling take so long ?

[Alexander Maassen]

Dear nanog members,

As current maintainer of DroneBL I happen to receive a lot of unwanted
packets in the form of DDoS attacks, now the DDoS itself is not the real
problem, dealing with it the fast way is.

Now most of you would think: Just filter it, put a big firewall in front
of it, bla bla bla bla. But what I'm really talking about is the
ignorance most providers show when it comes to handling the abuse when
it gets reported.
The issue in there being, it's way too slow, and my hoster needs to
temporary nullroute my ip range in order to protect his network.
We both mail all the involved providers and sometimes need to wait days
before hostings act upon the mail.

In most cases the only thing the abuse@ contacts do as hoster, is relay
the mail to the client but do not dare to do anything themself, even if
you provide them with a shitload of logs, even if you call them and say
that the attack from their source is still continueing, they refuse to
look into it and shutdown the source. And that pisses me off badly.

Why o why are isp's and hosters so ignorant in dealing with such issues
and act like they do not care?

Kind regards,
Alexander Maassen
Maintainer DroneBL



signature.asc
Description: OpenPGP digital signature