Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Rabbi Rob Thomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Dear team,

> Since we see >1Tbps DDOS attacs in the wild, I suppose
> out-of-the-box DDOS mitigation suppliers have lost this race. There
> is no operator in Switzerland which can handle 1Tbps DDOS attacks.
> 
> When we saw DDOS against digitec.ch and others earlier this year, I
> was a bit surprised that none of the so called "experts" proposed
> regional BGP propagation as a remedy.

May I offer up UTRS as a model or perhaps part of your solution?



UTRS is a system that helps mitigate large infrastructure attacks by
leveraging an existing network of cooperating BGP speakers such as
ISPs, hosting providers and educational institutions that
automatically distributes verified BGP-based filter rules from victim
to cooperating networks.

Victims can now effectively alleviate attacks quickly and across the
world at lightning speed. Additionally, by using UTRS, operators will
also be stopping the attack traffic at the source, saving many
would-be attack packets from their own network, as well as preventing
them from taking up unnecessary network resources at every other
network in between.

Be well,
Rob.
- -- 
Rabbi Rob Thomas   Team Cymru
   "It is easy to believe in freedom of speech for those with whom we
agree." - Leo McKern

-BEGIN PGP SIGNATURE-

iQIcBAEBCAAGBQJX8H6aAAoJEEPoYWL6hfKNSi0P/RVeq1gj9kW3q1avLcnPy0k/
1DIlz5e83iqTx2skqn+haHqvYuQ0G4xAz9QrqKW1oGSICkjVUhG/ZP6iDv1YG1wF
b0tFSZs+Halm7OKs57ZJ8xXX2+bwi9ydS6h1lhY1hs+fZPyJVsuxqvrLGvX6BLPB
8fXuUDBHmUjbsi3pVxXjC8L5IQgGk6UbrtbJGAxbSQj9n6rE93Tq28cQ7o0Hfj/W
jZvJDZmvFmSeidAfOGwtEH6ZEQnsg/7evclFq9qRRfjFu0Ypygd01AlFVaIqxzLH
IkLz/gD8CHkmt0ImKPO9F9sHCwimVvW784bCrAiBCgNp/iv7f6jjf4FDMJYP1+NH
hYhC3MNLsV41aEP/BjI8p9lZudlI7BweExu8RX4FEv3KQ7XWMAIZLte7gbNjXBMe
2DYlIrVBPwICa6k98cEpbl2n2YEo/D1kohkXDh2OWoy67B4WABJFVZ81AeDFvQU6
AvzXUgmgLhBJzTeagTQKQeO8gfgO6SrfwfcwpKeC8u+WDFE3CBAsRIbwgvyMwpkq
WXjC1wv93EAgiGkj3Ep6yWz+NbrdN26/IDf8zt59jbf6qQwvvwCInLYp4TCdJEDf
KbUmk8CngOOmU+Vk5krYcs2PIvf3ggXvBanN/p2FJX4vw84NdQCDu1fFqbwjzxyM
mYSHg+7/SJklcQ0VpMfK
=1gey
-END PGP SIGNATURE-


___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Will van Gulik]

binNDFG1Id5cb.bin
Description: PGP/MIME Versions Identification


encrypted.asc
Description: OpenPGP encrypted message

___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Jeroen Massar]

On 2016-10-01 20:24, Patrick Albrecht wrote:
> Hi
> 
> I'm a employee of a good known E-Commerce site here in switzerland and I
> would like to share some thoughts from my side if that's okay for all. I
> hope I understood well enough what you plan. Otherwise just ignore what
> I just wrote :)
>> Given that e-commerce such as digitec.ch is assumingly making 99.9% of
>> the revenue within Switzerland, their prefix doesn't need to reachable
>> from all over the world.
>
> That's correct, the /customer/ doesn't need to the reach the website
> from outsite switzerland normaly. But there're many 3rd-Party Provider
> for Newsletter, Monitoring etc. and distributors that need to be able to
> resolve digitec.ch outside of switzerland for example.

"resolve" implies DNS.

Peering is about BGP.

> (because there server are not located in switzerland) Mostly it's dispensable 
> if they
> can't reach the website or a ftp server for some minutes, but if they
> can't access the page for days the E-Commerce Site will have issue with
> orders, product availability, newsletter shipping etc. Also some
> 3rd-Party Scripts may use a dns lookup and would fail then.

You need to see that 'limited announce of prefix' would only happen in
the case of a DDoS, this, so that local sites / direct peers can reach
it, but it is 'dead' over transit, thus cutting off most DDoS traffic
that comes from strange countries (not .ch).

As for those external companies, if you are worried about them failing:
peer directly with them, setup a VPN or: move your stuff more local
where you have control.

Also, do realize that providing Swiss customer data to a foreign entity
might break various privacy regulations do ask your legal team and
of course inform your customers.

> There's also
> a possibilty that the employee reach the internet via a proxy outside
> of switzerland (due to a enterprise policy) so they wouldn't be able
> to access there site and couldn't work at all.

That is a weird "Enterprise policy". Doing business that way opens you
up to all kind of fun international laws concerning your business.

Also note that you can of course always announce to trusted peers that
are not in Switzerland...

The major point is "trusted peers". Ones that will clean up their
attacking hosts the moment you notify them.

> Of course if the site isn't available at all it's not a good experience
> for the customer and they may order there article on a other onlineshop,
> but if the website is online and doesn't work properly that's also not a
> optimal solution either.

Better test it out today what happens when your site gets DDoSsed to
bits, as the script kiddies have access to the same botnet know that
Krebs got sent after him... (see other mail).

> Addiontally to the fact that more and more E-Commerce Websites use
> DDoS-Protection services like akamai or cloudflare, only about half
> hosting there website on server in switzerland.

You might want to reconsider your hosting location ;)

Also, if you are paying those kind of companies: prepare to dig deep in
your pockets for DDoS protection... we are going to have a fun X-mas
this year...

Greets,
 Jeroen



___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Patrick Albrecht]

Hi

I'm a employee of a good known E-Commerce site here in switzerland and I
would like to share some thoughts from my side if that's okay for all. I
hope I understood well enough what you plan. Otherwise just ignore what
I just wrote :)
> Given that e-commerce such as digitec.ch is assumingly making 99.9% of
> the revenue within Switzerland, their prefix doesn't need to reachable
> from all over the world.
That's correct, the *customer* doesn't need to the reach the website
from outsite switzerland normaly. But there're many 3rd-Party Provider
for Newsletter, Monitoring etc. and distributors that need to be able to
resolve digitec.ch outside of switzerland for example. (because there
server are not located in switzerland) Mostly it's dispensable if they
can't reach the website or a ftp server for some minutes, but if they
can't access the page  for days the E-Commerce Site will have issue with
orders, product availability, newsletter shipping etc. Also some 3rd-
Party Scripts may use a dns lookup and would fail then. There's also a
possibilty that the employee reach the internet via a proxy outside of
switzerland (due to a enterprise policy) so they wouldn't be able to
access there site and couldn't work at all.

Of course if the site isn't available at all it's not a good experience
for the customer and they may order there article on a other onlineshop,
but if the website is online and doesn't work properly that's also not a
optimal solution either.

Addiontally to the fact that more and more E-Commerce Websites use DDoS-
Protection services like akamai or cloudflare, only about half  hosting
there website on server in switzerland.

--
  Patrick Albrecht
 Powered by FastMail

___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

[swinog] Krebs: Source Code for IoT Botnet ‘Mirai’ Released

[Jeroen Massar]

https://krebsonsecurity.com/2016/10/source-code-for-iot-botnet-mirai-released/

And now the script kiddies have their hands on it...

Enjoy that Internet...

Greets,
 Jeroen


___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Steven Glogger]

If I recall correctly 3303 has some communities which can be used for such a 
matter... 

gruss

-steven

> Am 01.10.2016 um 16:51 schrieb Fredy Kuenzler :
> 
> Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box
> DDOS mitigation suppliers have lost this race. There is no operator in
> Switzerland which can handle 1Tbps DDOS attacks.
> 
> When we saw DDOS against digitec.ch and others earlier this year, I was
> a bit surprised that none of the so called "experts" proposed regional
> BGP propagation as a remedy.
> 
> Given that e-commerce such as digitec.ch is assumingly making 99.9% of
> the revenue within Switzerland, their prefix doesn't need to reachable
> from all over the world. If the prefix of a Swiss e-commerce would be
> reachable from Swiss broadband providers only, the DDOS is mitigated, as
> the vast majority of the botnet is lacking a route to the targeted
> victim IP address.
> 
> To achieve this I think we need a collaborative community effort setting
> up a common procedure and define a BGP communitiy with the effect "do
> not announce beyond Switzerland".
> 
> An e-commerce should be able to hit the button injecting this defined
> BGP community when under attack (or permanently, of course).
> 
> I suppose to make this idea a success we need to have all major
> operators in Switzerland on board (3303, 6730, 6830) and I suppose the
> smaller operators will follow in their own interest to avoid blackholes.
> 
> Anyone? I think it's good if a somewhat "neutral body" with decent BGP
> knowledge could take the lead for such a working group, maybe SWITCH or
> SwissIX?
> 
> --
> Fredy Kuenzler
> 
> -
> Fiber7. No Limits.
> https://www.fiber7.ch
> -
> 
> Init7 (Switzerland) Ltd.
> AS13030
> St.-Georgen-Strasse 70
> CH-8400 Winterthur
> Skype:   flyingpotato
> Phone:   +41 44 315 4400
> Fax: +41 44 315 4401
> Twitter: @init7 / @kuenzler
> http://www.init7.net/
> 
> 
> ___
> swinog mailing list
> swinog@lists.swinog.ch
> http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog



___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Milan Trenka]

Agree, this has to be done.

See also https://fe.nix.cz/en/  in Czech republic some ISP realized this
idea about 3 year ago. 

And see also some IP-Transit-Providers already have regional restricted
route propagation in their BGP community. 
https://www.gtt.net/services/internet-services/ip-transit/bgp-communities/ 

So in case of DDOS it need only to add this community to the propagated
network instead of black holing one address. 

Best regards 

Milan

TRENKA INFORMATIK AG 
___
Seefeldstrasse 108
8008 Zürich
Tel: +41 44 383 63 07
mailto:m...@trenka.ch



-Ursprüngliche Nachricht-
Von: swinog-boun...@lists.swinog.ch [mailto:swinog-boun...@lists.swinog.ch]
Im Auftrag von Jeroen Massar
Gesendet: Samstag, 1. Oktober 2016 18:04
An: Fredy Kuenzler ; swi...@swinog.ch
Betreff: Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

On 2016-10-01 16:51, Fredy Kuenzler wrote:
[..]
> To achieve this I think we need a collaborative community effort 
> setting up a common procedure and define a BGP communitiy with the 
> effect "do not announce beyond Switzerland".

Great initiative! If you need extra hands, don't hesitate to yell...

Did you btw see:
 http://www.trustednetworksinitiative.nl/
 https://www.nl-ix.net/solutions/security-solutions/trusted-routing
 https://ams-ix.net/technical/trusted-networks-initiative

We should have a Swiss equivalent:
 - trusted and direct contacts
 - require BCP38 where possible
 - proper statistics/monitoring
 - proper & standardized "You are DDoS'ing" notifications
   providing Flow info as "proof".
 - proper & standardized "We put customer in walled garden"

The problem with the latter: VoIP... thus the walled garden needs to not
block that due to "emergency services". Thus a throttle and a call to the
customer might be needed to inform them...


As for the BGP thing... I thought folks had a deal like that per default for
all their prefixes :)

It is after all the reason why quite a few IRC servers live(d) in PI
/24:
 - always the prefix to local peers
 - when 'normal' also announce to transit providers

When DDoS comes:
 - stop announcing to transits
 - check monitoring/stats tools which local peers are sending crap
   traffic and kick them hard

Now, the more important part is actually that:
 - You have good relationship with your transit
 - You have amazing relationship with your local peers:
   so that you can call them and notify them of the problem
 - Have proper instrumentation

Of course, when you have that, you might also want to peek at:
 - RPF / BCP38 kinda stuff and 'force' or 'require' that from your peers
   thus avoiding any spoofed traffic from them.

Not that BCP38 actually solves anything for these DDoS's as there are just
thousands of botted devices involved...

Proper flows everywhere, proper notification and shutdowns at the source are
the only way to go there.

And that will involve people calling helpdesks because:
 - their botted host is sending too much traffic
   making "The Internet Slow" and them complaining
 - they are disconnected, as you caught them participating.

Which might not fly with management in many places as helpdesk == money.

Hence, maybe to cover that at least, having a admin.ch rule, BAKOM maybe,
that allows an ISP to "restrict access", eg wall-garden an endpoint that is
causing DDOS attack would be a good thing.

Though, does not have to go that high actually, having a general consensus
between ISPs that this is the case and putting it in the end-user agreement
could be good enough to cover their ass a bit.

Greets,
 Jeroen



___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog


smime.p7s
Description: S/MIME cryptographic signature

___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Jeroen Massar]

On 2016-10-01 16:51, Fredy Kuenzler wrote:
[..]
> To achieve this I think we need a collaborative community effort setting
> up a common procedure and define a BGP communitiy with the effect "do
> not announce beyond Switzerland".

Great initiative! If you need extra hands, don't hesitate to yell...

Did you btw see:
 http://www.trustednetworksinitiative.nl/
 https://www.nl-ix.net/solutions/security-solutions/trusted-routing
 https://ams-ix.net/technical/trusted-networks-initiative

We should have a Swiss equivalent:
 - trusted and direct contacts
 - require BCP38 where possible
 - proper statistics/monitoring
 - proper & standardized "You are DDoS'ing" notifications
   providing Flow info as "proof".
 - proper & standardized "We put customer in walled garden"

The problem with the latter: VoIP... thus the walled garden needs to not
block that due to "emergency services". Thus a throttle and a call to
the customer might be needed to inform them...


As for the BGP thing... I thought folks had a deal like that per default
for all their prefixes :)

It is after all the reason why quite a few IRC servers live(d) in PI
/24:
 - always the prefix to local peers
 - when 'normal' also announce to transit providers

When DDoS comes:
 - stop announcing to transits
 - check monitoring/stats tools which local peers are sending crap
   traffic and kick them hard

Now, the more important part is actually that:
 - You have good relationship with your transit
 - You have amazing relationship with your local peers:
   so that you can call them and notify them of the problem
 - Have proper instrumentation

Of course, when you have that, you might also want to peek at:
 - RPF / BCP38 kinda stuff and 'force' or 'require' that from your peers
   thus avoiding any spoofed traffic from them.

Not that BCP38 actually solves anything for these DDoS's as there are
just thousands of botted devices involved...

Proper flows everywhere, proper notification and shutdowns at the source
are the only way to go there.

And that will involve people calling helpdesks because:
 - their botted host is sending too much traffic
   making "The Internet Slow" and them complaining
 - they are disconnected, as you caught them participating.

Which might not fly with management in many places as helpdesk == money.

Hence, maybe to cover that at least, having a admin.ch rule, BAKOM
maybe, that allows an ISP to "restrict access", eg wall-garden an
endpoint that is causing DDOS attack would be a good thing.

Though, does not have to go that high actually, having a general
consensus between ISPs that this is the case and putting it in the
end-user agreement could be good enough to cover their ass a bit.

Greets,
 Jeroen



___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Fredy Kuenzler]

On 01.10.2016 17:35, Gert Doering wrote:
> I think this is an awesome idea.
> 
> The situation is similar here in DE - nobody could stand an 1 Tbit 
> DDoS attack, and a large number of content offerings are targeted 
> only to german speaking customers, so if DE/A/CH work, 99% of the 
> customers are still able to reach the site.

Maybe we should widen the approach and define a collaborative BGP
community "do announce only in country X", when X is some ISO-3166
country number? A prefix then can contain multiple communities, i.E. to
cover the whole DACH region.

https://de.wikipedia.org/wiki/ISO-3166-1-Kodierliste

> I'm not really sure how this would work in your example - what if
> you have two customers in a given BGP announcement, one of them
> *does* want to be reached world-wide (like, corporate VPNs) and the
> other one is attacked?  Split the aggregate, or bit the bullet and
> have all of them with limited reach, for the time being?

I suppose the e-commerces using such a mechanism would be able to afford
their own /24 and a decent block of IPv6 space (in other words: buy
legacy PI or become LIR). Another option is new business for managed
hosting "DDOS bullet proof Switzerland Hosting", where the hoster
dedicates a /24 or bigger for permanent limited propagation.

-- 
Fredy Kuenzler

-
Fiber7. No Limits.
https://www.fiber7.ch
-

Init7 (Switzerland) Ltd.
AS13030
St.-Georgen-Strasse 70
CH-8400 Winterthur
Skype:   flyingpotato
Phone:   +41 44 315 4400
Fax: +41 44 315 4401
Twitter: @init7 / @kuenzler
http://www.init7.net/


___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Re: [swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Gert Doering]

Hi,

On Sat, Oct 01, 2016 at 04:51:36PM +0200, Fredy Kuenzler wrote:
> To achieve this I think we need a collaborative community effort setting
> up a common procedure and define a BGP communitiy with the effect "do
> not announce beyond Switzerland".

I think this is an awesome idea.

The situation is similar here in DE - nobody could stand an 1 Tbit DDoS
attack, and a large number of content offerings are targeted only to
german speaking customers, so if DE/A/CH work, 99% of the customers
are still able to reach the site.

I'm not really sure how this would work in your example - what if you
have two customers in a given BGP announcement, one of them *does* want
to be reached world-wide (like, corporate VPNs) and the other one is
attacked?  Split the aggregate, or bit the bullet and have all of them
with limited reach, for the time being?

(We currently work this "the other way round" by using the "out of country"
and "out of continent" blackhole communities offered by NTT - so the customer 
under attack would be announced as a "faraway RTBH" route - but this isn't
good enough yet either, as not all transits offer this...)

Gert Doering
-- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AGVorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14  Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444   USt-IdNr.: DE813185279


___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

[swinog] DDOS >1Tbps - Swiss-wide (regional) BGP propagation?!

[Fredy Kuenzler]

Since we see >1Tbps DDOS attacs in the wild, I suppose out-of-the-box
DDOS mitigation suppliers have lost this race. There is no operator in
Switzerland which can handle 1Tbps DDOS attacks.

When we saw DDOS against digitec.ch and others earlier this year, I was
a bit surprised that none of the so called "experts" proposed regional
BGP propagation as a remedy.

Given that e-commerce such as digitec.ch is assumingly making 99.9% of
the revenue within Switzerland, their prefix doesn't need to reachable
from all over the world. If the prefix of a Swiss e-commerce would be
reachable from Swiss broadband providers only, the DDOS is mitigated, as
the vast majority of the botnet is lacking a route to the targeted
victim IP address.

To achieve this I think we need a collaborative community effort setting
up a common procedure and define a BGP communitiy with the effect "do
not announce beyond Switzerland".

An e-commerce should be able to hit the button injecting this defined
BGP community when under attack (or permanently, of course).

I suppose to make this idea a success we need to have all major
operators in Switzerland on board (3303, 6730, 6830) and I suppose the
smaller operators will follow in their own interest to avoid blackholes.

Anyone? I think it's good if a somewhat "neutral body" with decent BGP
knowledge could take the lead for such a working group, maybe SWITCH or
SwissIX?

--
Fredy Kuenzler

-
Fiber7. No Limits.
https://www.fiber7.ch
-

Init7 (Switzerland) Ltd.
AS13030
St.-Georgen-Strasse 70
CH-8400 Winterthur
Skype:   flyingpotato
Phone:   +41 44 315 4400
Fax: +41 44 315 4401
Twitter: @init7 / @kuenzler
http://www.init7.net/


___
swinog mailing list
swinog@lists.swinog.ch
http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog