Re: [j-nsp] Decoding DDOS messages

[adamv0025]

> Saku Ytti
> Sent: Wednesday, March 18, 2020 4:37 PM
> 
> On Wed, 18 Mar 2020 at 18:30, John Kristoff  wrote:
> 
> > Yep, I get all that.  I can tighten that up.  Care to show us how you
> > do loopback filters?
> 
> Really Juniper would be in the best position to automatically generate
> lo0 filter when none is provided, which would be really really good, not
> optimal, but really good. Bit of like generated-LPTS.
> 
That, but most importantly separate control-plane and management-plane
security like in XR.
If one could do this in Junos:
XR-example: control-plane management-plane inband interface xxx allow
SSH 
 -listing only my core facing and/or oob mgmt ports.
Then it would not matter that operator's iACL or lo0 filter has holes
(allowing ssh from BGP source port). 

adam   

___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Jason Healy]

Saku,

Thank you for your responses.  I'm trying to learn about this as I go...


On Mar 18, 2020, at 10:39 AM, Saku Ytti  wrote:
> 
> Your L2 should be in its virtual-switch/vpls (doesn't imply VPLS)
> instance with forwarding-plane filter policing BUM. But unrelatd to
> subject.

You might need to email me off-list for that one... I'm not sure if I'm 
following the theory on that.

>> IPMCAST-miss (lots of this one!)
> 
> Probably punts for programming flow, and subsequent will be HW
> switched. You may want to have ACL to drop all MCAST traffic at edge.
> This should be 0 if you don't actually run multicast.

We're applying an l2 filter at the vlan level to scrub all but well-known 
multicast on this switch.  Can it still get to the CPU even if blocked in this 
manner?  Or is the flow assignment done prior to l2 firewalling?

>> ARP
> 
> Self-explanatory? You shouldn't want to see this exceeded, ideally you
> should police this on IFD level, but I'm not sure if QFX5k can, MX
> can.

Assuming there is not malicious traffic, wouldn't exceeding this counter imply 
that the defaults are tuned too low?  We are a small school with ~1500 devices. 
 While we might get bursts of ARP traffic at peak times (like when students 
move between classes), I would be surprised if it was so high as to be a 
concern.

>> TTL
> 
> TTL exceeded message. Normal to hit this policer in uloops.

We're spoke-and-hub, static routing, so not expecting a lot of microloops due 
to convergence.  Possible we're seeing this from "lost" packets being misrouted 
to our ISP (then routed back).

>> Redirect
> 
> IP redirect, you probably want to disable them at network edge. This
> should be 0.

Is there an easy way to find/locate IP redirects?  I'm curious if these are 
sourced from our ISP.

>> L3MTU-fail
> 
> Egress MTU was too small for packet. It is punted for potentially ICMP
> message generation. Depending on config expected or unexpected.

We should be jumbo (9216) throughout, including uplink to our ISP.  Any way to 
narrow these down?

Thanks for all the replies, I'm starting to get a better idea on this.

Jason
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Saku Ytti]

On Wed, 18 Mar 2020 at 18:30, John Kristoff  wrote:

> Yep, I get all that.  I can tighten that up.  Care to show us how you
> do loopback filters?

It is situational, it's hard to come up with one-size-fits-all. One
approach would be basic skeleton, on top of which people then expand
what they need, which would likely be also then broken. Another option
would be to write exhaustive one, but exhaustive one necessarily has
compromises, so then people who don't need everything still take those
compromises.
Really Juniper would be in the best position to automatically generate
lo0 filter when none is provided, which would be really really good,
not optimal, but really good. Bit of like generated-LPTS.

I'm not sure if there is a utility in public template. But it's
something that I do occasionally think about, not just Junos or just
firewall, but also BGP, to show how to normalise BGP behaviour (no one
knows what their BGP policy is very accurately, as in almost every
case BGP policy is 'what ever is vendor default', and when you have
multivendor network, you have different policy in different  devices).




-- 
  ++ytti
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[John Kristoff]

On Wed, 18 Mar 2020 16:18:18 +
Saku Ytti  wrote:

> I set SPORT to 179
> I access your SSH port

Yep, I get all that.  I can tighten that up.  Care to show us how you
do loopback filters?

John
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Saku Ytti]

This wasn't the only problem, there are many issues, it's normal, I've
not read single lo0 filter in real network which isn't fundamentally
broken. Trying to tactically address the problems is waste of time
when redesign is needed.

On Wed, 18 Mar 2020 at 18:18, Saku Ytti  wrote:
>
> I'm your BGP speaker.
>
> I set SPORT to 179
> I access your SSH port
>
> On Wed, 18 Mar 2020 at 18:16, John Kristoff  wrote:
> >
> > On Wed, 18 Mar 2020 16:02:09 +
> > Saku Ytti  wrote:
> >
> > > It is completely broken, you use 'port' so you expose every port in your 
> > > system.
> >
> > Ha, OK thanks.  I think that would require some not so easy spoofing
> > unless I'm missing something.  We can convert any statement that just
> > uses port to directional, which I think will require additional rules
> > to tighten it up.  Feel free to submit example configs.
> >
> > John
>
>
>
> --
>   ++ytti



-- 
  ++ytti
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Saku Ytti]

I'm your BGP speaker.

I set SPORT to 179
I access your SSH port

On Wed, 18 Mar 2020 at 18:16, John Kristoff  wrote:
>
> On Wed, 18 Mar 2020 16:02:09 +
> Saku Ytti  wrote:
>
> > It is completely broken, you use 'port' so you expose every port in your 
> > system.
>
> Ha, OK thanks.  I think that would require some not so easy spoofing
> unless I'm missing something.  We can convert any statement that just
> uses port to directional, which I think will require additional rules
> to tighten it up.  Feel free to submit example configs.
>
> John



-- 
  ++ytti
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[John Kristoff]

On Wed, 18 Mar 2020 16:02:09 +
Saku Ytti  wrote:

> It is completely broken, you use 'port' so you expose every port in your 
> system.

Ha, OK thanks.  I think that would require some not so easy spoofing
unless I'm missing something.  We can convert any statement that just
uses port to directional, which I think will require additional rules
to tighten it up.  Feel free to submit example configs.

John
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Saku Ytti]

On Wed, 18 Mar 2020 at 17:01, John Kristoff  wrote:

>   

It is completely broken, you use 'port' so you expose every port in your system.


-- 
  ++ytti
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[John Kristoff]

On Wed, 18 Mar 2020 14:39:19 +
Saku Ytti  wrote:

> Unfortunately even non-broken lo0 filter is extremely uncommon, even
> MX book has fundamentally broken example, as is CYMRU example.

Team Cymru only lists a Cisco BGP, general NTP (which includes a
Juniper example), and Juniper IP multicast template publicly now:

  

If you are referring to one of those, there is an email right on the
page to contact them and you should if there are mistakes and
improvements.  They will welcome input.  I edited the NTP template and
helped facilitate the IP multicast one Lenny did, so if there is a
problem with either of those I'd be interested to know about it, but I
am no longer an employee of Team Cymru so I can't fix them.

The other templates, including a generic Juniper template you can
find via a net search, but not through Team Cymru's website navigation,
are many years old and no longer maintained. It would be unwise to
trust or relay on those.

I have some example templates for more recent stuff work I've done, but
does not cover currently this thread's case and may be less
generically applicable.  They are probably also not perfect, but people
are welcome to submit an issue there and I'll do my best to keep them
maintained:

  

John
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Tom Beecher]

This is the most recent Juniper document I had bookmarked for the QFX.

https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/protocols-edit-ddos-qfx-series.html

I agree with Saku that the ddos-policer is a good tool to use, but as he
said it requires turning for your specific environment to be at its most
useful. I don't want to discount the opinions of those who dislike it, but
many complaints about it seem to boil down to "the defaults didn't work, I
didn't tune it, therefore I hate it".

On Wed, Mar 18, 2020 at 10:42 AM Saku Ytti  wrote:

> Hey Jason,
>
> > Questions about the ddos-protection "features".  We're on a qfx5100-48
> running 16.1.  I know that folks on the list aren't always big fans of
> ddos-protection; I'm just trying to understand what is triggering it so I
> can make decisions about tuning/disabling/ignoring it.
>
> I am a big fan, it's great feature everyone should be running and
> tuning correctly. Unfortunately even non-broken lo0 filter is
> extremely uncommon, even MX book has fundamentally broken example, as
> is CYMRU example. And ddos-protection may be even more complicated.
>
> I'm not very familiar how it works on QFX5k, I'm more aware of MX
> behaviour, where it's really great.
>
> > We are not a service provider; we're an end site running a flat L2
> network (LAN) with the QFX as our L3 core for IRB and routing to our ISP.
> Since the QFX is seeing all the BUM traffic I'm curious if ddos-protection
> is being triggered as a result of seeing all the L2 packets.
>
> Your L2 should be in its virtual-switch/vpls (doesn't imply VPLS)
> instance with forwarding-plane filter policing BUM. But unrelatd to
> subject.
>
> > IPMCAST-miss (lots of this one!)
>
> Probably punts for programming flow, and subsequent will be HW
> switched. You may want to have ACL to drop all MCAST traffic at edge.
> This should be 0 if you don't actually run multicast.
>
> > ARP
>
> Self-explanatory? You shouldn't want to see this exceeded, ideally you
> should police this on IFD level, but I'm not sure if QFX5k can, MX
> can.
>
> > TTL
>
> TTL exceeded message. Normal to hit this policer in uloops.
>
> > Redirect
>
> IP redirect, you probably want to disable them at network edge. This
> should be 0.
>
> > L3MTU-fail
>
> Egress MTU was too small for packet. It is punted for potentially ICMP
> message generation. Depending on config expected or unexpected.
>
> > RESOLVE
>
> Traffic hitting connected DADDR which is not in ARP cache, we need to
> punt it for ARP resolution. Normal to see as there is constant
> background traffic to every DADDR.
>
> > L3NHOP
>
> Unsure.
>
> > So is that all ARP?  ARP that the switch needs to answer for?  Similar
> for the other packet types: are these thresholds for packets that the
> switch is processing (sent to the RE), or just for any traffic that is seen
> on any interface?  If it's just an issue of too much stuff going to the RE
> I can firewall it off so long as I know it's spurious.
>
> It's ARP packets verbatim (see RESOLVE which is non ARP packet
> triggering ARP resolution). Originally when ddos-protection was
> implemented resolve was not implemented, and RESOLVE packet was what
> ever classifier it hit, so if you sent BGP packets to unarpable DADDR
> it was eating BGP policer PPS, so you could easily get targets core
> iBGP down, and there was nothing they could do to stop you.
>
> > Sorry if I'm not asking the right questions... I'm just trying to figure
> out if these errors are actually problems that I need to track down, or if
> the default reporting is just too noisy.
> >
> > Thanks,
> >
> > Jason
> > ___
> > juniper-nsp mailing list juniper-nsp@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
>
>
>
> --
>   ++ytti
> ___
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] Decoding DDOS messages

[Saku Ytti]

Hey Jason,

> Questions about the ddos-protection "features".  We're on a qfx5100-48 
> running 16.1.  I know that folks on the list aren't always big fans of 
> ddos-protection; I'm just trying to understand what is triggering it so I can 
> make decisions about tuning/disabling/ignoring it.

I am a big fan, it's great feature everyone should be running and
tuning correctly. Unfortunately even non-broken lo0 filter is
extremely uncommon, even MX book has fundamentally broken example, as
is CYMRU example. And ddos-protection may be even more complicated.

I'm not very familiar how it works on QFX5k, I'm more aware of MX
behaviour, where it's really great.

> We are not a service provider; we're an end site running a flat L2 network 
> (LAN) with the QFX as our L3 core for IRB and routing to our ISP.  Since the 
> QFX is seeing all the BUM traffic I'm curious if ddos-protection is being 
> triggered as a result of seeing all the L2 packets.

Your L2 should be in its virtual-switch/vpls (doesn't imply VPLS)
instance with forwarding-plane filter policing BUM. But unrelatd to
subject.

> IPMCAST-miss (lots of this one!)

Probably punts for programming flow, and subsequent will be HW
switched. You may want to have ACL to drop all MCAST traffic at edge.
This should be 0 if you don't actually run multicast.

> ARP

Self-explanatory? You shouldn't want to see this exceeded, ideally you
should police this on IFD level, but I'm not sure if QFX5k can, MX
can.

> TTL

TTL exceeded message. Normal to hit this policer in uloops.

> Redirect

IP redirect, you probably want to disable them at network edge. This
should be 0.

> L3MTU-fail

Egress MTU was too small for packet. It is punted for potentially ICMP
message generation. Depending on config expected or unexpected.

> RESOLVE

Traffic hitting connected DADDR which is not in ARP cache, we need to
punt it for ARP resolution. Normal to see as there is constant
background traffic to every DADDR.

> L3NHOP

Unsure.

> So is that all ARP?  ARP that the switch needs to answer for?  Similar for 
> the other packet types: are these thresholds for packets that the switch is 
> processing (sent to the RE), or just for any traffic that is seen on any 
> interface?  If it's just an issue of too much stuff going to the RE I can 
> firewall it off so long as I know it's spurious.

It's ARP packets verbatim (see RESOLVE which is non ARP packet
triggering ARP resolution). Originally when ddos-protection was
implemented resolve was not implemented, and RESOLVE packet was what
ever classifier it hit, so if you sent BGP packets to unarpable DADDR
it was eating BGP policer PPS, so you could easily get targets core
iBGP down, and there was nothing they could do to stop you.

> Sorry if I'm not asking the right questions... I'm just trying to figure out 
> if these errors are actually problems that I need to track down, or if the 
> default reporting is just too noisy.
>
> Thanks,
>
> Jason
> ___
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp



-- 
  ++ytti
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp