Re: bgpd between two 6.4 boxes. IPv6 flapping, IPv4 rock solid

[Sebastian Benoit]

Stuart Henderson(s...@spacehopper.org) on 2019.04.03 16:22:26 -:
> On 2019-04-02, Rachel Roch  wrote:
> >
> >
> >
> > Mar 30, 2019, 11:10 AM by s...@spacehopper.org:
> >
> >> On 2019-03-29, Rachel Roch <> rr...@tutanota.de 
> >> > > wrote:
> >>
> >>> Hi,
> >>>
> >>> Has anyone encountered this before ?
> >>>
> >>> Neighbor?? AS?? MsgRcvd?? MsgSent?? OutQ 
> >>> Up/Down?? State/PrfRcvd
> >>> EXT-V6-R2 65515 50 
> >>> 40 0 00:02:55 Active
> >>> EXT-V4-R2 65515 38 
> >>> 37 0 00:27:42?? 1
> >>> After approx just over 2 minutes, the V6 flaps, bu the V4 remains rock 
> >>> solid.
> >>>
> >>> The boxes are sitting right next to each other, connected over an OpenBSD 
> >>> LACP trunk.
> >>>
> >>> I have made the pf rules as simple as possible:
> >>>
> >>> table  counters {self}
> >>> table  counters {192.0.2.1,2001:DB8::1}
> >>> pass in quick proto {tcp,udp,icmp} from  to 
> >>>  modulate state
> >>> pass out quick proto {tcp,udp,icmp} from  to 
> >>>  modulate state
> >>>
> >>
> >> A few tips:
> >>
> >> Start with an explicit "block any" rule so you don't have any traffic
> >> caught by the implicit "pass flags any no state" default. (If you want
> >> some "stateless" traffic as may often be the case on a BGP router, make
> >> it explicit in the ruleset). Otherwise you risk state being created 
> >> on something other than a SYN, so PF doesn't know the TCP window scaling
> >> value (which is *only* sent on SYN packets), which can result in the
> >> connection being killed after some traffic passes (state tracking gets
> >> out of sync).
> >>
> >> You don't have a rule for icmp6. IPv6's equivalent to ARP runs over icmp6
> >> and you do need a rule for that. It will currently be passed by the 
> >> implicit
> >> default rule but that will stop when you add "block any"..
> >>
> >> "modulate state" really isn't as simple as possible ;)
> >>
> >
> > A belated thanks for this !
> >
> > Re: icmp6:
> > pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol
> > pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv
> > pass quick inet6 proto ipv6-icmp all icmp6-type echoreq
> > pass quick inet6 proto ipv6-icmp all icmp6-type echorep
> >
> > Re: "modulate state" I thought that was meant to be a good option these 
> > days instead of one of the more traditional state techniques ?
> >
> >
> 
> "modulate state" can be useful for protecting machines with poor
> sequence number generation, but OpenBSD already has good randomness
> for this. At best (assuming no bugs in 'modulate state'), this
> adds complexity and burns cpu cycles for no benefit,

If it were better, it would be the default. It is useful, but can also cause
problems. Dont use it if you are not sure you need it _and_ know you will
get reports if there are problems.

My advise:

Regarding your config: As sthen said, start your policy with block any.

Then add rules to pass traffic for the networks you route. With "no state",
because if you have asymetric routing, stateful does not work.

Then add rules to pass traffic of the router itself, i.e. from/to ip
addresses on the router. Those can be stateful.

As for inet6, start with "pass inet6 proto icmp6 no state".
Then when everything is working, restrict it to what is actually needed.

Re: bgpd between two 6.4 boxes. IPv6 flapping, IPv4 rock solid

[Stuart Henderson]

On 2019-04-02, Rachel Roch  wrote:
>
>
>
> Mar 30, 2019, 11:10 AM by s...@spacehopper.org:
>
>> On 2019-03-29, Rachel Roch <> rr...@tutanota.de > 
>> > wrote:
>>
>>> Hi,
>>>
>>> Has anyone encountered this before ?
>>>
>>> Neighbor    AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
>>> EXT-V6-R2   65515 50 40 0 00:02:55 Active
>>> EXT-V4-R2   65515 38 37 0 00:27:42  1
>>> After approx just over 2 minutes, the V6 flaps, bu the V4 remains rock 
>>> solid.
>>>
>>> The boxes are sitting right next to each other, connected over an OpenBSD 
>>> LACP trunk.
>>>
>>> I have made the pf rules as simple as possible:
>>>
>>> table  counters {self}
>>> table  counters {192.0.2.1,2001:DB8::1}
>>> pass in quick proto {tcp,udp,icmp} from  to 
>>>  modulate state
>>> pass out quick proto {tcp,udp,icmp} from  to 
>>>  modulate state
>>>
>>
>> A few tips:
>>
>> Start with an explicit "block any" rule so you don't have any traffic
>> caught by the implicit "pass flags any no state" default. (If you want
>> some "stateless" traffic as may often be the case on a BGP router, make
>> it explicit in the ruleset). Otherwise you risk state being created 
>> on something other than a SYN, so PF doesn't know the TCP window scaling
>> value (which is *only* sent on SYN packets), which can result in the
>> connection being killed after some traffic passes (state tracking gets
>> out of sync).
>>
>> You don't have a rule for icmp6. IPv6's equivalent to ARP runs over icmp6
>> and you do need a rule for that. It will currently be passed by the implicit
>> default rule but that will stop when you add "block any"..
>>
>> "modulate state" really isn't as simple as possible ;)
>>
>
> A belated thanks for this !
>
> Re: icmp6:
> pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol
> pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv
> pass quick inet6 proto ipv6-icmp all icmp6-type echoreq
> pass quick inet6 proto ipv6-icmp all icmp6-type echorep
>
> Re: "modulate state" I thought that was meant to be a good option these days 
> instead of one of the more traditional state techniques ?
>
>

"modulate state" can be useful for protecting machines with poor
sequence number generation, but OpenBSD already has good randomness
for this. At best (assuming no bugs in 'modulate state'), this
adds complexity and burns cpu cycles for no benefit,

Re: bgpd between two 6.4 boxes. IPv6 flapping, IPv4 rock solid

[Rachel Roch]

Mar 30, 2019, 11:10 AM by s...@spacehopper.org:

> On 2019-03-29, Rachel Roch <> rr...@tutanota.de > > 
> wrote:
>
>> Hi,
>>
>> Has anyone encountered this before ?
>>
>> Neighbor    AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
>> EXT-V6-R2   65515 50 40 0 00:02:55 Active
>> EXT-V4-R2   65515 38 37 0 00:27:42  1
>> After approx just over 2 minutes, the V6 flaps, bu the V4 remains rock solid.
>>
>> The boxes are sitting right next to each other, connected over an OpenBSD 
>> LACP trunk.
>>
>> I have made the pf rules as simple as possible:
>>
>> table  counters {self}
>> table  counters {192.0.2.1,2001:DB8::1}
>> pass in quick proto {tcp,udp,icmp} from  to 
>>  modulate state
>> pass out quick proto {tcp,udp,icmp} from  to 
>>  modulate state
>>
>
> A few tips:
>
> Start with an explicit "block any" rule so you don't have any traffic
> caught by the implicit "pass flags any no state" default. (If you want
> some "stateless" traffic as may often be the case on a BGP router, make
> it explicit in the ruleset). Otherwise you risk state being created 
> on something other than a SYN, so PF doesn't know the TCP window scaling
> value (which is *only* sent on SYN packets), which can result in the
> connection being killed after some traffic passes (state tracking gets
> out of sync).
>
> You don't have a rule for icmp6. IPv6's equivalent to ARP runs over icmp6
> and you do need a rule for that. It will currently be passed by the implicit
> default rule but that will stop when you add "block any"..
>
> "modulate state" really isn't as simple as possible ;)
>

A belated thanks for this !

Re: icmp6:
pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol
pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv
pass quick inet6 proto ipv6-icmp all icmp6-type echoreq
pass quick inet6 proto ipv6-icmp all icmp6-type echorep

Re: "modulate state" I thought that was meant to be a good option these days 
instead of one of the more traditional state techniques ?

Re: bgpd between two 6.4 boxes. IPv6 flapping, IPv4 rock solid

[Stuart Henderson]

On 2019-03-29, Rachel Roch  wrote:
> Hi,
>
> Has anyone encountered this before ?
>
> Neighbor    AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
> EXT-V6-R2   65515 50 40 0 00:02:55 Active
> EXT-V4-R2   65515 38 37 0 00:27:42  1
> After approx just over 2 minutes, the V6 flaps, bu the V4 remains rock solid.
>
> The boxes are sitting right next to each other, connected over an OpenBSD 
> LACP trunk.
>
> I have made the pf rules as simple as possible:
>
> table  counters {self}
> table  counters {192.0.2.1,2001:DB8::1}
> pass in quick proto {tcp,udp,icmp} from  to 
>  modulate state
> pass out quick proto {tcp,udp,icmp} from  to 
>  modulate state

A few tips:

Start with an explicit "block any" rule so you don't have any traffic
caught by the implicit "pass flags any no state" default. (If you want
some "stateless" traffic as may often be the case on a BGP router, make
it explicit in the ruleset). Otherwise you risk state being created 
on something other than a SYN, so PF doesn't know the TCP window scaling
value (which is *only* sent on SYN packets), which can result in the
connection being killed after some traffic passes (state tracking gets
out of sync).

You don't have a rule for icmp6. IPv6's equivalent to ARP runs over icmp6
and you do need a rule for that. It will currently be passed by the implicit
default rule but that will stop when you add "block any"..

"modulate state" really isn't as simple as possible ;)

Re: bgpd between two 6.4 boxes. IPv6 flapping, IPv4 rock solid

[Rachel Roch]

29 Mar 2019, 18:57 by rr...@tutanota.de:

> Hi,
>
> Has anyone encountered this before ?
>
> Neighbor    AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
> EXT-V6-R2   65515 50 40 0 00:02:55 Active
> EXT-V4-R2   65515 38 37 0 00:27:42  1
> After approx just over 2 minutes, the V6 flaps, bu the V4 remains rock solid.
>
> The boxes are sitting right next to each other, connected over an OpenBSD 
> LACP trunk.
>
> I have made the pf rules as simple as possible:
>
> table  counters {self}
> table  counters {192.0.2.1,2001:DB8::1}
> pass in quick proto {tcp,udp,icmp} from  to 
>  modulate state
> pass out quick proto {tcp,udp,icmp} from  to 
>  modulate state
>
> The bgpd config is also simple:
>
> MY_ROUTER_ID_V4="192.0.2.2"
> MY_ROUTER_ID_V6="2001:DB8::2"
> MY_ASN="64515"
> AS $MY_ASN
> router-id $MY_ROUTER_ID_V4
> rde med compare always
> network inet connected
> network inet6 connected
> group its_internal_v4 {
>     remote-as 65515
>     announce IPv6 none
>     neighbor 192.0.2.1 {
>     local-address 192.0.2.2
>     descr "EXT-V4-R2"
>     }
> }
> group its_internal_v6 {
>     remote-as 65515
>     announce IPv6 none
>     neighbor 2001:DB8::1 {
>     local-address 2001:DB8::2
>     descr "EXT-V6-R2"
>     }
> }
>

Forgive me, I forgot to include what shows in the logs:

sending IPv4 unicast EOR marker
received IPv4 unicast EOR marker
received notification: HoldTimer expired
state change Established -> Idle, reason: NOTIFICATION received
state change Idle -> Connect, reason: Start
state change Connect -> OpenSent, reason: Connection opened
state change OpenSent -> OpenConfirm, reason: OPEN message received
state change OpenConfirm -> Established, reason: KEEPALIVE message received
sending IPv4 unicast EOR marker
received IPv4 unicast EOR marker

bgpd between two 6.4 boxes. IPv6 flapping, IPv4 rock solid

[Rachel Roch]

Hi,

Has anyone encountered this before ?

Neighbor    AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
EXT-V6-R2   65515 50 40 0 00:02:55 Active
EXT-V4-R2   65515 38 37 0 00:27:42  1
After approx just over 2 minutes, the V6 flaps, bu the V4 remains rock solid.

The boxes are sitting right next to each other, connected over an OpenBSD LACP 
trunk.

I have made the pf rules as simple as possible:

table  counters {self}
table  counters {192.0.2.1,2001:DB8::1}
pass in quick proto {tcp,udp,icmp} from  to  
modulate state
pass out quick proto {tcp,udp,icmp} from  to 
 modulate state

The bgpd config is also simple:

MY_ROUTER_ID_V4="192.0.2.2"
MY_ROUTER_ID_V6="2001:DB8::2"
MY_ASN="64515"
AS $MY_ASN
router-id $MY_ROUTER_ID_V4
rde med compare always
network inet connected
network inet6 connected
group its_internal_v4 {
    remote-as 65515
    announce IPv6 none
    neighbor 192.0.2.1 {
    local-address 192.0.2.2
    descr "EXT-V4-R2"
    }
}
group its_internal_v6 {
    remote-as 65515
    announce IPv6 none
    neighbor 2001:DB8::1 {
    local-address 2001:DB8::2
    descr "EXT-V6-R2"
    }
}