Re: rdomain/rtable 255 BGPd routes -> leaking to rdomain/rtable 0
just to make sure: changing the sshd configuration fixes your problem as well? i.e. there is no bug? Putting ListenAddress $IP rdomain 255 inside sshd_config did the trick, netstat -nr -f inet was looking at it's default route table which was 255 rtable in this case. Running sshd_rtable=255 I assume was doing what it's supposed to do, run the service within it's configured rtable only, so all running commands using network connections were confined to that rtable. I think there is no bug then, I'm sorry for the noise and thank you for your time and help !
Re: rdomain/rtable 255 BGPd routes -> leaking to rdomain/rtable 0
On Thu, Apr 12, 2018 at 07:14:57PM +0200, Sebastian Benoit wrote: > Nicolas Pence(nico...@pence.com.uy) on 2018.04.12 13:03:39 -0300: > > Thinking about it a little more, I've configured sshd_rtable=255 on > > /etc/rc.conf.local, > > on the non-working boxes, I'm re-checking this doing: > > > > rcctl set sshd rtable 0 > > > > and changing sshd_config: > > > > ListenAddress $IP rdomain 255 > > just to make sure: changing the sshd configuration fixes your problem as > well? i.e. there is no bug? > > For you convinience, this lets you display the rdomain on your shell prompt: > > rdomain=`ps -o rtable -p $$ | tail -n+2` or: rdomain=`id -R` > export PS1="[\u@$\h:\w]($rdomain)\$ " > > /Benno
Re: rdomain/rtable 255 BGPd routes -> leaking to rdomain/rtable 0
Nicolas Pence(nico...@pence.com.uy) on 2018.04.12 13:03:39 -0300: > Thinking about it a little more, I've configured sshd_rtable=255 on > /etc/rc.conf.local, > on the non-working boxes, I'm re-checking this doing: > > rcctl set sshd rtable 0 > > and changing sshd_config: > > ListenAddress $IP rdomain 255 just to make sure: changing the sshd configuration fixes your problem as well? i.e. there is no bug? For you convinience, this lets you display the rdomain on your shell prompt: rdomain=`ps -o rtable -p $$ | tail -n+2` export PS1="[\u@$\h:\w]($rdomain)\$ " /Benno
Re: rdomain/rtable 255 BGPd routes -> leaking to rdomain/rtable 0
Thinking about it a little more, I've configured sshd_rtable=255 on
/etc/rc.conf.local,
on the non-working boxes, I'm re-checking this doing:
rcctl set sshd rtable 0
and changing sshd_config:
ListenAddress $IP rdomain 255
Thank you
On 4/12/18 12:04, Nicolas Pence wrote:
I'm using 3 different rdomains, with one BGPd instance in each of them
with different configurations, when using rdomain 255 some routing
info is leaked into rtable 0 to the point that doesn't allow to route,
this happens over a few minutes/hours of functioning (tested on two
VM's with 6.2-stable and 6.3-release both amd64), network driver is
vmx(4) on all interfaces of both systems.
As I understand "netstat -nr -f inet" should have the same output as
"netstat -T0 -nr -f inet", this happens even after reboots.
* There is no BGPd running on rdomain 0, just on defined rdomains.
* Placing rtable $rdomain-number inside bgpd$RDOMAIN.conf doesn't
change the situation.
* Changing rdomain from 255 to 254 on interfaces vmx3 and carp25[45]
seems to solve the issue,
* No issue with routes belonging to different bgpd processess running
on other rdomains (179 & 253) were found.
*UPDATE* This error is only seen when logged in using SSH (user root,
key auth), if I test the same using the direct-attached console
(VMWare VMRC) the routes are shown correctly and there is no loss of
connection
(I know it sound nuts).
Tests on the non working
# route -n get 8.8.8.8
route: writing to routing socket: No such process
# netstat -T0 -nr -f inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 17.2.18.33 UGS 4 13 - 8 vmx3
224/4 127.0.0.1 URS 0 0 32768 8 lo0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHhl 1 2 32768 1 lo0
17.2.18.32/27 179.27.168.41 UCn 1 493 - 4 vmx3
17.2.18.33 0a:aa:dc:ff:10:02 UHLch 1 259 - 3 vmx3
17.2.18.41 0b:bb:57:a7:2a:e0 UHLl 0 18 - 1 vmx3
17.2.18.63 17.27.18.41 UHb 0 0 - 1 vmx3
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
10.25/16 10.24.2.1 UG 0 0 - 48 vmx0
10.25.6.20 ab:0c:5e:00:01:ff UHLl 0 0 - 1 carp255
10.25.6.20/32 10.25.6.20 UCn 0 0 - 19 carp255
10.25.6.21 ab:0c:5e:00:01:fe UHLl 0 36 - 1 carp254
10.25.6.21/32 10.25.6.21 Cn 0 0 - 19 carp254
[...]
# alias | grep bgp
bgpctl179='bgpctl -s /var/run/bgpd.sock.179'
bgpctl253='bgpctl -s /var/run/bgpd.sock.253'
bgpctl255='bgpctl -s /var/run/bgpd.sock.255'
# ls -lh /etc/rc.d/bgpd*
-r-xr-xr-x 1 root wheel 228B Mar 24 17:12 /etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd179 ->
/etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd253 ->
/etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd255 ->
/etc/rc.d/bgpd
# cat /etc/rc.conf.local
bgpd179_flags=-f /etc/bgpd179.conf -v
bgpd179_rtable=179
bgpd253_flags=-f /etc/bgpd253.conf -v
bgpd253_rtable=253
bgpd255_flags=-f /etc/bgpd255.conf -v
bgpd255_rtable=255
pkg_scripts=bgpd253 bgpd179 bgpd255
# bgpctl255 show ip bgp | head
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete
flags destination gateway lpref med aspath origin
*> 10.25.0.0/16 10.24.2.1 100 0 65510 i
*> 10.25.0.0/20 10.24.2.1 100 0 65510 65500 i
*> 10.25.8.0/24 10.24.2.1 100 0 65510 65500 i
*> 10.25.16.0/22 10.24.2.1 100 0 65510 65500 i
[...]
# ps aux -o rtable | grep -E '(_bgpd|USER)'
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND RTABLE
_bgpd 16269 0.0 0.5 944 1956 ?? Ip 12:53PM 0:00.00 bgpd:route deci
253
_bgpd 34173 0.0 0.5 940 1760 ?? Sp 12:53PM 0:00.34 bgpd:session en
253
_bgpd 48580 0.0 0.5 928 1940 ?? Ip 12:53PM 0:00.00 bgpd:route deci
179
_bgpd 49612 0.0 0.5 936 1768 ?? Sp 12:53PM 0:00.34 bgpd:session en
179
_bgpd 69090 0.0 0.6 1088 2248 ?? Ip 12:53PM 0:00.01 bgpd:route
deci 255
_bgpd 96380 0.0 0.5 1008 1876 ?? Sp 12:53PM 0:00.33 bgpd:session
en 255
# cat /etc/hostname.vmx0
rdomain 255
inet 10.24.2.3 255.255.255.224
description "Server -> Router"
up
# cat /etc/hostname.carp255
rdomain 255
vhid 255 carpdev vmx0 carppeer 10.24.2.4 advskew 0 pass mypass state
master
inet 10.25.6.20/32
up
# cat /etc/hostname.carp254
rdomain 255
vhid 254 carpdev vmx0 carppeer 10.24.2.4 advskew 100 pass myotherpass
state backup
inet 10.25.6.21/32
up
# grep -v ^# /etc/bgpd255.conf
peer="10.24.2.1"
AS 65512
router-id 10.24.2.3
log updates
network 10.25.6.20/32
network 10.25.6.21/32
rtable 255
group "AS65510 Router" {
rdomain/rtable 255 BGPd routes -> leaking to rdomain/rtable 0
I'm using 3 different rdomains, with one BGPd instance in each of them
with different configurations, when using rdomain 255 some routing info
is leaked into rtable 0 to the point that doesn't allow to route, this
happens over a few minutes/hours of functioning (tested on two VM's with
6.2-stable and 6.3-release both amd64), network driver is vmx(4) on all
interfaces of both systems.
As I understand "netstat -nr -f inet" should have the same output as
"netstat -T0 -nr -f inet", this happens even after reboots.
* There is no BGPd running on rdomain 0, just on defined rdomains.
* Placing rtable $rdomain-number inside bgpd$RDOMAIN.conf doesn't change
the situation.
* Changing rdomain from 255 to 254 on interfaces vmx3 and carp25[45]
seems to solve the issue,
* No issue with routes belonging to different bgpd processess running on
other rdomains (179 & 253) were found.
*UPDATE* This error is only seen when logged in using SSH (user root,
key auth), if I test the same using the direct-attached console (VMWare
VMRC) the routes are shown correctly and there is no loss of connection
(I know it sound nuts).
Tests on the non working
# route -n get 8.8.8.8
route: writing to routing socket: No such process
# netstat -T0 -nr -f inet
Routing tables
Internet:
Destination GatewayFlags Refs Use Mtu Prio Iface
default 17.2.18.33 UGS4 13 - 8 vmx3
224/4 127.0.0.1 URS00 32768 8 lo0
127/8 127.0.0.1 UGRS 00 32768 8 lo0
127.0.0.1 127.0.0.1 UHhl 12 32768 1 lo0
17.2.18.32/27 179.27.168.41 UCn1 493 - 4 vmx3
17.2.18.33 0a:aa:dc:ff:10:02 UHLch 1 259 - 3 vmx3
17.2.18.41 0b:bb:57:a7:2a:e0 UHLl 0 18 - 1 vmx3
17.2.18.63 17.27.18.41 UHb00 - 1 vmx3
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
10.25/16 10.24.2.1 UG 00 - 48 vmx0
10.25.6.20 ab:0c:5e:00:01:ff UHLl00 - 1 carp255
10.25.6.20/3210.25.6.20 UCn 00 - 19 carp255
10.25.6.21 ab:0c:5e:00:01:fe UHLl0 36 - 1 carp254
10.25.6.21/3210.25.6.21 Cn00 - 19 carp254
[...]
# alias | grep bgp
bgpctl179='bgpctl -s /var/run/bgpd.sock.179'
bgpctl253='bgpctl -s /var/run/bgpd.sock.253'
bgpctl255='bgpctl -s /var/run/bgpd.sock.255'
# ls -lh /etc/rc.d/bgpd*
-r-xr-xr-x 1 root wheel 228B Mar 24 17:12 /etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd179 ->
/etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd253 ->
/etc/rc.d/bgpd
lrwxr-xr-x 1 root wheel 14B Apr 9 11:01 /etc/rc.d/bgpd255 ->
/etc/rc.d/bgpd
# cat /etc/rc.conf.local
bgpd179_flags=-f /etc/bgpd179.conf -v
bgpd179_rtable=179
bgpd253_flags=-f /etc/bgpd253.conf -v
bgpd253_rtable=253
bgpd255_flags=-f /etc/bgpd255.conf -v
bgpd255_rtable=255
pkg_scripts=bgpd253 bgpd179 bgpd255
# bgpctl255 show ip bgp | head
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete
flags destination gateway lpref med aspath origin
*>10.25.0.0/1610.24.2.1 100 0 65510 i
*>10.25.0.0/2010.24.2.1 100 0 65510 65500 i
*>10.25.8.0/2410.24.2.1 100 0 65510 65500 i
*>10.25.16.0/22 10.24.2.1 100 0 65510 65500 i
[...]
# ps aux -o rtable | grep -E '(_bgpd|USER)'
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND RTABLE
_bgpd 16269 0.0 0.5 944 1956 ?? Ip 12:53PM 0:00.00 bgpd:route deci253
_bgpd 34173 0.0 0.5 940 1760 ?? Sp 12:53PM 0:00.34 bgpd:session en253
_bgpd 48580 0.0 0.5 928 1940 ?? Ip 12:53PM 0:00.00 bgpd:route deci179
_bgpd 49612 0.0 0.5 936 1768 ?? Sp 12:53PM 0:00.34 bgpd:session en179
_bgpd 69090 0.0 0.6 1088 2248 ?? Ip 12:53PM 0:00.01 bgpd:route deci255
_bgpd 96380 0.0 0.5 1008 1876 ?? Sp 12:53PM 0:00.33 bgpd:session en255
# cat /etc/hostname.vmx0
rdomain 255
inet 10.24.2.3 255.255.255.224
description "Server -> Router"
up
# cat /etc/hostname.carp255
rdomain 255
vhid 255 carpdev vmx0 carppeer 10.24.2.4 advskew 0 pass mypass state master
inet 10.25.6.20/32
up
# cat /etc/hostname.carp254
rdomain 255
vhid 254 carpdev vmx0 carppeer 10.24.2.4 advskew 100 pass myotherpass
state backup
inet 10.25.6.21/32
up
# grep -v ^# /etc/bgpd255.conf
peer="10.24.2.1"
AS 65512
router-id 10.24.2.3
log updates
network 10.25.6.20/32
network 10.25.6.21/32
rtable 255
group "AS65510 Router" {
remote-as 65510
descr "Server -> Router"
neighbor $peer {
announce IPv4 unicast
}
}
[...]
dmesg:
OpenBSD 6.3 (GENERIC) #100: Sat Mar 24 14:17:45 MDT 2018
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 385810432 (367MB)
avail mem =
