Re: wireguard reconfiguration reliability
On Thu, Mar 21, 2024 at 12:23:06PM +0300, Vitaliy Makkoveev wrote: > wg(4) diff was committed to -current. Does the problem exist in upcoming > 7.5? Oh, I didn't know a fix had been committed, the referenced thread didn't mention a final one. Thanks, I'll take a look.
Re: wireguard reconfiguration reliability
On 24/03/21 04:58PM, Kirill Miazine wrote: > in this case I added escaped newlines when posting, i have all on one line, > but I think i could have scripts where a pipe is followed by escaped > newline. have to check that one. BTW, this also applies to lists[1], for example: ! false && echo "Not false" || echo "False" Sorry for the offtopic. :-) [1]: https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/V3_chap02.html#tag_18_09_03
Re: wireguard reconfiguration reliability
• Страхиња Радић [2024-03-21 16:31]:
On 24/03/20 08:15AM, Kirill Miazine wrote:
#!/bin/sh
ifconfig wg1 | \
grep wgaip | \
awk '{print $2} ' | \
grep /32$ | \
sed 's/\/32//' | \
sort | while read x; do
ping -w 1 -c 1 $x 2>&1
done
Just FYI, you don't need backslashes (\) here, as the command ending with a
pipe is an incomplete pipeline. ;-)
ah, thanks a lot, good to know!
in this case I added escaped newlines when posting, i have all on one
line, but I think i could have scripts where a pipe is followed by
escaped newline. have to check that one.
i see some scripts in /etc also use escaped newlines after pipes:
root@stable ~ # grep -r '| \\$' /etc/
/etc/daily: baksize=`disklabel $bakdisk 2>/dev/null | \
/etc/daily: rootsize=`disklabel $rootdisk 2>/dev/null | \
/etc/weekly:echo "${UPDATEDB} --fcodes=-" | \
Re: wireguard reconfiguration reliability
On 24/03/20 08:15AM, Kirill Miazine wrote:
> #!/bin/sh
> ifconfig wg1 | \
> grep wgaip | \
> awk '{print $2} ' | \
> grep /32$ | \
> sed 's/\/32//' | \
> sort | while read x; do
> ping -w 1 -c 1 $x 2>&1
> done
Just FYI, you don't need backslashes (\) here, as the command ending with a
pipe is an incomplete pipeline. ;-)
Re: wireguard reconfiguration reliability
On Wed, Mar 20, 2024 at 03:43:52PM -0700, Paul B. Henson wrote: > On Wed, Mar 20, 2024 at 09:56:06PM +0100, Kirill Miazine wrote: > > > Like in this thread, I guess: > > > > https://marc.info/?t=16964239631=1=2 > > Yes, that is likely the issue we're hitting. Seems last message is from > 10/2023 and the issue wasn't resolved :(, so I guess it's a known > problem with no solution on the horizon. > wg(4) diff was committed to -current. Does the problem exist in upcoming 7.5? > Next time I'll try your workaround of batching the commands up (ifconfig > wg1 down; ifconfig wg1 delete; ifconfig wg1 destroy) rather than running > one at a time and keep my fingers crossed I win the race condition :). > > Thanks for the help... >
Re: wireguard reconfiguration reliability
• Paul B. Henson [2024-03-20 15:43]: > On Wed, Mar 20, 2024 at 09:56:06PM +0100, Kirill Miazine wrote: > > > Like in this thread, I guess: > > > > https://marc.info/?t=16964239631=1=2 > > Yes, that is likely the issue we're hitting. Seems last message is from > 10/2023 and the issue wasn't resolved :(, so I guess it's a known > problem with no solution on the horizon. > > Next time I'll try your workaround of batching the commands up (ifconfig > wg1 down; ifconfig wg1 delete; ifconfig wg1 destroy) rather than running > one at a time and keep my fingers crossed I win the race condition :). I could as well share how I reconfigure my wgpeers for wg1: I just remove them all first and then re-add whatever is in /etc/hostname.wg1, keeping the wg interface itself alone: ifconfig wg1 -wgpeerall grep ^wgpeer /etc/hostname.wg1|while read x;do ifconfig wg1 $x;done > Thanks for the help... > -- -- Kirill Miazine
Re: wireguard reconfiguration reliability
On 3/20/2024 20:56 Kirill Miazine wrote: Like in this thread, I guess: https://marc.info/?t=16964239631=1=2 Indeed. Thanks for the link.
Re: wireguard reconfiguration reliability
On Wed, Mar 20, 2024 at 09:56:06PM +0100, Kirill Miazine wrote: > Like in this thread, I guess: > > https://marc.info/?t=16964239631=1=2 Yes, that is likely the issue we're hitting. Seems last message is from 10/2023 and the issue wasn't resolved :(, so I guess it's a known problem with no solution on the horizon. Next time I'll try your workaround of batching the commands up (ifconfig wg1 down; ifconfig wg1 delete; ifconfig wg1 destroy) rather than running one at a time and keep my fingers crossed I win the race condition :). Thanks for the help...
Re: wireguard reconfiguration reliability
• Paul B. Henson [2024-03-20 21:14]: On 3/20/2024 9:21 AM, Zack Newman wrote: clients in rdomain(4) 0. Last week I ran ifconfig wg1 destroy, replaced the wgkey and wgpsk for one of the three wgpeers in the second interface, and ran sh /etc/netstart wg1. Once I did this, the server seemingly froze: That's similar to what we see, although generally the entire server doesn't die, just the ifconfig command wedges and can't be killed, and the box can't be rebooted cleanly. Like in this thread, I guess: https://marc.info/?t=16964239631=1=2 Thanks for the feedback…
Re: wireguard reconfiguration reliability
On 3/20/2024 9:21 AM, Zack Newman wrote: clients in rdomain(4) 0. Last week I ran ifconfig wg1 destroy, replaced the wgkey and wgpsk for one of the three wgpeers in the second interface, and ran sh /etc/netstart wg1. Once I did this, the server seemingly froze: That's similar to what we see, although generally the entire server doesn't die, just the ifconfig command wedges and can't be killed, and the box can't be rebooted cleanly. Thanks for the feedback…
Re: wireguard reconfiguration reliability
• Paul B. Henson [2024-03-20 20:38]: On 3/20/2024 1:44 AM, Kirill Miazine wrote: actually I checked, and I do use wgpka on clients, but not on the server -- I don't remember why I didn't... In our case the server is on an Internet accessible address, whereas the clients are behind a NAT firewall. We also have keepalives enabled on the clients (to maintain their NAT mapping) but not on the server (as if the client isn't sending its keepalives the server isn't going to get through anyway). this decribes my setup more or less, but some "clients" have stable, routable, reachable addresses. A scenario where it stops but then works again as soon as traffic is sent does kind of sound like a firewall or NAT timeout issue? We don't have that problem, if we leave it completely alone it generally works indefinitely with no issues. It's just when we try to modify the configuration that things sometimes go sideways. what makes flow stop is e.g. if server is rebooted, then clients wouldn't re-connect. it could also be that flushing wgpeers and then re-adding them also made clients go away. again, I haven't spent much time debugging and can't guarantee that described behaviour is what really is going on: I noticed the issue and that ping would seemingly resolve it, so I just added pings everywhere. Thanks for the data point…
Re: wireguard reconfiguration reliability
On 3/20/2024 1:44 AM, Kirill Miazine wrote: actually I checked, and I do use wgpka on clients, but not on the server -- I don't remember why I didn't... In our case the server is on an Internet accessible address, whereas the clients are behind a NAT firewall. We also have keepalives enabled on the clients (to maintain their NAT mapping) but not on the server (as if the client isn't sending its keepalives the server isn't going to get through anyway). A scenario where it stops but then works again as soon as traffic is sent does kind of sound like a firewall or NAT timeout issue? We don't have that problem, if we leave it completely alone it generally works indefinitely with no issues. It's just when we try to modify the configuration that things sometimes go sideways. Thanks for the data point…
Re: wireguard reconfiguration reliability
I have two wg(4) interfaces: one that is a site-to-site tunnel (i.e., exactly one wgpeer where both sides have wgendpoint configured) in rdomain(4) 1, and another that is used as the "server" for roaming VPN clients in rdomain(4) 0. Last week I ran ifconfig wg1 destroy, replaced the wgkey and wgpsk for one of the three wgpeers in the second interface, and ran sh /etc/netstart wg1. Once I did this, the server seemingly froze: it stopped responding to ICMPv6, ICMP, and UDP traffic sent to a vlan(4) interface in rdomain(4) 0 whose parent is an ixl(4) interface. After about a minute, the server started to respond to traffic; nonetheless I rebooted the server as a precautionary measure. I almost never change the wg(4) config, so I cannot say if this is a new issue or an issue at all-perhaps it was just a coincidence-and if an issue, not sure if this is related to what is happening to you. I thought I'd provide a possible data point though. The server is running OpenBSD 7.4-stable.
Re: wireguard reconfiguration reliability
• Lorenz (xha) [2024-03-20 09:29]:
[...]
> > I've seen some issues too, but has not identified a reproducible pattern.
> > What I've seen, however, is that WG packets start flowing when the other end
> > of the connection pings back, so in my setup with a central VPN server I
> > make it ping all the peers' WG IP adress periodically:
> >
> > #!/bin/sh
> > ifconfig wg1 | \
> > grep wgaip | \
> > awk '{print $2} ' | \
> > grep /32$ | \
> > sed 's/\/32//' | \
> > sort | while read x; do
> > ping -w 1 -c 1 $x 2>&1
> > done
> >
> > and then each peer also pings the server's WG IP periodically.
>
> i think that this is a different issue than the one paul has. are
> you aware that the "wgpka" option exists? (documented in ifconfig(8)).
> that might solve your problem.
could be a different issue FWIW.
yes, I am aware of and use wgpka, and yet the workaround still was
necessary.
Re: wireguard reconfiguration reliability
On Wed, Mar 20, 2024 at 08:15:55AM +0100, Kirill Miazine wrote:
> Hi there
>
> • Paul B. Henson [2024-03-20 05:40]:
> > We're using wireguard to set up VPN connections from various systems
> > deployed on-prem at customer sites to central openbsd boxes to route
> > internal traffic between the remote boxes and the internal network.
> >
> > After a fresh reboot with a given configuration, everything works great.
> > The problem we have is when we later add or remove a remote system and
> > try to reconfigure the wireguard interface on the central servers.
> >
> > Sometimes the new system just won't work, or oddly the new system works
> > fine but an existing system that was working breaks 8-/. When that
> > happens, we generally have to reboot it, at which point everything
> > works.
>
> I've seen some issues too, but has not identified a reproducible pattern.
> What I've seen, however, is that WG packets start flowing when the other end
> of the connection pings back, so in my setup with a central VPN server I
> make it ping all the peers' WG IP adress periodically:
>
> #!/bin/sh
> ifconfig wg1 | \
> grep wgaip | \
> awk '{print $2} ' | \
> grep /32$ | \
> sed 's/\/32//' | \
> sort | while read x; do
> ping -w 1 -c 1 $x 2>&1
> done
>
> and then each peer also pings the server's WG IP periodically.
i think that this is a different issue than the one paul has. are
you aware that the "wgpka" option exists? (documented in ifconfig(8)).
that might solve your problem.
Re: wireguard reconfiguration reliability
Hi there
• Paul B. Henson [2024-03-20 05:40]:
We're using wireguard to set up VPN connections from various systems
deployed on-prem at customer sites to central openbsd boxes to route
internal traffic between the remote boxes and the internal network.
After a fresh reboot with a given configuration, everything works great.
The problem we have is when we later add or remove a remote system and
try to reconfigure the wireguard interface on the central servers.
Sometimes the new system just won't work, or oddly the new system works
fine but an existing system that was working breaks 8-/. When that
happens, we generally have to reboot it, at which point everything
works.
I've seen some issues too, but has not identified a reproducible
pattern. What I've seen, however, is that WG packets start flowing when
the other end of the connection pings back, so in my setup with a
central VPN server I make it ping all the peers' WG IP adress periodically:
#!/bin/sh
ifconfig wg1 | \
grep wgaip | \
awk '{print $2} ' | \
grep /32$ | \
sed 's/\/32//' | \
sort | while read x; do
ping -w 1 -c 1 $x 2>&1
done
and then each peer also pings the server's WG IP periodically.
Occasionally ifconfig on the wg interface just wedges completely. When
that happens, it won't reboot cleaning, we have to hard reset it.
I've seen lockups upon destroying wg interface, but not during normal
operations (i.e. leaving wg alone).
Has anyone else seen this type of behavior? I'm not sure how common it
is to have regular ongoing changes to wireguard like we are doing, so it
might not pop up often.
Thanks much...
wireguard reconfiguration reliability
We're using wireguard to set up VPN connections from various systems deployed on-prem at customer sites to central openbsd boxes to route internal traffic between the remote boxes and the internal network. After a fresh reboot with a given configuration, everything works great. The problem we have is when we later add or remove a remote system and try to reconfigure the wireguard interface on the central servers. Sometimes the new system just won't work, or oddly the new system works fine but an existing system that was working breaks 8-/. When that happens, we generally have to reboot it, at which point everything works. Occasionally ifconfig on the wg interface just wedges completely. When that happens, it won't reboot cleaning, we have to hard reset it. Has anyone else seen this type of behavior? I'm not sure how common it is to have regular ongoing changes to wireguard like we are doing, so it might not pop up often. Thanks much...
