Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On 03.05.22 19:08, Gleb Smirnoff wrote: On Sun, Apr 24, 2022 at 09:49:48AM +0200, Florian Smeets wrote: F> On 23.04.22 01:38, Gleb Smirnoff wrote: F> >Hi Florian, F> > F> > here is a patch that should help with the IPv6 problem. I'm not F> > yet committing it, it might be not final. F> F> yes, the patch resolves the issue. There is just one SYN packet, and it F> gets a reply immediately. Alexander provided a patch against the ip6_output inconsistency: https://reviews.freebsd.org/D35117 You might be interested in testing it together with my patch. I will commit mine only after Alexander commits his. With both patches applied it's working fine and I cannot reproduce the initial issue. Thanks Florian OpenPGP_0xEF5BA4DCD5A9F3C0.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On Sun, Apr 24, 2022 at 09:49:48AM +0200, Florian Smeets wrote: F> On 23.04.22 01:38, Gleb Smirnoff wrote: F> >Hi Florian, F> > F> > here is a patch that should help with the IPv6 problem. I'm not F> > yet committing it, it might be not final. F> F> yes, the patch resolves the issue. There is just one SYN packet, and it F> gets a reply immediately. Alexander provided a patch against the ip6_output inconsistency: https://reviews.freebsd.org/D35117 You might be interested in testing it together with my patch. I will commit mine only after Alexander commits his. -- Gleb Smirnoff
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On 23.04.22 01:38, Gleb Smirnoff wrote: Hi Florian, here is a patch that should help with the IPv6 problem. I'm not yet committing it, it might be not final. Hi Gleb, yes, the patch resolves the issue. There is just one SYN packet, and it gets a reply immediately. Thanks, Florian OpenPGP_0xEF5BA4DCD5A9F3C0.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
> On 23. Apr 2022, at 02:24, Gleb Smirnoff wrote:
>
> Michael,
>
> On Sat, Apr 23, 2022 at 01:54:25AM +0200, Michael Tuexen wrote:
> M> > here is a patch that should help with the IPv6 problem. I'm not
> M> > yet committing it, it might be not final.
> M>
> M> when I was looking at the code, I was also wondering if it would make
> M> more sense to check for M_LOOP.
> M>
> M> However, isn't the rcvif wrong for the first two received packets? I
> M> would expect it always to be the loopback interface. Is that expectation
> M> wrong?
>
> The IPv6 has a special feature of calling (ifp->if_output)(origifp, ...
>
> I don't fully understand it, but Alexander does.
>
> What I can observe is that it works differently for the original packet,
> its first retransmit and second retransmit. Still unclear to me why.
I consider this also strange. The three packets are identical.
So I would expect, that all of these are handled the same way.
>
> Here is how to observe it:
>
> dtrace
>-n 'fbt::ip6_output:entry
>{ printf("ro %p ifp %p\n", args[2], args[2]->ro_nh ?
> args[2]->ro_nh->nh_ifp : 0); }'
>-n 'fbt::ip6_output_send:entry { printf("ifp %p origifp %p\n", args[1],
> args[2]); }'
>
> And you will see this:
>
> 1 45625 ip6_output:entry ro f800122c19a0 ifp 0
> 1 22539ip6_output_send:entry ifp f800027cb800 origifp
> f800020db000
>
> 0 45625 ip6_output:entry ro f800122c19a0 ifp
> f800027cb800
> 0 22539ip6_output_send:entry ifp f800027cb800 origifp
> f800020db000
>
> 0 45625 ip6_output:entry ro f800122c19a0 ifp
> f800027cb800
> 0 22539ip6_output_send:entry ifp f800027cb800 origifp
> f800027cb800
>
> So, on packet three (second retransmit) the origifp is equal to ifp (is lo0)
> and now
> packet passes validation. However, the more I read it, the more it seems to
> me that
> actually packet three is incorrect and first two are correct :)
>
> To cope with this self inflicted damage of (ifp->if_output)(origifp, IPV6
> introduced
> M_LOOP and uses it internally. Looks like a quick solution for IPv6 is to use
> it.
> However, I will commit it only once we got understanding why the hell a
> second retransmit
> is different.
>
> M> I also have an additional question:
> M> Why is this check protected by an (ia != NULL) condition? It does not make
> M> any use of ia?
>
> It is a host protection feature, so checks only packets that are destined to
> us.
> This allows to do basic antispoof checks for a host not equipped with any
> firewall.
Understood. I was confused, since all other code protected by (ia != NULL)
actually
depends on ia not being the NULL pointer.
Best regards
Michael
>
> For a machine acting as a router better behavior is not to drop anything
> routed
> through unless explicitly told so by a filtering policy.
>
> --
> Gleb Smirnoff
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
> On 23. Apr 2022, at 01:38, Gleb Smirnoff wrote: > > Hi Florian, > > here is a patch that should help with the IPv6 problem. I'm not > yet committing it, it might be not final. Hi Gleb, when I was looking at the code, I was also wondering if it would make more sense to check for M_LOOP. However, isn't the rcvif wrong for the first two received packets? I would expect it always to be the loopback interface. Is that expectation wrong? I also have an additional question: Why is this check protected by an (ia != NULL) condition? It does not make any use of ia? Best regards Michael > -- > Gleb Smirnoff >
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
Michael,
On Sat, Apr 23, 2022 at 01:54:25AM +0200, Michael Tuexen wrote:
M> > here is a patch that should help with the IPv6 problem. I'm not
M> > yet committing it, it might be not final.
M>
M> when I was looking at the code, I was also wondering if it would make
M> more sense to check for M_LOOP.
M>
M> However, isn't the rcvif wrong for the first two received packets? I
M> would expect it always to be the loopback interface. Is that expectation
M> wrong?
The IPv6 has a special feature of calling (ifp->if_output)(origifp, ...
I don't fully understand it, but Alexander does.
What I can observe is that it works differently for the original packet,
its first retransmit and second retransmit. Still unclear to me why.
Here is how to observe it:
dtrace
-n 'fbt::ip6_output:entry
{ printf("ro %p ifp %p\n", args[2], args[2]->ro_nh ? args[2]->ro_nh->nh_ifp
: 0); }'
-n 'fbt::ip6_output_send:entry { printf("ifp %p origifp %p\n", args[1],
args[2]); }'
And you will see this:
1 45625 ip6_output:entry ro f800122c19a0 ifp 0
1 22539ip6_output_send:entry ifp f800027cb800 origifp
f800020db000
0 45625 ip6_output:entry ro f800122c19a0 ifp
f800027cb800
0 22539ip6_output_send:entry ifp f800027cb800 origifp
f800020db000
0 45625 ip6_output:entry ro f800122c19a0 ifp
f800027cb800
0 22539ip6_output_send:entry ifp f800027cb800 origifp
f800027cb800
So, on packet three (second retransmit) the origifp is equal to ifp (is lo0)
and now
packet passes validation. However, the more I read it, the more it seems to me
that
actually packet three is incorrect and first two are correct :)
To cope with this self inflicted damage of (ifp->if_output)(origifp, IPV6
introduced
M_LOOP and uses it internally. Looks like a quick solution for IPv6 is to use
it.
However, I will commit it only once we got understanding why the hell a second
retransmit
is different.
M> I also have an additional question:
M> Why is this check protected by an (ia != NULL) condition? It does not make
M> any use of ia?
It is a host protection feature, so checks only packets that are destined to us.
This allows to do basic antispoof checks for a host not equipped with any
firewall.
For a machine acting as a router better behavior is not to drop anything routed
through unless explicitly told so by a filtering policy.
--
Gleb Smirnoff
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On Sat, Apr 16, 2022 at 09:19:57AM -0400, Michael Butler wrote: M> > Michael, can you please confirm or decline that you see the packets M> > that are dropped when you tcpdump on lo0? M> M> All the jails are aliased to share a single bridge interface. That M> results in the route to each jail being on lo0 so .. probably :-) This probably is somehow related to bridge. Can you please help me providing minimal configuration of bridge/jails where the problem shows up? -- Gleb Smirnoff
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
Hi Florian,
here is a patch that should help with the IPv6 problem. I'm not
yet committing it, it might be not final.
--
Gleb Smirnoff
diff --git a/sys/netinet6/ip6_input.c b/sys/netinet6/ip6_input.c
index 3a13d2a9dc7..625de6d3657 100644
--- a/sys/netinet6/ip6_input.c
+++ b/sys/netinet6/ip6_input.c
@@ -825,7 +825,7 @@ ip6_input(struct mbuf *m)
ip6_sprintf(ip6bufd, >ip6_dst)));
goto bad;
}
- if (V_ip6_sav && !(rcvif->if_flags & IFF_LOOPBACK) &&
+ if (V_ip6_sav && !(m->m_flags & M_LOOP) &&
__predict_false(in6_localip_fib(>ip6_src,
rcvif->if_fib))) {
IP6STAT_INC(ip6s_badscope); /* XXX */
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On 4/16/22 01:22, Gleb Smirnoff wrote: Hi Florian, Hi Michael, On Fri, Apr 15, 2022 at 06:11:13PM -0400, Michael Butler wrote: M> >> I can reproduce this locally, will try to figure out what is going on. M> >> If you can bisect it, it would be great. M> > M> > Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed M> > toggling net.inet6.ip6.source_address_validation makes the issue go away M> > on latest main. M> M> I found this commit and the ipv4 analog also cause packets between M> non-VNET jails on the same host and to the host itself to be dropped :-( I see your mails and will look into the problem ASAP. Meanwhile... Florian, can you please confirm you are using jails too? Michael, can you please confirm or decline that you see the packets that are dropped when you tcpdump on lo0? All the jails are aliased to share a single bridge interface. That results in the route to each jail being on lo0 so .. probably :-) Michael
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On 16.04.22 07:22, Gleb Smirnoff wrote: Hi Florian, Hi Michael, Hi Gleb, thanks for looking into it. On Fri, Apr 15, 2022 at 06:11:13PM -0400, Michael Butler wrote: M> > M> > Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed M> > toggling net.inet6.ip6.source_address_validation makes the issue go away M> > on latest main. M> M> I found this commit and the ipv4 analog also cause packets between M> non-VNET jails on the same host and to the host itself to be dropped :-( I see your mails and will look into the problem ASAP. Meanwhile... Florian, can you please confirm you are using jails too? No, two of the 3 hosts I tested on do not use jails. Florian OpenPGP_0xEF5BA4DCD5A9F3C0.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
Hi Florian, Hi Michael, On Fri, Apr 15, 2022 at 06:11:13PM -0400, Michael Butler wrote: M> >> I can reproduce this locally, will try to figure out what is going on. M> >> If you can bisect it, it would be great. M> > M> > Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed M> > toggling net.inet6.ip6.source_address_validation makes the issue go away M> > on latest main. M> M> I found this commit and the ipv4 analog also cause packets between M> non-VNET jails on the same host and to the host itself to be dropped :-( I see your mails and will look into the problem ASAP. Meanwhile... Florian, can you please confirm you are using jails too? Michael, can you please confirm or decline that you see the packets that are dropped when you tcpdump on lo0? -- Gleb Smirnoff
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
> On 16. Apr 2022, at 00:05, tue...@freebsd.org wrote: > >> On 15. Apr 2022, at 23:39, Florian Smeets wrote: >> >> On 15.04.22 21:24, tue...@freebsd.org wrote: On 15. Apr 2022, at 20:20, Florian Smeets wrote: Hi, there seems to be an issue with local IPv6 TCP connections on main. I have been seeing this for a couple of months at least. pkg upgr on my webserver hosting the pkg repo is very slow, all other hosts can connect to the pkg repo just fine. So IPv6 connections from external hosts are not affected. I thought I must have misconfigured something, as my setup is a bit weird. Yesterday I noticed the same issue on a different host, turns out all my 14.0 hosts seem to be affected, cognet@ could also reproduce it on one of his systems. The service/software used does not seem to matter, I tried with port 22, 25, 80 and 443. ICMP and UDP don't seem to be affected. ping6 gets replies immediately. And UDP connections with nc -l -u / nc -u don't have any delay, sent data is received immediately. Testing local TCP connections show this: flo@rp64:~ $ ifconfig dwc0|grep 2003 inet6 2003:cf:df49:c97:4c59:ebff:fec1:463d prefixlen 64 autoconf flo@rp64:~ $ nc -v 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 [3 second delay here] Connection to 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 port [tcp/ssh] succeeded! SSH-2.0-OpenSSH_8.9 FreeBSD-20220413 >> I need help debugging this, I don't know how to analyze this further. I will start bisecting this, but I thought maybe someone has an idea. >>> Hi Florian, >>> I can reproduce this locally, will try to figure out what is going on. If >>> you can bisect it, it would be great. >> >> Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed >> toggling net.inet6.ip6.source_address_validation makes the issue go away on >> latest main. > Cool, thanks for providing the information. However, I don't understand why > results in dropping the first two SYN segments (this should not happen) > and why the third is accepted (they should all treated the same). The first two packets don't have the loopback interface as its rcv interface, so Gleb's check for IFF_LOOPBACK fails. The second retransmission of the SYN has the loopback interface as its rcv interface, so Gleb check passes. I also do not see the problem with ICMP6, UDP or SCTP over IPv6. Don't know what is special about TCP, maybe Gleb knows out of his head... Best regards Michael > > Best regards > Michael >> >> Florian >> > >
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On 4/15/22 17:39, Florian Smeets wrote: On 15.04.22 21:24, tue...@freebsd.org wrote: On 15. Apr 2022, at 20:20, Florian Smeets wrote: Hi, there seems to be an issue with local IPv6 TCP connections on main. I have been seeing this for a couple of months at least. pkg upgr on my webserver hosting the pkg repo is very slow, all other hosts can connect to the pkg repo just fine. So IPv6 connections from external hosts are not affected. I thought I must have misconfigured something, as my setup is a bit weird. Yesterday I noticed the same issue on a different host, turns out all my 14.0 hosts seem to be affected, cognet@ could also reproduce it on one of his systems. The service/software used does not seem to matter, I tried with port 22, 25, 80 and 443. ICMP and UDP don't seem to be affected. ping6 gets replies immediately. And UDP connections with nc -l -u / nc -u don't have any delay, sent data is received immediately. Testing local TCP connections show this: flo@rp64:~ $ ifconfig dwc0|grep 2003 inet6 2003:cf:df49:c97:4c59:ebff:fec1:463d prefixlen 64 autoconf flo@rp64:~ $ nc -v 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 [3 second delay here] Connection to 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 port [tcp/ssh] succeeded! SSH-2.0-OpenSSH_8.9 FreeBSD-20220413 I need help debugging this, I don't know how to analyze this further. I will start bisecting this, but I thought maybe someone has an idea. Hi Florian, I can reproduce this locally, will try to figure out what is going on. If you can bisect it, it would be great. Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed toggling net.inet6.ip6.source_address_validation makes the issue go away on latest main. I found this commit and the ipv4 analog also cause packets between non-VNET jails on the same host and to the host itself to be dropped :-( Michael
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
> On 15. Apr 2022, at 23:39, Florian Smeets wrote: > > On 15.04.22 21:24, tue...@freebsd.org wrote: >>> On 15. Apr 2022, at 20:20, Florian Smeets wrote: >>> >>> >>> Hi, >>> >>> there seems to be an issue with local IPv6 TCP connections on main. I have >>> been seeing this for a couple of months at least. pkg upgr on my webserver >>> hosting the pkg repo is very slow, all other hosts can connect to the pkg >>> repo just fine. So IPv6 connections from external hosts are not affected. >>> >>> I thought I must have misconfigured something, as my setup is a bit weird. >>> Yesterday I noticed the same issue on a different host, turns out all my >>> 14.0 hosts seem to be affected, cognet@ could also reproduce it on one of >>> his systems. >>> >>> The service/software used does not seem to matter, I tried with port 22, >>> 25, 80 and 443. >>> >>> ICMP and UDP don't seem to be affected. ping6 gets replies immediately. And >>> UDP connections with nc -l -u / nc -u don't have any delay, sent data is >>> received immediately. >>> >>> Testing local TCP connections show this: >>> >>> flo@rp64:~ $ ifconfig dwc0|grep 2003 >>> inet6 2003:cf:df49:c97:4c59:ebff:fec1:463d prefixlen 64 autoconf >>> flo@rp64:~ $ nc -v 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 >>> [3 second delay here] >>> Connection to 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 port [tcp/ssh] >>> succeeded! >>> SSH-2.0-OpenSSH_8.9 FreeBSD-20220413 >>> > >>> >>> I need help debugging this, I don't know how to analyze this further. I >>> will start bisecting this, but I thought maybe someone has an idea. >> Hi Florian, >> I can reproduce this locally, will try to figure out what is going on. If >> you can bisect it, it would be great. > > Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed > toggling net.inet6.ip6.source_address_validation makes the issue go away on > latest main. Cool, thanks for providing the information. However, I don't understand why results in dropping the first two SYN segments (this should not happen) and why the third is accepted (they should all treated the same). Best regards Michael > > Florian >
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
On 15.04.22 21:24, tue...@freebsd.org wrote: On 15. Apr 2022, at 20:20, Florian Smeets wrote: Hi, there seems to be an issue with local IPv6 TCP connections on main. I have been seeing this for a couple of months at least. pkg upgr on my webserver hosting the pkg repo is very slow, all other hosts can connect to the pkg repo just fine. So IPv6 connections from external hosts are not affected. I thought I must have misconfigured something, as my setup is a bit weird. Yesterday I noticed the same issue on a different host, turns out all my 14.0 hosts seem to be affected, cognet@ could also reproduce it on one of his systems. The service/software used does not seem to matter, I tried with port 22, 25, 80 and 443. ICMP and UDP don't seem to be affected. ping6 gets replies immediately. And UDP connections with nc -l -u / nc -u don't have any delay, sent data is received immediately. Testing local TCP connections show this: flo@rp64:~ $ ifconfig dwc0|grep 2003 inet6 2003:cf:df49:c97:4c59:ebff:fec1:463d prefixlen 64 autoconf flo@rp64:~ $ nc -v 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 [3 second delay here] Connection to 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 port [tcp/ssh] succeeded! SSH-2.0-OpenSSH_8.9 FreeBSD-20220413 I need help debugging this, I don't know how to analyze this further. I will start bisecting this, but I thought maybe someone has an idea. Hi Florian, I can reproduce this locally, will try to figure out what is going on. If you can bisect it, it would be great. Found the culprit 1817be481b8703ae86730b151a6f49cc3022930f. And indeed toggling net.inet6.ip6.source_address_validation makes the issue go away on latest main. Florian OpenPGP_0xEF5BA4DCD5A9F3C0.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored
> On 15. Apr 2022, at 20:20, Florian Smeets wrote: > > [bcc to net@ for wider exposure] > > Hi, > > there seems to be an issue with local IPv6 TCP connections on main. I have > been seeing this for a couple of months at least. pkg upgr on my webserver > hosting the pkg repo is very slow, all other hosts can connect to the pkg > repo just fine. So IPv6 connections from external hosts are not affected. > > I thought I must have misconfigured something, as my setup is a bit weird. > Yesterday I noticed the same issue on a different host, turns out all my 14.0 > hosts seem to be affected, cognet@ could also reproduce it on one of his > systems. > > The service/software used does not seem to matter, I tried with port 22, 25, > 80 and 443. > > ICMP and UDP don't seem to be affected. ping6 gets replies immediately. And > UDP connections with nc -l -u / nc -u don't have any delay, sent data is > received immediately. > > Testing local TCP connections show this: > > flo@rp64:~ $ ifconfig dwc0|grep 2003 > inet6 2003:cf:df49:c97:4c59:ebff:fec1:463d prefixlen 64 autoconf > flo@rp64:~ $ nc -v 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 > [3 second delay here] > Connection to 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 port [tcp/ssh] > succeeded! > SSH-2.0-OpenSSH_8.9 FreeBSD-20220413 > > tcpdump on lo0 shows that the first two SYN packets are ignored / time out, > then the connection is successfully established. > > 19:28:38.685128 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > > 2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [S], seq 2489479594, win > 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 3410505643 ecr 0], > length 0 > > 19:28:39.696047 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > > 2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [S], seq 2489479594, win > 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 3410506654 ecr 0], > length 0 > > 19:28:41.897836 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > > 2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [S], seq 2489479594, win > 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 3410508856 ecr 0], > length 0 > > 19:28:41.897907 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.22 > > 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294: Flags [S.], seq 2857552476, ack > 2489479595, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val > 1858349482 ecr 3410508856], length 0 > > 19:28:41.897962 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > > 2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [.], ack 1, win 1276, options > [nop,nop,TS val 3410508856 ecr 1858349482], length 0 > > I need help debugging this, I don't know how to analyze this further. I will > start bisecting this, but I thought maybe someone has an idea. Hi Florian, I can reproduce this locally, will try to figure out what is going on. If you can bisect it, it would be great. Best regards Michael > > Thanks, > Florian >
