Bug#1050418: Conntrackd in Bookworm reverts byte order in src address sent by conntrackd in Bullseye
Control: tags -1 + patch
On 2023-08-24, at 20:33:13 +0100, Jeremy Sowden wrote:
> On 2023-08-24, at 12:55:30 +0200, Pavel Matěja wrote:
> > I'm upgrading our servers from Bullseye to Bookworm. Some of them
> > act as load balancers and they are using conntrackd to synchronize
> > TCP connection states using FTFW sync mode. I've noticed when I
> > have primary server running Bullseye (conntrack v1.4.6) and
> > secondary Bookworm (conntrack v1.4.7) I get
> >
> > bullseye:~$ sudo conntrack -L
> > ..
> > tcp 6 430554 ESTABLISHED src=x.y.49.137 dst=x.y.48.169
> > sport=35570 dport=636 src=10.170.0.153 dst=x.y.49.137
> > sport=636 dport=35570 [ASSURED] mark=0 use=1
> > ..
> >
> > bookworm:~$ sudo conntrack -L
> > ..
> > tcp 6 431388 ESTABLISHED src=x.y.49.137 dst=x.y.48.169
> > sport=35570 dport=636 src=153.0.170.10 dst=x.y.49.137
> > sport=636 dport=35570 [ASSURED] mark=0 use=1
> > ..
> >
> > Notice order of the 'src' address bytes.
> > When failover occures all TCP connections via secondary balancer are
> > broken as packets source addresses don't match those in conntrack
> > table anymore.
> >
> > [...]
> >
> > Core of this problem might be related to
> > https://git.netfilter.org/conntrack-tools/commit/?id=b55717d46ae3b7c3769192a66e565bc7c2d833a1
> > but I'm not familiar with conntrackd source code.
>
> I believe you are correct in identifying b55717d46ae3 ("conntrackd:
> fix endianness bug in IPv4 and IPv6 address").
>
> [...]
>
> I believe the upstream switch to NBO is correct, but I'm afraid that
> we in Debian didn't spot this consequence. I'll see about getting a
> notice added to the package documentation.
Something like this patch.
J.
From 04f032718c8ab3fcab7e101c988f67d2ebde5ab3 Mon Sep 17 00:00:00 2001
From: Jeremy Sowden
Date: Thu, 24 Aug 2023 21:03:57 +0100
Subject: [PATCH] d/NEWS: add notice about 1.4.6/1.4.7 little-endian
imcompatibility
Closes: #1050418
Signed-off-by: Jeremy Sowden
---
debian/NEWS | 16
1 file changed, 16 insertions(+)
create mode 100644 debian/NEWS
diff --git a/debian/NEWS b/debian/NEWS
new file mode 100644
index ..2b6e47aa77ad
--- /dev/null
+++ b/debian/NEWS
@@ -0,0 +1,16 @@
+conntrack-tools (1:1.4.7-2) unstable; urgency=medium
+
+ 1.4.6 conntrackd instances runnning on little-endian hosts are not
+ compatible with 1.4.7 instances.
+
+ Before 1.4.7, when syncing, conntrackd instances communicated NAT IP
+ addresses in host byte-order. This meant that one could not run
+ instances on hosts of different endianness, because if the byte-orders
+ of the sending and receiving instances were different, the receiving
+ instance would get the addresses inverted: 10.0.0.1 -> 1.0.0.10.
+
+ In 1.4.7, conntrackd uses network byte-order. Since network byte-order
+ is big-endian, all 1.4.7 instances are incompatible with 1.4.6 little-
+ endian instances.
+
+ -- Jeremy Sowden Thu, 24 Aug 2023 20:41:50 +0100
--
2.40.1
signature.asc
Description: PGP signature
Bug#1050418: Conntrackd in Bookworm reverts byte order in src address sent by conntrackd in Bullseye
Control: tags -1 confirmed upstream
On 2023-08-24, at 12:55:30 +0200, Pavel Matěja wrote:
> Package: conntrackd
> Version: 1:1.4.7-1+b2
> Conntrackd package on Bullseye is 1:1.4.6-2.
>
> I'm upgrading our servers from Bullseye to Bookworm. Some of them act
> as load balancers and they are using conntrackd to synchronize TCP
> connection states using FTFW sync mode.
> I've noticed when I have primary server running Bullseye (conntrack
> v1.4.6) and secondary Bookworm (conntrack v1.4.7) I get
>
> bullseye:~$ sudo conntrack -L
> ..
> tcp 6 430554 ESTABLISHED src=x.y.49.137 dst=x.y.48.169 sport=35570
> dport=636 src=10.170.0.153 dst=x.y.49.137 sport=636 dport=35570 [ASSURED]
> mark=0 use=1
> ..
>
> bookworm:~$ sudo conntrack -L
> ..
> tcp 6 431388 ESTABLISHED src=x.y.49.137 dst=x.y.48.169 sport=35570
> dport=636 src=153.0.170.10 dst=x.y.49.137 sport=636 dport=35570 [ASSURED]
> mark=0 use=1
> ..
>
> Notice order of the 'src' address bytes.
> When failover occures all TCP connections via secondary balancer are
> broken as packets source addresses don't match those in conntrack
> table anymore.
>
> Downgrade of conntrack and conntrackd packages on Bookworm server
> solved this problem.
> I was unable to create 1.4.7 package for Bullseye.
> I'm not sure which version is considered to be acting correctly.
>
> Core of this problem might be related to
> https://git.netfilter.org/conntrack-tools/commit/?id=b55717d46ae3b7c3769192a66e565bc7c2d833a1
> but I'm not familiar with conntrackd source code.
I believe you are correct in identifying b55717d46ae3 ("conntrackd: fix
endianness bug in IPv4 and IPv6 address"). Before 1.4.7, conntrackd
sent IP addresses in host byte order. However, this prevented one from
running conntrackd instances on hosts of different endianness:
https://marc.info/?l=netfilter=161886262729364=2
This commit changes conntrackd to use network byte order instead. The
consequence, of course, is that little-endian 1.4.6 instances are not
compatible with little-endian 1.4.7 instances.
I believe the upstream switch to NBO is correct, but I'm afraid that we
in Debian didn't spot this consequence. I'll see about getting a notice
added to the package documentation.
J.
signature.asc
Description: PGP signature
Bug#1050418: Conntrackd in Bookworm reverts byte order in src address sent by conntrackd in Bullseye
Package: conntrackd Version: 1:1.4.7-1+b2 Conntrackd package on Bullseye is 1:1.4.6-2. I'm upgrading our servers from Bullseye to Bookworm. Some of them act as load balancers and they are using conntrackd to synchronize TCP connection states using FTFW sync mode. I've noticed when I have primary server running Bullseye (conntrack v1.4.6) and secondary Bookworm (conntrack v1.4.7) I get bullseye:~$ sudo conntrack -L .. tcp 6 430554 ESTABLISHED src=x.y.49.137 dst=x.y.48.169 sport=35570 dport=636 src=10.170.0.153 dst=x.y.49.137 sport=636 dport=35570 [ASSURED] mark=0 use=1 .. bookworm:~$ sudo conntrack -L .. tcp 6 431388 ESTABLISHED src=x.y.49.137 dst=x.y.48.169 sport=35570 dport=636 src=153.0.170.10 dst=x.y.49.137 sport=636 dport=35570 [ASSURED] mark=0 use=1 .. Notice order of the 'src' address bytes. When failover occures all TCP connections via secondary balancer are broken as packets source addresses don't match those in conntrack table anymore. Downgrade of conntrack and conntrackd packages on Bookworm server solved this problem. I was unable to create 1.4.7 package for Bullseye. I'm not sure which version is considered to be acting correctly. Core of this problem might be related to https://git.netfilter.org/conntrack-tools/commit/?id=b55717d46ae3b7c3769192a66e565bc7c2d833a1 but I'm not familiar with conntrackd source code. I'm sorry I had to mask the public ip.
