Bug#800445: nf_conntrack: table full, dropping packet

[Michal Kašpar]

I've rebuilt the kernel with the patch this morning and the problem
didn't appear since then. Before the patch, the kernel had oopsed
within 10 minutes from start. Thanks for good work.

-- 
Michal Kašpar

Bug#800445: nf_conntrack: table full, dropping packet

[anomie]

On Sat, Oct 03, 2015 at 02:33:22AM +0100, Ben Hutchings wrote:
> 
> I think I found the fix for this, but please can you test the attached
> patch?  (Instructions for building a patched kernel package are at 
> .)

Sorry for the delay in testing. The attached patch does seem to work.

Thanks!

Bug#800445: nf_conntrack: table full, dropping packet

[anomie]

Package: linux-image-4.2.0-1-amd64
Version: 4.2.1-2

After booting the kernel image provided by this package and attempting
to connect to the Internet, network access doesn't actually work. The
message "nf_conntrack: table full, dropping packet" is repeatedly
logged.

Comparing the contents of the various /proc/sys conntrack files between
a working 4.1 kernel (from linux-image-4.1.0-2-amd64) and the broken
4.2, the only difference I see is that nf_conntrack_count has a value
"-5". Yes, negative 5. /proc/net/stat/nf_conntrack's "entries" column
matches this with a value of fffb. /proc/net/nf_conntrack is empty
on the 4.2 kernel, while it has a handful of expected entries on 4.1.

I have iptables rules set up by shorewall on this machine, including
configuration to forward/masq traffic on the interface used by vde2, in
case that helps reproduce this. Removing all the iptables rules and
removing the nf_conntrack_ipv4 module (and everything that depends on
it, of course) stops the error. Disabling shorewall at boot allows
network functionality and starting shorewall later didn't immediately
cause the problem, but in some experimentation after the system locked
up.

Bug#800445: nf_conntrack: table full, dropping packet

[C . Dominik Bódi]

severity: grave

I've got the same problem on my debian unstable system. I'm running shorewall, 
as well. 4.1 kernels seem to run fine but the latest 4.2 kernel in unstable 
shuts the firewall completely. I'm seeing the same error messages in dmesg, as 
well.

This makes a server unusable, as the firewall effectively shuts down all 
networking. Thus the severity of this bug report should be raised to "grave"

Regards,
C. Dominik Bódi

signature.asc
Description: This is a digitally signed message part.

Bug#800445: nf_conntrack: table full, dropping packet

[Ben Hutchings]

Control: tag -1 upstream fixed-upstream patch moreinfo

On Tue, 2015-09-29 at 08:20 -0400, ano...@users.sourceforge.net wrote:
> Package: linux-image-4.2.0-1-amd64
> Version: 4.2.1-2
> 
> After booting the kernel image provided by this package and attempting
> to connect to the Internet, network access doesn't actually work. The
> message "nf_conntrack: table full, dropping packet" is repeatedly
> logged.
> 
> Comparing the contents of the various /proc/sys conntrack files between
> a working 4.1 kernel (from linux-image-4.1.0-2-amd64) and the broken
> 4.2, the only difference I see is that nf_conntrack_count has a value
> "-5". Yes, negative 5. /proc/net/stat/nf_conntrack's "entries" column
> matches this with a value of fffb. /proc/net/nf_conntrack is empty
> on the 4.2 kernel, while it has a handful of expected entries on 4.1.
> 
> I have iptables rules set up by shorewall on this machine, including
> configuration to forward/masq traffic on the interface used by vde2, in
> case that helps reproduce this. Removing all the iptables rules and
> removing the nf_conntrack_ipv4 module (and everything that depends on
> it, of course) stops the error. Disabling shorewall at boot allows
> network functionality and starting shorewall later didn't immediately
> cause the problem, but in some experimentation after the system locked
> up.

I think I found the fix for this, but please can you test the attached
patch?  (Instructions for building a patched kernel package are at 
.)

Ben.

-- 
Ben Hutchings
compatible: Gracefully accepts erroneous data from any sourceFrom: Daniel Borkmann 
Date: Mon, 31 Aug 2015 19:11:02 +0200
Subject: netfilter: conntrack: use nf_ct_tmpl_free in CT/synproxy error paths
Origin: https://git.kernel.org/linus/9cf94eab8b309e8bcc78b41dd1561c75b537dd0b
Bug-Debian: https://bugs.debian.org/800445

Commit 0838aa7fcfcd ("netfilter: fix netns dependencies with conntrack
templates") migrated templates to the new allocator api, but forgot to
update error paths for them in CT and synproxy to use nf_ct_tmpl_free()
instead of nf_conntrack_free().

Due to that, memory is being freed into the wrong kmemcache, but also
we drop the per net reference count of ct objects causing an imbalance.

In Brad's case, this leads to a wrap-around of net->ct.count and thus
lets __nf_conntrack_alloc() refuse to create a new ct object:

  [   10.340913] xt_addrtype: ipv6 does not support BROADCAST matching
  [   10.810168] nf_conntrack: table full, dropping packet
  [   11.917416] r8169 :07:00.0 eth0: link up
  [   11.917438] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
  [   12.815902] nf_conntrack: table full, dropping packet
  [   15.688561] nf_conntrack: table full, dropping packet
  [   15.689365] nf_conntrack: table full, dropping packet
  [   15.690169] nf_conntrack: table full, dropping packet
  [   15.690967] nf_conntrack: table full, dropping packet
  [...]

With slab debugging, it also reports the wrong kmemcache (kmalloc-512 vs.
nf_conntrack_81ce75c0) and reports poison overwrites, etc. Thus,
to fix the problem, export and use nf_ct_tmpl_free() instead.

Fixes: 0838aa7fcfcd ("netfilter: fix netns dependencies with conntrack templates")
Reported-by: Brad Jackson 
Signed-off-by: Daniel Borkmann 
Signed-off-by: Pablo Neira Ayuso 
---
 include/net/netfilter/nf_conntrack.h | 1 +
 net/netfilter/nf_conntrack_core.c| 3 ++-
 net/netfilter/nf_synproxy_core.c | 2 +-
 net/netfilter/xt_CT.c| 2 +-
 4 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/include/net/netfilter/nf_conntrack.h b/include/net/netfilter/nf_conntrack.h
index 37cd391..4023c4c 100644
--- a/include/net/netfilter/nf_conntrack.h
+++ b/include/net/netfilter/nf_conntrack.h
@@ -292,6 +292,7 @@ extern unsigned int nf_conntrack_hash_rnd;
 void init_nf_conntrack_hash_rnd(void);
 
 struct nf_conn *nf_ct_tmpl_alloc(struct net *net, u16 zone, gfp_t flags);
+void nf_ct_tmpl_free(struct nf_conn *tmpl);
 
 #define NF_CT_STAT_INC(net, count)	  __this_cpu_inc((net)->ct.stat->count)
 #define NF_CT_STAT_INC_ATOMIC(net, count) this_cpu_inc((net)->ct.stat->count)
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index 3c20d02..0625a42 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -320,12 +320,13 @@ out_free:
 }
 EXPORT_SYMBOL_GPL(nf_ct_tmpl_alloc);
 
-static void nf_ct_tmpl_free(struct nf_conn *tmpl)
+void nf_ct_tmpl_free(struct nf_conn *tmpl)
 {
 	nf_ct_ext_destroy(tmpl);
 	nf_ct_ext_free(tmpl);
 	kfree(tmpl);
 }
+EXPORT_SYMBOL_GPL(nf_ct_tmpl_free);
 
 static void
 destroy_conntrack(struct nf_conntrack *nfct)
diff --git a/net/netfilter/nf_synproxy_core.c b/net/netfilter/nf_synproxy_core.c
index d7f1685..d6ee8f8 100644
--- a/net/netfilter/nf_synproxy_core.c
+++ b/net/netfilter/nf_synproxy_core.c
@@ -378,7 +378,7 @@ static int __net_init synproxy_net_init(s

Processed: Re: Bug#800445: nf_conntrack: table full, dropping packet

[Debian Bug Tracking System]

Processing control commands:

> tag -1 upstream fixed-upstream patch moreinfo
Bug #800445 [linux-image-4.2.0-1-amd64] nf_conntrack: table full, dropping 
packet
Added tag(s) upstream, fixed-upstream, moreinfo, and patch.

-- 
800445: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800445
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems