Memory leaks in conntrack
Hello, While testing my TC filter patches (so not related to conntrack), the following memory leaks are shown up: unreferenced object 0x9b19ba551228 (size 128): comm "chronyd", pid 338, jiffies 4294910829 (age 53.188s) hex dump (first 32 bytes): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 18 00 00 30 00 00 00 00 00 00 00 00 ...0 backtrace: [] create_object+0x169/0x2aa [] kmemleak_alloc+0x25/0x41 [] slab_post_alloc_hook+0x44/0x65 [] __kmalloc_track_caller+0x113/0x146 [] __krealloc+0x4a/0x69 [] nf_ct_ext_add+0xe1/0x145 [] init_conntrack+0x1f7/0x36e [] nf_conntrack_in+0x1d3/0x326 [] ipv4_conntrack_local+0x4d/0x50 [] nf_hook_slow+0x3c/0x9b [] nf_hook.constprop.40+0xbe/0xd8 [] __ip_local_out+0xb3/0xbf [] ip_local_out+0x1c/0x36 [] ip_send_skb+0x19/0x3d [] udp_send_skb+0x17e/0x1df [] udp_sendmsg+0x5a2/0x77c unreferenced object 0x9b19a69b3340 (size 336): comm "chronyd", pid 338, jiffies 4294910868 (age 53.032s) hex dump (first 32 bytes): 01 00 00 00 5a 5a 5a 5a 00 00 00 00 ad 4e ad de .N.. ff ff ff ff 5a 5a 5a 5a ff ff ff ff ff ff ff ff backtrace: [] create_object+0x169/0x2aa [] kmemleak_alloc+0x25/0x41 [] slab_post_alloc_hook+0x44/0x65 [] kmem_cache_alloc+0xd7/0x1f1 [] __nf_conntrack_alloc+0xa2/0x146 [] init_conntrack+0xb2/0x36e [] nf_conntrack_in+0x1d3/0x326 [] ipv4_conntrack_local+0x4d/0x50 [] nf_hook_slow+0x3c/0x9b [] nf_hook.constprop.40+0xbe/0xd8 [] __ip_local_out+0xb3/0xbf [] ip_local_out+0x1c/0x36 [] ip_send_skb+0x19/0x3d [] udp_send_skb+0x17e/0x1df [] udp_sendmsg+0x5a2/0x77c [] inet_sendmsg+0x37/0x5e This seems new because I never see this before. I don't touch chronyd in my VM, so I have no idea why it sends out UDP packets, my guess is it is some periodical packet. I don't think I use conntrack either, since /proc/net/ip_conntrack does not exist. Here are some related config of my kernel: $ grep CONNTRACK .config CONFIG_NF_CONNTRACK=y CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_CONNTRACK_SECMARK=y CONFIG_NF_CONNTRACK_ZONES=y CONFIG_NF_CONNTRACK_PROCFS=y CONFIG_NF_CONNTRACK_EVENTS=y # CONFIG_NF_CONNTRACK_TIMEOUT is not set CONFIG_NF_CONNTRACK_TIMESTAMP=y CONFIG_NF_CONNTRACK_AMANDA=y CONFIG_NF_CONNTRACK_FTP=y CONFIG_NF_CONNTRACK_H323=y CONFIG_NF_CONNTRACK_IRC=y CONFIG_NF_CONNTRACK_BROADCAST=y CONFIG_NF_CONNTRACK_NETBIOS_NS=y CONFIG_NF_CONNTRACK_SNMP=y CONFIG_NF_CONNTRACK_PPTP=y CONFIG_NF_CONNTRACK_SANE=y CONFIG_NF_CONNTRACK_SIP=y CONFIG_NF_CONNTRACK_TFTP=y CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y CONFIG_NF_CONNTRACK_IPV4=y CONFIG_NF_CONNTRACK_IPV6=y Please let me know if you need any other information. Thanks.
Re: Memory leaks in conntrack
Cong Wang wrote: > While testing my TC filter patches (so not related to conntrack), the > following memory leaks are shown up: > > unreferenced object 0x9b19ba551228 (size 128): > comm "chronyd", pid 338, jiffies 4294910829 (age 53.188s) > hex dump (first 32 bytes): > 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 00 00 00 00 18 00 00 30 00 00 00 00 00 00 00 00 ...0 > backtrace: > [] create_object+0x169/0x2aa > [] kmemleak_alloc+0x25/0x41 > [] slab_post_alloc_hook+0x44/0x65 > [] __kmalloc_track_caller+0x113/0x146 > [] __krealloc+0x4a/0x69 > [] nf_ct_ext_add+0xe1/0x145 > [] init_conntrack+0x1f7/0x36e > [] nf_conntrack_in+0x1d3/0x326 > [] ipv4_conntrack_local+0x4d/0x50 > [] nf_hook_slow+0x3c/0x9b > [] nf_hook.constprop.40+0xbe/0xd8 > [] __ip_local_out+0xb3/0xbf > [] ip_local_out+0x1c/0x36 > [] ip_send_skb+0x19/0x3d > [] udp_send_skb+0x17e/0x1df > [] udp_sendmsg+0x5a2/0x77c > unreferenced object 0x9b19a69b3340 (size 336): > comm "chronyd", pid 338, jiffies 4294910868 (age 53.032s) > hex dump (first 32 bytes): > 01 00 00 00 5a 5a 5a 5a 00 00 00 00 ad 4e ad de .N.. > ff ff ff ff 5a 5a 5a 5a ff ff ff ff ff ff ff ff > backtrace: > [] create_object+0x169/0x2aa > [] kmemleak_alloc+0x25/0x41 > [] slab_post_alloc_hook+0x44/0x65 > [] kmem_cache_alloc+0xd7/0x1f1 > [] __nf_conntrack_alloc+0xa2/0x146 > [] init_conntrack+0xb2/0x36e > [] nf_conntrack_in+0x1d3/0x326 > [] ipv4_conntrack_local+0x4d/0x50 > [] nf_hook_slow+0x3c/0x9b > [] nf_hook.constprop.40+0xbe/0xd8 > [] __ip_local_out+0xb3/0xbf > [] ip_local_out+0x1c/0x36 > [] ip_send_skb+0x19/0x3d > [] udp_send_skb+0x17e/0x1df > [] udp_sendmsg+0x5a2/0x77c > [] inet_sendmsg+0x37/0x5e > > I don't touch chronyd in my VM, so I have no idea why it sends out UDP > packets, my guess is it is some periodical packet. > > I don't think I use conntrack either, since /proc/net/ip_conntrack > does not exist. You probably do, can you try "cat /proc/net/nf_conntrack" instead? (otherwise there should be no ipv4_conntrack_local() invocation since we would not register this hook at all). I tried to reproduce this but so far I had no success. If you can identify something that could give a hint when this is happening (only once after boot, periodically, only with udp, etc) please let us know. (A reproducer would be even better of course ;-) ) Is this with current net tree? Thanks!
Re: Memory leaks in conntrack
Florian Westphal wrote: > Cong Wang wrote: > > While testing my TC filter patches (so not related to conntrack), the > > following memory leaks are shown up: > > > > unreferenced object 0x9b19ba551228 (size 128): > > comm "chronyd", pid 338, jiffies 4294910829 (age 53.188s) > > hex dump (first 32 bytes): > > 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > > 00 00 00 00 18 00 00 30 00 00 00 00 00 00 00 00 ...0 > > backtrace: > > [] create_object+0x169/0x2aa > > [] kmemleak_alloc+0x25/0x41 > > [] slab_post_alloc_hook+0x44/0x65 > > [] __kmalloc_track_caller+0x113/0x146 > > [] __krealloc+0x4a/0x69 > > [] nf_ct_ext_add+0xe1/0x145 > > [] init_conntrack+0x1f7/0x36e > > [] nf_conntrack_in+0x1d3/0x326 > > [] ipv4_conntrack_local+0x4d/0x50 > > [] nf_hook_slow+0x3c/0x9b > > [] nf_hook.constprop.40+0xbe/0xd8 > > [] __ip_local_out+0xb3/0xbf > > [] ip_local_out+0x1c/0x36 > > [] ip_send_skb+0x19/0x3d > > [] udp_send_skb+0x17e/0x1df > > [] udp_sendmsg+0x5a2/0x77c > > unreferenced object 0x9b19a69b3340 (size 336): > > comm "chronyd", pid 338, jiffies 4294910868 (age 53.032s) > > hex dump (first 32 bytes): > > 01 00 00 00 5a 5a 5a 5a 00 00 00 00 ad 4e ad de .N.. > > ff ff ff ff 5a 5a 5a 5a ff ff ff ff ff ff ff ff > > backtrace: > > [] create_object+0x169/0x2aa > > [] kmemleak_alloc+0x25/0x41 > > [] slab_post_alloc_hook+0x44/0x65 > > [] kmem_cache_alloc+0xd7/0x1f1 > > [] __nf_conntrack_alloc+0xa2/0x146 > > [] init_conntrack+0xb2/0x36e > > [] nf_conntrack_in+0x1d3/0x326 > > [] ipv4_conntrack_local+0x4d/0x50 > > [] nf_hook_slow+0x3c/0x9b > > [] nf_hook.constprop.40+0xbe/0xd8 > > [] __ip_local_out+0xb3/0xbf > > [] ip_local_out+0x1c/0x36 > > [] ip_send_skb+0x19/0x3d > > [] udp_send_skb+0x17e/0x1df > > [] udp_sendmsg+0x5a2/0x77c > > [] inet_sendmsg+0x37/0x5e > > > > I don't touch chronyd in my VM, so I have no idea why it sends out UDP > > packets, my guess is it is some periodical packet. > > > > I don't think I use conntrack either, since /proc/net/ip_conntrack > > does not exist. > > You probably do, can you try "cat /proc/net/nf_conntrack" instead? > > (otherwise there should be no ipv4_conntrack_local() invocation > since we would not register this hook at all). > > I tried to reproduce this but so far I had no success. > If you can identify something that could give a hint when this > is happening (only once after boot, periodically, only with udp, etc) > please let us know. FWIW i managed to obtain a similar backtrace, but in that case it was a false positive (peeking at the address content showed it was my ssh connection to the vm and timeout and tcp conntrackk struct fields were changing; i.e. the nf_conn reported was still in the conntrack hash. Why this address was reported i do not know, afaik kmemleak does scan for addresses anywhere in the object (we use container_of() to get back nf_conn from the hlist_node), so it should have found the address linked via the main conntrack hash table. Right now I don't have enough info to dig any further, sorry :-/
Re: Memory leaks in conntrack
On Wed, Sep 13, 2017 at 1:05 AM, Florian Westphal wrote: > Cong Wang wrote: >> While testing my TC filter patches (so not related to conntrack), the >> following memory leaks are shown up: >> >> unreferenced object 0x9b19ba551228 (size 128): >> comm "chronyd", pid 338, jiffies 4294910829 (age 53.188s) >> hex dump (first 32 bytes): >> 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 00 00 00 00 18 00 00 30 00 00 00 00 00 00 00 00 ...0 >> backtrace: >> [] create_object+0x169/0x2aa >> [] kmemleak_alloc+0x25/0x41 >> [] slab_post_alloc_hook+0x44/0x65 >> [] __kmalloc_track_caller+0x113/0x146 >> [] __krealloc+0x4a/0x69 >> [] nf_ct_ext_add+0xe1/0x145 >> [] init_conntrack+0x1f7/0x36e >> [] nf_conntrack_in+0x1d3/0x326 >> [] ipv4_conntrack_local+0x4d/0x50 >> [] nf_hook_slow+0x3c/0x9b >> [] nf_hook.constprop.40+0xbe/0xd8 >> [] __ip_local_out+0xb3/0xbf >> [] ip_local_out+0x1c/0x36 >> [] ip_send_skb+0x19/0x3d >> [] udp_send_skb+0x17e/0x1df >> [] udp_sendmsg+0x5a2/0x77c >> unreferenced object 0x9b19a69b3340 (size 336): >> comm "chronyd", pid 338, jiffies 4294910868 (age 53.032s) >> hex dump (first 32 bytes): >> 01 00 00 00 5a 5a 5a 5a 00 00 00 00 ad 4e ad de .N.. >> ff ff ff ff 5a 5a 5a 5a ff ff ff ff ff ff ff ff >> backtrace: >> [] create_object+0x169/0x2aa >> [] kmemleak_alloc+0x25/0x41 >> [] slab_post_alloc_hook+0x44/0x65 >> [] kmem_cache_alloc+0xd7/0x1f1 >> [] __nf_conntrack_alloc+0xa2/0x146 >> [] init_conntrack+0xb2/0x36e >> [] nf_conntrack_in+0x1d3/0x326 >> [] ipv4_conntrack_local+0x4d/0x50 >> [] nf_hook_slow+0x3c/0x9b >> [] nf_hook.constprop.40+0xbe/0xd8 >> [] __ip_local_out+0xb3/0xbf >> [] ip_local_out+0x1c/0x36 >> [] ip_send_skb+0x19/0x3d >> [] udp_send_skb+0x17e/0x1df >> [] udp_sendmsg+0x5a2/0x77c >> [] inet_sendmsg+0x37/0x5e >> >> I don't touch chronyd in my VM, so I have no idea why it sends out UDP >> packets, my guess is it is some periodical packet. >> >> I don't think I use conntrack either, since /proc/net/ip_conntrack >> does not exist. > > You probably do, can you try "cat /proc/net/nf_conntrack" instead? > > (otherwise there should be no ipv4_conntrack_local() invocation > since we would not register this hook at all). Yeah it is very weird but it is true: [root@localhost ~]# echo scan > /sys/kernel/debug/kmemleak [ 133.450823] kmemleak: 18 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [root@localhost ~]# cat /proc/net/ip_conntrack cat: /proc/net/ip_conntrack: No such file or directory [root@localhost ~]# cat /sys/kernel/debug/kmemleak unreferenced object 0x95c1e0b24040 (size 336): ... > > I tried to reproduce this but so far I had no success. > If you can identify something that could give a hint when this > is happening (only once after boot, periodically, only with udp, etc) > please let us know. > > (A reproducer would be even better of course ;-) ) Actually, it is even simpler to reproduce, nothing is needed but wait. I thought it is somewhat triggered by my tests, but actually no. For me, just boot the VM and wait for several seconds, memleak will show up. (chronyd is started by systemd during boot, not me.) > > Is this with current net tree? Yes, I just pulled DaveM's net tree and recompiled the kernel, still 100% reproducible here.
Re: Memory leaks in conntrack
On Wed, Sep 13, 2017 at 9:45 AM, Cong Wang wrote: > On Wed, Sep 13, 2017 at 1:05 AM, Florian Westphal wrote: >> Cong Wang wrote: >>> While testing my TC filter patches (so not related to conntrack), the >>> following memory leaks are shown up: >>> >>> unreferenced object 0x9b19ba551228 (size 128): >>> comm "chronyd", pid 338, jiffies 4294910829 (age 53.188s) >>> hex dump (first 32 bytes): >>> 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >>> 00 00 00 00 18 00 00 30 00 00 00 00 00 00 00 00 ...0 >>> backtrace: >>> [] create_object+0x169/0x2aa >>> [] kmemleak_alloc+0x25/0x41 >>> [] slab_post_alloc_hook+0x44/0x65 >>> [] __kmalloc_track_caller+0x113/0x146 >>> [] __krealloc+0x4a/0x69 >>> [] nf_ct_ext_add+0xe1/0x145 >>> [] init_conntrack+0x1f7/0x36e >>> [] nf_conntrack_in+0x1d3/0x326 >>> [] ipv4_conntrack_local+0x4d/0x50 >>> [] nf_hook_slow+0x3c/0x9b >>> [] nf_hook.constprop.40+0xbe/0xd8 >>> [] __ip_local_out+0xb3/0xbf >>> [] ip_local_out+0x1c/0x36 >>> [] ip_send_skb+0x19/0x3d >>> [] udp_send_skb+0x17e/0x1df >>> [] udp_sendmsg+0x5a2/0x77c >>> unreferenced object 0x9b19a69b3340 (size 336): >>> comm "chronyd", pid 338, jiffies 4294910868 (age 53.032s) >>> hex dump (first 32 bytes): >>> 01 00 00 00 5a 5a 5a 5a 00 00 00 00 ad 4e ad de .N.. >>> ff ff ff ff 5a 5a 5a 5a ff ff ff ff ff ff ff ff >>> backtrace: >>> [] create_object+0x169/0x2aa >>> [] kmemleak_alloc+0x25/0x41 >>> [] slab_post_alloc_hook+0x44/0x65 >>> [] kmem_cache_alloc+0xd7/0x1f1 >>> [] __nf_conntrack_alloc+0xa2/0x146 >>> [] init_conntrack+0xb2/0x36e >>> [] nf_conntrack_in+0x1d3/0x326 >>> [] ipv4_conntrack_local+0x4d/0x50 >>> [] nf_hook_slow+0x3c/0x9b >>> [] nf_hook.constprop.40+0xbe/0xd8 >>> [] __ip_local_out+0xb3/0xbf >>> [] ip_local_out+0x1c/0x36 >>> [] ip_send_skb+0x19/0x3d >>> [] udp_send_skb+0x17e/0x1df >>> [] udp_sendmsg+0x5a2/0x77c >>> [] inet_sendmsg+0x37/0x5e >>> >>> I don't touch chronyd in my VM, so I have no idea why it sends out UDP >>> packets, my guess is it is some periodical packet. >>> >>> I don't think I use conntrack either, since /proc/net/ip_conntrack >>> does not exist. >> >> You probably do, can you try "cat /proc/net/nf_conntrack" instead? >> >> (otherwise there should be no ipv4_conntrack_local() invocation >> since we would not register this hook at all). > > Yeah it is very weird but it is true: > > [root@localhost ~]# echo scan > /sys/kernel/debug/kmemleak > [ 133.450823] kmemleak: 18 new suspected memory leaks (see > /sys/kernel/debug/kmemleak) > [root@localhost ~]# cat /proc/net/ip_conntrack > cat: /proc/net/ip_conntrack: No such file or directory > [root@localhost ~]# cat /sys/kernel/debug/kmemleak > unreferenced object 0x95c1e0b24040 (size 336): > ... Oops, you mean nf_conntrack... Here we go: [root@localhost ~]# cat /proc/net/nf_conntrack ipv4 2 udp 17 116 src=192.168.124.6 dst=204.2.134.162 sport=123 dport=123 src=204.2.134.162 dst=192.168.124.6 sport=123 dport=123 [ASSURED] mark=0 zone=0 use=2 ipv4 2 udp 17 117 src=192.168.124.6 dst=45.79.187.10 sport=123 dport=123 src=45.79.187.10 dst=192.168.124.6 sport=123 dport=123 [ASSURED] mark=0 zone=0 use=2 ipv4 2 udp 17 110 src=192.168.124.6 dst=192.168.124.1 sport=35486 dport=53 src=192.168.124.1 dst=192.168.124.6 sport=53 dport=35486 [ASSURED] mark=0 zone=0 use=2 ipv4 2 udp 17 110 src=192.168.124.6 dst=192.168.124.1 sport=52373 dport=53 src=192.168.124.1 dst=192.168.124.6 sport=53 dport=52373 [ASSURED] mark=0 zone=0 use=2 ipv4 2 unknown 2 518 src=192.168.124.6 dst=224.0.0.22 [UNREPLIED] src=224.0.0.22 dst=192.168.124.6 mark=0 zone=0 use=2 ipv4 2 udp 17 110 src=192.168.124.6 dst=192.168.124.1 sport=43242 dport=53 src=192.168.124.1 dst=192.168.124.6 sport=53 dport=43242 [ASSURED] mark=0 zone=0 use=2 ipv4 2 udp 17 116 src=192.168.124.6 dst=96.226.123.196 sport=123 dport=123 src=96.226.123.196 dst=192.168.124.6 sport=123 dport=123 [ASSURED] mark=0 zone=0 use=2 ipv4 2 udp 17 110 src=192.168.124.6 dst=192.168.124.1 sport=42838 dport=53 src=192.168.124.1 dst=192.168.124.6 sport=53 dport=42838 [ASSURED] mark=0 zone=0 use=2 ipv4 2 udp 17 117 src=192.168.124.6 dst=97.127.104.4 sport=123 dport=123 src=97.127.104.4 dst=192.168.124.6 sport=123 dport=123 [ASSURED] mark=0 zone=0 use=2