[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Dan Kegel, would utilizing the Raring enablement stack in Precise work for you as per https://wiki.ubuntu.com/Kernel/LTSEnablementStack ? ** Changed in: linux (Ubuntu) Importance: Undecided = Medium ** Changed in: linux (Ubuntu) Status: Confirmed = Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
(Marking fix-released for lxc per comment #19) ** Changed in: lxc (Ubuntu) Status: Triaged = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Where does this stand? A fully updated 12.04.1 system is still seeing lots of interfaces; 2015 network-interface 4028 network-interface-security and toggling an lxc container up and down four times seemed to result in one extra network-interface and four extra network-interface-security's. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
I fixed the userspace side of this in 13.04 and the 3.8 kernel contains the matching kernel fix. I could probably backport the fix to 12.04, though it won't work until the 13.04 kernel is backported and people explicitly move to it. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Thanks. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
I'm running ltp against my proposed patch, and will send it to netdev (per the email thread) later today. ** Tags added: bot-stop-nagging ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Running this script periodically seems to work around the problem. Only lightly tested. ** Attachment added: gc.sh https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+attachment/3396509/+files/gc.sh -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Oops, that only deleted one of the jobs. This draft deletes both. ** Attachment added: gc.sh https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+attachment/3396518/+files/gc.sh -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
@Dan, note that you can also compare the list of running 'network-interface' jobs to the veth's in /sys/class/net/. For those which are not there, you can remove the network-interface job using sudo initctl emit net-device-removed INTERFACE=$thenicyoufound -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Once more with feeling. ** Attachment added: bug1065589-gc.sh https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+attachment/3396522/+files/bug1065589-gc.sh -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Email thread about a potential kernel patch to solve the problem: http://lists.linuxfoundation.org/pipermail/containers/2012-October/030519.html -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Alas, that archive doesn't show attachments. For the record, is there a better archive somewhere? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Hm, I didn't know it did that. The patch wasn't even an attachment, just inline. I'll attach the new version here. There is another archive at sf.net, but it doesn't seem to have oct 12 messages yet. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
** Patch added: 0001-dev_change_net_namespace-send-a-KOBJ_REMOVED-to-orig.patch https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+attachment/3396958/+files/0001-dev_change_net_namespace-send-a-KOBJ_REMOVED-to-orig.patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
** Changed in: lxc (Ubuntu) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Note that the count goes up by 1 for each container. But each container has 2 veths. My guess is that when lxc passes one of the 2 veths into the container, we need to emit a net-device-removed signal for the passed-in device. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Yup, I can reproduce this. However I believe the bug is in the init script, not in lxc. The veth interfaces are not around. ** Changed in: lxc (Ubuntu) Status: New = Triaged -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Ah, no, both veths have a network-interface job sticking around. Could this be seen as a kernel/udev bug, that when they veth is destroyed (and maybe even when passed to a new netns) a uevent should be sent? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
This can be worked around by manually stopping the jobs, for instance: sudo stop network-interface-security JOB=network-interface INTERFACE=vethCEDGKQ So lowering priority (per guidelines) since there is a workaround. ** Changed in: lxc (Ubuntu) Importance: High = Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
I've opened bug 1065684 to handle the network-interface-security jobs. Those can just be made to go away immediately. THe network-interface jobs are a bit more contraversial. The question is whether the kernel should emit a signal, or whether lxc should, when a veth is moved to another network namespace. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Looking at the kernel source, I believe the uevent is being sent. I think udev (or upstart-udev-bridge) may be tossing the event because the device is no longer valid. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1065589] Re: initctl list shows 11974 instances of network-interface-security after two days of uptime
Running 'udevadm monitor' while starting and stopping a container gives: KERNEL[17237.641503] add /devices/virtual/net/veth7t59jJ (net) KERNEL[17237.644300] add /devices/virtual/net/veth7t59jJ/queues/rx-0 (queues) KERNEL[17237.644577] add /devices/virtual/net/veth7t59jJ/queues/tx-0 (queues) KERNEL[17237.644855] add /devices/virtual/net/vethoLdAjv (net) KERNEL[17237.644906] add /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues) KERNEL[17237.644934] add /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues) KERNEL[17237.649498] add /kernel/slab/nf_conntrack_88007a72 (slab) KERNEL[17237.652309] add /devices/virtual/net/lo/queues/rx-0 (queues) KERNEL[17237.652348] add /devices/virtual/net/lo/queues/tx-0 (queues) UDEV [17237.657166] add /devices/virtual/net/lo/queues/tx-0 (queues) UDEV [17237.659337] add /devices/virtual/net/lo/queues/rx-0 (queues) UDEV [17237.659652] add /kernel/slab/nf_conntrack_88007a72 (slab) UDEV [17237.679686] add /devices/virtual/net/veth7t59jJ (net) UDEV [17237.680836] add /devices/virtual/net/veth7t59jJ/queues/rx-0 (queues) UDEV [17237.681041] add /devices/virtual/net/veth7t59jJ/queues/tx-0 (queues) UDEV [17237.698158] add /devices/virtual/net/vethoLdAjv (net) UDEV [17237.699761] add /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues) UDEV [17237.699948] add /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues) KERNEL[17244.427892] remove /devices/virtual/net/eth0/queues/rx-0 (queues) UDEV [17244.429127] remove /devices/virtual/net/eth0/queues/rx-0 (queues) KERNEL[17244.429466] remove /devices/virtual/net/eth0/queues/tx-0 (queues) UDEV [17244.429918] remove /devices/virtual/net/eth0/queues/tx-0 (queues) KERNEL[17244.431075] remove /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues) UDEV [17244.431564] remove /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues) KERNEL[17244.432570] remove /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues) UDEV [17244.433942] remove /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues) KERNEL[17244.435509] remove /devices/virtual/net/vethoLdAjv (net) UDEV [17244.436596] remove /devices/virtual/net/vethoLdAjv (net) KERNEL[17244.440529] remove /devices/virtual/net/lo/queues/rx-0 (queues) UDEV [17244.440979] remove /devices/virtual/net/lo/queues/rx-0 (queues) KERNEL[17244.441098] remove /devices/virtual/net/lo/queues/tx-0 (queues) UDEV [17244.441492] remove /devices/virtual/net/lo/queues/tx-0 (queues) KERNEL[17244.706760] remove /kernel/slab/nf_conntrack_88007a72 (slab) UDEV [17244.707342] remove /kernel/slab/nf_conntrack_88007a72 (slab) Note that while vethoLdAjv has both kernel and udev add and remove, veth7t59jJ has only the add messages. My previous comment is likely wrong, because the *renamed* veth7t59jJ (eth0 in the container namespace) does have remove messages, and no add messages. (My point here being that since we see those messages, for an eth0 in another network namespace, then we should also be seeing the messages for a veth7t59jJ while being moved to a new netns if those were being sent - the virtual 'eth0' does not have a /sys/class/net entry in the initial network namespace either) Therefore it seems most likely that the kernel is not actually sending the remove/add messages when a netdev is moved to a new namespace, nor giving hints when the netdev is renamed (which lxc does after moving it to a new namespace, obviously, to prevent name conflicts with 'eth0'). SO - lxc could finagle this by emitting upstart signals at src/lxc/conf.c:lxc_assign_network() or src/lxc/network.c:lxc_network_move_by_index(), but first it's worth considering whether the kernel should be sending signals. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1065589 Title: initctl list shows 11974 instances of network-interface-security after two days of uptime To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1065589/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
