[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-26 Thread Launchpad Bug Tracker
This bug was fixed in the package iptables - 1.8.3-2ubuntu2 --- iptables (1.8.3-2ubuntu2) eoan; urgency=medium * d/p/lp-1840633-nft-exit-in-case-we-can-t-fetch-current-genid.patch: avoid busy loop if cache can't be created (LP: #1840633) -- Christian Ehrhardt Wed, 21 Aug 201

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Christian Ehrhardt 
@Jamie - you are right and that is exactly the fix that I have up for review since a few hours in https://code.launchpad.net/~paelzer/ubuntu/+source/iptables/+git/iptables/+merge/371575 There are no "other cases needing a similar commit", I meant "this fix also helps other cases that ran into this

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Christian Ehrhardt 
Review is done, just waiting for the tests to confirm to be ok. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to iptables in Ubuntu. https://bugs.launchpad.net/bugs/1840633 Title: autopkgtests get stuck in Eoan with iptables

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Jamie Strandboge
Indeed, that is exactly what https://git.netfilter.org/iptables/commit/?id=e5cab728c40be88c541f68e4601d39178c36111f did. Are you saying there are other cases where a similar commit is needed? IMO, those should be patched before 1.8.3 goes into eoan. Unless I am missing something, iptables is correc

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Jamie Strandboge
It seems like iptables going into a busy loop as non-root is also a bug that should be fixed? At the very least, iptables should bail prior to that condition saying that root is needed. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscr

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/iptables/+git/iptables/+merge/371575 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to iptables in Ubuntu. https://bugs.launchpad.net/bugs/1840633

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Christian Ehrhardt 
As expected the upstream patch works even better. It gets -N and -L working. Most (but not all) the time this might be an issue of the build permissions. As now iptables works (correctly) with sudo but not without. The former hang also only happened with sudo. Maybe the build in the autopkgtest

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Christian Ehrhardt 
I have made good progress on the instance that I got shared. I have a fix for the busy loop, but while that will be helpful it isn't avoiding the root-cause and the system will be broken in regard to iptables. I have submitted an RFC with a long cover letter explaining the situation to upstream

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-21 Thread Christian Ehrhardt 
I have got a reply from upstream, since recently there is a similar fix which isn't in 1.8.3 I'll update my suggested fix to this one and see how the behavior will be with that one. => https://git.netfilter.org/iptables/commit/?id=e5cab728c40be88c541f68e4601d39178c36111f -- You received this b

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-20 Thread Christian Ehrhardt 
@jdstrand - agreed, but we might need/want to mask the test in UFW thou as the fix in iptables might need some time. The problem is that I can either make it skip the single test (need to check if then later on things fail) OR make it a force-badtest which would be sad as we'd loose so many othe

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-20 Thread Jamie Strandboge
Thanks for chasing this down! It seems clear that while the ufw autopkgtest found the issue, the bug is not in ufw. ** Changed in: ufw (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ipt

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-20 Thread Christian Ehrhardt 
A new -N call hangs as well, so I can debug "into" the failure Here is a strace from start into the looping call https://paste.ubuntu.com/p/YTQStdg6vm/ -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to iptables in Ubuntu. https

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-20 Thread Christian Ehrhardt 
It is this function our busy loop is in: 1581 static void __nft_build_cache(struct nft_handle *h) 1582 { 1583 »···uint32_t genid_start, genid_stop;

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-20 Thread Christian Ehrhardt 
Backtrce while in the loop (gdb) bt #0 0x77b120b82790 in mnl_cb_run () from /lib/powerpc64le-linux-gnu/libmnl.so.0 #1 0x013b1172e2a8 in mnl_talk (h=0x7fffd658cad0, nlh=, cb=0x13b11729570 , data=0x7fffd658c5e4) at nft.c:106 #2 0x013b1172e530 in mnl_genid_get (genid=0x7fffd658c5e4,

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
Since iptables seems to hang it might be more a bug in there, but I fail to recreate the case for further analysis no matter what I tried so far :-/ Adding iptables bug task and subscribing jdstrand in case he has another idea due to his experience in that area. ** Also affects: iptables (Ubuntu)

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
Note from my debug builds: TDEBUG: check OSError GNC: Enter get_netfilter_capabilities GNC: we are root GNC: chain ufw-caps-testSqLa5Y GNC: exe /sbin/iptables That means it is the first action of the test, just running /sbin/iptables -N ufw-caps-testSqLa5Y Nothing else of the python stack was r

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
I discussed if maybe it is part of the special net rules "autopkgtest@lgw01-14.secgroup" that I saw in the command - but according to Laney/Juliank those are just copies of the default group for scaling reasons. -- You received this bug notification because you are a member of Ubuntu Touch seeded

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
Original run command would be: $ #/home/ubuntu/autopkgtest/runner/autopkgtest --output-dir /tmp/autopkgtest-work.zn3q26vh/out --timeout-copy=6000 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-production/setup-canonical.sh --setup-commands /home/ubuntu/autopkgtest/setup-commands/s

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
But the ps output already has identified a hang inside of iptables itself hanging. The commandline identified it coming from 767 # First install a test chain 768 (rc, out) = cmd([exe, '-N', chain])

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
I ran a test on bileto ticket [1] with some extended debug output. A good run would look like: test_get_netfilter_capabilities (tests.unit.test_util.UtilTestCase) Test get_netfilter_capabilities() ...·

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
Checking the hanging process shows: https://paste.ubuntu.com/p/5Z745G963k/ <- that repeating endlessly And wchan is "0" which means it is really busy That is: sendto(4, {{len=20, type=NFNL_SUBSYS_NFTABLES<<8|NFT_MSG_GETGEN, flags=NLM_F_REQUEST, seq=0, pid=0}, {nfgen_family=AF_UNSPEC, version=NF

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
Thanks to Laney I now know that iptables is actually hanging in what seems to be a busy loop: ubuntu 717 0.0 0.7 22976 11968 ?S08:36 0:00 | \_ sshd: ubuntu@notty root 3715 0.0 0.6 16064 9536 ?Ss 08:36 0:00 | \_ sudo -n /tmp/autopkgtest-run-wrap

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
There were a bunch of mentions of that function in bug 1044361 bug 1039729 and bug 1062521. And while the issues back then are in the code since version 34 we might again face something that is special about the network environment that is present only in the autopkgtest-env for "build-needed" b

[Touch-packages] [Bug 1840633] Re: autopkgtests get stuck in Eoan with iptables 1.8.3

2019-08-19 Thread Christian Ehrhardt 
This ordering is interesting (from a good case): $ grep -e 'autopkgtest .*\[.*\]: test .*:' -e get_netfilter_capabilities old-iptables-good.txt test_get_netfilter_capabilities (tests.unit.test_util.UtilTestCase) Test get_netfilter_capabilities() ... ok test_get_netfilter_capabilities (tests.uni