[Bug 1652348] Re: initrd dhcp fails / ignores valid response

This bug was fixed in the package klibc - 2.0.3-0ubuntu1.14.04.3 --- klibc (2.0.3-0ubuntu1.14.04.3) trusty; urgency=medium * debian/patches/dhcp-one-socket-per-interface.patch: Use separate sockets for DHCP from multiple interfaces. Thanks to Jay Vosburgh

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

The trusty patch, 2.0.3-0ubuntu1.14.04.3 in -proposed, passed testing. ** Tags added: verification-done-trusty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails /

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

This bug was fixed in the package klibc - 2.0.4-8ubuntu3.1 --- klibc (2.0.4-8ubuntu3.1) yakkety; urgency=medium * debian/patches/dhcp-one-socket-per-interface.patch: Use separate sockets for DHCP from multiple interfaces. Thanks to Jay Vosburgh

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Verification-done-yakkety: I've verified that ipconfig can properly get an IP address via DHCP when there are multiple interfaces. I verified klibc 2.0.3-0ubuntu1.14.04.3. ** Tags added: verification-done-yakkety -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Hello Paul, or anyone else affected, Accepted klibc into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/klibc/2.0.3-0ubuntu1.14.04.3 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Also affects: klibc (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: klibc (Ubuntu Trusty) Assignee: (unassigned) => Mathieu Trudel-Lapierre (cyphermox) ** Changed in: klibc (Ubuntu Trusty) Status: New => In Progress ** Changed in: klibc (Ubuntu Trusty)

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Also affects: klibc (Ubuntu Zesty) Importance: Undecided Status: New ** Changed in: klibc (Ubuntu Zesty) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

That would be a different bug, unfortunately. Mine was specifically down to ipconfig not handling multiple network interfaces correctly, triaged and successfully fixed by Canonical, and exhaustively validated in our infrastructure. Roughly speaking, it would quickly loop through the interfaces

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Note: I've also discovered (unsurprisingly, I guess?) that I see the exact same behaviour on Ubuntu 14.04. Can this fix be backported? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

This bug was fixed in the package klibc - 2.0.4-8ubuntu1.16.04.3 --- klibc (2.0.4-8ubuntu1.16.04.3) xenial; urgency=medium * debian/patches/dhcp-one-socket-per-interface.patch: Use separate sockets for DHCP from multiple interfaces. Thanks to Jay Vosburgh

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about this bug go to:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I've tested and confirmed that this solves the issue on 16.04 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Hello Paul, or anyone else affected, Accepted klibc into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/klibc/2.0.4-8ubuntu3.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

Re: [Bug 1652348] Re: initrd dhcp fails / ignores valid response

On Tue, Jan 24, 2017 at 10:51:56PM -, Paul Graydon wrote: > Given this is a fundamental bug in klibc, is there a plan to try to > upstream this patch? It has been upstreamed to the Debian git repository. Given that we will be looking at removing klibc from the initramfs altogether in future

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Given this is a fundamental bug in klibc, is there a plan to try to upstream this patch? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

This bug was fixed in the package klibc - 2.0.4-8ubuntu4 --- klibc (2.0.4-8ubuntu4) zesty; urgency=medium * debian/patches/dhcp-one-socket-per-interface.patch: Use separate sockets for DHCP from multiple interfaces. Thanks to Jay Vosburgh .

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Description changed: - Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been - (re?)introduced that is breaking dhcp booting in the initrd environment. - This is stopping instances that use iscsi storage from being able to - connect. + [SRU justification] + Changes to ordering of kernel

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

The attachment "klibc-fix-1.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Patch proposal to modify ipconfig to use one packet socket per interface ** Patch added: "klibc-fix-1.patch" https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+attachment/4806861/+files/klibc-fix-1.patch -- You received this bug notification because you are a member of Ubuntu

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Changed in: klibc (Ubuntu) Status: Confirmed => In Progress ** Tags removed: kernel-bug-exists-upstream kernel-bug-exists- upstream-4.10-rc1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I have instrumented ipconfig, and determined that the ultimate source of the problem is that, for the case of multiple interfaces, ipconfig has a dependency on the kernel's probe order of the network interfaces. For whatever reason, the -31 kernel probes the network devices in one order (e.g.,

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I have reproduced the described issue locally using the instructions from comment 35; will start looking into the cause. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Changed in: linux (Ubuntu) Status: Incomplete => Triaged ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Xenial) Importance: Undecided => High ** Changed in: linux (Ubuntu Xenial) Status: New => Triaged -- You

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Just a note that I'm setting up to try the reproduction instructions from comment #35 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Tags removed: kernel-da-key ** Tags added: kernel-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about this

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul, thank you for the recreate instructions. This will help the support team immensely. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I took a step back from doing bisecting and focussed on creating a replication scenario, which I've done successfully. ipconfig is struggling to handle things when two interfaces are present and sending out DHCP requests, even if one interface doesn't get a response. Here's what I've done:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about this bug go to:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I'm continuing to bisect the mainline linux kernel, and also trying to see if I can create a straightforward reproducible example. First focus on bisecting was between 4.5 and 4.6, to figure out what changed to suddenly have ipconfig working. I've tracked it down to this using bisect, and

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I've tried every version in the v4 series, and a few in v3. None prior to (and including) v4.0.0 will boot, none output anything on the screen to give me a clue why they're not booting. So far: v4.0 = won't boot v4.1 = ipconfig bug v4.2 = ipconfig bug v4.3 = ipconfig bug v4.4 = ipconfig bug

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, thanks for the clarification. Paraphrasing Linus, "We don't break userspace!" So, kernel bits being flipped causing userspace issues would be considered, at least for now, a kernel issue. Despite this, the Ubuntu kernel commit bisect results are helpful here on Launchpad. However,

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

The more I look at this, the more I'm convinced *most* of the real problem lies in that ipconfig tool. Yes, various kernel changes seem to make it alter between working & not working under the circumstances (which is bizarre), but unless something is specifically interfering with the

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

My apologies for any lack of clarity. I tested against the head of ubuntu-xenial, reverting just that commit and it fixed it. I tested against the head of the mainstream kernel and it didn't (last night I tried 4.9, 4.8, 4.5, 4.4, 4.2 tags of the mainstream kernel and in every place I find the

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, you advised in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1652348/comments/26 reverting the commit worked consistently, but now in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1652348/comments/28 you are saying the opposite. Could you please clarify? ** Changed

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I tried reverting that specific commit from upstream, but that didn't resolve the issue. Time for a new round of bisecting the kernel, this time using mainline. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, I wouldn't get too hung up on what appears a non-related code change affecting DHCP. Honestly, this result isn't surprising given how kernel code is more inter-related than meets the eye. Despite this, the issue you are reporting is an upstream one. Could you please report this

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Tags removed: needs-reverse-bisect ** Tags added: reverse-bisect-done ** Tags added: cherry-pick ** Tags removed: cherry-pick kernel-fixed-upstream kernel-fixed-upstream-4.10-rc1 reverse-bisect-done ** Tags added: bisect-done kernel-bug-exists-upstream kernel-bug-exists-upstream-4.10-rc1

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

This seems to make no sense to me, as a layman anyway. I checked out the 4.4.0-58.79 tag, reverted that one commit and confirmed I have a booting 4.4.0-58-generic that'll happily DHCP in the initrd environment on multiple boots. It really does seem like, somehow, that commit is the source of the

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I bisected again, and again it came back to that mount point change. This seems so bizarre. $ git bisect log # bad: [6d4f0a79e5a307b6fd3ee3cc5bbb2fcb701b09db] UBUNTU: Ubuntu-4.4.0-57.78 # good: [db5f146d309e70067dae57798c9ea679af835aa7] UBUNTU: Ubuntu-4.4.0-53.74 git bisect start

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I see where I messed up.. I'll try the bisect again. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about this bug

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Okay... I can't help but think I made a mistake somewhere in the bisecting process, but it seems to have isolated fd4b5fa6e3487d15ede746f92601af008b2abbc0 as the bad commit $ git bisect log # bad: [6d4f0a79e5a307b6fd3ee3cc5bbb2fcb701b09db] UBUNTU: Ubuntu-4.4.0-57.78 # good:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I'll take a fresh look in the morning, but ran into this: make[1]: Leaving directory '/home/ubuntu/storage/ubuntu-xenial/debian/build/build-generic/zfs/module' Debug: module-check-generic install -d /home/ubuntu/storage/ubuntu-xenial/debian.master/abi/4.4.0-54.76/amd64 find

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I can give that a shot, following the instructions here: https://wiki.ubuntu.com/Kernel/KernelBisection#Bisecting_Ubuntu_kernel_versions -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, if the bug is reproducible at any interval, then perhaps a standard bisect between: linux-image-4.4.0-53-generic - Fine linux-image-4.4.0-57-generic - Affected would be more appropriate to understand which commit introduced the regression. -- You received this bug notification

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I should clarify, I know for certain that 4.4.0-51 is stable and reliable (and doesn't exhibit the bug). As part of our attempt to verify everything was correct with the installation we had a system run from Wednesday before Thanksgiving, all the way through to the following Monday, during which

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Okay.. this is interesting. It seems like the Ubuntu dev version of 4.10 is actually intermittently failing (?!) I guess the next thing to do here is keep rebooting on this version of the kernel and see how often the bug occurs vs doesn't occur, so I can get a feel for a reasonable number of

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, to advise, I have updated the article to move the mainline-build-one section out of the way, as it has been distracting here, and for other folks. Feel free to ignore it, as it is for those who build kernels all the time (i.e. N/A here). Also, I don't maintain it, so I won't be able

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Gah.. okay https://wiki.ubuntu.com/KernelTeam/GitKernelBuild -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Rolling that command against master fails too: ubuntu@Beta:~/linux$ mainline-build-one afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc xenial *** BUILDING: commit:afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc series:xenial abinum: ... full_version<4.4.0> version<4.4.0> long abinum<040400> fatal: 'xenial'

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Ahh, I see where the kteam tools stuff is supposed to come from. It's not clear if I'm supposed to go down that route and use the mainline-build-one script or not when trying to build the kernel in this case. If I use the mainline-build-one tool: $ mainline-build-one

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I'll get started on it. This might take a while to do. A couple of quick observations: 1) we haven't validated that mainline 4.4.0 actually works. I only know certain Ubuntu versions of the 4.4.0 kernel work. Given how much seems to be changing between Ubuntu releases of it, that seems a risky

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, the next step is to fully reverse commit bisect from kernel 4.4 to 4.10-rc1 in order to identify the last bad commit, followed immediately by the first good one. Once this good commit has been identified, it may be reviewed for backporting. Could you please do this following

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Tried and tested (the current up-to-date kernels at the time of posting): http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc1/linux- headers-4.10.0-041000rc1-generic_4.10.0-041000rc1.201612252031_amd64.deb http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc1/linux-

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

Paul Graydon, thank you for reporting this and helping make Ubuntu better. In order to allow additional upstream developers to examine the issue, at your earliest convenience, could you please test the latest upstream kernel available from http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I've also confirmed the bug is present all the way back in 4.4.0-21-generic, and is present in 4.8.0-34-generic from yakkety- proposed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

I've worked my way back through the kernels. The bug, as it was (avoided by ip=dhcp in the kernel command line), was in effect in version 4.4.0-38-generic. It was fixed in 4.4.0-42-generic. This is the state of play so far with kernels I've tested: linux-image-4.4.0-38-generic - Affected

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

apport-collect doesn't exist in initrd. I'm unable to supply the requested information. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Attachment added: "pcap from dhcp server side of inird startup doing dhcp" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1652348/+attachment/4795820/+files/failed.pcap -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

The checksum invalid mentioned in the pcap is interesting, but happens in both failed and successful, so I'm not sure it's relevant. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title:

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Attachment added: "pcap from dhcp server side of 'ipconfig -t "dhcp" -d "ens2f0" '" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1652348/+attachment/4795819/+files/worked.pcap -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Attachment added: "Working 4.4.0-53 initrd" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1652348/+attachment/4795794/+files/initrd.img-4.4.0-53-generic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Attachment added: "4.4.0-57 "broken" initrd" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1652348/+attachment/4795793/+files/initrd.img-4.4.0-57-generic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1652348] Re: initrd dhcp fails / ignores valid response

** Package changed: linux-meta (Ubuntu) => linux (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1652348 Title: initrd dhcp fails / ignores valid response To manage notifications about this