** No longer affects: linux-azure (Ubuntu)
** No longer affects: systemd (Ubuntu)
** Also affects: cloud-utils (Ubuntu Eoan)
Importance: Undecided
Status: New
** Also affects: cloud-initramfs-tools (Ubuntu Eoan)
Importance: Undecided
Status: New
--
You received this bug not
Yes, we see it on Eoan and blocking tests, so if you could SRU it there,
it'd be much appreciated. We have never seen it on Bionic to my
knowledge.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bu
Great, thanks for confirming.
I wonder if we should SRU this. You see it on Eoan I presume? The bug is
theoretically present in Bionic too aiui but if it never crops up I
don't know if it's worth it.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages,
Sorry, we were waiting on a bug regarding dictionary keys being modified
to resolve before we could verify this. We have not seen this in the
past couple of runs in our testing, so I think we're good to proceed.
--
You received this bug notification because you are a member of Ubuntu
Touch seede
Can someone from CPC confirm that this has fixed the issues in the Azure
tests?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with
This bug was fixed in the package cloud-utils -
0.31-7-gd99b2d76-0ubuntu1
---
cloud-utils (0.31-7-gd99b2d76-0ubuntu1) focal; urgency=medium
* New upstream snapshot.
- growpart: add flock support to prevent udev races (LP: #1834875)
-- Ryan Harper Tue, 25 Feb 2020 14:41:21
-0
This bug was fixed in the package cloud-initramfs-tools - 0.45ubuntu1
---
cloud-initramfs-tools (0.45ubuntu1) focal; urgency=medium
* Add dependency on flock for growroot's use of growpart.
(LP: #1834875)
-- Scott Moser Tue, 25 Feb 2020 13:08:22 -0500
** Changed in: cloud-i
On Tue, Feb 25, 2020 at 2:35 PM Scott Moser
wrote:
> this seemed to "just work" for me.
> http://paste.ubuntu.com/p/93dWDPZfZT/
Ah, I didn't check that there was an existing ubuntu/devel branch. Sorry.
I've pushed a MR here:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/
** Merge proposal linked:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379851
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
this seemed to "just work" for me.
http://paste.ubuntu.com/p/93dWDPZfZT/
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
** Patch added: "debdiff showing the changes to upload to fix the bug."
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5330894/+files/cloud-utils_0.31-6_to_0.31-7.debdiff
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is s
** Attachment added: "tarball of source package to upload"
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5330895/+files/cloud-utils_0.31-7-gd99b2d76-source.tar.xz
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscrib
@Scott,
cloud-utils isn't quite new-upstream-snapshot out of the box; the debian
dir does not contain the changelog; however, I think I've got this
sorted out. I've a MP I can put up; but it only will show the add of
the changelog file. I'll attach a debdiff and a source package.
--
You recei
a.) I gave the wrong link. ugh. It should have been:
https://code.launchpad.net/~smoser/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/379774
b.) the fixed link to 'a' probably makes more sense now. But basically
you need a newer cloud-initramfs-tools to adjust for the fact that
growp
@smoser
a looks merged
b has not had any changes since 2018 and looks up-to-date in Focal. What
am I missing?
c I see a devel branch, is it assumed that it uses a packaging structure
similar to cloud-init or can someone just grab and upload master?
--
You received this bug notification because
** Merge proposal linked:
https://code.launchpad.net/~smoser/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/379774
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs
The fix is in cloud-utils upstream now.
Still to do:
a.) review/merge cloud-initramfs-tools pull request
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379177
b.) upload cloud-initramfs-tools to focal
c.) upload cloud-utils to focal
d.) any SRU
the order of 'b' a
** Changed in: cloud-initramfs-tools (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with u
** Merge proposal linked:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379177
** Changed in: cloud-utils
Status: New => Fix Committed
** Also affects: cloud-utils (Ubuntu)
Importance: Undecided
Status: New
** Changed in: cloud-utils (Ubuntu)
Here's the upstream changes to growpart I'm suggesting:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-
utils/+merge/379177
I've also proposed on modifications to cloud-init's cc_growpart as a further
method
to aid debugging if this hit as well as some mitigation around the race.
h
** Merge proposal linked:
https://code.launchpad.net/~raharper/ubuntu/+source/cloud-utils/+git/cloud-utils/+merge/378839
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/18
Yes, this is my read on the issue as well. The trigger is related to
the inotify watch that systemd-udevd puts on the disk. Something that
might help that we could try per xnox's comment around use of flock.
if growpart were to flock /dev/sda (we need to sort out what flags are
needed to prevent
> Ah, no I think this might be along right lines: udev is calling blkid
> on the _partition_ of course, so it can probe for filesystem etc without
> looking at the partition table. After it's done that, it does look for
> the partition table so it can read the ID_PART_ENTRY_* values from it,
> but
Adding in a passing version of journalctl
** Attachment added: "passing_journalctl_output.txt"
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5326024/+files/passing_journalctl_output.txt
--
You received this bug notification because you are a member of Ubuntu
Touch seeded pac
Cloud-init service starts and will run growpart, etc
Feb 06 00:37:26 ubuntu systemd[1]: Starting Initial cloud-init job
(pre-networking)...
Feb 06 00:37:37 test-xrdpdnvfctsofyygmzan systemd[1]: Starting Initial
cloud-init job (metadata service crawler)...
Something has modified sdb1 (growpart/s
I'm attaching the output of a failing build
** Attachment added: "failing_journalctl_output.txt"
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5325941/+files/failing_journalctl_output.txt
--
You received this bug notification because you are a member of Ubuntu
Touch seeded p
On another tangent, I wonder if the change that brought this to light
was the switch to booting without initrd. The timing is about right, and
fits with the fact that it doesn't occur with the generic kernel (which
cannot boot without initrd in Azure). So if someone has an excess of
time to test ,
I can get that going, and will report back on this bug when I have more
information.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race
Hi, could someone prepare an image with
https://paste.ubuntu.com/p/HYGv6m8gCc/ at /etc/systemd/system/systemd-
udevd.service.d/udevd-debugging.conf, boot it on azure until it fails
and put the journalctl output (probably a few megs) somewhere I can read
it? Output from a successful boot would also
Oh yeah and one other thing I don't understand: why udev is processing
the partition while sgdisk is still running.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Ti
Ah, no I think this might be along right lines: udev is calling blkid on
the _partition_ of course, so it can probe for filesystem etc without
looking at the partition table. After it's done that, it does look for
the partition table so it can read the ID_PART_ENTRY_* values from it,
but if it fail
So I've been handed the job of moving this bug forward this iteration.
Having just re-read all the comments again and read some source code
while listening to loud techno music my observations/questions are
these:
1) this is a doozy
2) I would really like to see complete udevadm monitor and inot
On Thu, 7 Nov 2019 at 20:05, Scott Moser wrote:
>
> > > So that means we have this sequence of events:
> > > a.) growpart change partition table
> > > b.) growpart call partx
> > > c.) udev created and events being processed
>
> > That is not true. whilst sfdisk is deleting, creating, finishing
On Thu, 7 Nov 2019 at 17:50, Ryan Harper <1834...@bugs.launchpad.net> wrote:
>
> @ddstreet
>
> Yes, settle does not help.
>
> Re-triggering udevadm trigger --action=add /sys/class/block/sda
>
> Would re-run all of them after the partition change has occurred, which
> is what I'm currently suggestin
> We can't know how many devices are in the system. It maybe nearly zero,
> it could be minutes.
I'm not suggesting you trigger uevents for all devices, just the one
you're repartitioning...
> The cold plug is required (initramfs may or maynot
> have ran
> scripts/udevd etc but kernel events alre
On Thu, Nov 7, 2019 at 2:41 PM Dan Streetman
wrote:
> > Issuing a second
> > trigger will repeat this.
> > IMO, that's a non-zero amount of time that slows the boot down, so I'd
> like
> > to avoid that.
>
> systemd-udev-trigger.serivce retriggers *everything* at boot (except in
>
an unprivileged
On Thu, Nov 7, 2019 at 11:30 AM Dimitri John Ledkov
wrote:
> > So that means we have this sequence of events:
> > a.) growpart change partition table
> > b.) growpart call partx
> > c.) udev created and events being processed
>
> That is not true. whilst sfdisk is deleting, creating, finishing
> Issuing a second
> trigger will repeat this.
> IMO, that's a non-zero amount of time that slows the boot down, so I'd like
> to avoid that.
systemd-udev-trigger.serivce retriggers *everything* at boot (except in
an unprivileged container where it can't), so I'm not sure how much
added time we're
> > So that means we have this sequence of events:
> > a.) growpart change partition table
> > b.) growpart call partx
> > c.) udev created and events being processed
> That is not true. whilst sfdisk is deleting, creating, finishing
> partition table (a) and partx is called (b), udev events ar
On Thu, Nov 7, 2019 at 1:30 PM Dan Streetman
wrote:
> > Yes, settle does not help.
>
> Well, I didn't suggest just to settle ;-)
>
Sorry; long bug thread.
> > I'm currently suggesting as a heavy-handed workaround.
>
> I don't really see why you think this is heavy-handed, but I must be
> missi
> Yes, settle does not help.
Well, I didn't suggest just to settle ;-)
> I'm currently suggesting as a heavy-handed workaround.
I don't really see why you think this is heavy-handed, but I must be
missing something.
> I would like to understand *why* the udevd/kernel pair exhibits this racy
>
@ddstreet
Yes, settle does not help.
Re-triggering udevadm trigger --action=add /sys/class/block/sda
Would re-run all of them after the partition change has occurred, which
is what I'm currently suggesting as a heavy-handed workaround.
I would like to understand *why* the udevd/kernel pair exhi
Just curious, did anyone actually test with my suggestion from comment
40? That is, just make your partition table change (and update the
kernel with partx or whatever, of course), and after it's done trigger
--settle udev for the device. Be interesting to know if that actually
fixes it or not.
> So that means we have this sequence of events:
> a.) growpart change partition table
> b.) growpart call partx
> c.) udev created and events being processed
That is not true. whilst sfdisk is deleting, creating, finishing
partition table (a) and partx is called (b), udev events are already
fi
I really think you are all *way* over thinking this.
a. growpart made a change to the partition table (using sfdisk)
b. growpart called partx --update --nr 3 /dev/sda
c. growpart exited
With a and b growpart created udev events. If you create udev events,
you really need to wait for those event
> it will prevent udevd from running the rules against it. Thus
effectively the event will be fired and done, but nothing actually
executed for it.
Interesting, I suspect this is the race we see. The events emitted but
no actions taken (ie we didn't get our by-partuuid symlink created.
> I someh
flock will not block inotify or udev events being emitted.
See
https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L322
https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L409
it will prevent udevd from running the rules against it. Thus
effectively the event will be fire
A couple of comments on the suggested path:
> Imho the sequency of commands should be:
> * take flock on the device, to neutralise udev
+1 on this approach. Do you know if the flock will block
systemd's inotify write watch on the block device which triggers
udevd? This is the typical race we se
@ Cloud init team, do we want to try changing cloud-utils to use a lock?
And like have a canary "only use locked codepath on this region" such
that we can assert through testing that this no longer happens with new
code, but does with old code.
--
You received this bug notification because you a
Some observations:
* growpart uses sfdisk without --no-tell-kernel option, meaning that it does
notify kernel about partition changes
* growpart later calls partx, which may be redundant / cause no changes or
events
* as a side note, partprobe, blockdev --rereadpt can also be used to reread
part
** Changed in: systemd (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status i
** Tags added: id-5dbb311b74e3f364d8f98c56
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status in cloud-init:
Incomp
> The lack of events is not the issue
I'm not saying it is. But you seem to have it under control; I was just
offering my opinion.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net
The lack of events is not the issue. We see all the events from the
kernel that we would expect to see during a resize, and we see the same
number (and type) of events on both the passing and the failing
instances.
The problem is that if udev processes the sda1 event first then, at
least some of
> You're proposing retriggering events and _then_ waiting
I'm proposing make sure that the uevent(s) for the specific partition
you're interested in have started processing before you start waiting,
which is what the retriggering does.
But it could be a kernel bug, sure.
--
You received this bu
I don't think we're looking for a way to wait for udev. For this bug to
present, udev has to have processed all of the available events (i.e.
there's nothing left to wait for), and have done so incorrectly
(probably as a result of its interaction with the Azure kernel, as this
doesn't reproduce on
I think that is the correct way to wait for udev to process a specific
device's changes.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart
Dan, are you asking for that to get more debugging information, or are
you suggesting that as a fix for this?
(I'm yet to be convinced that this isn't a regression in (the
kernel|src:systemd), so I don't think that we should have to run _any_
additional commands to address this.)
--
You received
Pretty sure you need to actually try what I suggested in comment 40, but
of course use the specific dev/partition you're missing the udev symlink
for.
e.g. if you need the symlink for /dev/sda1 then instead of:
$ growpart /dev/sda 1 ; udevadm settle ; ls /dev/disk/by-uuid/$UUID
you should do
$
On Mon, Aug 26, 2019 at 08:58:46AM -, Tobias Koch wrote:
> > (Odds are that whatever causes it to be recreated later in boot would be
> > blocked by cloud-init waiting.)
>
> But that's not happening. The instance does boot normally, the only
> service degraded is cloud-init and there is no sig
On Mon, Aug 26, 2019 at 03:17:10PM -, Scott Moser wrote:
> If there is a race, or a need to wait, it almost certainly is in cloud-
> utils (growpart).
I would still like us to get a systemd/kernel person to take a look at
this, because I think that the race is somewhere further down than
cloud
If there is a race, or a need to wait, it almost certainly is in cloud-
utils (growpart). If you use growpart to grow a partition, the result
should be that that is done, and it is ready. The caller should not
expect that they have to know additional information (to call 'udevadm
settle') or shou
On Mon, Aug 26, 2019 at 4:05 AM Tobias Koch <1834...@bugs.launchpad.net>
wrote:
> > (Odds are that whatever causes it to be recreated later in boot would be
> > blocked by cloud-init waiting.)
>
> But that's not happening. The instance does boot normally, the only
> service degraded is cloud-init
** Also affects: cloud-utils
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
> (Odds are that whatever causes it to be recreated later in boot would be
> blocked by cloud-init waiting.)
But that's not happening. The instance does boot normally, the only
service degraded is cloud-init and there is no significant delay either.
So conversely, if I put a loop into cloud-init
On Fri, Aug 23, 2019 at 08:23:06PM -, Tobias Koch wrote:
> I may be missing the point, but the symlink in question is eventually
> recreated, does that tell us anything? This here
Yes, this is more supporting evidence that this is a race condition; the
state of the system both before and some
I may be missing the point, but the symlink in question is eventually
recreated, does that tell us anything? This here
> Dan had put a udevadm settle in this spot like so
>
> def get_size(filename)
>util.subp(['udevadm', 'settle'])
>os.open()
looks to me like the event queue should be
On Fri, Aug 23, 2019 at 10:21 AM Dan Watkins
wrote:
> Looking at the comment timestamps, Dan probably didn't see my comment,
> but to reiterate: all the events we expect to be processed _are_
> processed, the issue is that when they are processed they don't always
> end up with the correct partit
On Fri, Aug 23, 2019 at 10:00 AM Dan Streetman
wrote:
> > Dan had put a udevadm settle in this spot like so
> >
> > def get_size(filename)
> >util.subp(['udevadm', 'settle'])
> >os.open()
>
> if you know you've just changed (e.g.) /dev/sda, possibly its kernel-
> generated udev events
Looking at the comment timestamps, Dan probably didn't see my comment,
but to reiterate: all the events we expect to be processed _are_
processed, the issue is that when they are processed they don't always
end up with the correct partition information.
--
You received this bug notification becau
> Dan had put a udevadm settle in this spot like so
>
> def get_size(filename)
>util.subp(['udevadm', 'settle'])
>os.open()
if you know you've just changed (e.g.) /dev/sda, possibly its kernel-
generated udev events just haven't reached udev yet, so the immediate
call to 'udev settle'
[N.B. I wrote the below before I saw Ryan's comment, so there is some
repetition.]
OK, I've spent some time catching up on this properly so I can
summarise: per comment #24, the issue is that when udev processes the
events emitted by the kernel, it (sometimes) doesn't determine the
correct partiti
The sequence is:
exec growpart
exec sgdisk --info # read-only
exec sgdisk --pretend # read-only
exec sgdisk --backup # read-only copy
# modification of disk starts
exec sgdisk --move-second-header \
--delete=PART \
--new=PART \
--typecode --pa
** Changed in: cloud-init
Status: Invalid => New
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status in cloud-i
@daniel-thewatkins, I'm not convinced that this bug is invalid for
cloud-init. After reading through all of this again, I still don't
understand, what guarantee there is that when `udevadm settle` is
called, all relevant events have already been queued.
Copying udevadm over, and with that suppress
So, to reset on status: I believe that we've narrowed this down to a
systemd/udev problem, as it goes away when a different udevadm binary is
used. I don't have a good sense of how to go about digging into that
issue, so I don't think I can take it much further.
(Also, as it appears to be a syste
When I say unorthodox, I mean I copied the binary ;) So that should
narrow it down quite a bit. Unless there is more funniness involved.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad
That certainly is unorthodox! When you say "copied udevadm", what
specifically do you mean? Just the binary, the udev package, ...?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.ne
Oh, good observation, Dan. Yes, it was happening on Cosmic and later.
But I cannot say when it started. Now in a moment of despair I tried
something very unorthodox: I copied udevadm from Bionic to the Eoan
image and ran the tests again. Look and behold, all pass.
** No longer affects: linux-azure
79 matches
Mail list logo
