Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Michael Hudson-Doyle]

On Fri, 3 May 2024 at 03:05, Heinrich Schuchardt <
heinrich.schucha...@canonical.com> wrote:

> On 02.05.24 16:46, Robie Basak wrote:
> > On Thu, May 02, 2024 at 04:05:31PM +0200, Heinrich Schuchardt wrote:
> >> Often I see apt-get update downloads exceeding 100 MiB. That is without
> a
> >> single package download.
> >
> > I think it might be worth quantifying this. Right now, for amd64
> > proposed pocket Packages.xz files for the following:
>
> This is what I see:
>
> $ sudo apt-get update
> Get:1 http://archive.ubuntu.com/ubuntu oracular InRelease [64,6 kB]
> Hit:2 http://dl.google.com/linux/chrome/deb stable InRelease
>
>
> Hit:3 https://ppa.launchpadcontent.net/xypron/qemu/ubuntu noble
> InRelease
>
> Hit:4 https://ppa.launchpadcontent.net/xypron/rsyslog2312/ubuntu noble
> InRelease
>
> Get:5 http://security.ubuntu.com/ubuntu oracular-security InRelease
> [64,6 kB]
> Hit:6
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/private/ubuntu
> mantic InRelease
> Get:7 http://archive.ubuntu.com/ubuntu oracular-updates InRelease [64,6
> kB]
> Get:8 http://archive.ubuntu.com/ubuntu oracular-backports InRelease
> [64,7 kB]
> Hit:9
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/private/ubuntu
> noble InRelease
> Get:10 http://archive.ubuntu.com/ubuntu oracular/multiverse Sources [299
> kB]
> Hit:11
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/ventana/ubuntu
> noble InRelease
> Hit:12
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/ventana/ubuntu
> mantic InRelease
> Get:13 http://archive.ubuntu.com/ubuntu oracular/restricted Sources
> [18,7 kB]
> Get:14 http://archive.ubuntu.com/ubuntu oracular/universe Sources [19,9
> MB]
> Hit:15
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team/private/ubuntu
> mantic InRelease
> Get:16 http://archive.ubuntu.com/ubuntu oracular/main Sources [1.378 kB]
> Get:17 http://archive.ubuntu.com/ubuntu oracular/main amd64 Packages
> [1.401 kB]
> Get:18 http://archive.ubuntu.com/ubuntu oracular/main i386 Packages
> [1.041 kB]
> Get:19 http://archive.ubuntu.com/ubuntu oracular/main Translation-en
> [512 kB]
> Get:20 http://archive.ubuntu.com/ubuntu oracular amd64 Contents (deb)
> [51,3 MB]
> Get:21 http://archive.ubuntu.com/ubuntu oracular i386 Contents (deb)
> [40,3 MB]
> Get:22 http://archive.ubuntu.com/ubuntu oracular/restricted i386
> Packages [14,7 kB]
> Get:23 http://archive.ubuntu.com/ubuntu oracular/restricted amd64
> Packages [93,9 kB]
> Get:24 http://archive.ubuntu.com/ubuntu oracular/restricted
> Translation-en [18,7 kB]
> Get:25 http://archive.ubuntu.com/ubuntu oracular/universe amd64 Packages
> [15,5 MB]
> Get:26 http://archive.ubuntu.com/ubuntu oracular/universe i386 Packages
> [8.166 kB]
> Get:27 http://archive.ubuntu.com/ubuntu oracular/universe Translation-en
> [5.980 kB]
> Get:28 http://archive.ubuntu.com/ubuntu oracular/multiverse amd64
> Packages [269 kB]
> Get:29 http://archive.ubuntu.com/ubuntu oracular/multiverse i386
> Packages [126 kB]
> Get:30 http://archive.ubuntu.com/ubuntu oracular/multiverse
> Translation-en [118 kB]
> Fetched 147 MB in 10s (14,3 MB/s)
>
>
> Reading package lists... Done
>

Yes sure but that's not the common experience for at least three reasons:

 1. it's the devel series so the release pocket gets republished all the
time
 2. you have apt-file installed and are downloading the Contents files.
those are always big
 3. you have deb-src enabled (this makes much less difference than the
previous 2 though)

If we want to make apt update quicker / lighter on resources we should
figure out if we can stop publishing some of the hashes (which entirely
dominate the size of the compressed package lists). We currently have 4
hashes in the lists (md5, sha1, sha256, sha512) -- I know Dimitri was
trying to get us to the point that we could stop publishing MD5 at least
but there are a few things out there that hardcode a dependence on it.
Maybe oracular is a good time to turn off some hashes and see what breaks.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: pastebinit default target on Ubuntu

[Michael Hudson-Doyle]

On Tue, 16 Apr 2024 at 14:37, Steve Langasek 
wrote:

> On Mon, Apr 15, 2024 at 04:42:37PM -0400, Stéphane Graber wrote:
> > > And if there are issues with the usability of paste.ubuntu.com, uh,
> we own
> > > that service?  So let's work with our IS team to make it fit for
> purpose.
> > > (I don't know why it currently requires a login to *view* paste
> contents;
> > > that seems straightforwardly a bug that we should just get sorted.)
>
> > That's because pastebin servers are frequently abused as a way to get
> > free mass storage.
>
> > It's not very practical to require login to post to a pastebin as the
> > whole point is for a tool like "pastebinit" to work without needing
> > user configuration as it's commonly used as a debug tool on cloud
> > instances and other random servers random than a user's personal
> > system.
>
> > With that in mind, a bunch of folks noticed that you could abuse a
> > service like paste.ubuntu.com by pushing large files (base64 encoded
> > or the like) and then retrieve them with a very trivial amount of html
> > parsing (if no raw option is offered directly).
>
> > There are obviously alternatives to this, but they tend to require a
> > bunch more server side logic, basically trying to find the right set
> > of restrictions to both poster and reader so that legitimate users can
> > use the service normally while abusers get sufficiently annoyed to
> > stay away from it.
>
> The current behavior of paste.ubuntu.com, and what I assumed was the
> driver
> for moving away from this as a default, was that it requires a login to
> VIEW
> the contents of the pastebin.  AFAICS this is not justifiable on the basis
> of preventing abuse with illicit/illegal pastes, that's already addressed
> by
> requiring login on the submission side.
>

I think the current behaviour is to require login for at least one of
submission or view, so a paste created while logged in can be viewed
anonymously and a paste created anonymously (e.g. by pastebinit, which I
don't think supports logging in?) requires a login to view.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Bumping apt RSA key length requirements to 3072-bit (2048 w/ warning) for 24.04

[Michael Hudson-Doyle]

On Wed, 24 Jan 2024 at 20:48, Adrien Nader  wrote:

> On Wed, Jan 24, 2024, Michael Hudson-Doyle wrote:
> > On Tue, 23 Jan 2024 at 02:31, Jeremy Bícha 
> > wrote:
> >
> > > On Mon, Jan 22, 2024 at 7:36 AM Dimitri John Ledkov
> > >  wrote:
> > > > > Sadly shipping this in 24.04 means that PPAs owned by user
> > > > > accounts created prior to 2014-03-11[3] until the key rotation
> > > > > mechanism(s) [4][5] have been implemented.
> > > > >
> > > >
> > > > I do wonder how many active old PPA owners remain in action.
> > > >
> > > > And if we can reset per-series signing keys on all of those for any
> > > > new PPAs, and noble series (meaning single signe, new key for
> noble+).
> > > >
> > > > I have personally created a new team for myself, only added myself to
> > > > be a member of said team, to gain access to PPAs signed with 4k RSA
> > > > key, as I can no longer use my own ppas. I guess I should ask to
> > > > delete them all, and request removal of the signing key to gain back
> > > > personal PPAs with 4k signing key.
> > >
> > > Many of Ubuntu's core teams are older than 2014. This includes
> > > Desktop, Checkbox, Kernel, Pythoneers, Security, Mozilla, LibreOffice,
> > > Kubuntu, Lubuntu.
> > >
> > > I suspect that this change would break most of the heaviest used PPAs.
> > > We need a coordinated transition.
> > >
> >
> > I agree with Jeremy that we can't just blithely assume all PPAs created
> > before 2014 are no longer much used.
> >
> > Unfortunately I don't know what that means for a way forward. Clearly
> 1024R
> > keys should be retired. From one angle, I can imagine a scheme were a
> repo
> > is dual-signed and signs the new key with the old to convince apt to
> update
> > it but from another this seems impossible (and clearly very unlikely to
> > land before noble GA).
>
> We know of at least one active PPA with a 1024-bit key:
> https://launchpad.net/~videolan/+archive/ubuntu/master-daily .
>

I kind of misspoke in a way -- it's not the PPA that has to be old, as all
PPAs from a given owner are signed with the same key. It sounds slightly
more tractable to have Launchpad generate new keys for each owner and sign
new PPAs with that key. But a) not in time for noble b) this doesn't really
solve the problem anyway.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Bumping apt RSA key length requirements to 3072-bit (2048 w/ warning) for 24.04

[Michael Hudson-Doyle]

On Tue, 23 Jan 2024 at 02:31, Jeremy Bícha 
wrote:

> On Mon, Jan 22, 2024 at 7:36 AM Dimitri John Ledkov
>  wrote:
> > > Sadly shipping this in 24.04 means that PPAs owned by user
> > > accounts created prior to 2014-03-11[3] until the key rotation
> > > mechanism(s) [4][5] have been implemented.
> > >
> >
> > I do wonder how many active old PPA owners remain in action.
> >
> > And if we can reset per-series signing keys on all of those for any
> > new PPAs, and noble series (meaning single signe, new key for noble+).
> >
> > I have personally created a new team for myself, only added myself to
> > be a member of said team, to gain access to PPAs signed with 4k RSA
> > key, as I can no longer use my own ppas. I guess I should ask to
> > delete them all, and request removal of the signing key to gain back
> > personal PPAs with 4k signing key.
>
> Many of Ubuntu's core teams are older than 2014. This includes
> Desktop, Checkbox, Kernel, Pythoneers, Security, Mozilla, LibreOffice,
> Kubuntu, Lubuntu.
>
> I suspect that this change would break most of the heaviest used PPAs.
> We need a coordinated transition.
>

I agree with Jeremy that we can't just blithely assume all PPAs created
before 2014 are no longer much used.

Unfortunately I don't know what that means for a way forward. Clearly 1024R
keys should be retired. From one angle, I can imagine a scheme were a repo
is dual-signed and signs the new key with the old to convince apt to update
it but from another this seems impossible (and clearly very unlikely to
land before noble GA).

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu iso

[Michael Hudson-Doyle]

On Mon, 20 Nov 2023 at 07:14, Jerry Geis  wrote:

> I am remastering the DVD. No issues there.
>
> Is there a way to remove the /pool directory on the ISO?
> When I do that, remaster, and use the iso to boot, I get errors - about
> not finding packages.
>
> I "desire" to install "everything" over the internet-  I want to remove
> the /pool directory to save space...  How can I do that and let the
> installer be happy ?
>

 With the current installers there needs to be a package repository on the
ISO but it could be empty I suppose, something that can be arranged with a
hint of apt-ftparchive?

Cheers,
mwh
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: new seeds feature: alternatives in germinate 2.4.2

[Michael Hudson-Doyle]

On Fri, 1 Sept 2023 at 04:23, Julian Andres Klode <
julian.kl...@canonical.com> wrote:

> This means that only the first package ends up getting a Task
> field, and hence installation during image build - which uses
> the task feature of apt (apt install task-name^)


This isn't how image builds install tasks, for better or worse.

In lucid and earlier, the Task header is used but it's matched by hand
rather than using the ^ syntax -- I'm not sure why though, it may be
something to do with not wanting to install the packages that have the task
header but are from a foreign architecture.

In mantic though, livecd-rootfs uses the package lists produced by
germinate directly.

Given these package lists are what drive the Task header in the archive, I
expect it doesn't make any difference though.

Cheers,
mwh

is still as
> predictable as before - the resolving there doesn't change
> (task-name^ expands to all packages with Task: task-name).
>

>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Call for testing: 22.04.3 release candidate images ready!

[Michael Hudson-Doyle]

On Wed, 9 Aug 2023 at 02:18, Paride Legovini  wrote:

> Hello everyone,
>
> We just finished building our first set of 22.04.3 release candidate
> images, which are now available for testing from the ISO tracker:
>
> https://iso.qa.ubuntu.com/qatracker/milestones/447/builds
>
> Please pick your favorite flavor and start testing! And be sure to
> report your results on the ISO tracker above. Caveats:
>
> - A couple of risc-v images aren't ready yet (nezha, unmatched).
>   They should be appear in the ISO tracker within a few hours.
>
> - The ISO tracker link to the Ubuntu Desktop images does not point
>   to the right location. The correct location is:
>   https://cdimage.ubuntu.com/jammy/daily-live/20230807.2/
>
> As with every recent release, we have a discourse thread for tracking
> progress which we try to keep up to date as much as possible:
>
>
> https://discourse.ubuntu.com/t/focal-fossa-20-04-5-lts-point-release-status-tracking/29969


I think you meant
https://discourse.ubuntu.com/t/jammy-jellyfish-22-04-3-lts-point-release-status-tracking/37439
here :-)

Cheers
mwh


>
> --
> Paride
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Mantic QEMU, segfault in nested VMs and autopkgtest.u.c

[Michael Hudson-Doyle]

On Wed, 2 Aug 2023 at 13:30, Sergio Durigan Junior 
wrote:

> Hi,
>
> Just a heads up that the QEMU package on Mantic is currently affected by
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2016252.  This about
> using QEMU in a nested VM environment, and as such is impacting dep8
> tests that make use of such feature (systemd tests come to mind, but
> there are other packages affected like QEMU itself).
>
> If you see a failure in a dep8 test and it's a QEMU segfault, be aware.
> This is actually a glibc regression that hasn't yet been fixed upstream.
> I'm following the discussion there and plan to backport the patch ASAP
> (unfortunately it's not part of glibc 2.38 either).
>

Can you coordinate that with the impending upload of 2.38 please? (Hi Simon)

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Michael Hudson-Doyle]

On Tue, 11 Jul 2023 at 11:17, Benjamin Drung  wrote:

> On Sun, 2023-07-09 at 15:29 -0700, Steve Langasek wrote:
> > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote:
> > > > > Benjamin Drung  schrieb am Sa., 8. Juli 2023,
> 02:19:
> >
> > > > > > Hi all,
> >
> > > > > > a year ago we changed the default compression and level for the
> > > > > > initramfs to zstd -1. This fixed the very slow creation times on
> > > > > > development boards (see bug #1958148), but that leads to bigger
> > > > > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > > > > Big initramfs sizes can also fill up small sized /boot
> partitions easily
> > > > > > (grooming the 850 initramfs-tools bugs revealed several such
> reports).
> >
> > > > > > Using xz -9 would give very good compression, but it takes very
> long
> > > > > > (especially on slow development boards) and a lot of memory
> (good luck
> > > > > > on Raspberry Pis with small memory like Pi Zeros).
> >
> > > > > > I propose following approach to address the drawback: Create cpio
> > > > > > archives (compressed with xz -9) for the kernel modules and
> firmware
> > > > > > files when building the kernel/firmware Debian package. Then
> ship those
> > > > > > cpio archives in the package (or in a separate binary package).
> Then the
> > > > > > CPU load it put on the builders. The cpio archives would contain
> the
> > > > > > modules for MODULES=most.
> >
> > > > > > mkinitramfs will then look for those cpio archives and uses
> those in
> > > > > > case they are present. Such a initramfs would look like this:
> >
> > > > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > > > * main cpio archive compressed with zstd -1
> > > > > > * kernel modules from the Debian package compressed with xz -9
> > > > > > * firmware files from the Debian package compressed with xz -9
> >
> > > > > > After working on initramfs-tools as part my day job, my fingers
> were
> > > > > > itching and I had to create a quick and dirty draft in my free
> night
> > > > > > time. You can find the result of the last two hours in [1]. This
> draft
> > > > > > has a mkinitramfs-kernel script that creates a cpio archive
> containing
> > > > > > the kernel modules and firmware (that needs to be split later
> on).
> >
> > > > > > The lunar test result on my AMD Ryzen 7 5700G look promising:
> Building
> > > > > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds
> and is
> > > > > > 54.9 MiB in size. Creating the initramfs speeds up from around
> 8.7
> > > > > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1
> MiB to
> > > > > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but
> > > > > > /lib/modules need 54.9 MiB for the cpio archive.
> >
> > > > > > The drawback is that building the kernel would take longer, the
> package
> > > > > > takes more space on the archive and mirrors, and downloading
> them could
> > > > > > take longer on slow connections.
> >
> > > > > > Implementing my proposal would be relative easy for
> initramfs-tools, but
> > > > > > would mean some work for the kernel team.
> >
> > > > > > What do you think?
> >
> > > Will the user still be able to add further modules and will machine
> specific
> > > configuration files (e.g. for booting from iSCSI) still be included
> into the
> > > initrd?
> >
> > I think a robust implementation of this on the initramfs-tools side looks
> > like:
> >
> >  - identify all the contents that belong in the initramfs
> >  - among those contents, find all zstd-compressed files, if any, and
> store
> >them in an uncompressed initramfs
> >  - put the rest of the contents in a compressed initramfs
> >
> > This would be compatible with kernel packages whether they are changed to
> > ship zstd-compressed modules or not and allow for a smooth transition.
>
> This is exactly what I tried to implement over the weekend. The result
> looked promising. The creation time and initramfs size decreased by a
> few percent. Booting that initramfs failed and today I figured out why.
> Now I have a new draft version that works, but is slower to compress,
> because it moves the compressed files around after running depmod.
> Fiddling with the list of files passed to cpio should give back the
> speed, but that is a task for later.
>
> So without further ado here is the result so far:
>

These look like excellent results, and this approach makes a lot of sense
to me. I've wanted to see this improved for ages, so I'm happy to see
someone actually taking steps to make this happen! Spending builder time on
maximum compression vs doing it on every Ubuntu system in existence is
surely better for everyone.

Would the idea be that linux-modules-$VER-generic would contain only the
compressed modules?

I was wondering if it make sense to construct a zstd dictionary for
compressing kernel modules but I didn't realize they need to be available
at decompression time, I'm not sure the kernel would 

Re: 23.04: desktop iso peculiarity compared to the live-server one

[Michael Hudson-Doyle]

On Tue, 27 Jun 2023 at 13:59, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> On Fri, 23 Jun 2023 at 11:38, Adam Vodopjan 
> wrote:
>
>> Mby you're the correct person to report another issue with install images.
>> Since 21.04 the boot/grub/loopback.cfg in live-server is broken: the
>> 'iso-scan/filename=' piece is gone.
>>
>
> Argh that was an unintended consequence of removing 'quiet' from the
> default kernel command line. This should fix it:
>
> https://code.launchpad.net/~mwhudson/debian-cd/+git/ubuntu/+merge/445395
>

So this should be fixed in the next mantic and jammy dailies. Thanks for
the report!

Cheers,
mwh

>
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: 23.04: desktop iso peculiarity compared to the live-server one

[Michael Hudson-Doyle]

On Fri, 23 Jun 2023 at 11:38, Adam Vodopjan  wrote:

> Mby you're the correct person to report another issue with install images.
> Since 21.04 the boot/grub/loopback.cfg in live-server is broken: the
> 'iso-scan/filename=' piece is gone.
>

Argh that was an unintended consequence of removing 'quiet' from the
default kernel command line. This should fix it:

https://code.launchpad.net/~mwhudson/debian-cd/+git/ubuntu/+merge/445395

Cheers,
mwh
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: 23.04: desktop iso peculiarity compared to the live-server one

[Michael Hudson-Doyle]

On Fri, 23 Jun 2023 at 10:24, Adam Vodopjan  wrote:

>
> THE QUESTION I WANTED TO ASK, is: was it intentional to not set the default
> layer in initrd in the desktop iso case?
>
>
I don't think we ever got around to discussing it properly. I think the
desktop ISO should be changed to do things the same way as the server ISO
does, for the same sort of reason you talk about (although my motivation
for the server ISOs was mostly to make netbooting easier, it's the same
thing really).

Cheers,
mwh
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: incoming change to task handling in livecd-rootfs in mantic

[Michael Hudson-Doyle]

On Tue, 23 May 2023 at 03:34, Steve Langasek 
wrote:

> On Mon, May 22, 2023 at 01:04:51PM +1200, Michael Hudson-Doyle wrote:
> > Thinking about this a bit more... what is germinate actually for in this
> > context? :-)
>
> > The way packages from a seed end up in an image goes like this at the
> > moment:
>
> > 1. it is listed in the seed
> > 2. germinate runs and puts the package and all its dependencies into a
> text
> > file
> > 3. live-build reads this text file and uses apt to install the packages
> > (4. we run mimimize-manual at some point so removing a seeded package and
> > running autoremove does something useful)
>
> > The thing about this is, *apt* obviously knows how to find the
> dependencies
> > of a package. Why don't we just shove the packages listed into the seed
> > straight into the file live-build reads?
>
> > (I suppose one answer might be to do with alternatives handling? cf
> > https://imgflip.com/i/7mmgcz -- does germinate do anything clever to
> > satisfy alternatives with a minimal number of extra packages or anything
> > like that?)
>
> It does not.  Indeed, several of the recent seed changes made while landing
> this were to fix the fact that germinate from livecd-rootfs (though not
> when
> run on the archive) was seeding two conflicting implementations one of
> which
> satisfies all the dependencies.




> (Actually this was a virtual package rather
> than an alternative, but AFAIK it doesn't do anything clever with
> alternatives either!)
>

Well potato potahto.


> I would expect feeding the list directly to apt rather than using germinate
> would result in some further changes to the exact packages included, which
> would then need to be examined to confirm they're what we want, since there
> are often multiple ways to resolve a seed.
>
> There's also the fact that you'd be reimplementing germinate's own parsing
> of the seeds and resolution of seed dependencies, so I'm not sure it's
> worth
> it?
>

Yes it's probably not worth it. But maybe we can lean this way for when we
implement things in ubuntu-image (although I think that has support for
running germinate already).

(One thing that would be nice, in addition to dropping the time-consuming
> use of germinate at runtime,


The slow part IME is downloading the package lists, the computational part
germinate only takes a few seconds.


> would be that today germinate silently ignores
> listed packages that are unavailable, whereas apt would enforce correctness
> of the list...)
>

That does sound like it would be an improvement.


> > ((Clearly we need something like germinate to generate the component
> > mismatches reports. But maybe not at image build time?))
>
> That's already run on ubuntu-archive-toolbox independently of the image
> builds, so would continue to do so.
>

Yes, I think my point here was that we can't just delete germinate
completely.

Cheers,
mwh


> --
> Steve Langasek   Give me a lever long enough and a Free OS
> Debian Developer   to set it on, and I can move the world.
> Ubuntu Developer   https://www.debian.org/
> slanga...@ubuntu.com vor...@debian.org
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: incoming change to task handling in livecd-rootfs in mantic

[Michael Hudson-Doyle]

On Fri, 19 May 2023 at 22:38, Iain Lane  wrote:

> On Fri, May 19, 2023 at 09:05:28PM +1200, Michael Hudson-Doyle wrote:
> > After a few rounds of fixups, this change passed all tests and has now
> > migrated to release, so the next round of mantic image builds will be
> built
> > with it. Let me know if you see anything strange in them!
>
> Really nice, great work. This has been long overdue, glad it finally got
> sorted out.
>

Thanks! I think it makes things a bit saner.


> Although the backslash line was one of my favourite lines of code in the
> archive and I'll be sad to see it go. :-)
>

Heh yes. At least version control will preserve it for future generations...

I was reminded of
>
>   https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1921862
>
> When reading this thread. tl;dr is that it's not (was not at the time)
> possible to seed new packages post-release because germinate isn't being
> run for -updates in the publisher so the newly-seeded packages don't get
> Task headers. Would that be any more possible to achieve now? (For
> seeding only: *un*seeding is more gnarly, it involves knowing how to
> have -updates override release.)
>

Modulo what Steve said I think this mostly fixed now, to the extent that
additions and removals to seeds will just be reflected in the next image
build.

The gap we have is that if a seeded package has different dependencies in
different pockets, that might not get picked up. But this seems a touch
edge casey.

Thinking about this a bit more... what is germinate actually for in this
context? :-)

The way packages from a seed end up in an image goes like this at the
moment:

1. it is listed in the seed
2. germinate runs and puts the package and all its dependencies into a text
file
3. live-build reads this text file and uses apt to install the packages
(4. we run mimimize-manual at some point so removing a seeded package and
running autoremove does something useful)

The thing about this is, *apt* obviously knows how to find the dependencies
of a package. Why don't we just shove the packages listed into the seed
straight into the file live-build reads?

(I suppose one answer might be to do with alternatives handling? cf
https://imgflip.com/i/7mmgcz -- does germinate do anything clever to
satisfy alternatives with a minimal number of extra packages or anything
like that?)

((Clearly we need something like germinate to generate the component
mismatches reports. But maybe not at image build time?))

Cheers,
mwh

Cheers!
>
> --
> Iain Lane  [ i...@orangesquash.org.uk ]
> Debian Developer   [ la...@debian.org ]
> Ubuntu Developer   [ la...@ubuntu.com ]
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: incoming change to task handling in livecd-rootfs in mantic

[Michael Hudson-Doyle]

On Mon, 15 May 2023, 09:47 Michael Hudson-Doyle, <
michael.hud...@canonical.com> wrote:

> Hi all,
>
> I've just uploaded a change[1] to livecd-rootfs that changes how the
> "add_task" function works.
>
> It switches away from using the "Task" headers in the archive's package
> lists to find the packages (and snaps) that make up a task to reading the
> information directly from the output of germinate.
>
> I originally wrote this because I wanted to make an image from a rebuild
> archive (which don't have Task headers) but it also makes the handling of
> tasks that much more direct and hopefully a touch less confusing -- for
> example, changes to seeds will be reflected in image builds without having
> to wait for an archive publication cycle (it also removes two sequence of
> six consecutive backslashes from the source, which has to be a win).
>
> Now this _should_ not result in any changes to the contents of images --
> I've tested a few builds before landing and it all looks fine -- but to be
> sure we will hold this new version of livecd-rootfs in proposed until we
> have done a bunch more test images.
>

Hi again,

After a few rounds of fixups, this change passed all tests and has now
migrated to release, so the next round of mantic image builds will be built
with it. Let me know if you see anything strange in them!

Cheers,
mwh

Cheers,
> mwh
>
> [1]
> https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/425047
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

incoming change to task handling in livecd-rootfs in mantic

[Michael Hudson-Doyle]

Hi all,

I've just uploaded a change[1] to livecd-rootfs that changes how the
"add_task" function works.

It switches away from using the "Task" headers in the archive's package
lists to find the packages (and snaps) that make up a task to reading the
information directly from the output of germinate.

I originally wrote this because I wanted to make an image from a rebuild
archive (which don't have Task headers) but it also makes the handling of
tasks that much more direct and hopefully a touch less confusing -- for
example, changes to seeds will be reflected in image builds without having
to wait for an archive publication cycle (it also removes two sequence of
six consecutive backslashes from the source, which has to be a win).

Now this _should_ not result in any changes to the contents of images --
I've tested a few builds before landing and it all looks fine -- but to be
sure we will hold this new version of livecd-rootfs in proposed until we
have done a bunch more test images.

Cheers,
mwh

[1]
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/425047
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Michael Hudson-Doyle]

On Mon, 1 Aug 2022 at 11:51, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

>
>
> On Mon, 1 Aug 2022 at 11:06, Steve Langasek 
> wrote:
>
>> On Sun, Jul 31, 2022 at 05:48:25PM -0400, Jeremy Bicha wrote:
>> > On Sun, Jul 31, 2022 at 4:02 PM Steve Langasek
>> >  wrote:
>> > > On Fri, Jul 29, 2022 at 03:36:32PM +1200, Michael Hudson-Doyle wrote:
>> > > > There is some stuff on NBS due to a ldc transition: r-to-d,
>> > > > appstream-generator, tilix, etc. AFAICT all of these packages have
>> been
>> > > > removed from Debian testing and I guess we should follow along.
>>
>> > > Unfortunately, Ubuntu Budgie has tilix seeded so a removal is not
>> > > straightforward.  Do you want to check with the Budgie developers
>> about
>> > > unseeding this?
>>
>> > ldc being broken is common and I don't think we've done a full removal
>> > from Ubuntu before for it. Although one thing that is different is
>> > that we allow libraries to smoothly migrate out of proposed in Ubuntu
>> > now.
>>
>> It is my understanding that ldc is not broken but that gir-to-d is not
>> compatible with the current version.  There are 6 source packages in the
>> archive that have successfully rebuilt against the new version of
>> libphobos2-ldc-shared, it's only gir-to-d and it's reverse-dependencies
>> that
>> are broken.
>>
>> And Debian has removed all of these from testing, which implies they agree
>> with this analysis.
>>
>> > Could we instead revert to an older ldc?
>>
>> We could, but what is going to get gir-to-d fixed to allow the transition
>> to
>> proceed later?
>>
>
> gir-to-d people appear to think this is a compiler bug:
> https://github.com/ldc-developers/ldc/issues/4000
>
> Not really sure what the takeaway for Debian/Ubuntu is here. A dh-dlang
> change?
>

I chatted briefly to the Debian maintainer on IRC and they said that
they'll upload a workaround to gir-to-d (roughly this
https://paste.ubuntu.com/p/bVHGFXQRJF/) "soon". We can fix it in Ubuntu
sooner, clearly, but IMHO it's worth just waiting a few days to see if we
get the fix via Debian with no effort.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Michael Hudson-Doyle]

On Mon, 1 Aug 2022 at 11:06, Steve Langasek 
wrote:

> On Sun, Jul 31, 2022 at 05:48:25PM -0400, Jeremy Bicha wrote:
> > On Sun, Jul 31, 2022 at 4:02 PM Steve Langasek
> >  wrote:
> > > On Fri, Jul 29, 2022 at 03:36:32PM +1200, Michael Hudson-Doyle wrote:
> > > > There is some stuff on NBS due to a ldc transition: r-to-d,
> > > > appstream-generator, tilix, etc. AFAICT all of these packages have
> been
> > > > removed from Debian testing and I guess we should follow along.
>
> > > Unfortunately, Ubuntu Budgie has tilix seeded so a removal is not
> > > straightforward.  Do you want to check with the Budgie developers about
> > > unseeding this?
>
> > ldc being broken is common and I don't think we've done a full removal
> > from Ubuntu before for it. Although one thing that is different is
> > that we allow libraries to smoothly migrate out of proposed in Ubuntu
> > now.
>
> It is my understanding that ldc is not broken but that gir-to-d is not
> compatible with the current version.  There are 6 source packages in the
> archive that have successfully rebuilt against the new version of
> libphobos2-ldc-shared, it's only gir-to-d and it's reverse-dependencies
> that
> are broken.
>
> And Debian has removed all of these from testing, which implies they agree
> with this analysis.
>
> > Could we instead revert to an older ldc?
>
> We could, but what is going to get gir-to-d fixed to allow the transition
> to
> proceed later?
>

gir-to-d people appear to think this is a compiler bug:
https://github.com/ldc-developers/ldc/issues/4000

Not really sure what the takeaway for Debian/Ubuntu is here. A dh-dlang
change?

Cheers,
mwh


> > appstream-generator also seems important since we use it to build
> > https://appstream.ubuntu.com/ although I guess it doesn't have to be
> > packaged in the latest Ubuntu for that service.
>
> It's a temporary removal, and it would have to remain unfixed for 2 LTS
> cycles to impact our ability to run appstream.u.c from debs.
>
> --
> Steve Langasek   Give me a lever long enough and a Free OS
> Debian Developer   to set it on, and I can move the world.
> Ubuntu Developer   https://www.debian.org/
> slanga...@ubuntu.com vor...@debian.org
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Thanks to sickness and leave and other interruptions I didn't get a whole
lot done in my shift, and I didn't do a very good job of recording the
things I did do I'm afraid.

# ffmpeg 5

I spent a while looking at the ongoing ffmpeg 5 transition and it's a huge
mess. ffmpeg itself has now migrated but there are a pretty large set of
packages on NBS depending on the old ffmpeg libraries. Dan did a great job
of sorting out all the easy and easy-ish stuff here but most (all?) of the
remaining packages seem to require at best moving to much newer upstreams
than are present in the archive or at worse extensive upstream work (some
packages may be able to be built without ffpmeg support until upstream
catches up, which degrades functionality but might be worth doing as a stop
gap). I feel like at this point it is not really something that is suitable
for +1 maintenance work. Unless someone who already has experience with
ffmpeg can devote an entire shift to it, I'm not sure what our plan should
be.

# opencv

This is the other big in progress transition. I don't really know what the
hold up is here: it seems that opencv is failing to build on armhf and
arm64 due to infrastructure issues?

# simpler NBS stuff

vzctl was the only package depending on libcgroups1. I uploaded a no-change
rebuild and it migrated.

astrometry.net's failing autopkgtests were the only thing keeping
libnetpbm10 files around. Steve had patched astrometry.net's dependencies
to point at libnetpbm11-dev but an include path fix was also needed, which
I uploaded.

autofdo (last package depending on libgoogle-glog0v5) ftbfs, probably
because our llvm is too new. Upstream /may/ have this fixed in git,
unfortunately in an omnibus commit with the message "Update to the latest
internal version." I suspect the two sensible options here are to just take
upstream git as a new upstream version or remove the package (it has no
rdeps).

gst-plugins-bad1.0: some tests time out on arm64. I don't know why -- the
build completes in <15 minutes on all other architectures.

There is some stuff on NBS due to a ldc transition: r-to-d,
appstream-generator, tilix, etc. AFAICT all of these packages have been
removed from Debian testing and I guess we should follow along.

The 'apertium' source package changed the name of the binary package for
its library part and apertium-recursive needed a no-change rebuild to pick
this up.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance day report

[Michael Hudson-Doyle]

On Tue, 19 Jul 2022 at 05:31, Paride Legovini  wrote:

> mathcomp-multinomials:
>  - holding 3 packages
>  - missing builds on all archs but riscv64
>  - reason: missing b-deps (riscv64 got lucky because it slow)
>  - retriggered builds; built everywhere but on arm64.
>  - reason for missing arm64 build:
>.
>The following packages have unmet dependencies:
> libcoq-mathcomp-bigenough : Depends: libcoq-mathcomp-ssreflect-94ef7
>E: Unable to correct problems, you have held broken packages.
>.
>  - HANDOVER: I think we just need a no-change rebuild of
>src:mathcomp-bigenough to rebuild against a newer ssreflect.
>

I did this.

qiime:
>  - holding 2 packages
>  - direct autopkgregression on armhf, with error:
>AssertionError: dtype('int64') != 
>  - I wasn't expecting this to ever pass on armhf and did a
>migration-reference/0 retrigger, but it actually passed!
>  - I expect the package to now be a candiate.
>

This is backwards, I think? The reference test succeeded but the new
version fails --> this is a regression on armhf. However the Debian
maintainer has requested removal on 32 bit arches:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1014692. I guess we
should follow.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: libgit2 switch from mbedTLS to OpenSSL

[Michael Hudson-Doyle]

On Wed, 29 Jun 2022 at 20:33, Simon Chopin 
wrote:

> Hi!
>
> As part of our efforts to support the Rust toolchain in main, we need to
> have libgit2 in main (dependency of cargo). However, it currently links
> against mbedTLS for its HTTPS backend rather than OpenSSL, for licensing
> reasons IIUC. Those reasons would now be invalid with the new OpenSSL
> 3.0 licensing.
>
> I'd like to switch it back to OpenSSL to avoid pulling yet another TLS
> implementation in main, however I'm a bit fuzzy whether this would
> constitute a breaking change for the libgit2 package. The libgit2
> library does not expose anything from its crypto implem as part of its
> API, nor does it re-export any of their symbols (assuming I understand
> the output of readelf -s correctly).
>
> Could someone confirm that this does not represent a breaking change?
>

I can't see any way that the selection of the backend leaks into the ABI in
a quick poke around in libgit2. I presume you've built the .so both ways
and looked at the dynamic symbol tables? (actually the symbols file
probably helps here!)

If the same names are exported then we'd only be in trouble if the
arguments to a function have changed somehow and I can't see how that would
happen given the libgit2 headers.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance _day_ report

[Michael Hudson-Doyle]

On Thu, 23 Jun 2022 at 01:33, Paride Legovini  wrote:

> - I polished some local changes I had to allow autopkgtest-virt-lxd to
> run tests in LXD VMs, and submitted this upstream [3]. I think
> testing/feedback from Ubuntu developers is needed there. If that gets
> merged we could consider running the armhf tests in LXD VMs, avoiding
> issues like the firejail one I mentioned above.
>

Do we publish images that work with lxc launch --vm on armhf? I didn't
think we did. Getting armhf autopktests into VMs would be great though.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Building initrd for custom ISO

[Michael Hudson-Doyle]

On Tue, 21 Jun 2022 at 03:06, Renato Botelho do Couto 
wrote:

> I'm working implementing support for a hardware that requires one kernel
> change to have console working.  I already built a custom kernel that
> solves the problem and have console working.
>
> Now next step is to build a custom ISO containing this kernel so it will
> be able to boot and run installer on fixed console.  I understand I need
> to build a custom initrd and copy it to /casper on ISO but I couldn't
> find any documentation about how to do it.
>
> If anyone knows the steps to do it please let me know
>

You should be able to use https://github.com/mwhudson/livefs-editor to do
this. There's no direct support for replacing a kernel with a hand built
one but it should be a start...

Cheers,
mwh


> Thanks!
> --
> Renato Botelho do Couto
> Software Engineer
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Demote ltrace from standard to universe

[Michael Hudson-Doyle]

On Fri, 8 Apr 2022 at 05:55, Julian Andres Klode 
wrote:

> Hi
>
> I was running ltrace today and noticed it doesn't really work at
> all anymore for binaries (tried ls, dpkg, apt, hello) in jammy,
> presumably due to PIE.
>

I think strictly speaking it's actually BIND_NOW rather than PIE directly
(although in practice the two come as a package):
https://alioth-lists-archive.debian.net/pipermail/ltrace-devel/2016-May/001378.html

It also fails to build on various architectures, unmaintained
> since 4 years, and not really up to our quality standards
> anymore IMO.
>
> I'm proposing to remove this:
>
>
> https://code.launchpad.net/~juliank/ubuntu-seeds/+git/platform/+merge/418876
>
> vorlon asked me to raise this here and get some feedback,
> does anyone have an objection to this?
>

I think it's a good idea. FWIW, there is another tool that does a similar
thing but using a more supported facility: latrace. This uses the LD_AUDIT
stuff in glibc. If there is desire to have this functionality in main
(something I'm not particularly confident of, to be sure), it would seem to
be a better choice.

Cheers,
mwh


> I know it's late in the cycle, but if the tool is (mostly) useless,
> demoting it should actually help users to not waste their time.
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer  i speak de, en
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Revisiting default initramfs compression

[Michael Hudson-Doyle]

On Fri, 11 Mar 2022 at 02:39, Dave Jones  wrote:

> On Thu, Mar 10, 2022 at 12:10:39PM +0100, Julian Andres Klode wrote:
> >On Wed, Mar 09, 2022 at 01:24:55PM +, Dave Jones wrote:
> [snip]
> >> Firstly I actually think lz4 -2 is probably the ideal level for that
> >> compressor. There's a large difference in compression performance
> >> between lz4 -1 and lz4 -2 across all platforms tested, but no
> >> difference in memory usage, and only a minimal increase in
> >> compression & decompression time. However, lz4 is currently
> >> configured to use level -9 which takes a considerable amount of extra
> >> time for little to no gain in compression performance (at least with
> >> our initramfs inputs anyway).
> >>
> >> On machines with more generous RAM allowances, zstd -T0 -1 does
> >> appear to be the ideal. The incremental gains in compression at
> >> higher levels are outweighed by the extra time spent compressing
> >> (i.e. for our initramfs inputs at least, the extra time spent on the
> >> compression is not gained back on reading the compressed data at I/O
> >> speeds typical for their respective platforms).
> >>
> >> [snipped some data]
> >>
> >> At this point, if you want some data to play with I'd highly
> >> recommend cloning the following repo and following the instructions
> >> in the README:
> >>
> >> https://github.com/waveform80/compression
> >
> >I'm still not convinced by the data as it does not align at all what I
> >see on my laptop or does it? It certainly does not _feel_ like it, as I
> >was arguing that -12 makes most sense.
>
> I've included the gather.py script in the compression repo if you want
> to run it against your laptop? The entries can be keyed by an arbitrary
> label (like "Julian's laptop") so they wouldn't clobber the existing PC
> based results.
>
> >I did some reameasurements
> >
> >Compression levels:
> >
> >uncompressed157MB
> >lz4 -2   75MB (42%)
> >lz4 -9   63MB (40%)
> >zstd -T0 -1  56MB (36%)
> >zstd -T0 -2  52MB (33%)
> >zstd -T0 -3  47MB (30%)
> >zstd -T0 -6  45MB (29%)
> >zstd -T0 -12 40MB (22%)
> >
> >I don't know where 19 is, but a switch to lz4 -2 would roughly double
> >the size, that's for sure, so how would this affect /boot size?
>
> Sorry -- I should have clarified:
>
> I'm definitely *not* suggesting we switch PC users from zstd to lz4. As
> you point out, that would balloon the size of the compressed initrd. My
> one concern there was that even at level -1, zstd still uses a *teensy*
> bit too much RAM for my comfort on the extremely limited Pi Zero 2 and
> hence that we should be (and indeed, are) using lz4 instead there.
>
> When lz4 is in use (as on the Pi images on jammy), it is currently
> hard-coded to use level -9 which (as you observe below) is quite silly
> in spending a great deal more time achieving effectively no extra
> compression. In other words, my desire for lz4 -2 applies to the Pi
> images on Jammy alone and nothing else (it's a change that, if made,
> should *not* be back-ported to Focal for the reasons you've noted).
>
> >Looking at size, zstd clearly is the correct choice, if we reverted to
> >lz4 -2, sizes would even grow relative to older lz4 -9 choice, meaning
> >those users upgrading from focal run out of boot space.
> >
> >Ignoring non-LTS users for a moment, we essentially need to find a
> >compressor that accomodates the size increase in kernel initramfs due
> >to new code and stuff, and I think zstd -1 does that reasonably well.
>
> Agreed.
>
> >Times spent (compressor/total update-initramfs)
> >   usersystem  total
> >lz4 -2 0.3/ 6.2s   0.1/ 2.6s   0.3/ 8.2s (3% of
> update-initramfs time)
> >lz4 -9 4.8/10.8s   0.1/ 2.6s   4.9/12.9s <- this is totally
> silly
> >zstd -T0 -10.7/ 5.6s   0.1/ 1.7s   0.2/ 6.2s (um, faster than
> lz4?)
> >zstd -T0 -10.7/ 7.1s   0.1/ 3.5s   0.2/ 9.3s (um, much slower in
> 2nd run)
> >zstd -T0 -20.9/ 7.1s   0.1/ 3.0s   0.3/ 8.8s (more noise than
> difference)
> >zstd -T0 -31.6/ 7.8s   0.1/ 2.9s   0.5/ 8.8s
> >zstd -30.9/ 7.2s   0.1/ 3.1s   0.8/ 9.5s
> >zstd -30.9/ 7.7s   0.1/ 3.8s   0.8/10.7s (noise, lots of
> noise)
> >zstd -T0 -66.2/12.8s   0.1/ 3.9s   1.7/11.4s
> >zstd -T0 -12  13.1/19.7s   0.2/ 3.4s   4.0/13.0s
> >
> >It shows us that looking at the compressor does not tell us all the
> >story; for low-level zstd and lz4 values, you will absolutely not
> >notice the time spent compressing; in fact, there is more noise from
> >I/O or whatever despite the laptop essentially idling.
>
> Indeed; this is one of the reasons I stuck to the pure (de)compression
> times in my analysis and ignored the rest of update-initramfs as there's
> also *huge* variety across the architectures there (Pi I/O time is
> vastly different to a PC, and of course the inputs vary in size as well
> as the Pi has a much smaller 

Re: Revisiting default initramfs compression

[Michael Hudson-Doyle]

On Thu, 10 Mar 2022 at 02:24, Dave Jones  wrote:

> Hi Michael (and others),
>
> Julian's summarised this near perfectly, but I'll try and add a little
> detail from the data I've gathered [1] (with others' generous help, in
> particular Heinrich for the RISC-V bits):
>
> On Wed, Mar 09, 2022 at 09:46:19AM +0100, Julian Andres Klode wrote:
> >On Wed, Mar 09, 2022 at 02:10:57PM +1300, Michael Hudson-Doyle wrote:
> [snip]
> >> > some time ago, the default compressor for initramfs was changed
> >> > from lz4 -9 to zstd -19. This caused significant problems:
> >> >
> >>
> >> Exactly three months later... we still haven't taken any action on this.
> >> Time to do something!
>
> Agreed!
>
> >> I have a few questions below but tl;dr: unless there are immediate
> >> objections, I'm going to make a change to initramfs-tools to allow the
> >> compression level to be configured and set the default to 12 for zstd.
> >
> >So xypron had a patch to change the default level to 9 for sponsoring
> >out for a couple of months now (no idea how that level came up).
> >
> >We pushed back on that as it does not account for low-memory systems
> >which we need to take care of as well.
>
> Yes, zstd -12 is not safe on low memory systems. In particular it fails
> to even run successfully on an otherwise entirely idle and unloaded Pi
> Zero 2 (which, on arm64, has a little more than 200MB of RAM free at
> runtime, whilst zstd -T0 -12 on a PC requested ~415MB resident at
> runtime).
>
> In fact, I'm not convinced *any* of zstd's levels are actually useful on
> a machine with as limited RAM as the Zero 2 (or 3A+) are. For example,
> with ~200MB of RAM free, if the user is running a daemon that eats, say,
> 100MB resident and we start up a compressor that eats 50MB (as zstd -T0
> does at level -1) we stand a fair chance of pushing the daemon into OOM.
>

Fair enough.


> Now, in practice this doesn't actually matter right now as I've already
> overridden initramfs' default to lz4 in ubuntu-raspi-settings, however I
> think there are adjustments that should be made there too (and there is
> the question of whether this is relevant for, say, minimal memory cloud
> instances).
>
> >We then postponed any implementation to after a discussion in
> >Frankfurt.
> >
> >I think the summary from the Frankfurt discussion was:
> >
> >- lz4 -1 is the right choice for low-memory systems
> >- if you have more memory, zstd -1 becomes the best choice
> >- pigz is outperforming both a bunch of times
> >
> >But that's really for waveform to share.
>
> Just to clarify a couple of things here:
>
> Firstly I actually think lz4 -2 is probably the ideal level for that
> compressor. There's a large difference in compression performance
> between lz4 -1 and lz4 -2 across all platforms tested, but no difference
> in memory usage, and only a minimal increase in compression &
> decompression time. However, lz4 is currently configured to use level -9
> which takes a considerable amount of extra time for little to no gain in
> compression performance (at least with our initramfs inputs anyway).
>
> On machines with more generous RAM allowances, zstd -T0 -1 does appear
> to be the ideal. The incremental gains in compression at higher levels
> are outweighed by the extra time spent compressing (i.e. for our
> initramfs inputs at least, the extra time spent on the compression is
> not gained back on reading the compressed data at I/O speeds typical for
> their respective platforms).
>

Moving all the way to -1 does feel like quite a change, but well. I'm not
opposed to it.


> [snipped some data]
>
> At this point, if you want some data to play with I'd highly recommend
> cloning the following repo and following the instructions in the README:
>
> https://github.com/waveform80/compression
>
> >> Are people really going to run an arm64 userland on a Pi 0?
>
> I have no specific figures on this, but given the *vast* majority of
> Ubuntu downloads for the Pi skew towards arm64 (about 10:1 last time I
> looked) I think it's a safe assumption that at least some do.
>
> >> Any "real" solution for pi 0 has to involve doing at least the bulk
> >> of the compression not on the pi, there's no real way around that.
> >> Which is something we should do, but realistically it's not happening
> >> for jammy.
>
> I'm not so sure about that. On a Pi Zero 2 under the arm64 arch, lz4 -2
> takes ~8MB of resident RAM, and ~1.9s of time to compress the initramfs
> to approximately half its size. If we're willing to wait a little
> longer, lz4 -4 takes 8

Re: Revisiting default initramfs compression

[Michael Hudson-Doyle]

On Thu, 9 Dec 2021 at 06:13, Julian Andres Klode 
wrote:

> Hi all,
>
> some time ago, the default compressor for initramfs was changed
> from lz4 -9 to zstd -19. This caused significant problems:
>

Exactly three months later... we still haven't taken any action on this.
Time to do something!

I have a few questions below but tl;dr: unless there are immediate
objections, I'm going to make a change to initramfs-tools to allow the
compression level to be configured and set the default to 12 for zstd.

- it is very slow
> - it uses a lot of memory
>
> The former is a problem for everyone, the latter means that
> zstd just crashes on a Pi Zero.
>
> This is an analysis of what we have in terms of time spent,
> memory spent, and file size achieved, and where we can
> go from here.
>
> # Comparison of different compression levels
>
> ## Desktop (ThinkPad T480s, jammy)
>
> levelusertime   elapsed memory fileSize
> lz4 9.65s11.09s12M  64M
> -1  5.69s 6.99s24M  57M
> -6 12.59s 8.58s99M  47M
> -1219.85s10.82s   249M  41M
> -1971.29s26.95s   519M  35M
>
> -> I believe that somewhere around -12 is a decent
>compromise between size and speed.
>

I would agree that it's certainly better than -19.


> ## Pi 4 (arm64, focal)
>
> Times have been measured for mkinitramfs only. A full
> update-initramfs call spends much more time copying
> some firmware bits to boot partition with flash-kernel
>
> levelusertime   elapsed memory fileSize
> lz421.10s64.85s21M  29M
> -1 13.73s44.55s21M  27M
> -6 26.07s49.09s91M  24M
> -1248.18s54.67s   203M  22M
> -19   130.07s92.80s   350M  20M
>
> -> 6 is essentially free if the Pi 4 is idle. Nice.
> -> -6 is still 20% of total RAM of a Pi 0
>

Are people really going to run an arm64 userland on a Pi 0?

Any "real" solution for pi 0 has to involve doing at least the bulk of the
compression not on the pi, there's no real way around that. Which is
something we should do, but realistically it's not happening for jammy.


> -> There's no meaningful difference between -6 and -12
>in terms of time elapsed. -6 uses 116% CPU, -12 uses
>145% CPU.
>
> ## Adaptive compression
>
> zstd also supports adaptive compression, compressing as hard as
> it can while not impacting I/O speed. So hardware with slow I/O
> like a Pi would compress harder to avoid idling.
>
> This is somewhat suboptimal with recent update-initramfs though,
> as it first writes the cpio archive to the disk and then compresses
> it rather than doing it in a pipe where that would be more
> meaningful.
>
> Question: Does zstd --adapt adapt to memory available?
>

While attractive, this does feel a bit risky. We want to be able to make
reasonable predictions about the size of the initrd.


> # Way(s) forward
>
> To remedy the issue the proposal is to build with
>
> - zstd -1 on hardware with 512 MB or less memory
> - zstd between -1 and -19 on other hardware
> - zstd -19 during image building
>

I think this broadly makes sense. I'll notice that currently
initramfs-tools doesn't allow tuning the compression level at all :/
Probably fairly routine to add support for that though.


> Finding the right level between -1 and -19 is hard. The more
> cores you have, the less penalty you pay for higher level.
>

You suggested -12 above. How about we try that to start with?

Do you have an idea how to detect "512 MB or less memory"?


> Going for adaptive compression would remove the guess work, but
> will result in larger images on faster machines. Maybe that's
> fine, though - they probably have more space on /boot anyway?
>
> If we want to aim for 5% of total memory, we should probably
> aim for something like:
>
> -1  on <= 512MB
> -6  on <= 2 GB (or --adapt=min=1,max=6)
> -12 on the rest (or --adapt=min=12)
>
> It's clear that in all cases, zstd -1 is at least better than the
> lz4 -9 we used before; both in terms of space used, and time spent.
>

Yeah that's interesting.


> # Concerns
>
> Lowering the compression level will reduce the boot speed by fractions
> of a second on hardware with fast I/O.
>

Well speaking selfishly, the mean "number of boots per mkinitramfs call"
for me is about 1...

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

I was on +1 this week and managed to help a few things along, at least (it
was a bit of a disrupted week).

The python transition. Oh boy, the python transition.

When I first looked at this, I was lead to sssd and samba, both of which
reported regressions in adsys. This turned out to be samba's fault
(incompatible command line options in 4.15) and I filed
https://github.com/ubuntu/adsys/pull/289 as a sort of simplest
possible solution. I uploaded this to the archive but then the build failed
on ppc64el and dep8 tests failed on armhf. Steve removed the package from
the archive to let the transitions go through.

The new version of avogadrolibs failed in strange ways. Steve removed this
for now (fortunately there was already a version that worked with Python
3.10).

But no! Then britney found more things to be unhappy about. One of these
was an "implicit dependency" chain pointing at astropy / poliastro, which
was dealt with with a hint. A few more things got in the way, including a
component mismatch, and the next thing was a build failure of capnproto on
ppc64el. This turned out to be some undefined behaviour that upstream had
already fixed, so I backported that and uploaded it.

The final obstacle turned out to be wireshark ftbfs on s390x. Eventually it
turned out that previous versions had ignored test failures on s390x and
this was only removed by a typo. So doko put that back and finally finally
everything migrated.

It has to be said, the way britney only revealed one blocking implicit
dependency chain at a time wasted a whole lot of time here. I wonder if
there is anything we can do about that?

In between all this I started looking at the nbs report for libssl1.1.

The no change rebuild of scrypt had failed to build on ppc64el so on a
total hunch I tried rebuilding it without -O3 in a PPA and that succeeded
so I uploaded it and it migrated.

ntpsec was also on the nbs report too. We'd synced a version from Debian a
few days ago that built with openssl3 but the debian/control file still had
libssl1.1 as an explicit dependency so I removed that and uploaded it. It
migrated. This got fixed in Debian very quickly as well so I  synced it
when it became possible.

I looked very briefly at xml-security-c-utils and xmltooling and got
confused.

I merged swi-prolog to fix the build with openssl3 but lots of dep8 tests
now fail.

sblim-sfcb ftbfs when rebuilt for ssl3 but the failure was not ssl related;
rather it was missing dependencies in the automake stuff and some
assumptions that -fcommon was the default. I fixed these and uploaded the
package and it migrated.

mumble ftbfs with openssl3 which is fixed upstream but the patch does not
apply cleanly. There is a new upstream version which will be easier to
backport the fix too it seems but that would require a FFe...

yapet fails to build with openssl3 so i reported this upstream:
https://github.com/RafaelOstertag/yapet/issues/23

After finding another package (libpython2.7-stdlib) that had an explicit
dependency on libssl1.1 I got Seth Arnold to search his collection of every
unpacked source package from the archive and found a small number of other
packages with similar dependencies:

 - nim, where the openssl support works by dlopening a bunch of libcrypto
sonames. Unclear to me if it will work with openssl3.
 - rhash also dlopens some libcrypto sonames but this one was simple enough
that I could tell it would still work with openssl3 so I added
libcrypto.so.3 to the list and uploaded it.
 - qt6-base also dlopens a bunch of libcrypto sonames but appears to have
support for openssl3 so it's just a matter of fiddling the deps around,
mitya57 has some changes
https://salsa.debian.org/qt-kde-team/qt/qtbase/-/commit/531c31ae58aefc7c
that I / someone should get around to testing / proposing properly
 - nvidia-cuda-toolkit -- this was actually in Build-Depends not Depends
and is clearly a special package (the source package is 14 gigs
uncompressed!) so I asked Graham who appears to have uploaded it in the
past.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: proposed-migration, "SMOOTH_UPDATES", and NBS reports

[Michael Hudson-Doyle]

On Thu, 9 Dec 2021 at 05:42, Julian Andres Klode 
wrote:

> On Wed, Dec 08, 2021 at 05:29:19PM +0100, Sebastien Bacher wrote:
> > Hey again,
> >
> > Le 08/12/2021 à 00:14, Steve Langasek a écrit :
> > > I expect that with this option set, we will find much fewer problems
> with
> > > entanglement of library transitions, and in turn I hope developers
> will be
> > > less frustrated by migration delays.
> > Right, I expect that to be the case. It's going to come at the cost of
> > reducing the pressure for the team to complete the transitions since
> that's
> > not going to get in the way of having their updates to land.
> > My personal feeling is that it's going to turn out to be an issue for the
> > archive and the release teams and that we aren't going to find proper
> > staffing to deal with the problems, but hopefully I'm wrong there, we
> will
> > see.
>
> I think we should
>
> - disable smoothing at feature freeze
>

So we're now a little past feature freeze and smooth_updates is still in
effect. This made the recent python/perl/the world transition a bit easier
so that was probably a good thing overall but now that's behind us, maybe
we should disable it now?


> - work on getting NBS to 0 by beta freeze
>

So this becomes less of a whackamole.

Cheers.
mwh


> If it doesn't work, maybe we need to revert to not autosmoothing
> and force or add a smooth hint.
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer  i speak de, en
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Should we change the behaviour of -P for lsblk in util-linux for Jammy?

[Michael Hudson-Doyle]

I too vote for (3). I think (2) would be the next least-bad option but
given the fundamental-ness of util-linux I think it's better to be safe.

Cheers,
mwh

On Sat, 19 Feb 2022 at 02:43, Paride Legovini  wrote:

> Bryce Harrington wrote on 18/02/2022:
> > On Thu, Feb 17, 2022 at 01:00:07PM +1300, Matthew Ruffell wrote:
> >> Hi all,
> >>
> >> I'm looking for a bit of advice about landing a new feature in
> util-linux, as
> >> things have gotten a little complicated, and with feature freeze
> looming, a
> >> second opinion would be much appreciated.
> >>
> >> e.g. lsblk -P now outputs LOG_SEC instead of LOG-SEC.
> >>
> >> So, what I need advice on is the next steps. Should we:
> >>
> >> 1) Do nothing, accept 2.32.2 behaviour for -P in Jammy, which is a
> change from
> >> Focal/Impish, and will abruptly change again with the release of 2.38
> likely to
> >> land in KK. MAAS and Curtin are already fixed, no issues there, users
> must
> >> upgrade to latest stable MAAS and curtin on Jammy release. Older Curtin
> and MAAS
> >> will break.
> >>
> >> 2) Backport the new 10+ commits into 2.27.2 in Jammy and hope we don't
> cause any
> >> regressions with the significant amount of code being changed. We keep
> the same
> >> behaviour that users expect from Focal/Impish, and users can now use
> >> -y / --shell if they want. Older MAAS and Curtin continue to work.
> >>
> >> 3) Revert the single patch which caused all of this,
> >> 58b510e580 libsmartcols: sanitize variable names on export output
> >> which is a tidier and well tested solution, and drop the patch when
> util-linux
> >> is rebased to 2.38 in KK. This keeps existing behaviour in
> Focal/Impish, and
> >> enables older MAAS and Curtin to keep working.
> >>
> >> I'm leaning toward suggesting 3) at this stage, but this is mostly to
> keep
> >> existing users happy on their older versions of MAAS.
> >
> > I think your hunch for #3 does sound like the safer approach to me as
> > well.  Unless there's a huge number of users asking for the new feature,
> > those who do need it can either wait until it's generally available, or
> > use a workaround with some awk filters or whatnot.
>
> I'm also +1 on option (3). The issue I can see with (3) is with
> downstream software parsing  `lsblk --version` to know how to deal with
> the output of -P. However (2) is not a solution for this (we'd still
> diverge from the upstream behavior for that version) and option (1) has
> the issues you mentioned.
>
> Paride
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: glibc 2.35 is coming to jammy

[Michael Hudson-Doyle]

On Fri, 4 Feb 2022 at 14:03, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> Hello again,
>
> On Fri, 28 Jan 2022 at 10:47, Michael Hudson-Doyle <
> michael.hud...@canonical.com> wrote:
>
>> glibc 2.35 is due to be released next week and the plan is to get it into
>> Ubuntu pretty soon after that. As usual, this will run a very large number
>> of autopkgtests and so be a bit disruptive, but I'm not expecting enormous
>> fallout from this update based on the testing we have done so far. Assuming
>> things line up, I'll try to upload on my Friday so that the infrastructure
>> can grind away over the weekend.
>>
>
> I've just uploaded the release to jamy. Hopefully all the tests will be
> done by next week!
>

It took a while, but 2.35-0ubuntu1 has migrated to release now. There is a
known problem with fpc (a pascal compiler) on armhf which will require
2.35-0ubuntu2 pretty soon, but that should not be as disruptive.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: glibc 2.35 is coming to jammy

[Michael Hudson-Doyle]

Hello again,

On Fri, 28 Jan 2022 at 10:47, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> glibc 2.35 is due to be released next week and the plan is to get it into
> Ubuntu pretty soon after that. As usual, this will run a very large number
> of autopkgtests and so be a bit disruptive, but I'm not expecting enormous
> fallout from this update based on the testing we have done so far. Assuming
> things line up, I'll try to upload on my Friday so that the infrastructure
> can grind away over the weekend.
>

I've just uploaded the release to jamy. Hopefully all the tests will be
done by next week!

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

glibc 2.35 is coming to jammy

[Michael Hudson-Doyle]

Hi all,

glibc 2.35 is due to be released next week and the plan is to get it into
Ubuntu pretty soon after that. As usual, this will run a very large number
of autopkgtests and so be a bit disruptive, but I'm not expecting enormous
fallout from this update based on the testing we have done so far. Assuming
things line up, I'll try to upload on my Friday so that the infrastructure
can grind away over the weekend.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Change unattended-upgrades from Depends to Recommends on ubuntu-server-minimal

[Michael Hudson-Doyle]

So I think we should probably change the server-minimal seed to only
recommend, not depend, on unattended-upgrades. Changing to a hard
dependency was not intended when I wrote that seed and a change like this
should probably be a conscious thing.

On Thu, 6 Jan 2022 at 22:25, John Chittum 
wrote:

> Going to backtrack slightly to answer Matthew's question:
>
> Why is Jammy Server semantically different from Cloud images or
>> Container images?
>
>
> The Cloud Image and the LXD Container image are the same. Both are
> generated by the `ubuntu-cpc` project, and thus take the same initial paths
> in `livecd-rootfs` for choosing seed and base install packages. Neither
> install `ubuntu-server-minimal` at this time.
>
> There were decisions made (predating me on CPC) that the cloud image (and
> lxd rootfs) would be separate entities from the Server product. I'm not
> going to weigh in on correctness, but they are separate products at this
> point.
>

It would be super great to consolidate the vaguely-but-not-quite
overlapping "external product"/"livecd-rootfs project"/"seed definitions"
stuff but there are subtleties all over :/ Something for a sprint, a marker
pen and a very large sheet of paper.

>From Steve:
>
> First, I think unattended-upgrades should be directly seeded everywhere;
>> its
>> inclusion in the images should not be a side-effect of including
>> software-properties.
>
>
> On one hand, I generally agree with this statement.
>

Me too.


> It seems odd that it is not directly seeded in the cloud images. the LXD
> container, I'm a little less worried about, and I'd almost argue the
> opposite -- that in a container image `unattended-upgrades` should not be
> installed by default. Or, if it is, the default settings are to not be
> running. That fits more to the current world stance on containers (you
> don't upgrade them, you replace them). For instance, the Docker container
> does not have it installed. From the Impish container (which i have handy)
>
> docker run -it --rm ubuntu /bin/bash
> root@671453626e88:/# dpkg -l | grep unattended
>

Docker images should definitely not have it installed, seeing is there is
no generic mechanism for having it _do_ anything in docker containers. lxd
containers are a bit different.


> This gets into some of the limitations put in place by livecd-rootfs when
> creating images. There is a mapping of $PROJECT to $SEED done in
> livecd-rootfs/live-build/auto/config. And cloud-images, while having their
> own seed, appear to be using ubuntu.$SUITE/server, and then additional
> meta-packages and seed files. the relevant code is here:
>
> https://git.launchpad.net/livecd-rootfs/tree/live-build/auto/config#n893
>
> The gist is anything built by the ubuntu-cpc starts from the server seed,
> then runs the seed task minimal, standard, and cloud-image, then installs
> the meta-package ubuntu-minimal. It's certainly different than the server
> path, and should be different considering the use cases.
>
> From Dan:
>
> And that isn't necessarily a bad choice - large deployments
>> really *do not* want software changing outside of predefined
>> maintenance windows, where actual admins are online to handle any kind
>> of issues caused by the upgrades. That's obviously not the only case
>> where unattended upgrades are not desirable, but a very frequent and
>> large example.
>>
>
> I very much agree with the ease of removal and the large scale
> deployments. Previous life, I was one of those people, doing exactly
> quarterly updates, and our initial Ubuntu image deployed by our IT
> department did not have unattended-upgrades installed. I'm also thinking
> about things in a cloud context, where the approach has been that servers
> are not "pets." I'd say that statement is not a universal truth, but is a
> good starting point for considering the importance of unattended-upgrades
> in a cloud context.
>

The larger scale the deployment, the more expertise we can assume on the
part of the deployer though. I think the defaults have to be targeted at
the "mail server in the closet of a small business" use case.

Cheers,
mwh


> ---
> Dr. John Chittum
> Engineering Manager, Canonical, CPC team
>
>
> On Tue, Jan 4, 2022 at 5:15 PM Dan Streetman 
> wrote:
>
>> On Thu, Dec 16, 2021 at 6:18 PM Steve Langasek
>>  wrote:
>> >
>> > Matthew, Jay, thanks for pressing on this.
>> >
>> > On Tue, Dec 14, 2021 at 05:36:15PM -0800, Jay Vosburgh wrote:
>> > > Steve Langasek  wrote:
>> >
>> > > >Hi Matthew,
>> >
>> > > >On Tue, Dec 14, 2021 at 03:28:32PM +1300, Matthew Ruffell wrote:
>> >
>> > > >It's not necessary to remove the unattended-upgrades package in
>> order to
>> > > >achieve this.  unattended-upgrades is configurable, and it's
>> sufficient to
>> > > >set 'APT::Periodic::Unattended-Upgrade "0";' in
>> > > >/etc/apt/apt.conf.d/20auto-upgrades (or, in a separate file that
>> sorts
>> > > >lexically after, if that works better for the user's configuration
>> > > 

Re: status of ccache and libhiredis in main

[Michael Hudson-Doyle]

On Tue, 7 Dec 2021 at 03:19, Lukas Märdian 
wrote:

>
>
> On Sun, Dec 5, 2021 at 7:38 PM Jeremy Bicha  wrote:
>
>> On Tue, Nov 30, 2021 at 10:22 PM Michael Hudson-Doyle
>>  wrote:
>> > It turns out that ccache has been in main essentially forever, since
>> 2004 and there is no formal record of the motivations for including it in
>> main. However it was and is a useful tool for developers.
>>
>> Until about 5 or 6 years ago, Build-Depends for main packages had to
>> be in main. Because that's no longer required, I think it's expected
>> that most developers will have access to universe when building
>> packages.
>
>
You're right but it seems a bit unlikely that a package would build-depend
on ccache. It's not very useful in the buildd environment...


> My opinion is that it's not a problem to move ccache to
>> universe.
>>
>
> I agree! ccache should be demoted to universe, as it is not used by any
> supported package or the official build process.
> Ubuntu developers can still easily access it from "universe". If there is
> a need to re-promote ccache into "main" in the future, it should go through
> a full MIR process [0] instead of just being in "main" for legacy reasons.
>

That said, I agree too. The seed change got merged a few hours ago and
presumably ccache will get moved to universe by an AA pretty soon.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Revisiting default initramfs compression

[Michael Hudson-Doyle]

On Thu, 9 Dec 2021 at 10:02, Julian Andres Klode 
wrote:

> On Thu, Dec 09, 2021 at 09:21:35AM +1300, Michael Hudson-Doyle wrote:
> > On Thu, 9 Dec 2021 at 08:52, Julian Andres Klode <
> julian.kl...@canonical.com>
> > wrote:
> >
> > > The most interesting solution would be to create the cpip archive
> > > dynamically by giving cpio a list of files over a pipe that we create
> > > dynamically in the hooks as we copy files in, then pipe that to zstd
> > > --adapt; then it would all work super nicely.
> > >
> >
> > If we're going to spend significant effort, I think I'd prefer we try to
> > find a way to not recompress the modules when making the initrd at all,
> at
> > least for MODULES=most builds. We could just ship a zstd -19'ed
> > MODULES=most cpio archive in the kernel packaging and have four layers of
> > initrd (intel firmware, amd64 firmware, modules, everything else) but
> that
> > would bloat the kernel package rather a lot...
>
> I'd kind of like us to ship "default" initramfs in like
> linux-initrd-$uname-r
> and linux-initrd-generic and so on. Maybe even signed somehow so that
> the kernel can verify its integrity when booting. Such that booting with
> authenticated FDE is fully authenticated.
>
> But oh well, those are all long term wishes :)
>

Yeah, let's not let that sort of thing distract us from fixing the issues
that lead to this thread (but also let's not put *too* much effort into
this).

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Revisiting default initramfs compression

[Michael Hudson-Doyle]

On Thu, 9 Dec 2021 at 08:52, Julian Andres Klode 
wrote:

> The most interesting solution would be to create the cpip archive
> dynamically by giving cpio a list of files over a pipe that we create
> dynamically in the hooks as we copy files in, then pipe that to zstd
> --adapt; then it would all work super nicely.
>

If we're going to spend significant effort, I think I'd prefer we try to
find a way to not recompress the modules when making the initrd at all, at
least for MODULES=most builds. We could just ship a zstd -19'ed
MODULES=most cpio archive in the kernel packaging and have four layers of
initrd (intel firmware, amd64 firmware, modules, everything else) but that
would bloat the kernel package rather a lot...

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

status of ccache and libhiredis in main

[Michael Hudson-Doyle]

Hi all,

A while ago I synced ccache from Debian and then discovered:

(a) ccache is in main
(b) the new binary packages depend on libhiredis, which is not

This is because the new upstream version now has a redis storage backend.

It turns out that ccache has been in main essentially forever, since 2004
and there is no formal record of the motivations for including it in main.
However it was and is a useful tool for developers.

There are three broad ways forward here:

(1) MIR libhiredis (it's a pretty small package that itself has no
non-trivial dependencies, it doesn't actually depend on redis so that can
stay in universe, but otoh it is a library that does network access in C
that has had a couple of CVEs already)
(2) Build ccache without support for the redis storage backend.
(3) Drop ccache from main.

(2) would involve the least paperwork but I don't know if it's the option
that best serves our users. (3) is also simple to achieve.

What do you think?

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

It would have been very easy to have spend my entire +1 shift on the Python
3.10 transition but as other people are actively working on that, I tried
to look at other things too.

hdf5 ftbfs on s390x, failing test test_aws_canonical_request (not what I
expected!), retried, failed in same way. The build log gives very little
indication what the failure is about, so I built on a canonistack instance.
Some gdb time time later I found an out of bounds write that could have
affected any architecture and filed
https://github.com/HDFGroup/hdf5/issues/1168. I applied a simple patch and
uploaded and it built fine. Then a confusing dolfin/armhf autopkgtest
failure which went away on retry and this migrated.

subversion fails on ppc64el because ruby uses gcc-10 on ppc64el and this
leads to LTO-related failures. Apparently this will all become irrelevant
when ruby3 is the default but I don't know what the ETA on that is.

zodbpickle breaks with python 3.10. There is a patch upstream but Debian
seems way out of date and there are no reverse dependencies so it's a bit
hard to care. Maybe I'll come back later in the week (I didn't).

siphashc is another package broken by python 3.10 with one upload in debian
and no reverse dependencies.

sphinxbase ftbfs i think because python 3.10 added a deprecation warning to
the distutils module. This turns out to be a problem from
https://www.gnu.org/software/autoconf-archive/ax_python_devel.html. Graham
uploaded a new version of autoconf-archive but sphinxbase still required
some hacking to use the ax_python_devel.m4 from there. This migrated
eventually.

pythonmagick also has autotools problems. I tried a local build with
dh-autoreconf but that didn't seem to help. Graham uploaded some related
things but then the package still doesn't work because it links the
python3.9 extension to the python 3.10 version of the boost_python library
for some reason.

h5py failed tests on armhf with a bus error (turned out this had been
reported upstream already at
https://github.com/h5py/h5py/issues/1927#issuecomment-963639147) This was
an unaligned access that was easy to find on canonistack with gdb and I
uploaded a simple patch to fix it and it built fine. The autopkgtests were
very angry but not, afaict, particularly because of h5py, more just fallout
from the python transition.

There were a few packages (eckit, fckit, metkit) that upstream no longer
supported on 32 bit, so I got an AA to remove the lingering armhf binaries
and they migrated.

supertuxkart ftbfs on arm because an assembly file used vfp instructions
without passing an -march option or using a .arch directive to enable FPU
instructions (something that only became necessary fairly recently). I
patched it to use a .fpu directive (but maybe .arch would have been better
-- oh well). It built and migrated after this.

ncurses was blocked by an autopkgtest failure in cunit. I found a patch in
the Debian BTS fixing this so I nmu-ed this to debian's DELAYED/5 queue. We
could upload this to Ubuntu as delta but it will end up in Ubuntu soon
enough so I don't know if it's worth it.

I retried lots of r things, mostly successfully.

Something in the R stack had regressed on s390x (and probably all big
endian systems). It showed up in r-cran-s2 and its dependencies, but I
suspected the problem was actually in r-cran-wk (whose autopkgtests have
never passed on s390x, sigh, so there was no reported regression for this
package). After a bit of hunting I found the problem and filed
https://github.com/paleolimbot/wk/issues/105 (spoiler, it may cause a
laugh/snort, it certainly did for me). I uploaded the simple fix and it
helped, with r-cran-wk itself migrating followed by a couple of other r
packages.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Release vmlinuz and initrd alongside iso

[Michael Hudson-Doyle]

Hi,

On Tue, 9 Nov 2021 at 02:19, MonkZ  wrote:

> Hi,
>
> currently enabling booting via ipxe (https://ipxe.org/) over http needs
> a dedicated mirror that has vmlinuz and initrd extracted from the iso.
>
> Would it be possible to release those files - already extracted from the
> iso - alongside those very isos?
>

This is something we should do, yes. I've created an internal ticket to
look into it (after failing to find the one I was *sure* already existed)
-- I personally am not very familiar with the relevant bits and pieces.

Cheers,
mwh


> This would enable ubuntu to create one iso, that can boot every version
> available directly from one usb stick or pxe server.
>
> #!ipxe
> dhcp
> set http-server https://releases.ubuntu.com
> kernel
> http://${http-server}/ubuntu/21.10/ubuntu-21.10-desktop-amd64.iso.vmlinuz
> initrd
> http://${http-server}/ubuntu/21.10/ubuntu-21.10-desktop-amd64.iso.initrd
> imgargs ubuntu-21.10-desktop-amd64.iso.vmlinuz root=/dev/ram0
> ramdisk_size=300 boot=casper ip=dhcp netboot=url
> url=http://${http-server}/os/ubuntu/21.10/ubuntu-21.10.x-desktop-amd64.iso
> boot || shell
>
> Something similar is done by Arch Linux. But they included the Let's
> Encrypt Root Certificate too, to enable boot via https, and they
> chainloaded a menu from their mirror to just have one iso or usbstick
> image to work indefinitely, as the critical information, what to boot
> would be loaded after the boot of ipxe.
>
>
> Regards
>
> MonkZ
>
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

My +1 shift this week (good timing after the glibc migration unclogged
things a bit).

I started by scrolling about halfway down excuses and seeing what I found.

The first thing I found was radicale, which was failing its own tests but
they passed when I tried to reproduce locally so I retried. It still fails
so I assumed without any real evidence that this is going to be down to
proxy nonsense, which it was. I uploaded a simple change and got it right
on the second try and it migrated.

xorg-server: 9base was apparently still waiting for test results on ppc64el
after 20 odd days. I retried this from the autopkgtest result page, and it
migrated.

monero has some odd linker error on riscv64. I retried it and it failed
again.

pmdk ftbfs on amd64, apparently because the version of valgrind used does
not know about the clone3 syscall. There are a few valgrind commits around
glibc 2.34 which I cherry-picked and uploaded. Once this was built and
published, a retry of pmdk succeeded and it migrated.

pmemkv was in depwait due to libpmemobj-cpp failing to build with valgrind
complaints. My valgrind fixes above fixed this everywhere apart from
ppc64el, not sure what's going on there. pmemkv built on amd64 and arm64
now but is still in depwait for ppc64el.

New version of imx-code-signing-tool falls foul of -fcommon default.

kdiff3 ftbfs due to gcc-11. It's fixed upstream so I cherry-picked the
patch and uploaded it and it migrated.

golang-golang-x-net was blocked on a few packages having trouble with
proxies, which Sergio seemed to have fixed in his shift so I retried them.
In addition it was blocked by golang-github-smartystreets-assertions on
s390x which has a test that fails on s390x with Go 1.16 or newer (
https://github.com/smartystreets/assertions/issues/42) so a hint is
probably appropriate. I filed
https://code.launchpad.net/~mwhudson/britney/+git/hints-ubuntu/+merge/408185,
which got merged and then golang-golang-x-net migrated.

secsipidx was failing to build from source due to the change to Go 1.16 by
default. This is now the default in Debian sid too so I fixed this in
Debian (by adding GO111MODULE=off to rules), synced it and it migrated
without fuss.

I looked into why go was segfaulting in the snapd/ppc64el autopkgtests and
it turns out this was a grotty problem that had already been fixed
upstream: https://go-review.googlesource.com/c/go/+/328110. Impish already
had the fix but the snapd tests use my Go snap and that hadn't been updated
yet, so I updated the snap. This issue is going to get fixed in the kernel
too, so it isn't going to have the impact I feared it might.

It might not strictly count as +1 duties but I looked into a bug involving
the snapd autopkgtests https://bugs.launchpad.net/snapd/+bug/1943077 and
filed a proposed fix https://github.com/snapcore/snapd/pull/10757. This got
merged so the tests should start passing again when this hits the stable
channel.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: glibc 2.34 is coming to impish tomorrow

[Michael Hudson-Doyle]

Unfortunately, a bug (
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1942276) meant that
I've had to upload glibc again. This will trigger all the tests in the
world again, but I believe it's both possible and reasonable to not
actually run all these tests. We should definitely run a subset, like
letting a random 10% of the tests run would make sense to me, although I
don't know how to do that . Hopefully someone else does :)

Cheers,
mwh

On Tue, 17 Aug 2021 at 10:12, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> So glibc 2.34 is now in impish proposed and all the autopkgtests have run.
> I've started a discourse thread to coordinate handling the fallout:
> https://discourse.ubuntu.com/t/migrating-glibc-2-34/23749
>
> On Wed, 11 Aug 2021 at 11:49, Michael Hudson-Doyle <
> michael.hud...@canonical.com> wrote:
>
>> Hi everyone,
>>
>> This is just a bit of advance notice that we'll be uploading the new
>> release of glibc, 2.34, to impish in about 24 hours and are hoping we can
>> get it migrated before feature freeze next week. So expect autopkgtest
>> queues to be a bit clogged for a while.
>>
>> As Matthias mailed yesterday we have done a test rebuild with the new
>> glibc and while it causes a few build failures we're working through there
>> is no evidence of trouble running existing binaries so hopefully the new
>> version won't be too disruptive. But finding that out is what all
>> those autopkgtests are for :-)
>>
>> Cheers,
>> mwh
>>
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Impact of removing ubuntu-server metapackage

[Michael Hudson-Doyle]

On Fri, 20 Aug 2021 at 10:55, Matthew Smith  wrote:

>
> Hello,
>
> I have a handful of applications that will run on a ubuntu 20.04 server
> install. I want to be able to manage core files that could be generated if
> any of those applications crash using systemd-coredump. I noticed that when
> I install systemd-coredump, it removes apport and ubuntu-server. Both
> systemd-coredump and apport provide "core-dump-handler" so systemd-coredump
> and apport have a conflict which results in apport being removed. Since
> ubuntu-server has a dependency on apport it's removed when apport is
> removed.
>
> The description of ubuntu-server says:
>
>  This package depends on all of the packages in the Ubuntu Server system
> .
>  It is also used to help ensure proper upgrades, so it is recommended
> that it not be removed.
>
>
> I am wondering exactly how ubuntu-server is related to ensuring proper
> upgrades. If I choose to use systemd-coredump and ubuntu-server gets
> removed as a side effect, will that cause trouble if I want to upgrade or
> dist-upgrade at some point in the future?
>

The main thing is that you won't get any packages that have been added to
Ubuntu server in the new version. For example multipath-tools got added to
the default install between 18.04 and 20.04 -- if ubuntu-server is still
present on the system, upgrading will cause multipath-tools to be
installed, if the metapackage has been removed then it won't.

Cheers,
mwh
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: glibc 2.34 is coming to impish tomorrow

[Michael Hudson-Doyle]

So glibc 2.34 is now in impish proposed and all the autopkgtests have run.
I've started a discourse thread to coordinate handling the fallout:
https://discourse.ubuntu.com/t/migrating-glibc-2-34/23749

On Wed, 11 Aug 2021 at 11:49, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> Hi everyone,
>
> This is just a bit of advance notice that we'll be uploading the new
> release of glibc, 2.34, to impish in about 24 hours and are hoping we can
> get it migrated before feature freeze next week. So expect autopkgtest
> queues to be a bit clogged for a while.
>
> As Matthias mailed yesterday we have done a test rebuild with the new
> glibc and while it causes a few build failures we're working through there
> is no evidence of trouble running existing binaries so hopefully the new
> version won't be too disruptive. But finding that out is what all
> those autopkgtests are for :-)
>
> Cheers,
> mwh
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

glibc 2.34 is coming to impish tomorrow

[Michael Hudson-Doyle]

Hi everyone,

This is just a bit of advance notice that we'll be uploading the new
release of glibc, 2.34, to impish in about 24 hours and are hoping we can
get it migrated before feature freeze next week. So expect autopkgtest
queues to be a bit clogged for a while.

As Matthias mailed yesterday we have done a test rebuild with the new glibc
and while it causes a few build failures we're working through there is no
evidence of trouble running existing binaries so hopefully the new version
won't be too disruptive. But finding that out is what all
those autopkgtests are for :-)

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report part 2

[Michael Hudson-Doyle]

I looked a bit at sbcl. It is being held in proposed by a regression in
cl-xmls, but looking at the cl-xmls failure, it is actually a failure of
the clisp tests, not the sbcl tests. There is a version of clisp in
proposed but it's failing to build on ppc64el and s390x with the error
"oint_addr_mask
does not cover CODE_ADDRESS_RANGE" (i think this might be caused by kernel
changes perhaps?). clisp seems nearly dead upstream, so perhaps not testing
it in autopkgtests makes sense? Or removing it? Or ignoring the test
failure in cl-xmls.

suricata fails autopkgtest in a strange way, and it turns out Steve
reported the reason three years ago:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895342 (the service does
not start unless there is a nic called eth0). I filed a bug in launchpad
and tagged it to show up in excuses so hopefully people won't waste time on
this again.

stylish-haskell ftbfs on some arches because the new version has a new
dependency (from src:haskell-ghc-lib-parser) that did not build on those
architectures because the build runs out of memory. I'm testing a build in
my ppa with parallel builds disabled (
https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages?field.name_filter=haskell-ghc-lib-parser_filter=published_filter=)
to see if that helps but even then it's not going to build on s390x (
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=967075) so there is still
something to be figured out here. And it still OOMed on arm64.

acpica-unix ftbfs on armhf. Digging uncovered a segfault and more digging
revealed that this was caused by the Debian maintainer "ack"ing previous
NMUs by throwing part of one of them away. I uploaded it again and it
migrated.

request-tracker5 ftbfs for fairly trivial reasons. I reported a bug in
debian about this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988905

python-iptables was failing its (superficial) autopkgtest due to changes in
the output of ldconfig -v. I filed a PR upstream
https://github.com/ldx/python-iptables/pull/322 and uploaded it.

ogre-1.9 ftbfs on armhf. I found an upstream PR with a fix (
https://github.com/OGRECave/ogre/pull/1592) and uploaded it.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report part 1

[Michael Hudson-Doyle]

I'm on +1 maintenance this week but off on Wednesday so here is a report
from my first two days.

I looked at php stuff. I sent a small update on PHP stuff to Bryce and this
list.

I figured out xfig after far too much debugging (
https://sourceforge.net/p/mcj/tickets/120/).

I took a look at thrift, which has been broken by Go 1.16's switch to
GO111MODULES defaulting to on. Upstream has a fix (
https://github.com/apache/thrift/commit/b71f11e251a711604cea8caad7d493ea57fe8a8f)
which will presumably filter through to the distros sooner or later. It
doesn't backport cleanly, although I didn't really look into this -- it's
probably mostly mechanical to apply it to the version in the archive.

thin is failing (new) tests on armhf because it is trying to connect to a
test server via 0.0.0.0 but the request is going out to the squid proxy
which replies "lol no". I guess this could be fixed by adding 0.0.0.0 to
no_proxy but maybe we should do this always? I also don't understand how
this works on other architectures -- if you do (python) s=socket.socket();
s.bind(('0.0.0.0', 0)); print(s.getsockname()[0]) will you ever get an
answer other than 0.0.0.0?

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Michael Hudson-Doyle]

On Sat, 8 May 2021 at 21:30, Dan Bungert 
wrote:

> +1 Maintenance Report
> Dan Bungert, Week of May-03-2021
>
> ### xfig ###
>
> Local tests fine, there is a valgrind complaint which I chased for a
> while before concluding that the ghostscript library that xfig is using
> has some valgrind magic in there.  My hunch is that the valgrind stuff
> is a false positive and just not interacting well with the valgrind
> settings on ghostscript / this xfig test.  It bugs me that this seems to
> test fine for Debian but not for Ubuntu but I currently lack a better
> answer.
>
> I was able to verify at least that this doesn't seem to be something
> that could be remediated by big_packages:
> autopkgtest -- qemu --ram-size=600 --cpus=1
> passed for me.


So on digging into this it turns out the testsuite output is copied into
the autopkgtest artifacts folder, so if you click the "artifacts" link on a
failing run in https://autopkgtest.ubuntu.com/packages/x/xfig/impish/amd64
you do get a bit more detail as to what is going on, and the failure turns
out to be because there is a message on stderr ("Error: XtCreatePopupShell
requires non-NULL parent") that causes the test to fail. This test is new
in the new version of xfig and so possibly would have failed in all
versions of Ubuntu, so in some sense waving it through would be justified
but I also thing it's easy to patch the test to not care about stderr, so
I'm going to try and do that (although. Did you know autoconf included a
test harness? It does --
https://www.gnu.org/software/autoconf/manual/autoconf-2.61/html_node/Writing-testsuite_002eat.html
-- and I'm not sure I'm any happier for knowing this).

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: PHP 8.0 transition plan

[Michael Hudson-Doyle]

On Sat, 15 May 2021 at 15:54, Bryce Harrington <
bryce.harring...@canonical.com> wrote:

> Hi mwhudson,
>
> The other day you mentioned you'd be on +1 duty next week and expressed
> interest in how the php transition is going.
>
> Here's a wiki page with the current state of things:
>
> https://wiki.ubuntu.com/ServerTeam/Transition/Php8.0
>
> I've put in rebuilds for most of the stack by now, and a healthy chunk
> has at least built and entered -proposed, and some has fully migrated.
> A bunch of stuff is blocked by php-apcu and if that can migrate, a lot
> more of the stack should migrate too.  The task here is getting symfony
> and php-doctrine-cache to migrate, but they have some non-obvious test
> failures that need investigation.
>

So what seems to be going on here is that php-apcu-bc is not going to be
updated for PHP 8.0 (https://github.com/krakjoe/apcu-bc/issues/34) and so
should be removed from the archive (the package currently in
impish-proposed has a patch to only build for php 7 and so is almost empty).

Its only reverse-build-depends are ... symfony and php-doctrine-cache (and
it has no reverse-depends at all, only a reverse recommends from php-apcu
which presumably should also be removed).

Dropping the build-dependency from symfony leaves this failure:

Running src/Symfony/Component/HttpFoundation tests
PHPUnit 9.5.2 by Sebastian Bergmann and contributors.

Runtime:   PHP 8.0.5
Configuration: /<>/phpunit.xml.dist
Warning:   Your XML configuration validates against a deprecated schema.
Suggestion:Migrate your XML configuration using
"--migrate-configuration"!

Testing /<>/src/Symfony/Component/HttpFoundation
.   61 / 1233 (
 4%)
.  122 / 1233 (
 9%)
.  183 / 1233 (
14%)
.  244 / 1233 (
19%)
..S..  305 / 1233 (
24%)
.  366 / 1233 (
29%)
.  427 / 1233 (
34%)
.  488 / 1233 (
39%)
.  549 / 1233 (
44%)
.  610 / 1233 (
49%)
.  671 / 1233 (
54%)
.  732 / 1233 (
59%)
.  793 / 1233 (
64%)
.  854 / 1233 (
69%)
.  915 / 1233 (
74%)
..
Legacy deprecation notices (4)
KO src/Symfony/Component/HttpFoundation
PHP Fatal error:  Cannot use int as default value for parameter $with_cas
of type bool in
/usr/share/php/PHPUnit/Framework/MockObject/MockClass.php(51) : eval()'d
code on line 141

php-doctrine-cache still ftbfs without the build-dep, like this:

There were 2 failures:

1)
Doctrine\Tests\Common\Cache\MemcacheCacheTest::testSetContainsFetchDelete
with data set "null" (null)
Scalar and array data retrieved from the cache must be the same as the
original, e.g. same type
Failed asserting that false is identical to null.

/<>/tests/Doctrine/Tests/Common/Cache/CacheTest.php:35

2) Doctrine\Tests\Common\Cache\MemcacheCacheTest::testUpdateExistingEntry
with data set "null" (null)
Scalar and array data retrieved from the cache must be the same as the
original, e.g. same type
Failed asserting that false is identical to null.

/<>/tests/Doctrine/Tests/Common/Cache/CacheTest.php:59

but it also seems the package is deprecated now?

Cheers,
mwh

For items in that list that are FTBFS I've done some preliminary
> investigation and noted possible solutions.  Some of these will be
> pretty easy to fix.
>
> The stuff under To Review probably just need nochange rebuilds but I
> haven't had a chance yet to look through them.
>
> I'll be out first half of Monday but will continue on this transition
> through the rest of the week.
>
> Bryce
>
> On Wed, May 12, 2021 at 09:12:03AM -0700, Bryce Harrington wrote:
> > Hi devs,
> >
> > For impish the server team will be transitioning PHP to 8.0 over the
> > coming weeks. (LP: #1927264) [0]
> >
> > Since version 7.0, upstream PHP has adopted a regular release cadence[1],
> > with one release per year. Each release is supported for 2 years, plus a
> > third year of security critical fixes. PHP 8.0 was available last cycle
> > but since changes from 7.4 to 8.0 were significant[2], we opted to
> postpone
> > it to the 21.10 cycle and focused instead on resolving some phpunit
> > 8.5->9.5 transition troubles. Landing PHP 8.0 this cycle will enable it
> 

Re: autoinstall network question

[Michael Hudson-Doyle]

On Fri, 15 Jan 2021 at 02:23, Jerry Geis  wrote:

>
>
> On Thu, Jan 14, 2021 at 7:42 AM Jerry Geis  wrote:
>
>> On my boot line for append I have biosdevname=0 net.ifnames=0 (using
>> 20.04 LTS).
>> append  initrd=/casper/initrd debian-installer/local=en_US autoinstall
>> ds=nocloud;s=/cdrom/  quiet splash biosdevname=0 net.ifnames=0 ---
>>
>> After the auto install I have no network.
>>
>>  network:
>> network:
>>   version: 2
>>   ethernets:
>> eth0:
>>   dhcp4: yes
>>   dhcp-identifier: mac
>>
>> This is my network section.
>>
>> So now I do "dmesg | grep eth" and it says enp1s0 is the network.
>> Why is that ? I told the kernel to use eth0 type names ?
>>
>> Jerry
>>
>
>
> Ok - so I discovered that while I boot autoinstall with the "biosdevname=0
> net.ifnames=0" - this does NOT - automatically get set during the install
> and then reboot.
>

It will if you put it after the --- characters.

Cheers,
mwh


> If I login and manually add them to the /etc/default/grug file and reboot
> it works.  So question then is how do my "late-commands" - update the
> /etc/default/grub ?
>
> I found an example where it using /target and curtin in-target --target
> /target update-grub - - --- WOW - can I not do "chroot /target" and just
> have normal commands ?
>
> Jerry
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Disable File Checking Boot Parameter

[Michael Hudson-Doyle]

On Wed, 9 Dec 2020 at 01:27,  wrote:

> Hello
>
> I need to boot a Ubuntu LiveUSB multiple times a day. Each time, I have
> to remove the "maybe-ubiquity" from the kernel parameters and the side
> effect is that I would not be able to skip the .deb files check which
> lasted quite long for each boot (Pressing Ctrl-C does nothing). Is there
> a boot parameter I can pass to disable this checking?
>

fsck.mode=skip on the kernel command line should do it.

Cheers,
mwh


> Thanks
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

This shift I spent a while working on a ghc bug that is blocking the
haskell transition. It turned out to really be a gold (linker) bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=26902 which fortunately has
a simple fix which should land upstream and then in Debian and Ubuntu soon.

I then spent a while looking at python things. pandas should migrate when
https://code.launchpad.net/~mwhudson/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/393889
is merged and the dask tests retried and numpy is close to migrating with a
bit of retrying. I think there is a genuine regression though in numpy on
s390x causing python-meshio to fail tests. It is a tolerance based test and
upstream has reacted to previous failure reports by just increasing the
tolerance but I don't know if that's appropriate this time. Scipy is
reported as triggering heaps of "regressions" but most of those are just
because numpy hasn't migrated yet, I'm not sure it's worth retriggering
them all with the corresponding triggers vs just getting numpy migrated.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Because of some family stuff, I only had one slightly shortened day
this time around. I focused on ftbfs from the recent report -- I scrolled
down to universe and just started from the top.

I fixed 4g8 in Ubuntu. I should probably NMU it to debian but I've not done
that before!

I fixed accelio by turning off some warnings (don't build distro code with
Werror pls)

I found an upstream patch for the accounts-qml-module failure and
backported that. It turned out that libaccounts-qt also needed a patch
before accounts-qml-module would build.

acelib ftbfs with glibc 2.32 but I found a newer version of the upstream
source with a fix. But then it is broken by the removal of sunrpc from
glibc.

achilles failed on armhf but it looks spurious.

acm is broken by the removal of sunrpc from glibc. I think this might be
fixed upstream, at least in this fork
http://www.icosaedro.it/acm/download.html.

adequate was removed from Debian testing almost a year ago as part of the
python2 removal, so probably should be removed from Ubuntu too.

Then my next shift rolled around quicker than usual and this time I started
at the bottom of the list. I fixed simple things in zookeeper, ziproxy,
zhmcclient.

I whipped up a patch to switch zfs-fuse to tirpc and sent it to Debian to
see if the maintainers could be prodded into giving it a quick sanity check
(https://bugs.debian.org/971688)

I fixed a failure in zeromq3 on s390x (apparently a consequence of more
aggressive inlining there?) and sent it upstream
https://github.com/zeromq/libzmq/pull/4055 (it's trivial).

zemberek-ooo fails in the same way as debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=802416 -- which was filed
in 2015, and it was removed from testing in 2017 -- probably a removal
candidate then...

z3 fails on s390x in a way I find totally opaque.

I fixed simple issues in yersinia, yazpp, xtron and xsd.

I then read some documentation and NMUed my fixes to Debian where
appropriate.

I had a quick look at the openexr transition, currently blocked by an
autopkgtest failure and the failure of vips and kio-extras to build. I
retried the autopkgtest failure a couple of times (
https://autopkgtest.ubuntu.com/packages/b/bambam/groovy/s390x), synced a
build fix for vips from Debian, looked at kio-extras and mostly got a bit
angry at cmake but found a fix and uploaded that. vips 8.10 in proposed
seems to break ruby-vips on arm64 and ppc64el though - I don't think this
is a flake so this needs some kind of resolution before the transition will
go through.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

I forgot to send my +1 report for my last shift. This was possibly because
it was amazingly frustrating, it was in the middle of the ghc/libffi
transition and mostly consisted of waiting for riscv and armhf builds to
take ages to fail and have to be restarted. But at least those transitions
did get completed in the end.

Notes from my 31/8 - 01/9 shift:

I started by looking at the libffi transition.

I saw that ecl was blocked by slime tests failing on armhf and arm64. The
slime tests that are failing were the sbcl ones rather than the ecl ones
though so I retriggered them with the version of sbcl in groovy-proposed.
They passed (and sbcl migrated).

allure ftbfs on riscv64 because, I think haskell-lambdahack failed to build
on riscv. Someone has already retried the build 4 hours ago but the last
version took 21 hours to build so I'll have to come back to this tomorrow.

I then spent a good long while digging into haskell packages which seemed
to have been built out of order and realized that this was why Steve and
Gianfranco were talking about getting haskell-gi-harfbuzz out of Debian NEW
and into Ubuntu.

glib2.0 ftbfs because of a doc test. It seems that it would build with the
version of gtk-doc in unstable, but we've synced gtk-doc from experimental
and it doesn't work with that? seb128 sorted this out.

gjs is failing autopkgtests on amd64 with lots of segfaults but it passed
when tested with all-proposed=1...

bagel has regressed in release on s390x
https://code.launchpad.net/~mwhudson/britney/hints-ubuntu/+merge/390049

I retried the cffi/amd64 test with the ecl from proposed.

Notes from 14/9 - 15/9 shift:

I looked at cl-usocket. The version in proposed actually runs the testsuite
now, and this makes connections to internet hosts that are not going to
work on our infrastructure. So I filed a MP to force-reset-test this
package:
https://code.launchpad.net/~mwhudson/britney/hints-ubuntu-cl-usocket/+merge/390651,
then I filed a Debian bug http://bugs.debian.org/970345.

I looked at bilibop for too long before finding that Steve had already
filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969606

python-biopython tests fail because clustalo segfaults on s390x, which is
also why that package fails to build on s390x. It turned out Steve had
already looked into this, and it's very strange: it only happens on the
buildds, not on more-or-less identically configured canonistack instances.

mksh fails because the build process runs chmod -x when a binary fails the
testsuite but still ships it and then the testsuite breaks. I filed debian
bug 970268 about this which got fixed, so I synced the fix.

libgraphics-colornames-perl is held in proposed by libcolor-calc-perl
failing tests, but the latter ftbfs and has been removed from testing so we
should follow along. I filed
https://bugs.launchpad.net/ubuntu/+source/libcgi-application-plugin-authorization-perl/+bug/1895489
for this.

acpica-unix fails on s390x and seems pretty broken (initialization fails).
It seems upstream does not support big-endian and this is added in a debian
patch, so I guess this needs updating.

lintian-brush tests fail on s390x but it seems very likely the problem is
really in "ruamel.yaml.clib", a Python C extension library which does not
have (afaics) any tests (!).

elpy seems a bit flaky on (at least) s390x, I retried it and after a couple
of goes it migrated.

pygalmesh is failing tests on s390x, I investigated a little and filed an
upstream bug: https://github.com/nschloe/pygalmesh/issues/111

cp2k fails on armhf and I suspect would also on other 32 bit arches. ginngs
filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=948909 back in
January with no reply about this. Maybe the next +1 person can action the
plan there of not building on 32 bit and removing the existing binaries.

minimap2 was failing to build on most architectures (
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969596) so I whipped up a
simple patch, uploaded it and attached it to the Debian report, which got
uploaded so I synced it.

Then Steve pointed me at golang-github-grpc-ecosystem-grpc-gateway which
currently ftbfs. It seems that the issue here is that groovy-proposed has
moved golang-goprotobuf to the 1.4 series, and this is quite a big
change. golang-github-grpc-ecosystem-grpc-gateway does have a "v2" branch
upstream that has support for goprotobuf 1.4 but that is not yet released
and it's not clear to me when it will be released:
https://github.com/grpc-ecosystem/grpc-gateway/issues/1223. Balint says
he'll look at this. One option would be to make a golang-goprotobuf13
package that contains the older branch of the protobuf package but well.
That's pretty ugly.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: installation kit modified during installation

[Michael Hudson-Doyle]

On Sat, 29 Aug 2020 at 01:25, Doru Georgescu  wrote:

> Short version:
>
> Byte 480 of the Ubuntu desktop 20.04.1 LTS installation kit has been
> modified during installation. Is this by design?
>

Yes, the logs of the installation process are now written to the USB stick
by default. I guess the change you see at byte 480 is the change to the
partition table.


> Detailed version:
>
> I use to create my usb stick install kit with:
>
> # dd if=downloads/ubuntu-20.04.1-desktop-amd64.iso of=/dev/sdd
> and it worked for me.
>
> I also use to verify the install kit before and after install with:
>
> # cmp downloads/ubuntu-20.04.1-desktop-amd64.iso /dev/sdd
> and this also worked for me until now. It exits with end of file error,
> because the kit is shorter than /dev/sdd.
>
> Now, however, for the first time, there is a difference after install at
> byte 480, line 4.
>
> The kit has been created on a compromised system.
>
> However, I have doubts that it has been modified by malicious code.
>
> So I ran:
>
> # mount /dev/sdd1 mnt
> # mount -o loop ubuntu-20.04.1-desktop-amd64.iso mnt1
> # find mnt/ -exec bash -c 'file={}; cmp $file ${file/mnt/mnt1}' \; | grep
> differ
> and found no difference, only that cmp does not compare directories.
>
> # lsblk -fm /dev/sdd
> NAME FSTYPE LABEL UUID FSAVAIL FSUSE%
> MOUNTPOINT  SIZE OWNER GROUP MODE
> sdd  iso966 Ubuntu 20.04.1 LTS amd64
> │ 2020-07-31-16-51-12-00
> 7,2G root  disk  brw-rw
> ├─sdd1
> │iso966 Ubuntu 20.04.1 LTS amd64
> │ 2020-07-31-16-51-12-00
> 2,6G root  disk  brw-rw
> ├─sdd2
> │vfat C26E-047E
>3,9M root  disk  brw-rw
> └─sdd3
>  ext4   writable
>   a83a9b1c-36cb-4312-9aba-0359f74c0374
> 4,7G root  disk  brw-rw
>

This writable directory was created during installtion.


> What could be the cause? Should I worry about this?
>

No :)

Cheers,
mwh


> Also aked here:
> https://askubuntu.com/questions/1269405/installation-kit-modified-during-install-is-this-a-security-issue
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: pam 1.1.8-3.6ubuntu2.18.04.2 packaging issue

[Michael Hudson-Doyle]

On Sat, 5 Sep 2020 at 06:39, Arnold Czeman (aczeman) <
arnold.cze...@oneidentity.com> wrote:

> Hello,
>
> I have a little problem with the newest version of pam source package on
> Bionic.
> https://bugs.launchpad.net/ubuntu/+source/pam/1.1.8-3.6ubuntu2.18.04.2/
> I would like to import to a git repo with gbp tool, but this package seems
> to have
> a wrong a package format.
> The earlier version of this package were non-native debian packages, they
> had extra
> archives which name match a *.diff.gz or a *.debian.tar.* pattern. They
> had
> two component version string (where a '-' concatenates the upstream
> version and the
> debian version).
>

Right, the package should be non-native. But for some reason it is. All
versions in bionic since release have been native though.


> The current version of this pam package has no extra archives, so it seems
> to be a
> native debian package, but it cannot be native package because it has a
> wrong version
> string format for this package format.
> I have found a description of the debian source package formats here:
>
> https://wiki.debian.org/DebianMentorsFaq#What_is_the_difference_between_a_native_Debian_package_and_a_non-native_package.3F
> I have also checked the gbp tool source code, and it seems to be a
> compatible one with
> this description.
>
> Could you please make a correct version string if you want this will be a
> native package, otherwise
> please generate a *.diff.gz or a *.debian.tar.* archive, just like an
> older version of this
> package!
>

It's back to non-native in focal and groovy. It might make sense to fix it
to be non-native in the next SRU, but it doesn't seem worth performing a
SRU just for this to me.

Cheers,
mwh


> If I can help anything, please let me know!
>
> Thank you in advance,
> Arnold Czeman
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

I started my shift with the welcome news that britney "Will attempt
migration" of icu and the ~million associated packages in proposed but
after a little while it became clear that it was not in fact migrating
these packages.

I wondered if britney was crashing so looked at the logs -- it wasn't, but
I did find the reason (which was also in update_output.txt): migrating
json-c (which is entangled in the icu transition) makes the version of
gce-compute-image-packages in release uninstallable, and the version in
proposed cannot migrate because it depends on google-guest-agent which is
not in main (it does not yet have an MIR).

Fixing this required removing not only gce-compute-image-packages from
proposed but also google-compute-engine-oslogin, because the version of
gce-compute-image-packages in release builds packages that are built by the
version of google-compute-engine-oslogin in proposed (I wonder if this
added confusion is why excuses.html is opaque about this). Anyway, I got
Chris Halse Rogers to remove those and then uploaded a no change rebuild of
the version of gce-compute-image-packages from release, crossed my fingers
and waited.

Then britney started crashing. I filed a MR upstream to at least try to
understand why a bit better:
https://salsa.debian.org/release-team/britney2/-/merge_requests/50

Then icu and a bunch of other packages migrated, but surprisingly not
json-c. I think this was prevented by another package (kamailio)
disappearing from the archive as it was being published from proposed into
updates, and it migrated on the next run. \o/ I then
put gce-compute-image-packages and google-compute-engine-oslogin back.

While all the above was going on, I looked into the emcee failure on
ppc64el a bit and filed a bug upstream:
https://github.com/dfm/emcee/issues/353

Same routine for dired-du/armhf:
https://github.com/calancha/dired-du/issues/4

debian-cloud-images isn't migrating because src:fai needs a merge. It's not
completely trivial so I left it for now.

android-platform-art ftbfs on arm64, this is part of AOSP (already reported
in Debian as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963058).
There are maybe relevant commits upstream but they refer to private bugs in
the AOSP bug tracker so e.

I retried the hugo/armhf test. It migrated.

sphinx was held up by 4 all-platforms "regressions", none of which were
caused by sphinx.
astropy: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966713. I
uploaded a fix for this which didn't work and had another go. will send it
to debian if it works.
beets is already fixed in proposed, so I just need to do some retriggering
same story for python-gmpy2
And the python-pbr tests got bad-tested, so sphinx migrated.

And then my second day of +1 mostly got wiped out by having to look after a
sick child. I worked on some ftbfs stuff, mostly by filing removal bugs.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

I noticed golang-gopkg-square-go-jose.v2 times out on armhf, filed
https://github.com/square/go-jose/issues/326.

Then I looked at the icu transition.

0ad fails to build due to gcc-10, I found the fix for this upstream and
backported it.

I retried some ucto builds which appeared to have run before a build
dependency had been built (maybe fallout from the build farm outage). And
retried "frog" builds when these completed.

doxygen autopkgtests are failing because the json-c in proposed has moved
its Doxyfile.

multipath-tools has britney complaining about impossible depends -- turns
out its udebs are still in main but its dependencies are now in universe.
Apparently a bunch of udebs turned up on component mismatches and got
demoted, but shouldn't have. They got repromoted again and if they appear
on mismatches again someone should debug it :) (or mass demote all udebs to
universe or stop building or at least publishing udebs or or ...)

I noticed that udisks2 was failing autopkgtests, found a bug about this,
found the bug had a comment to a potential fix upstream so I applied the
patch, tested locally and uploaded it.

I tricked xnox into uploading s390-tools-signed, which should unstick
s390-tools.

frr is failing autopkgtests, and has the most unhelpful autopkgtests I've
seen in a while (I have no idea what the tests are expressing). The package
has no rdeps and is the last thing keeping json-c in proposed so maybe
removal is appropriate for now. I filed a bug for this:
https://bugs.launchpad.net/ubuntu/+source/frr/+bug/1890259

libyang is failing to build on riscv64 because of symbols file differences
in libyang-cpp1. I don't understand how to fix this, other than by removing
the symbols file entirely.

dee's tests are timing out on riscv64 :(

libreoffice's tests seem to be very flaky on armhf.

cmake-extras autopkgtest was failing, which I fixed and also submitted
upstream https://github.com/ubports/cmake-extras/pull/14

doxygen's tests fail with the json-c in proposed: it builds the
documentation for json-c in an autopkgtest but json-c has moved it's
Doxyfile into a different directory. Easy to fix once json-c has migrated.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Michael Hudson-Doyle]

On Wed, 22 Jul 2020 at 10:01, Reinhard Tartler  wrote:

>
> Hi Michael,
>
> On Tue, Jun 23, 2020 at 5:55 PM Michael Hudson-Doyle <
> michael.hud...@canonical.com> wrote:
>
>> continuity no longer builds the continuity binary package so I filed a
>> bug to get that removed.
>>
>
> The continuity package in debian continues to build the binary package
> 'golang-github-containerd-continuity-dev', which is a build dependency for
> other packages, such as docker. I'm actually trying to reconcile the
> docker.io packages between debian and ubuntu (see LP#1883978 and
> https://lists.ubuntu.com/archives/ubuntu-server/2020-July/008406.html for
> context), and the lack of an uptodate continuity package is an issue.
>
> Can you please elaborate on the removal? Would it be an issue to sync
> continuity_0.0~git20200107.26c1120-1 from unstable to groovy at this point?
>

Ah yes, I think this was a result of miscommunication, the archive admin
who responded to my bug removed the source package rather than the binary
package. I'm not sure if we can simply sync the package again from debian
-- let's find out :) Hm no it got rejected. This is going to take an AA to
fix I think.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

popcon to be removed from the standard seed

[Michael Hudson-Doyle]

Hi all,

We are planning to remove the “popularity-contest” package from the
standard seed, meaning that new installs will not include it. This package
is intended to report (anonymously) the most installed packages on Ubuntu
systems to the Ubuntu developers. It turns out the package and backend have
both been broken since 18.04 LTS without being much missed so we have
decided to remove the package from the default install and discard the
Ubuntu delta for the package.

Cheers,

mwh

for the Ubuntu Foundations team
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Hi all,

I've been doing +1 maintenance for the last couple of days. I didn't keep
as detailed notes this time. Yesterday, I mostly focused on Go things and
got a few things to migrate. golang-github-hashicorp-memberlist is now
blocking several packages from migrating. I think it's just a very flaky
test but it would be good for someone to at least try to understand what's
going on before badtesting it.

Today, I had the importance of the haskell transition explained to me and
so kicked that along a bit. It took me a while to grok how the haskell
packages work, but I fixed a couple of packages and file removal bugs for
another few. One of the packages I uploaded, agda, ooms during compilation
on arm64, maybe someone can look at that.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Hi,

I had to take a day of my shift off and the other day also ended up being a
bit disrupted so I didn't make as much progress as I would have hoped.

My basic plan was to work on the go packages by scrolling to the bottom of
excuses and searching upwards for "debian go packaging team".

I looked at the nomad build failures on armhf and arm64 and got a bit sad.

golang-gomega was reported as causing a regression in
golang-github-go-redis-redis on s390x. This seems a bit unlikely on the
face of it. golang-gomega itself does not have autopkgtests and builds an
arch: all package so I ran the build on s390x but it passed. I triggered a
test of golang-github-go-redis-redis on s390x in release to see what
happens.

golang-go.uber-atomic reported a regression in golang-go.uber-zap, upstream
acknowledges the test is flaky (https://github.com/uber-go/zap/issues/334)
so I was going to disable it but then noticed that Andreas fixed it better
and his fix had migrated to I retriggered the tests (and on a couple of
other packages it had "regressed" against, golang-github-apex-log and
golang-github-pkg-errors). golang-go.uber-atomic and golang-github-apex-log
migrated.

golang-github-yl2chen-cidranger reported regressions on all arches. It
looks like it needs a newer version of golang-github-stretchr-testify-dev.
It looks like a new version of that has migrated since the tests ran, so I
just retriggered the tests on all architectures. It migrated.

There were versions of both and golang-github-tdewolff-parse and
golang-github-tdewolff-minify in proposed and it looked like they needed to
be tested together, so I triggered that and they both migrated.

golang-yaml.v2 is failing autopkgtests on s390x that was chased down to a
compiler bug that is now fixed upstream, so I backported that and uploaded
it after doing some testing with a build in my PPA.

golang-github-smira-go-aws-auth is failing with a request to route53 timing
out. I suspect this will have regressed in release but will check. If so,
I'll make an upload to skip the test.

golang-github-pkg-errors triggers a regression in golang-github-src-d-gcfg
which is a fork of some other go package and oh god why

golang-github-miekg-dns fails on armhf. Andreas reported this upstream (
https://github.com/miekg/dns/issues/1129) which has a suggestion on it,
which I'll try when I've got my build environment set up on my arm64
canonistack node... The suggestion helps but there is another failure as
well.

https://github.com/gophercloud/gophercloud/issues/1634

continuity no longer builds the continuity binary package so I filed a bug
to get that removed.

golang-github-roaringbitmap-roaring was another package that needed the new
testify, so I retriggered its tests on all arches. If it migrates, it
should be retriggered against some other packages.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

I've been on +1 maintenance over the last two days. I'm not sure I've
achieved anything very significant but oh well. I'll come back tonight to
check on things and probably file a few removal request bugs.

I mostly started at the bottom of excuses and worked my way up.

There are two big messes I saw: rust-* and golang-*. I think the rust
packages can mostly be ignored or deleted but there are some useful
packages held up by the go mess. Maybe at some point one of the people on
the +1 rotation should focus on just that for a couple of days.

I looked at the r transition.
https://people.canonical.com/~ubuntu-archive/transitions/html/html/r-api-combined.html
showed
only three bad packages, all of which were due to rgtk2's failure to build
on s390x. Completely failed to understand the build failure (if you run the
tests a second time in a failing tree, they pass!?). Given the package is
dead upstream, has few rdeps, depends on the obsolete gtk2 and that the
number of people running a graphical r application on s390x must be 0, the
pragmatic way forward is likely to delete these binaries.

Looked into the pcs/python-tornado/mitmproxy knot. Tried a simple upstream
update of mitmproxy but it now depends on
https://pypi.org/project/zstandard/ which does not appear to be packaged.

I fixed sctk's ftbfs and sent the patch to debian (
bugs.debiagolang-github-prometheus-client-golangn.org/962439
).

rocksdb from debian experimental builds fine on Ubuntu. I wonder what's
preventing it from being uploaded to unstable?

I spent a while on qosmic before finding doko's debian bug report had been
moved to another package and realizing that the builds just needed to be
retried.

python-gmpy2 hangs on armhf... need to look into that

There is a knot of go packages related to prometheus held up by what might
be flaky tests on arm64.

postgresql-filedump's autopkgtest was failing on s390x and it turns out
that pg dump files are endian dependent and the test uses one generated on
a little endian system. I uploaded a version that will skip the test on a
big endian system (and send the patch to Debian).

I fixed a couple of build issues in nux and uploaded.

nss-wrapper needs its i386 hint bumping, filed an MP for that.

There is a mess around ndpi / ntopng. Debian started a transition to ndpi 3
some time ago but it ftbfs on some arches and the maintainer seems to have
given up. Some guy called "Dimitri" tried to engage with upstream on the
big endian problems: https://github.com/ntop/nDPI/issues/843

gnuift ftbfs, has for a long time, is orphaned and not in unstable --> we
should remove it

gkrellm2-cpufreq now depends on a package that the debian src:linux builds
but ours does not (libcpupower-dev)

ggcov also seems fairly dead

I fixed python-django-gravatar2's autopkgtest in debian, presumably it will
sync over soon.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Michael Hudson-Doyle]

On Wed, 27 May 2020 at 06:00, Dimitri John Ledkov  wrote:

> On Tue, 26 May 2020 at 11:02, Michael Hudson-Doyle
>  wrote:
> >
> > Hi,
> >
> > I've spend today working on +1 maintenance (
> https://wiki.ubuntu.com/PlusOneMaintenanceTeam,
> https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status).
> >
> > We have a few transitions ongoing (gsl, hdf5, perl, ...) which are
> showing signs of all getting entangled together.
> >
> > I re-uploaded haskell-hmatrix-gsl which had been uploaded too early
> (before gsl has been published on riscv64). It built fine.
> >
> > I found some more blockers for the gsl transition:
> https://people.canonical.com/~ubuntu-archive/transitions/html/html/auto-gsl.html.
> The cpl-* ones are semi-typical mysterious failures for a new port,
> probably down to bugs in our emulated builders. The ea-utils one is a bit
> of a mess though, in some ways: it should never have built in the first
> place! There is a package (pegjs) in focal and groovy release that
> Provides: nodejs. This means that things that transitively build-depend on
> nodejs don't end up in depwait like they should. There is a pegjs in groovy
> proposed that fixes this, I guess it needs to be forced to migrate and any
> binaries this makes uninstallable removed (it's not like packages which
> depend on nodejs and are erroneously installable are actually going to
> _work_).
> >
>
> That's a nice triange of nodejs / pegjs. Did you open an RM bug report
> against pegjs, and subscribe ~ubuntu-archive to it, such that they can
> process the binary-only removal on riscv64?
>

No because I wasn't sure what the best approach was. It would also be
possible to force pegjs to migrate to release and remove things that become
uninstallable on riscv as a result.


> As documented at
> https://wiki.ubuntu.com/UbuntuDevelopment/PackageArchive#Removing_Packages
>
>
> > I fixed pbsuite's autopkgtest and sent a patch off to Debian, this has
> migrated and been applied to the Debian repo already.
> >
> > I looked at the pbcopper ftbfs on armhf and ppc64el. Debian has a report
> about the armhf failure and the packaging repo on salsa already has changes
> to disable the build on 32 bit architectures so I guess we should follow
> that. The ppc64el failure was strange and I spent probably too long coming
> to the conclusion that it's a bug in the "simde" header library Debian uses
> to compile the x86-intrinsic-using code on other architectures:
> https://github.com/nemequ/simde/issues/325. I've uploaded a workaround
> and sent it to Debian.
> >
> > I'm doing this again tomorrow, I expect I'll mostly keep on picking at
> proposed migration.
>

This is mostly what I've been doing. I don't remember exactly which
packages I poked at, I'm afraid.

pbcopper got through new (it has a new library name) but its rdep pbbam
ftbfs (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959643). There's a
new upstream version, 1.0.7 which does build but it might be best to wait
to see what Debian does before moving in Ubuntu. My pbbam package is in my
PPA at https://launchpad.net/~mwhudson/+archive/ubuntu/devirt if someone
wants to take it and upload it to the main archive (the other deps in this
mini transition, pbseqlib and blasr, appear to build fine with this new
pbbam).

One thing I did a lot of was retrying of things with all-proposed=1 on the
end. I thought that autopkgtest was supposed to do that by itself when a
test dependency wasn't installable? Did that regress somehow?

I spent quite a long time looking into the r-cran-lwgeom failures. Steve
removed the upstream version that appears to have endianness issues (
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961211) and patched it to
build. But the autopkgtests still failed. I've cribbed a patch by looking
at some upstream changes which passes locally but I think in general
upgrading to the new upstream version would be a good idea.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

+1 maintenance report

[Michael Hudson-Doyle]

Hi,

I've spend today working on +1 maintenance (
https://wiki.ubuntu.com/PlusOneMaintenanceTeam,
https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status).

We have a few transitions ongoing (gsl, hdf5, perl, ...) which are showing
signs of all getting entangled together.

I re-uploaded haskell-hmatrix-gsl which had been uploaded too early (before
gsl has been published on riscv64). It built fine.

I found some more blockers for the gsl transition:
https://people.canonical.com/~ubuntu-archive/transitions/html/html/auto-gsl.html.
The cpl-* ones are semi-typical mysterious failures for a new port,
probably down to bugs in our emulated builders. The ea-utils one is a bit
of a mess though, in some ways: it should never have built in the first
place! There is a package (pegjs) in focal and groovy release that
Provides: nodejs. This means that things that transitively build-depend on
nodejs don't end up in depwait like they should. There is a pegjs in groovy
proposed that fixes this, I guess it needs to be forced to migrate and any
binaries this makes uninstallable removed (it's not like packages which
depend on nodejs and are erroneously installable are actually going to
_work_).

I fixed pbsuite's autopkgtest and sent a patch off to Debian, this has
migrated and been applied to the Debian repo already.

I looked at the pbcopper ftbfs on armhf and ppc64el. Debian has a report
about the armhf failure and the packaging repo on salsa already has changes
to disable the build on 32 bit architectures so I guess we should follow
that. The ppc64el failure was strange and I spent probably too long coming
to the conclusion that it's a bug in the "simde" header library Debian uses
to compile the x86-intrinsic-using code on other architectures:
https://github.com/nemequ/simde/issues/325. I've uploaded a workaround and
sent it to Debian.

I'm doing this again tomorrow, I expect I'll mostly keep on picking at
proposed migration.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: How to handle tmpfiles.d in non-systemd environments

[Michael Hudson-Doyle]

On Tue, 18 Feb 2020 at 13:37, Robie Basak  wrote:

> On Tue, Feb 18, 2020 at 10:31:40AM +1300, Michael Hudson-Doyle wrote:
> > On the face of it, the package is buggy. tmpfiles configs are processed
> by
> > systemd, the package depends on the tmpfiles config being processed -> it
> > should Depend: on systemd. This is not a path forward to having it work
> in
> > a docker container though.
>
> This is a good point. I believe this has been discussed in Debian, and
> that there was an argument that it's better not to enforce that via a
> dependency for now since doing so would preclude alternative init
> systems from being usable. I don't know if that discussion concluded.
>

There was some discussion along some of these lines, yes. One approach
would be to have several packages to use alternatives to provide
/bin/tmpfiles (and Provide: tmpfiles in their control file). But that
hasn't happened yet.


> For Ubuntu specifically, I believe it's not buggy because we can assume
> that systemd is present.
>

systemd is not present in docker containers!


> As you said though, this is largely unrelated to the Docker use case we
> want to enable.
>
> I commented in the bug about that.
>

I think my previous email outlined a possible solution. I could copy my
idea to the bug I guess...

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: How to handle tmpfiles.d in non-systemd environments

[Michael Hudson-Doyle]

On Mon, 17 Feb 2020 at 21:34, Christian Ehrhardt <
christian.ehrha...@canonical.com> wrote:

> Hi,
> There is a bug [1] showcasing an issue failing in a docker container as
> tmpfiles.d there does nothing.
>
> On 6th of December I already wrote to ubuntu-server as part of my daily
> bug duties and subscribed people to the bug. Not much happened since then,
> a few small IRC exchanges and the Debian init system GR was resolved, but
> other than that I didn't hear anything else on the topic.
>
> This class of issues we should IMHO solve/ignore/handle in a general and
> not package specific fashion.
>
> I was wondering if anyone on the core-system or systemd side of things has
> already thought/decided on that class of issues and could chime in on the
> bug.
>
> To everyone - have you seen similar bugs that we should Dup' together in
> regard to this?
>
> [1]: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1855140
>

Fun.

On the face of it, the package is buggy. tmpfiles configs are processed by
systemd, the package depends on the tmpfiles config being processed -> it
should Depend: on systemd. This is not a path forward to having it work in
a docker container though.

One way to make this work might be this:

 1. have src:systemd make a systemd-tmpfiles binary package that just ships
systemd-tmpfiles (upstream has made commitments that systemd-tmpfiles does
not depend on systemd running:
https://lists.debian.org/debian-devel/2019/12/msg00060.html).

 2. Have any package that ship tmpfiles configs depend on this new package
and invoke systemd-tmpfiles in postinst (debhelper / dh_installsystemd does
in fact arrange for systemd-tmpfiles to be invoked in postinst but only
if /run/systemd/system exists, so we'd change that to be unconditional and
get something to add systemd-tmpfiles to misc:Depends).

This is a bit sketchy because nothing will process the tmpfiles fragments
when the container starts, but /run and /tmp and just regular directories
in a docker container (by default anyway) so it might just work (tm).

We should probably coordinate this with Debian. The recent thread
"opentmpfiles & opensysusers, and its use in the Debian policy" is somewhat
related, maybe I should fork a thread off that.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Details of what is happening with python3.8 in Ubuntu?

[Michael Hudson-Doyle]

Hi Seb,

On Tue, 22 Oct 2019 at 02:27, Sebastien Bacher  wrote:

> Hey there,
>
> Matthias announced that F would have python3.8 and from the recent
> upload, it looks like that's being worked on actively at the moment.
>

Yes. To be clear, "F would have python3.8 (at archive opening)" means
having python3.8 as a supported version, not just having 3.8 available
(eoan has 3.8 available).


> While looking at the update_excuses_by_team.html#desktop-packages report
> I noticed some delta-over-debian being added to packages where I
> can't really make sense of what's going on in the changelog.
>
> Some examples
>
> http://launchpadlibrarian.net/447713946/pygobject_3.34.0-1build1_3.34.0-1ubuntu1.diff.gz
>
> http://launchpadlibrarian.net/447727952/pycairo_1.16.2-1ubuntu1_1.16.2-1ubuntu3.diff.gz
>
> http://launchpadlibrarian.net/447772089/cracklib2_2.9.6-2build1_2.9.6-2ubuntu1.diff.gz
>
> Some changes for example add a Build-Depends on dh-exec and hacks in the
> .install, I guess those doing the changes understand why but it triggers
> some questions to me
>
> - Could someone explain why those .install tricks are needed exactly?
>

It's because the "abi tag" for python c extensions has changed in a way
that makes it a bit annoying / impossible to write a glob that matches 3.7
and 3.8 release build extensions and not the debug extensions.


> Couldn't the issue be solved in the python packaging tools instead?
>

Yes, probably.

- Can/should those changes be forwarded to Debian? (they are not at the
> moment?) I would be happy to help with that once I understand the
> technical approach being taken.
>

I've made a bug report for cracklib2 now.

- Is that a transition we are under pressure to get through?


Well we want it to complete before opening the archive and I assume people
want the archive to open...


> Should we
> maybe take the time to document what's going on


Well there was this mail:
https://www.mail-archive.com/ubuntu-devel@lists.ubuntu.com/msg09144.html

And there is a transition tracker:
https://people.canonical.com/~ubuntu-archive/transitions/html/python3.8-add.html
(not
sure this has been advertised but it's not hard to find).

and have more people
> helping and do things in a way were we collectively understand what is
> happening


Maybe roping in more people would help, maybe not. The plan was for
Matthias and I to work on this more or less full time and just crank
through it.


> so we know what to do from those deltas in the futur/ help
> forwarding to Debian for example?
>

While in general we should certainly forward things to Debian I do think
the people who end up doing this transition in Debian in general know to
look for potential fixes in Ubuntu (when the changes will presumably make
more sense if those people are diagnosing the same or similar build
failures). And some of them may not be necessary, if like you say dh-python
can grow smarts to make installing the right extensions into the right
packages more easily.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: On the py2 topic: DEP8 tests

[Michael Hudson-Doyle]

On Thu, 26 Sep 2019 at 09:42, Robie Basak  wrote:

> On Wed, Sep 25, 2019 at 02:33:35PM -0700, Steve Langasek wrote:
> > It should be possible to track this by looking at the contents of the
> > Testsuite-Triggers field in Sources.gz.
>
> I wonder if it would be appropriate for archive admins to check this
> before package removals as they do for reverse dependencies? I
> understand that the archive won't end up "broken" by the same
> definition, but it will make the autopkgtest situation in the archive
> worse.
>
> Maybe, to help, the "reverse-depends" tool needs adjusting?
>

In progress, and apparently fixed if you use ubuntu-dev-tools from git?
https://bugs.launchpad.net/reverse-depends/+bug/1843614

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: New Ubuntu Core Developer - Thomas Ward

[Michael Hudson-Doyle]

Congrats!!

Cheers,
mwh

On Mon, 17 Jun 2019, 21:01 Lukasz Zemczak, 
wrote:

> Hello everyone,
>
> Please congratulate teward on his today's successful core-dev application!
> Welcome to the team!
>
> Cheers,
>
> --
> Łukasz 'sil2100' Zemczak
>  Foundations Team
>  lukasz.zemc...@canonical.com
>  www.canonical.com
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: mlocate - what is it good for?

[Michael Hudson-Doyle]

On Thu, 23 May 2019 at 09:07, Brian Murray  wrote:

> The Ubuntu Foundations team was recently looking at an issue with
> mlocate[1] and the effect it has on all users of Ubuntu. While that
> specific issue is fixable there are also issues[2,3] with keeping
> PRUNEFS and PRUNEPATHS current in updatedb.conf. So we ended up
> questioning the usefulness of installing mlocate by default on systems
> at all. We believe that find is an adequate replacement for mlocate but
> want to hear from you about use cases where it may not be.


I think this is a cattle vs pet sort of thing as well: on a system that
being used for development or adminned by multiple humans, locate is quite
useful. And the overhead of having to install it for systems like this
seems acceptable (users like these are exactly the sort of users for whom
installing a package is easy!).

On systems that are part of a fleet of automatically maintained machines,
all locate does is waste IO bandwidth -- which might well be a shared
resource if the systems are VMs.

My vote would be for removing it from the default install.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

the in-progress transition, icu, arm64 and nans

[Michael Hudson-Doyle]

Hi,

By my reading, the single thing blocking the migration of a whole bunch of
packages out of disco-proposed is the autopkgtest failure of symfony on
arm64.

Unfortunately, this failure is not a flaky test: it is due to a behaviour
change in ICU.

On disco-proposed:

root@Mcdivitt-B0-Cartridge37:/# cat int64test.c
#include 
#include 
#include 

int main(int argc, char** argv) {
UErrorCode status;
UNumberFormat* nf = unum_open(UNUM_DEFAULT, NULL, -1, NULL, NULL,
);
UChar u_str[1000];
int32_t length;
u_strFromUTF8(u_str, 1000, , argv[1], strlen(argv[1]),
);
int64_t r = unum_parseInt64(nf, u_str, length, NULL, );
printf("result %ld status %d\n", r, U_FAILURE(status));
}
root@Mcdivitt-B0-Cartridge37:/# gcc -o int64test int64test.c  -licuuc
-licui18n
root@Mcdivitt-B0-Cartridge37:/# ./int64test nan
result 0 status 0

Compiling the same source without proposed enabled:

ubuntu@Mcdivitt-B0-Cartridge37:~$ ./int64test nan
result 0 status 1

FWIW, ./int64test NaN behaves like disco-proposed always (on arm64...), and
symfony has special code handling "NaN", so a reasonable expedient fix is
probably to patch symfony to not be case sensitive there.

Given that this only seems to happen on arm64 and not, say, amd64, my guess
is that it's also an ICU bug but ICU is pretty confusing and documentation
on intended behaviour of how NaN should be handled by number parsing seems
to be non-existent.

Hope this helps someone.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Just some observations on first time install of 18.10

[Michael Hudson-Doyle]

On Sun, 28 Oct 2018 at 10:45, Jim Pye  wrote:

> All
>
> Not bugs, but some observations on the new installer of Ubuntu 18.10 Server
>
> Using .iso in a VirtualBox VM on a Mac
>
> Not sure of the etiquette of attaching screen shots, but have them if
> needed.
>
> 1. Getting to step 7 of 11 in the install, screen showing choice of snaps,
> I noticed the install was continuing in the progress bar at bottom of
> screen, cool. However, as I was looking for things like Apache, Puppet etc.
> The install completed, so when I pressed done it took me directly to the 11
> of 11 screen, the Install Complete! screen and I am not sure what I missed
> on screens 8, 9, and 10 (maybe choosing install of Apache which is not
> snapped yet?)
>

Hmm, I've not seen that. You didn't miss anything though, there are no
screens after the snap screen :)


> 2. On screen 11 of 11 there were two "Reboot Now" options in the choice of
> three, the other being "View Full Log".
>

https://bugs.launchpad.net/subiquity/+bug/1783119 -- I can't reproduce this
one either though (it's actually possible this and the previous one are
related).


> 3. After rebooting I was presented with the Login prompt after all the
> usual startup.  However, before I could enter my details there seemed to be
> a lot (nearly two screenfuls) of additional output from services starting,
> stopping and restarting, ssh keygen stuff etc. which meant the login prompt
> scrolled off and was hidden/gobbled up.  As this was ssh keys etc. I assume
> this was a once only effect, but it might need a bit more tidy up in the
> startup dependency area before the bash login prompt is displayed.
>

https://bugs.launchpad.net/subiquity/+bug/1778226


> As I said not bugs, just observations.
>

Cheers,
mwh


> Kia Ora
> Jim
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: uploading golang-1.10 everywhere

[Michael Hudson-Doyle]

On Thu, 27 Sep 2018 at 06:29, Jamie Strandboge  wrote:

> On Wed, 26 Sep 2018, Michael Hudson-Doyle wrote:
>
> > Hi all,
> >
> > At the recent sprint in Brussels, there was some discussion of providing
> a way
> > for packages in older Ubuntu releases a way to build with a newer
> version of
> > Go. After some discussion with the security team, it was decided that it
> made
> > sense to upload Go 1.10 (which is after all going to be supported as
> part of
> > Bionic for 5 years) to the older releases. I'm mentioning this plan here
> (a)
> > for awareness (b) to get the security team to publicly acknowledge their
> > agreement to the plan :)
> >
> > There is a bug to track the uploads:
> >
> > https://bugs.launchpad.net/ubuntu/+source/golang-1.10/+bug/1794395
> >
> > All comments here or in the bug gratefully received.
> >
> This is fine with the security team (if it was a golang version that was
> in an interim release, that would be a different story, but we are
> already supporting this in bionic, so why not in earlier?). I commented
> in the bug too.
>

Thanks, I've uploaded golang-1.10 to the various UNAPPROVED queues now.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

uploading golang-1.10 everywhere

[Michael Hudson-Doyle]

Hi all,

At the recent sprint in Brussels, there was some discussion of providing a
way for packages in older Ubuntu releases a way to build with a newer
version of Go. After some discussion with the security team, it was decided
that it made sense to upload Go 1.10 (which is after all going to be
supported as part of Bionic for 5 years) to the older releases. I'm
mentioning this plan here (a) for awareness (b) to get the security team to
publicly acknowledge their agreement to the plan :)

There is a bug to track the uploads:

https://bugs.launchpad.net/ubuntu/+source/golang-1.10/+bug/1794395

All comments here or in the bug gratefully received.

Cheers,
mwh

PS: although it's always hard to predict the future, the plan as it is
today is also to backport whatever version of Go ends up in 20.04 LTS to
the releases that are still supported at that time, etc.
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: glibc 2.28 autopkgtest failures

[Michael Hudson-Doyle]

On Tue, 4 Sep 2018 at 13:26, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:


> The bug is that when you lie to cffi about the return type of a variable,
> in particular when you say a function returning a float does not return a
> float, the x87 stack is not wiped to the extent required by the calling
> convention. glibc 2.28 rewrote the cos function in a way that apparently
> now cares about this (or I guess gcc now compiles the function in a way
> that cares about this) but make no mistake, it only ever worked by chance
> before. So I think the test_sin_no_return_value test should be skipped on
> i386. (I'll file an upstream issue I guess).
>

https://bitbucket.org/cffi/cffi/issues/382/test_sin_no_return_value-violates-calling


> Cheers,
> mwh
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

glibc 2.28 autopkgtest failures

[Michael Hudson-Doyle]

tl;dr we should force things through :)

There are at the time of writing two autopkgtest failures preventing glibc
from migrating: r-cran-rgenoud and python-cffi

The r-cran-genoud failure is mystifying in that I have no idea how the
glibc version is affecting the package (it doesn't call many libc or libm
functions!). I've spent far too long trying on this but as the package has
no rdeps we should probably just remove it and whoever brings it back can
try to understand why it changes behaviour with new libc. (The test also
looks a bit fragile, it is asserting that optimizing a particular function
with the rng seeded a particular way finds a different maxima to seeding
the rng a different way, afaict).

The python-cffi is exposing a bug that exists already. A minimal failing
example is this:

root@cosmic32:~# cat n.py
import math
from cffi.backend_ctypes import CTypesBackend
from cffi import FFI
ffi = FFI(backend=CTypesBackend())
ffi.cdef(""" void fabs(double x); """)
ffi.dlopen('m').fabs(1.0)
print(math.cos(1.23))
root@cosmic32:~# python3 n.py
Traceback (most recent call last):
  File "n.py", line 7, in 
print(math.cos(1.23))
ValueError: math domain error

The bug is that when you lie to cffi about the return type of a variable,
in particular when you say a function returning a float does not return a
float, the x87 stack is not wiped to the extent required by the calling
convention. glibc 2.28 rewrote the cos function in a way that apparently
now cares about this (or I guess gcc now compiles the function in a way
that cares about this) but make no mistake, it only ever worked by chance
before. So I think the test_sin_no_return_value test should be skipped on
i386. (I'll file an upstream issue I guess).

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: python3-numpy depending on *both* python 3.6 and 3.7

[Michael Hudson-Doyle]

On Sun, 26 Aug 2018 at 23:48, Dmitry Shachnev  wrote:

> On Sun, Aug 26, 2018 at 10:01:38PM +1200, Michael Hudson-Doyle wrote:
> > So the problem is that python3-numpy contains a version of 'f2py' for
> each
> > supported version of Python. I guess the proper fix involves creating a
> > separate package for the each version of f2py? python3.6-f2py,
> > python3.7-f2py, and have python3-numpy depend on the one for the default
> > version of Python?
>
> This will mean having a new package name whenever we add a new supported
> Python version. It will be very inconvenient to maintain.
>
> Better remove the versioned scripts altogether and use “python3.x -m
> numpy.f2py” when one really needs a non-default Python version.
>
>
Yes, that's probably a better plan.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: python3-numpy depending on *both* python 3.6 and 3.7

[Michael Hudson-Doyle]

On Sat, 25 Aug 2018 at 05:07, Steve Langasek 
wrote:

> Hi Andreas,
>
> On Fri, Aug 24, 2018 at 10:05:23AM -0300, Andreas Hasenack wrote:
> > Hi,
> >
> > while investigating some DEP8 failures currently in cosmic's
> > migration, I came across this:
> > $ dpkg -s python3-numpy|grep Depends
> > Depends: python3 (<< 3.8), python3 (>= 3.6~), python3.6:any,
> > python3.7:any, python3:any (>= 3.3.2-2~), libblas3 | libblas.so.3,
> > libc6 (>= 2.27), liblapack3 | liblapack.so.3
> >
> > Is it ok/correct to depend on two python versions like that? Is the
> > point of it making sure numpy is available regardless which python 3
> > you are using? But at the cost of pulling in both?
> >
> > This is breaking python3-libcloud, which does not support python3.7,
> > when pulled in via fdroidserver:
> > https://pastebin.ubuntu.com/p/Kn77DXfxR5/
> >
> > The correct fix is to have python3-libcloud work with python 3.7, by
> > replacing "async" (a reserved keyword in python3.7) with something
> > else, like async_, but what caught my eye was this numpy dependency in
> > two python versions. And how libcloud ended up chosing 3.7 over 3.6
> > I'm not sure.
>
> This issue was reported in Debian during the previous python transition,
> and
> was closed without a permanent resolution:
>
>  https://bugs.debian.org/878281
>
> And a bug is now open again about the same issue wrt the python3.7
> transition:
>
>  https://bugs.debian.org/903663
>
> I agree that the current behavior is incorrect and not what we expect from
> packages that support multiple versions of python3.
>

So the problem is that python3-numpy contains a version of 'f2py' for each
supported version of Python. I guess the proper fix involves creating a
separate package for the each version of f2py? python3.6-f2py,
python3.7-f2py, and have python3-numpy depend on the one for the default
version of Python?

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: libv8-3.14-dev

[Michael Hudson-Doyle]

On 18 May 2018 at 21:58, jaroslav.svoboda 
wrote:

> Hi,
>
> is there a reason why libv8-3.14-dev and libv8-3.14 packages are not
> available for arm64 (particularly in Bionic)? I read there was an issue few
> years ago when V8 did not recognized ARMv8 but that should be fixed by now.
> https://packages.ubuntu.com/bionic/arm64/libv8-3.14-dev just does not
> exist even though search links there.
>
>
v8 *3.14* is very old and I think does not support arm64. My understanding
is that v8 moves too quickly to be sanely packageable in either Debian or
Ubuntu.

Cheers,
mwh
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Golang Package.

[Michael Hudson-Doyle]

It's up to date in Bionic. If you want a newer release in an older Ubuntu,
there is this PPA: https://launchpad.net/~gophers/+archive/ubuntu/archive
or you can use the snap (snap install --classic --channel 1.10/stable go).

Cheers,
mwh

On 22 February 2018 at 11:36, Mike Lloyd  wrote:

> Hey team. I've noticed the golang package is always several major releases
> behind. What is the process for getting this package updated? I'd like to
> help if I can.
>
> Mike.
>
> Get Outlook for iOS 
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/ubuntu-devel-discuss
>
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: linting wrapper for dput

[Michael Hudson-Doyle]

On 6 September 2017 at 08:14, Brian Murray <br...@ubuntu.com> wrote:

> On Fri, Sep 01, 2017 at 12:54:02PM +1200, Michael Hudson-Doyle wrote:
> > Hi all,
> >
> > If, like me, you spend time preparing uploads for ppas, the ubuntu
> > development release, SRUs and/or debian, sometimes you no doubt screw up
> > and upload a package without ~ppa1 to your ppa or upload a package with
> > version X.Y~17.04 to xenial (at least, I do this sort of thing quite a
> > lot). I've written a wrapper around dput that checks for many of these
> > mistakes before invoking real dput:
> >
> > https://gist.github.com/mwhudson/616499edb1191bd99c987bbbd8781ce9
> >
> > (it also checks that for SRU bugs the listed bugs have open tasks for the
> > appropriate distro / source packages).
>
> One problem the SRU team sometimes encounters is bugs linked in SRUs
> being private, perhaps you could also check for those?
>

Ah yes, that makes sense. A very lazy way of doing this would just be to
log in to Launchpad anonymously I guess!

I did that, and created a repo on github for this:
https://github.com/mwhudson/dput-wrap/tree/master



Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: The transition to add support for Python 3.6 is beginning

[Michael Hudson-Doyle]

On 12 July 2017 at 11:30, Barry Warsaw <ba...@ubuntu.com> wrote:

> On Jul 12, 2017, at 07:33 AM, Michael Hudson-Doyle wrote:
>
> >Yeah, it's all good so long as we don't rebuild lazr.delegates thanks to
> >https://github.com/pypa/setuptools/issues/912 related fun...
>
> Too true. :(
>
> I haven't yet investigated the suggested linked to that issue, but if you
> have
> the time and inclination, you might see if this helps.  E.g. is lazr.*
> using
> these inconsistently?
>
> https://bitbucket.org/ambv/configparser/issues/17/#comment-36669436


Well they both do this:

try:
import pkg_resources
pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)

But both python3-lazr.delegates and python3-lazr.config Depend: on
python3-pkg-resources, so the ImportError case should never occur...

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: The transition to add support for Python 3.6 is beginning

[Michael Hudson-Doyle]

On 30 June 2017 at 03:12, Barry Warsaw  wrote:

> On Jun 28, 2017, at 07:22 PM, Barry Warsaw wrote:
>
> >I want to keep lazr.config in the archive, and I help maintain it both in
> >Debian and upstream.
>
> I'd forgotten that I had released a new upstream version a while ago, and
> had
> an update sitting in my Debian packaging git repo waiting for the Stretch
> release.  I just uploaded lazr.config 2.2-1 to unstable so after this winds
> its way through the system, it should be all good.
>

Yeah, it's all good so long as we don't rebuild lazr.delegates thanks to
https://github.com/pypa/setuptools/issues/912 related fun...

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: The transition to add support for Python 3.6 is beginning

[Michael Hudson-Doyle]

On 29 June 2017 at 11:22, Barry Warsaw <ba...@ubuntu.com> wrote:

> Thanks for working on this Michael.


Thanks for looking into it!


>
> On Jun 29, 2017, at 10:43 AM, Michael Hudson-Doyle wrote:
>
> >The 100 or so failures are now summarised in
> >https://docs.google.com/spreadsheets/d/1Y8dy5cyu8DrmTvT4CNSaVSP2ZZT79
> sYhT_rAQ19bFUQ/edit?usp=sharing,
> >please do check if any packages you care about are on the list and fix any
> >that you can see how to. Uploading fixes direct to the archive is
> preferred,
> >but if you lack the rights to do that, attaching a bug to a debdiff and
> >subscribe me (mwhudson on lp) and I'll sponsor it v. quickly!
>
> What's the process for claiming a package?


Good question. Making a comment makes sense. I've just sent you an
invitation to edit, and am happy to do so for anyone else who is interested.


> I want to keep lazr.config in the
> archive, and I help maintain it both in Debian and upstream.  I'm
> motivated to
> keep it working as it will soon be a dep of mailman3-core which should be
> entering unstable soon-ish.
>

Fair enough, I've removed it's threat of removal :-)


> I'll probably look at nose and pyflakes as part of my normal DD workflow.
>

I've uploaded the new pyflakes to Ubuntu but if you upload it to unstable &
sync it that would make me happy :)


> I'm not interested in system-image any more, but if someone else adopts it,
> I'm happy to answer questions.
>
> I've added some comments to the spreadsheet.
>

Thanks!

Cheers
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: The transition to add support for Python 3.6 is beginning

[Michael Hudson-Doyle]

On 21 June 2017 at 15:13, Michael Hudson-Doyle <michael.hud...@canonical.com
> wrote:

> Hi all,
>
> An update on the transition to Python 3.6: Python 3.6 is now a supported
> version in artful release, and almost all packages that build C extensions
> have been rebuilt (pandas is still a problem).
>
> We have created a PPA where python3.6 is the default and rebuilt all
> python packages: https://launchpad.net/~canonical-foundations/+
> archive/ubuntu/python3.6-as-default/+packages and the next step is to fix
> all the failures this reveals.  The initial failing source packages are
> listed in http://paste.ubuntu.com/24903638/ although some of those have
> been fixed now.
>

The 100 or so failures are now summarised in
https://docs.google.com/spreadsheets/d/1Y8dy5cyu8DrmTvT4CNSaVSP2ZZT79sYhT_rAQ19bFUQ/edit?usp=sharing,
please do check if any packages you care about are on the list and fix any
that you can see how to. Uploading fixes direct to the archive is
preferred, but if you lack the rights to do that, attaching a bug to a
debdiff and subscribe me (mwhudson on lp) and I'll sponsor it v. quickly!

Cheers,
mwh

Cheers,
> mwh
>
> On 12 May 2017 at 11:29, Michael Hudson-Doyle <
> michael.hud...@canonical.com> wrote:
>
>> Hi everyone,
>>
>> The process of adding Python 3.6 as a supported version has begun.
>>
>> The transition tracker is here:
>>
>> https://people.canonical.com/~ubuntu-archive/transitions/htm
>> l/python3.5-6.html
>>
>> and I'm currently working my way down the list.
>>
>> I did a test rebuild of all/most dependent packages in a ppa:
>>
>> https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages
>>
>> And compiled some terse notes on the failures here:
>>
>> http://paste.ubuntu.com/24557408/
>>
>> And help resolving the failures would be appreciated (especially pandas!)
>>
>> Cheers,
>> mwh
>>
>
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: The transition to add support for Python 3.6 is beginning

[Michael Hudson-Doyle]

Hi all,

An update on the transition to Python 3.6: Python 3.6 is now a supported
version in artful release, and almost all packages that build C extensions
have been rebuilt (pandas is still a problem).

We have created a PPA where python3.6 is the default and rebuilt all python
packages:
https://launchpad.net/~canonical-foundations/+archive/ubuntu/python3.6-as-default/+packages
and the next step is to fix all the failures this reveals.  The initial
failing source packages are listed in http://paste.ubuntu.com/24903638/
although some of those have been fixed now.

Once the failures are accounted for, we can flip the switch to make python
3.6 the default in the archive.

Cheers,
mwh

On 12 May 2017 at 11:29, Michael Hudson-Doyle <michael.hud...@canonical.com>
wrote:

> Hi everyone,
>
> The process of adding Python 3.6 as a supported version has begun.
>
> The transition tracker is here:
>
> https://people.canonical.com/~ubuntu-archive/transitions/
> html/python3.5-6.html
>
> and I'm currently working my way down the list.
>
> I did a test rebuild of all/most dependent packages in a ppa:
>
> https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages
>
> And compiled some terse notes on the failures here:
>
> http://paste.ubuntu.com/24557408/
>
> And help resolving the failures would be appreciated (especially pandas!)
>
> Cheers,
> mwh
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

The transition to add support for Python 3.6 is beginning

[Michael Hudson-Doyle]

Hi everyone,

The process of adding Python 3.6 as a supported version has begun.

The transition tracker is here:


https://people.canonical.com/~ubuntu-archive/transitions/html/python3.5-6.html

and I'm currently working my way down the list.

I did a test rebuild of all/most dependent packages in a ppa:

https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages

And compiled some terse notes on the failures here:

http://paste.ubuntu.com/24557408/

And help resolving the failures would be appreciated (especially pandas!)

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: How to push a security update to Xenial and Zesty

[Michael Hudson-Doyle]

Apologies for the terseness but are you looking for
https://wiki.ubuntu.com/StableReleaseUpdates ?

On 16 December 2016 at 23:49, Roman Tsisyk  wrote:

> Hi everyone,
>
> I'm upstream and Debian maintainer of `tarantool` package [1][2].
> We received two CVE notifications yesterday.  A fix is already in git repo
> and I plan to push an update to Debian Unstable today.
>
> How I can push a security update for my package to xenial and yakkety?
>
> http://packages.ubuntu.com/search?keywords=tarantool;
> searchon=names=all=all
> https://packages.debian.org/search?keywords=tarantool;
> searchon=names=all=all
>
> xenial (16.04LTS) (database): In-memory database with a Lua application
> server [universe]
> 1.6.7.588.g76bbd9c-1build1: amd64 i386
>
> yakkety (16.10) (database): In-memory database with a Lua application
> server [universe]
> 1.6.8.740.g45fc77f-1build1: amd64 arm64 armhf i386
>
> zesty (database): In-memory database with a Lua application server
> [universe]
> 1.7.2.314.gedffdc0-1: amd64 arm64 armhf i386
>
> stretch (testing) (database): In-memory database with a Lua application
> server
> 1.7.2.314.gedffdc0-1: amd64 arm64 armhf i386
>
> sid (unstable) (database): In-memory database with a Lua application server
> 1.7.2.314.gedffdc0-1: amd64 arm64 armhf i386
>
> I have the fixes both for 1.6.x and 1.7.x series and it is possible to
> upgrade xenial users without breaking changes.
>
> Could you please help me? Thanks.
>
> [1]: https://tracker.debian.org/pkg/tarantool
> [2]: https://github.com/tarantool/tarantool
>
> --
> WBR,
>   Roman Tsisyk 
>   http://tarantool.org/ - an efficient in-memory data store and a Lua
> application server
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: git workflows for general Ubuntu development

[Michael Hudson-Doyle]

On 15 November 2016 at 01:51, Robie Basak  wrote:

> On the server team, we've been working on a process that uses git to do
> our "Ubuntu merges". As a consequence, we now have a mechanism that can
> import package histories into git on Launchpad. We think that this work
> opens up a bunch of new possibilities, such as for drive-by contributors
> submitting merge proposals entirely through git.
>

This is pretty exciting! Do you think your work will fulfil the goals of
the UDD project or is there still some stuff that's out of scope?


> We'll be talking about our work tomorrow, as part of the Ubuntu Online
> Summit. The session page is at
> http://summit.ubuntu.com/uos-1611/meeting/22710/git-based-merge-workflow/
> and is currently scheduled for 2016-11-15 18:00 UTC. Times may change,
> so be sure to check the schedule tomorrow.
>
> I originally wrote about this in August 2014
> (https://lists.ubuntu.com/archives/ubuntu-devel/2014-August/038418.html).
> We've come on a long way since then. Our automated git imports preserve
> histories. Where relevant, Ubuntu packages are correctly parented from
> their Debian origins. We hope to start running this live soon, which
> will import package uploads into our git trees as they happen rather
> than on-demand. Once this is live, anyone will be able to easily clone
> from the "current" Ubuntu packaging git trees, which we think is useful
> in itself.
>
> If you're interested, we'd love your feedback. What are your use cases?
> What sort of workflows would you like to see? You can see some further
> notes of ours as they form in the pad on the session page linked above.
> What have we missed?
>

The two questions I have (which are touched on but not afaics really
answered in the notes are) 1) how does this work if I already maintain the
packaging for some package in git? 2) what about dgit?


> Note that developer time, as always, is limited. We've developed what we
> have so far to speed up our own work. Our use cases are probably quite
> different from non-core contributors. Possibly enabling the drive-by
> contribution use case is a happy consequence. Volunteers able to work on
> additional things are welcome to join us. However I am specifically
> looking for things we can tweak without much effort that will help
> others. I regret that we don't have the time to take on a big project
> that doesn't benefit our own use cases. So please understand that if you
> make a proposal that involves significant developer work, it is unlikely
> to happen unless you also find developers to volunteer their time to
> work on it.
>
> At the UOS session we'll be able to discuss this in real-time on Google
> Hangouts and concurrently on IRC. But our time in the session is limited
> to one hour, so replying to this thread to first distill any
> conversation would be helpful if possible.


Cheers,
mwh
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

plans for golang packages in z

[Michael Hudson-Doyle]

Hi,

It'd be nice to switch the default golang package to 1.7 during the
toolchain update part of the z development cycle. This is just a matter of
uploading a golang-defaults package with a version like 2:1.7+0ubuntu1. It
the stars align, the go team will release Go 1.7.2 before this happens but
I don't think waiting very long for that makes sense.

I guess in practice this is just a pre-emptive request to the release team
to allow my golang-defaults upload through nice and early.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

patch pilot report 2016-07-08

[Michael Hudson-Doyle]

I was patch pilot yesterday. I didn't manage to spend all day on it,
but I reduced the queue a bit:

https://bugs.launchpad.net/ubuntu/+source/quassel/+bug/1589128

 - Proposed syncing package instead, set to Incomplete.

https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1546565

 - Asked for clarification, unsubscribed sponsors after I got it.

https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/253119

 - Unsubscribed sponsors as this seems to be being addressed on the
   merge proposal on the upstream project (i.e. the correct place ;-p)

https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1584485

 - Unsubscribed sponsors as there is nothing to upload yet.

https://bugs.launchpad.net/ubuntu/+source/bacula/+bug/1570923

 - Unsubscribed sponsors as there is nothing to upload yet.

https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1534538

 - Rebase patch on current version in trusty-proposed, uploaded.

https://bugs.launchpad.net/neutron/+bug/1414218

 - Uploaded to trusty queue.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

patch pilot report 2016-06-13

[Michael Hudson-Doyle]

Hi all,

I did my first stint at patch piloting today (was supposed to be last
thursday but other things came up). I didn't get a whole lot done but
I'll get better :-)


https://code.launchpad.net/~cbjchen/horizon/lp1382079/+merge/289023

 - marked as merged

https://bugs.launchpad.net/ubuntu/+source/php7.0/+bug/1569609

 - set up as SRU bug

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1571456

 - unsubscribed sponsors as it looks like this is going to happen via
   upstream.

https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1578540

 - suggested this be sent to debian

https://bugs.launchpad.net/ubuntu/+source/pixfrogger/+bug/1585058

 - commented, think this needs AA attention

https://bugs.launchpad.net/ubuntu/+source/libmime-types-perl/+bug/1589322

 - synced

golang-github-miekg-mmark

 - synced

golang-github-miekg-mmark

 - merged (was trivial)

Cheers,
mwh

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ANN: DNS resolver changes in yakkety

[Michael Hudson-Doyle]

On 7 June 2016 at 04:41, Dimitri John Ledkov  wrote:
> On 6 June 2016 at 17:27, Stéphane Graber  wrote:
>> On Mon, Jun 06, 2016 at 03:17:51PM +0100, Robie Basak wrote:
>>
>> Unless the above can be fixed somehow, and I very much doubt resolved
>> will grow a DNS server any time soon, the switch to resolved mostly
>> feels like a regression over the existing resolvconf+dnsmasq setup we've
>> got right now and which in my experience at least, has been working
>> pretty well for us.
>>
>
> I have in the past tried to drop all config files from /etc.
>
> Dropping /etc/nsswitch.conf is trivial. Apart from libc and shadow
> very little else parses that, so that has minimal breakage so things
> that do call into libc end up doing the right thing.

Go binaries parse nsswitch.conf. If they find something they don't
recognize they use the cgo-based resolver (if that is built into the
binary, which I think is the case for most binaries we care about), so
this shouldn't break anything, just perform slightly worse.

Cheers,
mwh

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Go shared libraries are coming

[Michael Hudson-Doyle]

On 31 May 2016 at 12:48, Martin Packman <martin.pack...@canonical.com> wrote:
> On 25/05/2016, Michael Hudson-Doyle <michael.hud...@canonical.com> wrote:
>>
>> I've attempted to document the new world at
>> https://docs.google.com/document/d/1IOlBWWgcDeB9PfRORENESYj8iJt4W2EwsbYcpg4akBE/edit#
>
> Thank you for the clear write-up.

I'm glad it came across clearly!

> Is the thought that for instance all the -dev packages juju currently
> depends on should move to providing a shared library? I presume the
> current line we have over not splitting up the github.com/juju
> namespace into multiple packages till we expect to have multiple
> consumers remains.

Yes, that sounds right to me. TBH, I've not really thought super hard
about juju as it is still in this strange "some deps from source
package / some deps from archive" middle ground, but I assume that
some sanity will emerge here (or at least stability). There's no real
reason to make separate packages for the different github.com/juju
things until there is an actual reason.

Cheers,
mwh
PS: can we stop including jujud in the juju-2.0 binary packages yet?

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Go shared libraries are coming

[Michael Hudson-Doyle]

On 26 May 2016 at 10:22, Michael Hudson-Doyle
<michael.hud...@canonical.com> wrote:
> On 26 May 2016 at 06:42, Matthias Klose <d...@ubuntu.com> wrote:
>> On 25.05.2016 10:51, Michael Hudson-Doyle wrote:
>>>
>>> Hi all.
>>>
>>> I'm planning to start uploading the various bits and pieces of Go
>>> shared libraries to yakkety in the next few days. I'll start with
>>> golang-1.6, golang-defaults and dh-golang, and then move on through
>>> the libraries used by the packages we really care about (lxd, snapd,
>>> juju-core).
>>
>>
>> what's the story with golang-defaults on powerpc and gccgo?
>
> I wasn't planning to change anything on powerpc for now. It should
> work though (with some more dh-golang hacking), so I'll spend a little
> while seeing how hard it is.

It's not that hard it seems, so I'm now testing the version with gccgo
support out in my 'devirt' PPA. Not having lots of architecture lists
everywhere will be nice!

Cheers,
mwh

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Go shared libraries are coming

[Michael Hudson-Doyle]

On 26 May 2016 at 06:42, Matthias Klose <d...@ubuntu.com> wrote:
> On 25.05.2016 10:51, Michael Hudson-Doyle wrote:
>>
>> Hi all.
>>
>> I'm planning to start uploading the various bits and pieces of Go
>> shared libraries to yakkety in the next few days. I'll start with
>> golang-1.6, golang-defaults and dh-golang, and then move on through
>> the libraries used by the packages we really care about (lxd, snapd,
>> juju-core).
>
>
> what's the story with golang-defaults on powerpc and gccgo?

I wasn't planning to change anything on powerpc for now. It should
work though (with some more dh-golang hacking), so I'll spend a little
while seeing how hard it is.

Cheers,
mwh

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Go shared libraries are coming

[Michael Hudson-Doyle]

Hi all.

I'm planning to start uploading the various bits and pieces of Go
shared libraries to yakkety in the next few days. I'll start with
golang-1.6, golang-defaults and dh-golang, and then move on through
the libraries used by the packages we really care about (lxd, snapd,
juju-core).

I've attempted to document the new world at
https://docs.google.com/document/d/1IOlBWWgcDeB9PfRORENESYj8iJt4W2EwsbYcpg4akBE/edit#
but if anything is unclear, please do let me know. A package will not
start linking against the shared libraries until it is changed to do
so, so there should not be any immediate disruption, at least until I
come and start proposing patches to your packages :-) But do let me
know if you have any concerns about the direction this is going in.

Cheers,
mwh

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: [PATCH] Please fix squid3 bug #3769 and squid3 trusty package bug 1405351 in trusty

[Michael Hudson-Doyle]

Hi, thanks! I've reformatted the patch and put it in my PPA. Could you
please fill out the test case and regression potential sections of the
bug report? I don't know enough about squid3 to do that.

Cheers,
mwh

On 24 March 2016 at 03:21, Lukas Erlacher  wrote:
> Hi,
>
> there was no bug so I made one:
> https://bugs.launchpad.net/ubuntu/+source/squid3/+bug/1561007
>
> Best,
> Luke
>
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

plans for go in xenial

[Michael Hudson-Doyle]

Hi,

I'm starting to make plans for packaging Go and packages written in Go
for Xenial. I thought I'd sketch out my ideas here for feedback and
avoidance of surprises later in the piece.

Some background:

1. Go 1.6 is due to be released around the time of feature freeze for Xenial.

2. As part of the work involved in getting juju and go into main, the
security team received a promise that we would at least try to use
shared libraries in the juju packaging.

3. Go 1.5 only has support for shared libraries on amd64, but Go 1.6
will have support for shared libraries on all supported architectures
(well, apart from s390 and powerpc, which only have gccgo anyway)

The timeline means that getting 1.6 into Xenial is probably
reasonable, but waiting for the release or even a release candidate
would be too late to explore how shared libraries are working out --
that needs to start soon, and so with a 1.6 pre-release.

There are two broad options here: do all the shared library
exploration in a staging PPA or create a golang1.6 package that can be
uploaded to the distro without affecting everything that build depends
on "golang-go".

The advantages of the former are that it's less pollution in the
archive, especially if we end up not using shared libraries this
cycle. The latter has the advantage that one wouldn't need to add a
ppa to every image involved in testing. (Tangentially, I'm also
exploring the idea of having separate and co-installable golang1.5-go
and golang1.6-go packages, partly to make this easier and partly so we
can backport golang1.5-go cleanly to trusty).

Well, this is mostly a braindump, but let me know what you think.

Cheers,
mwh

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel