Upgrades to Ubuntu 24.04.1 LTS are enabled again
Hello everyone, upgrades are now enabled again. For more details, please refer to the discourse post: https://discourse.ubuntu.com/t/upgrades-to-ubuntu-24-04-1-lts-are-enabled-again/47920 Thanks! On Mon, Sep 9, 2024 at 11:46 AM Utkarsh Gupta wrote: > Hello everyone, > > We’ve discovered a bug in ubuntu-release-upgrader in the code that enables > a new apt algorithm to handle obsolete packages. > > Therefore, updates to Noble have been temporarily suspended while the > issue is being addressed. > > The fixes are in the pipeline already and we should be re-enabling the > upgrades soon. Once done, expect to hear from us along with the > retrospective on what happened and what we are doing to ensure that this > doesn’t happen again. > > > On behalf of the Ubuntu Release team, > Utkarsh Gupta > -- > Ubuntu-release mailing list > Ubuntu-release@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-release > -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Upcoming APT cryptography changes for 24.04.1
On Tue, Jul 30, 2024 at 01:23:34PM GMT, Robie Basak wrote:
> To clarify my opinion, I should add that for LTS purposes, I think it's
> important that we have a path to "ratchet up" the minimum ciphersuite
> standard, and therefore it's appropriate to do in SRU as a policy.
> Theoretically this would be for any release, including past releases,
> and future ratchets that we might decide are appropriate after later
> releases have been around a while.
>
> But we do want to avoid user suprise and where possible do it with
> plenty of notice, so it would be preferable for the actual mechanism to
> support this as well as is possible.
>
> On Tue, Jul 30, 2024 at 05:27:20PM +0900, Julian Andres Klode wrote:
> > One option could be:
> >
> > +{
> > +"begin": "2024-08-05T00:00:00Z",
> > +"lines": [
> > +"All PPAs are now dual signed with 4096-bit RSA keys.",
> > +"Remove and readd PPAs to migrate to the new keys.",
> > +"For more details see:
> > https://ubuntu.com/blog/ubuntu-ppa-resigning";
> > +]
> > }
>
> What I intended in my suggestion was the following. Some of this you
> were already doing, but for completeness of my logic I'm listing all of
> the bits that I think are important:
>
> 1. Make the change a configuration item that can be dropped into a .d/
> directory.
The default configuration is in apt-pkg/init.cc; we drop an override
in the maintainer script to downgrade it to a warning on upgrades from
earlier apt in noble.
This means that after rolling out the update:
1. Existing noble installs continue trusting rsa1024
2. Upgrades from mantic/jammy done after that point will no longer trust it
3. Installs form 24.04.2 media will not trust it
Importantly, this means LTS -> LTS upgrades should not ever get into an
in-between warning state which was the goal.
>
> 2. The configuration should include a date from which it will happen, so
> that apt does it dynamically without any further configuration on the
> flag day.
I've talked to more people about making this a time bomb, but there's
concerns that this breaks stuff like test suites in the release pocket;
so there's an inclination to actually enable new flags with package
upgrades such that the change itself gets testing, and e.g. Azure
with its snapshot based rollout can do its normal testing rather than
having to special case it.
We don't yet have a date mechanism. The mechanism implemented provides
three levels, an unlabeled active one, a next one causing warnings, and
a 'future' level allowing messages in --audit messages (--audit is not
yet available in the noble SRU either). Each of which specify the list
of allowed algorithms in that level, in the format given to us by
GnuPG (or rather a strict subset).
This is the natural and minimal extension of the existing mechanism;
it's not necessarily the best mechanism yet. The mechanism was rejected
as inadequate by the Rust-Sequioa maintainers; hence their gpgv
reimplementation doesn't support it (installing gpgv-from-sq hence
disables the feature).
A new mechanism is in the works that instead of relying on the GnuPG
public key assertions on the signatures, parses the keyring itself
hence it knows *all* your configured keys (it also gives you the apt-key
list replacement people were asking for) [I read the pgpdump code and
roughly translated it to APT style].
That new mechanism is backportable. And we can choose to only issue
warnings, not errors, as they are now more effective. No longer is
there a case where you could have a repository with 2 keys configured,
no warnings because they now sign with a strong key and a MITM
downgrades you to a weak key.
But in any case, it seems advisable to improve the mechanism to specify
target dates for when we revoke trust, that said, this hasn't made it
in yet, this is something to look at for 24.04.2; trying to insert the
date algorithm last minute seems more regression potential.
>
> 3. Prior to that date, apt would know it's coming and could then display
> a warning indicating that it is configured to change from that date.
>
> And then there would be a path to making this enforced sooner for fresh
> installs for .1, but eventually converging with those who have installed
> .0, if that's what we choose to do.
>
> The key thing is that then we can warn users in advance, and users would
> also be able to bump the date and/or manually override to avoid the
> "ratchet" entirely, without being surprised by interruption of service
> and frantic searching to mitigate an issue after it has affected them.
>
> Ideally, if a
Re: Upcoming APT cryptography changes for 24.04.1
On Mon, Jul 15, 2024 at 08:31:13PM GMT, Julian Andres Klode wrote:
> Hello!
>
> As you all know, and the release notes state, we want to promote
> the weak public key algorithms in APT from warnings to errors for
> the upcoming 24.04.1 release to ensure people _actually_ get a
> security benefit.
>
> Now warnings provide some advantage already as you can see weak
> repositories; but they do not provide full safety: If a repository
> has multiple keys configured, but only signs with safe ones, you
> do not get a warning, but an attacker could downgrade the key.
>
> This was skipped for the main 24.04 release as PPAs had not been
> resigned yet, but the resigning is at the final stages now, and
> having the final policy in place for 24.04.1 means that all those
> upgrades from 22.04 and new installs from the new media get the
> safe experience.
>
> Some repositories have been inadvertently affected by the way this
> was implemented as we also revoked ECC curves other than Ed25519
> and Ed448, such as NIST P-256; raising this to errors now would
> break those users entirely which is not intended.
>
> To mitigate risk my plan is:
>
> 1. Introduce new apt 2.8.1 release that will add the other
>256-bit or stronger curves to the allowed list. It will
>also introduce new levels for warnings and audit messages
>to ensure that you still get warnings for unusual curves,
>and audit messages going forward.
>
>See https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2073126 for
>more details.
>
>I have asked foundations and ubuntu security for a review
>of this change.
After consulting with Robie I have added an additional mitigation;
if you upgrade from 2.7.14 in the release pocket (or older 2.7.x
from the devel series), the apt.postinst creates a new file
/etc/apt/apt.conf.d/00-temporary-rsa1024
Which temporarily downgrades weak RSA keys to warnings again;
this allows us to decouple rolling out the new enforcement to
new installs from rolling out the enforcement to existing systems,
mitigating the risk for regressions.
We can then work to migrate existing systems to stronger keys
(semi?)automatically following the point release until the next
point release at which point we should release an apt that deletes
that mitigation file again.
>
> 2. Once the PPAs have been switched to the 4096-bit RSA key,
>issue a message to apt users using the Pro client 'apt-news'
>mechanism on 24.04 stating that:
>
>> All PPAs have been resigned with 4096-bit keys now, and support for old
>> signing keys will be disabled starting August 05. If apt update lists
>> any affecteds PPAs, please use add-apt-repository -r to remove them
>a> nd re-add them again.
>
>(or similar wording)
>
>A similar message can be issued for older releases so they
>can upgrade their security, however I have not yet checked
>if it actually removes the key from trusted.gpg.d back then.
The wording needs to be shorter such that it can fit on 2 lines
and have a URL link in the 3rd line.
One option could be:
+{
+"begin": "2024-08-05T00:00:00Z",
+"lines": [
+"All PPAs are now dual signed with 4096-bit RSA keys.",
+"Remove and readd PPAs to migrate to the new keys.",
+"For more details see:
https://ubuntu.com/blog/ubuntu-ppa-resigning";
+]
}
Though it's arguably nicer to not include instructions there
and move them to the blog post (and or only share instructions
once we have automatic tooling)
>
> 4. Release the apt update to the -updates pocket on August 05, override
>the phasing to 0% and build images with that. This ensures that new
>installs have the new policy enabled. Note that there's some policy
>about not including phased updates in images that will have to be
>temporarily lifted.
>
>Also do note that phasing is ignored in chroots due to image
>building concerns, so some use cases will still see the update.
>
> 5. Slowly phase the apt update. It seems fine to roll this out
>over a long time frame such as 3 months assuming there are
>no other upcoming urgent apt updates.
>
>I do not know how much control we have over the phasing speed.
This is obsolete now due to the added apt.conf.d mitigation.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
--
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Upcoming APT cryptography changes for 24.04.1
Hello! As you all know, and the release notes state, we want to promote the weak public key algorithms in APT from warnings to errors for the upcoming 24.04.1 release to ensure people _actually_ get a security benefit. Now warnings provide some advantage already as you can see weak repositories; but they do not provide full safety: If a repository has multiple keys configured, but only signs with safe ones, you do not get a warning, but an attacker could downgrade the key. This was skipped for the main 24.04 release as PPAs had not been resigned yet, but the resigning is at the final stages now, and having the final policy in place for 24.04.1 means that all those upgrades from 22.04 and new installs from the new media get the safe experience. Some repositories have been inadvertently affected by the way this was implemented as we also revoked ECC curves other than Ed25519 and Ed448, such as NIST P-256; raising this to errors now would break those users entirely which is not intended. To mitigate risk my plan is: 1. Introduce new apt 2.8.1 release that will add the other 256-bit or stronger curves to the allowed list. It will also introduce new levels for warnings and audit messages to ensure that you still get warnings for unusual curves, and audit messages going forward. See https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2073126 for more details. I have asked foundations and ubuntu security for a review of this change. 2. Once the PPAs have been switched to the 4096-bit RSA key, issue a message to apt users using the Pro client 'apt-news' mechanism on 24.04 stating that: > All PPAs have been resigned with 4096-bit keys now, and support for old > signing keys will be disabled starting August 05. If apt update lists > any affecteds PPAs, please use add-apt-repository -r to remove them a> nd re-add them again. (or similar wording) A similar message can be issued for older releases so they can upgrade their security, however I have not yet checked if it actually removes the key from trusted.gpg.d back then. 3. Optionally, release an update for software-properties that refreshes the signing key for affected PPAs in postinst. This will catch everyone who is online and did not see the message. We can use the remove & re-add logic here, but it may be more suitable to just replace exactly the key in the file which can be done with a little bit of apt_pkg.TagSection.write() :) Figuring out which PPAs need a key refresh is a bit annoying, essentially it boils down to looking at all configured PPAs, calling gpgv on their release files with a ">=rsa2048" assertion and see if it passes it or not. While the benefits are clear; it is important to note that key refresh is almost TOFU: The only integrity assurance we have about the authenticity of the new key is launchpad's https certificate, so it is worth considering if this is something we feel comfortable about doing automatically. 4. Release the apt update to the -updates pocket on August 05, override the phasing to 0% and build images with that. This ensures that new installs have the new policy enabled. Note that there's some policy about not including phased updates in images that will have to be temporarily lifted. Also do note that phasing is ignored in chroots due to image building concerns, so some use cases will still see the update. 5. Slowly phase the apt update. It seems fine to roll this out over a long time frame such as 3 months assuming there are no other upcoming urgent apt updates. I do not know how much control we have over the phasing speed. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: mantic unsatisfiable build-depends report (Sep 14, 19:38:18)
On Thu, Sep 14, 2023 at 10:41:01PM +0200, Julian Andres Klode wrote: > (forward from https://magenta.jak-linux.org/ubuntu-archive/distcheck/devel/, > the release.txt file) the idea is that you want to look at the file regularly and go fix stuff, not for me to send these emails regularly. Plan to move this to something more official at some point; but basically this is a reminder that the service exists (and now is current again, and always tracking devel). -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
mantic unsatisfiable build-depends report (Sep 14, 19:38:18)
(forward from https://magenta.jak-linux.org/ubuntu-archive/distcheck/devel/, the release.txt file) Unsatisfiable build-depends (release sources against release binaries) === As of release Release file: Thu, 14 Sep 2023 19:38:18 UTC As of proposed Release file: Thu, 14 Sep 2023 19:39:02 UTC Packages build-depending on librust*: * git-deltabuild-depends on missing librust-sysinfo-0.28-dev:amd64 * newsboat build-depends on missing librust-lexopt-0.2-dev:amd64 * rust-cargo-debstatus build-depends on missing librust-env-logger-0.9+default-dev:amd64 * rust-condure build-depends on missing librust-miniz-oxide-0.6+default-dev:amd64 * rust-dfrsbuild-depends on missing librust-env-logger-0.9+default-dev:amd64 * rust-droid-juicerbuild-depends on missing librust-goblin-0.2+default-dev:amd64 * rust-elfx86exts build-depends on missing librust-object-0.30+default-dev:amd64 * rust-process-viewer build-depends on missing librust-sysinfo-0.28+default-dev:amd64 (>= 0.28.2-~~) * rust-sequoia-sq build-depends on missing librust-sequoia-autocrypt-0.24-dev:amd64 * sccache build-depends on missing librust-daemonize-0.4+default-dev:amd64 Packages build-depending on python3-*: * hyperspy build-depends on missing python3-ptable:amd64 * pytorch-text build-depends on missing python3-torch:amd64 (>= 1.12.0~) * unicorn-hat build-depends on missing python3-rpi-ws281x:amd64 Packages build-depending on emacs-nox:amd64: * emacs-non-dfsg build-depends on missing emacs-nox:amd64 (>= 1:29) | emacs:amd64 (>= 1:29) | emacs:amd64 (>= 1:29) Packages build-depending on golang-*: * golang-gitlab-gitlab-org-labkit build-depends on missing golang-google-grpc-dev:amd64 (< 1.38~) Packages build-depending on libbind-dev:amd64: * milter-greylist build-depends on missing libbind-dev:amd64 Packages build-depending on libglademm-2.4-dev:amd64: * bibshelf build-depends on missing libglademm-2.4-dev:amd64 (>= 2.6) Packages build-depending on libltdl7-dev:amd64: * ripoff build-depends on missing libltdl7-dev:amd64 Packages build-depending on libmaven-compiler-plugin-java:amd64: * cbmc build-depends on missing libmaven-compiler-plugin-java:amd64 (< 3.8.2~) Packages build-depending on libqzxing-dev:amd64: * lomiri-camera-appbuild-depends on missing libqzxing-dev:amd64 Packages build-depending on libwine-development-dev:amd64: * dxvk build-depends on missing libwine-development-dev:amd64 Packages build-depending on linux-generate-aws:amd64: * linux-signed-aws build-depends on missing linux-generate-aws:amd64 (= 6.5.0-1005.5) Packages build-depending on linux-generate-azure:amd64: * linux-signed-azure build-depends on missing linux-generate-azure:amd64 (= 6.5.0-1004.4) Packages build-depending on linux-generate-gcp:amd64: * linux-signed-gcp build-depends on missing linux-generate-gcp:amd64 (= 6.5.0-1004.4) Packages build-depending on linux-generate-ibm:amd64: * linux-signed-ibm build-depends on missing linux-generate-ibm:amd64 (= 6.5.0-1005.5) Packages build-depending on linux-generate-lowlatency:amd64: * linux-signed-lowlatency build-depends on missing linux-generate-lowlatency:amd64 (= 6.5.0-5.5.1) Packages build-depending on linux-generate-oracle:amd64: * linux-signed-oracle build-depends on missing linux-generate-oracle:amd64 (= 6.5.0-1005.5) Packages build-depending on linux-generate:amd64: * linux-signed build-depends on missing linux-generate:amd64 (= 6.5.0-5.5) Packages build-depending on pep8:amd64: * backup2swift build-depends on missing pep8:amd64 Packages build-depending on pylint3:amd64: * python-jenkinsapibuild-depends on missing pylint3:amd64 Packages build-depending on python-*: * python-jujuclientbuild-depends on missing python-setuptools:amd64 Packages build-depending on ruby-*: * nanocbuild-depends on missing ruby-uglifier:amd64 Packages build-depending on s390-tools-zkey:amd64: * s390-tools-signedbuild-depends on missing s390-tools-zkey:amd64 (= 2.28.0-0ubuntu2) -- Ubuntu-release mailing
Re: Transitioning the sync-blacklist vcs to git
On Wed, Jul 05, 2023 at 11:43:32AM +0200, Sebastien Bacher wrote: > Hey there, > > I've imported the sync-blacklist bzr repository [1] to git [2] and submitted > a merge request [3] to update the update-sync-blacklist job to use the git > location Can we rename it to sync-blocklist while we're at it? -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: adsys SRU
On Wed, Jun 14, 2023 at 10:00:44PM +0100, Robie Basak wrote: > On Wed, Jun 14, 2023 at 09:31:42AM +0200, Didier Roche wrote: > > Unfortunately, like many projects, there is a constant tension between the > > request for new features backport (adsys, as being an enterprise product, > > only really makes sense in a LTS context) and bug fixes. Most of the new > > features are developed due to industry requirements, which are: > > - evolution of their own security practices (for instance, certificates > > support) > > - request for other platform supports (winbind in addition to > > already-existing sssd) > > So on one hand, enterprise users want to use the LTS because they don't > want feature changes. On the other hand, they demand new features so > they do want change. > > How does this fit with our stable release policies with respect to > adsys? Is it possible that one enterprise wants to use the current > version of adsys in a stable release and doesn't want it to change > because that's what they validated, and another enterprise wants a new > feature and requires it to be updated in a stable release? > > If that conflict does arise, how are we to resolve it, keeping in mind > the need to maintain the reputation of our LTS as a stable platform that > generally and very deliberately doesn't do feature changes? > > Could these cases, for example, be served better by a snap, the > backports pocket or a PPA instead? Let me propose another way to limit issues: Backport the adsys in the latest stable release to LTS series, and keep that as a release series, while developing new stuff in stable. Basically that's what I'm trying to do with grub too (except we currently binary copy the grub binaries rather than do rebuilds per series, but different story). You can also split this up further and have an adsys that just receives security updates, and an "hwe" like rolling adsys, think like kernels. I think the backports pocket could work, but it may be easier to use an adsys-rolling package or something so users don't have to enable "scary" repositories. Also backports and PPAs don't really have the same guarantees as -updates, when that is something that customers want. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en signature.asc Description: PGP signature -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Partially phased updates on release media
On Wed, May 19, 2021 at 03:59:06PM -0700, Brian Murray wrote: > The release team was recently discussing the possibility of point > release media containing updates which are not fully phased and how this > creates a difference between upgraded systems and freshly installed > ones[1]. Looking at the packages which are not currently phasing[2] I've > discovered that the 16.04.7 installation iso contains > unity-control-center 15.04.0+16.04.20171130-0ubuntu1 and the 18.04.5 iso > contains nautilus 1:3.26.4-0~ubuntu18.04.5. To ensure that both types of > systems are the same I think we should fully phase these two packages. > > Additionally, a step should be added to the point release process to nag > the uploader of a package which has stopped phasing to investigate the > reasons for the stoppage well before (2 weeks?) the point release. If > the package is still not fully phased in time for the point release then > the release team should make a decision about fully phasing it or > reverting to the previous version of the package. I just filed https://bugs.launchpad.net/launchpad/+bug/1929082 against launchpad to keep the previous version of the update in updates while a new one is being phased. This would then allow image builders to set the never-include-phased-updates option in apt.conf, and then images will always build with whatever is fully phased, and we don't have to take as much special care (just need to disable phasing changes during image building time) for future LTS releases. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Retrospective on today's britney outage (10:42-20:55 UTC)
britney outage from 10:42 to 20:55 (UTC) At 17:47 (all times UTC) today I was informed by RikMills that britney had not been running successfully since about 11:00, because autopkgtest.com/results was eventually 503-ing. By 18:15, I had mistakingly identified this as a swift issue and asked IS to investigate. By 18:40 I heard back from IS that they did not see errors, and looked at the autopkgtest-web logs a second. Investigation showed that the issue was haproxy disabling our cloud workers which both failed at the same time (well, regularly) due to "database disk image is malformed" errors from sqlite. By 19:00 Laney started working on moving the /results proxying out of the Apache servers and directly into haproxy, and released that at 20:34. Meanwhile I started working on fixing the error by replacing our simple file copying code with the SQLite online backup API, as waveform had suggested earlier. That work was finished at 20:56, after we finally figured out how to give me charm store access (grant --channel unpublished did the trick). I identified some Work items for the future: * some alerts other for britney failing, as relying on community members to report 7 hours after the first failure is not super helpful. * We probably also need some monitoring that alerts us of the high failure rate we had on the web servers. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
New force-reset-test hint
Hello everyone! Laney just merged his branch based off my branch from May 2018 that adds the "force-reset-test" hint. This resets the baseline to always failed, and is a better alternative to force-badtest for a lot of things that started failing for whatever reason but might eventually come back working. So, less bad testing, more test reseting please! Thanks! -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
