Hi everyone, Firstly, I deeply apologise for causing the regression.
Even with three separate people testing the test packages and the packages in -proposed, the failure still went unnoticed. I should have considered the impacts of changing the default behaviour of adcli a little more deeply than treating it like a normal SRU. Here are the facts: The failure is limited to adcli, version 0.8.2-1ubuntu1 on Bionic. At the time of writing, it is still in the archive. To archive admins, this needs to be pulled. adcli versions 0.9.0-1ubuntu0.20.04.1 in Focal, 0.9.0-1ubuntu1.2 in Groovy and 0.9.0-1ubuntu2 in Hirsute are not affected. sssd 1.16.1-1ubuntu1.7 in Bionic, and 2.2.3-3ubuntu0.1 in Focal are not affected. Bug Reports: There are two launchpad bugs open: LP #1906627 "adcli fails, can't contact LDAP server" https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627 LP #1906673 "Realm join hangs" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673 Customer Cases: SF 00298839 "Ubuntu Client Not Joining the Nasdaq AD Domain" https://canonical.my.salesforce.com/5004K000003u9EW SF 00299039 "Regression Issue due to https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673" https://canonical.my.salesforce.com/5004K000003uAkL Root Cause: The recent SRU in LP #1868703 "Support "ad_use_ldaps" flag for new AD requirements (ADV190023)" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703 introduced two changes for adcli on Bionic. The first, was to change from GSS-API to GSS-SPNEGO, and the second was to implement support for the flag --use-ldaps. I built a upstream master of adcli, and it still fails on Ubuntu. This indicates that the failure is not actually in the adcli package. adcli does not implement GSS-SPNEGO, it is linked in from the libsasl2-modules-gssapi-mit package, which is a part of cyrus-sasl2. I built the source of cyrus-sasl2 2.1.27+dfsg-2 from Focal on Bionic, and it works with the problematic adcli package. The root cause is that the implementation of GSS-SPNEGO in cyrus-sasl2 on Bionic is broken, and has never worked. There is more details about commits which the cyrus-sasl2 package in Bionic is missing in comment #5 in LP #1906627. https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/comments/5 Steps taken yesterday: I added regression-update to LP #1906627, and I pinged ubuntu-archive in #ubuntu-release with these details, but they seem to have been lost in the noise. Located root cause to cryus-sasl2 on Bionic. Next steps: We don't need to revert any changes for adcli or sssd on Focal onward. We don't need to revert any changes on sssd on Bionic. We need to push a new adcli into Bionic with the recent patches reverted. We need to fix the GSS-SPNEGO implementation in cyrus-sasl2 in Bionic. We need to re-release all the SRUs from LP #1868703 after some very thorough testing and validation. Again, I am deeply sorry for causing this regression. I will fix it, starting with getting adcli removed from the Bionic archive. Thanks, Matthew On Fri, Dec 4, 2020 at 10:40 PM Lukasz Zemczak <lukasz.zemc...@canonical.com> wrote: > > Hey! > > I prefer broken upgrades to get pulled anyway. Besides, packages are > updated by unattended-upgrades in up-to 24 hours, so some users might > have not gotten it yet. And there's also those not using > undattended-upgrades. Let me demote it back to -proposed from -updates > as well. > > On Fri, 4 Dec 2020 at 10:00, Christian Ehrhardt > <christian.ehrha...@canonical.com> wrote: > > > > On Fri, Dec 4, 2020 at 9:49 AM Lukasz Zemczak > > <lukasz.zemc...@canonical.com> wrote: > > > > > > Hey Christian! > > > > > > This sounds bad indeed, let's see what Matthew has to say. In the > > > meantime I have backed it out from both bionic-security and > > > focal-security. > > > > Thank you > > > > > Should we also consider dropping it from -updates? > > > > Well, compared to other cases in this case we don't even yet have a > > "ok this is a mess, but this is how you can resolve it afterwards to > > work again". > > Therefore I think pulling it from -updates as well makes sense until > > Matthew had time to look at it in detail and give all-clear (or not). > > > > P.S.: you slightly raced vorlon who had a different assessment > > [09:30] <vorlon> cpaelzer: well, by this point almost everyone will > > have picked it up from security via unattended-upgrades so there's not > > much point > > But having it pulled for now is on the safe-side and we can re-instate > > it at any time once we know more. > > > > > Cheers, > > > > > > On Fri, 4 Dec 2020 at 09:01, Christian Ehrhardt > > > <christian.ehrha...@canonical.com> wrote: > > > > > > > > I was looking at 16 recently touched bugs. Of these a few needed a > > > > comment or > > > > task update but not a lot of work. Worth to mention are two of them. > > > > > > > > First we've had "one more" kind of conflicting mysql packages from > > > > third party breaking install/upgrade of the one provided by Ubuntu. I > > > > dupped it onto bug 1771630 which is our single place to unite all > > > > those. > > > > > > > > > > > > A recent sssd update (driven by SEG) seems to have regressed users > > > > that now end in a hang. > > > > I've pinged on [1], subscribed Matthew (and our Team) on [2]. I've > > > > marked it regression-update and also pinged Matthew him via Chat. > > > > Furthermore I've set him on CC on this mail. > > > > @Matthew - once you've done your initial assessment would you mind > > > > replying here with the next steps on this case please? > > > > I've marked it prio high, if other triagers see more such reports > > > > please mark it even critical then (in that case it is less likely to > > > > be just one odd special setup) > > > > The release is 21h ago, I'll ping ubuntu-archive (also on CC) if we > > > > should - for now until clarified by Matthew - remove it from > > > > -security. > > > > > > > > > > > > [1]: > > > > https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703/comments/86 > > > > [2]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673 > > > > > > > > > > > > -- > > > > Christian Ehrhardt > > > > Staff Engineer, Ubuntu Server > > > > Canonical Ltd > > > > > > > > -- > > > > ubuntu-archive mailing list > > > > ubuntu-arch...@lists.ubuntu.com > > > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-archive > > > > > > > > > > > > -- > > > Łukasz 'sil2100' Zemczak > > > Foundations Team > > > lukasz.zemc...@canonical.com > > > www.canonical.com > > > > > > > > -- > > Christian Ehrhardt > > Staff Engineer, Ubuntu Server > > Canonical Ltd > > > > -- > Łukasz 'sil2100' Zemczak > Foundations Team > lukasz.zemc...@canonical.com > www.canonical.com -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam