I have two releases that I consider high-priority: github-branch-source and github-api .
Users have been able to rollback to the previous release to unblock themselves, but people who cannot rollback (new installations) remain blocked. On Friday, June 12, 2020 at 10:05:33 AM UTC-7, Oleg Nenashev wrote: > > Dear all, > > June 12 update: > > - We continue to work on the accounts migration and will share the > next update on Monday > - Jenkins releases are still blocked. If there are any emergency > releases you need to perform, please reply in this thread. > > Best regards, > Oleg Nenashev > > On Friday, June 12, 2020 at 6:00:32 PM UTC+2, Oleg Nenashev wrote: >> >> Hi Dave, >> >> This is an email from the *Step 2. We’ll reset every user password from >> the LDAP database*. This one includes a temporary password, and we >> expect users to change it after they login into the system. >> >> For those who wonder: Yes, the temporary password is sent in plain text >> as mentioned above. This is how our current password reset system is >> designed. As other projects, we have a decent amount of technical debt in >> our infrastructure which we gradually resolve. I have already added >> changing the account password reset flow to the outage retrospective list, >> an we will be reviewing what to do there after the outage is fully >> resolved. Apart from fixing it, migrating to a 3rd-party identity service >> is on the table for me (Linux Foundation or GitHub). If anyone is >> interested to participate and to improve the project, the Jenkins >> infrastructure team <https://www.jenkins.io/projects/infrastructure/> is >> always looking for more contributors! >> >> If anyone has concerns about such method and wants to use alternate >> channels for encrypted password transfer, please send us a message through >> the Jenkins Infrastructure mailing list from your email registered in >> Jenkins. In this email please provide your public GPG key so that we can >> reset a password again in a secure way. >> >> Best regards, >> Oleg >> >> On Friday, June 12, 2020 at 5:07:27 PM UTC+2, Dave Pedu wrote: >>> >>> Hello, >>> >>> I have received an email linking to this thread. However, it contains a >>> plaintext password for my account, despite this: >>> >>> > There will be no temporary password in these emails, but there will >>> be information pointing to this thread. >>> >>> Is this email legitimate or am I being phished? Screenshot attached. >>> >>> Thanks, >>> Dave >>> >>> On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote: >>>> >>>> Dear all, >>>> >>>> We are ready to proceed with restoration of the Jenkins account >>>> database. Today we are going to restore user LDAP accounts that were >>>> created since the First of February 2020 based on the data from Jenkins >>>> Jira and the repository Permission Manager metadata data. We will also >>>> reset passwords for all users registered in the database. >>>> >>>> Step 1. All users who lost their account will receive an email saying >>>> that their accounts were re-created. There will be no temporary password >>>> in >>>> these emails, but there will be information pointing to this thread. >>>> >>>> Step 2. We’ll reset every user password from the LDAP database, it is >>>> more than 100 000 users. Once done, you’ll receive an email telling you >>>> that your password was reset with a reason containing a link to this mail >>>> thread. >>>> >>>> Step 3. We will delete accounts of users who requested such deletion >>>> between February and June 2020. These users were restored from the backup, >>>> so we have to delete them again.The list of users is based on Jira tickets >>>> and private messages to the Jenkins Infra officer. If for some reason you >>>> notice that your account still exists, feel free to raise a ticket in >>>> Jenkins >>>> Jira <https://issues.jenkins-ci.org/> (project=INFRA, component=account >>>> ). >>>> >>>> Please do not hesitate to contact us using the #jenkins-infra channel >>>> on Freenode IRC or the Jenkins Infrastructure mailing list if you have any >>>> questions or suggestions. If you see a security issue related to the >>>> accounts, please follow the vulnerability reporting guidelines >>>> <https://www.jenkins.io/security/#reporting-vulnerabilities>. >>>> >>>> Best regards, >>>> >>>> Olivier Vernin && Jenkins Infrastructure Team >>>> >>>> >>>> On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote: >>>>> >>>>> Dear all, >>>>> >>>>> As you may have noticed, the release artifact uploads are currently >>>>> blocked in the Jenkins Artifactory instances ( >>>>> https://repo.jenkins-ci.org/). We are doing a security investigation >>>>> due to a partial user database loss on June 02. Today we blocked releases >>>>> to the Jenkins artifactory, and there also was a temporary outage of the >>>>> Artifactory downloads which was a collateral damage of the temporary >>>>> permissions. You can find more details about it in this Jenkins Infra >>>>> Thread >>>>> <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> >>>>> and in this Dev List thread >>>>> <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ> >>>>> . >>>>> >>>>> Current status: >>>>> >>>>> - >>>>> >>>>> Downloads are restored for all artifacts on >>>>> https://repo.jenkins-ci.org/, Jenkins core historical releases, >>>>> Remoting library and Windows Service Wrapper which were among ones >>>>> reported >>>>> by Jenkins users. >>>>> - >>>>> >>>>> Uploads: Jenkins artifact uploads are blocked for the most of >>>>> Jenkins plugin maintainers and contributors. It affects releases of >>>>> Jenkins >>>>> plugins, Jenkins core and modules, developer tools and all libraries >>>>> hosted >>>>> on https://repo.jenkins-ci.org/. Incremental and Snapshot >>>>> deployments are not affected. >>>>> >>>>> >>>>> Quick summary: >>>>> >>>>> - >>>>> >>>>> Jun 02 - There was a Kubernetes Cluster outage on June 02. During >>>>> this outage we had to rebuild the cluster from scratch to get some >>>>> services >>>>> working again. >>>>> - >>>>> >>>>> Jun 02 - After the recovery we lost three months of LDAP changes. >>>>> It has happened due to the broken backup of the LDAP database. >>>>> - >>>>> >>>>> Jun 02 - We identified a number of potential security risks which >>>>> may be caused by the LDAP outage. Account overtake and malicious >>>>> upload was >>>>> one of the identified risks. FTR this issue is tracked as >>>>> SECURITY-1895 as >>>>> a follow-up to these discussions. Only the Security team members have >>>>> access to it, so I am not sharing a link here. >>>>> - >>>>> >>>>> Jun 09 - After the security risk was independently reported in >>>>> public by a plugin maintainer in the dev list thread >>>>> <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we >>>>> decided to block uploads of release artifacts to the Jenkins >>>>> Artifactory >>>>> instance. >>>>> - >>>>> >>>>> Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked >>>>> (plugins, Jenkins core and modules, developer tools, etc.). Downloads >>>>> of >>>>> some binaries were also blocked as an unexpected collateral damage. >>>>> Jenkins >>>>> core historical releases, Remoting library and Windows Service Wrapper >>>>> are >>>>> among the affected binaries >>>>> - >>>>> >>>>> Jun 09, 10AM UTC - We finished reviews of all artifact releases to >>>>> https://repo.jenkins-ci.org/, which happened between the infra >>>>> outage on June 02 and the blockage of the releases. There are no >>>>> maliciously uploaded artifacts. Note that the common plugin release >>>>> flow >>>>> requires access to GitHub in order to push the release commits, so a >>>>> malicious attacker would need to overtake both Jenkins and GitHub >>>>> accounts >>>>> of a single user to submit a legitimately-looking release. >>>>> - >>>>> >>>>> Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch >>>>> >>>>> <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569> >>>>> >>>>> in the Repository Permission Updater was applied to prevent uploads. >>>>> Artifact uploads are still blocking >>>>> - >>>>> >>>>> Jun 09, 2PM UTC, based on repo.jenkins-ci.org and >>>>> issues.jenkins-ci.org data, we restored maintainers accounts. >>>>> >>>>> >>>>> Our next steps would be to communicate the issue to all maintainers >>>>> and contributors who might have been affected by the LDAP history loss. >>>>> We >>>>> will likely need to perform additional user verification steps for plugin >>>>> maintainers to ensure that there are no contributors affected by the >>>>> issues. Today at 3:30PM UTC we will also have a Jenkins >>>>> Infrastructure team meeting where this issue will be discussed in more >>>>> detail. This is a public meeting, and everyone is welcome to join. >>>>> Calendar >>>>> link >>>>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com> >>>>> >>>>> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and >>>>> Security team members who contributed to this investigation. >>>>> >>>>> Best regards, >>>>> >>>>> Oleg Nenashev >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ab94c691-ab74-4eaa-a166-c2e16f8e2b28o%40googlegroups.com.
