Dear all, As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> and in this Dev List thread <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ>.
Current status: - Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users. - Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected. Quick summary: - Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again. - Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database. - Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here. - Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we decided to block uploads of release artifacts to the Jenkins Artifactory instance. - Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries - Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release. - Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569> in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking - Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts. Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation. Best regards, Oleg Nenashev -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLD4AWGkQCh0mGTRtViyHT-UBXoE3SbaKgVe5%3DsbSjBE%3Dg%40mail.gmail.com.
