June 12th update:
- We are still working on the account migration
- Step 1 is completed, all users have been restored in the database
based on the data from Jenkins Jira and repository permissions updater.
- Step 2 is in progress. Tens of thousands users have already
received the password reset notifications, we had 2 batches of password
resets today. We will continue the migration tomorrow
- Step 3 - not started
- Plugin uploads are still blocked at the moment
- Tomorrow we plan to double-check the account resets for plugin
maintainers, and we will consider reenabling uploads after that
Best regards,
Oleg
On Thursday, June 11, 2020 at 12:07:01 PM UTC+2, Olblak wrote:
>
> Dear all,
>
> We are ready to proceed with restoration of the Jenkins account database.
> Today we are going to restore user LDAP accounts that were created since
> the First of February 2020 based on the data from Jenkins Jira and the
> repository Permission Manager metadata data. We will also reset passwords
> for all users registered in the database.
>
> Step 1. All users who lost their account will receive an email saying that
> their accounts were re-created. There will be no temporary password in
> these emails, but there will be information pointing to this thread.
>
> Step 2. We’ll reset every user password from the LDAP database, it is more
> than 100 000 users. Once done, you’ll receive an email telling you that
> your password was reset with a reason containing a link to this mail thread.
>
> Step 3. We will delete accounts of users who requested such deletion
> between February and June 2020. These users were restored from the backup,
> so we have to delete them again.The list of users is based on Jira tickets
> and private messages to the Jenkins Infra officer. If for some reason you
> notice that your account still exists, feel free to raise a ticket in Jenkins
> Jira <https://issues.jenkins-ci.org/> (project=INFRA, component=account).
>
> Please do not hesitate to contact us using the #jenkins-infra channel on
> Freenode IRC or the Jenkins Infrastructure mailing list if you have any
> questions or suggestions. If you see a security issue related to the
> accounts, please follow the vulnerability reporting guidelines
> <https://www.jenkins.io/security/#reporting-vulnerabilities>.
>
> Best regards,
>
> Olivier Vernin && Jenkins Infrastructure Team
>
>
> On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:
>>
>> Dear all,
>>
>> As you may have noticed, the release artifact uploads are currently
>> blocked in the Jenkins Artifactory instances (
>> https://repo.jenkins-ci.org/). We are doing a security investigation due
>> to a partial user database loss on June 02. Today we blocked releases to
>> the Jenkins artifactory, and there also was a temporary outage of the
>> Artifactory downloads which was a collateral damage of the temporary
>> permissions. You can find more details about it in this Jenkins Infra
>> Thread
>> <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> and
>> in this Dev List thread
>> <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ>.
>>
>> Current status:
>>
>> -
>>
>> Downloads are restored for all artifacts on
>> https://repo.jenkins-ci.org/, Jenkins core historical releases,
>> Remoting library and Windows Service Wrapper which were among ones
>> reported
>> by Jenkins users.
>> -
>>
>> Uploads: Jenkins artifact uploads are blocked for the most of Jenkins
>> plugin maintainers and contributors. It affects releases of Jenkins
>> plugins, Jenkins core and modules, developer tools and all libraries
>> hosted
>> on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments
>> are not affected.
>>
>>
>> Quick summary:
>>
>> -
>>
>> Jun 02 - There was a Kubernetes Cluster outage on June 02. During
>> this outage we had to rebuild the cluster from scratch to get some
>> services
>> working again.
>> -
>>
>> Jun 02 - After the recovery we lost three months of LDAP changes. It
>> has happened due to the broken backup of the LDAP database.
>> -
>>
>> Jun 02 - We identified a number of potential security risks which may
>> be caused by the LDAP outage. Account overtake and malicious upload was
>> one
>> of the identified risks. FTR this issue is tracked as SECURITY-1895 as a
>> follow-up to these discussions. Only the Security team members have
>> access
>> to it, so I am not sharing a link here.
>> -
>>
>> Jun 09 - After the security risk was independently reported in public
>> by a plugin maintainer in the dev list thread
>> <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we decided
>> to block uploads of release artifacts to the Jenkins Artifactory instance.
>> -
>>
>> Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked
>> (plugins, Jenkins core and modules, developer tools, etc.). Downloads of
>> some binaries were also blocked as an unexpected collateral damage.
>> Jenkins
>> core historical releases, Remoting library and Windows Service Wrapper
>> are
>> among the affected binaries
>> -
>>
>> Jun 09, 10AM UTC - We finished reviews of all artifact releases to
>> https://repo.jenkins-ci.org/, which happened between the infra outage
>> on June 02 and the blockage of the releases. There are no maliciously
>> uploaded artifacts. Note that the common plugin release flow requires
>> access to GitHub in order to push the release commits, so a malicious
>> attacker would need to overtake both Jenkins and GitHub accounts of a
>> single user to submit a legitimately-looking release.
>> -
>>
>> Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch
>>
>> <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569>
>> in the Repository Permission Updater was applied to prevent uploads.
>> Artifact uploads are still blocking
>> -
>>
>> Jun 09, 2PM UTC, based on repo.jenkins-ci.org and
>> issues.jenkins-ci.org data, we restored maintainers accounts.
>>
>>
>> Our next steps would be to communicate the issue to all maintainers and
>> contributors who might have been affected by the LDAP history loss. We will
>> likely need to perform additional user verification steps for plugin
>> maintainers to ensure that there are no contributors affected by the
>> issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure
>> team meeting where this issue will be discussed in more detail. This is a
>> public meeting, and everyone is welcome to join. Calendar link
>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com>
>>
>> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and
>> Security team members who contributed to this investigation.
>>
>> Best regards,
>>
>> Oleg Nenashev
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/9f9cf846-f5b6-4906-87a7-6f2faf969c9fo%40googlegroups.com.
