Hi Everybody, Some updates regarding this topics, while almost everything is back to normal, we decided to not pursue with the "every user password reset" as we announced initially but instead we focused on maintainers and administrators access. The reason for that is because we don't have ready to use tooling so it requires us to write custom scripts. While we initially tried to go down that path, we reset +-30% of the database, we realized that because of the amount of garbage we have in that database, it was hard and time-consuming to finish this so we decided to look after alternatives for accounts.jenkins.io.
At the moment we have two promising alternatives: Keycloak as a replacement for accounts.jenkins.io. keycloak is an opensource identity management tool, which supports many integrations like Github SSO or LDAP. It's deployed and only available from our VPN at the moment, configuration is defined here <https://github.com/jenkins-infra/charts/pull/256>. It uses a RDS PostgreSQL database running on AWS and containers are running on our AKS cluster. It was easy to deploy, configured, *seems* easy to maintain, and its database is running on a managed service. It sounds very promising as it does exactly everything that accounts.jenkins.io do with a lot more like: * Enforce email verification * OTP * Safe reset password workflow * Using your social account like Github for login * And many more So we could stop losing our time patching our custom identity management tool. The second option would be to totally or partially delegate identity management to the Linux Foundation infrastructure team. We had a first exploratory meeting with them this week and we have another one planned next week The whole idea is the vast majority of Jenkins account are used to report/update issues while the smallest amount of accounts are used by plugin maintainers (+-1700)and Jenkins administrators(+-20). So if we can delegate the management of Jira to them, we wouldn't need to maintain an identity management tool anymore. While implementation details still need to be discussed with them what seems to be clear at the moment are: * Identify management would be a black box, as it would also contain other Linux Foundation accounts. * We could use it for Artifactory (repo.jenkins-ci.org) as they are already doing the same for other communities that they are managing. While we would lose flexibility on this, we wouldn't have to maintain it or care about GDPR. Therefore it will give us more time to focus on other initiatives. If you have any advice, questions, concerns on this topic, feel free to raise them. Thanks for your patience Olivier On Wednesday, 17 June 2020 at 16:59:31 UTC+2 Oleg Nenashev wrote: > Duly noted about the Documentation. > I will migrate https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins to > jenkins.io and extend it to cover the use-case tonight > > On Wed, Jun 17, 2020 at 4:53 PM <kjesc...@d2iq.com> wrote: > >> Hm, that does not work. I am using the Gradle JPI plugin. It does not >> seem to pick up ~/.m2/settings.xml nor ~/.jenkins-ci.org. >> >> >> On June 17, 2020 at 15:52:17, Tim Jacomb (timja...@gmail.com) wrote: >> >> it's just the same as a password to maven, so use the api key instead of >> a password. >> >> On Wed, 17 Jun 2020 at 14:39, <kjesc...@d2iq.com> wrote: >> >>> Hi, >>> >>> thanks for the advice. Hm, my ~/.m2/settings.xml had my encrypted >>> password. The docs ( >>> https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins#HostingPlugins-Releasingtojenkins-ci.org) >>> >>> don’t mention the API key. How can I configure Maven to use the API key >>> instead? >>> >>> Many thanks. >>> Karsten. >>> >>> >>> On June 17, 2020 at 14:53:22, Mark Waite (mark.ea...@gmail.com) wrote: >>> >>> >>> >>> On Wed, Jun 17, 2020 at 6:44 AM Karsten Jeschkies <kjesc...@d2iq.com> >>> wrote: >>> >>>> Hi, >>>> >>>> thanks for you hard work. I reset my password successfully but cannot >>>> upload a release for the Mesos plugin. Are releases still blocked? >>>> >>>> >>> Releases are not blocked but a password reset will also reset your >>> password to the artifact repository. If you're receiving an HTTP 401 when >>> you try to `mvn release perform` you may need to update your password in >>> the ~/.m2/settings.xml. >>> >>> I had to do that in order to release a new version of a plugin >>> yesterday. I logged into the Jenkins Artifactory instance and had it >>> generate an encrypted password from my profile page on that server. I >>> inserted that encrypted password into my ~/.m2/settings.xml file. I'm not >>> sure if that is the preferred way to do it, but it worked for me. >>> >>> Mark Waite >>> >>> >>>> Best. >>>> Karsten. >>>> >>>> On Tuesday, June 9, 2020 at 5:00:25 PM UTC+2, Oleg Nenashev wrote: >>>>> >>>>> Dear all, >>>>> >>>>> As you may have noticed, the release artifact uploads are currently >>>>> blocked in the Jenkins Artifactory instances ( >>>>> https://repo.jenkins-ci.org/). We are doing a security investigation >>>>> due to a partial user database loss on June 02. Today we blocked releases >>>>> to the Jenkins artifactory, and there also was a temporary outage of the >>>>> Artifactory downloads which was a collateral damage of the temporary >>>>> permissions. You can find more details about it in this Jenkins Infra >>>>> Thread >>>>> <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> >>>>> and in this Dev List thread >>>>> <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ> >>>>> . >>>>> >>>>> Current status: >>>>> >>>>> - >>>>> >>>>> Downloads are restored for all artifacts on >>>>> https://repo.jenkins-ci.org/, Jenkins core historical releases, >>>>> Remoting library and Windows Service Wrapper which were among ones >>>>> reported >>>>> by Jenkins users. >>>>> - >>>>> >>>>> Uploads: Jenkins artifact uploads are blocked for the most of >>>>> Jenkins plugin maintainers and contributors. It affects releases of >>>>> Jenkins >>>>> plugins, Jenkins core and modules, developer tools and all libraries >>>>> hosted >>>>> on https://repo.jenkins-ci.org/. Incremental and Snapshot >>>>> deployments are not affected. >>>>> >>>>> >>>>> Quick summary: >>>>> >>>>> - >>>>> >>>>> Jun 02 - There was a Kubernetes Cluster outage on June 02. During >>>>> this outage we had to rebuild the cluster from scratch to get some >>>>> services >>>>> working again. >>>>> - >>>>> >>>>> Jun 02 - After the recovery we lost three months of LDAP changes. >>>>> It has happened due to the broken backup of the LDAP database. >>>>> - >>>>> >>>>> Jun 02 - We identified a number of potential security risks which >>>>> may be caused by the LDAP outage. Account overtake and malicious >>>>> upload was >>>>> one of the identified risks. FTR this issue is tracked as >>>>> SECURITY-1895 as >>>>> a follow-up to these discussions. Only the Security team members have >>>>> access to it, so I am not sharing a link here. >>>>> - >>>>> >>>>> Jun 09 - After the security risk was independently reported in >>>>> public by a plugin maintainer in the dev list thread >>>>> <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we >>>>> decided to block uploads of release artifacts to the Jenkins >>>>> Artifactory >>>>> instance. >>>>> - >>>>> >>>>> Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked >>>>> (plugins, Jenkins core and modules, developer tools, etc.). Downloads >>>>> of >>>>> some binaries were also blocked as an unexpected collateral damage. >>>>> Jenkins >>>>> core historical releases, Remoting library and Windows Service Wrapper >>>>> are >>>>> among the affected binaries >>>>> - >>>>> >>>>> Jun 09, 10AM UTC - We finished reviews of all artifact releases to >>>>> https://repo.jenkins-ci.org/, which happened between the infra >>>>> outage on June 02 and the blockage of the releases. There are no >>>>> maliciously uploaded artifacts. Note that the common plugin release >>>>> flow >>>>> requires access to GitHub in order to push the release commits, so a >>>>> malicious attacker would need to overtake both Jenkins and GitHub >>>>> accounts >>>>> of a single user to submit a legitimately-looking release. >>>>> - >>>>> >>>>> Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch >>>>> >>>>> <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569> >>>>> >>>>> in the Repository Permission Updater was applied to prevent uploads. >>>>> Artifact uploads are still blocking >>>>> - >>>>> >>>>> Jun 09, 2PM UTC, based on repo.jenkins-ci.org and >>>>> issues.jenkins-ci.org data, we restored maintainers accounts. >>>>> >>>>> >>>>> Our next steps would be to communicate the issue to all maintainers >>>>> and contributors who might have been affected by the LDAP history loss. >>>>> We >>>>> will likely need to perform additional user verification steps for plugin >>>>> maintainers to ensure that there are no contributors affected by the >>>>> issues. Today at 3:30PM UTC we will also have a Jenkins >>>>> Infrastructure team meeting where this issue will be discussed in more >>>>> detail. This is a public meeting, and everyone is welcome to join. >>>>> Calendar >>>>> link >>>>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com> >>>>> >>>>> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and >>>>> Security team members who contributed to this investigation. >>>>> >>>>> Best regards, >>>>> >>>>> Oleg Nenashev >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to jenkinsci-de...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-de...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-de...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YXJzCV7jFGLz18BnjePsTVjbf5ch%2B9nVBcjC-QUO5z2T8Q%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YXJzCV7jFGLz18BnjePsTVjbf5ch%2B9nVBcjC-QUO5z2T8Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-de...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3BieEJ4aA5zax1SqVf%2B_EuNL9C3-qEhCjxkiw6PRPXXakEg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3BieEJ4aA5zax1SqVf%2B_EuNL9C3-qEhCjxkiw6PRPXXakEg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> > You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Developers" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> jenkinsci-de...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YX%2BC_yqey%2B8Da5q7oj-grWh15Hz4-JmVY_GTxynYsk%2B7bg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YX%2BC_yqey%2B8Da5q7oj-grWh15Hz4-JmVY_GTxynYsk%2B7bg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/de3cae26-59ae-4544-98fc-8c30e32dda14n%40googlegroups.com.
