Hi Andrew and all I apologize for the late answer. I was pretty sick last week. Again Andrew, thanks for having taken care of the new keyword for GHA and the time to give me all these precisions. It helped me a lot to understand the tricky situation where we are. I feel quite concerned by the budget and head reduction you mentionned in your team.
I am glad you welcomed well my idea of moving this to a project level master repo. I think it would really ease committers' life, which looks important to me. We surely need to attract more contributors and sponsors to reverse the budget trend. I do agree that this proposal would take some re-engineering much deeper than a single change. And sure, it is preferable to have a broader consensus at the community at large for this. I now wonder how I can help with that. I guess the TAC would be ultimately the right place for this kind of topic. But first, I can start by getting others feedbacks here. @TSC What's your opinion about this ? Hope this helps Best Regards Guillaume ________________________________ De : Andrew Grimberg <agrimb...@linuxfoundation.org> Envoyé : vendredi 17 novembre 2023 18:45 À : LAMBERT Guillaume INNOV/NET; Robert Varga; OpenDaylight Discuss; OpenDaylight Dev; t...@lists.opendaylight.org Objet : Re: [OpenDaylight TSC] Gerrit verify changes Guillaume, Again, I'll inline: On 11/17/23 06:23, guillaume.lamb...@orange.com wrote: --[snip]-- > [GL] 'rerun-gha' would suit perfectly as far as I am concerned. Thanks for > this proposal. I've opened an internal change to make this happen. It should active later today. --[snip]-- >>> And with such an enforcement on GHA verified +1 , there would be no way >> for committers to force a merge, even if they performed all tests locally to >> ensure it is safe. >> >> While I understand this sentiment, we've had far too many instances where >> committers ignored or didn't even wait for CI to validate on INFO.yaml >> changes which have caused many problems for those project and for RelEng. >> This is, unfortunately, one of those cases of a few too many rotten apples >> spoiling it for everyone. We just can't allow anyone that right anymore when >> it comes to the configuration files that actually drive access control. >> > [GL] I was contributing before INFO.yaml were introduced and I totally > understand your concern too. > I agree many contributors don't handle it properly. But what about all the > other ones ? > Should we really cut the whole tree because of a few rotten apples ? > AFAIK, this file was introduced only to ease the LFN management of Gerrit > rights. > IMO we should bother contributors at least as possible with such purely LFN > sysadmin concerns. > So I am not convinced at all we need to check this file at every change. > Jenkins allow to check the existence or a modification of a single file > and such a job could be run only when this condition is met. > I would be surprised GHA doesn't handle this too. > But if not, why not running this job separately once a week ? > Is this INFO.yaml used so often by LFN process ? > I guess this is mostly to reconfigure Gerrit rights, isn't it? > > My opinion is that all these INFO.yaml files are likely not at their right > place at the root of every project repo. > IMO, they'd better all be together in a separate DB. > I would suggest a specific repository that lists every projects and where TSC > members have the +2/verified +1 > since every committers' elections must be validated by the TSC. > This would avoid many caveats and ease many process. To answer your question about cutting the whole tree because of a few rotten apples. Unfortunately, we really do have to enforce this way across all of our projects. Over the years budget reductions have reduced forced us to cut back on personnel in Release Engineering. This means that we have a smaller team than when we first rolled out Jenkins Job Builder or the INFO.yaml system for management. As we've had to reduce head count we have to standardize the projects more and more to be able to effectively support them. Since we've spent quiet a large amount of support time on issues related to bad INFO.yaml files we can no longer leave it to the good practices of most committers as that leaves the door open for the bad ones. On the topic of how Jenkins does triggering, you're correct that the triggering can happen more granularly. We do not exactly have the luxury with the integration with GitHub Actions. Even though we wrote the integration the operational design of how Jenkins operates and how GitHub Actions operates are fundamentally different. The Gerrit Trigger plugin for Jenkins effectively does following: * Provide Jenkins with a way to trigger jobs meet certain conditions from Gerrit Events * Have a singular start of _all_ jobs notice and a singular final resolution of _all_ jobs notice to Gerrit (this is effectively a single GitHub Action workflow that votes at the start of all jobs and at the end of all jobs) Our methodology for the tie between Gerrit and GitHub is fundamentally lighter than the way that plugin for Jenkins operates. The reason being that we cannot directly integrate a plugin in GitHub the way Jenkins can. We could implement a GitHub bot, but then there would be additional infrastructure that would need to be setup and maintained. Instead, we elected to utilize triggering based on Gerrit events using the hooks system that Gerrit natively has. This means that the onus of voting moves from a gathering of all the jobs in a single event (virtual workflow) to disparate GitHub workflows and introducing potential voting race conditions. How does this affect the INFO.yaml validation? In a couple of ways: 1) Gerrit hooks have a lot less information available to them when they trigger and a Gerrit event picked up by the Jenkins plugin, thus we don't actually know _what_ files changed when the hook fires. This means that we have to do that checking in a job. 2) Since we now have to do this file change checking in the job, we have to make sure that INFO.yaml files are being properly validated. Historically all projects got a Jenkins job to do this, but as the entire workflow for GitHub Actions moves into the project repository and out of a manged global job definition, we lose the ability to make sure / enforce that those changes are still valid. As such, we introduce the concept of Gerrit wide required workflows that allow us to hard enforce this system. We built the job to be as lightweight as possible. 3) With global required validation this opens the possibility for the TSCs to say projects must run any N sets of jobs. While ODL, FDio, and Anuket do not presently do that (outside of the one now being mandated by LF Release Engineering in support of the Gerrit platform) ONAP _is_ using this as they already had that mandate in place for documentation related jobs. Our moving this into GitHub Actions as a non-overridable job has already helped with managing that project. So, given these reasons, yes, it's actually required that this job fire on every change. We had to move the validation of INFO.yaml to a completely separate CI user just to make sure that we could enforce the voting from that user _and_ not impact GitHub Actions jobs that a given project may desire to use in the future. Now, as to what the INFO.yaml file is doing for projects, you're correct that it's defining rights. Is it used often by LFN process? By ODL, not much as you rarely change committers or add new repositories. By ONAP, we see changes regularly to their INFO.yaml files, mostly from new repositories being created, but also from committer changes that happen semi-regularly. Could this be moved to something that happens once a week or less often? No, as a bad change getting into the system needs to be prevented at the time the change is being made, not after the fact. Otherwise, we're back in the situation where Release Engineering is having to fix broken things because some committers don't want to wait for validation from the CI systems. Now, should the INFO.yaml live somewhere else? Technically, they already do. The LF Gerrit system has a `releng/info-master` repository which is the _actual_ configuration definition. INFO.yaml all live in the repos that they manage for a very simple reason, the community at large when we were creating the system wanted the definitions to be somewhere that was easy for them to update and vote on changes for. Could we dismantle the fact that the files live in the repositories? Yes, but then we would have to rework how the files get properly voted on _and_ the fact that over 95% of all active committers have never logged into the LF Gerrit system. Could this all be moved into some project level master repository? Potentially, but as I said, the community at large did not like that idea at the time. I'm open to such an idea, it would just take some semi-significant re-engineering of the INFO management system on our part so it wouldn't be a simple change. I will also note that committer ratification by TSC is not required by all projects in LFN. As such, while that's the norm for ODL, it is not for everyone else. For instance ONAP allows committer modification with no ratification from the TSC as long as the modification gets quorum from current committers of a project. Additionally, ONAP also allows for new repository creation under an existing top level repository with no requirement of ratification from the TSC. --[snip]-- -- Andrew J Grimberg Senior Manager, Release Engineering The Linux Foundation NOTICE: The Linux Foundation supports their employees with flexible work hours. If you recieve mail from me outside of standard business hours please be aware that I do not expect a response until the next standard business day. ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9047): https://lists.opendaylight.org/g/Discuss/message/9047 Mute This Topic: https://lists.opendaylight.org/mt/102585993/21656 Group Owner: discuss+ow...@lists.opendaylight.org Unsubscribe: https://lists.opendaylight.org/g/Discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
