FYI it looks like the transition to new Jenkins CI is visible on Nexmark performance graphs[1][2]. Are new VM nodes less performant than old ones?
[1] hhttp:// 104.154.241.245/d/ahuaA_zGz/nexmark?orgId=1&from=1587597387737&to=1595373387737&var-processingType=batch&var-ID=All&var-runner=All [2] https://issues.apache.org/jira/browse/BEAM-10542?focusedCommentId=17162374&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17162374 On Thu, Jun 18, 2020 at 3:32 PM Tyson Hamilton <[email protected]> wrote: > Currently no. We're already experiencing a backlog of builds so the > additional load would be a problem. I've opened two related issues that I > think need completion before allowing non-committers to trigger tests: > > Load sharing improvements: > https://issues.apache.org/jira/browse/BEAM-10281 > Admin access (maybe not required but nice to have): > https://issues.apache.org/jira/browse/BEAM-10280 > > I created https://issues.apache.org/jira/browse/BEAM-10282 to track > opening up triggering for non-committers. > > On Thu, Jun 18, 2020 at 3:30 PM Luke Cwik <[email protected]> wrote: > >> Was about to ask the same question, so can non-committers trigger the >> tests now? >> >> On Thu, Jun 18, 2020 at 11:54 AM Heejong Lee <[email protected]> wrote: >> >>> This is awesome. Could non-committers also trigger the test now? >>> >>> On Wed, Jun 17, 2020 at 6:12 AM Damian Gadomski < >>> [email protected]> wrote: >>> >>>> Hello, >>>> >>>> Good news, we've just migrated to the new CI: >>>> https://ci-beam.apache.org. As from now beam projects at >>>> builds.apache.org are disabled. >>>> >>>> If you experience any issues with the new setup please let me know, >>>> either here or on ASF slack. >>>> >>>> Regards, >>>> Damian >>>> >>>> On Mon, Jun 15, 2020 at 10:40 PM Damian Gadomski < >>>> [email protected]> wrote: >>>> >>>>> Happy to see your positive response :) >>>>> >>>>> @Udi Meiri, Thanks for pointing that out. I've checked it and indeed >>>>> it needs some attention. >>>>> >>>>> There are two things basing on my research: >>>>> >>>>> - data uploaded by performance and load tests by the jobs, >>>>> directly to the influx DB - that should be handled automatically as new >>>>> jobs will upload the same data in the same way >>>>> - data fetched using Jenkins API by the metrics tool >>>>> (syncjenkins.py) - here the situation is a bit more complex as the >>>>> script >>>>> relies on the build number (it's used actually as a time reference and >>>>> primary key in the DB is created from it). To avoid refactoring of the >>>>> script and database migration to use timestamp instead of build number >>>>> I've >>>>> just "fast-forwarded" the numbers on the new >>>>> https://ci-beam.apache.org to follow current numbering from the >>>>> old CI. Therefore simple replacement of the Jenkins URL in the metrics >>>>> scripts should do the trick to have continuous metrics data. I'll check >>>>> that tomorrow on my local grafana instance. >>>>> >>>>> Please let me know if there's anything that I missed. >>>>> >>>>> Regards, >>>>> Damian >>>>> >>>>> On Mon, Jun 15, 2020 at 8:05 PM Alexey Romanenko < >>>>> [email protected]> wrote: >>>>> >>>>>> Great! Thank you for working on this and letting us know. >>>>>> >>>>>> On 12 Jun 2020, at 16:58, Damian Gadomski < >>>>>> [email protected]> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> During the last few days, I was preparing for the Beam Jenkins >>>>>> migration from builds.apache.org to ci-beam.apache.org. The new >>>>>> Jenkins Master will be dedicated only for Beam related jobs, all Beam >>>>>> Committers will have build configure access, and Beam PMC will have Admin >>>>>> (GUI) Access. >>>>>> >>>>>> We (in cooperation with Infra) are almost ready for the migration >>>>>> itself and I want to share with you the details of our plan. We are >>>>>> planning to start the migration next week, most likely on Tuesday. I'll >>>>>> keep you updated on the progress. We do not expect any issues nor the >>>>>> outage of the CI services, everything should be more or less >>>>>> unnoticeable. >>>>>> Just don't be surprised that the Jenkins URL will change to >>>>>> https://ci-beam.apache.org >>>>>> >>>>>> If you are curious, here are the steps that we are going to take: >>>>>> >>>>>> 1. Create 16 new CI nodes that will be connected to the new CI. We >>>>>> will then have simultaneously running two CI servers. >>>>>> 2. Verify that new builds work as expected on the new instance >>>>>> (compare results of cron builds). (a day or two would be sufficient) >>>>>> 3. Move the responsibility of Phrase/PR/Commit builds to the new CI, >>>>>> disable on the old one. >>>>>> 4. Modify the .test-infra/jenkins/README.md to point to the new >>>>>> instance and replace Post-commit tests status in README.md and >>>>>> .github/PULL_REQUEST_TEMPLATE.md >>>>>> 5. Disable the jobs on the old Jenkins and add a description to each >>>>>> job with the URL to the corresponding one on the new CI. >>>>>> 6. Turn off VM instances of the old nodes. >>>>>> 7. Remove VM instances of the old nodes. >>>>>> >>>>>> In case of any questions or doubts feel free to ask :) >>>>>> >>>>>> Regards, >>>>>> Damian >>>>>> >>>>>> >>>>>>
