Thanks for reviving the efforts here Matthias! +1 for the transition to GitHub Actions.
As for ASF Infra Jenkins, it works fine. Jenkins is extremely feature-rich. Not sure about the spare capacity though. I know that for Apache Beam, Google donated a bunch of servers to get additional build capacity. -Max On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl <matthias.p...@aiven.io.invalid> wrote: > > Btw. even though we've been focusing on GitHub Actions with this FLIP, I'm > curious whether somebody has experience with Apache Infra's Jenkins > deployment. The discussion I found about Jenkins [1] is quite out-dated > (2014). I haven't worked with it myself but could imagine that there are > some features provided through plugins which are missing in GitHub Actions. > > [1] https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4 > > On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl <matthias.p...@aiven.io> > wrote: > > > That's a valid point. I updated the FLIP accordingly: > > > >> Currently, the secrets (e.g. for S3 access tokens) are maintained by > >> certain PMC members with access to the corresponding configuration in the > >> Azure CI project. This responsibility will be moved to Apache Infra. They > >> are in charge of handling secrets in the Apache organization. As a > >> consequence, updating secrets is becoming a bit more complicated. This can > >> be still considered an improvement from a legal standpoint because the > >> responsibility is transferred from an individual company (i.e. Ververica > >> who's the maintainer of the Azure CI project) to the Apache Foundation. > > > > > > On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser <martijnvis...@apache.org> > > wrote: > > > >> Hi Matthias, > >> > >> Thanks for the write-up and for the efforts on this. I really hope > >> that we can move away from Azure towards GHA for a better integration > >> as well (directly seeing if a PR can be merged due to CI passing for > >> example). > >> > >> The one thing I'm missing in the FLIP is how we would setup the > >> secrets for the nightly runs (for the S3 tests, potential tests with > >> external services etc). My guess is we need to provide the secret to > >> ASF Infra and then we would be able to refer to them in a pipeline? > >> > >> Best regards, > >> > >> Martijn > >> > >> On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl > >> <matthias.p...@aiven.io.invalid> wrote: > >> > > >> > I realized that I mixed up FLIP IDs. FLIP-395 is already reserved [1]. I > >> > switched to FLIP-396 [2] for the sake of consistency. 8) > >> > > >> > [1] https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv > >> > [2] > >> > > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions > >> > > >> > On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl <matthias.p...@aiven.io> > >> > wrote: > >> > > >> > > Hi everyone, > >> > > > >> > > The Flink community discussed migrating from Azure CI to GitHub > >> Actions > >> > > quite some time ago [1]. The efforts around that stalled due to > >> limitations > >> > > around self-hosted runner support from Apache Infra’s side. There > >> were some > >> > > recent developments on that topic. Apache Infra is experimenting with > >> > > ephemeral runners now which might enable us to move ahead with GitHub > >> > > Actions. > >> > > > >> > > The goal is to join the trial phase for ephemeral runners and > >> experiment > >> > > with our CI workflows in terms of stability and performance. At the > >> end we > >> > > can decide whether we want to abandon Azure CI and move to GitHub > >> Actions > >> > > or stick to the former one. > >> > > > >> > > Nico Weidner and Chesnay laid the groundwork on this topic in the > >> past. I > >> > > picked up the work they did and continued experimenting with it in my > >> own > >> > > fork XComp/flink [2] the past few weeks. The workflows are in a state > >> where > >> > > I think that we start moving the relevant code into Flink’s > >> repository. > >> > > Example runs for the basic workflow [3] and the extended (nightly) > >> workflow > >> > > [4] are provided. > >> > > > >> > > This will bring a few more changes to the Flink contributors. That is > >> why > >> > > I wanted to bring this discussion to the mailing list first. I did a > >> write > >> > > up on (hopefully) all related topics in FLIP-395 [5]. > >> > > > >> > > I’m looking forward to your feedback. > >> > > > >> > > Matthias > >> > > > >> > > [1] https://lists.apache.org/thread/vcyx2nx0mhklqwm827vgykv8pc54gg3k > >> > > > >> > > [2] https://github.com/XComp/flink/actions > >> > > > >> > > [3] https://github.com/XComp/flink/actions/runs/6926309782 > >> > > > >> > > [4] https://github.com/XComp/flink/actions/runs/6927443941 > >> > > > >> > > [5] > >> > > > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-395%3A+Migration+to+GitHub+Actions > >> > > > >> > > > >> > > -- > >> > > > >> > > [image: Aiven] <https://www.aiven.io> > >> > > > >> > > *Matthias Pohl* > >> > > Opensource Software Engineer, *Aiven* > >> > > matthias.p...@aiven.io <i...@aiven.io> | +49 170 9869525 > >> > > aiven.io <https://www.aiven.io> | > >> > > <https://www.facebook.com/aivencloud> > >> > > <https://www.linkedin.com/company/aiven/> < > >> https://twitter.com/aiven_io> > >> > > *Aiven Deutschland GmbH* > >> > > Alexanderufer 3-7, 10117 Berlin > >> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > >> > > Amtsgericht Charlottenburg, HRB 209739 B > >> > > > >> > >
