Hi Krisztian,
Thanks for bringing this up. This is definitely becoming a
high-priority topic for Arrow development.
I don't believe there is much opportunity for reducing the number of
builds or their runtime. We simply have a lot of development going on,
and the number of different CI jobs we have is simply because we need to
support many different configurations (and past experience has shown
that they quickly stop working if we don't monitor them on a regular basis).
So I think the only path forward is to build up (== buy, probably) our
own execution resources for CI. Whether that entails using Github
self-hosted runners, Buildkite, or yet another system, I have no idea.
I'll submit two requirements though:
- the configuration for CI builds must be kept in the Arrow repository
(as they are currently in .github, etc.)
- CI builds must be runnable from PRs
Regards
Antoine.
Le 15/04/2021 à 00:14, Krisztián Szűcs a écrit :
Hi,
The Apache Github Actions agent pool seems to be oversubscribed as
more Apache projects migrate their CI setup to GHA. We experienced
pretty solid feedback times (~20-30m) when we originally moved to GHA
but now we are roughly 5hrs behind [1].
Based on other projects' complaints and discussions [2][3] (doesn't
have all the links at hand) we can't expect a short term solution from
infra. I think we *need* to figure out something on the project level
instead to maintain the overall project health and to improve the
development velocity.
I don't have a concrete proposal at the moment, but we should start to
collect the available options. Ideas?
Thanks, Krisztian
[1]: https://github.com/apache/arrow/actions?query=is%3Ain_progress
[2]: https://github.com/apache/pulsar/issues/9154
[3]: https://issues.apache.org/jira/browse/SPARK-34053
