As we know this is a common issue among Apache projects. While the projects do not have the final solution, Apache Spark project has a mechanism [1][2] to run a test in own local (forked) repository. Can we alleviate the problem a little bit?
[1] https://github.com/apache/spark/pull/29504 [2] https://github.com/apache/spark-website/pull/286 Regards, Kazuaki Ishizaki Weston Pace <weston.p...@gmail.com> wrote on 2021/04/15 09:13:05: > From: Weston Pace <weston.p...@gmail.com> > To: dev@arrow.apache.org > Date: 2021/04/15 09:13 > Subject: [EXTERNAL] Re: CI feedback time > > It may be worth reaching out to the Airflow project. Based on > INVALID URI REMOVED > u=https-3A__cwiki.apache.org_confluence_display_BUILDS_GitHub-2BActions-2Bstatus&d=DwIFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=jvAxQ5CeH5B- > RJFoUC0C578xEYg1v24trTR- > NGxnIJU&s=r_4tQJDEfMyTcFxbBMmOFuNrn_yH8bi2dKTMnqUkVrU&e= > it seems they have been investing time into figuring how to make > self-hosted runners work (it seems Github's patching model makes this > somewhat difficult). > > On Wed, Apr 14, 2021 at 12:28 PM Antoine Pitrou <anto...@python.org> wrote: > > > > > > Hi Krisztian, > > > > Thanks for bringing this up. This is definitely becoming a > > high-priority topic for Arrow development. > > > > I don't believe there is much opportunity for reducing the number of > > builds or their runtime. We simply have a lot of development going on, > > and the number of different CI jobs we have is simply because we need to > > support many different configurations (and past experience has shown > > that they quickly stop working if we don't monitor them on a regular basis). > > > > So I think the only path forward is to build up (== buy, probably) our > > own execution resources for CI. Whether that entails using Github > > self-hosted runners, Buildkite, or yet another system, I have no idea. > > > > I'll submit two requirements though: > > - the configuration for CI builds must be kept in the Arrow repository > > (as they are currently in .github, etc.) > > - CI builds must be runnable from PRs > > > > Regards > > > > Antoine. > > > > > > Le 15/04/2021 à 00:14, Krisztián Szűcs a écrit : > > > Hi, > > > > > > The Apache Github Actions agent pool seems to be oversubscribed as > > > more Apache projects migrate their CI setup to GHA. We experienced > > > pretty solid feedback times (~20-30m) when we originally moved to GHA > > > but now we are roughly 5hrs behind [1]. > > > > > > Based on other projects' complaints and discussions [2][3] (doesn't > > > have all the links at hand) we can't expect a short term solution from > > > infra. I think we *need* to figure out something on the project level > > > instead to maintain the overall project health and to improve the > > > development velocity. > > > > > > I don't have a concrete proposal at the moment, but we should start to > > > collect the available options. Ideas? > > > > > > Thanks, Krisztian > > > > > > [1]: INVALID URI REMOVED > u=https-3A__github.com_apache_arrow_actions-3Fquery-3Dis-253Ain-5Fprogress&d=DwIFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=jvAxQ5CeH5B- > RJFoUC0C578xEYg1v24trTR-NGxnIJU&s=GN2DAt- > n72kMqfoaVZd9aNn_6eLGhTKb4uMBgJvLrNs&e= > > > [2]: INVALID URI REMOVED > u=https-3A__github.com_apache_pulsar_issues_9154&d=DwIFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=jvAxQ5CeH5B- > RJFoUC0C578xEYg1v24trTR- > NGxnIJU&s=2GvMYXYBAdKvDWPLlxgQYhkk0pyPyzq9mRvBevvVqsM&e= > > > [3]: INVALID URI REMOVED > u=https-3A__issues.apache.org_jira_browse_SPARK-2D34053&d=DwIFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=jvAxQ5CeH5B- > RJFoUC0C578xEYg1v24trTR-NGxnIJU&s=fiQY4K7tuzBqXG8csD- > tN1nDKAh5S_gn7Sotng6GMdg&e= > > > >
