Re: Github Actions feedback time

Hi Wes, Just a quick update here. I spawned a small ubuntu VM on Azure with a mounted volume to try out the dockerization. I used rootless mode since we would like to run PRs but we have no control over what people PR to the repo and the pipelin

Re: Github Actions feedback time

Good point re re-use of our other Docker images. Probably the best path would be to attach a few bare-metal machines or spin up some persistent cloud instances (if we figure out how they're going to be paid for). On Thu, Jan 7, 2021 at 2:15 PM Jorge Cardoso Leitão wrote: > > Hi Wes, > > Thanks fo

Re: Github Actions feedback time

Hi Wes, Thanks for the suggestion and for the offer; much appreciated. I was able to run it on my own macbook on the dockerized agent , and it just ran out of the box. The main limitation of this is that it is more difficult to run ou

Re: Github Actions feedback time

Jorge -- if you want to test a Linux agent, you could run the buildkite-agent in a Docker container. We (Ursa) could possibly look into adding Dockerized agents on some of our physical machines, particularly if we set up a well-documented procedure for setting this up on a new machine. On Wed, Jan

Re: Github Actions feedback time

Hi Jacob, Neal already requested those (and other) actions to be whitelisted here https://issues.apache.org/jira/browse/INFRA-21239, but there was no response yet. Best, Jorge On Thu, Jan 7, 2021 at 6:20 AM Jacob Quinn wrote: > From this page, it looks like there have been certain github orga

Re: Github Actions feedback time

>From this page, it looks like there have been certain github organizations that have been "whitelisted" to allow their github actions to run. Is there a process to do this whitelisting? If the `julia-actions` github org was allowed to run, that would enable everything needed for Julia CI to run.

Re: Github Actions feedback time

Hi, > I wasn't following the build queue's state lately, but I think we > should consolidate the build configurations. > Possible candidates are the PR* workflows https://github.com/apache/arrow/pull/9120 Thanks, -- kou In "Github Actions feedback time" on Tue, 5 Jan 2021 13:33:38 +0100,

Re: Github Actions feedback time

Hi, Sorry for not noticing this thread earlier. Looks like in addition to unusually slow feedback time that did not happen last Sunday or earlier there are also weird installation errors such as ‘can not install numpy’ as well. Can these be due to some form of timeout? Here is my C++ PR: https

Re: Github Actions feedback time

Hi Wes, I fielded ARROW-11140 with what I would consider a successful POC from Rust implementation's point of view. Let's move the discussion there. I was already able to run Rust's first steps there (using my own mac as a runner), and, so far, l

Re: Github Actions feedback time

Random question - could Github be reached out to for the limits to be increased/investigated in the mean time, given this is an opensource project? On Wed, 6 Jan 2021 at 06:57, Andrew Lamb wrote: > Makes sense -- thank you > > On Tue, Jan 5, 2021 at 5:38 PM Wes McKinney wrote: > > > There are a

Re: Github Actions feedback time

Makes sense -- thank you On Tue, Jan 5, 2021 at 5:38 PM Wes McKinney wrote: > There are a lot of companies now with a business interest in the success of > Apache Arrow. We should do a back of the envelope estimate of the number of > compute hours we use per year and arrive at an annual cost est

Re: Github Actions feedback time

There are a lot of companies now with a business interest in the success of Apache Arrow. We should do a back of the envelope estimate of the number of compute hours we use per year and arrive at an annual cost estimate. My guess (and maybe I’m wildly off) is that it’s somewhere around $50,000-100,

Re: Github Actions feedback time

One thing that is not clear to me is who will pay for the dedicated infrastructure costs with something like Build Kite? I don't fully understand how moving to self hosted (and thus also presumably limited) arrow-only infrastructure will reduce build queue times, unless there is a deep pocketed sp

Re: Github Actions feedback time

hi Jorge -- as far as I know, all that's necessary to use BK is the pipeline configuration in the repo and to attach workers to Arrow's (free) account (which I'm happy to add you to). We can likely get a lot of free cloud credits from Google, Microsoft, or Amazon if we wanted to go that route to ke

Re: Github Actions feedback time

Krisztián, I agree with you that there is an ongoing problem with the queue. Thanks for the tips wrt to WIP. Wes, I would be up for moving Rust workflows to Buildkite. Is the integration in place wrt to reporting to triggers and reporting back to gihtub? I.e. can we just place a `pipeline.yml` on

Re: Github Actions feedback time

At the risk of sounding like a broken record -- we are almost certainly going to have to move our builds to dedicated infrastructure that this community has complete agency over sometime between now and 2025. Maybe it will be this year, maybe next year, but to me it is an inevitability. I spent qui

Re: Github Actions feedback time

On Tue, Jan 5, 2021 at 4:40 PM Jeroen Ooms wrote: > > On Tue, Jan 5, 2021 at 1:34 PM Krisztián Szűcs > wrote: > > > > Hi, > > > > I'm concerned about the overall feedback time we have on pull requests. > > I have a simple PR to make the comment bot working again, but no > > builds are running eve

Re: Github Actions feedback time

On Tue, Jan 5, 2021 at 1:34 PM Krisztián Szűcs wrote: > > Hi, > > I'm concerned about the overall feedback time we have on pull requests. > I have a simple PR to make the comment bot working again, but no > builds are running even after 30 minutes. FWIW, GitHub seems to queue jobs by pusher, not

Re: Github Actions feedback time

On Tue, Jan 5, 2021 at 3:21 PM Jorge Cardoso Leitão wrote: > > Hi Krisztian, > > Could you describe which flows you are referring to? > > Rust builds are taking 4-6m on non-(windows/mac) and 8-15m on windows/mac, > with almost immediate feedback. E.g. I force-pushed recently > https://github.com/a

Re: Github Actions feedback time

Hi Krisztian, Could you describe which flows you are referring to? Rust builds are taking 4-6m on non-(windows/mac) and 8-15m on windows/mac, with almost immediate feedback. E.g. I force-pushed recently https://github.com/apache/arrow/pull/9089 and the first complete test came 5m later. One hypo