There are a lot of companies now with a business interest in the success of Apache Arrow. We should do a back of the envelope estimate of the number of compute hours we use per year and arrive at an annual cost estimate. My guess (and maybe I’m wildly off) is that it’s somewhere around $50,000-100,000/year, which shouldn’t be difficult to come by. Costs could also be mitigated by organizations hooking up physical on-prem build machines.
On Tue, Jan 5, 2021 at 4:29 PM Andrew Lamb <al...@influxdata.com> wrote: > One thing that is not clear to me is who will pay for the dedicated > infrastructure costs with something like Build Kite? > > I don't fully understand how moving to self hosted (and thus also > presumably limited) arrow-only infrastructure will reduce build queue > times, unless there is a deep pocketed sponsor. I can certainly see how > moving to self hosted infrastructure will increase the amount of control / > visibility the project would have > > Andrew > > > > On Tue, Jan 5, 2021 at 12:46 PM Jorge Cardoso Leitão < > jorgecarlei...@gmail.com> wrote: > > > Krisztián, > > > > I agree with you that there is an ongoing problem with the queue. Thanks > > for the tips wrt to WIP. > > > > Wes, I would be up for moving Rust workflows to Buildkite. Is the > > integration in place wrt to reporting to triggers and reporting back to > > gihtub? I.e. can we just place a `pipeline.yml` on the repo, or is there > > any PMC-specific activity that is blocking us from working on this? > > > > Best, > > Jorge > > > > > > > > > > On Tue, Jan 5, 2021 at 5:10 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > At the risk of sounding like a broken record -- we are almost > > > certainly going to have to move our builds to dedicated infrastructure > > > that this community has complete agency over sometime between now and > > > 2025. Maybe it will be this year, maybe next year, but to me it is an > > > inevitability. I spent quite a bit of time getting Apache Infra to > > > enable Buildkite for apache/* repositories precisely so that we have a > > > coherent path. I hope to be able to have folks on my team focus on > > > this at some point and contribute it to the community once we have the > > > bandwidth. > > > > > > On Tue, Jan 5, 2021 at 10:01 AM Krisztián Szűcs > > > <szucs.kriszt...@gmail.com> wrote: > > > > > > > > On Tue, Jan 5, 2021 at 4:40 PM Jeroen Ooms <jeroeno...@gmail.com> > > wrote: > > > > > > > > > > On Tue, Jan 5, 2021 at 1:34 PM Krisztián Szűcs > > > > > <szucs.kriszt...@gmail.com> wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm concerned about the overall feedback time we have on pull > > > requests. > > > > > > I have a simple PR to make the comment bot working again, but no > > > > > > builds are running even after 30 minutes. > > > > > > > > > > FWIW, GitHub seems to queue jobs by pusher, not by repo. I have the > > > > > same problem sometimes with pull requests taking long to build, but > > > > > this usually happens when there are large jobs running under my > name > > > > > in completely unrelated projects. Also note that if you use e.g. > > > > > scheduled (cron) actions, then the jobs are executed in the queue > of > > > > > the github user who last touched the yml file for that workflow. > > > > Based on the "Actions" tab the queueing seems chronological to me, > > > > although I can imagine the behaviour you described. > > > > > > > > Has anyone experienced similar clogging on apache/arrow's gha queue > > > lately? > > > > > >
