Re: Travis CI delays
I'd like to mention that we have the option to reuse the buildbot builders as well. I've put up a PR [1] which runs the exact same builders (without the CUDA ones) what we have in the buildbot setup [2]. The required GitHub Actions configuration is fairly small [3]. [1]: https://github.com/apache/arrow/pull/5536 [2]: https://ci.ursalabs.org/#/builders [3]: https://github.com/apache/arrow/pull/5536/files On Sat, Sep 28, 2019 at 1:36 AM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > Hello, > > I suggest we tackle https://jira.apache.org/jira/browse/ARROW-5801. > For Rust, that would be > https://jira.apache.org/jira/browse/ARROW-5809. Once ported to > docker/docker-compose, it's trivial to activate github action for the > same test (see https://github.com/apache/arrow/pull/5530). As I'm > writing this email (4 hours after the PR was pushed), the travis > checks are still not completed due to the queue. While the github > action was completed after 22 minutes > (https://github.com/apache/arrow/pull/5530/checks?check_run_id=239290338). > > François > > > On Fri, Sep 27, 2019 at 11:22 AM Krisztián Szűcs > wrote: > > > > On Fri, Sep 27, 2019 at 5:07 PM Andy Grove > wrote: > > > > > Thanks Krisztian. That's very helpful. I will create a CI page on the > wiki > > > and add this info. > > > > > > Does anyone have any objections to me trying out GitHub Actions for > running > > > the Rust tests on PR builds? I could try this out on my own fork first. > > > > > I think GitHub Actions is a good idea, especially for easier build setups > > like Rust > > has. Go, Node as similarly straightforward, so we could decommission the > > travis > > counterparts hopefully speeding up the rest of the builds a bit. > > > > > > > > On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs < > szucs.kriszt...@gmail.com > > > > > > > wrote: > > > > > > > On Fri, Sep 27, 2019 at 3:25 PM Andy Grove > > > wrote: > > > > > > > > > I've been poking around on the Arrow website and wiki and I can't > find > > > > > documentation relating to CI. Do we have any documentation on how > > > things > > > > > work today or what the goals are? > > > > > > > > I don't think so. > > > > > > > > > For Rust builds it isn't immediately > > > > > obvious why they are building on both Travis CI and Ursabot. > > > > > > > > > We have another thread [1], where we discuss multiple things about > > > > Buildbot (ursabot). I've enabled the buildbot builder for rust > because it > > > > provides much quicker feedback than travis does. > > > > > > > > [1]: > > > > > > > > > > > > https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney > > > > wrote: > > > > > > > > > > > FYI, using the kibble.dev link from INFRA-18533, it seems that > this > > > > > > September we're using about 15% of the ASF's total Travis CI > capacity > > > > > > (60 concurrent workers I think) > > > > > > > > > > > > https://imgur.com/a/oOrbPsj > > > > > > > > > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. > > > Suffice > > > > > > to say the ASF's Travis couldn't accommodate us if we had twice > as > > > > > > many pull requests > > > > > > > > > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney < > wesmck...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > I've been sounding the alarm bells about this for a while. We > need > > > to > > > > > > > work to get ourselves off of Travis CI, but it is not going to > be > > > > > > > easy.t > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > > > > > emkornfi...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > My understanding is the Travis CI queue is shared among all > > > apache > > > > > > > > projects, and there are few including Arrow that make heavy > use > > > of > > > > > the > > > > > > > > resources. Hence, a lot of time waiting for jobs to start. > I > > > > think > > > > > > there > > > > > > > > are some open JIRAs to finish dockerization of builds, I > don't > > > know > > > > > the > > > > > > > > current status of finding alternative CI sources though. > > > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove < > > > andygrov...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I know this has been discussed in the past, and I > apologize for > > > > not > > > > > > paying > > > > > > > > > attention at the time (and searching for arrow + travis in > > > email > > > > > > isn't very > > > > > > > > > effective) but why does it take so long for our Travis CI > > > builds > > > > > and > > > > > > are > > > > > > > > > there open JIRA issues related to this? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Andy. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Travis CI delays
Hi, We can't use cache on GitHub Actions for now: https://github.community/t5/GitHub-Actions/Caching-files-between-GitHub-Action-executions/m-p/30974/highlight/true#M630 > We're working on caching packages and artifacts between > workflow executions, we'll have it by mid-November. Thanks, -- kou In <479f115c-4640-8fef-9e72-d085d09ea...@python.org> "Re: Travis CI delays" on Sat, 28 Sep 2019 10:11:19 +0200, Antoine Pitrou wrote: > > Is it easy to enable caching (e.g. ccache) with docker-compose and > Github Actions? > > Regards > > Antoine. > > > Le 28/09/2019 à 01:36, Francois Saint-Jacques a écrit : >> Hello, >> >> I suggest we tackle https://jira.apache.org/jira/browse/ARROW-5801. >> For Rust, that would be >> https://jira.apache.org/jira/browse/ARROW-5809. Once ported to >> docker/docker-compose, it's trivial to activate github action for the >> same test (see https://github.com/apache/arrow/pull/5530). As I'm >> writing this email (4 hours after the PR was pushed), the travis >> checks are still not completed due to the queue. While the github >> action was completed after 22 minutes >> (https://github.com/apache/arrow/pull/5530/checks?check_run_id=239290338). >> >> François >> >> >> On Fri, Sep 27, 2019 at 11:22 AM Krisztián Szűcs >> wrote: >>> >>> On Fri, Sep 27, 2019 at 5:07 PM Andy Grove wrote: >>> >>>> Thanks Krisztian. That's very helpful. I will create a CI page on the wiki >>>> and add this info. >>>> >>>> Does anyone have any objections to me trying out GitHub Actions for running >>>> the Rust tests on PR builds? I could try this out on my own fork first. >>>> >>> I think GitHub Actions is a good idea, especially for easier build setups >>> like Rust >>> has. Go, Node as similarly straightforward, so we could decommission the >>> travis >>> counterparts hopefully speeding up the rest of the builds a bit. >>> >>>> >>>> On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs >>>> >>>> wrote: >>>> >>>>> On Fri, Sep 27, 2019 at 3:25 PM Andy Grove >>>> wrote: >>>>> >>>>>> I've been poking around on the Arrow website and wiki and I can't find >>>>>> documentation relating to CI. Do we have any documentation on how >>>> things >>>>>> work today or what the goals are? >>>>> >>>>> I don't think so. >>>>> >>>>>> For Rust builds it isn't immediately >>>>>> obvious why they are building on both Travis CI and Ursabot. >>>>>> >>>>> We have another thread [1], where we discuss multiple things about >>>>> Buildbot (ursabot). I've enabled the buildbot builder for rust because it >>>>> provides much quicker feedback than travis does. >>>>> >>>>> [1]: >>>>> >>>>> >>>> https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E >>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney >>>>> wrote: >>>>>> >>>>>>> FYI, using the kibble.dev link from INFRA-18533, it seems that this >>>>>>> September we're using about 15% of the ASF's total Travis CI capacity >>>>>>> (60 concurrent workers I think) >>>>>>> >>>>>>> https://imgur.com/a/oOrbPsj >>>>>>> >>>>>>> The highest is Apache Druid (incubating) at 18%, so we are #2. >>>> Suffice >>>>>>> to say the ASF's Travis couldn't accommodate us if we had twice as >>>>>>> many pull requests >>>>>>> >>>>>>> On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney >>>>>> wrote: >>>>>>>> >>>>>>>> I've been sounding the alarm bells about this for a while. We need >>>> to >>>>>>>> work to get ourselves off of Travis CI, but it is not going to be >>>>>>>> easy.t >>>>> >>>>>>> >>>>>>>> On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < >>>>>> emkornfi...@gmail.com> >>>>>>> wrote: >>>>>>>>> >>>>>>>>> My understanding is the Travis CI queue is shared among all >>>> apache >>>>>>>>> projects, and there are few including Arrow that make heavy use >>>> of >>>>>> the >>>>>>>>> resources. Hence, a lot of time waiting for jobs to start. I >>>>> think >>>>>>> there >>>>>>>>> are some open JIRAs to finish dockerization of builds, I don't >>>> know >>>>>> the >>>>>>>>> current status of finding alternative CI sources though. >>>>>>>>> >>>>>>>>> On Thu, Sep 26, 2019 at 10:24 PM Andy Grove < >>>> andygrov...@gmail.com >>>>>> >>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I know this has been discussed in the past, and I apologize for >>>>> not >>>>>>> paying >>>>>>>>>> attention at the time (and searching for arrow + travis in >>>> email >>>>>>> isn't very >>>>>>>>>> effective) but why does it take so long for our Travis CI >>>> builds >>>>>> and >>>>>>> are >>>>>>>>>> there open JIRA issues related to this? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Andy. >>>>>>>>>> >>>>>>> >>>>>> >>>>> >>>>
Re: Travis CI delays
Is it easy to enable caching (e.g. ccache) with docker-compose and Github Actions? Regards Antoine. Le 28/09/2019 à 01:36, Francois Saint-Jacques a écrit : > Hello, > > I suggest we tackle https://jira.apache.org/jira/browse/ARROW-5801. > For Rust, that would be > https://jira.apache.org/jira/browse/ARROW-5809. Once ported to > docker/docker-compose, it's trivial to activate github action for the > same test (see https://github.com/apache/arrow/pull/5530). As I'm > writing this email (4 hours after the PR was pushed), the travis > checks are still not completed due to the queue. While the github > action was completed after 22 minutes > (https://github.com/apache/arrow/pull/5530/checks?check_run_id=239290338). > > François > > > On Fri, Sep 27, 2019 at 11:22 AM Krisztián Szűcs > wrote: >> >> On Fri, Sep 27, 2019 at 5:07 PM Andy Grove wrote: >> >>> Thanks Krisztian. That's very helpful. I will create a CI page on the wiki >>> and add this info. >>> >>> Does anyone have any objections to me trying out GitHub Actions for running >>> the Rust tests on PR builds? I could try this out on my own fork first. >>> >> I think GitHub Actions is a good idea, especially for easier build setups >> like Rust >> has. Go, Node as similarly straightforward, so we could decommission the >> travis >> counterparts hopefully speeding up the rest of the builds a bit. >> >>> >>> On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs >>> >>> wrote: >>> On Fri, Sep 27, 2019 at 3:25 PM Andy Grove >>> wrote: > I've been poking around on the Arrow website and wiki and I can't find > documentation relating to CI. Do we have any documentation on how >>> things > work today or what the goals are? I don't think so. > For Rust builds it isn't immediately > obvious why they are building on both Travis CI and Ursabot. > We have another thread [1], where we discuss multiple things about Buildbot (ursabot). I've enabled the buildbot builder for rust because it provides much quicker feedback than travis does. [1]: >>> https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > >> FYI, using the kibble.dev link from INFRA-18533, it seems that this >> September we're using about 15% of the ASF's total Travis CI capacity >> (60 concurrent workers I think) >> >> https://imgur.com/a/oOrbPsj >> >> The highest is Apache Druid (incubating) at 18%, so we are #2. >>> Suffice >> to say the ASF's Travis couldn't accommodate us if we had twice as >> many pull requests >> >> On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > wrote: >>> >>> I've been sounding the alarm bells about this for a while. We need >>> to >>> work to get ourselves off of Travis CI, but it is not going to be >>> easy.t >> >>> On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > emkornfi...@gmail.com> >> wrote: My understanding is the Travis CI queue is shared among all >>> apache projects, and there are few including Arrow that make heavy use >>> of > the resources. Hence, a lot of time waiting for jobs to start. I think >> there are some open JIRAs to finish dockerization of builds, I don't >>> know > the current status of finding alternative CI sources though. On Thu, Sep 26, 2019 at 10:24 PM Andy Grove < >>> andygrov...@gmail.com > >> wrote: > I know this has been discussed in the past, and I apologize for not >> paying > attention at the time (and searching for arrow + travis in >>> email >> isn't very > effective) but why does it take so long for our Travis CI >>> builds > and >> are > there open JIRA issues related to this? > > Thanks, > > Andy. > >> > >>>
Re: Travis CI delays
Hello, I suggest we tackle https://jira.apache.org/jira/browse/ARROW-5801. For Rust, that would be https://jira.apache.org/jira/browse/ARROW-5809. Once ported to docker/docker-compose, it's trivial to activate github action for the same test (see https://github.com/apache/arrow/pull/5530). As I'm writing this email (4 hours after the PR was pushed), the travis checks are still not completed due to the queue. While the github action was completed after 22 minutes (https://github.com/apache/arrow/pull/5530/checks?check_run_id=239290338). François On Fri, Sep 27, 2019 at 11:22 AM Krisztián Szűcs wrote: > > On Fri, Sep 27, 2019 at 5:07 PM Andy Grove wrote: > > > Thanks Krisztian. That's very helpful. I will create a CI page on the wiki > > and add this info. > > > > Does anyone have any objections to me trying out GitHub Actions for running > > the Rust tests on PR builds? I could try this out on my own fork first. > > > I think GitHub Actions is a good idea, especially for easier build setups > like Rust > has. Go, Node as similarly straightforward, so we could decommission the > travis > counterparts hopefully speeding up the rest of the builds a bit. > > > > > On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs > > > > wrote: > > > > > On Fri, Sep 27, 2019 at 3:25 PM Andy Grove > > wrote: > > > > > > > I've been poking around on the Arrow website and wiki and I can't find > > > > documentation relating to CI. Do we have any documentation on how > > things > > > > work today or what the goals are? > > > > > > I don't think so. > > > > > > > For Rust builds it isn't immediately > > > > obvious why they are building on both Travis CI and Ursabot. > > > > > > > We have another thread [1], where we discuss multiple things about > > > Buildbot (ursabot). I've enabled the buildbot builder for rust because it > > > provides much quicker feedback than travis does. > > > > > > [1]: > > > > > > > > https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > > > > > > > > > > > > > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney > > > wrote: > > > > > > > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > > > > September we're using about 15% of the ASF's total Travis CI capacity > > > > > (60 concurrent workers I think) > > > > > > > > > > https://imgur.com/a/oOrbPsj > > > > > > > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. > > Suffice > > > > > to say the ASF's Travis couldn't accommodate us if we had twice as > > > > > many pull requests > > > > > > > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > > > > wrote: > > > > > > > > > > > > I've been sounding the alarm bells about this for a while. We need > > to > > > > > > work to get ourselves off of Travis CI, but it is not going to be > > > > > > easy.t > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > > > > emkornfi...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > My understanding is the Travis CI queue is shared among all > > apache > > > > > > > projects, and there are few including Arrow that make heavy use > > of > > > > the > > > > > > > resources. Hence, a lot of time waiting for jobs to start. I > > > think > > > > > there > > > > > > > are some open JIRAs to finish dockerization of builds, I don't > > know > > > > the > > > > > > > current status of finding alternative CI sources though. > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove < > > andygrov...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > I know this has been discussed in the past, and I apologize for > > > not > > > > > paying > > > > > > > > attention at the time (and searching for arrow + travis in > > email > > > > > isn't very > > > > > > > > effective) but why does it take so long for our Travis CI > > builds > > > > and > > > > > are > > > > > > > > there open JIRA issues related to this? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Andy. > > > > > > > > > > > > > > > > > > > > > >
Re: Travis CI delays
Thanks Krisztian. That's very helpful. I will create a CI page on the wiki and add this info. Does anyone have any objections to me trying out GitHub Actions for running the Rust tests on PR builds? I could try this out on my own fork first. On Fri, Sep 27, 2019 at 7:40 AM Krisztián Szűcs wrote: > On Fri, Sep 27, 2019 at 3:25 PM Andy Grove wrote: > > > I've been poking around on the Arrow website and wiki and I can't find > > documentation relating to CI. Do we have any documentation on how things > > work today or what the goals are? > > I don't think so. > > > For Rust builds it isn't immediately > > obvious why they are building on both Travis CI and Ursabot. > > > We have another thread [1], where we discuss multiple things about > Buildbot (ursabot). I've enabled the buildbot builder for rust because it > provides much quicker feedback than travis does. > > [1]: > > https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > > > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney > wrote: > > > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > > September we're using about 15% of the ASF's total Travis CI capacity > > > (60 concurrent workers I think) > > > > > > https://imgur.com/a/oOrbPsj > > > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > > > to say the ASF's Travis couldn't accommodate us if we had twice as > > > many pull requests > > > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > > wrote: > > > > > > > > I've been sounding the alarm bells about this for a while. We need to > > > > work to get ourselves off of Travis CI, but it is not going to be > > > > easy.t > > > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > > emkornfi...@gmail.com> > > > wrote: > > > > > > > > > > My understanding is the Travis CI queue is shared among all apache > > > > > projects, and there are few including Arrow that make heavy use of > > the > > > > > resources. Hence, a lot of time waiting for jobs to start. I > think > > > there > > > > > are some open JIRAs to finish dockerization of builds, I don't know > > the > > > > > current status of finding alternative CI sources though. > > > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > > > > wrote: > > > > > > > > > > > I know this has been discussed in the past, and I apologize for > not > > > paying > > > > > > attention at the time (and searching for arrow + travis in email > > > isn't very > > > > > > effective) but why does it take so long for our Travis CI builds > > and > > > are > > > > > > there open JIRA issues related to this? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andy. > > > > > > > > > > > >
Re: Travis CI delays
On Fri, Sep 27, 2019 at 3:25 PM Andy Grove wrote: > I've been poking around on the Arrow website and wiki and I can't find > documentation relating to CI. Do we have any documentation on how things > work today or what the goals are? I don't think so. > For Rust builds it isn't immediately > obvious why they are building on both Travis CI and Ursabot. > We have another thread [1], where we discuss multiple things about Buildbot (ursabot). I've enabled the buildbot builder for rust because it provides much quicker feedback than travis does. [1]: https://lists.apache.org/thread.html/02f7981176b67a12618b96b7d3b13e38b8f862e14c735dcf0ae359e0@%3Cdev.arrow.apache.org%3E > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > September we're using about 15% of the ASF's total Travis CI capacity > > (60 concurrent workers I think) > > > > https://imgur.com/a/oOrbPsj > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > > to say the ASF's Travis couldn't accommodate us if we had twice as > > many pull requests > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney > wrote: > > > > > > I've been sounding the alarm bells about this for a while. We need to > > > work to get ourselves off of Travis CI, but it is not going to be > > > easy.t > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield < > emkornfi...@gmail.com> > > wrote: > > > > > > > > My understanding is the Travis CI queue is shared among all apache > > > > projects, and there are few including Arrow that make heavy use of > the > > > > resources. Hence, a lot of time waiting for jobs to start. I think > > there > > > > are some open JIRAs to finish dockerization of builds, I don't know > the > > > > current status of finding alternative CI sources though. > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > > wrote: > > > > > > > > > I know this has been discussed in the past, and I apologize for not > > paying > > > > > attention at the time (and searching for arrow + travis in email > > isn't very > > > > > effective) but why does it take so long for our Travis CI builds > and > > are > > > > > there open JIRA issues related to this? > > > > > > > > > > Thanks, > > > > > > > > > > Andy. > > > > > > > >
Re: Travis CI delays
There are a bunch of other mailing list discussions over the last few months I recommend digging up and reviewing. As one practical issue with the Ursabot builds, those are running on servers that are hosted at my physical home in Nashville. If there's a power outage https://ci.ursalabs.org and all the CI builds will go down. If you're comfortable turning off the Rust Travis builds and relying on the Ursabot builds, and docker-compose for running the build locally, you're free to do that, but I think you need to make sure you have a contingency plan for verifying patches for the times when the network is unavailable. I travel about a third of the time and have no backup support if it goes down while I'm away from home. - Wes On Fri, Sep 27, 2019 at 8:25 AM Andy Grove wrote: > > I've been poking around on the Arrow website and wiki and I can't find > documentation relating to CI. Do we have any documentation on how things > work today or what the goals are? For Rust builds it isn't immediately > obvious why they are building on both Travis CI and Ursabot. > > > > On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > > > FYI, using the kibble.dev link from INFRA-18533, it seems that this > > September we're using about 15% of the ASF's total Travis CI capacity > > (60 concurrent workers I think) > > > > https://imgur.com/a/oOrbPsj > > > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > > to say the ASF's Travis couldn't accommodate us if we had twice as > > many pull requests > > > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney wrote: > > > > > > I've been sounding the alarm bells about this for a while. We need to > > > work to get ourselves off of Travis CI, but it is not going to be > > > easy. > > > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield > > wrote: > > > > > > > > My understanding is the Travis CI queue is shared among all apache > > > > projects, and there are few including Arrow that make heavy use of the > > > > resources. Hence, a lot of time waiting for jobs to start. I think > > there > > > > are some open JIRAs to finish dockerization of builds, I don't know the > > > > current status of finding alternative CI sources though. > > > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > > wrote: > > > > > > > > > I know this has been discussed in the past, and I apologize for not > > paying > > > > > attention at the time (and searching for arrow + travis in email > > isn't very > > > > > effective) but why does it take so long for our Travis CI builds and > > are > > > > > there open JIRA issues related to this? > > > > > > > > > > Thanks, > > > > > > > > > > Andy. > > > > > > >
Re: Travis CI delays
I've been poking around on the Arrow website and wiki and I can't find documentation relating to CI. Do we have any documentation on how things work today or what the goals are? For Rust builds it isn't immediately obvious why they are building on both Travis CI and Ursabot. On Fri, Sep 27, 2019 at 6:33 AM Wes McKinney wrote: > FYI, using the kibble.dev link from INFRA-18533, it seems that this > September we're using about 15% of the ASF's total Travis CI capacity > (60 concurrent workers I think) > > https://imgur.com/a/oOrbPsj > > The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice > to say the ASF's Travis couldn't accommodate us if we had twice as > many pull requests > > On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney wrote: > > > > I've been sounding the alarm bells about this for a while. We need to > > work to get ourselves off of Travis CI, but it is not going to be > > easy. > > > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield > wrote: > > > > > > My understanding is the Travis CI queue is shared among all apache > > > projects, and there are few including Arrow that make heavy use of the > > > resources. Hence, a lot of time waiting for jobs to start. I think > there > > > are some open JIRAs to finish dockerization of builds, I don't know the > > > current status of finding alternative CI sources though. > > > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove > wrote: > > > > > > > I know this has been discussed in the past, and I apologize for not > paying > > > > attention at the time (and searching for arrow + travis in email > isn't very > > > > effective) but why does it take so long for our Travis CI builds and > are > > > > there open JIRA issues related to this? > > > > > > > > Thanks, > > > > > > > > Andy. > > > > >
Re: Travis CI delays
I've been sounding the alarm bells about this for a while. We need to work to get ourselves off of Travis CI, but it is not going to be easy. On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield wrote: > > My understanding is the Travis CI queue is shared among all apache > projects, and there are few including Arrow that make heavy use of the > resources. Hence, a lot of time waiting for jobs to start. I think there > are some open JIRAs to finish dockerization of builds, I don't know the > current status of finding alternative CI sources though. > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove wrote: > > > I know this has been discussed in the past, and I apologize for not paying > > attention at the time (and searching for arrow + travis in email isn't very > > effective) but why does it take so long for our Travis CI builds and are > > there open JIRA issues related to this? > > > > Thanks, > > > > Andy. > >
Re: Travis CI delays
FYI, using the kibble.dev link from INFRA-18533, it seems that this September we're using about 15% of the ASF's total Travis CI capacity (60 concurrent workers I think) https://imgur.com/a/oOrbPsj The highest is Apache Druid (incubating) at 18%, so we are #2. Suffice to say the ASF's Travis couldn't accommodate us if we had twice as many pull requests On Fri, Sep 27, 2019 at 7:26 AM Wes McKinney wrote: > > I've been sounding the alarm bells about this for a while. We need to > work to get ourselves off of Travis CI, but it is not going to be > easy. > > On Thu, Sep 26, 2019 at 11:42 PM Micah Kornfield > wrote: > > > > My understanding is the Travis CI queue is shared among all apache > > projects, and there are few including Arrow that make heavy use of the > > resources. Hence, a lot of time waiting for jobs to start. I think there > > are some open JIRAs to finish dockerization of builds, I don't know the > > current status of finding alternative CI sources though. > > > > On Thu, Sep 26, 2019 at 10:24 PM Andy Grove wrote: > > > > > I know this has been discussed in the past, and I apologize for not paying > > > attention at the time (and searching for arrow + travis in email isn't > > > very > > > effective) but why does it take so long for our Travis CI builds and are > > > there open JIRA issues related to this? > > > > > > Thanks, > > > > > > Andy. > > >
Re: Travis CI delays
My understanding is the Travis CI queue is shared among all apache projects, and there are few including Arrow that make heavy use of the resources. Hence, a lot of time waiting for jobs to start. I think there are some open JIRAs to finish dockerization of builds, I don't know the current status of finding alternative CI sources though. On Thu, Sep 26, 2019 at 10:24 PM Andy Grove wrote: > I know this has been discussed in the past, and I apologize for not paying > attention at the time (and searching for arrow + travis in email isn't very > effective) but why does it take so long for our Travis CI builds and are > there open JIRA issues related to this? > > Thanks, > > Andy. >
