hi Krisz -- I just left comments on the PR. This definitely looks like a good step forward. My main comment is that I think there are too many C++/Python tasks to run on an each-commit basis. Ideally many of these would be run nightly. There is also a certain amount of redundancy in rebuilding the C++ library multiple times before running each dependent set of tests, whereas in Travis we build the C++ library once then test both C++ and Python. If there is sufficient number of builders then perhaps it doesn't matter so much
It seems there are a few things, like action filtering (similar to "detect-changes.py") based on what was changed that would need to get done before this can be merged. - Wes On Fri, Oct 25, 2019 at 7:25 PM Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > > Hi, > > During the release of 0.15.1-RC0 I literally had to wait days > to ensure that the Travis, Appveyor and Crossbow builds > were all passing for the release branch. Additionally each > newly added patch was delaying the process by 8 hrs or so > (actually felt like 16). > > Recently I've been working on to incorporate the advantages > of the Buildbot setup into our current docker-compose > configuration, including support for multiple architectures > and platforms, reusing docker images and caching dependency > installation steps. It tries to follow the semantics of ursabot, > but using only docker-compose and tiny shell scripts. > > This refactoring also includes GitHub Actions workflows for > Windows and macOS as well, reusing the same (bash) builds > scripts. The docker configuration and the scripts are CI agnostic. > Last but not least, I've managed to clean up a lot of things > including every travis builds, and three Appveyor builds. > As an example the ci [3] and dev [4] folders got much cleaner. > > The majority of the builds are passing [2], but due to the size > of the pull request [1] reviews for relevant workflows like the > JavaScript, C#, Rust, JNI, etc. would be much appreciated. > I'll be on vacation until Wednesday, but will try to respond on > both GH and the ML. > > Thanks, Krisztian > > [1]: https://github.com/apache/arrow/pull/5589 > [2]: https://github.com/apache/arrow/runs/275685241 > [3]: > https://github.com/apache/arrow/tree/9c7e7289b9c9486c13a02e7cb5682a0f9f274ec6/ci > [4]: > https://github.com/apache/arrow/tree/9c7e7289b9c9486c13a02e7cb5682a0f9f274ec6/dev
