Hi, On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <wesmck...@gmail.com> wrote: > > hi folks, > > I have noticed that many of our GitHub Actions configurations are very > similar to each other They are indeed. > > https://www.diffchecker.com/eF4tHdzo > > Aside from the "copy-paste" issue, some work would have to be done to > generate a Crossbow configuration using GHA. Do you mean having GHA as a backend for crossbow? @Kou has just added support for that in [1] > > It seems like a solution to these issues is to create a program to > generate the GHA configurations (using some templates or other tools). > So what is in .github/workflows would not be edited by human hands in > general but rather generated by this program. That would be quite similar to what I've implemented in ursabot [2], just generating GHA flavored ymls instead of buildbot objects, so it seems doable. Of course we'll need commit hooks to force the regeneration of these configuration files. > > This program could also assist with local automation for > reproducibility purposes (for example, locally executing a cascade of > dependent docker-compose steps). Another independent improvement could be to ditch docker-compose completely. I'd say that 70% of the docker-compose.yml [3] and the relating dockerfiles are filled with duplications necessary because of the limited parametrization and reusability of docker and docker-compose. It also makes harder to use new docker features like https://docs.docker.com/buildx/working-with-buildx/
Again I'm referring ursabot where I've already implemented the ideas, the docker files [4] and the image hierarchy from the compose file [3] could be replaced by something similar like the ursabot docker utility [6]. The builder definitions [7] which are the counterparts for the scripts [8] compose command [9] and compose parameters [10]. I'm not saying that we should use the exact build definition syntax from ursabot, we can have arbitrary ymls etc. but the configuration I've referenced from ursabot approximates what we can achieve by increasing the abstraction level. Note, that the ursabot sources I've referenced are not GPL contaminated. > > Thoughts? On the other hand I see one big advantage of the current setup. By using plain GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash scripts, our CI configuration is more idiomatic. It sits closer to the developers' expectations. It's hard for me to judge the developer friendliness of either of "CI tools" (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator of those, but I suppose that the new GHA setup is the easiest to work with. > Thanks, Krisztian [1] https://github.com/apache/arrow/pull/6286 [2] https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256 [3] https://github.com/apache/arrow/blob/master/docker-compose.yml [4] https://github.com/apache/arrow/tree/master/ci/docker [6] https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py [7] https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332 [8] https://github.com/apache/arrow/tree/master/ci/scripts [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306 [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264 > - Wes
