What if we wrote our own action(s) to wrap up some of the boilerplate? It doesn't seem that there are any off-the-shelf actions we could use to drive docker-compose: https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose but I don't think it would be that difficult to wrap `docker-compose pull $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in an action.
Neal On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <[email protected]> wrote: > Hi, > > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <[email protected]> wrote: > > > > hi folks, > > > > I have noticed that many of our GitHub Actions configurations are very > > similar to each other > They are indeed. > > > > https://www.diffchecker.com/eF4tHdzo > > > > Aside from the "copy-paste" issue, some work would have to be done to > > generate a Crossbow configuration using GHA. > Do you mean having GHA as a backend for crossbow? @Kou has just > added support for that in [1] > > > > It seems like a solution to these issues is to create a program to > > generate the GHA configurations (using some templates or other tools). > > So what is in .github/workflows would not be edited by human hands in > > general but rather generated by this program. > That would be quite similar to what I've implemented in ursabot [2], just > generating GHA flavored ymls instead of buildbot objects, so it seems > doable. Of course we'll need commit hooks to force the regeneration of > these configuration files. > > > > This program could also assist with local automation for > > reproducibility purposes (for example, locally executing a cascade of > > dependent docker-compose steps). > Another independent improvement could be to ditch docker-compose > completely. > I'd say that 70% of the docker-compose.yml [3] and the relating > dockerfiles are > filled with duplications necessary because of the limited parametrization > and > reusability of docker and docker-compose. It also makes harder to use new > docker > features like https://docs.docker.com/buildx/working-with-buildx/ > > Again I'm referring ursabot where I've already implemented the ideas, the > docker files [4] and the image hierarchy from the compose file [3] could be > replaced by something similar like the ursabot docker utility [6]. > The builder definitions [7] which are the counterparts for the scripts > [8] compose > command [9] and compose parameters [10]. > I'm not saying that we should use the exact build definition syntax > from ursabot, > we can have arbitrary ymls etc. but the configuration I've referenced from > ursabot approximates what we can achieve by increasing the abstraction > level. > > Note, that the ursabot sources I've referenced are not GPL contaminated. > > > > Thoughts? > On the other hand I see one big advantage of the current setup. By using > plain > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash > scripts, our CI configuration is more idiomatic. It sits closer to the > developers' > expectations. > It's hard for me to judge the developer friendliness of either of "CI > tools" > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator > of those, but I suppose that the new GHA setup is the easiest to work with. > > > Thanks, Krisztian > > [1] https://github.com/apache/arrow/pull/6286 > [2] > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256 > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml > [4] https://github.com/apache/arrow/tree/master/ci/docker > [6] > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py > [7] > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332 > [8] https://github.com/apache/arrow/tree/master/ci/scripts > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306 > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264 > > - Wes >
