I've just looked in Docker settings for it's Automated builds, and it is possible to set up a URL that we can post to that will then trigger a daily build.
https://hub.docker.com/repository/registry-1.docker.io/apache/airflow/builds/05570a90-f8bf-4803-b935-f93c455ab5bb was me testing it out (needs auth, most people won't be able to see that) So we can set up a travis job (say, since we can put encrypted info in there. I don't think we can put secrets in our Github Actions as we aren't admins on the repo) that would make a PSOT to this special URL once a day, causing DockerHub to build for us. Would that get us the behaviour we need without polluting our git tags? -ash On Apr 21 2020, at 10:59 am, Ash Berlin-Taylor <[email protected]> wrote: > What is the goal in having daily-master-ci-2020-04-21 etc docker image > tags? When would we want to use anything than "current latest CI > master" image? > > -a > > On Apr 21 2020, at 8:53 am, Jarek Potiuk <[email protected]> wrote: > >> Hello Everyone, >> >> TL;DR; I have a proposal regarding using the Airflow CI/Prod images >> generation for our daily builds/PR/local development. This is mostly >> internal detail on the code/image registry - the development workflow >> will remain practically the same with my proposal. >> >> Context: >> >> Co far we have used DockerHub as a temporary cache for images (CI and >> Prod) that were build during the PR/master CI runs. This caused us >> some problems as the number of builds we had exceeded Docker's >> capacity and I had to implement some workarounds (which I am not happy >> with - skipping builds on DockerHub when they are < 10 hours old. >> >> However, with the new GIthubActions approach, we can use the internal >> registry of GitHub for build image caching (I tested it and it works >> as expected in https://github.com/apache/airflow/pull/8393 ) >> >> Unfortunately, GithubRegistry requires authentication (please comment >> on that thread >> https://github.community/t5/GitHub-Actions/docker-pull-from-public-GitHub-Package-Registry-fail-with-quot/m-p/52505/highlight/false#M8536) >> so we cannot use it for local development/breeze easily (everyone will >> have to authenticate with the registry ???). So we have to keep >> DockerHub for regular snapshots of images for local development. >> >> Proposal: >> >> I think that opens us an interesting possibility once we move >> everything to GA: >> >> 1) We use Github Registry for all CI image caches - that will make >> them always "fresh" >> >> 2) Use DockerHub to keep released images (1.10.10, 2.0.0 in the >> future ...) >> >> 3) We have daily TAGs applied automatically by CRON jobs in github >> that will be daily "snapshots" of Airflow code and images and we use >> those daily snapshots for local development/Breeze. >> >> This way we can trigger the builds in Dockerhub based on TAGs rather >> than on every master merge as we have now. This will add number of >> tags we have in our repo (but they will be well organised - they will >> follow for example 'daily-master-ci-2020-04-21' for CI image and >> `daily-master-2020-04-21` and `daily-master-build-2020-04-21` >> >> Such daily tags/snapshots are usual approach for many projects. Plus >> they will give us the opportunity to track down history a bit easier. >> >> WDYT? >> >> J. >> >> -- >> Jarek Potiuk >> Polidea | Principal Software Engineer >> >> M: +48 660 796 129 >>
