When you say tags, are you meaning `git tag` here, so we'd eventually see 100s of such daily tags in git?
If so -1 veto from me -- that would make many parts of Github UI and git almost unusable and feels very counter to git+github. (Also I've never seen any other project have daily tags like that?!) I don't know about everyone else, but I've been running with a version from two-ish weeks ago and I just say "no, don't rebuild" every time breeze asks. Nothing has broken as a result. I don't see why we need daily tags in git? Can't the daily cron build publish to dockerhub once a day exactly as it is right now? -a On Apr 21 2020, at 8:53 am, Jarek Potiuk <[email protected]> wrote: > Hello Everyone, > > TL;DR; I have a proposal regarding using the Airflow CI/Prod images > generation for our daily builds/PR/local development. This is mostly > internal detail on the code/image registry - the development workflow > will remain practically the same with my proposal. > > Context: > > Co far we have used DockerHub as a temporary cache for images (CI and > Prod) that were build during the PR/master CI runs. This caused us > some problems as the number of builds we had exceeded Docker's > capacity and I had to implement some workarounds (which I am not happy > with - skipping builds on DockerHub when they are < 10 hours old. > > However, with the new GIthubActions approach, we can use the internal > registry of GitHub for build image caching (I tested it and it works > as expected in https://github.com/apache/airflow/pull/8393 ) > > Unfortunately, GithubRegistry requires authentication (please comment > on that thread > https://github.community/t5/GitHub-Actions/docker-pull-from-public-GitHub-Package-Registry-fail-with-quot/m-p/52505/highlight/false#M8536) > so we cannot use it for local development/breeze easily (everyone will > have to authenticate with the registry ???). So we have to keep > DockerHub for regular snapshots of images for local development. > > Proposal: > > I think that opens us an interesting possibility once we move > everything to GA: > > 1) We use Github Registry for all CI image caches - that will make > them always "fresh" > > 2) Use DockerHub to keep released images (1.10.10, 2.0.0 in the future ...) > > 3) We have daily TAGs applied automatically by CRON jobs in github > that will be daily "snapshots" of Airflow code and images and we use > those daily snapshots for local development/Breeze. > > This way we can trigger the builds in Dockerhub based on TAGs rather > than on every master merge as we have now. This will add number of > tags we have in our repo (but they will be well organised - they will > follow for example 'daily-master-ci-2020-04-21' for CI image and > `daily-master-2020-04-21` and `daily-master-build-2020-04-21` > > Such daily tags/snapshots are usual approach for many projects. Plus > they will give us the opportunity to track down history a bit easier. > > WDYT? > > J. > > -- > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 >
