I'm still not quite sure what problem are we solving here either...? What is broken with the current/already merged solution?
>From a philosophical view I don't like deleting tags, and this feels like a bit of a hack to work around limitations in other systems. (Welcome to being a developer I guess.) What you have proposed is better than having the tags build up, certainly, but I'm still not wild about it (And to check: we can't just re-push a single "nightly" tag as Docker Hub will not rebuild when a tag changes? Have we confirmed this?) I've read the discussion you linked to, but the only thing I see is this comment https://github.com/apache/airflow/pull/8400#issuecomment-614796124 > But is it safe to store such URL somewhere? Is it something that is > sustainable long term (who will take care that it is actually still > working :)) .... Who will watch the watcher. ? Yes, if we store that build URL in a secure secret, for instance using the encryption approach suggested here https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets#limits-for-secrets we can get Apache Infra to add a single secret then we can add/change values easily in the future. There is a lot of precedent in Infra tickets of creating a secret for Github Actions: https://issues.apache.org/jira/browse/INFRA-19602?jql=text%20~%20%22github%20actions%20secrets%22 for example In the past I've used https://github.com/voxpupuli/hiera-eyaml even outside of puppet as it only encrypts the values, not the whole file, which makes it a bit easier to see what setting is changed, even if the setting is not visible in the diff. So what I'd suggest is we ask Infra to create an random GPG key, put the private key in a Secret in Github and then provide us with the public key. I'm happy to set this up if it's the route we want to go down. If it's a nightly Github action, so we'd see CRON failures as we did with Travis, no? -a On Apr 21 2020, at 12:17 pm, Jarek Potiuk <[email protected]> wrote: > On Tue, Apr 21, 2020 at 12:05 PM Ash Berlin-Taylor <[email protected]> wrote: > >> I've just looked in Docker settings for it's Automated builds, and it is >> possible to set up a URL that we can post to that will then trigger a >> daily build. >> >> https://hub.docker.com/repository/registry-1.docker.io/apache/airflow/builds/05570a90-f8bf-4803-b935-f93c455ab5bb >> was me testing it out (needs auth, most people won't be able to see that) >> >> Yes. I know this option. This problem (regular builds) and possibly > triggering them via some kind of CRON job was already discussed it in > detail with Daniel in > https://github.com/apache/airflow/pull/8400#issuecomment-614783967 - that > was PR entitle "Less frequent DockerHub Builds" which we merged already > (but I am not particularly happy with this approach). Please take a look > there Ash - we discussed all the options we saw at this time > (including URL > triggering). > > >> So we can set up a travis job (say, since we can put encrypted info in >> there. I don't think we can put secrets in our Github Actions as we >> aren't admins on the repo) that would make a PSOT to this special URL >> once a day, causing DockerHub to build for us. >> > > I believe a big problem with external URL that it might be to use to DDOS > our builds. And we cannot (For now) manage secrets in our Github > Actions. I > opened INFRA ticket and Gavin assigned it himself so likely there will be > soon answered and maybe we will have a proposal from INFRA soon: > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124. If > we had > this possibility, URL triggered by CRON Github Action would be a > possibility. We are waiting for INFRA to help with that. And I think we > want to move out Travis eventually. And I do not want to add another "CRON" > service just for that - it should be available to all committers > to modify/fix/change and we do not want to add additional > service/credentials/hidden URL secret mechanism. I think we definitely do > not want to keep both GA and Travis at the same time. This is quite a bad > idea to keep Travis running and complicating our toolset. > > Would that get us the behaviour we need without polluting our git tags? >> > > I think I have a better solution :) See below. > > -ash >> >> On Apr 21 2020, at 10:59 am, Ash Berlin-Taylor <[email protected]> wrote: >> >> > What is the goal in having daily-master-ci-2020-04-21 etc docker image >> > tags? When would we want to use anything than "current latest CI >> > master" image? >> > > Agree. It does clutter the namespace. And some projects are ok with that. > If we do not think it might be useful we can even implement retention > policy and keep only 2-3 latest tags (or even just the latest one). I think > this might be a very good solution - every night when the master CRON build > succeeds we delete previous "daily-master-ci-*" and create a new one with > today's date. That will give us what we want, it will not clutter the > namespace and additionally, we will immediately see when the last daily > build succeeded. The builds in DockerHub can be triggered by regular > expression for the tags so this will work. > > I think in this form it should all your concerns Ash (no clutter, full > automation) and mine (no extra services to manage) and provides a robust > solution without. > > Why do you think? Ash, any other concerns? Others? > > J. >
