I am going to check the nightly tag now. If the nightly tag works (quite likely) - would that still be problem Ash/ Daniel? Do you still think that tag-triggered solution is worse in this case than URL-triggered one ?
J. On Tue, Apr 21, 2020 at 4:01 PM Daniel Imberman <[email protected]> wrote: > Yeah, I'm not worried about DDOS as long as the URL is stored in a > secret/doesn't show up in the github action UI. > > > On Tue, Apr 21, 2020 at 6:29 AM Ash Berlin-Taylor <[email protected]> wrote: > > > I'm still not quite sure what problem are we solving here either...? > > What is broken with the current/already merged solution? > > > > From a philosophical view I don't like deleting tags, and this feels > > like a bit of a hack to work around limitations in other systems. > > (Welcome to being a developer I guess.) What you have proposed is better > > than having the tags build up, certainly, but I'm still not wild about > > it (And to check: we can't just re-push a single "nightly" tag as Docker > > Hub will not rebuild when a tag changes? Have we confirmed this?) > > > > I've read the discussion you linked to, but the only thing I see is this > > comment > https://github.com/apache/airflow/pull/8400#issuecomment-614796124 > > > > > But is it safe to store such URL somewhere? Is it something that is > > > sustainable long term (who will take care that it is actually still > > > working :)) .... Who will watch the watcher. ? > > > > Yes, if we store that build URL in a secure secret, for instance using > > the encryption approach suggested here > > > > > https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets#limits-for-secrets > > we can get Apache Infra to add a single secret then we can add/change > > values easily in the future. > > > > There is a lot of precedent in Infra tickets of creating a secret for > > Github Actions: > > > > > https://issues.apache.org/jira/browse/INFRA-19602?jql=text%20~%20%22github%20actions%20secrets%22 > > for example > > > > In the past I've used https://github.com/voxpupuli/hiera-eyaml even > > outside of puppet as it only encrypts the values, not the whole file, > > which makes it a bit easier to see what setting is changed, even if the > > setting is not visible in the diff. So what I'd suggest is we ask Infra > > to create an random GPG key, put the private key in a Secret in Github > > and then provide us with the public key. I'm happy to set this up if > > it's the route we want to go down. > > > > If it's a nightly Github action, so we'd see CRON failures as we did > > with Travis, no? > > > > -a > > > > > > On Apr 21 2020, at 12:17 pm, Jarek Potiuk <[email protected]> > > wrote: > > > > > On Tue, Apr 21, 2020 at 12:05 PM Ash Berlin-Taylor <[email protected]> > > wrote: > > > > > >> I've just looked in Docker settings for it's Automated builds, and it > is > > >> possible to set up a URL that we can post to that will then trigger a > > >> daily build. > > >> > > >> > > > https://hub.docker.com/repository/registry-1.docker.io/apache/airflow/builds/05570a90-f8bf-4803-b935-f93c455ab5bb > > >> was me testing it out (needs auth, most people won't be able to see > > that) > > >> > > >> Yes. I know this option. This problem (regular builds) and possibly > > > triggering them via some kind of CRON job was already discussed it in > > > detail with Daniel in > > > https://github.com/apache/airflow/pull/8400#issuecomment-614783967 - > > that > > > was PR entitle "Less frequent DockerHub Builds" which we merged already > > > (but I am not particularly happy with this approach). Please take a > look > > > there Ash - we discussed all the options we saw at this time > > > (including URL > > > triggering). > > > > > > > > >> So we can set up a travis job (say, since we can put encrypted info in > > >> there. I don't think we can put secrets in our Github Actions as we > > >> aren't admins on the repo) that would make a PSOT to this special URL > > >> once a day, causing DockerHub to build for us. > > >> > > > > > > I believe a big problem with external URL that it might be to use to > DDOS > > > our builds. And we cannot (For now) manage secrets in our Github > > > Actions. I > > > opened INFRA ticket and Gavin assigned it himself so likely there will > be > > > soon answered and maybe we will have a proposal from INFRA soon: > > > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124. If > > > we had > > > this possibility, URL triggered by CRON Github Action would be a > > > possibility. We are waiting for INFRA to help with that. And I think we > > > want to move out Travis eventually. And I do not want to add another > > "CRON" > > > service just for that - it should be available to all committers > > > to modify/fix/change and we do not want to add additional > > > service/credentials/hidden URL secret mechanism. I think we definitely > do > > > not want to keep both GA and Travis at the same time. This is quite a > bad > > > idea to keep Travis running and complicating our toolset. > > > > > > Would that get us the behaviour we need without polluting our git tags? > > >> > > > > > > I think I have a better solution :) See below. > > > > > > -ash > > >> > > >> On Apr 21 2020, at 10:59 am, Ash Berlin-Taylor <[email protected]> > wrote: > > >> > > >> > What is the goal in having daily-master-ci-2020-04-21 etc docker > image > > >> > tags? When would we want to use anything than "current latest CI > > >> > master" image? > > >> > > > > > > Agree. It does clutter the namespace. And some projects are ok with > that. > > > If we do not think it might be useful we can even implement retention > > > policy and keep only 2-3 latest tags (or even just the latest one). I > > think > > > this might be a very good solution - every night when the master CRON > > build > > > succeeds we delete previous "daily-master-ci-*" and create a new one > with > > > today's date. That will give us what we want, it will not clutter the > > > namespace and additionally, we will immediately see when the last daily > > > build succeeded. The builds in DockerHub can be triggered by regular > > > expression for the tags so this will work. > > > > > > I think in this form it should all your concerns Ash (no clutter, full > > > automation) and mine (no extra services to manage) and provides a > robust > > > solution without. > > > > > > Why do you think? Ash, any other concerns? Others? > > > > > > J. > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
