Thinking about it a bit I'd suggest we remove "v2.0.0dev0" when master - at the point we release 2.0.0 and master becomes 2.1.0dev0 we want master to be that (and I don't think we want to have the old tags hanging around)
What's the difference between Main non-CI and prod? The "prod" flavour should be the default, so `airflow:1.10.4` is designed/suitable for prod (without the `v` prefix seems to be the common pattern in docker, for example 'postgres:9.6') And the difference bwteeen main non-ci (as you had it) and CI is that one contains [all], and the other is that + devel? > On 17 Jun 2019, at 16:20, Jarek Potiuk <[email protected]> wrote: > > OK. The next build should prepare "master" tags for all the "master" > images: Would such revised labelling make sense? > > V*ersions from master (development use only):* > > - Main non-CI images (small) *: airflow:master-v2.0.0dev0-python3.5, > airflow:**master-v2.0.0dev0-python3.6, airflow:master* > ==airflow:master-v2.0.0dev0-python3.5 > - CI images (big) *: airflow:master-v2.0.0dev0-ci-python3.5, > airflow:**master-v2.0.0dev0-ci-python3.6, > airflow:master-ci*==airflow:master-v2.0.0dev0-ci-python3.5 > - Production optimised images: (future): > *airflow:master-v2.0.0dev0-prod-python3.5, > airflow:**master-v2.0.0dev0-prod-python3.6, airflow:master-prod* > ==airflow:master-v2.0.0dev0-prod-python3.5 > > *Release versions (future):* > > - Main non-CI images (small): *airflow:v1.10.4-python3.5, * > *airflow:v1.10.4**-python3.6, airflow:latest*==*airflow:v1.10.4 (should > we have them ? I think it might be useful to have a reference image for > tests)* > - CI images (big): *airflow:v1.10.4-ci-python3.5, > **airflow:v1.10.4**-ci-python3.6, > airflow:latest-ci*==*airflow:v1.10.4-ci **(should we have them ? I think > it might be useful to have a reference image for tests)* > - Production optimised images: *airflow:v1.10.4-prod-python3.5, * > *airflow:v1.10.4**-prod-python3.6, airflow:latest-prod*== > *airflow:v1.10.4-pro* > > > *J.* > > On Mon, Jun 17, 2019 at 4:56 PM Jarek Potiuk <[email protected]> > wrote: > >> Sure. I agree "latest" might be misleading until we work out how we >> release it so I am fine with changing to master :). It's super easy - >> merely changing tags in DockerHub. >> >> On Mon, Jun 17, 2019 at 4:25 PM Ash Berlin-Taylor <[email protected]> wrote: >> >>> That page does mention "Nightly" builds which is close to what building >>> master would be. The other thing that matters is what we actual call A >>> Release. >>> >>>> Do not include any links on the project website that might encourage >>> non-developers to download and use nightly builds, snapshots, release >>> candidates, or any other similar package >>> >>> I think we're find so long as we don't do that -- or in this case, since >>> we will probably want to link to the docker hub page once we have versioned >>> images there if we make it clear that `:master` is not intended for end >>> users, and by the same argument if we have anything as `:latest` it should >>> be a docker image relating to an official Release. >>> >>> Jarek: no `latest` pointing at CI images please. >>> >>> -a >>> >>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <[email protected]> >>> wrote: >>>> >>>> One thing: we talked about releasing images under a "master" tag >>> (perhaps in another thread?), we should check if this is compatible with >>> Apache's release policy [1]. It's not clear to me if this is allowable or >>> not after a cursory reading. >>>> >>>> [1] http://www.apache.org/legal/release-policy.html#what >>>> >>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk <[email protected]> >>> wrote: >>>> Anyone has more comments. I think prevailing opnion is: >>>> 1) To keep all images in one repo (apache/airflow) >>>> 2) I am not sure about labelling but I might try to document all cases >>> in a >>>> "production" image proposal that I would like to start as soon as we >>> merge >>>> the current CI image (which I think is quite close to finalisation). >>>> >>>> J. >>>> >>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk <[email protected] >>>> >>>> wrote: >>>> >>>>> It's super easy to do :) >>>>> >>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor <[email protected]> >>> wrote: >>>>> >>>>>> I'm fine with us just publishing release images using the newest >>> python >>>>>> release (i.e. 3.7) as the main reason we support older python >>> versions is >>>>>> to support distros thats ship those versions.(i.e. Deb stable), but >>> I don't >>>>>> think we need to support that in docker. >>>>>> >>>>>> (But if it's easy to do since we want them for ci then sure) >>>>>> >>>>>> -ash >>>>>> >>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk <[email protected] >>>> >>>>>> wrote: >>>>>>> >>>>>>> Yeah Kamil - python 3.5 is the default one for now. I think we >>> should have >>>>>>> another discussion here - how many versions to support. There is >>> this >>>>>>> ticket opened today : >>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about >>>>>>> supporting python 3.6 and 3.7 in tests. Anyone has a strong opinion >>> on >>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it increases >>> the >>>>>>> build/test time on Travis. There are a number of differences >>> between those >>>>>>> major versions (I have a blog post about it in writing ) but I >>> think there >>>>>>> is concern about eating Apache Travis time. >>>>>>> >>>>>>> Anyone against those three ? >>>>>>> >>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil BreguĊa < >>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> 1) I would prefer to use one repository. >>>>>>>> +1 >>>>>>>> >>>>>>>> 2) The presented schema looks logical to me. I had doubts whether >>>>>>>> Python 3.5 was a good choice for "latest" version, but I checked >>> that >>>>>>>> travis uses only this version. >>>>>>>> >>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk < >>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hello everyone, >>>>>>>>> >>>>>>>>> We are close to finish AIP-10 (Airlfow image for CI) and seems >>> that we >>>>>>>>> >>>>>>>> will >>>>>>>> >>>>>>>>> start working soon on an official image AIP, but in the meantime >>> we have >>>>>>>>> 1.10.4 release coming and we would like to agree tagging scheme >>> used for >>>>>>>>> the current CI images. We discussed it a bit on Slack, but it's >>> time to >>>>>>>>> bring it here. I created a JIRA issue for it: >>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764 and my >>> proposals >>>>>>>>> >>>>>>>> after >>>>>>>> >>>>>>>>> the initial discussion are those: >>>>>>>>> >>>>>>>>> First of all we have different images that we can talk about : >>>>>>>>> >>>>>>>>> 1. "base" one - with bare development-ready airflow with >>> minimum set >>>>>>>>> >>>>>>>> of >>>>>>>> >>>>>>>>> dependencies >>>>>>>>> 2. "CI" with all the tools packages that are needed for CI >>> tests >>>>>>>>> 3. Soon we will likely have an "official" one which might be >>> used in >>>>>>>>> similar fashion as the "puckel" one. >>>>>>>>> >>>>>>>>> There are two decisions to make: >>>>>>>>> >>>>>>>>> 1) How to keep those images - in one repository or whether we >>> should have >>>>>>>>> separate repos. >>>>>>>>> >>>>>>>>> It is easier for now to keep all of them within apache/airflow >>>>>>>>> < >>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow> >>>>>>>>> >>>>>>>> repository >>>>>>>> >>>>>>>>> it seems and use a labelling scheme to separate those (there is >>> nothing >>>>>>>>> wrong with that but it might seem a bit hacky). It's a bit >>> easier to >>>>>>>>> maintain with access and CI. >>>>>>>>> >>>>>>>>> We could also think about separate apache/airflow-ci, >>> apache/airflow-dev, >>>>>>>>> apache/airflow-prod or smth similar - that would require some >>>>>>>>> infrastructure tickets and is not very common. >>>>>>>>> >>>>>>>>> 2) What labelling scheme to use(apache/airflow:label). My >>> proposal is >>>>>>>>> similar to this (if we keep everything in the airflow repository) >>>>>>>>> >>>>>>>>> - *latest* = latest released version (python 3.5) = * >>>>>>>>> >>>>>>>> v1.10.3-python3.5* >>>>>>>> >>>>>>>>> - *master* = latest master version (python 3.5) = >>>>>>>>> >>>>>>>> *v2.0.0dev0-python3.5* >>>>>>>> >>>>>>>>> - *v1.10.3-python3.5,v1.10.3-python3.6* - released 1.10.3 >>> with python >>>>>>>>> 3.5/3.6 >>>>>>>>> - *latest-ci *= latest released version of CI variant (python >>> 3.5) >>>>>>>>> *v1.10.3-ci-python3.5* >>>>>>>>> - *master-ci* = latest master version of CI variant (python >>> 3.5) >>>>>>>>> *v2.0.0dev0-ci-python3.5* >>>>>>>>> - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released >>> 1.10.3 with >>>>>>>>> python 3.5/3.6 >>>>>>>>> >>>>>>>>> >>>>>>>>> My preference is to keep all the images in one repo and use >>> labelling >>>>>>>>> scheme as above, >>>>>>>>> but I am open to discuss this. >>>>>>>>> >>>>>>>>> J, >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Jarek Potiuk >>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>>>> >>>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
