I know this is bikeshedding at this point but I think something more alone the lines of:
`apache/airflow:1.10.4-python3.5[-ci][-slim]` would be more appropriate as a standard. Here is my rationale: - The airflow version should definitely come first. - The python version is the second-most significant information (other than airflow itself's version) - Everything else are extra "flavor" tags which we can then combine to create more specialized images if the need arises. We could define that extra tags should be ordered alphabetically. On Tue, Jun 18, 2019 at 6:02 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Great :). I'd prefer to use single word rather than ci-slim (then we can > use "-" as separator when splitting image name). The "slimci" seem like > most appropriate : > > I also think that it might make sense to build production-optimised images > all the time. This way we will notice when they break and will be able to > test them before they are actually released. It's usually much more painful > to see that something does not work *just* at the moment you do release. > The common theme which I learned - if something is painful in software > releasing - simply do it more often, then you learn how to cope with it. So > I'd leave it even in master. Looking at yesterday's discussion in slack > <https://apache-airflow.slack.com/archives/CCPRP7943/p1560712788074100> > and > the nice stats generated by Bas (attached > < > https://drive.google.com/file/d/1BVB_SqBDAqaxxTNOP2reTrgY_7C74xK0/view?usp=sharing > >) > I'd propose python 3.6 to become the default version (while we are still > supporting 3.5). > > It looks like we are converging to this: > > *Versions from master (development use only):* > > - CI slim image *: airflow:master-slimci-python3.5, > airflow:**master-slimci-python3.6, > airflow:master-slimci*==airflow:master-slimci-python3.6 > - CI full image *: airflow:master-ci-python3.5, > airflow:**master-ci-python3.6, > airflow:master-ci*==airflow:master-ci-python3.6 > - Production optimised images: (future): *airflow:master-python3.5, > airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 > > *Release versions (future):* > > - CI slim image: *airflow:1.10.4-slimci-python3.5, * > *airflow:1.10.4-slimci**-python3.6, airflow:latest-slimci* > ==airflow:1.10.4-slimci-python3.5 > - CI full image: *airflow:1.10.4-ci-python3.5, > **airflow:1.10.4**-ci-python3.6, > airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > - Production optimised images (future): *airflow:1.10.4-python3.5, * > *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6 > > If no-one objects, I will update AIP-10 and will take it into account when > creating AIP for "production" image. > > J. > > On Tue, Jun 18, 2019 at 11:24 AM Ash Berlin-Taylor <a...@apache.org> wrote: > > > "slim" is common amongst docker, so that sounds good. > > > > I think the "primary" docker images should be production focused, and > > anything else tagged (i.e. it is prod unless it says otherwise.) Since > > master is not meant for end use we could also _only_ have the CI versions > > of those images. > > > > > > *Versions from master (development use only):* > > > > - CI images (big) *: airflow:master-ci-python3.5, > > airflow:**master-ci-python3.6, > > airflow:master-ci*==airflow:master-ci-python3.6 > > > > *Release versions (future):* > > > > - Main non-CI images (small): *airflow:1.10.4-python3.5-slim, * > > *airflow:1.10.4**-python3.6-slim, > > airflow:latest-slim*==airflow:1.10.4-python3.5-slim > > - CI images (big): *airflow:1.10.4-ci-python3.5, > > **airflow:1.10.4**-ci-python3.6, > > airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > > - Production optimised images: *airflow:1.10.4-python3.5, * > > *airflow:1.10.4**-python3.6, airflow:latest* > > ==airflow:1.10.4-python3.6 > > > > > > > On 18 Jun 2019, at 10:16, James Coder <jcode...@gmail.com> wrote: > > > > > > Just my 2 cents, at work we tend to use “slim” to denote things that > are > > pared down. > > > How about “ci-slim”? > > > > > > James Coder > > > > > >> On Jun 18, 2019, at 4:06 AM, Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > >> > > >> Ok so then next iteration of proposal: The only doubt I have myself is > > the > > >> "master" vs. "master-prod". Maybe we should rather have "master" for > > >> "production-ready" image and use a different name for the "small-ci" > > >> image". Maybe "trimci" or "ci-lite" or "liteci" ? > > >> What do you think? > > >> > > >> *Versions from master (development use only):* > > >> > > >> - Main non-CI images (small) *: airflow:master-python3.5, > > >> airflow:**master-python3.6, > > >> airflow:master*==airflow:master-python3.6 > > >> - CI images (big) *: airflow:master-ci-python3.5, > > >> airflow:**master-ci-python3.6, > > >> airflow:master-ci*==airflow:master-ci-python3.6 > > >> - Production optimised images: (future): > > *airflow:master-prod-python3.5, > > >> airflow:**master-prod-python3.6, airflow:master-prod* > > >> ==airflow:master-prod-python3.6 > > >> > > >> *Release versions (future):* > > >> > > >> - Main non-CI images (small): *airflow:1.10.4-python3.5, * > > >> *airflow:1.10.4**-python3.6, > airflow:latest*==airflow:1.10.4-python3.5 > > >> - CI images (big): *airflow:1.10.4-ci-python3.5, > > >> **airflow:1.10.4**-ci-python3.6, > > >> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > > >> - Production optimised images: *airflow:1.10.4-prod-python3.5, * > > >> *airflow:1.10.4**-prod-python3.6, airflow:latest-prod* > > >> ==airflow:1.10.4-prod-python3.6 > > >> > > >> > > >> > > >> On Mon, Jun 17, 2019 at 10:31 PM Philippe Gagnon < > philgagn...@gmail.com > > > > > >> wrote: > > >> > > >>> That makes sense. The reason I had doubts is because of the way > docker > > hub > > >>> lists image tags together -- there's no real differentiation between > > >>> pre-release and release builds. But then I suppose that if the > tagging > > >>> scheme is explicit enough it shouldn't be an issue. > > >>> > > >>> +1 on `:latest` being an official release. > > >>> > > >>>> On Mon, Jun 17, 2019 at 10:25 AM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > >>>> > > >>>> That page does mention "Nightly" builds which is close to what > > building > > >>>> master would be. The other thing that matters is what we actual > call A > > >>>> Release. > > >>>> > > >>>>> Do not include any links on the project website that might > encourage > > >>>> non-developers to download and use nightly builds, snapshots, > release > > >>>> candidates, or any other similar package > > >>>> > > >>>> I think we're find so long as we don't do that -- or in this case, > > since > > >>>> we will probably want to link to the docker hub page once we have > > >>> versioned > > >>>> images there if we make it clear that `:master` is not intended for > > end > > >>>> users, and by the same argument if we have anything as `:latest` it > > >>> should > > >>>> be a docker image relating to an official Release. > > >>>> > > >>>> Jarek: no `latest` pointing at CI images please. > > >>>> > > >>>> -a > > >>>> > > >>>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <philgagn...@gmail.com> > > >>> wrote: > > >>>>> > > >>>>> One thing: we talked about releasing images under a "master" tag > > >>>> (perhaps in another thread?), we should check if this is compatible > > with > > >>>> Apache's release policy [1]. It's not clear to me if this is > > allowable or > > >>>> not after a cursory reading. > > >>>>> > > >>>>> [1] http://www.apache.org/legal/release-policy.html#what > > >>>>> > > >>>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk < > > jarek.pot...@polidea.com > > >>>> > > >>>> wrote: > > >>>>> Anyone has more comments. I think prevailing opnion is: > > >>>>> 1) To keep all images in one repo (apache/airflow) > > >>>>> 2) I am not sure about labelling but I might try to document all > > cases > > >>>> in a > > >>>>> "production" image proposal that I would like to start as soon as > we > > >>>> merge > > >>>>> the current CI image (which I think is quite close to > finalisation). > > >>>>> > > >>>>> J. > > >>>>> > > >>>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk < > > >>> jarek.pot...@polidea.com> > > >>>>> wrote: > > >>>>> > > >>>>>> It's super easy to do :) > > >>>>>> > > >>>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor < > a...@apache.org> > > >>>> wrote: > > >>>>>> > > >>>>>>> I'm fine with us just publishing release images using the newest > > >>>> python > > >>>>>>> release (i.e. 3.7) as the main reason we support older python > > >>>> versions is > > >>>>>>> to support distros thats ship those versions.(i.e. Deb stable), > but > > >>> I > > >>>> don't > > >>>>>>> think we need to support that in docker. > > >>>>>>> > > >>>>>>> (But if it's easy to do since we want them for ci then sure) > > >>>>>>> > > >>>>>>> -ash > > >>>>>>> > > >>>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk < > > >>> jarek.pot...@polidea.com> > > >>>>>>> wrote: > > >>>>>>>> > > >>>>>>>> Yeah Kamil - python 3.5 is the default one for now. I think we > > >>>> should have > > >>>>>>>> another discussion here - how many versions to support. There is > > >>> this > > >>>>>>>> ticket opened today : > > >>>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about > > >>>>>>>> supporting python 3.6 and 3.7 in tests. Anyone has a strong > > opinion > > >>>> on > > >>>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it > > increases > > >>>> the > > >>>>>>>> build/test time on Travis. There are a number of differences > > >>> between > > >>>> those > > >>>>>>>> major versions (I have a blog post about it in writing ) but I > > >>> think > > >>>> there > > >>>>>>>> is concern about eating Apache Travis time. > > >>>>>>>> > > >>>>>>>> Anyone against those three ? > > >>>>>>>> > > >>>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil Breguła < > > >>>> kamil.breg...@polidea.com> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>> 1) I would prefer to use one repository. > > >>>>>>>>> +1 > > >>>>>>>>> > > >>>>>>>>> 2) The presented schema looks logical to me. I had doubts > whether > > >>>>>>>>> Python 3.5 was a good choice for "latest" version, but I > checked > > >>>> that > > >>>>>>>>> travis uses only this version. > > >>>>>>>>> > > >>>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk < > > >>>> jarek.pot...@polidea.com> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Hello everyone, > > >>>>>>>>>> > > >>>>>>>>>> We are close to finish AIP-10 (Airlfow image for CI) and seems > > >>>> that we > > >>>>>>>>>> > > >>>>>>>>> will > > >>>>>>>>> > > >>>>>>>>>> start working soon on an official image AIP, but in the > meantime > > >>>> we have > > >>>>>>>>>> 1.10.4 release coming and we would like to agree tagging > scheme > > >>>> used for > > >>>>>>>>>> the current CI images. We discussed it a bit on Slack, but > it's > > >>>> time to > > >>>>>>>>>> bring it here. I created a JIRA issue for it: > > >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764 and my > > >>>> proposals > > >>>>>>>>>> > > >>>>>>>>> after > > >>>>>>>>> > > >>>>>>>>>> the initial discussion are those: > > >>>>>>>>>> > > >>>>>>>>>> First of all we have different images that we can talk about : > > >>>>>>>>>> > > >>>>>>>>>> 1. "base" one - with bare development-ready airflow with > > >>>> minimum set > > >>>>>>>>>> > > >>>>>>>>> of > > >>>>>>>>> > > >>>>>>>>>> dependencies > > >>>>>>>>>> 2. "CI" with all the tools packages that are needed for CI > > >>>> tests > > >>>>>>>>>> 3. Soon we will likely have an "official" one which might be > > >>>> used in > > >>>>>>>>>> similar fashion as the "puckel" one. > > >>>>>>>>>> > > >>>>>>>>>> There are two decisions to make: > > >>>>>>>>>> > > >>>>>>>>>> 1) How to keep those images - in one repository or whether we > > >>>> should have > > >>>>>>>>>> separate repos. > > >>>>>>>>>> > > >>>>>>>>>> It is easier for now to keep all of them within apache/airflow > > >>>>>>>>>> < > > >>>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow> > > >>>>>>>>>> > > >>>>>>>>> repository > > >>>>>>>>> > > >>>>>>>>>> it seems and use a labelling scheme to separate those (there > is > > >>>> nothing > > >>>>>>>>>> wrong with that but it might seem a bit hacky). It's a bit > > >>> easier > > >>>> to > > >>>>>>>>>> maintain with access and CI. > > >>>>>>>>>> > > >>>>>>>>>> We could also think about separate apache/airflow-ci, > > >>>> apache/airflow-dev, > > >>>>>>>>>> apache/airflow-prod or smth similar - that would require some > > >>>>>>>>>> infrastructure tickets and is not very common. > > >>>>>>>>>> > > >>>>>>>>>> 2) What labelling scheme to use(apache/airflow:label). My > > >>>> proposal is > > >>>>>>>>>> similar to this (if we keep everything in the airflow > > >>> repository) > > >>>>>>>>>> > > >>>>>>>>>> - *latest* = latest released version (python 3.5) = * > > >>>>>>>>>> > > >>>>>>>>> v1.10.3-python3.5* > > >>>>>>>>> > > >>>>>>>>>> - *master* = latest master version (python 3.5) = > > >>>>>>>>>> > > >>>>>>>>> *v2.0.0dev0-python3.5* > > >>>>>>>>> > > >>>>>>>>>> - *v1.10.3-python3.5,v1.10.3-python3.6* - released 1.10.3 > > >>>> with python > > >>>>>>>>>> 3.5/3.6 > > >>>>>>>>>> - *latest-ci *= latest released version of CI variant > (python > > >>>> 3.5) > > >>>>>>>>>> *v1.10.3-ci-python3.5* > > >>>>>>>>>> - *master-ci* = latest master version of CI variant (python > > >>>> 3.5) > > >>>>>>>>>> *v2.0.0dev0-ci-python3.5* > > >>>>>>>>>> - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released > > >>>> 1.10.3 with > > >>>>>>>>>> python 3.5/3.6 > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> My preference is to keep all the images in one repo and use > > >>>> labelling > > >>>>>>>>>> scheme as above, > > >>>>>>>>>> but I am open to discuss this. > > >>>>>>>>>> > > >>>>>>>>>> J, > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> -- > > >>>>>>>>>> > > >>>>>>>>>> Jarek Potiuk > > >>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > > >>> Engineer > > >>>>>>>>>> > > >>>>>>>>>> M: +48 660 796 129 <+48660796129> > > >>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> > > >>>>>> Jarek Potiuk > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>>>> > > >>>>>> M: +48 660 796 129 <+48660796129> > > >>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Jarek Potiuk > > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>>> > > >>>>> M: +48 660 796 129 <+48660796129> > > >>>>> [image: Polidea] <https://www.polidea.com/> > > >>>> > > >>>> > > >>> > > >> > > >> > > >> -- > > >> > > >> Jarek Potiuk > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > >> M: +48 660 796 129 <+48660796129> > > >> [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
