I like the ordering too. Below is updated proposal: Ash:
* FROM: - not really, we are using optimised multi-staging Dockerfile to generate different variants of the images. The variants are independent from each other but we use one common Dockerfile to generate all of them (AKA one source of truth). This way they do not not contain unnecessary layers - each image will only contain what it needs. The only common part is base (python:3.x official images). * I think we still need the released CI images - it's much more future-proof with very little overhead. It can make testing much easier and we will be able to release and test bug-fixes as well in case we decide to do so. There is little harm in keeping those in the repo (just few extra layers) and I think it's just good to have such frozen images. It is much better for caching - we are using previously build images as source of cache for subsequent builds, so 1.10.3-python3.5-ci will be used as cache to build 1.10.3.1 in case we decide to make such release. Also if we keep separate 1.10.3, 1.10.4 then the first time we push branch with 1.10.4 the whole image will be rebuild from scratch, which is pretty good idea (rather than base it on the previously cached 1.10.x). This will all happen automatically via the hook/build script so we will not have to do anything to get it working this way. Plus it will be much easier if you would like to run tests using specific release. If we keep the released versions in repo you will not have to rebuild them, you will get them pre-build and they will be automatically downloaded when you use Breeze. *I hope this is the last iteration :): * *Versions from master (development use only):* - CI slim image *: airflow:master-python3.5-ci, airflow:**master-python3.6-ci-slim, airflow:master-ci-slim*==airflow:master-python3.6-ci-slim - CI full image *: airflow:master-python3.5-ci, airflow:**master-python3.6-ci, airflow:master-ci*==airflow:master-python3.6-ci - Production optimised images: (future): *airflow:master-python3.5, airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 *Release versions (future):* - CI slim image: *airflow:1.10.4-python3.5-ci-slim, **airflow:1.10.4-**python3.6-ci-slim, airflow:latest-ci-slim*==airflow:1.10.4-python3.6-ci-slim - CI full image: *airflow:1.10.4-python3.5-ci, **airflow:1.10.4**-python3.6-ci, airflow:latest-ci*==airflow:1.10.4-python3.6-ci - Production optimised images (future): *airflow:1.10.4-python3.5, * *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6 J. On Tue, Jun 18, 2019 at 3:00 PM Ash Berlin-Taylor <[email protected]> wrote: > I like this ordering and the reasoning given for it. > > > On 18 Jun 2019, at 13:54, Philippe Gagnon <[email protected]> wrote: > > > > I know this is bikeshedding at this point but I think something more > alone > > the lines of: > > > > `apache/airflow:1.10.4-python3.5[-ci][-slim]` > > > > would be more appropriate as a standard. Here is my rationale: > > > > - The airflow version should definitely come first. > > - The python version is the second-most significant information (other > than > > airflow itself's version) > > - Everything else are extra "flavor" tags which we can then combine to > > create more specialized images if the need arises. We could define that > > extra tags should be ordered alphabetically. > > > > > > On Tue, Jun 18, 2019 at 6:02 AM Jarek Potiuk <[email protected]> > > wrote: > > > >> Great :). I'd prefer to use single word rather than ci-slim (then we can > >> use "-" as separator when splitting image name). The "slimci" seem like > >> most appropriate : > >> > >> I also think that it might make sense to build production-optimised > images > >> all the time. This way we will notice when they break and will be able > to > >> test them before they are actually released. It's usually much more > painful > >> to see that something does not work *just* at the moment you do release. > >> The common theme which I learned - if something is painful in software > >> releasing - simply do it more often, then you learn how to cope with > it. So > >> I'd leave it even in master. Looking at yesterday's discussion in slack > >> <https://apache-airflow.slack.com/archives/CCPRP7943/p1560712788074100> > >> and > >> the nice stats generated by Bas (attached > >> < > >> > https://drive.google.com/file/d/1BVB_SqBDAqaxxTNOP2reTrgY_7C74xK0/view?usp=sharing > >>> ) > >> I'd propose python 3.6 to become the default version (while we are still > >> supporting 3.5). > >> > >> It looks like we are converging to this: > >> > >> *Versions from master (development use only):* > >> > >> - CI slim image *: airflow:master-slimci-python3.5, > >> airflow:**master-slimci-python3.6, > >> airflow:master-slimci*==airflow:master-slimci-python3.6 > >> - CI full image *: airflow:master-ci-python3.5, > >> airflow:**master-ci-python3.6, > >> airflow:master-ci*==airflow:master-ci-python3.6 > >> - Production optimised images: (future): *airflow:master-python3.5, > >> airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 > >> > >> *Release versions (future):* > >> > >> - CI slim image: *airflow:1.10.4-slimci-python3.5, * > >> *airflow:1.10.4-slimci**-python3.6, airflow:latest-slimci* > >> ==airflow:1.10.4-slimci-python3.5 > >> - CI full image: *airflow:1.10.4-ci-python3.5, > >> **airflow:1.10.4**-ci-python3.6, > >> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > >> - Production optimised images (future): *airflow:1.10.4-python3.5, * > >> *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6 > >> > >> If no-one objects, I will update AIP-10 and will take it into account > when > >> creating AIP for "production" image. > >> > >> J. > >> > >> On Tue, Jun 18, 2019 at 11:24 AM Ash Berlin-Taylor <[email protected]> > wrote: > >> > >>> "slim" is common amongst docker, so that sounds good. > >>> > >>> I think the "primary" docker images should be production focused, and > >>> anything else tagged (i.e. it is prod unless it says otherwise.) Since > >>> master is not meant for end use we could also _only_ have the CI > versions > >>> of those images. > >>> > >>> > >>> *Versions from master (development use only):* > >>> > >>> - CI images (big) *: airflow:master-ci-python3.5, > >>> airflow:**master-ci-python3.6, > >>> airflow:master-ci*==airflow:master-ci-python3.6 > >>> > >>> *Release versions (future):* > >>> > >>> - Main non-CI images (small): *airflow:1.10.4-python3.5-slim, * > >>> *airflow:1.10.4**-python3.6-slim, > >>> airflow:latest-slim*==airflow:1.10.4-python3.5-slim > >>> - CI images (big): *airflow:1.10.4-ci-python3.5, > >>> **airflow:1.10.4**-ci-python3.6, > >>> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > >>> - Production optimised images: *airflow:1.10.4-python3.5, * > >>> *airflow:1.10.4**-python3.6, airflow:latest* > >>> ==airflow:1.10.4-python3.6 > >>> > >>> > >>>> On 18 Jun 2019, at 10:16, James Coder <[email protected]> wrote: > >>>> > >>>> Just my 2 cents, at work we tend to use “slim” to denote things that > >> are > >>> pared down. > >>>> How about “ci-slim”? > >>>> > >>>> James Coder > >>>> > >>>>> On Jun 18, 2019, at 4:06 AM, Jarek Potiuk <[email protected]> > >>> wrote: > >>>>> > >>>>> Ok so then next iteration of proposal: The only doubt I have myself > is > >>> the > >>>>> "master" vs. "master-prod". Maybe we should rather have "master" for > >>>>> "production-ready" image and use a different name for the "small-ci" > >>>>> image". Maybe "trimci" or "ci-lite" or "liteci" ? > >>>>> What do you think? > >>>>> > >>>>> *Versions from master (development use only):* > >>>>> > >>>>> - Main non-CI images (small) *: airflow:master-python3.5, > >>>>> airflow:**master-python3.6, > >>>>> airflow:master*==airflow:master-python3.6 > >>>>> - CI images (big) *: airflow:master-ci-python3.5, > >>>>> airflow:**master-ci-python3.6, > >>>>> airflow:master-ci*==airflow:master-ci-python3.6 > >>>>> - Production optimised images: (future): > >>> *airflow:master-prod-python3.5, > >>>>> airflow:**master-prod-python3.6, airflow:master-prod* > >>>>> ==airflow:master-prod-python3.6 > >>>>> > >>>>> *Release versions (future):* > >>>>> > >>>>> - Main non-CI images (small): *airflow:1.10.4-python3.5, * > >>>>> *airflow:1.10.4**-python3.6, > >> airflow:latest*==airflow:1.10.4-python3.5 > >>>>> - CI images (big): *airflow:1.10.4-ci-python3.5, > >>>>> **airflow:1.10.4**-ci-python3.6, > >>>>> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > >>>>> - Production optimised images: *airflow:1.10.4-prod-python3.5, * > >>>>> *airflow:1.10.4**-prod-python3.6, airflow:latest-prod* > >>>>> ==airflow:1.10.4-prod-python3.6 > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Jun 17, 2019 at 10:31 PM Philippe Gagnon < > >> [email protected] > >>>> > >>>>> wrote: > >>>>> > >>>>>> That makes sense. The reason I had doubts is because of the way > >> docker > >>> hub > >>>>>> lists image tags together -- there's no real differentiation between > >>>>>> pre-release and release builds. But then I suppose that if the > >> tagging > >>>>>> scheme is explicit enough it shouldn't be an issue. > >>>>>> > >>>>>> +1 on `:latest` being an official release. > >>>>>> > >>>>>>> On Mon, Jun 17, 2019 at 10:25 AM Ash Berlin-Taylor <[email protected] > > > >>> wrote: > >>>>>>> > >>>>>>> That page does mention "Nightly" builds which is close to what > >>> building > >>>>>>> master would be. The other thing that matters is what we actual > >> call A > >>>>>>> Release. > >>>>>>> > >>>>>>>> Do not include any links on the project website that might > >> encourage > >>>>>>> non-developers to download and use nightly builds, snapshots, > >> release > >>>>>>> candidates, or any other similar package > >>>>>>> > >>>>>>> I think we're find so long as we don't do that -- or in this case, > >>> since > >>>>>>> we will probably want to link to the docker hub page once we have > >>>>>> versioned > >>>>>>> images there if we make it clear that `:master` is not intended for > >>> end > >>>>>>> users, and by the same argument if we have anything as `:latest` it > >>>>>> should > >>>>>>> be a docker image relating to an official Release. > >>>>>>> > >>>>>>> Jarek: no `latest` pointing at CI images please. > >>>>>>> > >>>>>>> -a > >>>>>>> > >>>>>>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <[email protected]> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> One thing: we talked about releasing images under a "master" tag > >>>>>>> (perhaps in another thread?), we should check if this is compatible > >>> with > >>>>>>> Apache's release policy [1]. It's not clear to me if this is > >>> allowable or > >>>>>>> not after a cursory reading. > >>>>>>>> > >>>>>>>> [1] http://www.apache.org/legal/release-policy.html#what > >>>>>>>> > >>>>>>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk < > >>> [email protected] > >>>>>>> > >>>>>>> wrote: > >>>>>>>> Anyone has more comments. I think prevailing opnion is: > >>>>>>>> 1) To keep all images in one repo (apache/airflow) > >>>>>>>> 2) I am not sure about labelling but I might try to document all > >>> cases > >>>>>>> in a > >>>>>>>> "production" image proposal that I would like to start as soon as > >> we > >>>>>>> merge > >>>>>>>> the current CI image (which I think is quite close to > >> finalisation). > >>>>>>>> > >>>>>>>> J. > >>>>>>>> > >>>>>>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk < > >>>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> It's super easy to do :) > >>>>>>>>> > >>>>>>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor < > >> [email protected]> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> I'm fine with us just publishing release images using the newest > >>>>>>> python > >>>>>>>>>> release (i.e. 3.7) as the main reason we support older python > >>>>>>> versions is > >>>>>>>>>> to support distros thats ship those versions.(i.e. Deb stable), > >> but > >>>>>> I > >>>>>>> don't > >>>>>>>>>> think we need to support that in docker. > >>>>>>>>>> > >>>>>>>>>> (But if it's easy to do since we want them for ci then sure) > >>>>>>>>>> > >>>>>>>>>> -ash > >>>>>>>>>> > >>>>>>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk < > >>>>>> [email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Yeah Kamil - python 3.5 is the default one for now. I think we > >>>>>>> should have > >>>>>>>>>>> another discussion here - how many versions to support. There > is > >>>>>> this > >>>>>>>>>>> ticket opened today : > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about > >>>>>>>>>>> supporting python 3.6 and 3.7 in tests. Anyone has a strong > >>> opinion > >>>>>>> on > >>>>>>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it > >>> increases > >>>>>>> the > >>>>>>>>>>> build/test time on Travis. There are a number of differences > >>>>>> between > >>>>>>> those > >>>>>>>>>>> major versions (I have a blog post about it in writing ) but I > >>>>>> think > >>>>>>> there > >>>>>>>>>>> is concern about eating Apache Travis time. > >>>>>>>>>>> > >>>>>>>>>>> Anyone against those three ? > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil Breguła < > >>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> 1) I would prefer to use one repository. > >>>>>>>>>>>> +1 > >>>>>>>>>>>> > >>>>>>>>>>>> 2) The presented schema looks logical to me. I had doubts > >> whether > >>>>>>>>>>>> Python 3.5 was a good choice for "latest" version, but I > >> checked > >>>>>>> that > >>>>>>>>>>>> travis uses only this version. > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk < > >>>>>>> [email protected]> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hello everyone, > >>>>>>>>>>>>> > >>>>>>>>>>>>> We are close to finish AIP-10 (Airlfow image for CI) and > seems > >>>>>>> that we > >>>>>>>>>>>>> > >>>>>>>>>>>> will > >>>>>>>>>>>> > >>>>>>>>>>>>> start working soon on an official image AIP, but in the > >> meantime > >>>>>>> we have > >>>>>>>>>>>>> 1.10.4 release coming and we would like to agree tagging > >> scheme > >>>>>>> used for > >>>>>>>>>>>>> the current CI images. We discussed it a bit on Slack, but > >> it's > >>>>>>> time to > >>>>>>>>>>>>> bring it here. I created a JIRA issue for it: > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764 and my > >>>>>>> proposals > >>>>>>>>>>>>> > >>>>>>>>>>>> after > >>>>>>>>>>>> > >>>>>>>>>>>>> the initial discussion are those: > >>>>>>>>>>>>> > >>>>>>>>>>>>> First of all we have different images that we can talk about > : > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. "base" one - with bare development-ready airflow with > >>>>>>> minimum set > >>>>>>>>>>>>> > >>>>>>>>>>>> of > >>>>>>>>>>>> > >>>>>>>>>>>>> dependencies > >>>>>>>>>>>>> 2. "CI" with all the tools packages that are needed for CI > >>>>>>> tests > >>>>>>>>>>>>> 3. Soon we will likely have an "official" one which might be > >>>>>>> used in > >>>>>>>>>>>>> similar fashion as the "puckel" one. > >>>>>>>>>>>>> > >>>>>>>>>>>>> There are two decisions to make: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) How to keep those images - in one repository or whether we > >>>>>>> should have > >>>>>>>>>>>>> separate repos. > >>>>>>>>>>>>> > >>>>>>>>>>>>> It is easier for now to keep all of them within > apache/airflow > >>>>>>>>>>>>> < > >>>>>>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow > > > >>>>>>>>>>>>> > >>>>>>>>>>>> repository > >>>>>>>>>>>> > >>>>>>>>>>>>> it seems and use a labelling scheme to separate those (there > >> is > >>>>>>> nothing > >>>>>>>>>>>>> wrong with that but it might seem a bit hacky). It's a bit > >>>>>> easier > >>>>>>> to > >>>>>>>>>>>>> maintain with access and CI. > >>>>>>>>>>>>> > >>>>>>>>>>>>> We could also think about separate apache/airflow-ci, > >>>>>>> apache/airflow-dev, > >>>>>>>>>>>>> apache/airflow-prod or smth similar - that would require some > >>>>>>>>>>>>> infrastructure tickets and is not very common. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2) What labelling scheme to use(apache/airflow:label). My > >>>>>>> proposal is > >>>>>>>>>>>>> similar to this (if we keep everything in the airflow > >>>>>> repository) > >>>>>>>>>>>>> > >>>>>>>>>>>>> - *latest* = latest released version (python 3.5) = * > >>>>>>>>>>>>> > >>>>>>>>>>>> v1.10.3-python3.5* > >>>>>>>>>>>> > >>>>>>>>>>>>> - *master* = latest master version (python 3.5) = > >>>>>>>>>>>>> > >>>>>>>>>>>> *v2.0.0dev0-python3.5* > >>>>>>>>>>>> > >>>>>>>>>>>>> - *v1.10.3-python3.5,v1.10.3-python3.6* - released 1.10.3 > >>>>>>> with python > >>>>>>>>>>>>> 3.5/3.6 > >>>>>>>>>>>>> - *latest-ci *= latest released version of CI variant > >> (python > >>>>>>> 3.5) > >>>>>>>>>>>>> *v1.10.3-ci-python3.5* > >>>>>>>>>>>>> - *master-ci* = latest master version of CI variant (python > >>>>>>> 3.5) > >>>>>>>>>>>>> *v2.0.0dev0-ci-python3.5* > >>>>>>>>>>>>> - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released > >>>>>>> 1.10.3 with > >>>>>>>>>>>>> python 3.5/3.6 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> My preference is to keep all the images in one repo and use > >>>>>>> labelling > >>>>>>>>>>>>> scheme as above, > >>>>>>>>>>>>> but I am open to discuss this. > >>>>>>>>>>>>> > >>>>>>>>>>>>> J, > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> > >>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >>>>>> Engineer > >>>>>>>>>>>>> > >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> Jarek Potiuk > >>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>>>>> > >>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> > >>>>>>>> Jarek Potiuk > >>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>>>> > >>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> Jarek Potiuk > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>> > >>>>> M: +48 660 796 129 <+48660796129> > >>>>> [image: Polidea] <https://www.polidea.com/> > >>> > >>> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > >> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
