Yes I know. Being perfectionist, I could not help myself to correct it ;). *Versions from master (development use only):*
- CI slim image *: airflow:master-python3.5-ci-slim, airflow:**master-python3.6-ci-slim, airflow:master-ci-slim*==airflow:master-python3.6-ci-slim - CI full image *: airflow:master-python3.5-ci, airflow:**master-python3.6-ci, airflow:master-ci*==airflow:master-python3.6-ci - Production optimised images: (future): *airflow:master-python3.5, airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 *Release versions (future):* - CI slim image: *airflow:1.10.4-python3.5-ci-slim, **airflow:1.10.4-**python3.6-ci-slim, airflow:latest-ci-slim*==airflow:1.10.4-python3.6-ci-slim - CI full image: *airflow:1.10.4-python3.5-ci, **airflow:1.10.4**-python3.6-ci, airflow:latest-ci*==airflow:1.10.4-python3.6-ci - Production optimised images (future): *airflow:1.10.4-python3.5, * *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6 On Tue, Jun 18, 2019 at 3:35 PM Jarek Potiuk <[email protected]> wrote: > I like the ordering too. Below is updated proposal: > > Ash: > > * FROM: - not really, we are using optimised multi-staging Dockerfile to > generate different variants of the images. The variants are independent > from each other but we use one common Dockerfile to generate all of them > (AKA one source of truth). This way they do not not contain unnecessary > layers - each image will only contain what it needs. The only common part > is base (python:3.x official images). > > * I think we still need the released CI images - it's much more > future-proof with very little overhead. It can make testing much easier and > we will be able to release and test bug-fixes as well in case we decide to > do so. There is little harm in keeping those in the repo (just few extra > layers) and I think it's just good to have such frozen images. It is much > better for caching - we are using previously build images as source of > cache for subsequent builds, so 1.10.3-python3.5-ci will be used as cache > to build 1.10.3.1 in case we decide to make such release. Also if we keep > separate 1.10.3, 1.10.4 then the first time we push branch with 1.10.4 the > whole image will be rebuild from scratch, which is pretty good idea (rather > than base it on the previously cached 1.10.x). This will all happen > automatically via the hook/build script so we will not have to do anything > to get it working this way. Plus it will be much easier if you would like > to run tests using specific release. If we keep the released versions in > repo you will not have to rebuild them, you will get them pre-build and > they will be automatically downloaded when you use Breeze. > > *I hope this is the last iteration :): * > > *Versions from master (development use only):* > > - CI slim image *: airflow:master-python3.5-ci, > airflow:**master-python3.6-ci-slim, > airflow:master-ci-slim*==airflow:master-python3.6-ci-slim > - CI full image *: airflow:master-python3.5-ci, > airflow:**master-python3.6-ci, > airflow:master-ci*==airflow:master-python3.6-ci > - Production optimised images: (future): *airflow:master-python3.5, > airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 > > *Release versions (future):* > > - CI slim image: *airflow:1.10.4-python3.5-ci-slim, > **airflow:1.10.4-**python3.6-ci-slim, > airflow:latest-ci-slim*==airflow:1.10.4-python3.6-ci-slim > - CI full image: *airflow:1.10.4-python3.5-ci, > **airflow:1.10.4**-python3.6-ci, > airflow:latest-ci*==airflow:1.10.4-python3.6-ci > - Production optimised images (future): *airflow:1.10.4-python3.5, * > *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6 > > J. > > On Tue, Jun 18, 2019 at 3:00 PM Ash Berlin-Taylor <[email protected]> wrote: > >> I like this ordering and the reasoning given for it. >> >> > On 18 Jun 2019, at 13:54, Philippe Gagnon <[email protected]> >> wrote: >> > >> > I know this is bikeshedding at this point but I think something more >> alone >> > the lines of: >> > >> > `apache/airflow:1.10.4-python3.5[-ci][-slim]` >> > >> > would be more appropriate as a standard. Here is my rationale: >> > >> > - The airflow version should definitely come first. >> > - The python version is the second-most significant information (other >> than >> > airflow itself's version) >> > - Everything else are extra "flavor" tags which we can then combine to >> > create more specialized images if the need arises. We could define that >> > extra tags should be ordered alphabetically. >> > >> > >> > On Tue, Jun 18, 2019 at 6:02 AM Jarek Potiuk <[email protected]> >> > wrote: >> > >> >> Great :). I'd prefer to use single word rather than ci-slim (then we >> can >> >> use "-" as separator when splitting image name). The "slimci" seem like >> >> most appropriate : >> >> >> >> I also think that it might make sense to build production-optimised >> images >> >> all the time. This way we will notice when they break and will be able >> to >> >> test them before they are actually released. It's usually much more >> painful >> >> to see that something does not work *just* at the moment you do >> release. >> >> The common theme which I learned - if something is painful in software >> >> releasing - simply do it more often, then you learn how to cope with >> it. So >> >> I'd leave it even in master. Looking at yesterday's discussion in slack >> >> <https://apache-airflow.slack.com/archives/CCPRP7943/p1560712788074100 >> > >> >> and >> >> the nice stats generated by Bas (attached >> >> < >> >> >> https://drive.google.com/file/d/1BVB_SqBDAqaxxTNOP2reTrgY_7C74xK0/view?usp=sharing >> >>> ) >> >> I'd propose python 3.6 to become the default version (while we are >> still >> >> supporting 3.5). >> >> >> >> It looks like we are converging to this: >> >> >> >> *Versions from master (development use only):* >> >> >> >> - CI slim image *: airflow:master-slimci-python3.5, >> >> airflow:**master-slimci-python3.6, >> >> airflow:master-slimci*==airflow:master-slimci-python3.6 >> >> - CI full image *: airflow:master-ci-python3.5, >> >> airflow:**master-ci-python3.6, >> >> airflow:master-ci*==airflow:master-ci-python3.6 >> >> - Production optimised images: (future): *airflow:master-python3.5, >> >> airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 >> >> >> >> *Release versions (future):* >> >> >> >> - CI slim image: *airflow:1.10.4-slimci-python3.5, * >> >> *airflow:1.10.4-slimci**-python3.6, airflow:latest-slimci* >> >> ==airflow:1.10.4-slimci-python3.5 >> >> - CI full image: *airflow:1.10.4-ci-python3.5, >> >> **airflow:1.10.4**-ci-python3.6, >> >> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 >> >> - Production optimised images (future): *airflow:1.10.4-python3.5, * >> >> *airflow:1.10.4**-python3.6, >> airflow:latest*==airflow:1.10.4-python3.6 >> >> >> >> If no-one objects, I will update AIP-10 and will take it into account >> when >> >> creating AIP for "production" image. >> >> >> >> J. >> >> >> >> On Tue, Jun 18, 2019 at 11:24 AM Ash Berlin-Taylor <[email protected]> >> wrote: >> >> >> >>> "slim" is common amongst docker, so that sounds good. >> >>> >> >>> I think the "primary" docker images should be production focused, and >> >>> anything else tagged (i.e. it is prod unless it says otherwise.) Since >> >>> master is not meant for end use we could also _only_ have the CI >> versions >> >>> of those images. >> >>> >> >>> >> >>> *Versions from master (development use only):* >> >>> >> >>> - CI images (big) *: airflow:master-ci-python3.5, >> >>> airflow:**master-ci-python3.6, >> >>> airflow:master-ci*==airflow:master-ci-python3.6 >> >>> >> >>> *Release versions (future):* >> >>> >> >>> - Main non-CI images (small): *airflow:1.10.4-python3.5-slim, * >> >>> *airflow:1.10.4**-python3.6-slim, >> >>> airflow:latest-slim*==airflow:1.10.4-python3.5-slim >> >>> - CI images (big): *airflow:1.10.4-ci-python3.5, >> >>> **airflow:1.10.4**-ci-python3.6, >> >>> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 >> >>> - Production optimised images: *airflow:1.10.4-python3.5, * >> >>> *airflow:1.10.4**-python3.6, airflow:latest* >> >>> ==airflow:1.10.4-python3.6 >> >>> >> >>> >> >>>> On 18 Jun 2019, at 10:16, James Coder <[email protected]> wrote: >> >>>> >> >>>> Just my 2 cents, at work we tend to use “slim” to denote things that >> >> are >> >>> pared down. >> >>>> How about “ci-slim”? >> >>>> >> >>>> James Coder >> >>>> >> >>>>> On Jun 18, 2019, at 4:06 AM, Jarek Potiuk <[email protected] >> > >> >>> wrote: >> >>>>> >> >>>>> Ok so then next iteration of proposal: The only doubt I have myself >> is >> >>> the >> >>>>> "master" vs. "master-prod". Maybe we should rather have "master" for >> >>>>> "production-ready" image and use a different name for the "small-ci" >> >>>>> image". Maybe "trimci" or "ci-lite" or "liteci" ? >> >>>>> What do you think? >> >>>>> >> >>>>> *Versions from master (development use only):* >> >>>>> >> >>>>> - Main non-CI images (small) *: airflow:master-python3.5, >> >>>>> airflow:**master-python3.6, >> >>>>> airflow:master*==airflow:master-python3.6 >> >>>>> - CI images (big) *: airflow:master-ci-python3.5, >> >>>>> airflow:**master-ci-python3.6, >> >>>>> airflow:master-ci*==airflow:master-ci-python3.6 >> >>>>> - Production optimised images: (future): >> >>> *airflow:master-prod-python3.5, >> >>>>> airflow:**master-prod-python3.6, airflow:master-prod* >> >>>>> ==airflow:master-prod-python3.6 >> >>>>> >> >>>>> *Release versions (future):* >> >>>>> >> >>>>> - Main non-CI images (small): *airflow:1.10.4-python3.5, * >> >>>>> *airflow:1.10.4**-python3.6, >> >> airflow:latest*==airflow:1.10.4-python3.5 >> >>>>> - CI images (big): *airflow:1.10.4-ci-python3.5, >> >>>>> **airflow:1.10.4**-ci-python3.6, >> >>>>> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 >> >>>>> - Production optimised images: *airflow:1.10.4-prod-python3.5, * >> >>>>> *airflow:1.10.4**-prod-python3.6, airflow:latest-prod* >> >>>>> ==airflow:1.10.4-prod-python3.6 >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Mon, Jun 17, 2019 at 10:31 PM Philippe Gagnon < >> >> [email protected] >> >>>> >> >>>>> wrote: >> >>>>> >> >>>>>> That makes sense. The reason I had doubts is because of the way >> >> docker >> >>> hub >> >>>>>> lists image tags together -- there's no real differentiation >> between >> >>>>>> pre-release and release builds. But then I suppose that if the >> >> tagging >> >>>>>> scheme is explicit enough it shouldn't be an issue. >> >>>>>> >> >>>>>> +1 on `:latest` being an official release. >> >>>>>> >> >>>>>>> On Mon, Jun 17, 2019 at 10:25 AM Ash Berlin-Taylor < >> [email protected]> >> >>> wrote: >> >>>>>>> >> >>>>>>> That page does mention "Nightly" builds which is close to what >> >>> building >> >>>>>>> master would be. The other thing that matters is what we actual >> >> call A >> >>>>>>> Release. >> >>>>>>> >> >>>>>>>> Do not include any links on the project website that might >> >> encourage >> >>>>>>> non-developers to download and use nightly builds, snapshots, >> >> release >> >>>>>>> candidates, or any other similar package >> >>>>>>> >> >>>>>>> I think we're find so long as we don't do that -- or in this case, >> >>> since >> >>>>>>> we will probably want to link to the docker hub page once we have >> >>>>>> versioned >> >>>>>>> images there if we make it clear that `:master` is not intended >> for >> >>> end >> >>>>>>> users, and by the same argument if we have anything as `:latest` >> it >> >>>>>> should >> >>>>>>> be a docker image relating to an official Release. >> >>>>>>> >> >>>>>>> Jarek: no `latest` pointing at CI images please. >> >>>>>>> >> >>>>>>> -a >> >>>>>>> >> >>>>>>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <[email protected] >> > >> >>>>>> wrote: >> >>>>>>>> >> >>>>>>>> One thing: we talked about releasing images under a "master" tag >> >>>>>>> (perhaps in another thread?), we should check if this is >> compatible >> >>> with >> >>>>>>> Apache's release policy [1]. It's not clear to me if this is >> >>> allowable or >> >>>>>>> not after a cursory reading. >> >>>>>>>> >> >>>>>>>> [1] http://www.apache.org/legal/release-policy.html#what >> >>>>>>>> >> >>>>>>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk < >> >>> [email protected] >> >>>>>>> >> >>>>>>> wrote: >> >>>>>>>> Anyone has more comments. I think prevailing opnion is: >> >>>>>>>> 1) To keep all images in one repo (apache/airflow) >> >>>>>>>> 2) I am not sure about labelling but I might try to document all >> >>> cases >> >>>>>>> in a >> >>>>>>>> "production" image proposal that I would like to start as soon as >> >> we >> >>>>>>> merge >> >>>>>>>> the current CI image (which I think is quite close to >> >> finalisation). >> >>>>>>>> >> >>>>>>>> J. >> >>>>>>>> >> >>>>>>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk < >> >>>>>> [email protected]> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> It's super easy to do :) >> >>>>>>>>> >> >>>>>>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor < >> >> [email protected]> >> >>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> I'm fine with us just publishing release images using the >> newest >> >>>>>>> python >> >>>>>>>>>> release (i.e. 3.7) as the main reason we support older python >> >>>>>>> versions is >> >>>>>>>>>> to support distros thats ship those versions.(i.e. Deb stable), >> >> but >> >>>>>> I >> >>>>>>> don't >> >>>>>>>>>> think we need to support that in docker. >> >>>>>>>>>> >> >>>>>>>>>> (But if it's easy to do since we want them for ci then sure) >> >>>>>>>>>> >> >>>>>>>>>> -ash >> >>>>>>>>>> >> >>>>>>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk < >> >>>>>> [email protected]> >> >>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Yeah Kamil - python 3.5 is the default one for now. I think we >> >>>>>>> should have >> >>>>>>>>>>> another discussion here - how many versions to support. There >> is >> >>>>>> this >> >>>>>>>>>>> ticket opened today : >> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about >> >>>>>>>>>>> supporting python 3.6 and 3.7 in tests. Anyone has a strong >> >>> opinion >> >>>>>>> on >> >>>>>>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it >> >>> increases >> >>>>>>> the >> >>>>>>>>>>> build/test time on Travis. There are a number of differences >> >>>>>> between >> >>>>>>> those >> >>>>>>>>>>> major versions (I have a blog post about it in writing ) but I >> >>>>>> think >> >>>>>>> there >> >>>>>>>>>>> is concern about eating Apache Travis time. >> >>>>>>>>>>> >> >>>>>>>>>>> Anyone against those three ? >> >>>>>>>>>>> >> >>>>>>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil Breguła < >> >>>>>>> [email protected]> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> 1) I would prefer to use one repository. >> >>>>>>>>>>>> +1 >> >>>>>>>>>>>> >> >>>>>>>>>>>> 2) The presented schema looks logical to me. I had doubts >> >> whether >> >>>>>>>>>>>> Python 3.5 was a good choice for "latest" version, but I >> >> checked >> >>>>>>> that >> >>>>>>>>>>>> travis uses only this version. >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk < >> >>>>>>> [email protected]> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Hello everyone, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We are close to finish AIP-10 (Airlfow image for CI) and >> seems >> >>>>>>> that we >> >>>>>>>>>>>>> >> >>>>>>>>>>>> will >> >>>>>>>>>>>> >> >>>>>>>>>>>>> start working soon on an official image AIP, but in the >> >> meantime >> >>>>>>> we have >> >>>>>>>>>>>>> 1.10.4 release coming and we would like to agree tagging >> >> scheme >> >>>>>>> used for >> >>>>>>>>>>>>> the current CI images. We discussed it a bit on Slack, but >> >> it's >> >>>>>>> time to >> >>>>>>>>>>>>> bring it here. I created a JIRA issue for it: >> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764 and my >> >>>>>>> proposals >> >>>>>>>>>>>>> >> >>>>>>>>>>>> after >> >>>>>>>>>>>> >> >>>>>>>>>>>>> the initial discussion are those: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> First of all we have different images that we can talk >> about : >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 1. "base" one - with bare development-ready airflow with >> >>>>>>> minimum set >> >>>>>>>>>>>>> >> >>>>>>>>>>>> of >> >>>>>>>>>>>> >> >>>>>>>>>>>>> dependencies >> >>>>>>>>>>>>> 2. "CI" with all the tools packages that are needed for CI >> >>>>>>> tests >> >>>>>>>>>>>>> 3. Soon we will likely have an "official" one which might >> be >> >>>>>>> used in >> >>>>>>>>>>>>> similar fashion as the "puckel" one. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> There are two decisions to make: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 1) How to keep those images - in one repository or whether >> we >> >>>>>>> should have >> >>>>>>>>>>>>> separate repos. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> It is easier for now to keep all of them within >> apache/airflow >> >>>>>>>>>>>>> < >> >>>>>>> >> https://cloud.docker.com/u/apache/repository/docker/apache/airflow> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> repository >> >>>>>>>>>>>> >> >>>>>>>>>>>>> it seems and use a labelling scheme to separate those (there >> >> is >> >>>>>>> nothing >> >>>>>>>>>>>>> wrong with that but it might seem a bit hacky). It's a bit >> >>>>>> easier >> >>>>>>> to >> >>>>>>>>>>>>> maintain with access and CI. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> We could also think about separate apache/airflow-ci, >> >>>>>>> apache/airflow-dev, >> >>>>>>>>>>>>> apache/airflow-prod or smth similar - that would require >> some >> >>>>>>>>>>>>> infrastructure tickets and is not very common. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 2) What labelling scheme to use(apache/airflow:label). My >> >>>>>>> proposal is >> >>>>>>>>>>>>> similar to this (if we keep everything in the airflow >> >>>>>> repository) >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> - *latest* = latest released version (python 3.5) = * >> >>>>>>>>>>>>> >> >>>>>>>>>>>> v1.10.3-python3.5* >> >>>>>>>>>>>> >> >>>>>>>>>>>>> - *master* = latest master version (python 3.5) = >> >>>>>>>>>>>>> >> >>>>>>>>>>>> *v2.0.0dev0-python3.5* >> >>>>>>>>>>>> >> >>>>>>>>>>>>> - *v1.10.3-python3.5,v1.10.3-python3.6* - released 1.10.3 >> >>>>>>> with python >> >>>>>>>>>>>>> 3.5/3.6 >> >>>>>>>>>>>>> - *latest-ci *= latest released version of CI variant >> >> (python >> >>>>>>> 3.5) >> >>>>>>>>>>>>> *v1.10.3-ci-python3.5* >> >>>>>>>>>>>>> - *master-ci* = latest master version of CI variant (python >> >>>>>>> 3.5) >> >>>>>>>>>>>>> *v2.0.0dev0-ci-python3.5* >> >>>>>>>>>>>>> - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released >> >>>>>>> 1.10.3 with >> >>>>>>>>>>>>> python 3.5/3.6 >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> My preference is to keep all the images in one repo and use >> >>>>>>> labelling >> >>>>>>>>>>>>> scheme as above, >> >>>>>>>>>>>>> but I am open to discuss this. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> J, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -- >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Jarek Potiuk >> >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software >> >>>>>> Engineer >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> >> >>>>>>>>> Jarek Potiuk >> >>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software >> Engineer >> >>>>>>>>> >> >>>>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> >> >>>>>>>> Jarek Potiuk >> >>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>>>>>> >> >>>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> >> >>>>> Jarek Potiuk >> >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>>> >> >>>>> M: +48 660 796 129 <+48660796129> >> >>>>> [image: Polidea] <https://www.polidea.com/> >> >>> >> >>> >> >> >> >> -- >> >> >> >> Jarek Potiuk >> >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> >> >> M: +48 660 796 129 <+48660796129> >> >> [image: Polidea] <https://www.polidea.com/> >> >> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
