Only objection is around airflow:1.10.4-slimci-python3.5 - the way our release process works that should probably be airflow:1.10.x-slimci-python3.5 or similar - to represent the release branch, but we don't need a CI version for the tagged release, just the release branch. I think.
Would the various CI images would be FROM the production images or not? -a > On 18 Jun 2019, at 11:02, Jarek Potiuk <[email protected]> wrote: > > Great :). I'd prefer to use single word rather than ci-slim (then we can > use "-" as separator when splitting image name). The "slimci" seem like > most appropriate : > > I also think that it might make sense to build production-optimised images > all the time. This way we will notice when they break and will be able to > test them before they are actually released. It's usually much more painful > to see that something does not work *just* at the moment you do release. > The common theme which I learned - if something is painful in software > releasing - simply do it more often, then you learn how to cope with it. So > I'd leave it even in master. Looking at yesterday's discussion in slack > <https://apache-airflow.slack.com/archives/CCPRP7943/p1560712788074100> and > the nice stats generated by Bas (attached > <https://drive.google.com/file/d/1BVB_SqBDAqaxxTNOP2reTrgY_7C74xK0/view?usp=sharing>) > I'd propose python 3.6 to become the default version (while we are still > supporting 3.5). > > It looks like we are converging to this: > > *Versions from master (development use only):* > > - CI slim image *: airflow:master-slimci-python3.5, > airflow:**master-slimci-python3.6, > airflow:master-slimci*==airflow:master-slimci-python3.6 > - CI full image *: airflow:master-ci-python3.5, > airflow:**master-ci-python3.6, > airflow:master-ci*==airflow:master-ci-python3.6 > - Production optimised images: (future): *airflow:master-python3.5, > airflow:**master-python3.6, airflow:master*==airflow:master-python3.6 > > *Release versions (future):* > > - CI slim image: *airflow:1.10.4-slimci-python3.5, * > *airflow:1.10.4-slimci**-python3.6, airflow:latest-slimci* > ==airflow:1.10.4-slimci-python3.5 > - CI full image: *airflow:1.10.4-ci-python3.5, > **airflow:1.10.4**-ci-python3.6, > airflow:latest-ci*==airflow:1.10.4-ci-python3.6 > - Production optimised images (future): *airflow:1.10.4-python3.5, * > *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6 > > If no-one objects, I will update AIP-10 and will take it into account when > creating AIP for "production" image. > > J. > > On Tue, Jun 18, 2019 at 11:24 AM Ash Berlin-Taylor <[email protected]> wrote: > >> "slim" is common amongst docker, so that sounds good. >> >> I think the "primary" docker images should be production focused, and >> anything else tagged (i.e. it is prod unless it says otherwise.) Since >> master is not meant for end use we could also _only_ have the CI versions >> of those images. >> >> >> *Versions from master (development use only):* >> >> - CI images (big) *: airflow:master-ci-python3.5, >> airflow:**master-ci-python3.6, >> airflow:master-ci*==airflow:master-ci-python3.6 >> >> *Release versions (future):* >> >> - Main non-CI images (small): *airflow:1.10.4-python3.5-slim, * >> *airflow:1.10.4**-python3.6-slim, >> airflow:latest-slim*==airflow:1.10.4-python3.5-slim >> - CI images (big): *airflow:1.10.4-ci-python3.5, >> **airflow:1.10.4**-ci-python3.6, >> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 >> - Production optimised images: *airflow:1.10.4-python3.5, * >> *airflow:1.10.4**-python3.6, airflow:latest* >> ==airflow:1.10.4-python3.6 >> >> >>> On 18 Jun 2019, at 10:16, James Coder <[email protected]> wrote: >>> >>> Just my 2 cents, at work we tend to use “slim” to denote things that are >> pared down. >>> How about “ci-slim”? >>> >>> James Coder >>> >>>> On Jun 18, 2019, at 4:06 AM, Jarek Potiuk <[email protected]> >> wrote: >>>> >>>> Ok so then next iteration of proposal: The only doubt I have myself is >> the >>>> "master" vs. "master-prod". Maybe we should rather have "master" for >>>> "production-ready" image and use a different name for the "small-ci" >>>> image". Maybe "trimci" or "ci-lite" or "liteci" ? >>>> What do you think? >>>> >>>> *Versions from master (development use only):* >>>> >>>> - Main non-CI images (small) *: airflow:master-python3.5, >>>> airflow:**master-python3.6, >>>> airflow:master*==airflow:master-python3.6 >>>> - CI images (big) *: airflow:master-ci-python3.5, >>>> airflow:**master-ci-python3.6, >>>> airflow:master-ci*==airflow:master-ci-python3.6 >>>> - Production optimised images: (future): >> *airflow:master-prod-python3.5, >>>> airflow:**master-prod-python3.6, airflow:master-prod* >>>> ==airflow:master-prod-python3.6 >>>> >>>> *Release versions (future):* >>>> >>>> - Main non-CI images (small): *airflow:1.10.4-python3.5, * >>>> *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.5 >>>> - CI images (big): *airflow:1.10.4-ci-python3.5, >>>> **airflow:1.10.4**-ci-python3.6, >>>> airflow:latest-ci*==airflow:1.10.4-ci-python3.6 >>>> - Production optimised images: *airflow:1.10.4-prod-python3.5, * >>>> *airflow:1.10.4**-prod-python3.6, airflow:latest-prod* >>>> ==airflow:1.10.4-prod-python3.6 >>>> >>>> >>>> >>>> On Mon, Jun 17, 2019 at 10:31 PM Philippe Gagnon <[email protected] >>> >>>> wrote: >>>> >>>>> That makes sense. The reason I had doubts is because of the way docker >> hub >>>>> lists image tags together -- there's no real differentiation between >>>>> pre-release and release builds. But then I suppose that if the tagging >>>>> scheme is explicit enough it shouldn't be an issue. >>>>> >>>>> +1 on `:latest` being an official release. >>>>> >>>>>> On Mon, Jun 17, 2019 at 10:25 AM Ash Berlin-Taylor <[email protected]> >> wrote: >>>>>> >>>>>> That page does mention "Nightly" builds which is close to what >> building >>>>>> master would be. The other thing that matters is what we actual call A >>>>>> Release. >>>>>> >>>>>>> Do not include any links on the project website that might encourage >>>>>> non-developers to download and use nightly builds, snapshots, release >>>>>> candidates, or any other similar package >>>>>> >>>>>> I think we're find so long as we don't do that -- or in this case, >> since >>>>>> we will probably want to link to the docker hub page once we have >>>>> versioned >>>>>> images there if we make it clear that `:master` is not intended for >> end >>>>>> users, and by the same argument if we have anything as `:latest` it >>>>> should >>>>>> be a docker image relating to an official Release. >>>>>> >>>>>> Jarek: no `latest` pointing at CI images please. >>>>>> >>>>>> -a >>>>>> >>>>>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <[email protected]> >>>>> wrote: >>>>>>> >>>>>>> One thing: we talked about releasing images under a "master" tag >>>>>> (perhaps in another thread?), we should check if this is compatible >> with >>>>>> Apache's release policy [1]. It's not clear to me if this is >> allowable or >>>>>> not after a cursory reading. >>>>>>> >>>>>>> [1] http://www.apache.org/legal/release-policy.html#what >>>>>>> >>>>>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk < >> [email protected] >>>>>> >>>>>> wrote: >>>>>>> Anyone has more comments. I think prevailing opnion is: >>>>>>> 1) To keep all images in one repo (apache/airflow) >>>>>>> 2) I am not sure about labelling but I might try to document all >> cases >>>>>> in a >>>>>>> "production" image proposal that I would like to start as soon as we >>>>>> merge >>>>>>> the current CI image (which I think is quite close to finalisation). >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk < >>>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> It's super easy to do :) >>>>>>>> >>>>>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor <[email protected]> >>>>>> wrote: >>>>>>>> >>>>>>>>> I'm fine with us just publishing release images using the newest >>>>>> python >>>>>>>>> release (i.e. 3.7) as the main reason we support older python >>>>>> versions is >>>>>>>>> to support distros thats ship those versions.(i.e. Deb stable), but >>>>> I >>>>>> don't >>>>>>>>> think we need to support that in docker. >>>>>>>>> >>>>>>>>> (But if it's easy to do since we want them for ci then sure) >>>>>>>>> >>>>>>>>> -ash >>>>>>>>> >>>>>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk < >>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Yeah Kamil - python 3.5 is the default one for now. I think we >>>>>> should have >>>>>>>>>> another discussion here - how many versions to support. There is >>>>> this >>>>>>>>>> ticket opened today : >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about >>>>>>>>>> supporting python 3.6 and 3.7 in tests. Anyone has a strong >> opinion >>>>>> on >>>>>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it >> increases >>>>>> the >>>>>>>>>> build/test time on Travis. There are a number of differences >>>>> between >>>>>> those >>>>>>>>>> major versions (I have a blog post about it in writing ) but I >>>>> think >>>>>> there >>>>>>>>>> is concern about eating Apache Travis time. >>>>>>>>>> >>>>>>>>>> Anyone against those three ? >>>>>>>>>> >>>>>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil Breguła < >>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> 1) I would prefer to use one repository. >>>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> 2) The presented schema looks logical to me. I had doubts whether >>>>>>>>>>> Python 3.5 was a good choice for "latest" version, but I checked >>>>>> that >>>>>>>>>>> travis uses only this version. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk < >>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hello everyone, >>>>>>>>>>>> >>>>>>>>>>>> We are close to finish AIP-10 (Airlfow image for CI) and seems >>>>>> that we >>>>>>>>>>>> >>>>>>>>>>> will >>>>>>>>>>> >>>>>>>>>>>> start working soon on an official image AIP, but in the meantime >>>>>> we have >>>>>>>>>>>> 1.10.4 release coming and we would like to agree tagging scheme >>>>>> used for >>>>>>>>>>>> the current CI images. We discussed it a bit on Slack, but it's >>>>>> time to >>>>>>>>>>>> bring it here. I created a JIRA issue for it: >>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764 and my >>>>>> proposals >>>>>>>>>>>> >>>>>>>>>>> after >>>>>>>>>>> >>>>>>>>>>>> the initial discussion are those: >>>>>>>>>>>> >>>>>>>>>>>> First of all we have different images that we can talk about : >>>>>>>>>>>> >>>>>>>>>>>> 1. "base" one - with bare development-ready airflow with >>>>>> minimum set >>>>>>>>>>>> >>>>>>>>>>> of >>>>>>>>>>> >>>>>>>>>>>> dependencies >>>>>>>>>>>> 2. "CI" with all the tools packages that are needed for CI >>>>>> tests >>>>>>>>>>>> 3. Soon we will likely have an "official" one which might be >>>>>> used in >>>>>>>>>>>> similar fashion as the "puckel" one. >>>>>>>>>>>> >>>>>>>>>>>> There are two decisions to make: >>>>>>>>>>>> >>>>>>>>>>>> 1) How to keep those images - in one repository or whether we >>>>>> should have >>>>>>>>>>>> separate repos. >>>>>>>>>>>> >>>>>>>>>>>> It is easier for now to keep all of them within apache/airflow >>>>>>>>>>>> < >>>>>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow> >>>>>>>>>>>> >>>>>>>>>>> repository >>>>>>>>>>> >>>>>>>>>>>> it seems and use a labelling scheme to separate those (there is >>>>>> nothing >>>>>>>>>>>> wrong with that but it might seem a bit hacky). It's a bit >>>>> easier >>>>>> to >>>>>>>>>>>> maintain with access and CI. >>>>>>>>>>>> >>>>>>>>>>>> We could also think about separate apache/airflow-ci, >>>>>> apache/airflow-dev, >>>>>>>>>>>> apache/airflow-prod or smth similar - that would require some >>>>>>>>>>>> infrastructure tickets and is not very common. >>>>>>>>>>>> >>>>>>>>>>>> 2) What labelling scheme to use(apache/airflow:label). My >>>>>> proposal is >>>>>>>>>>>> similar to this (if we keep everything in the airflow >>>>> repository) >>>>>>>>>>>> >>>>>>>>>>>> - *latest* = latest released version (python 3.5) = * >>>>>>>>>>>> >>>>>>>>>>> v1.10.3-python3.5* >>>>>>>>>>> >>>>>>>>>>>> - *master* = latest master version (python 3.5) = >>>>>>>>>>>> >>>>>>>>>>> *v2.0.0dev0-python3.5* >>>>>>>>>>> >>>>>>>>>>>> - *v1.10.3-python3.5,v1.10.3-python3.6* - released 1.10.3 >>>>>> with python >>>>>>>>>>>> 3.5/3.6 >>>>>>>>>>>> - *latest-ci *= latest released version of CI variant (python >>>>>> 3.5) >>>>>>>>>>>> *v1.10.3-ci-python3.5* >>>>>>>>>>>> - *master-ci* = latest master version of CI variant (python >>>>>> 3.5) >>>>>>>>>>>> *v2.0.0dev0-ci-python3.5* >>>>>>>>>>>> - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released >>>>>> 1.10.3 with >>>>>>>>>>>> python 3.5/3.6 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> My preference is to keep all the images in one repo and use >>>>>> labelling >>>>>>>>>>>> scheme as above, >>>>>>>>>>>> but I am open to discuss this. >>>>>>>>>>>> >>>>>>>>>>>> J, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Jarek Potiuk >>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software >>>>> Engineer >>>>>>>>>>>> >>>>>>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Jarek Potiuk >>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>>> >>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jarek Potiuk >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>> >>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
