Re: Upgrading Airflow in Prod

2019-06-18 Thread ramandumcs
Thanks David & Teresa, We are thinking of similar kind of approach where we would have a dedicated Test env to test the Airflow upgrades. We have to preserve the metadata so we will be upgrading the Airflow version using Airflow cmd. Following steps would be done -> Setup Mysql Slave to Airflo

Re: [DISCUSS] Pull Request Improvements

2019-06-18 Thread Chao-Han Tsai
+1 Watched the pull panda video and I think the offering is great and beneficial to Airflow. Chao-Han On Tue, Jun 18, 2019 at 8:24 AM Ry Walker wrote: > +1 we use pull-reminders at Astronomer, good experience. Airflow might be > a bit rough at first with 193 pending PRs, but it will help get th

Re: [DISCUSS] Pull Request Improvements

2019-06-18 Thread Ry Walker
+1 we use pull-reminders at Astronomer, good experience. Airflow might be a bit rough at first with 193 pending PRs, but it will help get that # down :) Sent via Superhuman iOS ( https://sprh.mn/?vip=r...@rywalker.com ) On Tue, Jun 18 2019 at 7:31 AM, < fo...@driesprong.frl > wrote: > > > >

Re: Tagging of the airflow images

2019-06-18 Thread Jarek Potiuk
Yes I know. Being perfectionist, I could not help myself to correct it ;). *Versions from master (development use only):* - CI slim image *: airflow:master-python3.5-ci-slim, airflow:**master-python3.6-ci-slim, airflow:master-ci-slim*==airflow:master-python3.6-ci-slim - CI full image *:

Re: Tagging of the airflow images

2019-06-18 Thread Jarek Potiuk
I like the ordering too. Below is updated proposal: Ash: * FROM: - not really, we are using optimised multi-staging Dockerfile to generate different variants of the images. The variants are independent from each other but we use one common Dockerfile to generate all of them (AKA one source of tru

Re: Tagging of the airflow images

2019-06-18 Thread Ash Berlin-Taylor
I like this ordering and the reasoning given for it. > On 18 Jun 2019, at 13:54, Philippe Gagnon wrote: > > I know this is bikeshedding at this point but I think something more alone > the lines of: > > `apache/airflow:1.10.4-python3.5[-ci][-slim]` > > would be more appropriate as a standard.

Re: Tagging of the airflow images

2019-06-18 Thread Ash Berlin-Taylor
Only objection is around airflow:1.10.4-slimci-python3.5 - the way our release process works that should probably be airflow:1.10.x-slimci-python3.5 or similar - to represent the release branch, but we don't need a CI version for the tagged release, just the release branch. I think. Would the v

Re: Tagging of the airflow images

2019-06-18 Thread Philippe Gagnon
I know this is bikeshedding at this point but I think something more alone the lines of: `apache/airflow:1.10.4-python3.5[-ci][-slim]` would be more appropriate as a standard. Here is my rationale: - The airflow version should definitely come first. - The python version is the second-most signif

Re: [DISCUSS] Pull Request Improvements

2019-06-18 Thread Driesprong, Fokko
It looks great, and I think it Airflow would benefit from it. I would say go for it, and open up an infra ticket to get it set up :-) Cheers, Fokko Op di 18 jun. 2019 om 13:27 schreef Jiajie Zhong : > I watch the pull-panda video and think it is great! > I think maybe we could have a try on it.

Re: [DISCUSS] Pull Request Improvements

2019-06-18 Thread Jiajie Zhong
I watch the pull-panda video and think it is great! I think maybe we could have a try on it. Best Wish — Jiajie On Jun 18, 2019, at 17:45, Jarek Potiuk mailto:jarek.pot...@polidea.com>> wrote: Just started this morning to test Pull Panda on our internal repo. Looks like really useful - it's

Re: Tagging of the airflow images

2019-06-18 Thread Jarek Potiuk
Great :). I'd prefer to use single word rather than ci-slim (then we can use "-" as separator when splitting image name). The "slimci" seem like most appropriate : I also think that it might make sense to build production-optimised images all the time. This way we will notice when they break and w

Re: [DISCUSS] Pull Request Improvements

2019-06-18 Thread Jarek Potiuk
Just started this morning to test Pull Panda on our internal repo. Looks like really useful - it's slack integration is really nice. All for it! J. On Tue, Jun 18, 2019 at 11:44 AM Felix Uellendall wrote: > Hey all, > > Pull request are currently often forgotten after they moved to the second

[DISCUSS] Pull Request Improvements

2019-06-18 Thread Felix Uellendall
Hey all, Pull request are currently often forgotten after they moved to the second page and I think this is really frustrating for pr's authors to wait so long for their pr's getting merged and we can and should do more about it. Our number of committers is growing, but I think we can also impr

Re: Tagging of the airflow images

2019-06-18 Thread Ash Berlin-Taylor
"slim" is common amongst docker, so that sounds good. I think the "primary" docker images should be production focused, and anything else tagged (i.e. it is prod unless it says otherwise.) Since master is not meant for end use we could also _only_ have the CI versions of those images. *Version

Re: Tagging of the airflow images

2019-06-18 Thread James Coder
Just my 2 cents, at work we tend to use “slim” to denote things that are pared down. How about “ci-slim”? James Coder > On Jun 18, 2019, at 4:06 AM, Jarek Potiuk wrote: > > Ok so then next iteration of proposal: The only doubt I have myself is the > "master" vs. "master-prod". Maybe we should

Re: Tagging of the airflow images

2019-06-18 Thread Jarek Potiuk
Ok so then next iteration of proposal: The only doubt I have myself is the "master" vs. "master-prod". Maybe we should rather have "master" for "production-ready" image and use a different name for the "small-ci" image". Maybe "trimci" or "ci-lite" or "liteci" ? What do you think? *Versions from m