Re: Release cycle and patch releases

2022-05-11 Thread Robin Edwards
ut for a while this time to > hear from others first :) > > On Wed, May 4, 2022 at 6:32 PM Robin Edwards wrote: >> >> This is probably slightly touching on the issues Jarek and Kevin were >> discussing in the release announcement however i think it warrants its >>

Release cycle and patch releases

2022-05-04 Thread Robin Edwards
This is probably slightly touching on the issues Jarek and Kevin were discussing in the release announcement however i think it warrants its own thread. Firstly i'd like to thank everyone for their hard work in 2.3, I haven't had time to try it out yet but i do look forward giving it a spin. We r

Re: [VOTE] Release Airflow 2.2.2 from 2.2.2rc2

2021-11-15 Thread Robin Edwards
+1 (non-binding) On Mon, 15 Nov 2021 at 02:47, Josh Fell wrote: > > +1 (non-binding) > > Verified some bug fixes and ran a few DAGs. > > On Sun, Nov 14, 2021 at 2:39 PM Leon Smith wrote: >> >> +1 (non-binding) >> >> On Sun, 14 Nov 2021 at 16:28, Jarek Potiuk wrote: >>> >>> +1 (binding) - signat

Re: [VOTE] Release Airflow 2.2.2 from 2.2.2rc1

2021-11-10 Thread Robin Edwards
e a look at this issue today regardless. > > -ash > > On Wed, Nov 10 2021 at 10:57:08 +, Robin Edwards > wrote: > > -1 (non binding) As https://github.com/apache/airflow/issues/19461 is not > fixed which i consider critical. On Tue, 9 Nov 2021 at 22:31, Jed Cunning

Re: [VOTE] Release Airflow 2.2.2 from 2.2.2rc1

2021-11-10 Thread Robin Edwards
-1 (non binding) As https://github.com/apache/airflow/issues/19461 is not fixed which i consider critical. On Tue, 9 Nov 2021 at 22:31, Jed Cunningham wrote: > > Hey fellow Airflowers, > > I have cut Airflow 2.2.2rc1. This email is calling a vote on the release, > which will last for 72 hours, f

Re: [VOTE] Release Airflow 2.2.1 from 2.2.1rc2

2021-10-29 Thread Robin Edwards
+1 non binding On Fri, 29 Oct 2021, 10:09 Jarek Potiuk, wrote: > +1 (binding) : basic tests work, signature, licence, checksum - all OK. > > On Fri, Oct 29, 2021 at 7:50 AM Elad Kalif wrote: > >> +1 (binding) >> >> בתאריך יום ו׳, 29 באוק׳ 2021, 11:44, מאת Leon Smith ‏: >> >>> +1 (non-binding) >

Re: Language matters - don't use the term "whitelist" in Apache Airflow

2020-06-08 Thread Robin Edwards
Nice one Ash, +1 R On Mon, 8 Jun 2020 at 05:04, Jarek Potiuk wrote: > > Indeed. Thanks Ash! Much appreciated! > > J. > > On Mon, Jun 8, 2020 at 1:40 AM Kevin Yang wrote: > > > Thank you Ash. +1 and stamped. > > > > > > Cheers, > > Kevin Y > > > > On Sun, Jun 7, 2020 at 2:46 PM Deng Xiaodong w

Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection balancing HA )

2020-04-24 Thread Robin Edwards
+1 On Fri, 24 Apr 2020, 11:32 Kaxil Naik, wrote: > +1 > > On Fri, Apr 24, 2020 at 11:31 AM Ash Berlin-Taylor wrote: > > > As we discussed in the thread on Spec thread for our new API, the idea > > came up of making conn_id unique in Airflow > > > > < > > > https://lists.apache.org/thread.html/r

Re: API spec questions

2020-04-11 Thread Robin Edwards
On Sat, 11 Apr 2020, 01:20 QP Hou, wrote: > It looks like we need to first make a decision on whether we want to > enforce uniqueness for conn_id and implement connection load balance > properly in another way. > Yea I feel there are plenty of other tools and services for implementing load balan

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc4

2020-04-08 Thread Robin Edwards
Just loaded up our dev environment and done a few sanity checks and also ran our test suite. Thank you everyone for their hard work very much looking forward to rolling this out. Python 3.7 / Postgres / Celery +1 (non-binding) On Tue, 7 Apr 2020 at 23:19, Xinbin Huang wrote: > > Test with Pyth

Re: Let's talk Airflow 2.0

2020-04-02 Thread Robin Edwards
ook at Cypress for this over Selenium. What > > ever > > > we choose we need to pay careful attention to avoid slow or flakey UI > > tests. > > > > > > > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've > > b

Re: New committer: Jiajie Zhong

2020-04-01 Thread Robin Edwards
Nice one Jiajie :-) Cheers, R On Wed, 1 Apr 2020 at 07:58, Michał Słowikowski wrote: > > Congratulations! > > Best regards > M. > > wt., 31 mar 2020, 21:00 użytkownik Xinbin Huang > napisał: > > > Congratulations! > > > > On Tue, Mar 31, 2020, 11:13 AM Felix Uellendall > > wrote: > > > > > We

Re: Let's talk Airflow 2.0

2020-03-22 Thread Robin Edwards
I feel some of the stuff for instance Schedular HA could wait for a point release of version 2 (although maybe this a lot further a long than I am aware). Like you mentioned Spark did with K8s. Also does the new API need to be feature complete or just enough functionality to warrant removing the

Re: [VOTE] Switch from using Jira to Github Issues

2020-03-18 Thread Robin Edwards
woohoo! +1 none binding On Wed, 18 Mar 2020, 08:36 Michał Słowikowski, < michal.slowikow...@polidea.com> wrote: > +1 (non-binding) > > On Tue, Mar 17, 2020 at 10:13 PM Sid Anand wrote: > > > (resending from my apache account .. sorry) > > > > +1 Binding > > > > Airflow's returning full-circle t

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

2020-02-25 Thread Robin Edwards
This is brilliant work, thank you! Looking forward to watching my RDS metrics when this gets deployed :-) On Tue, 25 Feb 2020, 07:08 Driesprong, Fokko, wrote: > Sweet work Kamil and others! I'll try to go through them today! > > Cheers, Fokko > > Op ma 24 feb. 2020 om 22:37 schreef Tao Feng : >

Re: DAG Runs Archival Policy

2020-01-24 Thread Robin Edwards
Hey Vardan, I also run a system with a large number of DAGs. Regarding the slowness in the UI there are a few fixes that went into 1.10.7 which reduced the number of DAGs Airflow loads when browsing. There is also a couple more changes going into the next release (I hope!) which will speed it up

Re: [AirFlow]: Pandas DataFrame Between Tasks

2019-12-24 Thread Robin Edwards
Feather is probably a good option for data frames: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_feather.html R On Tue, 24 Dec 2019 at 07:52, Deng Xiaodong wrote: > > Hi David. > > The only “out of box” way to share data/information between tasks is XCom ( > htt

Re: [VOTE] Release 1.10.7 from 1.10.7rc3

2019-12-21 Thread Robin Edwards
+1 (none binding) Tested Python 3.7 on Postgresql Thanks for everyones hard work, Rob On Sat, 21 Dec 2019, 19:10 Jarek Potiuk, wrote: > +1 (binding). > > Tested: > >- Python 3.5 + mysql >- Python 3.6 + postgres >- Python 2.7 + postgres >- Python 2.7 + sqlite >- I also doub

Re: [VOTE] Release Airflow 1.10.4 from RC5

2019-08-06 Thread Robin Edwards
+1 (none-biding) Done a nightly run in production on python 3.7 with no issues. Thanks guys! R On Tue, 6 Aug 2019 at 09:31, Jarek Potiuk wrote: > > +1 (non-binding) as I am not PMC member > > Tested installation and run airflow locally in several configurations > (including Python 2.7/3.5). No

Re: [VOTE] Release Airflow 1.10.4 from RC3

2019-07-17 Thread Robin Edwards
+1 (none binding) - been running in production since RC2. Thanks for all your hard work R On Tue, 16 Jul 2019 at 21:15, Ash Berlin-Taylor wrote: > > Thanks for testing. > > On 1) everyone should run upgradedb on every upgrade. The behaviour of not > running it wasn't great. > > 2) I thought we

Handling look back periods in data sources

2019-04-26 Thread Robin Edwards
I was wondering if anyone has a pattern for handling look back periods when ingesting data. One of our sources attributes cost up to 90 days in arrears. So currently I am ingesting @daily but re-writing the last 90 days with depends_on_past=True on my operator. This feels wrong as I am effecting da

Re: Airflow 1.10.3 has been released!

2019-04-12 Thread Robin Edwards
Yes thanks everyone for their hard work :-) R On Fri, 12 Apr 2019 at 18:38, Feng Lu wrote: > > Great work, thank you Ash! > > On Thu, Apr 11, 2019 at 4:28 PM Beau Barker > wrote: > > > Thanks Ash and contributors. > > > > > On 12 Apr 2019, at 3:38 am, Gabriel Silk > > wrote: > > > > > > Nice w

Strange subdag task SIGTERM issue

2019-01-04 Thread Robin Edwards
Hello, I have an issue with my subdags occasionally getting stuck which seems to be related to their subtasks being terminated. The subdag task its self will be marked as failed but the child tasks will be left in the state 'up_for_retry'. I am using the SequentialExecutor for the SubDag and the C