[VOTE] Airflow Backport Providers 2020.5.20rc2

2020-05-19 Thread Jarek Potiuk
Hey all, I have cut Airflow Backport Providers 2020.5.20rc2. This email is calling a vote on the release, which will last for 72 hours - which means that it will end on Fri May 22 18:15:00 CEST 2020. The main differences from RC1 are (Thanks Ash for those comments and suggestions. it's the first

Re: [VOTE] Airflow Backport Providers - 2020.5.20rc1

2020-05-19 Thread Jarek Potiuk
As explained before I am cancelling the vote for rc1 and will send a vote for rc2 in a moment. J. On Mon, May 18, 2020 at 3:02 PM Jarek Potiuk wrote: > Yep. Good point. It has no chance to work. I will remove it from release > list. > > J. > > On Mon, May 18, 2020 at 2:40 PM Ash Berlin-Taylor

Re: Celery Executor test failing - help needed to solve it

2020-05-19 Thread Jarek Potiuk
Fantastic! Great job Ash! On Tue, May 19, 2020 at 12:14 PM Ash Berlin-Taylor wrote: > PR created that I think will fix it > https://github.com/apache/airflow/pull/8909 > > -ash > > On May 19 2020, at 10:08 am, Jarek Potiuk > wrote: > > > Issue created:

Re: Celery Executor test failing - help needed to solve it

2020-05-19 Thread Ash Berlin-Taylor
PR created that I think will fix it https://github.com/apache/airflow/pull/8909 -ash On May 19 2020, at 10:08 am, Jarek Potiuk wrote: > Issue created: https://github.com/apache/airflow/issues/8908 > > On Tue, May 19, 2020 at 11:06 AM Asif Saif Uddin wrote: > >> value = Exception('>

Re: Celery Executor test failing - help needed to solve it

2020-05-19 Thread Asif Saif Uddin
value = Exception('((\'(psycopg2.errors.UniqueViolation) duplicate key value violates..."pg_type_typname_nsp_index"\\nDETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.\\n\',))') tb = None def reraise(tp, value, tb=None): """Reraise exception.""" if value.__traceback__

Re: [PROPOSAL] Secret Backend Hooks

2020-05-19 Thread Jarek Potiuk
Cool. I thought it's a misunderstanding :). Great it is clear now! J. On Tue, May 19, 2020 at 11:17 AM Ash Berlin-Taylor wrote: > Yes, sorry I got completely the wrong idea somehow. This makes sense, > and thank you for patiently explaining it to me until I got it! > > My main reason for

Re: [PROPOSAL] Secret Backend Hooks

2020-05-19 Thread Ash Berlin-Taylor
Yes, sorry I got completely the wrong idea somehow. This makes sense, and thank you for patiently explaining it to me until I got it! My main reason for questioning was not this specific feature, but the gradual "scope creep" of Airflow operators. One of the hardest things we as project

Re: Celery Executor test failing - help needed to solve it

2020-05-19 Thread Jarek Potiuk
Issue created: https://github.com/apache/airflow/issues/8908 On Tue, May 19, 2020 at 11:06 AM Asif Saif Uddin wrote: > value = Exception(' \'sqlalchemy.exc.IntegrityError\'>((\'(psycopg2.errors.UniqueViolation) > duplicate key value violates..."pg_type_typname_nsp_index"\\nDETAIL: Key >

Re: Celery Executor test failing - help needed to solve it

2020-05-19 Thread Tomasz Urbaszek
Should we create an issue? Maybe Asif (CC) from Celery team will be able to help us? Tomek On Tue, May 19, 2020 at 10:39 AM Ash Berlin-Taylor wrote: > > I'll take a look. I'm surprised by that error though, it looks like two > processes are racing to create a table, and not dealing with

Re: Celery Executor test failing - help needed to solve it

2020-05-19 Thread Ash Berlin-Taylor
I'll take a look. I'm surprised by that error though, it looks like two processes are racing to create a table, and not dealing with losing. (I suspect we can fix it by pre-creating that tbl,/calling something before we spawn processes, but this really is a bug in celery.) On 19 May 2020

Re: [PROPOSAL] Secret Backend Hooks

2020-05-19 Thread Jarek Potiuk
Yep. That is exactly why I think we need the hooks. Is it possible you donate your code for the Vault Hook implementation ? I would love to use it for my implementation. (And make you or whoever the author is as co-author :) J. wt., 19 maj 2020, 09:41 użytkownik Nathan Hadfield napisał: >

Re: [PROPOSAL] Secret Backend Hooks

2020-05-19 Thread Nathan Hadfield
Jarek, We are already using the secret backend for Airflow variables. But, because of the example I explained and also a programmatic need to update our GCP Airflow connections every day, then we still have to maintain a secondary, custom method for Vault authentication and manipulation of

Celery Executor test failing - help needed to solve it

2020-05-19 Thread Jarek Potiuk
Hello Everyone, but mostly I think Kamil, Ash, Kaxil, I think we need someone who knows a bit more about Celery Executor to stabilise one of the tests that is transiently failing. I think this started to appear recently (so likely it is one of the optimizations implemented recently) You can see

Re: [PROPOSAL] Secret Backend Hooks

2020-05-19 Thread Jarek Potiuk
Let me start again from scratch and use KMS as an example. Maybe - again - we understand things differently: Just to start KMSHook -> has two methods: "encrypt" and "decrypt". I would continue to use that as a base. Again let me repeat that. I do not want to implement a generic SecretHook! I