Re: MySQL version support for Airflow 5.6 ->? 5.7

2020-03-01 Thread Jarek Potiuk
Or actually we can force the charset to utf8mb3 to keep the potential to use most utf8 characters in the ids/keys I will test it shortly but it should work. On Mon, Mar 2, 2020 at 8:38 AM Jarek Potiuk wrote: > Another option that came to me - and possibly that's the simplest and best > one. > >

Re: MySQL version support for Airflow 5.6 ->? 5.7

2020-03-01 Thread Jarek Potiuk
Another option that came to me - and possibly that's the simplest and best one. We can force charset for only the _id + key columns to be latin1. This might solve all the problems at once with pretty much zero impact on the rest of the system. It seems super-simple. https://stackoverflow.com/ques

Re: [PROPOSAL][AIP-32] Airflow REST API

2020-03-01 Thread Matt Buell
@Jarek your points have fully convinced me. I like the spec first approach On Sun, Mar 1, 2020 at 2:03 PM Vikram Koka wrote: > In response to the point raised by Jarek, i.e. 'API first' or 'Code first', > looking at the document, one of the stated goals is: > "The API will be intended for use

Re: MySQL version support for Airflow 5.6 ->? 5.7

2020-03-01 Thread Jarek Potiuk
> > > Shouldn't be a problem in practice as we delete XComs for a task before > running out again. Might happen in very rare edge casesi guess. > > Experience has taught me that if you want the mysql optimiser to do > something sensible: it won't. This may be better in newer versions, but was > s

Re: MySQL version support for Airflow 5.6 ->? 5.7

2020-03-01 Thread Ash Berlin-Taylor
Oh the ti_id idea I mentioned below won't work very well for the case when we query by non-exact execution date On 1 March 2020 20:06:08 GMT, Ash Berlin-Taylor wrote: > > >On 1 March 2020 19:40:59 GMT, Jarek Potiuk >wrote: >>> >>> >>> >>> does c) really help? We'd still need a unique key on the

Re: MySQL version support for Airflow 5.6 ->? 5.7

2020-03-01 Thread Ash Berlin-Taylor
On 1 March 2020 19:40:59 GMT, Jarek Potiuk wrote: >> >> >> >> does c) really help? We'd still need a unique key on the existing PK >> columns - i.e. does Mysql just complain about the length/size of the >PK >> index, but not other unique indexes? >> > >Right indeed - so the problem (and I unde

Re: MySQL version support for Airflow 5.6 ->? 5.7

2020-03-01 Thread Jarek Potiuk
> > > > does c) really help? We'd still need a unique key on the existing PK > columns - i.e. does Mysql just complain about the length/size of the PK > index, but not other unique indexes? > Right indeed - so the problem (and I understand the reason why we changed 1.10 -> 2.0 ) is also the uniqu

Re: [PROPOSAL][AIP-32] Airflow REST API

2020-03-01 Thread Vikram Koka
In response to the point raised by Jarek, i.e. 'API first' or 'Code first', looking at the document, one of the stated goals is: "The API will be intended for use by any third party. It should not be related to a specific application, e.g. a React UI" In my opinion, the 'Code first' approach on

Re: [PROPOSAL][AIP-32] Airflow REST API

2020-03-01 Thread Ash Berlin-Taylor
On Mar 1 2020, at 5:11 pm, Jarek Potiuk wrote: > At this point you immediately lose all the flexibility that comes with "code > first" approach: you released the API already and you are not supposed to > change how the API is designed, yet the design is by-product of the code so > it is very di

Re: [PROPOSAL][AIP-32] Airflow REST API

2020-03-01 Thread Jarek Potiuk
I have not yet reviewed the API in detail - and I am sure there will be more things coming up while implementing it, but just one general comment (to Ash and Matt concerns). I think we need to agree on the "API first" vs. "Code First" approach (and possibly vote if we have disagreement after discu