Re: [Proposal] Component changes for Apache Airflow

2020-03-03 Thread Jiajie Zhong
I'm not quick familiar with JIRA, but after we remove the ability to create them from the bug-creating screen, Do we could avoid someone create components by accident? Best Wish — Jiajie

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-03-03 Thread Evgeny Shulman
Hey Everybody I am happy to contribute to this discussion! It's something that everybody is missing at Airflow. If I am part of a discussion around orchestration framework comparison, "functional DAGs” arguments are the most used ones. We are building a similar approach and would be happy to bas

Re: [Proposal] Component changes for Apache Airflow

2020-03-03 Thread Aizhamal Nurmamat kyzy
Please, disregard the link in my previous email that was added accidentally :/ On Tue, Mar 3, 2020, 10:27 AM Aizhamal Nurmamat kyzy wrote: > Thank you for spotting it, Jiajie! We should definitely combine them, and > remove duplicates. > > My guess is that the new components were added accident

Re: [PROPOSAL] Approach for releasing the backported "providers" packages

2020-03-03 Thread Jarek Potiuk
Going back to the topic i rethought it and had some discussions with Tomek and I think the best and least invasive way to implement the dependencies between provider packages will be to use extras - similar to airflow. We can generate the extras automatically for each providers package: Then if fo

Re: [Proposal] Component changes for Apache Airflow

2020-03-03 Thread Aizhamal Nurmamat kyzy
Thank you for spotting it, Jiajie! We should definitely combine them, and remove duplicates. My guess is that the new components were added accidentally by committers / pmc members in the bug-creating screen. It seems one can create a component by pressing enter twice on the new component. If po

Re: [Proposal] Component changes for Apache Airflow

2020-03-03 Thread Jiajie Zhong
Hey guys, sad to said that we have some similar component in JIRA now, such as `db` vs `database`, or `test` vs `flaky_test`, or `executor` vs `executor-kubernetes`. Should we combine them, and limit fix the component to avoid regression? Best Wish — Jiajie

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

2020-03-03 Thread Kaxil Naik
The goal would be to support both MySQL and PostgreSQL for production as we know many of Airflow users use MySQL as Metadata DB. On Tue, Mar 3, 2020 at 12:25 PM Ash Berlin-Taylor wrote: > It _shouldn't_, and we will test extensively with mysql. > > Worse case is we'll have to fall back to managi

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

2020-03-03 Thread Ash Berlin-Taylor
It _shouldn't_, and we will test extensively with mysql. Worse case is we'll have to fall back to managing the lock ourselves with a column rather than relying on db/row level locks. This might be a case where we have different/specialised behaviour for different dbs, or even db versions, if sa

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

2020-03-03 Thread Kamil Breguła
Hello, Will reliance on the database cause problems with MySQL? A lot of my users use this database. I am afraid that the lock mechanism in MySQL is much less stable and predictable than PostgresSQL and this can cause various stability problems. I know that Astronomer uses PostgreSQL, but Airflow

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

2020-03-03 Thread Kaxil Naik
Good work on the Proposal Ash & Vikram. On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka wrote: > Team, > > > > We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better > Scheduling Performance' on Confluence and would very much appreciate > feedback and suggestions from the community.