Re: [VOTE] Switch from using Jira to Github Issues

+1 binding On Mon, Mar 16, 2020 at 11:44 PM Kevin Yang wrote: > +1 binding > > On Mon, Mar 16, 2020 at 9:34 AM Tao Feng wrote: > > > +1 > > > > On Mon, Mar 16, 2020 at 7:36 AM Ash Berlin-Taylor > wrote: > > > > > The original discuss thread is here. > > > > > > https://lists.apache.org/thread.

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

Haven't checked the math in the AIP but I believe with the given formula, with 5 schedulers and 100 DAGs there is already a 9% chance of conflict and the larger users of Airflow have many more DAGs than that. I'm a bit concerned putting about putting more load on the DB which is already a scalabil

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

Would be happy to give +1 for this AIP later! XD On Mon, Mar 16, 2020 at 11:08 PM Ash Berlin-Taylor wrote: > Does anyone have any other opinions about this? If not I'd like to call a > vote (and start working on the code!) > > -ash > On Mar 3 2020, at 12:34 pm, Kaxil Naik wrote: > > The goal

Re: [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

Does anyone have any other opinions about this? If not I'd like to call a vote (and start working on the code!) -ash On Mar 3 2020, at 12:34 pm, Kaxil Naik wrote: > The goal would be to support both MySQL and PostgreSQL for production as we > know many of Airflow users use MySQL as Metadata DB.

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Hi guys, I am new here, and usually I try to create tickets on Jira before starting to work on a task. I think the problem here is not the tool like Jira but the way we are using it. I do not know if there is a process for creating tasks but I assume it looks like this: check keywords if such an

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding On Mon, Mar 16, 2020 at 9:34 AM Tao Feng wrote: > +1 > > On Mon, Mar 16, 2020 at 7:36 AM Ash Berlin-Taylor wrote: > > > The original discuss thread is here. > > > https://lists.apache.org/thread.html/r19014fac5aa2aa15a86890aa438940da263e430cb535ca451ef451a4%40%3Cdev.airflow.apache.or

Re: [VOTE] Switch from using Jira to Github Issues

+1 On Mon, Mar 16, 2020 at 7:36 AM Ash Berlin-Taylor wrote: > The original discuss thread is here. > https://lists.apache.org/thread.html/r19014fac5aa2aa15a86890aa438940da263e430cb535ca451ef451a4%40%3Cdev.airflow.apache.org%3E > > Proposal is: > > - We create a few appropriate issue templates in

RE: [DISCUSS] Stop using Jira (since we aren't using it properly)

I would just like to add some extra positive thoughts for this. Firstly as a newcomer JIRA is confusing, even coming from a word that does use JIRA internally it's not what you see for most open source projects so it's far more familiar to use GitHub issues. (actually as a side note one negati

Re: [VOTE] Switch from using Jira to Github Issues

I wasn't think so, not for now. The Confluence based-flow isn't causing (me) any pain points. -ash On 2020/03/16 15:11:33, Darren Weber wrote: > +1 > > What about AIPs - will they migrate also? > > On Mon, Mar 16, 2020 at 8:01 AM Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > >

Re: [VOTE] Switch from using Jira to Github Issues

+1 What about AIPs - will they migrate also? On Mon, Mar 16, 2020 at 8:01 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > +1 binding > > On Mon, Mar 16, 2020, 7:45 AM Jarek Potiuk > wrote: > > > +1 binding. > > > > Re: contention: I think we can split reviewers by components. I thin

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding On Mon, Mar 16, 2020, 7:45 AM Jarek Potiuk wrote: > +1 binding. > > Re: contention: I think we can split reviewers by components. I think the > assignment to components was done recently so duplicates will be rare. I am > happy to be one of the reviewers. > > On Mon, Mar 16, 2020 at 3

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding. Re: contention: I think we can split reviewers by components. I think the assignment to components was done recently so duplicates will be rare. I am happy to be one of the reviewers. On Mon, Mar 16, 2020 at 3:42 PM Kaxil Naik wrote: > +1 binding > > BTW, we can enable Github Issues

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding T. On Mon, Mar 16, 2020 at 3:42 PM Kaxil Naik wrote: > > +1 binding > > BTW, we can enable Github Issues by just adding the following block in > *airflow/.asf.yaml* > > github: > features: > # Enable issues management > issues: true > # Enable projects for project manage

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding BTW, we can enable Github Issues by just adding the following block in *airflow/.asf.yaml* github: features: # Enable issues management issues: true # Enable projects for project management boards projects: true On Mon, Mar 16, 2020 at 2:36 PM Ash Berlin-Taylor wrot

[VOTE] Switch from using Jira to Github Issues

The original discuss thread is here. https://lists.apache.org/thread.html/r19014fac5aa2aa15a86890aa438940da263e430cb535ca451ef451a4%40%3Cdev.airflow.apache.org%3E Proposal is: - We create a few appropriate issue templates in ./github -- specifically also one for "Ask for help" that basically s

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Yep. Vote and switch :) On Mon, Mar 16, 2020 at 3:12 PM Tomasz Urbaszek wrote: > Yes, vote and switch. > > > On Mon, Mar 16, 2020 at 3:06 PM Ash Berlin-Taylor wrote: > > > Yeah, Superset is using GIthub Issues instead of Jira. > > > > This is probably the third or fourth time the Github/Jira su

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Yes, vote and switch. On Mon, Mar 16, 2020 at 3:06 PM Ash Berlin-Taylor wrote: > Yeah, Superset is using GIthub Issues instead of Jira. > > This is probably the third or fourth time the Github/Jira subject has been > brought up, something just finally pushed me over the edge of "why are we > bo

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

VOTE and switch. On Mon, Mar 16, 2020 at 2:06 PM Ash Berlin-Taylor wrote: > Yeah, Superset is using GIthub Issues instead of Jira. > > This is probably the third or fourth time the Github/Jira subject has been > brought up, something just finally pushed me over the edge of "why are we > botherin

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Yeah, Superset is using GIthub Issues instead of Jira. This is probably the third or fourth time the Github/Jira subject has been brought up, something just finally pushed me over the edge of "why are we bothering with this" today. It seems like we have fairly broad agreement this time. AIP wort

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

+1 for Github issues. Github allows creating issue template (feature, bug, custom) so this should help. And I have a feeling that GH issues are indexed better than JIRA tickets. JIRA gives the possibility to interlink between ASF projects but I don't think is something important for us. I've also

Re: [VOTE] AIP-33 Secrets Backend

+1 binding On Mon, Mar 16, 2020 at 1:16 PM Driesprong, Fokko wrote: > +1 > > Op ma 16 mrt. 2020 om 12:53 schreef Bas Harenslak > : > > > +1 > > > > > On 16 Mar 2020, at 12:27, Ash Berlin-Taylor wrote: > > > > > > +1 binding > > > > > > On Mar 16 2020, at 11:20 am, Kaxil Naik wrote: > > > > > >

Re: Stateless Webserver with DAG Serialization

Thank you all :) On Sun, Mar 15, 2020 at 8:35 AM Driesprong, Fokko wrote: > Thanks for the update Kaxil, this is awesome! 💯 > > Cheers, Fokko > > Op za 14 mrt. 2020 om 13:40 schreef Tomasz Urbaszek >: > > > Thanks Kaxil, Ash, Anita and Zhou! This is a big step forward :) > > > > T. > > > > On

Re: [VOTE] AIP-33 Secrets Backend

+1 Op ma 16 mrt. 2020 om 12:53 schreef Bas Harenslak : > +1 > > > On 16 Mar 2020, at 12:27, Ash Berlin-Taylor wrote: > > > > +1 binding > > > > On Mar 16 2020, at 11:20 am, Kaxil Naik wrote: > > > >> +1 (binding) > >> > >> On Mon, Mar 16, 2020 at 11:19 AM Jarek Potiuk > > >> wrote: > >> > >>>

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

+1 One other problem it would help us solve is *closing issues where the PR is merged*. This is one of the pain-points for us, some of the JIRA issues are open even though the PR is merged. With Github issues, if there is a PR solving an existing issue just adding "fixes #20" would close that iss

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

And for the clean state -> I am rather for a "brute" approach. I.e. review them and only move those that people reviewing them find necessary to keep. Mark the others as stale, add comment "we are closing them in a week - please create a github issue if you want to keep it" and close the remaining

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

> Maybe we could have some clear guidelines on when the issues should be > created - only when there is a problem someone wants to report and we have no > code for it yet. Yes, exactly. If you want to submit a fix directly: great, open a PR; if you want to report it but arent able/willing to su

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

I am all for it. We can easily rely just on PR# to uniquely identify commit rather than Github issue id - and remove the requirement to have an issue altogether? The issue can be added optionally but it should not be a requirement. I think PRs and Issues are pretty equivalent when you follow the "

Re: [VOTE] AIP-33 Secrets Backend

+1 > On 16 Mar 2020, at 12:27, Ash Berlin-Taylor wrote: > > +1 binding > > On Mar 16 2020, at 11:20 am, Kaxil Naik wrote: > >> +1 (binding) >> >> On Mon, Mar 16, 2020 at 11:19 AM Jarek Potiuk >> wrote: >> >>> Clarify: +1 (binding) >>> >>> On Mon, Mar 16, 2020 at 12:19 PM Jarek Potiuk >>>

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

> I'm totally in favor of not using Jira, as they are serving hardly any > purpose other than just a useless step before creating a PR. However, I don't > think to make a GitHub issue mandatory is also a good step, as eventually, > it'll meet the same fate of being opened just before opening a P

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

I'm totally in favor of not using Jira, as they are serving hardly any purpose other than just a useless step before creating a PR. However, I don't think to make a GitHub issue mandatory is also a good step, as eventually, it'll meet the same fate of being opened just before opening a PR. So IMO

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Yeah, Github issues are far from perfect, it's mainly just I feel we have a lot of "busy-work" in our process that is no longer really serving much benefit to us as a community. -a On Mar 16 2020, at 11:35 am, Bolke de Bruin wrote: > Honestly, I think both suck. So I can go either way > > > On

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Honestly, I think both suck. So I can go either way On 16 March 2020 at 12:33:27, Ash Berlin-Taylor (a...@firemirror.com) wrote: The subject pretty much says it all. We aren't using Jira very well in most cases, and the requirement for a Jira ticket for a code change leads to people just creati

Re: Let's agree to guidelines for AIP (Airflow Improvement Proposal)

Yes, I like the procedural issues one (that includes lazy consensus) too. On Mon, Mar 16, 2020, 11:31 Jarek Potiuk wrote: > (and BTW when we vote on this procedure we should follow voting process on > procedural issues (same link - above) > > "Votes on procedural issues follow the common forma

[DISCUSS] Stop using Jira (since we aren't using it properly)

The subject pretty much says it all. We aren't using Jira very well in most cases, and the requirement for a Jira ticket for a code change leads to people just creating new Jira tickets, rather than searching to see if there already exists a ticket for that feature. For example: https://issues.a

Re: Let's agree to guidelines for AIP (Airflow Improvement Proposal)

(and BTW when we vote on this procedure we should follow voting process on procedural issues (same link - above) "Votes on procedural issues follow the common format of majority rule unless otherwise stated. That is, if there are more favourable votes than unfavourable ones, the issue is considere

Re: Let's agree to guidelines for AIP (Airflow Improvement Proposal)

Agree with the proposal in general. However I think this is about code modification, so we should rather follow Votes on code modifications rather than releases: https://www.apache.org/foundation/voting.html#votes-on-code-modification That means: - we sum all votes and positive means "passed

Re: [VOTE] AIP-33 Secrets Backend

+1 binding On Mar 16 2020, at 11:20 am, Kaxil Naik wrote: > +1 (binding) > > On Mon, Mar 16, 2020 at 11:19 AM Jarek Potiuk > wrote: > >> Clarify: +1 (binding) >> >> On Mon, Mar 16, 2020 at 12:19 PM Jarek Potiuk >> wrote: >> >> > +1 >> > >> > On Mon, Mar 16, 2020 at 12:16 PM Kaxil Naik wro

Re: [VOTE] AIP-33 Secrets Backend

+1 (binding) On Mon, Mar 16, 2020 at 11:19 AM Jarek Potiuk wrote: > Clarify: +1 (binding) > > On Mon, Mar 16, 2020 at 12:19 PM Jarek Potiuk > wrote: > > > +1 > > > > On Mon, Mar 16, 2020 at 12:16 PM Kaxil Naik wrote: > > > >> Hey all, > >> > >> This email calls for a vote on the scope describe

Re: [VOTE] AIP-33 Secrets Backend

Clarify: +1 (binding) On Mon, Mar 16, 2020 at 12:19 PM Jarek Potiuk wrote: > +1 > > On Mon, Mar 16, 2020 at 12:16 PM Kaxil Naik wrote: > >> Hey all, >> >> This email calls for a vote on the scope described in >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-33+Secrets+backend >> and >

Re: [VOTE] AIP-33 Secrets Backend

+1 On Mon, Mar 16, 2020 at 12:16 PM Kaxil Naik wrote: > Hey all, > > This email calls for a vote on the scope described in > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-33+Secrets+backend > and > the PR: https://github.com/apache/airflow/pull/6376 > > *Note*: The PR is already merged

[VOTE] AIP-33 Secrets Backend

Hey all, This email calls for a vote on the scope described in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-33+Secrets+backend and the PR: https://github.com/apache/airflow/pull/6376 *Note*: The PR is already merged but if this vote does not pass we will revert it. Documentation on th

Re: [DISCUSS] Back to (some) dependency pinning

On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko wrote: > Personally I don't like to have two versions in the PyPi repo. This also > complicates the releases, since we need to test, release and verify two > versions of Airflow > Not necessarily. Release is automated (and I run both package bui

Re: [DISCUSS] AIP-33 Creds Backend

I will mark this AIP as accepted, given we now have 4 +1 binding votes and already 8 days have passed since this AIP was created. +1 bindings: - Kaxil - Jarek - Tomek - Fokko I will now create a VOTE thread

Let's agree to guidelines for AIP (Airflow Improvement Proposal)

Hey all, I feel like we have good guidelines on creating an AIP, however, there we don't have "clear" guidelines on the following (We might already do, in which case please correct me): 1. How long should the *Vote *on AIP go on? 2. Minimum number of votes required to marked the AIP as "ac

Re: [DISCUSS] AIP-33 Creds Backend

I am also +1 for the idea of secrets backend. And agree with Fokko that we should support other backends as well. T. On Mon, Mar 16, 2020 at 11:19 AM Kaxil Naik wrote: > > Bumping this thread again. We have 2 "+1" votes on it currently (Me & > Jarek). > > Would love to hear other's opinion and

Re: [DISCUSS] AIP-33 Creds Backend

Here is the PR https://github.com/apache/airflow/pull/6376 that I have merged already (Apologies for merging this before everyone voted on the AIP) The backend is pluggable and I am planning to extend it Google KMS and Hashicorp Vault over the weekend but if someone wants to go ahead please feel f

Re: [DISCUSS] AIP-33 Creds Backend

Yes, big +1 from my side as well. I think this is a requirement for any bigger company. 1. I think we should add support for KMS, Vault etc. 2. Make the current backend pluggable, and set the current as default (Airflow metastore) Cheers, Fokko Op ma 16 mrt. 2020 om 11:19 schreef Kaxi

Re: [DISCUSS] AIP-33 Creds Backend

Bumping this thread again. We have 2 "+1" votes on it currently (Me & Jarek). Would love to hear other's opinion and thoughts. Personally I feel this is a great addition and the one I was looking forward to for a long time. Regards, Kaxil On Sun, Mar 8, 2020 at 8:33 PM Kaxil Naik wrote: > Than

Re: [DISCUSS] Back to (some) dependency pinning

Personally I don't like to have two versions in the PyPi repo. This also complicates the releases, since we need to test, release and verify two versions of Airflow. I'm afraid that this might confuse users. Besides that, it feels a bit like we're reinventing certain mechanisms that are already in

Re: [DISCUSS] Back to (some) dependency pinning

Does anyone think this one needs voting? I think it's an incremental change (but one that helps having stable installation mechanism on installing any released version of Airflow). J. On Sun, Mar 15, 2020 at 1:17 PM Jarek Potiuk wrote: > I think I have finally found a good and simple solution