Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Sid Anand
Good points Ben. Gitter has Communities and Rooms. Currently, we seem to be represented in 2 Gitter communities and a few rooms: - Airflow Developers (This was our first Gitter community) - 2 rooms - Dev - Lobby - Apache - incubator-airflow The Airflow

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Ben Gregory
+1 to Slack over Gitter but understand the concerns of disruption to community and message retention. IMO - Gitter's main disadvantage is forcing all conversations through one channel which means following any one issue is confusing and inefficient. I've stopped checking it so much precisely

Re: Use xcom in task retry

2018-09-06 Thread Ben Gregory
Hi Mishika -- Posting this question on StackOverflow with some code examples you're using will likely be the fastest way to have this addressed. Just make sure to tag it with "airflow" so people can find it. - Ben On Thu, Sep 6, 2018 at 12:12 AM Mishika Singh wrote: > I am pushing some

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Sid Anand
Some good points. @Arthur Wiedmer The Apache mailing list is always the official record and Google indexes it well, so that any previous discussions on the mailing lists are available to future generations. I originally started this thread with the thought of having everyone on the mailing list,

Re: Best Practice of Airflow Setting-Up & Usage

2018-09-06 Thread Deng Xiaodong
Thanks for sharing, Raman. Based on what you shared, I think there are two points that may be worth further discussing/thinking. *Scaling up (given thousands of DAGs):* If you have thousands of DAGs, you may encounter longer scheduling latency (actual start time minus planned start time). For

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread William Horton
+1 *William Horton* Backend Engineer 90 Fifth Avenue, 3rd Floor New York NY 10011 m: 404.432.8480 On Thu, Sep 6, 2018 at 10:03 AM, Ravi Kotecha wrote: > -1 > > On Thu, Sep 6, 2018 at 2:01 PM Chandu Kavar wrote: > > > +1 > > > > On Thu, Sep 6, 2018, 10:30 AM Sid Anand wrote: > > > > > Hi

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Ravi Kotecha
-1 On Thu, Sep 6, 2018 at 2:01 PM Chandu Kavar wrote: > +1 > > On Thu, Sep 6, 2018, 10:30 AM Sid Anand wrote: > > > Hi Folks! > > In the Apache tradition, I'd like to ask the community to vote on > replacing > > Gitter with Slack. > > > > For more information about what was recently discussed,

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Chandu Kavar
+1 On Thu, Sep 6, 2018, 10:30 AM Sid Anand wrote: > Hi Folks! > In the Apache tradition, I'd like to ask the community to vote on replacing > Gitter with Slack. > > For more information about what was recently discussed, refer to > >

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread James Meickle
-1 unless we get a Slack with full retention. On Thu, Sep 6, 2018 at 8:16 AM Daniel Cohen wrote: > +1 > > On Thu, Sep 6, 2018 at 3:04 PM Adam Boscarino > wrote: > > > +1 > > > > On Wed, Sep 5, 2018 at 10:30 PM Sid Anand wrote: > > > > > Hi Folks! > > > In the Apache tradition, I'd like to ask

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Adam Boscarino
+1 On Wed, Sep 5, 2018 at 10:30 PM Sid Anand wrote: > Hi Folks! > In the Apache tradition, I'd like to ask the community to vote on replacing > Gitter with Slack. > > For more information about what was recently discussed, refer to > >

Re: Security issue being ignored?

2018-09-06 Thread Bolke de Bruin
Both are not security vulnerabilities: either it is in an upstream project or it is due to the way Airflow can be used. PR is welcome for the second JIRA. B. Verstuurd vanaf mijn iPad > Op 6 sep. 2018 om 11:07 heeft airflowuser > het volgende geschreven: > > Another example: >

Re: Best Practice of Airflow Setting-Up & Usage

2018-09-06 Thread ramandumcs
Hi, We have a requirement to scale to run 1000(s) concurrent dags. With celery executor we observed that Airflow worker gets stuck sometimes if connection to redis/mysql breaks (https://github.com/celery/celery/issues/3932 https://github.com/celery/celery/issues/4457) Currently we are using

Re: Security issue being ignored?

2018-09-06 Thread airflowuser
Another example: https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-2283 Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐ Original Message ‐‐‐ On September 3, 2018 10:20 AM, airflowuser wrote: > Hi, > I noticed you opened a disccusion about the neccesity of

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Kaxil Naik
0 (binding) I prefer using StackOverflow + Mailing list which are searchable in Google. And as someone said before the free slack version has only 10,000 messages that are searchable. If ASF provides a paid version, than I am happy to move to Slack - as it is for sure better than gitter.

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Eamon Keane
+1 On Thu, Sep 6, 2018 at 8:28 AM Bolke de Bruin wrote: > 0 (binding) abstaining from the vote. > > I am not convinced we gain anything except disruption for an existing > community. We cannot even force people to move away only make it “less” > official. My question would be do we get more

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread Bolke de Bruin
0 (binding) abstaining from the vote. I am not convinced we gain anything except disruption for an existing community. We cannot even force people to move away only make it “less” official. My question would be do we get more attendance of contributors and committers if we move to slack?

Re: [VOTE] Replace with Gitter with Slack?

2018-09-06 Thread airflowuser
-1 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On September 6, 2018 5:30 AM, Sid Anand wrote: > Hi Folks! > In the Apache tradition, I'd like to ask the community to vote on replacing > Gitter with Slack. > > For more information about what was recently discussed,

Use xcom in task retry

2018-09-06 Thread Mishika Singh
I am pushing some [key,value] using xcom in an operator, which I want to use when that task fails and comes for retry, for which I am using xcom_pull in that operator. But it is returning None instead of that value. Any pointer around this will be helpful. -- Regards Mishika Singh

Re: Call for fixes for Airflow 1.10.1

2018-09-06 Thread Deng Xiaodong
Hi Ash, May you consider including JIRA ticket 2848 (PR 3693, Ensure dag_id in metadata "job" for LocalTaskJob) in 1.10.1 as well? https://issues.apache.org/jira/browse/AIRFLOW-2848 https://github.com/apache/incubator-airflow/pull/3693 This is a bug in terms of metadata, which also affects