Re: Tip for Running Unit Testing against Your Own Fork

Let's add this info to https://github.com/apache/incubator-airflow/blob/master/dev/README.md . Can someone send a PR ?? On 22/11/2018, 09:52, "Jarek Potiuk" wrote: Seems that the dev list does not like attachments. The screenshot showing the right setting is here:

Re: [VOTE] Airflow 1.10.1 RC2

ly annoyed at the Apache release process and how long it takes, it means each release ends up getting big) -ash > Kaxil Naik Data Reply Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com www.reply.com On 17 Nov 2018, at 01:0

Re: [VOTE] Airflow 1.10.1 RC2

-1 (binding) . Tested it on Python 2.7.14, got expected result but had 1 security concern that I want to get in the release. Even when 'expose_config'=False, RBAC you still shows the configs which can contain sensitive information like airflow metadb passwords, etc. If we can get that in +1

Re: Airflow 1.10.1b1 release available - PLEASE TEST

Good work Ash. Appreciate the clean and categorised Change Log. Regards, Kaxil On 09/11/2018, 13:45, "Ash Berlin-Taylor" wrote: Hi Everyone, I've just released a beta version of 1.10.1! Please could you test this and report back any problems you notice, and also report back if

Re: Python 3.6 Support for Airflow 1.10.0

//stackoverflow.com/cv/taylor> | LinkedIn > > <https://www.linkedin.com/in/tedmiston/> | AngelList > > <https://angel.co/taylor> | Stack Overflow > > <https://stackoverflow.com/users/149428/taylor-edmiston> > > > &g

Re: Python 3.6 Support for Airflow 1.10.0

update setup.py to include 3.6 as well? @bolke – Thughts? Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com<mailto:k.n...@reply.com> www.reply.com [Data Reply] From: Naik Kaxil Reply-To

Python 3.6 Support for Airflow 1.10.0

Hi all, @fokko – I remember that you had test Airflow on 3.6 . Can we include 3.6 in setup.py then ? Regards, Kaxil Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com

Airflow 1.10.0 is released on PyPI

Dear Airflow community, Airflow 1.10.0 was just released on PyPI (`pip install apache-airflow`). https://pypi.python.org/pypi/apache-airflow The documentation is available on: * https://airflow.apache.org * https://airflow.readthedocs.io/en/stable/ and

Re: [RESULT][VOTE] Release Airflow 1.10.0

ld be awesome to get 1.10 on Pypi :-) > > Op wo 22 aug. 2018 om 23:43 schreef Naik Kaxil : > > > Mine is "kaxil" > > > > On 22/08/2018, 16:18, "Bolke de Bruin" wrote: > > > > @max > >

Re: Jira cleanup and triage

Thanks @Gerado, Appreciate that. I have closed/resolved the JIRAs. On 24/08/2018, 05:07, "Gerardo Curiel" wrote: Hi Kaxil, On Wed, Aug 22, 2018 at 6:53 PM Naik Kaxil wrote: > Wherever you see a an open issue associated with merged PR, p

Re: [VOTE] Airflow 1.10.0rc4

+1 (binding) Tested it on Python2.7 with flask UI On 09/08/2018, 23:58, "Daniel Imberman" wrote: -1 (non-binding), There is a k8s bug fix that should be PRed by @jordan.zucker this weekend (relating to the tracking of resourceVersions). There have also been multiple users

Re: Multiple hosts for a single connection

Agree that we should have the same functionality over the WebUI and CLI. We need multiple connections for a single conn-id for load balancing, so +1 for adding something like `--allow-multiple` or just a warning when a connection with the same id is created with cli. On 07/08/2018, 15:29,

Re: [VOTE] Airflow 1.10.0rc3

@bolke Can we also include the following commit to 1.10 release as we would need this commit to generate docs at ReadTheDocs? - https://github.com/apache/incubator-airflow/commit/8af0aa96bfe3caa51d67ab393db069d37b0c4169 Regards, Kaxil On 06/08/2018, 14:59, "James Meickle" wrote: Not a

Readthedocs is Broken

Hi Guys, The readthedocs builds are failing due to https://github.com/apache/incubator-airflow/pull/3660: https://readthedocs.org/projects/airflow/builds/7585546/ Does someone know how we can pass an environment variable (SLUGIFY_USES_TEXT_UNIDECODE=yes) to readthedocs environment? Is it

Re: Airflow committers (a list)

Last time I updated to add my name, it was reflected the next day. I had added the instructions here when I joined PPMC: https://cwiki.apache.org/confluence/display/AIRFLOW/Committers%27+Guide#Committers'Guide-AddingNewCommitters/PMCMemberstotheAirflowIncubationStatusPage Jenkins job runs

Re: Podling Report Reminder - August 2018

Just reviewed it and looks good to me. Just one thing, do we just need to resolve the license issue for graduation? Regards, Kaxil On 05/08/2018, 22:15, "Sid Anand" wrote: Done. Folks, Please review and correct if anything is off:

Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

Hi Airflow'ers, Please join the Apache Airflow PMC in welcoming its newest member and co-committer, Feng Tao (a.k.a. feng-tao). Welcome Feng, great to have you on board! Cheers, Kaxil Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

Awesome. Thanks @Verdan On 23/07/2018, 07:58, "Verdan Mahmood" wrote: Heads-up!! This frontend change has been merged in master branch recently. This will impact the users working on Airflow RBAC UI only. That means: *If you are a contributor/developer of Apache Airflow:*

Re: Sep Airflow Bay Area Meetup @ Google

Hi Feng, Will the session be recorded? Will love if it is. :) On 19/07/2018, 22:26, "Feng Lu" wrote: Hi all, Hope you are enjoying your summer! This is Feng Lu from Google and we'll host the next Airflow meetup in our Sunnyvale campus

Re: DAG Level permissions (was Re: RBAC Update)

Awesome work Feng. Well done. On 17/07/2018, 13:27, "James Meickle" wrote: Really excited for this one - we have a lot of internal access controls and this will help us implement them properly. It's going to be great being able to give everyone access to see the overall state of

Re: [VOTE] Airflow 1.10.0rc2

I have run tests on python 2.7 and 3.5 and it works fine. My vote: +1 (binding) Thanks @bolke. Regards, K On 16/07/2018, 09:03, "fo...@driesprongen.nl on behalf of Driesprong, Fokko" wrote: My vote is ​+1 (binding)​ Cheers, Fokko 2018-07-16 9:52 GMT+02:00 Driesprong,

Re: [VOTE] Airflow 1.10.0rc1

gt; Op 10 jul. 2018 om 03:22 heeft Naik Kaxil het volgende geschreven: > > +1 > > Regards, > Kaxil > > On 10/07/2018, 02:09, "Kengo Seki" wrote: > >+1 (non-binding) > >- Checked signatures an

Re: [VOTE] Airflow 1.10.0rc1

+1 Regards, Kaxil On 10/07/2018, 02:09, "Kengo Seki" wrote: +1 (non-binding) - Checked signatures and checksums - Confirmed the source distribution corresponds to v1-10-test's HEAD (f1083cb). - Installed the binary distribution with cryptography package ran some

Environment variables not working when using systemd

Hi guys, I have recently setup Airflow on a new VM with system integration. I have added some environment variables in my bash_rc file for airflow user. Now when I try to run a BashOperator by first starting airflow using `airflow webserver -D` and `airflow scheduler -D` it seem to have access

Securing Connections

I would like to get thoughts on how you guys secure connections i.e. Role-based control of connection. For example I don’t want Person A to use connection X, or in other words I only want Person B to have access of connection X. With RBAC in the master, it is possible but how do you guys

Re: SparkSubmitOperator with Azure HDInsight

equire you to open ports/copy the hadoop configuration to your airflow machine. Niels 2018-06-22 14:17 GMT+02:00 Naik Kaxil : > Hi all, > > > > Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure > HDInsight cluster? Are yo

SparkSubmitOperator with Azure HDInsight

Hi all, Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure HDInsight cluster? Are you using Livy or spark-submit to run remote Spark jobs? Regards, Kaxil Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730

Re: Apache Airflow 1.10.0b2

It is working on the FAB UI after removing the old airflow.cfg file and running `airflow initdb`. But with that the old webserver seems to be broken (Issue has already raised by Kevin here: https://issues.apache.org/jira/browse/AIRFLOW-2624). Regards, Kaxil On 20/06/2018, 22:35, "Naik

Re: Apache Airflow 1.10.0b2

Thanks Bolke. I have been testing this today and I see that the logs are not showing up for me in the Web UI (old UI using Flask Admin). The path of my log is: /Users/kaxil/airflow/logs/example_bash_operator/runme_0/2018-06-20T21:31:36.996580+00:00/1.log However, The log tab in the UI is

Re: Airflow 1.10.0

Yes it does make sense. I will try to help out in testing. On 15/06/2018, 18:01, "Bolke de Bruin" wrote: Ok to kick this a bit forward. I can spend 1 hour every day coming week to get 1.10 out. This means I will rebranch 1.10 from master coming Monday. Every day until Friday I will make

Re: Master is broken

Hi Gerado, As Fokko mentioned, sometimes the CI breaks because of some flaky test and sometime just fails due to connectivity issues while downloading required packages. If a certain commits break the master, we try to fix it asap. I have merged the PR that reverts the commit that broke the

Re: Airflow - High Availability and Scale Up vs Scale Out

ervices. In my > limited trial, it did what we wanted but we need more time to test. > > On Fri, Jun 8, 2018, 4:23 AM Naik Kaxil wrote: > > > Hi guys, > > > > > > > > I have 2 specific questions for the guys using Airflow in

Airflow - High Availability and Scale Up vs Scale Out

Hi guys, I have 2 specific questions for the guys using Airflow in production? 1. How have you achieved High availability? How does the architecture look like? Do you replicate the master node as well? 2. Scale Up vs Scale Out? * What is the preferred approach you take? 1 beefy

Re: Airflow-related Talk: Functional Data Engineering, a set of Best Practices

Thanks for sharing this Max. On 06/06/2018, 01:23, "Maxime Beauchemin" wrote: I'm taking the freedom to share my talk from DataEngConf 2018 with this group since it's somewhat related to Airflow. https://www.youtube.com/watch?v=4Spo2QRTz1k Related blog post:

Re: Make sure your Airflow instance isn't public and isn't Google indexed

+1 for securing it by default. I also suggest to create a admin user password while running "initdb". On 05/06/2018, 22:49, "Bolke de Bruin" wrote: Tbh I like to go to a setup where it is secure by default. Airflow is getting more and more used so it also increases the attack surface.

Re: Adding moderators to the email lists for an incubating project

Hi Sid, Count me in. My Apache id is 'kaxilnaik'. Regards, Kaxil Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com www.reply.com [Data Reply]

Re: Airflow London Meetup Group

- https://twitter.com/ApacheAirflow/status/1002594373849120769 >- https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements > > > On Fri, Jun 1, 2018 at 5:42 AM, Naik Kaxil wrote: > >> We have just created the Airflow London meetup and hope to grow the >> A

Re: Adding moderators to the email lists for an incubating project

Sid, Count me in as well. My apache id is 'kaxilnaik'. Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com www.reply.com [Data Reply] From:

Re: Airflow London Meetup Group

We have just created the Airflow London meetup and hope to grow the Airflow community in London. Meetup URL: https://www.meetup.com/London-Apache-Airflow-Meetup/ Looking forward to meet Airflow users in London. Regards, Kaxil On 01/06/2018, 10:31, "Tautvydas" wrote: I'd be interested

Airflow London Meetup Group

Hi all, We(me and my colleagues at Data Reply UK) are planning to start a meetup for Airflow users. Me & one of my friend already gave a talk at PyData London last month and people showed huge interest in it. We have a good space for meetup at our offices in central London (in Victoria to be

1.10.0 Release

Hi Bolke, Fokko, Are we going to rebranch master for v1.10 and delete the current v1.10-test branch? What are the plans for it and in case Bolke if you are busy, I am happy to assist alongwith @Fokko or anyone who is free? Regards, Kaxil On 21/05/2018, 18:58, "Bolke de Bruin"

Re: Where to put docs on configuring specific kinds of connections? (Or restructuring the docs the way Django does)

Hi Tim, My personal opinion is to write the docs on GCP connection down at GCP Integration page would be an ideal fit but would like to hear others opinion as well. +1 for creating separate How-To guides.

Re: celery problem, AttributeError: async

Thanks. Please do that. On 21/05/2018, 14:59, "Craig Rodrigues" wrote: celery 4.1.1 was just released last night which has all the async problems fixed: https://github.com/celery/celery/commits/v4.1.1 I'll test this out, and then submit a PR to bump

Re: Cannot access https://cms.apache.org/incubator/publish

Hi Sid, I think Jenkins trigger a build on incubator svn repo daily as seen here https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/426/console When I checked http://incubator.apache.org/projects/airflow.html the page has already been updated. However, I still tried to

Re: Cannot access https://cms.apache.org/incubator/publish

Sure I will do this and update the wiki On 15/05/2018, 14:51, "siddharth anand" wrote: Kaxil, Can you try these steps and update the airflow wiki (committer guide) based on your findings? -s Sent from Sid's iPhone Kaxil Naik Data

Re: Facing SSL issue with dev/airflow-pr tool

Hi Sumit, I think it is unrelated to the PR tool. Please check the below link: https://stackoverflow.com/questions/44316292/ssl-sslerror-tlsv1-alert-protocol-version Try running: pip install requests[security] Regards, Kaxil On 15/05/2018, 07:46, "Sumit Maheshwari"

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

Hi all, My apache account creation has been sorted out. Few of the questions I have are: - How should I verify that I have enough permissions to merge a PR? - Can someone grant my Jira user (kaxilnaik) permission to edit/close tickets? - Same for github (userid= kaxil) - Same

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

and co-committer, Naik Kaxil (a.k.a. kaxil <https://github.com/kaxil>). Welcome Kaxil, great to have you on board! Cheers, Fokko Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1W 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com www.reply.com

Re: Managed Apache Airflow Service on Google Cloud Platform

Finally!! ( Was quite excited for the beta release after trying the alpha. Nice work Feng Lu and the team. Regards, Kaxil On 01/05/2018, 20:07, "Gerard Toonstra" wrote: Really good stuff. About 1.5 years ago I talked to some googlers on how awesome it would be

Re: Airflow 1.10 Release Date

iPad > Op 11 apr. 2018 om 11:01 heeft Naik Kaxil <k.n...@reply.com> het volgende geschreven: > > Hi, > > Me & my friend at going to present on Apache Airflow on Cloud at PyData London this month. > > Do we have a release data for

Airflow 1.10 Release Date

Hi, Me & my friend at going to present on Apache Airflow on Cloud at PyData London this month. Do we have a release data for Airflow 1.10 that I can announce there and features it might include? Regards, Kaxil Kaxil Naik Data Reply 38 Grosvenor Gardens London SW1W 0EB - UK phone: +44

Slides / Presentations for Airflow

Hi all, Do we maintain slides or presentation materials like Apache Beam does ? https://beam.apache.org/contribute/presentation-materials/ Or if anyone has just presented somewhere can I get the link to slides and videos if possible? Regards, Kaxil Kaxil Naik Data Reply 38 Grosvenor

Re: Readthedocs

sername is Fokko Cheers, Fokko 2018-04-10 12:56 GMT+02:00 Naik Kaxil <k.n...@reply.com>: > Additionally, I have also updated the docs to reflect the latest change. > > https://airflow.readthedocs.io/en/latest/index.html > > https://airflo

Re: Readthedocs

Additionally, I have also updated the docs to reflect the latest change. https://airflow.readthedocs.io/en/latest/index.html https://airflow.readthedocs.io/en/latest/start.html Regards, Kaxil On 10/04/2018, 11:54, "Naik Kaxil" <k.n...@reply.com> wrote: Can you c

Re: Readthedocs

Can you create an account on readthedocs and give me your user id? I'll add you to it. Also, You can run the "sphinx-build -b html . ~/Desktop/airflowdoc" command on your local machine to generate the docs and "incubator-airflow/docs/_build/html' directory to see the generated docs.

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Congrats Ash ( On 24/02/2018, 20:23, "fo...@driesprongen.nl on behalf of Driesprong, Fokko" wrote: Welcome Ash! 2018-02-23 18:45 GMT+01:00 Andy Hadjigeorgiou : > Congrats Ash! > > >

Re: Airflow Documentation - Readthedocs

. Have created a PR (https://github.com/apache/incubator-airflow/pull/3033). Regards, Kaxil On 12/02/2018, 01:54, "Naik Kaxil" <k.n...@reply.com> wrote: Hi all, I think instead of the old "airflow" docs on PyPi mentioned by @ash, I think the latest docs

Re: Airflow Documentation - Readthedocs

e "Destroy" button. > > Max > > On Sat, Feb 10, 2018 at 12:53 PM, Naik Kaxil <k.n...@reply.com> wrote: > > > From what I see the package owners at PyPi are mistercrunch, aoen, artwr > . > > Will it be possible for one of you to ta

Re: Airflow Documentation - Readthedocs

s/airflow/builds/6724340/) On Fri, Feb 9, 2018 at 12:56 PM, Naik Kaxil <k.n...@reply.com> wrote: > Thanks Andy for doing that. Can you please do the same for version 1.9 and > 1.8.2 if possible? If Arthur adds me I will do it but if you can spare some

Re: Airflow Documentation - Readthedocs

could be > > > > picked up so it can be made opart of the release process. > > > > > > > > @Arthur @max: can you share / make available credentials? > > > > > > > > Cheers > > > > Bolke > > > > > > >

Re: Airflow Documentation - Readthedocs

p so it can be made opart of the release process. > > @Arthur @max: can you share / make available credentials? > > Cheers > Bolke > > > > On 4 Feb 2018, at 15:16, Naik Kaxil <k.n...@reply.com> wrote: > > > > Yes, the same h

Re: Airflow Documentation - Readthedocs

you share / make available credentials? Cheers Bolke > On 4 Feb 2018, at 15:16, Naik Kaxil <k.n...@reply.com> wrote: > > Yes, the same happened to me. One of my colleague wanted to use a Google Cloud Storage operator but he was looking at the ol

Re: Airflow Documentation - Readthedocs

project, maybe they can fix the build? Or add some more Airflow committers as readthedocs project owners. I get quite some questions from colleagues because they are reading old docs :) Cheers, Fokko 2018-02-04 14:32 GMT+01:00 Naik Kaxil <k.n...@reply.com&g

Airflow Documentation - Readthedocs

Hi guys, Are we still using http://airflow.readthedocs.io/ for latest documentation? I see that the last build was 2 months ago which failed.. http://readthedocs.org/projects/airflow/builds/ The documentation at http://pythonhosted.org/airflow/ will only be for the latest airflow version at

Re: How to trigger the dag to run immediately

low") my_dag_run.execute({"config": effective_config}) On Thu, Feb 1, 2018 at 9:05 PM Naik Kaxil <k.n...@reply.com> wrote: > Hi Song Liu, > > In Airflow 1.8 and higher there is a button for each dag on the dashboard > that looks like a play

Re: How to trigger the dag to run immediately

Hi Song Liu, In Airflow 1.8 and higher there is a button for each dag on the dashboard that looks like a play button, you will need to click on that button to Trigger that DAG. More info: https://stackoverflow.com/a/43375518/5691525 . Regards, Kaxil On 01/02/2018, 18:38, "Song Liu"