Re: Graduation resolution passed - Airflow is a TLP

Yeaaah! Great moment to witness! Arthur Wiedmer 于2018年12月21日 周五上午5:58写道： > Amazing! > > Great work all! > > On Thu, Dec 20, 2018, 13:13 Jakob Homan > > Hey all- > >The Board minutes haven't been published yet (probably due to > > Holiday-related slowness), but I can see through the admin

Re: Large number of SQL Connection from airflow scheduler

Hi Anand, >From my experience that reusing connection pool across those parsing processes would be dangerous( recall I got some connection corruption error because of that). If you find that to be too many connection you can probably tune down the max_threads

Re: [VOTE] Graduate the Apache Airflow as a TLP

+1 (non-binding) On Fri, Nov 30, 2018 at 2:15 PM Stefan Seelmann wrote: > +1 (non-binding) > > On 11/30/18 10:33 PM, Jakob Homan wrote: > > Hey all! > > > > Following a very successful DISCUSS[1] regarding graduating Airflow to > > Top Level Project (TLP) status, I'm starting the official VOTE.

Re: Call for fixes for Airflow 1.10.2

Hi Kaxil, Ty for running 1.10.2, that's pretty fast. Can you please also include the following bug fix in 1.10.2 please? AIRFLOW-3414 : Fix reload_module in DagFileProcessorAgent Cheers, Kevin Y On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik

Re: Remove airflow from pypi

tall package apache-airflow instead of airflow > > > > -------- > > Command "python setup.py egg_info" failed with error code 1 in > > /private/var/folders/km/xypq2kxs4ys3dt6bwtd4fbj0gn/T/pip-install-7y_awdxy/airflow/ > &

Re: Airflow issues

Have you checked the task log locally on the worker box (default under airflow_home/logs)? If you configured remote logging it is possible that the task exited and the failure handler didn’t catch that and thus not uploading log? If there’s no such log, maybe try look at the worker process log to

Re: Remove airflow from pypi

+1 for having a placeholder for security reason. Not sure about adding dependency on apache-airflow tho--if the warning is overlooked and stuff starting to break because of backward incompatibility, it can be quite confusing. Cheers, Kevin Y On Sat, Nov 24, 2018 at 5:19 PM soma dhavala wrote:

Re: programmatically creating and airflow quirks

I believe that is mostly because we want to skip parsing/loading .py files that doesn't contain DAG defs to save time, as scheduler is going to parse/load the .py files over and over again and some files can take quite long to load. Cheers, Kevin Y On Fri, Nov 23, 2018 at 12:44 AM soma dhavala

Re: Airflow 1.10.1 is released

 Thank you Ash for cutting the release! On Wed, Nov 21, 2018 at 6:09 PM Sid Anand wrote: > Excellent work Ash! Thanks for doing the needful!! > > -s > > On Wed, Nov 21, 2018 at 5:40 PM Tao Feng wrote: > > > Thanks Ash for running the release! > > > > On Wed, Nov 21, 2018 at 2:20 PM Ash

Re: Guidelines around how to scale with worker nodes?

Hi Jerry, This may require you to profile the tasks running on your machines yourself--so you can get an idea how much computation resources your tasks are consuming. Generally # of worker nodes is the root cause for tasks gettign stuck in QUEUED state. You can verify that by comparing the # of

Re: A Naive Multi-Scheduler Architecture Experiment of Airflow

Finally we start to talk about this seriously? Yeah! :D For your approach, a few thoughts: 1. Shard by # of files may not yield same load--even very different load since we may have some framework DAG file producing 500 DAG and take forever to parse. 2. I think Alex Guziel

Re: Slides from London Airflow Meetup #1

Wow the slides captured much more info that I thought and the talk covered much wider range about Airflow than I thought. Very useful, great job, thank you very much! Cheers, Kevin Y On Tue, Sep 25, 2018 at 3:36 AM Sumit Maheshwari wrote: > Thanks a lot, Ash. > > > > On Tue, Sep 25, 2018 at

Re: Airflow 1.10 Migration Duration

Thank you Taylor, you are right there are changes that are not backwards compatible and that should not be an expectation, e.g. table name change from user to users. To provide on data point from our test, the upgrade from our alembic revsion cc1e65623dc7 to the head of 1.10.0rc4, alembic revision