Re: execution_date - can we stop the confusion?

What about (aliasing) execution_date to period_start, and next_execution_date to period_end? Would this help any do we think? (Though things like ds and ts might still be confusing? This is probably where the OP got the idea for run_stamped from? One step at a time.) Ash On 27 September 2018 2

Re: Solved: suppress PendingDeprecationWarning messages in airflow logs

Doesn't this warning come from the BaseOperator class - https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L2511 ? Are you passing unused arguments to the QuboleOperator, or do you not control the instantiation of those operators? Chris On Fri, Sep 28, 2018 at 7:39 PM Sean

Re: Solved: suppress PendingDeprecationWarning messages in airflow logs

Haha yes, good point. Thanks! Sean On 9/28/18, 6:13 PM, "Ash Berlin-Taylor" wrote: Sounds good for your use, certainly. I mainly wanted to make sure other people knew before blindly equipping a foot-canon :) -ash > On 29 Sep 2018, at 00:09, Sean Carey wrote

Re: Solved: suppress PendingDeprecationWarning messages in airflow logs

Sounds good for your use, certainly. I mainly wanted to make sure other people knew before blindly equipping a foot-canon :) -ash > On 29 Sep 2018, at 00:09, Sean Carey wrote: > > Thanks, Ash. I understand what you're saying. The warnings are coming from > the Qubole operator. We get a lo

Re: Solved: suppress PendingDeprecationWarning messages in airflow logs

Thanks, Ash. I understand what you're saying. The warnings are coming from the Qubole operator. We get a lot of this: PendingDeprecationWarning: Invalid arguments were passed to QuboleOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args:

Re: Solved: suppress PendingDeprecationWarning messages in airflow logs

What deprecation warnings are you getting? Are they from Airflow itself (i.e. things Airflow is calling like flask_wtf, etc) or of your use of Airflow? If it is the form could you check and see if someone has already reported a Jira issue so we can fix them? https://issues.apache.org/jira/issue

Solved: suppress PendingDeprecationWarning messages in airflow logs

Hello, I’ve been looking for a way to suppress the PendingDeprecationWarning messages cluttering our airflow logs and I have a working solution which I thought I would share. In order to do this, you first need to configure airflow for custom logging using steps 1-4 here: https://airflow.read

Re: Fundamental change - Separate DAG name and id.

Exactly that Ash! This awesome community never fails to impress. Thanks folks! On Tue, Sep 25, 2018 at 3:29 AM Ash Berlin-Taylor wrote: > > > On 24 Sep 2018, at 23:12, Alex Tronchin-James 949-412-7220 > <(949)%20412-7220> wrote: > > > > Re: [Brian Greene] "How does filename matter? Frankly I w

Jira issue updating adventures & dashboard (aka holdensmagicalunicorn)

So I _think_ I figured out why my latest attempt at updating JIRA issues went off-the-rails, right now general user account don't have permission to start progress on issues which they did not create in the airflow project even if they are assigned to them. For now the dashboard is up in a read-on

Flask App Builder [FAB] support

The new [experimental] web UI is based on Flask App Builder [FAB] to which we contributed security fixes recently. It's been hard to get the main maintainer of the project's attention to release to Pypi. For context, I have write access to the repository, but no access to release to Pypi. Please

Re: Travis CI tests failing in master

Thanks everyone for quick resolve! I can confirm it also works for us now and we are unblocked with sending our first PR very soon :). J. On Fri, Sep 28, 2018 at 12:13 PM Kaxil Naik wrote: > Haha. Well, what can I say? The solution was changing `pip` tp `pip2`. > Banging my head against the wal

Re: Travis CI tests failing in master

Haha. Well, what can I say? The solution was changing `pip` tp `pip2`. Banging my head against the wall. I just read yesterday that it wasn't working and now 😁... well well... On Fri, Sep 28, 2018 at 10:02 AM Ash Berlin-Taylor wrote: > Looks like someone has beaten you to the punch > https://git

Re: Travis CI tests failing in master

Looks like someone has beaten you to the punch https://github.com/apache/incubator-airflow/pull/3968#pullrequestreview-159772413 :) -ash > On 28 Sep 2018, at 09:59, Kaxil Naik wrote: > > The problem is 80% s

Re: Travis CI tests failing in master

The problem is 80% solved. The only failing error are related to TestPythonVirtualenvOperator . More details are discussed at https://github.com/apache/incubator-airflow-ci/pull/3 It is because protocol=3 is used instead of 2. Can someone take a dig to resolve this? Regards, Kaxil On Thu, Sep

Re: Slides from London Airflow Meetup #1

Wow the slides captured much more info that I thought and the talk covered much wider range about Airflow than I thought. Very useful, great job, thank you very much! Cheers, Kevin Y On Tue, Sep 25, 2018 at 3:36 AM Sumit Maheshwari wrote: > Thanks a lot, Ash. > > > > On Tue, Sep 25, 2018 at 3:4

Re: SqlAlchemy Pool config parameters to minimize connectivity issue impact

Hi Kevin, We are using Airflow 1.9 with Local Excecutor and we are facing various mysql connectivity error/exceptions like -> sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE task_inst

Re: SqlAlchemy Pool config parameters to minimize connectivity issue impact

Hi Raman, Would you elaborate a bit more on what exactly is the connectivity issues that you were facing and which version of Airflow are you on? We previously had some connectivity issue when the # of connection was too large and we had it fixed with this PR

Re: Airflow 1.10 Migration Duration

Thank you Taylor, you are right there are changes that are not backwards compatible and that should not be an expectation, e.g. table name change from user to users. To provide on data point from our test, the upgrade from our alembic revsion cc1e65623dc7 to the head of 1.10.0rc4, alembic revision