Re: DockerOperator - old API? What version/package API does this use?

Wow, this is actually quite complex owing to the unit tests... Russell Jurney @rjurney russell.jur...@gmail.com LI FB datasyndrome.com On Thu, Apr 27, 2017 at 9:12 PM, Russell Jurney wrote: > I crea

Re: DockerOperator - old API? What version/package API does this use?

I created a JIRA: https://issues.apache.org/jira/browse/AIRFLOW-1159 Russell Jurney @rjurney russell.jur...@gmail.com LI FB datasyndrome.com On Thu, Apr 27, 2017 at 9:00 PM, Russell Jurney wrote: >

Re: DockerOperator - old API? What version/package API does this use?

Ok, after some more poking around I found that this is the docker-py 1.6 API, which is pretty old. I'm going to update it to the latest and see if I can make it work, since we are using the newer docker API elsewhere. Russell Jurney @rjurney russell.jur...@gmail.com LI

DockerOperator - old API? What version/package API does this use?

I am trying to use DockerOperator and I get this error: from docker import Client, tls ImportError: cannot import name 'Client' There is no requirements.txt, so I can't figure out what docker API this is, but it hasn't existed in the official docker Python module for ... ever? I can't figure

Re: Run job multiple times with same execution date

Ah, I see what you're saying. That works, thanks! Alek On Thu, Apr 27, 2017 at 3:12 PM, Paul Zaczkiewicz wrote: > Right, but I was asking about next_execution_date, not execution_date. > > run time execution_date next_execution_date next_execution_date - > timedelta(days=1) > 2017-01-02 02:00 2

Re: Run job multiple times with same execution date

Right, but I was asking about next_execution_date, not execution_date. run time execution_date next_execution_date next_execution_date - timedelta(days=1) 2017-01-02 02:00 2017-01-01 17:00 2017-01-02 02:00 2017-01-01 02:00 2017-01-02 05:00 2017-01-02 02:00 2017-01-02 05:00 2017-01-01 05:00 2017-01

Re: Run job multiple times with same execution date

It's `0 2-20/3 * * *`. The problem with using an offset from the execution date is that, for the very first run of the day, its execution date will *already* be the previous day (since it covers 8pm the previous day til 2am today) - subtracting from that would yield a date two days in the past, whi

Re: Run job multiple times with same execution date

First off, what schedule are you giving your DAG? I assume it's something like "0 */4 * * *". Why don't you subtract 1 day from next_execution_date instead? On Thu, Apr 27, 2017 at 2:28 PM, Alek Storm wrote: > Right, an easier way to do that would probably be to use the > `prev_execution_date`

Re: Run job multiple times with same execution date

Right, an easier way to do that would probably be to use the `prev_execution_date` macro, but either way, it would give the wrong result on the first run of the day, since its start_date would already be the previous day. Alek On Thu, Apr 27, 2017 at 1:17 PM, Shah Altaf wrote: > Hello. You can

Re: Run job multiple times with same execution date

Hello. You can get access to the execution date of today and subtract a day from it. You can use a PythonOperator with provide_context=True, that will give your Python method a ds argument that has the execution date in it. Manipulate it and return an XCOM. For example, here's a filename being d

[CANCEL] [VOTE] Release Airflow 1.8.1 RC1

Canceling this vote due to discussion on incubator general list. Going to do an RC2 on Monday that includes AIRFLOW-1142, AIRFLOW-492, and has ONLY the tarball with an RC (not the files--they'll just be 1.8.1). On Mon, Apr 24, 2017 at 10:31 AM, Sumit Maheshwari wrote: > +1 (binding) > > > > On

Run job multiple times with same execution date

I'd like to create a job that runs six times a day, all with the same execution date (the previous day). Is there a way to do this without generating six separate DAGs, or running an `airflow clear` command in between runs? Thanks, Alek

Re: Accessing configuration parameters passed to Airflow through CLI

There was a discussion on google groups about that: https://groups.google.com/forum/#!topic/airbnb_airflow/GRdoW30PNUI On Thu, Apr 27, 2017 at 9:42 AM, Devjyoti Patra wrote: > Hi, > > I am trying to pass the following configuration parameters to Airflow CLI > while triggering a dag run. Followi

Accessing configuration parameters passed to Airflow through CLI

Hi, I am trying to pass the following configuration parameters to Airflow CLI while triggering a dag run. Following is the trigger_dag command I am using. airflow trigger_dag -c '{"account_list":"[1,2,3,4,5]", "start_date":"2016-04-25"}' insights_assembly_9900 My problem is that how can I ac