Re: [Tutor] airflow dag

2017-05-29 Thread shubham goyal
Hello, See this is the code: from airflow import DAG from datetime import datetime,timedelta default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime.now(), 'email': ['airf...@airflow.com'], 'email_on_failure': False, 'email_on_retry': False }

Re: [Tutor] airflow dag

2017-05-28 Thread Francois Dion
My mailbox if full of similar stories: companies dumping airflow on their ETL (or similar) group. Those who knew Python succeeded, those who didn't failed, and some even moved to other companies because they couldn't cope with all this complexity dumped on them all at once. Moral of the story, it

Re: [Tutor] airflow dag

2017-05-28 Thread Alan Gauld via Tutor
On 28/05/17 04:37, shubham goyal wrote: > Does anybody have answer? You received two answers, both of which asked you to try something and get back to us for more information. Did you try printing sys.argv? What was the result? And did you try Peter's argparse code? You still haven't explained w

Re: [Tutor] airflow dag

2017-05-28 Thread Steven D'Aprano
On Sat, May 27, 2017 at 02:25:24PM +1000, Cameron Simpson wrote: > On 25May2017 18:02, Alan Gauld wrote: > >On 25/05/17 13:15, shubham goyal wrote: > >>I want to ask that can we pass the parameters as commandline arguments in > >>airflow when we are triggering the dag and access them inside the da

Re: [Tutor] airflow dag

2017-05-28 Thread Steven D'Aprano
On Sun, May 28, 2017 at 09:07:10AM +0530, shubham goyal wrote: > Does anybody have answer? Answer to what question? Peter has already answered your question about passing parameters as command line arguments. Do you have another question? -- Steve

Re: [Tutor] airflow dag

2017-05-28 Thread shubham goyal
Does anybody have answer? On May 27, 2017 1:53 PM, "Cameron Simpson" wrote: > On 25May2017 18:02, Alan Gauld wrote: > >> On 25/05/17 13:15, shubham goyal wrote: >> >>> I want to ask that can we pass the parameters as commandline arguments in >>> airflow when we are triggering the dag and access

Re: [Tutor] airflow dag

2017-05-27 Thread Cameron Simpson
On 25May2017 18:02, Alan Gauld wrote: On 25/05/17 13:15, shubham goyal wrote: I want to ask that can we pass the parameters as commandline arguments in airflow when we are triggering the dag and access them inside the dag's python script/file. I've no idea what a dag is. It's a directed acy

Re: [Tutor] airflow dag

2017-05-25 Thread Peter Otten
Alan Gauld via Tutor wrote: >>dag=dag > > I'm not sure what you think the line above does but > in normal Python it would have zero effect. The context makes it a keyword argument. dag = DAG(...) ... SimpleHttpOperator( ... dag=dag ) ___ Tut

Re: [Tutor] airflow dag

2017-05-25 Thread Peter Otten
shubham goyal wrote: > He guys, > > I want to ask that can we pass the parameters as commandline arguments in > airflow when we are triggering the dag and access them inside the dag's > python script/file. > script: > like this here i am trying to create a cluster but i need to pass password >

Re: [Tutor] airflow dag

2017-05-25 Thread Alan Gauld via Tutor
On 25/05/17 13:15, shubham goyal wrote: > He guys, > > I want to ask that can we pass the parameters as commandline arguments in > airflow when we are triggering the dag and access them inside the dag's > python script/file. I've no idea what a dag is. This list is for people learning Python as a

[Tutor] airflow dag

2017-05-25 Thread shubham goyal
He guys, I want to ask that can we pass the parameters as commandline arguments in airflow when we are triggering the dag and access them inside the dag's python script/file. script: from airflow import DAG from datetime import datetime,timedelta default_args = { 'owner': 'airflow', 'depe