Re: programmatically creating and airflow quirks

2018-11-22 Thread Alex Guziel
Yup. On Thu, Nov 22, 2018 at 3:16 PM soma dhavala wrote: > > > On Nov 23, 2018, at 3:28 AM, Alex Guziel wrote: > > It’s because of this > > “When searching for DAGs, Airflow will only consider files where the > string “airflow” and “DAG” both appear in the contents of the .py file.” > > > Have

Re: programmatically creating and airflow quirks

2018-11-22 Thread soma dhavala
> On Nov 23, 2018, at 3:28 AM, Alex Guziel wrote: > > It’s because of this > > “When searching for DAGs, Airflow will only consider files where the string > “airflow” and “DAG” both appear in the contents of the .py file.” > Have not noticed it. From airflow/models.py, in process_file — (

Re: programmatically creating and airflow quirks

2018-11-22 Thread Alex Guziel
It’s because of this “When searching for DAGs, Airflow will only consider files where the string “airflow” and “DAG” both appear in the contents of the .py file.” On Thu, Nov 22, 2018 at 2:27 AM soma dhavala wrote: > > > On Nov 22, 2018, at 3:37 PM, Alex Guziel wrote: > > I think this is what

Request for review PR

2018-11-22 Thread airflowuser
Can one of the committers please review this one ? https://github.com/apache/incubator-airflow/pull/2709

Re: programmatically creating and airflow quirks

2018-11-22 Thread soma dhavala
> On Nov 22, 2018, at 3:37 PM, Alex Guziel wrote: > > I think this is what is going on. The dags are picked by local variables. > I.E. if you do > dag = Dag(...) > dag = Dag(…) from my_module import create_dag for file in yaml_files: dag = create_dag(file) globals()[dag.dag_i

Re: programmatically creating and airflow quirks

2018-11-22 Thread Alex Guziel
I think this is what is going on. The dags are picked by local variables. I.E. if you do dag = Dag(...) dag = Dag(...) Only the second dag will be picked up. On Thu, Nov 22, 2018 at 2:04 AM Soma S Dhavala wrote: > Hey AirFlow Devs: > In our organization, we build a Machine Learning WorkBench wi

programmatically creating and airflow quirks

2018-11-22 Thread Soma S Dhavala
Hey AirFlow Devs: In our organization, we build a Machine Learning WorkBench with AirFlow as an orchestrator of the ML Work Flows, and have wrapped AirFlow python operators to customize the behaviour. These work flows are specified in YAML. We drop a DAG loader (written python) in the default loca

Re: Tip for Running Unit Testing against Your Own Fork

2018-11-22 Thread Naik Kaxil
Let's add this info to https://github.com/apache/incubator-airflow/blob/master/dev/README.md . Can someone send a PR ?? On 22/11/2018, 09:52, "Jarek Potiuk" wrote: Seems that the dev list does not like attachments. The screenshot showing the right setting is here: https://dr

Re: Tip for Running Unit Testing against Your Own Fork

2018-11-22 Thread Jarek Potiuk
Seems that the dev list does not like attachments. The screenshot showing the right setting is here: https://drive.google.com/open?id=1-6_Fec0vLiKqEamCoY_xOh4Wb8tV-R37 J. On Thu, Nov 22, 2018 at 10:49 AM Sai Phanindhra wrote: > Some admin has to check. I am not sure what options they will be

Re: Tip for Running Unit Testing against Your Own Fork

2018-11-22 Thread Sai Phanindhra
Some admin has to check. I am not sure what options they will be seeing. They should have option to toggle auto cancel builds just like i had. On Thu, 22 Nov 2018 at 14:21, Jarek Potiuk wrote: > There is - "auto cancellation" (see the screenshot). So maybe it's just a > matter of flipping the to

Re: Tip for Running Unit Testing against Your Own Fork

2018-11-22 Thread Jarek Potiuk
There is - "auto cancellation" (see the screenshot). So maybe it's just a matter of flipping the toggle by the admins? [image: Screenshot 2018-11-22 at 09.49.27 (1).png] J. On Wed, Nov 21, 2018 at 6:19 PM Sai Phanindhra wrote: > Hi, > I am not sure about that, there has to be a provision to