Limitations of templating

Hi, I was going through the jinja templating in airflow and in the generic operators, it seemed like, it is limited to the bash and python operators? I would like to get a clarification on this? Best regards Niranda Perera Research Assistant Dept of CSE, University of Moratuwa niranda...@cse.mr

EMROperator templating

Hello, I would like to use EmrCreateJobFlowOperator, using job_flow_overrides attribute with dynamic attribute. Example EmrCreateJobFlowOperator(job_flow_overrides ={'foo' : '{{ xcom.pull }}'}) job_flow_overrides seems very nested to use jinja templating and template_fields is set to []. Any

Re: Limitations of templating

Hi Niranda - the best way I find is to have a quick peek at the source code. Just as an example the MsSql operator, go to the source code https://pythonhosted.org/airflow/_modules/mssql_operator.html And look for template_fields. Anything you see in the template_fields will allow for templating

Re: large number of messages from celeryev pile up in rabbitmq

I have tried using "CELERY_IGNORE_RESULT = True" but my experience with this was that while task instances would be executed the Airflow webserver would not show them as having executed successfully. I.e. I lacked the visual feedback in the webserver that tasks / dags had actually been executed suc

Passing Variables

Hi, I would like to pass a variable to my airflow dag and would like to know if there is a recommended method for doing this. I am hoping to create a dag with python operators and tasks that read data from a parquet table, perform a calculation then write the results into a new table. I'd like to

Re: Passing Variables

We use Airflow variables heavily. from airflow.models import Variable # Load an environment variable as a string ENV = Variable.get('ENV').strip() # Load an environment variable as JSON and access a JSON field named PLATFORM PLATFORM = 'EP' SSH_KEY = Variable.get('ep_platform_ssh_keys', deser

Re: Passing Variables

One DAG cannot have multiple shapes at one time, by design. You cannot parameterize things that will affect the shape of your DAG (though note that you can fully parameterize what happens within individual task instances). Think about it, a DAG is one (and only one) graph. It's NOT a shapeshifting

Re: large number of messages from celeryev pile up in rabbitmq

Interesting. The webserver doesn't use the celery broker to identify whether the task is complete, rather it takes task status information from the database. So it seems that in that case for some reason the worker wasn't updating the database. Any ways good to know that you resolved the issue a

Re: EMROperator templating

Looks like the author didn't setup any attributes as templated... https://github.com/apache/incubator-airflow/blob/ff45d8f2218a8da9328161aa66d004c3db3b367e/airflow/contrib/operators/emr_create_job_flow_operator.py#L35 You can do it easily with a simple hack: class TemplatedEmrCreateJobFlowOperato

Re: Limitations of templating

Thanks Shah for the heads up! Coming to think of it, why can't 'provide_context' be a argument in the BaseOperator? wouldn't this allow all the operators to make use of the variables? Best regards Niranda Perera Research Assistant Dept of CSE, University of Moratuwa niranda...@cse.mrt.ac.lk +94

Re: Limitations of templating

All operators' `execute` method receive a `context` dict, that is essentially the same as what you find in the main Jinja namespace. `provide_context` is specific to `PythonOperator` and its derivatives and defines whether you want the operator to unpack that context dict into your callable as kwa

Re: Passing Variables

Ah.. I completely missed the question.. in my haste to do too many things. Assuming you have a DAG named process_my_data with 3 tasks : read__from_source_table --> transform --> write_to_new_table. This dag should have a @none schedule. You could write a script to read your list of source tables

DAGs dont get refreshed ?

Greetings, We are using airflow (1.7) to manage our ETL pipeline and we are having issues related to refreshing of DAGs. When we update the DAG python script inside "dag folder" ,they don't get updated in the UI.(DAG tree as well as the Code in the UI). We have to kill and restart the "airflow we

Re: DAGs dont get refreshed ?

We actually do restart both Web and Schedulers. I know the scheduler does reparse the files in the dag folder, but the current state of the web ui does require a restart. -s On Tue, Jun 20, 2017 at 6:07 PM, Ashika Umanga Umagiliya < umanga@gmail.com> wrote: > Greetings, > > We are using airf

Re: DAGs dont get refreshed ?

To clarify, at Agari, we use monitd (like systemd) to restart both webserver and scheduler (running local executor) after deploying new dags to the dag folder. The Web UI does not discover new DAGs and it does automatically reload changes to existing files either. -s On Tue, Jun 20, 2017 at 6:34

Re: DAGs dont get refreshed ?

There is a manual DAG refresh option in the UI for DAGs the UI is already aware of -- this will reload a DAG. But that's not a complete solution to the the more general DAG refresh problem. -s On Tue, Jun 20, 2017 at 6:36 PM, siddharth anand wrote: > To clarify, at Agari, we use monitd (like sy

Re: Limitations of templating

Shah and Maxime, Thank you very much for the explanations. I went through the code in a little detail and now I understand how fields and files templates are getting resolved by the template_fields and template_ext in the BaseOperator API. Best regards Niranda Perera Research Assistant Dept of