[GitHub] [airflow] casassg commented on a change in pull request #10587: Add @dag decorator
casassg commented on a change in pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#discussion_r511031290
##
File path: docs/concepts.rst
##
@@ -162,6 +162,31 @@ Example DAG with decorated style
html_content=email_info['body']
)
+DAG decorator
+-
+
+.. versionadded:: 2.0.0
+
+In addition to creating DAGs using context managed, in Airflow 2.0 you can
also create DAGs from a function.
+DAG decorator creates a DAG generator function. This function when called
returns a DAG.
+
+DAG decorator also sets up the parameters you have in the function as DAG
params. This allows you to parametrize
+your DAGs and set the parameters when triggering the DAG manually. See
+:ref:`Passing Parameters when triggering dags ` to learn
how to pass parameters when triggering DAGs.
+
+Example DAG with decorator:
+
+.. exampleinclude:: /../airflow/example_dags/example_dag_decorator.py
+:language: python
+:dedent: 4
+:start-after: [START dag_decorator_usage]
+:end-before: [END dag_decorator_usage]
+
+.. note:: Note that Airflow will only load DAGs that appear in``globals()``.
Which means you need to make sure to have
+ a variable for your returned DAG in the module scope. Otherwise Airflow
won't detect your decorated DAG. In addition,
+ you may want to make your dag variable named ``DAG`` such that Airflow
doesn't skip this module when the
+ ``DAG_DISCOVERY_SAFE_MODE`` is activated.
Review comment:
Added to updating.md as well.
##
File path: docs/concepts.rst
##
@@ -162,6 +162,31 @@ Example DAG with decorated style
html_content=email_info['body']
)
+DAG decorator
+-
+
+.. versionadded:: 2.0.0
+
+In addition to creating DAGs using context managed, in Airflow 2.0 you can
also create DAGs from a function.
+DAG decorator creates a DAG generator function. This function when called
returns a DAG.
+
+DAG decorator also sets up the parameters you have in the function as DAG
params. This allows you to parametrize
+your DAGs and set the parameters when triggering the DAG manually. See
+:ref:`Passing Parameters when triggering dags ` to learn
how to pass parameters when triggering DAGs.
+
+Example DAG with decorator:
+
+.. exampleinclude:: /../airflow/example_dags/example_dag_decorator.py
+:language: python
+:dedent: 4
+:start-after: [START dag_decorator_usage]
+:end-before: [END dag_decorator_usage]
+
+.. note:: Note that Airflow will only load DAGs that appear in``globals()``.
Which means you need to make sure to have
+ a variable for your returned DAG in the module scope. Otherwise Airflow
won't detect your decorated DAG. In addition,
+ you may want to make your dag variable named ``DAG`` such that Airflow
doesn't skip this module when the
+ ``DAG_DISCOVERY_SAFE_MODE`` is activated.
Review comment:
Yup. let me know if that's okay
##
File path: docs/concepts.rst
##
@@ -162,6 +162,32 @@ Example DAG with decorated style
html_content=email_info['body']
)
+DAG decorator
+-
+
+.. versionadded:: 2.0.0
+
+In addition to creating DAGs using context managed, in Airflow 2.0 you can
also create DAGs from a function.
+DAG decorator creates a DAG generator function. This function when called
returns a DAG.
+
+DAG decorator also sets up the parameters you have in the function as DAG
params. This allows you to parameterize
+your DAGs and set the parameters when triggering the DAG manually. See
+:ref:`Passing Parameters when triggering dags ` to learn
how to pass parameters when triggering DAGs.
+
+You can also use the parameters on jinja templates by using the
``{{context.params}}`` dictionary.
+
+Example DAG with decorator:
+
+.. exampleinclude:: /../airflow/example_dags/example_dag_decorator.py
+:language: python
+:start-after: [START dag_decorator_usage]
+:end-before: [END dag_decorator_usage]
+
+.. note:: Note that Airflow will only load DAGs that appear in ``globals()``.
Which means you need to make sure to have
Review comment:
I dont think so, if you look above, other notes have it inlined. See:
https://raw.githubusercontent.com/apache/airflow/3d99f29fbf5da3e28bb26feb12d17ef5928b147e/docs/concepts.rst
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] casassg commented on a change in pull request #10587: Add @dag decorator
casassg commented on a change in pull request #10587: URL: https://github.com/apache/airflow/pull/10587#discussion_r506673787 ## File path: docs/concepts.rst ## @@ -162,6 +162,31 @@ Example DAG with decorated style html_content=email_info['body'] ) +DAG decorator +- + +.. versionadded:: 2.0.0 + +In addition to creating DAGs using context managed, in Airflow 2.0 you can also create DAGs from a function. +DAG decorator creates a DAG generator function. This function when called returns a DAG. + +DAG decorator also sets up the parameters you have in the function as DAG params. This allows you to parametrize +your DAGs and set the parameters when triggering the DAG manually. See +:ref:`Passing Parameters when triggering dags ` to learn how to pass parameters when triggering DAGs. + +Example DAG with decorator: + +.. exampleinclude:: /../airflow/example_dags/example_dag_decorator.py +:language: python +:dedent: 4 +:start-after: [START dag_decorator_usage] +:end-before: [END dag_decorator_usage] + +.. note:: Note that Airflow will only load DAGs that appear in``globals()``. Which means you need to make sure to have + a variable for your returned DAG in the module scope. Otherwise Airflow won't detect your decorated DAG. In addition, + you may want to make your dag variable named ``DAG`` such that Airflow doesn't skip this module when the + ``DAG_DISCOVERY_SAFE_MODE`` is activated. Review comment: Thoughts? I'm not sure how to run the system dags to make sure it runs CC @turbaszek This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] casassg commented on a change in pull request #10587: Add @dag decorator
casassg commented on a change in pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#discussion_r506649853
##
File path: docs/concepts.rst
##
@@ -162,6 +162,52 @@ Example DAG with decorated style
html_content=email_info['body']
)
+DAG decorator
+-
+
+.. versionadded:: 2.0.0
+
+In addition to creating DAGs using context managed, in Airflow 2.0 you can
also create DAGs from a function.
+DAG decorator creates a DAG generator function. This function when called
returns a DAG.
+
+DAG decorator also sets up the parameters you have in the function as DAG
params. This allows you to parametrize
+your DAGs and set the parameters when triggering the DAG manually. See
+:ref:`Passing Parameters when triggering dags ` to learn
how to pass parameters when triggering DAGs.
+
+Example DAG with decorator:
+
+.. code-block:: python
+
+ from airflow.decorators import dag, task
+
+ @dag(default_args=default_args, schedule_interval=None)
+ def send_server_ip(email: 'exam...@example.com')
+
+# Using default connection as it's set to httpbin.org by default
+get_ip = SimpleHttpOperator(
+task_id='get_ip', endpoint='get', method='GET', xcom_push=True
+)
+
+@task(multiple_outputs=True)
+def prepare_email(raw_json: str) -> Dict[str, str]:
+ external_ip = json.loads(raw_json)['origin']
+ return {
+'subject':f'Server connected from {external_ip}',
+'body': f'Seems like today your server executing Airflow is connected
from the external IP {external_ip}'
+ }
+
+email_info = prepare_email(get_ip.output)
+
+send_email = EmailOperator(
+task_id='send_email',
+to=email,
+subject=email_info['subject'],
+html_content=email_info['body']
+)
+
+ my_dag = send_server_ip()
Review comment:
Fine by me. I will add as a todo. Prioritizing adding unit tests for now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] casassg commented on a change in pull request #10587: Add @dag decorator
casassg commented on a change in pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#discussion_r506649427
##
File path: airflow/models/dag.py
##
@@ -2187,6 +2199,33 @@ def calculate_dagrun_date_fields(
log.info("Setting next_dagrun for %s to %s", dag.dag_id,
self.next_dagrun)
+def dag(*dag_args, **dag_kwargs):
+"""
+Python dag decorator. Wraps a function into an Airflow DAG.
+Accepts kwargs for operator kwarg. Can be used to parametrize DAGs.
+
+:param dag_args: Arguments for DAG object
+:type dag_args: list
+:param dag_kwargs: Kwargs for DAG object.
+:type dag_kwargs: dict
+"""
+def wrapper(f: Callable):
+dag_sig = signature(DAG.__init__)
+dag_sig = dag_sig.bind_partial(*dag_args, **dag_kwargs)
+
+@functools.wraps(f)
+def factory(*args, **kwargs):
+f_sig = signature(f).bind(*args, **kwargs)
+f_sig.apply_defaults()
+with DAG(*dag_sig.args, dag_id=f.__name__, **dag_sig.kwargs) as
dag_obj:
+f_kwargs = {}
+for name, value in f_sig.arguments.items():
+f_kwargs[name] = dag_obj.param(name, value)
+f(**f_kwargs)
+return dag_obj
Review comment:
Lmk if this is enough. Adding more unit tests as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
