Re: [PR] Impove dynamic DAG generation docs [airflow]
Lee-W closed pull request #50441: Impove dynamic DAG generation docs URL: https://github.com/apache/airflow/pull/50441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
potiuk commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083508516 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation Review Comment: `Optimizing` is just above (line 161) I think it's fine to leave ll that here - this is just **one** of the ways you can do dynamic dag generation - I think if anything - the whole document can be linked to from best practices - just a short paragraph titled dynamic dag generation and link to the "dynamic dag generation.rst". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
sanederchik commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083498262 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation Review Comment: Could you please give a link to the Optimizing chapter? I'm sorry I couldn't find it... Also I just got an idea - maybe it'd be better to move all of my commit to Best practice chapter? Because it looks like my examples with your proposals are more about clean code and DRY, not about specifically dynamic DAGs generation. And here, in Dynamic Dag Generation chapter, I could leave an example with generating dynamic DAG in a way I propose with the link to Best practice. Anyway I'd be glad to make it as you consider the best for users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
sanederchik commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083498262 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation Review Comment: Could you please give a link to the Optimizing chapter? I'm sorry I couldn't find it... Also I just got an idea - maybe it'd be better to move all of my commit to Best practice chapter? Because it looks like my examples with your proposals are more about clean code and DRY, not about specifically dinamic DAGs generation. And here, in Dynamic Dag Generation chapter, I could leave an example with generating dynamic DAG in a way I propose with the link to Best practice. Anyway I'd be glad to make it as you consider the best for users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
potiuk commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083254466 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation Review Comment: I think it would be nice to move it before the "Optimizing" chapter - Optimizing is really a bit more of applying to all other cases where dynamic dags are generated :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
sanederchik commented on code in PR #50441:
URL: https://github.com/apache/airflow/pull/50441#discussion_r2083264235
##
airflow-core/docs/howto/dynamic-dag-generation.rst:
##
@@ -218,3 +217,74 @@ of the context are set to ``None``.
with DAG(dag_id=dag_id, ...):
...
+
+
+Abstraction of DAG objects generation
+.
+
+Assume you want to create a *DAG template* which you can use for:
+
+ - Hiding the logics of your DAG from a viewer
+ - Dynamic DAG generation without a need for writing main logics of a DAG
inside cycles
+ - Convenient testing.
+
+Then you can create a special function which takes a `DAG` object as an
argument
+and returns the same `DAG` object as the result.
+
+For example, assume you store the following `DAG` generating function inside
your ``reusables/reusable_dag.py`` module file:
+
+.. code-block:: python
+
+ import datetime
+
+ from airflow.decorators import task
+ from airflow import DAG
+
+ DEFAULT_DAG_CONFIG = {
+ 'schedule': None,
+ 'catchup': False,
+ 'start_date': datetime.datetime.fromisoformat('2025-04-29'),
+ 'default_args': {
+ 'depends_on_past': False
+ },
+ 'max_active_tasks': 1,
+ 'max_active_runs': 1
+ }
+
+ def reusable_dag_generator(
+ dag_obj: DAG,
+ printable_msg: str
+ ):
+ """Reusable dag generator used as a template for DAG generation"""
+
+ with dag_obj:
+
+ @task()
+ def start_dag():
+ pass
+
+ @task()
+ def print_msg():
+ print(printable_msg)
+
+ @task()
+ def finish_dag():
+ pass
+
+ start_dag() >> print_msg() >> finish_dag()
+ return dag_obj
+
+Then you can import a ``reusable_dag_generator`` function in your DAG file and
use it:
Review Comment:
Sure, I'll make a dynamic DAG generation example as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
sanederchik commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083264139 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation +. + +Assume you want to create a *DAG template* which you can use for: + + - Hiding the logics of your DAG from a viewer + - Dynamic DAG generation without a need for writing main logics of a DAG inside cycles + - Convenient testing. Review Comment: Sure, I just thought it would make an example look too difficult for people. I will write a `test_requsable_dag` example. ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation +. + +Assume you want to create a *DAG template* which you can use for: + + - Hiding the logics of your DAG from a viewer + - Dynamic DAG generation without a need for writing main logics of a DAG inside cycles + - Convenient testing. Review Comment: Sure, I just thought it would make an example look too difficult for users. I will write a `test_requsable_dag` example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
sanederchik commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083263890 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation +. + +Assume you want to create a *DAG template* which you can use for: + + - Hiding the logics of your DAG from a viewer + - Dynamic DAG generation without a need for writing main logics of a DAG inside cycles Review Comment: Sure, my fault, I'll fix it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
potiuk commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083254519 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation +. + +Assume you want to create a *DAG template* which you can use for: + + - Hiding the logics of your DAG from a viewer + - Dynamic DAG generation without a need for writing main logics of a DAG inside cycles Review Comment: ```suggestion - Dynamic DAG generation without a need for writing main logics of a DAG inside loops ``` Maybe? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
potiuk commented on code in PR #50441:
URL: https://github.com/apache/airflow/pull/50441#discussion_r2083254350
##
airflow-core/docs/howto/dynamic-dag-generation.rst:
##
@@ -218,3 +217,74 @@ of the context are set to ``None``.
with DAG(dag_id=dag_id, ...):
...
+
+
+Abstraction of DAG objects generation
+.
+
+Assume you want to create a *DAG template* which you can use for:
+
+ - Hiding the logics of your DAG from a viewer
+ - Dynamic DAG generation without a need for writing main logics of a DAG
inside cycles
+ - Convenient testing.
+
+Then you can create a special function which takes a `DAG` object as an
argument
+and returns the same `DAG` object as the result.
+
+For example, assume you store the following `DAG` generating function inside
your ``reusables/reusable_dag.py`` module file:
+
+.. code-block:: python
+
+ import datetime
+
+ from airflow.decorators import task
+ from airflow import DAG
+
+ DEFAULT_DAG_CONFIG = {
+ 'schedule': None,
+ 'catchup': False,
+ 'start_date': datetime.datetime.fromisoformat('2025-04-29'),
+ 'default_args': {
+ 'depends_on_past': False
+ },
+ 'max_active_tasks': 1,
+ 'max_active_runs': 1
+ }
+
+ def reusable_dag_generator(
+ dag_obj: DAG,
+ printable_msg: str
+ ):
+ """Reusable dag generator used as a template for DAG generation"""
+
+ with dag_obj:
+
+ @task()
+ def start_dag():
+ pass
+
+ @task()
+ def print_msg():
+ print(printable_msg)
+
+ @task()
+ def finish_dag():
+ pass
+
+ start_dag() >> print_msg() >> finish_dag()
+ return dag_obj
+
+Then you can import a ``reusable_dag_generator`` function in your DAG file and
use it:
Review Comment:
Maybe worth mentioning that you can do it multiple times in different dag
files or maybe even good idea to show how you can create two similar dags in
one file here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
potiuk commented on code in PR #50441: URL: https://github.com/apache/airflow/pull/50441#discussion_r2083254263 ## airflow-core/docs/howto/dynamic-dag-generation.rst: ## @@ -218,3 +217,74 @@ of the context are set to ``None``. with DAG(dag_id=dag_id, ...): ... + + +Abstraction of DAG objects generation +. + +Assume you want to create a *DAG template* which you can use for: + + - Hiding the logics of your DAG from a viewer + - Dynamic DAG generation without a need for writing main logics of a DAG inside cycles + - Convenient testing. Review Comment: I think it would be nice to show example of how you can achieve convenient testing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Impove dynamic DAG generation docs [airflow]
boring-cyborg[bot] commented on PR #50441: URL: https://github.com/apache/airflow/pull/50441#issuecomment-2869042016 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points: - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices). - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
