Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
Michael-cd30 commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2960270703 > > For example, each "transform" DAG will wait for the "extract_load" DAGs to finish. > > [@Michael-cd30](https://github.com/Michael-cd30), this is a really good use-case for Assets. I'd give those a look! > > https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html#schedule-dags-with-assets Assets won't help here as we have several “extract_load” DAGs and several “transform” DAGs without them knowing each other. So we can add new DAGs without having to change the logic and just by tagging the DAGs correctly. Here's how it works. When a “transform” DAG launches, it executes a first sensor that says “Hey guys, is anyone extracting/loading here?" If yes, then the transform DAG says “OK, no problem, I'll reschedule in 10 minutes”. Otherwise, it actually launches the transformations. On the other hand, when a DAG “extract_load” is launched, it also executes a first sensor that says “Hey guys, is there anyone out there transforming?”. If so, then the “extract_load” DAG says “Ok, in that case I'll stop wisely (soft_fail)". In this way, transformations are only executed once all extractions/loads have been properly completed. This guarantees data consistency. But it also guarantees that when a transformation starts, it's no longer possible for an “extract_load” to start. This ensures that there is no starvation of transformation processes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
kaxil commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2960082561 > The need is to know whether DAGs with specific tags are running or not. > > We use this information to synchronize the DAGs. > > For example, each "transform" DAG will wait for the "extract_load" DAGs to finish. > > Therfore, I don't think the task_context will help. > Yeah, for that use-case Task Context won't help. The option then is using [official REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) or [Python Client](https://github.com/apache/airflow-client-python) instead. > I think that if this isn't a bug, the documentation should at least be updated. Yup, it is WIP -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
jroachgolf84 commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2960116814 > For example, each "transform" DAG will wait for the "extract_load" DAGs to finish. @Michael-cd30, this is a really good use-case for Assets. I'd give those a look! https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html#schedule-dags-with-assets -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
kaxil commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2960101728 Created https://github.com/apache/airflow/issues/51590 to track the doc update -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
Michael-cd30 commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2960065613 The need is to know whether DAGs with specific tags are running or not. We use this information to synchronize the DAGs. For example, each "transform" DAG will wait for the "extract_load" DAGs to finish. Therfore, I don't think the task_context will help. I think that if this isn't a bug, the documentation should at least be updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
kaxil commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2960010078 Depending on what you want to do, you can also use Task Context: ```python from airflow.sdk import dag, get_current_context, task from airflow.utils.state import DagRunState from datetime import datetime @dag(dag_id = "test_airflow3", start_date = datetime(2025, 1, 1), schedule = "@hourly", tags = ["misc"], catchup = False) def mon_dag(): @task(task_id = "appel_dagrun_find") def ma_premiere_tache(): context = get_current_context() ti = context["ti"] dag_run = context["dag_run"] ma_premiere_tache() mon_dag() ``` and then other options: https://github.com/apache/airflow/blob/bf0bfe9dc1f3812989acfe6ed7118acf9ba5b586/task-sdk/src/airflow/sdk/types.py#L100-L118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
kaxil closed issue #51563: DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error URL: https://github.com/apache/airflow/issues/51563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
kaxil commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2959986408 This is not an issue, it is by design that in Airflow 3.0+ you cannot use Airflow metadata DB in your task. You should only use anything in `airflow.sdk` namespace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
kevinhongzl commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2959762254 @jroachgolf84 I'd like to help with this issue :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
jroachgolf84 commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2959742908 I was able to reproduce this exception using the logic provided above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] DagRun.find() throws an "Direct database access via the ORM is not allowed in Airflow 3.0" error [airflow]
boring-cyborg[bot] commented on issue #51563: URL: https://github.com/apache/airflow/issues/51563#issuecomment-2958014868 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
