[GitHub] [airflow] jedcunningham commented on issue #17343: Dags added by DagBag interrupt randomly
jedcunningham commented on issue #17343:
URL: https://github.com/apache/airflow/issues/17343#issuecomment-900681488
If it were me, I'd use `TriggerDagRunOperator` directly and put a
`ShortCircuirOperator` in front of it to check your `random_condition`,
something like:
```
check = ShortCircuitOperator(
task_id="check",
python_callable=lamda: random_condition,
)
trigger = TriggerDagRunOperator(
task_id="trigger",
trigger_dag_id=dag_name,
conf={"group_nr": next_group_nr, "group": next_group},
)
check >> trigger
```
Generally, I'd say using `.execute()` on an operator directly is an
anti-pattern.
Also, as mentioned in #17344 Airflow really does expect a single DAG folder.
Finding a simpler way to do multi-tenancy would probably be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17343: Dags added by DagBag interrupt randomly
jedcunningham commented on issue #17343:
URL: https://github.com/apache/airflow/issues/17343#issuecomment-900681488
If it were me, I'd use `TriggerDagRunOperator` directly and put a
`ShortCircuirOperator` in front of it to check your `random_condition`,
something like:
```
check = ShortCircuitOperator(
task_id="check",
python_callable=lamda: random_condition,
)
trigger = TriggerDagRunOperator(
task_id="trigger",
trigger_dag_id=dag_name,
conf={"group_nr": next_group_nr, "group": next_group},
)
check >> trigger
```
Generally, I'd say using `.execute()` on an operator directly is an
anti-pattern.
Also, as mentioned in #17344 Airflow really does expect a single DAG folder.
Finding a simpler way to do multi-tenancy would probably be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17343: Dags added by DagBag interrupt randomly
jedcunningham commented on issue #17343:
URL: https://github.com/apache/airflow/issues/17343#issuecomment-890215689
Try removing the `trigger.execute({})` line. This is causing the scheduler
to trigger the DAG every time the DAG is parsed (which can be roughly every
30s), probably not what you intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
