Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
github-actions[bot] closed issue #49857: Event Driven Dags can miss/delay runs when paused URL: https://github.com/apache/airflow/issues/49857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
github-actions[bot] commented on issue #49857: URL: https://github.com/apache/airflow/issues/49857#issuecomment-2892558753 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
github-actions[bot] commented on issue #49857: URL: https://github.com/apache/airflow/issues/49857#issuecomment-2874628475 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
darkag commented on issue #49857: URL: https://github.com/apache/airflow/issues/49857#issuecomment-2836423671 thanks for your response, I've managed to understand what happens when file is delete but no asset_event is created when pausing the dag. In fact, it seems that the asset_trigger link is deleted between the moment trigger yield event and the moment register_asset_change is called due to dag paused. It may not be considered as a bug since event driven dag should not be paused, but it's limit their usage to handle case when you have the possibility to pause the initial event source or if events doesn't have significant payload and you can wait for the next event/create a dummy event from UI after unpausing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
vincbeck commented on issue #49857: URL: https://github.com/apache/airflow/issues/49857#issuecomment-2834557250 I confirm everything that has been said here. `example_asset_with_watchers` might not be the best example and I might replace with a real world scenario. A paused DAG will never be executed if it receives an event (because it is paused). If you want to emit event -> store event -> process event then you might want to use a queue. It needs to be configured outside of Airflow. You configure you env to be send to a queue of choice and then you, in Airflow, your configure your DAG to be executed whenever a new message lands in the queue. Today only SQS is available (with `SqsMessageQueueProvider`) as queue provider to be compatible with event driven scheduling but other providers should be soon available. > Other question, during my tests I guess that one time between the time that dags was paused and the trigger stop whatching for file, the /tmp/test was deleted but no asset event was generated, but since I was not able to find which part of code manage the return of trigger and create the asset event I was not able to push more investigation. Could you tell me where I can find these code ? https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/assets/manager.py#L109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
darkag commented on issue #49857: URL: https://github.com/apache/airflow/issues/49857#issuecomment-2834445450 thanks for your reponses, if I understand event driven dags should never be paused and if I need to prevent it to run (ex in case of maintenance of a database) I need to block what causes trigger to return event (in case of example_asset_with_watchers I need to stop what create /tmp/test file) ? Other question, during my tests I guess that one time between the time that dags was paused and the trigger stop whatching for file, the /tmp/test was deleted but no asset event was generated, but since I was not able to find which part of code manage the return of trigger and create the asset event I was not able to push more investigation. Could you tell me where I can find these code ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Event Driven Dags can miss/delay runs when paused [airflow]
vatsrahul1001 commented on issue #49857: URL: https://github.com/apache/airflow/issues/49857#issuecomment-2834072594 Yeah DAG needs to be active. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
