vincbeck commented on issue #48885: URL: https://github.com/apache/airflow/issues/48885#issuecomment-2786644614
> Agree re queue working perfectly with event-driven paradigm, but I can appreciate the idea of building up a catalogue of Watchers in Airflow that can be used without extra infrastructure. > > > Your trigger would use asset_events to only detect files added since the last event. > > > With this solution you can run into race condition issues, if the asset event is not inserted fast enough in the DB, you can trigger the DAG multiple times before you read the last asset event from the DB. > > Ah yeah, that's a problem. I think time windows can also be problematic though? Like, if Airflow goes down / is unresponsive and watcher is only looking at `t0 - config` (ex: 30s), what would happen to files that were dropped during the gap? Yup, that's another issue, they would be ignored. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
