cmarteepants commented on issue #48885:
URL: https://github.com/apache/airflow/issues/48885#issuecomment-2786635018
Agree re queue working perfectly with event-driven paradigm, but I can
appreciate the idea of building up a catalogue of Watchers in Airflow that can
be used without extra infrastructure.
>Your trigger would use asset_events to only detect files added since the
last event.
>With this solution you can run into race condition issues, if the asset
event is not inserted fast enough in the DB, you can trigger the DAG multiple
times before you read the last asset event from the DB.
Ah yeah, that's a problem. I think time windows can also be problematic
though? Like, if Airflow goes down / is unresponsive and watcher is only
looking at `t0 - config` (ex: 30s), what would happen to files that were
dropped during the gap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]