cmarteepants commented on issue #48885:
URL: https://github.com/apache/airflow/issues/48885#issuecomment-2786635018

   Agree re queue working perfectly with event-driven paradigm, but I can 
appreciate the idea of building up a catalogue of Watchers in Airflow that can 
be used without extra infrastructure. 
   
   >Your trigger would use asset_events to only detect files added since the 
last event.
    
   >With this solution you can run into race condition issues, if the asset 
event is not inserted fast enough in the DB, you can trigger the DAG multiple 
times before you read the last asset event from the DB.
   
   Ah yeah, that's a problem. I think time windows can also be problematic 
though? Like, if Airflow goes down / is unresponsive and watcher is only 
looking at `t0 - config` (ex: 30s), what would happen to files that were 
dropped during the gap?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to