vincbeck commented on issue #48885:
URL: https://github.com/apache/airflow/issues/48885#issuecomment-2786601443

   > I chatted with [@RNHTTR](https://github.com/RNHTTR) earlier this morning. 
We thought it might make sense to leverage the `asset_events` table to avoid 
"infinite scheduling". For example, when the trigger runs, it checks for any 
files in Snowflake stage that have landed since the last Asset Event. Thoughts?
   
   Your flow would be basically like:
   
   ```
   New file in Snowflake stage -> trigger -> AssetWatcher -> Asset
   ```
   
   Your trigger would use `asset_events` to only detect files added since the 
last event.
   
   With this solution you can run into race condition issues, if the asset 
event is not inserted fast enough in the DB, you can trigger the DAG multiple 
times before you read the last asset event from the DB.
   
   Another solution would be to detect new files by time window. You could 
implement your trigger to work with time windows (e.g. 30 seconds). The trigger 
would detect files added in the last 30 seconds, and if so, it fires and wait 
for 30 seconds.
   
   Another solution would be to use a queue, whenever a new file is added, you 
send a message to a queue. Then your trigger would monitor the queue. Queues 
work perfectly with the event-driven paradigm.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to