jroachgolf84 commented on issue #48885:
URL: https://github.com/apache/airflow/issues/48885#issuecomment-2786807188

   I believe that this object/file storage pattern will be one of the more 
common use-cases for `AssetWatcher`'s, so it's something I'm inclined to dig a 
bit deeper into. IMO, there are a couple of other "gotchas" to keep in mind:
   
   * In the Snowflake Stage example here (and others), do I want to process 
each file individually, or files in a batch since the object store was last 
"polled"?
   * How can we avoid the race condition mentioned above, which is going to be 
prevalent with just about any object/file-based approach?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to