GitHub user joshwebtimestwo added a comment to the discussion: Event-based-Trigger
Part of this is available already in Airflow 3.0+: [event-driven scheduling](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/event-scheduling.html) uses an `AssetWatcher` to update an asset when an external event arrives, with the DAG scheduled on that asset. For GCS, an existing route is: ```text GCS OBJECT_FINALIZE notification -> Pub/Sub topic and subscription -> Airflow MessageQueueTrigger / AssetWatcher -> DAG scheduled on the asset ``` The Google provider has a [complete Pub/Sub scheduling example](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/message-queues/index.html#wait-for-a-message-in-a-queue), using `MessageQueueTrigger(scheme="google+pubsub", ...)`, `AssetWatcher`, and `schedule=[asset]`. [Cloud Storage notifications](https://docs.cloud.google.com/storage/docs/pubsub-notifications) can publish completed object creations directly to the topic, so that route doesn't require a Lambda or Cloud Function. It **still requires** the bucket notification, topic/subscription and permissions. It therefore only partly meets your proposal: an Airflow configuration field containing a bucket path would need additional provisioning or a different watcher implementation. Also, using a trigger that simply checks “this object exists” can repeatedly schedule the DAG after the first arrival; the event-scheduling docs explain this pitfall. Which Airflow version are you running, and is it Cloud Composer or self-managed? That determines whether the existing watcher approach is available in your environment. I verified the documented APIs and provider implementation, but haven't tested this against your GCP deployment. GitHub link: https://github.com/apache/airflow/discussions/75981#discussioncomment-19272821 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
