GitHub user AI-DATA-ENGINEER-YASH created a discussion: Event-based-Trigger
I am from a Fintech, Banking sector and there is scarcity of event based trigger in our airflow on GCP. Current Pain Points - High Latency: Critical data sits idle waiting for the next scheduled execution window. Wasted Compute & Resources: Pipelines run on fixed schedules even when no new source data has arrived. Friction with External Workarounds: Reacting instantly to external events (such as a file landing in Cloud Storage, Pub/Sub messages, or webhooks) currently requires auxiliary services like AWS Lambda, Pub/Sub, or complex dataset configurations. These solutions introduce heavy operational overhead, slow setup times, and lack an intuitive, native user experience. Expected Solution - We propose introducing an inbuilt, native event-based trigger functionality directly within Airflow—designed to be as straightforward, reliable, and seamless as our existing schedule-based setup. Specifically, we want the ability to: Configure an event trigger (such as specifying a GCS bucket path) directly inside Airflow configuration. Automatically kick off DAG executions upon file arrival without maintaining external event-routing infrastructure. GitHub link: https://github.com/apache/airflow/discussions/75981 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
