Can you please explain why it should be a provider managed by Airflow rather than on your own? Redis provider is not very popular as normally redis isn't that involved with batch processing. Why should Valkey be different?
Please also clarify if Amazon are committing to the maintenance of the provider including bug fixes, issues triage and any other issue that comes up with upstream library. On Tue, Jul 15, 2025 at 10:03 PM Ghaeli, Sean <gha...@amazon.com.invalid> wrote: > Hi all, > > > > I’d like to propose the addition of a new provider to support Valkey< > https://valkey.io/>: a fully open-source, community-governed fork of > Redis 7.2, now maintained under the Linux Foundation. > > > > Valkey has been gaining traction recently, in part due to changes by Redis > moving their license to one unrecognized by the Open Source Initiative. > Valkey represents a future-proof alternative. The proposed provider would > closely mirror the existing Redis provider in functionality and serve as a > drop-in, OSS-aligned option for workflows that depend on key-value storage, > pub/sub, or caching patterns. > > > > Tentatively, here are the components I plan on implementing (mirroring the > Redis provider): > > > > * ValkeyHook – Manage connections and interactions with the Valkey > server, enabling reuse across operators and sensors. > * ValkeyPublishOperator – Allows tasks to publish messages to Valkey > pub/sub channels, useful for event-driven workflows or inter-task signaling. > * ValkeyKeySensor – Waits for a specific key to exist or reach a > value, supporting synchronization patterns where one task's output controls > downstream logic. > * ValkeyPubSubSensor – Subscribes to a Valkey channel and triggers on > incoming messages, enabling real-time DAG triggering. > > > > Here's an example of how we could use the components to publish a message > and then trigger a DAG: > > > > ` > > publish_message = ValkeyPublishOperator( > > task_id="publish_ready_signal", > > channel="processing_status", > > message="READY", > > valkey_conn_id="valkey_default", > > ) > > ` > > > > This task publishes the message "READY" to the processing_status channel > on a Valkey server. Other processes listening to this channel will receive > the message and proceed with their execution. > > > > ` > > wait_for_ready = ValkeyPubSubSensor( > > task_id="wait_for_ready_message", > > channel="processing_status", > > valkey_conn_id="valkey_default", > > timeout=600, > > ) > > ` > > > > Here, a sensor waits for the message on the same channel before > continuing. Once the message is received, the DAG can continue with further > tasks—such as triggering another DAG—in an event-driven manner. > > > > Given the community's interest in event-driven scheduling (AIP 82< > https://github.com/apache/airflow/issues/52712>) as well as concerns > about limited provider optionality (PR #52712< > https://github.com/apache/airflow/issues/52712>), a Valkey provider would > expand user choice. > > > > I’d be happy to begin working on the initial PR and documentation. > > > > Looking forward to feedback. > > Sean > >
