Hi all,
I’d like to propose the addition of a new provider to support Valkey<https://valkey.io/>: a fully open-source, community-governed fork of Redis 7.2, now maintained under the Linux Foundation. Valkey has been gaining traction recently, in part due to changes by Redis moving their license to one unrecognized by the Open Source Initiative. Valkey represents a future-proof alternative. The proposed provider would closely mirror the existing Redis provider in functionality and serve as a drop-in, OSS-aligned option for workflows that depend on key-value storage, pub/sub, or caching patterns. Tentatively, here are the components I plan on implementing (mirroring the Redis provider): * ValkeyHook – Manage connections and interactions with the Valkey server, enabling reuse across operators and sensors. * ValkeyPublishOperator – Allows tasks to publish messages to Valkey pub/sub channels, useful for event-driven workflows or inter-task signaling. * ValkeyKeySensor – Waits for a specific key to exist or reach a value, supporting synchronization patterns where one task's output controls downstream logic. * ValkeyPubSubSensor – Subscribes to a Valkey channel and triggers on incoming messages, enabling real-time DAG triggering. Here's an example of how we could use the components to publish a message and then trigger a DAG: ` publish_message = ValkeyPublishOperator( task_id="publish_ready_signal", channel="processing_status", message="READY", valkey_conn_id="valkey_default", ) ` This task publishes the message "READY" to the processing_status channel on a Valkey server. Other processes listening to this channel will receive the message and proceed with their execution. ` wait_for_ready = ValkeyPubSubSensor( task_id="wait_for_ready_message", channel="processing_status", valkey_conn_id="valkey_default", timeout=600, ) ` Here, a sensor waits for the message on the same channel before continuing. Once the message is received, the DAG can continue with further tasks—such as triggering another DAG—in an event-driven manner. Given the community's interest in event-driven scheduling (AIP 82<https://github.com/apache/airflow/issues/52712>) as well as concerns about limited provider optionality (PR #52712<https://github.com/apache/airflow/issues/52712>), a Valkey provider would expand user choice. I’d be happy to begin working on the initial PR and documentation. Looking forward to feedback. Sean
