Yes. Valkey is a good replacement/parallel implementation to Redis. One
thing however is important - I would see this even more useful if it also
allows for the Celery Redis replacement for broker - means that it is fully
supported in our:
* unit tests (of course)
* integration tests with ("valkey") integration - including running
docker-compose integration to run in CI
* Helm Chart support to run valley as Celery Broker instead of Redis
I think I'd only consider valley as "successful" provider integration if
all that above is completed.
J.
On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean <gha...@amazon.com.invalid>
wrote:
> Hi all,
>
>
>
> I’d like to propose the addition of a new provider to support Valkey<
> https://valkey.io/>: a fully open-source, community-governed fork of
> Redis 7.2, now maintained under the Linux Foundation.
>
>
>
> Valkey has been gaining traction recently, in part due to changes by Redis
> moving their license to one unrecognized by the Open Source Initiative.
> Valkey represents a future-proof alternative. The proposed provider would
> closely mirror the existing Redis provider in functionality and serve as a
> drop-in, OSS-aligned option for workflows that depend on key-value storage,
> pub/sub, or caching patterns.
>
>
>
> Tentatively, here are the components I plan on implementing (mirroring the
> Redis provider):
>
>
>
> * ValkeyHook – Manage connections and interactions with the Valkey
> server, enabling reuse across operators and sensors.
> * ValkeyPublishOperator – Allows tasks to publish messages to Valkey
> pub/sub channels, useful for event-driven workflows or inter-task signaling.
> * ValkeyKeySensor – Waits for a specific key to exist or reach a
> value, supporting synchronization patterns where one task's output controls
> downstream logic.
> * ValkeyPubSubSensor – Subscribes to a Valkey channel and triggers on
> incoming messages, enabling real-time DAG triggering.
>
>
>
> Here's an example of how we could use the components to publish a message
> and then trigger a DAG:
>
>
>
> `
>
> publish_message = ValkeyPublishOperator(
>
> task_id="publish_ready_signal",
>
> channel="processing_status",
>
> message="READY",
>
> valkey_conn_id="valkey_default",
>
> )
>
> `
>
>
>
> This task publishes the message "READY" to the processing_status channel
> on a Valkey server. Other processes listening to this channel will receive
> the message and proceed with their execution.
>
>
>
> `
>
> wait_for_ready = ValkeyPubSubSensor(
>
> task_id="wait_for_ready_message",
>
> channel="processing_status",
>
> valkey_conn_id="valkey_default",
>
> timeout=600,
>
> )
>
> `
>
>
>
> Here, a sensor waits for the message on the same channel before
> continuing. Once the message is received, the DAG can continue with further
> tasks—such as triggering another DAG—in an event-driven manner.
>
>
>
> Given the community's interest in event-driven scheduling (AIP 82<
> https://github.com/apache/airflow/issues/52712>) as well as concerns
> about limited provider optionality (PR #52712<
> https://github.com/apache/airflow/issues/52712>), a Valkey provider would
> expand user choice.
>
>
>
> I’d be happy to begin working on the initial PR and documentation.
>
>
>
> Looking forward to feedback.
>
> Sean
>
>
