GitHub user starkmarkus deleted a comment on the discussion: storage for tracking rate limits
No, not the broker in the task-execution sense. This is just storage for the rate limiter state, so it keeps short-lived counters such as request counts in the current window and when the window resets. It is not meant for metrics or Airflow metadata. On Redis vs Valkey: Valkey is a Redis fork and is generally used as a Redis-compatible drop-in replacement for common use cases. If Redis is already available, that is the safest default. If your deployment already uses Valkey, I would still test it in that exact setup, but it should usually be fine for this kind of backend. The multiple webserver/API instances part just means shared state matters if you run more than one Airflow web-facing process. In that case, a shared backend like Redis or Valkey keeps the rate limiting consistent across instances. GitHub link: https://github.com/apache/airflow/discussions/68630#discussioncomment-17329027 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
