GitHub user starkmarkus edited a comment on the discussion: storage for tracking rate limits
It is not a broker in the task-execution sense. This is just storage for the rate limiter state, so the backend keeps short-lived counters such as request counts in the current window and when the window resets. It is not meant for metrics or Airflow metadata. For Valkey vs Redis: Valkey is broadly Redis-compatible. If Redis is already available, that is the safest default. If your deployment already uses Valkey and the limiter backend supports it, that should usually work too. The “multiple webserver/API instances” part just means shared state matters if you run more than one Airflow web-facing process. In that case, a shared backend like Redis or Valkey keeps the rate limiting consistent across instances. GitHub link: https://github.com/apache/airflow/discussions/68630#discussioncomment-17325734 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
