andreyvital commented on issue #39717: URL: https://github.com/apache/airflow/issues/39717#issuecomment-2133534216
Hi @nghilethanh-atherlabs I've been experimenting with those configs as well: ``` # https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#task-acks-late # https://github.com/apache/airflow/issues/16163#issuecomment-1563704852 task_acks_late = False # https://github.com/apache/airflow/blob/2b6f8ffc69b5f34a1c4ab7463418b91becc61957/airflow/providers/celery/executors/default_celery.py#L52 # https://github.com/celery/celery/discussions/7276#discussioncomment-8720263 # https://github.com/celery/celery/issues/4627#issuecomment-396907957 [celery_broker_transport_options] visibility_timeout = 300 max_retries = 120 interval_start = 0 interval_step = 0.2 interval_max = 0.5 # sentinel_kwargs = {} ``` For the redis persistency, you can refer to their config file to enable persistency. Not sure it will sort out. But let's keep trying folks. ``` # redis.conf bind 0.0.0.0 protected-mode no requirepass REDACTED maxmemory 6gb # https://redis.io/docs/manual/eviction/ maxmemory-policy noeviction port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no supervised no pidfile /var/run/redis.pid loglevel notice logfile "" databases 16 always-show-logo no save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /bitnami/redis/data appendonly no appendfilename "appendonly.aof" appendfsync everysec # appendfsync no no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble no aof-rewrite-incremental-fsync yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 ``` ```YML # docker-compose.yml redis: image: bitnami/redis:7.2.5 container_name: redis environment: - REDIS_DISABLE_COMMANDS=CONFIG # The password will come from the config file, but we need to bypass the validation - ALLOW_EMPTY_PASSWORD=yes ports: - 6379:6379 # command: /opt/bitnami/scripts/redis/run.sh --maxmemory 2gb command: /opt/bitnami/scripts/redis/run.sh volumes: - ./redis/redis.conf:/opt/bitnami/redis/mounted-etc/redis.conf - redis:/bitnami/redis/data restart: always healthcheck: test: - CMD - redis-cli - ping interval: 5s timeout: 30s retries: 10 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
