turbaszek edited a comment on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-617194692
Duplicate of https://github.com/apache/airflow/issues/8044 @jsmodic you are right, autoscaling is weird in Celery in 4.x and there are issues reporting this problem in Celery itself. If you want to perform production-grade autoscaling and you are using k8s cluster this may be interesting to you: https://www.astronomer.io/blog/the-keda-autoscaler/ I tested it on master and it seems to be working fine (`-c n` spawns main celery process and `n` child processes). Same on 1.10.10: ``` root 412 48.3 2.0 660108 123128 pts/0 Sl+ 14:07 0:02 [celeryd: celery@3715dff2587f:MainProcess] -active- (worker -c 2) root 422 50.5 1.7 374028 107824 pts/0 S+ 14:07 0:02 /usr/local/bin/python /usr/local/bin/airflow serve_logs root 426 0.0 1.5 383524 92124 pts/0 S+ 14:07 0:00 [celeryd: celery@3715dff2587f:ForkPoolWorker-1] root 427 0.0 1.5 383528 92124 pts/0 S+ 14:07 0:00 [celeryd: celery@3715dff2587f:ForkPoolWorker-2] ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
