potiuk commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1914580294
And just to add a bit more context to the discussion. It's a known fact that a number of our use have some troubles with stability running Celery-based deployment. I think if we know a way to make it more stable, keeping it as "secret-sauce" makes little sense. Also some people proposed other solutions to improve stability (which might or might not be related and they are way more invasive) and possibly the stability issues might be helped with better default configuration (in docs and in chart). I am particualarly referring to that one https://github.com/apache/airflow/pull/31829 which proposes to disable the "ack late" feature to improve stability of Airflow Celery deployment, which comes from the creator of the "user community" chart. We - so far - generally ignored that one, mostly I guessm becasue we did not want to spend time on analysing all the consequences of disabling ack late. I personally have a feeling that it changes a lot of promises about delivery and taks handling in Celery, and shifts the problem to other set of edge cases, but maybe, by making the defaults working "better" out-of-the-box, the reason for the "ack late disabling" migh be generally gone as that might improve Celery deployment stability? That's just a thought and more context, the "ack late" ase bothers me for quite some time, but I do not have enough experience/knowledge in Celery to be able to provide more than just intuition, that maybe this one applied properly might solve problems that manifest in others trying other, worse effectively, things to improve the stability of celery. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
