Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2023-11-17 Thread via GitHub
RNHTTR commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1816984824 > Yes. You need to run similar monitoring and restarting of Celery on your own. Liveness Probe runs periodicaly check if any component is responding and will restart it when it is n

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-22 Thread via GitHub
dstandish commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1904486710 We were able to improve the behavior with these settings: ``` AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT=30 AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-26 Thread via GitHub
kim-youngseop-developer commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1912341464 @dstandish dude, it's passed about 30 minutes since after applying your solution. my celery worker was throwing `connection reset by peer.` errors and freezin

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-26 Thread via GitHub
potiuk commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1912349988 > @dstandish dude, it's passed about 30 minutes since after applying your solution. my celery worker was throwing `connection reset by peer.` errors and freezing very frequently, bu

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-26 Thread via GitHub
dstandish commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1912386701 > @dstandish dude, it's passed about 30 minutes since after applying your solution. my celery worker was throwing `connection reset by peer.` errors and freezing very frequently,

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-26 Thread via GitHub
kaxil commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1913058540 > We were able to improve the behavior with these settings: > > ``` > AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT=30 > AIRFLOW__CELERY_BROKER_TRANSPORT_OPTION

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-27 Thread via GitHub
potiuk commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1913286153 > We could set those in the Helm Chart too ? Yeah. Good idea. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1914564113 > I'd wait with the doc changes to see if celery 5.4.0 solves this problem from the root then we can simply bump version I think it depends if those are "Generically applicabl

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1914580294 And just to add a bit more context to the discussion. It's a known fact that a number of our use have some troubles with stability running Celery-based deployment. I think if we kno

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1914638507 And the new PR for "ack_late" - to complete the picture: https://github.com/apache/airflow/pull/37066 -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Celery worker enters a catatonic state after redis restart [airflow]

2024-01-29 Thread via GitHub
eladkal commented on issue #26542: URL: https://github.com/apache/airflow/issues/26542#issuecomment-1916233877 > So I think if we ALSO have a way how to bring older celery configuration to "reasonable defaults", it's worth considering to explain that in the documentation and even our Helm C