shivaam commented on issue #59707:
URL: https://github.com/apache/airflow/issues/59707#issuecomment-4174346189

   @auvipy wondering if you have any thoughts on this issue. Any inputs on what 
we can check further?
   
   ```
   The duplicate hostname check calls 
celery_app.control.inspect().active_queues() before celery_app.worker_main(). 
The inspect() call opens broker connections and initializes internal 
connection/producer pools on the Celery app. When worker_main() starts the 
worker with the prefork pool (which uses fork()), this pre-initialized state 
interferes with the worker's internal communication — tasks are received from 
the broker but never dispatched to pool workers.
   
   I confirmed this by observing that app.amqp._producer_pool is non-None at 
worker_main() time when the bug occurs, and None when it works correctly. 
Clearing this state with kombu.pools.reset() after the inspect() call restores 
correct behavior.
   
   I have not fully traced the exact internal mechanism by which the 
pre-initialized pools break the prefork dispatch path, but the correlation is 
consistent across all tests.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to