Hi everyone, I've been trying to debug this issue for a couple days now without too much luck. The basic problem is that starting a worker using 'airflow worker' with celery.pool = gevent or eventlet results in tasks reaching the worker but never being executed. However, using prefork as the pool or starting a worker with 'celery worker -P eventlet' and passing all the configs manually results in the expected behaviour.
Some observations in differences between airflow worker (aw) and celery worker (cw) using eventlet: 1. aw passes a hub object to celery.worker.consumer.consumer.Consumer.__init__ whereas cw does not 2. After receiving a message, a stack trace printed in eventlet.greenpool.GreenPool.spawn_n reaches celery.worker.consumer.consumer.Evloop.start for both aw + cw then they diverge: aw: -> celery.worker.loops.asynloop -> kombu.asynchronous.hub.Hub.create_loop -> kombu.transport.base.Transport.on_readable -> kombu.transport.base.Transport._read cw: -> celery.worker.loops.synloop -> kombu.common.QoS.update -> kombu.common.QoS.set -> celery.worker.consumer.tasks.Tasks.start.set_prefetch_count -> kombu.messaging.Consumer.qos -> amqp.channel.Channel.basic_qos -> amqp.abstract_channel.AbstractChannel.send_method -> amqp.abstract_channel.AbstractChannel.wait 3. They both converge at amqp.connection.Connection.drain_events and follow the same path to eventlet.greenpool.GreenPool.spawn_n. However, aw stops at this function while cw continues. The last line that I've been able to trace aw to is eventlet/greenpool.py:118 (self.coroutines_running.add(g)). Execution seems to hang there for each thread and I only see heartbeat printouts with debug logging. cw continues to eventlet.greenpool.GreenPool._spawn_n_impl and executes the task. 4. Possibly related: I've noticed that celery flower will often say "unknown worker" after clicking a worker hostname when aw + eventlet/gevent is used while cw doesn't have this issue. Any suggestions about how to get airflow worker with eventlet/gevent working would be much appreciated. However, I'm also wondering if there are any negative side-effects of creating my workers through celery worker if I'm not able to get airflow worker working? Thanks, Aaron
