In answer to posed questions: - RHEL 6 x86_64 - qpid-cpp-server - Every 60 seconds for me as well. /var/log/messages loos as below.
-Jeffrey -----Original Message----- From: pulp-list-boun...@redhat.com [mailto:pulp-list-boun...@redhat.com] On Behalf Of Ashby, Jason (IMS) Sent: Wednesday, November 18, 2015 10:38 AM To: Brian Bouterse <bbout...@redhat.com>; pulp-list@redhat.com Subject: Re: [Pulp-list] Missing celery workers For me it is consistently every 60 seconds. Here are logs from the last 10 minutes. I also set loglevel to DEBUG for each config file in /etc/default/pulp* and restarted the services, but I'm not seeing any DEBUG stuff in the logs. $ sudo grep missing /var/log/messages Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:25:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:25:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:25:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its queue. Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. -----Original Message----- From: pulp-list-boun...@redhat.com [mailto:pulp-list-boun...@redhat.com] On Behalf Of Brian Bouterse Sent: Wednesday, November 18, 2015 11:05 AM To: pulp-list@redhat.com Subject: Re: [Pulp-list] Missing celery workers Jason and Jeffrey, Thanks for reporting this. I've written up a bug [0] and I am investigating the root cause. On the bug are you able to leave some answers to these questions? - Can you confirm that it affects both RabbitMQ and Qpid usage? - Can you confirm that the workers "go missing" and then return, and then "go missing" in a continuous cycle? I expect it to happen every 90 seconds. - Jeffrey specifically, what OS are you using? [0]: https://pulp.plan.io/issues/1380 Thanks, Brian On 11/18/2015 09:33 AM, Miller, Jeffrey L wrote: > I am seeing this behavior as well after upgrading from 2.6 to 2.7. > However, I am using qpid not rabbitmq. > > > > -Jeffrey > > > > > > > > *From:* pulp-list-boun...@redhat.com > [mailto:pulp-list-boun...@redhat.com] *On Behalf Of *Ashby, Jason > (IMS) > *Sent:* Wednesday, November 18, 2015 8:29 AM > *To:* pulp-list@redhat.com > *Subject:* [Pulp-list] Missing celery workers > > > > Hi all, > > I'm hitting another issue with the upgrade to Pulp 2.7.0 + changing > from qpid to rabbitmq for messaging. The workers are continuously > going missing, every minute or so. The effect is that the tasks in > the task list stay in a Waiting state and are never completed. > > > > Rabbitmq looks healthy; I see successful accepted connections per the > logs and can see a bunch of connections in the rabbitmq management GUI. > I'm kind of stuck as far as troubleshooting goes. Any tips on what > else to investigate? > > > > Pulp and rabbitmq servers are both CentOS 6. > > > > # /var/log/messages > > Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed > heartbeat from resource_manager@pulp01 > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: > New worker 'reserved_resource_worker-3@pulp01' discovered > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: > New worker 'reserved_resource_worker-1@pulp01' discovered > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: > New worker 'reserved_resource_worker-2@pulp01' discovered > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: > New worker 'reserved_resource_worker-0@pulp01' discovered > > Nov 18 09:05:56 pulp01 pulp: pulp.server.async.worker_watcher:INFO: > New worker 'resource_manager@pulp01' discovered > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-3@pulp01' has gone missing, removing from > list of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-3@pulp01 is missing. Canceling the > tasks in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-1@pulp01' has gone missing, removing from > list of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-1@pulp01 is missing. Canceling the > tasks in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-2@pulp01' has gone missing, removing from > list of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-2@pulp01 is missing. Canceling the > tasks in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-0@pulp01' has gone missing, removing from > list of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-0@pulp01 is missing. Canceling the > tasks in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'resource_manager@pulp01' has gone missing, removing from list of > workers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named resource_manager@pulp01 is missing. Canceling the tasks in its queue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: There > are 0 pulp_resource_manager processes running. Pulp will not operate > correctly without > > at least one pulp_resource_mananger process running. > > > > ---------------------------------------------------------------------- > -- > > > Information in this e-mail may be confidential. It is intended only > for the addressee(s) identified above. If you are not the > addressee(s), or an employee or agent of the addressee(s), please note > that any dissemination, distribution, or copying of this communication > is strictly prohibited. If you have received this e-mail in error, > please notify the sender of the error. > > > > _______________________________________________ > Pulp-list mailing list > Pulp-list@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-list > _______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list ________________________________ Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. _______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list _______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
