Hi all, I'm hitting another issue with the upgrade to Pulp 2.7.0 + changing from qpid to rabbitmq for messaging. The workers are continuously going missing, every minute or so. The effect is that the tasks in the task list stay in a Waiting state and are never completed.
Rabbitmq looks healthy; I see successful accepted connections per the logs and can see a bunch of connections in the rabbitmq management GUI. I'm kind of stuck as far as troubleshooting goes. Any tips on what else to investigate? Pulp and rabbitmq servers are both CentOS 6. # /var/log/messages Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed heartbeat from resource_manager@pulp01 Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-3@pulp01' discovered Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-1@pulp01' discovered Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-2@pulp01' discovered Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-0@pulp01' discovered Nov 18 09:05:56 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'resource_manager@pulp01' discovered Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of work ers Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its q ueue. Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of work ers Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its q ueue. Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of work ers Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its q ueue. Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of work ers Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its q ueue. Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue. Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running. ________________________________ Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list
