Title: Message Title
Adam Brousseau created an issue
Jenkins / JENKINS-58390
Docker cloud provisioning 1 slave behind
Issue Type:
Bug
Assignee:
Nicolas De Loof
Components:
docker-plugin
Created:
2019-07-08 18:08
Environment:
Jenkins 2.164.3 Docker Plugin 1.1.6
Priority:
Major
Reporter:
Adam Brousseau
We have multiple docker cloud hosts setup with multiple templates per host. Some templates are duplicated across clouds, some are unique to a host. I'm not sure yet how to reproduce or get into this state but I can help diagnose on my end with some guidance. After a period of time successfully provisioning docker agents, we'll get into a state whereby a job is waiting for a container that never comes. If another job is launched which requests a container which happens to match the criteria of the already queued job, the new container will provision and the first job will take it. This leaves the new job waiting until another job requests a matching container. If another job requests a unique container, it too will wait indefinitely until another job requests the same container. Restarting master resolves the issue temporarily (few days). I noticed in the logs once we have hit this state and I launch a job requesting container-A the log shows the following and the container will not start until a second job requests a container.
Jul 08, 2019 12:05:24 PM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'docker-003jf7ab2t864'.
Jul 08, 2019 12:05:24 PM INFO hudson.remoting.Request$2 run
Failed to send back a reply to the request hudson.remoting.Request$2@17