The issue with the 16.04 Ubuntu image is fixed now (but we may require some additional actions which I'll send to Vanessa to in case this issue comes up again). We fixed this issue tonight by rebuilding ubuntu1604 and deploying the new image.
I'm going to close this ticket as resolved and we'll take the additional task to find a way to ensure this doesn't appear again off of this ticket. If you're not interested in the detailed analysis you can stop reading now. For those interested I suspect that the lock issue will appear again (although I could be wrong). The reason I believe so is that our vm init script runs "apt-get update" as an initialization step when the VM boots up at creation time via this script [0]. Ed mentioned that we didn't see this in the past and it only started appear again recently as we deployed another patch to disable Ubuntu's unattended updates. I believe a possible reason we will see this issue appear again due to [0] is because of we switched from using JClouds to OpenStack Jenkins plugins for node spinnup and there's difference in how the init-script is executed depending on which plugin is being used. JClouds Plugin: 1) boot vm 2) wait for ssh access 3) copies init-script into vm via ssh 4) executes init-script, and doesn't continue processing until script is complete 5) once init-script is complete, passes vm over to job and job starts OpenStack Plugin: 1) boot vm and passes init-script in as User Data 2) init-script runs inside vm without Jenkins intervention, thus is a non-blocking function 3) in parallel jenkins waits for ssh access to vm 4) ssh's into vm and passes vm over to job and job starts running In the OpenStack plugin case step 4 can execute while step 2 is still running apt-get update in the background because it was a non-blocking function. A few ideas I have to get around this. a) Allow init-script to continue running apt-get update however have a shell script at the start of Ubuntu jobs that waits for the lock to get released before allowing the job to start b) Remove apt-get update from init-script and make the job run apt-get update at the beginning of it's execution c) Regularly update VMs to ensure that apt-get update always runs quickly Regards, Thanh [0] https://git.fd.io/ci-management/tree/jenkins-scripts/basic_settings.sh#n14 On Thu Jan 19 19:23:59 2017, hagbard wrote: > FYI... helpdesk is on it, and its being worked in #fdio-infra on IRC > > Ed > > On Thu, Jan 19, 2017 at 4:31 PM, Ed Warnicke <hagb...@gmail.com> wrote: > > > Looping in help desk. > > On Thu, Jan 19, 2017 at 4:16 PM Dave Barach (dbarach) <dbar...@cisco.com> > > wrote: > > > >> Folks, > >> > >> > >> > >> See https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/3378/console > >> > >> > >> > >> 11:00:46 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource > >> temporarily unavailable) > >> > >> 11:00:46 E: Unable to lock the administration directory (/var/lib/dpkg/), > >> is another process using it? > >> > >> > >> > >> I recognize this failure from my own Ubuntu 16.04 system: a cron-job > >> starts “apt-get -q”, which for whatever reason does not terminate. As a > >> workaround, “sudo killall apt-get || true” before trying to acquire build > >> dependencies... > >> > >> > >> > >> HTH... Dave > >> > >> > >> _______________________________________________ > >> > >> vpp-dev mailing list > >> > >> vpp-dev@lists.fd.io > >> > >> https://lists.fd.io/mailman/listinfo/vpp-dev > > > > _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
