[jira] [Created] (IMPALA-8115) some jenkins workers slow to spawn to to dpkg lock conflicts
Michael Brown created IMPALA-8115:
-
Summary: some jenkins workers slow to spawn to to dpkg lock
conflicts
Key: IMPALA-8115
URL: https://issues.apache.org/jira/browse/IMPALA-8115
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Reporter: Michael Brown
A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start
doing real work. I noticed that it was retrying {{apt-get update}}:
{noformat}
++ sudo apt-get --yes install openjdk-8-jdk
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily
unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another
process using it?
++ date
Thu Jan 24 23:37:33 UTC 2019
++ sudo apt-get update
++ sleep 10
++ sudo apt-get --yes install openjdk-8-jdk
[etc]
{noformat}
I ssh'd into a host and saw that, yes, something else was holding onto the dpkg
log (confirmed with lsof and not pasted here. dpkg process PID 11459 was the
culprit)
{noformat}
root 1750 0.0 0.0 4508 1664 ?Ss 23:21 0:00 /bin/sh
/usr/lib/apt/apt.systemd.daily
root 1804 12.3 0.1 141076 80452 ?S23:22 1:24 \_
/usr/bin/python3 /usr/bin/unattended-upgrade
root 3263 0.0 0.1 140960 72896 ?S23:23 0:00 \_
/usr/bin/python3 /usr/bin/unattended-upgrade
root 11459 0.6 0.0 45920 25184 pts/1Ss+ 23:24 0:03 \_
/usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure
/var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb
/var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb
/var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb
/var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb
/var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb
/var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb
/var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb
/var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb
/var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb
/var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb
/var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb
/var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb
/var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb
/var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb
/var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb
/var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb
/var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb
/var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb
/var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb
/var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb
/var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb
/var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb
/var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb
/var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb
/var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb
/var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb
/var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb
/var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb
/var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb
/var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb
/var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb
/var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb
/var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb
/var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb
/var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb
[jira] [Created] (IMPALA-8115) some jenkins workers slow to spawn to to dpkg lock conflicts
Michael Brown created IMPALA-8115:
-
Summary: some jenkins workers slow to spawn to to dpkg lock
conflicts
Key: IMPALA-8115
URL: https://issues.apache.org/jira/browse/IMPALA-8115
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Reporter: Michael Brown
A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start
doing real work. I noticed that it was retrying {{apt-get update}}:
{noformat}
++ sudo apt-get --yes install openjdk-8-jdk
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily
unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another
process using it?
++ date
Thu Jan 24 23:37:33 UTC 2019
++ sudo apt-get update
++ sleep 10
++ sudo apt-get --yes install openjdk-8-jdk
[etc]
{noformat}
I ssh'd into a host and saw that, yes, something else was holding onto the dpkg
log (confirmed with lsof and not pasted here. dpkg process PID 11459 was the
culprit)
{noformat}
root 1750 0.0 0.0 4508 1664 ?Ss 23:21 0:00 /bin/sh
/usr/lib/apt/apt.systemd.daily
root 1804 12.3 0.1 141076 80452 ?S23:22 1:24 \_
/usr/bin/python3 /usr/bin/unattended-upgrade
root 3263 0.0 0.1 140960 72896 ?S23:23 0:00 \_
/usr/bin/python3 /usr/bin/unattended-upgrade
root 11459 0.6 0.0 45920 25184 pts/1Ss+ 23:24 0:03 \_
/usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure
/var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb
/var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb
/var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb
/var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb
/var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb
/var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb
/var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb
/var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb
/var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb
/var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb
/var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb
/var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb
/var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
/var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb
/var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb
/var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb
/var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb
/var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
/var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb
/var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb
/var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb
/var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb
/var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb
/var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb
/var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb
/var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb
/var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb
/var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb
/var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb
/var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb
/var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb
/var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb
/var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb
/var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb
/var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb
/var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb
/var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb
