[jira] [Commented] (IMPALA-8115) some jenkins workers slow to do real work due to dpkg lock conflicts
[
https://issues.apache.org/jira/browse/IMPALA-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041739#comment-17041739
]
Laszlo Gaal commented on IMPALA-8115:
-
By now all currently used build jobs on jenkins.impala.io have been switched to
the "configured" images, preventing the issue above.
Work is also in progress to make building and maintaining these image types
more regular and automated, but that work is tracked elsewhere; thus resolving
this issue as fixed.
> some jenkins workers slow to do real work due to dpkg lock conflicts
>
>
> Key: IMPALA-8115
> URL: https://issues.apache.org/jira/browse/IMPALA-8115
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Michael Brown
>Assignee: Laszlo Gaal
>Priority: Major
>
> A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start
> doing real work. I noticed that it was retrying {{apt-get update}}:
> {noformat}
> ++ sudo apt-get --yes install openjdk-8-jdk
> E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily
> unavailable)
> E: Unable to lock the administration directory (/var/lib/dpkg/), is another
> process using it?
> ++ date
> Thu Jan 24 23:37:33 UTC 2019
> ++ sudo apt-get update
> ++ sleep 10
> ++ sudo apt-get --yes install openjdk-8-jdk
> [etc]
> {noformat}
> I ssh'd into a host and saw that, yes, something else was holding onto the
> dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was
> the culprit)
> {noformat}
> root 1750 0.0 0.0 4508 1664 ?Ss 23:21 0:00 /bin/sh
> /usr/lib/apt/apt.systemd.daily
> root 1804 12.3 0.1 141076 80452 ?S23:22 1:24 \_
> /usr/bin/python3 /usr/bin/unattended-upgrade
> root 3263 0.0 0.1 140960 72896 ?S23:23 0:00 \_
> /usr/bin/python3 /usr/bin/unattended-upgrade
> root 11459 0.6 0.0 45920 25184 pts/1Ss+ 23:24 0:03 \_
> /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure
> /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb
> /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb
> /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
>
> /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb
> /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb
> /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb
> /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb
> /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb
> /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb
> /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb
> /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb
> /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb
> /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb
> /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb
> /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb
> /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb
> /var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb
> /var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb
> /var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb
> /var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb
> /var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb
>
[jira] [Commented] (IMPALA-8115) some jenkins workers slow to do real work due to dpkg lock conflicts
[
https://issues.apache.org/jira/browse/IMPALA-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021495#comment-17021495
]
Laszlo Gaal commented on IMPALA-8115:
-
I have reconfigured several jobs to use labels with names ending in
"configured".
These labels run on prebaked AMIs instead of the stock Ubuntu AMIs published by
the Ubuntu project.
The prebaked AMIs hav openjdk-8, git and (in some of the cases) Docker
preinstalled; unattended upgrades
have also been disabled.
This should make job startup suffer much less latency.
Other jobs (e.g. the Python 2.6 incompatibility checker) have the base AMI
updated to a much more recent Ubuntu AMI, making the autoupdate process faster
(less delta between the current state and the state recorded in the published
AMI).
Unfortunately both solutions need to be reapplied periodically to keep the
images and labels up to date.
> some jenkins workers slow to do real work due to dpkg lock conflicts
>
>
> Key: IMPALA-8115
> URL: https://issues.apache.org/jira/browse/IMPALA-8115
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Michael Brown
>Assignee: Laszlo Gaal
>Priority: Major
>
> A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start
> doing real work. I noticed that it was retrying {{apt-get update}}:
> {noformat}
> ++ sudo apt-get --yes install openjdk-8-jdk
> E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily
> unavailable)
> E: Unable to lock the administration directory (/var/lib/dpkg/), is another
> process using it?
> ++ date
> Thu Jan 24 23:37:33 UTC 2019
> ++ sudo apt-get update
> ++ sleep 10
> ++ sudo apt-get --yes install openjdk-8-jdk
> [etc]
> {noformat}
> I ssh'd into a host and saw that, yes, something else was holding onto the
> dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was
> the culprit)
> {noformat}
> root 1750 0.0 0.0 4508 1664 ?Ss 23:21 0:00 /bin/sh
> /usr/lib/apt/apt.systemd.daily
> root 1804 12.3 0.1 141076 80452 ?S23:22 1:24 \_
> /usr/bin/python3 /usr/bin/unattended-upgrade
> root 3263 0.0 0.1 140960 72896 ?S23:23 0:00 \_
> /usr/bin/python3 /usr/bin/unattended-upgrade
> root 11459 0.6 0.0 45920 25184 pts/1Ss+ 23:24 0:03 \_
> /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure
> /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb
> /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb
> /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
>
> /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb
> /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb
> /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb
> /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb
> /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb
> /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb
> /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb
> /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb
> /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb
> /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb
> /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb
> /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb
> /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb
>
[jira] [Commented] (IMPALA-8115) some jenkins workers slow to do real work due to dpkg lock conflicts
[
https://issues.apache.org/jira/browse/IMPALA-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751703#comment-16751703
]
Michael Brown commented on IMPALA-8115:
---
I noticed this because I posted https://gerrit.cloudera.org/#/c/12272/ at 2:42
PT and at 3:45 PT there is still no update that the pre-review job completed.
https://jenkins.impala.io/job/gerrit-code-review-checks/1879/console shows a
slow timeline:
{noformat}
Triggered by Gerrit: http://gerrit.cloudera.org:8080/12272
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] timestamps
[Pipeline] {
[Pipeline] ansiColor
[Pipeline] {
[Pipeline] timeout
22:42:09 Timeout set to expire in 10 hr
[Pipeline] {
[Pipeline] parallel
[Pipeline] { (Branch: Tidy)
[Pipeline] { (Branch: BuildOnly)
[Pipeline] { (Branch: Python26Compatibility)
[Pipeline] { (Branch: Rat)
[Pipeline] build (Building clang-tidy-ub1604)
22:42:09 Scheduling project: clang-tidy-ub1604
[Pipeline] build (Building ubuntu-16.04-build-only)
22:42:09 Scheduling project: ubuntu-16.04-build-only
[Pipeline] build (Building python26-incompatibility-check)
22:42:09 Scheduling project: python26-incompatibility-check
[Pipeline] build (Building rat-check-ub1604)
22:42:09 Scheduling project: rat-check-ub1604
22:44:22 Starting building: python26-incompatibility-check #1548
[Pipeline] }
23:35:15 Starting building: ubuntu-16.04-build-only #5027
23:36:34 Starting building: rat-check-ub1604 #5404
[Pipeline] }
23:37:14 Starting building: clang-tidy-ub1604 #4922
{noformat}
It's taken nearly an hour to start rat-check-ub1604 #5404 and clang-tidy-ub1604
#4922.
My 15 minutes math doesn't account for all of this. I'm not yet sure where else
we could accelerate/improve the process.
> some jenkins workers slow to do real work due to dpkg lock conflicts
>
>
> Key: IMPALA-8115
> URL: https://issues.apache.org/jira/browse/IMPALA-8115
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Michael Brown
>Priority: Major
>
> A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start
> doing real work. I noticed that it was retrying {{apt-get update}}:
> {noformat}
> ++ sudo apt-get --yes install openjdk-8-jdk
> E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily
> unavailable)
> E: Unable to lock the administration directory (/var/lib/dpkg/), is another
> process using it?
> ++ date
> Thu Jan 24 23:37:33 UTC 2019
> ++ sudo apt-get update
> ++ sleep 10
> ++ sudo apt-get --yes install openjdk-8-jdk
> [etc]
> {noformat}
> I ssh'd into a host and saw that, yes, something else was holding onto the
> dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was
> the culprit)
> {noformat}
> root 1750 0.0 0.0 4508 1664 ?Ss 23:21 0:00 /bin/sh
> /usr/lib/apt/apt.systemd.daily
> root 1804 12.3 0.1 141076 80452 ?S23:22 1:24 \_
> /usr/bin/python3 /usr/bin/unattended-upgrade
> root 3263 0.0 0.1 140960 72896 ?S23:23 0:00 \_
> /usr/bin/python3 /usr/bin/unattended-upgrade
> root 11459 0.6 0.0 45920 25184 pts/1Ss+ 23:24 0:03 \_
> /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure
> /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb
> /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb
> /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
>
> /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb
> /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb
> /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb
> /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb
> /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb
> /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb
> /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb
> /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb
> /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb
> /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb
> /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb
> /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb
> /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb
> /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb
>
