Your message dated Fri, 23 Feb 2024 17:32:31 +0000
with message-id <e1rdzpt-00gftq...@fasolo.debian.org>
and subject line Bug#1028212: fixed in prometheus-node-exporter-collectors
0.0~git20230203.6f710f8-1+deb12u1
has caused the Debian Bug report #1028212,
regarding prometheus-node-exporter-collectors: APT update deadlock - prevents
unattended security upgrades
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)
--
1028212: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028212
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: prometheus-node-exporter-collectors
Version: 0.0~git20221011.8f6be63-1
Severity: important
Tags: security
X-Debbugs-Cc: Salvatore Bonaccorso <car...@debian.org>, steve+...@tecwec.eu,
Debian Security Team <t...@security.debian.org>
As requested, by Salvatore lowering prio and avoiding embargo.
-----
Hello, happy new year, and thanks.
This looks like an apt deadlock, which prevents updates, unattended upgrades,
and so critical security updates
for systems where they are enabled.
(Yes, we can just manually kill the offending apt_info.py process to
temporarily solve the issue - but this is not the good solution).
As it prevents security updates, and despite it unlikely to happen massively,
and be practically exploited, I feel this requires real attention.
Symptoms:
Persistent apt update locking error:
# apt update
Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock. It is held by process 65553
(python3)
N: Be aware that removing the lock file is not a solution and may break your
system.
E: Unable to lock directory /var/lib/apt/lists/
# 1 hour later, same issue, same holding PID 65553
# Concerned processes:
# ps aux |grep pyth
root 1259 0.0 0.1 121076 27528 ? Ssl Jan06 0:00
/usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgra>
root 65553 0.0 0.4 89640 76908 ? S 12:09 0:03 python3
/usr/share/prometheus-node-exporter-collectors/apt_info.py
ee 70395 0.0 0.2 124164 42844 ? Sl 12:35 0:00
/bin/python3.11 /home/ee/.vscode-oss/extensions/ms-python.python> (not
suspected)
# ps aux |grep apt
root 65551 0.0 0.0 9552 4252 ? Ss 12:09 0:00 /bin/bash -c
/usr/share/prometheus-node-exporter-collectors/apt_>
root 65553 0.0 0.4 89640 76908 ? S 12:09 0:03 python3
/usr/share/prometheus-node-exporter-collectors/apt_info.>
root 65554 0.0 0.0 2464 884 ? S 12:09 0:00 sponge
/var/lib/prometheus/node-exporter/apt.prom
_apt 65814 0.0 0.0 27192 13204 ? S 12:09 0:00
/usr/lib/apt/methods/https
_apt 65815 0.0 0.0 24420 10236 ? S 12:09 0:00
/usr/lib/apt/methods/http
_apt 65816 0.0 0.0 27192 13204 ? S 12:09 0:00
/usr/lib/apt/methods/https
_apt 65817 0.0 0.0 24420 10272 ? S 12:09 0:00
/usr/lib/apt/methods/http
_apt 65819 0.0 0.0 17572 7624 ? S 12:09 0:00
/usr/lib/apt/methods/gpgv
_apt 65826 0.0 0.0 27192 13464 ? S 12:09 0:00
/usr/lib/apt/methods/https
_apt 65829 0.0 0.0 24420 10292 ? S 12:09 0:00
/usr/lib/apt/methods/http
_apt 66110 0.0 0.0 17528 7500 ? S 12:10 0:00
/usr/lib/apt/methods/store
_apt 66112 0.0 0.0 18436 8636 ? S 12:10 0:00
/usr/lib/apt/methods/rred
_apt 66113 0.0 0.0 18576 8860 ? S 12:10 0:00
/usr/lib/apt/methods/rred
The deadlock is obviously between the unattended-upgrade proc (1259), and the
prometheus tryptic: 65551/53/54.
# 65553 seems to be the culprit - as apt update tells us
# strace -p 65553
strace: Process 65553 attached
pselect6(29, [12 13 14 16 18 20 22 24 26 28], [], NULL, {tv_sec=0,
tv_nsec=499419645}, NULL) = 0 (Timeout)
pselect6(29, [12 13 14 16 18 20 22 24 26 28], [], NULL, {tv_sec=0,
tv_nsec=500000000}, NULL) = 0 (Timeout)
... repeats 'forever' ....
All fds are pipes, I could not get more info until the processed crashed due to
my diagnostic atttempts.
An apt/python/prom collector specialist should instantly identify these pipes
and make more deductions, from the following state:
# gdb -p 65553 and bt:
#0 0x00007fa4bf65f794 in __GI___select (nfds=29, readfds=0x7ffc24f8e7c0,
writefds=0x7ffc24f8e840, exceptfds=0x0,
timeout=0x7ffc24f8e750) at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x00007fa4bebad338 in pkgAcquire::Run(int) () from
/lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#2 0x00007fa4becb1485 in AcquireUpdate(pkgAcquire&, int, bool, bool) () from
/lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#3 0x00007fa4becb1976 in ListUpdate(pkgAcquireStatus&, pkgSourceList&, int) ()
from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0
#4 0x00007fa4bed32fe1 in ?? () from
/usr/lib/python3/dist-packages/apt_pkg.cpython-311-x86_64-linux-gnu.so
#5 0x0000000000521cf0 in ?? ()
#6 0x000000000053983c in PyObject_Vectorcall ()
#7 0x000000000052a570 in _PyEval_EvalFrameDefault ()
#8 0x000000000052222b in PyEval_EvalCode ()
#9 0x0000000000647f07 in ?? ()
#10 0x00000000006457cf in ?? ()
#11 0x0000000000651920 in ?? ()
#12 0x000000000065166b in _PyRun_SimpleFileObject ()
#13 0x0000000000651494 in _PyRun_AnyFileObject ()
#14 0x000000000065022f in Py_RunMain ()
#15 0x00000000006248b7 in Py_BytesMain ()
#16 0x00007fa4bf58818a in __libc_start_call_main (main=main@entry=0x624820,
argc=argc@entry=2,
argv=argv@entry=0x7ffc24f8f298) at ../sysdeps/nptl/libc_start_call_main.h:58
#17 0x00007fa4bf588245 in __libc_start_main_impl (main=0x624820, argc=2,
argv=0x7ffc24f8f298, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc24f8f288)
at ../csu/libc-start.c:381
#18 0x0000000000624751 in _start ()
This seems to suggest that the location of the deadlock, for 65553, is:
(apt_info.py)
def _main():
cache = apt.cache.Cache()
# First of all, attempt to update the index. If we don't have permission
# to do so (or it fails for some reason), it's not the end of the world,
# we'll operate on the old index.
with contextlib.suppress(apt.cache.LockFailedException,
apt.cache.FetchFailedException):
cache.update() <<<<<<<<<<<< VERY LIKELY
I could not confirm the precise location, as trying to get a python backtrace
from the process generated a SEGV:
(gdb) call PyRun_SimpleString("print('toto\n')") # to test
'PyRun_SimpleString' has unknown return type; cast the call to its declared
return type
(gdb) call (void*)PyRun_SimpleString("print('toto\n')")
Program received signal SIGSEGV, Segmentation fault.
# Oops... will not get a python trace now.
Hopefully, I collected the core (~27MB) - if interested, tell me - keeping it
for a few weeks:
#0 0x000000000063187a in ?? ()
#1 0x00000000006349b2 in PyImport_AddModuleObject ()
#2 0x0000000000634688 in PyImport_AddModule ()
#3 0x000000000063e323 in PyRun_SimpleStringFlags ()
(but I feel it unrelated, and not so usefull - but I may be wrong)
I feel I can't help more now, so throwing the potato 😉
Best,
Eric 'Steve' Estievenart
-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (990, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.0.0-6-amd64 (SMP w/4 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages prometheus-node-exporter-collectors depends on:
ii moreutils 0.67-1
ii prometheus-node-exporter 1.5.0-1+b1
ii python3-apt 2.5.0
ii systemd-sysv 252.4-1
Versions of packages prometheus-node-exporter-collectors recommends:
ii ipmitool 1.8.19-4
ii jq 1.6-2.1
ii nvme-cli 2.2.1-3
ii python3 3.11.1-1
ii smartmontools 7.3-1+b1
prometheus-node-exporter-collectors suggests no packages.
-- no debconf information
--- End Message ---
--- Begin Message ---
Source: prometheus-node-exporter-collectors
Source-Version: 0.0~git20230203.6f710f8-1+deb12u1
Done: Antoine Beaupré <anar...@debian.org>
We believe that the bug you reported is fixed in the latest version of
prometheus-node-exporter-collectors, which is due to be installed in the Debian
FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 1028...@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Antoine Beaupré <anar...@debian.org> (supplier of updated
prometheus-node-exporter-collectors package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmas...@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Tue, 31 Oct 2023 13:57:52 -0400
Source: prometheus-node-exporter-collectors
Architecture: source
Version: 0.0~git20230203.6f710f8-1+deb12u1
Distribution: bookworm
Urgency: medium
Maintainer: Debian Go Packaging Team <team+pkg...@tracker.debian.org>
Changed-By: Antoine Beaupré <anar...@debian.org>
Closes: 1028212
Changes:
prometheus-node-exporter-collectors (0.0~git20230203.6f710f8-1+deb12u1)
bookworm; urgency=medium
.
* Team upload
* Fix deadlock with other apt update runs (Closes: #1028212)
Checksums-Sha1:
1871dadd592acdef225fd159c0b9bce182964292 1845
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1.dsc
1167c16e9cee42df8174669f13254810995972f9 6600
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1.debian.tar.xz
d98d3e02c151a188172cc4faf415a573e4d5bb7c 5785
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1_amd64.buildinfo
Checksums-Sha256:
745c39eca7b9f314d89347fe894d7f4a6097162a164a8c7ed2752ead5f532e37 1845
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1.dsc
8d1b9162152e2eae7ac9160b97dd111328a66fa92b0d59232ecee6ba2045120d 6600
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1.debian.tar.xz
4375e55e11f338f039f30b3dc8be36528f338c20c970af619ebbaadcc5c47660 5785
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1_amd64.buildinfo
Files:
6b045aee739ba60a424081c7e04f6da2 1845 net optional
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1.dsc
47af1dea96ed961fb859fe2cc3943f9c 6600 net optional
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1.debian.tar.xz
9cdeab9a29ca80322f531acd76b974e7 5785 net optional
prometheus-node-exporter-collectors_0.0~git20230203.6f710f8-1+deb12u1_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----
iHUEARYIAB0WIQS7ts1MmNdOE1inUqYCKTpvpOU0cwUCZdYJUwAKCRACKTpvpOU0
cz35AP4kbSbM8kGeII/+QBBptpEapQF8aUBxIgtpEu5QGhN0sQEA/uneK/w4WgDG
hLOTUYHJoEBEhbyP2qnIUO9dM7PcDgA=
=Lx29
-----END PGP SIGNATURE-----
pgpgwWDs0ohLB.pgp
Description: PGP signature
--- End Message ---
