I'm starting to think that there is something wrong with the horsea
machine indeed: still not able to upgrade to lunar. Starting from jammy
or from kinetic failed.
#Jammy
Deploying again jammy and doing do-release-upgrade again failed with
ssh, with differents things after rebooting and trying the do the
release upgrade (I have outputs, but I don't remember exactly at what
point I did the reboot of the machine, i.e: "Missing privilege
separation directory: /run/sshd" could happen after reboot, but not
sure).
#Kinetic
So, ok... let's think is something wrong with the jammy image that is being
deployed (a remote possibility, I think, but ...).... trying the upgrading from
Kinetic: failed. I checked the system status before starting the upgrade and it
was not degraded.
It was stucked at 97% on:
Setting up snapd (2.58+22.10.1) ...
Installing new version of config file /etc/apt/apt.conf.d/20snapd.conf ...
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
snapd.failure.service is a disabled or a static unit not running, not starting
it.
snapd.snap-repair.service is a disabled or a static unit not running, not
starting it.
Failed to restart snapd.mounts-pre.target: Operation refused, unit
snapd.mounts-pre.target may be requested by dependenc
y only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status snapd.mounts-pre.target' for details.
system status was:
│ └─unattended-upgrades.service
│ ├─ 1227 /usr/bin/python3
/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
│ └─11628 /usr/bin/python3
/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
└─user.slice
└─user-1000.slice
└─session-1.scope
├─ 1856 "sshd: ubuntu [priv]"
├─ 1937 "sshd: ubuntu@pts/0"
├─ 1941 -bash
├─ 2302 sudo apt upgrade -y
├─ 2303 sudo apt upgrade -y
├─ 2304 apt upgrade -y
├─ 5750 /usr/bin/dpkg --status-fd 39 --configure --pending
├─ 5751 sh -c "(test -x /usr/lib/needrestart/dpkg-status &&
/usr/lib/needrestart/dpkg-status || cat > /dev/null>
├─ 5752 sh -c "(test -x /usr/lib/needrestart/dpkg-status &&
/usr/lib/needrestart/dpkg-status || cat > /dev/null>
├─ 5753 /bin/sh /usr/lib/needrestart/dpkg-status
├─11336 /bin/sh /var/lib/dpkg/info/snapd.postinst configure
2.57.5+22.10ubuntu0.1
├─11501 /usr/bin/perl /usr/bin/deb-systemd-invoke restart
snapd.aa-prompt-listener.service snapd.apparmor.servi>
├─11563 systemctl --quiet --system restart
snapd.aa-prompt-listener.service snapd.apparmor.service snapd.autoim>
└─11564 /bin/systemd-tty-ask-password-agent --watch
After that, it failed with:
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Setting up openssh-sftp-server (1:9.0p1-1ubuntu7.3) ...
Setting up needrestart (3.6-1ubuntu0.1) ...
Setting up openssh-server (1:9.0p1-1ubuntu7.3) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess returned
error exit status 1
Setting up bind9-dnsutils (1:9.18.12-0ubuntu0.22.10.2) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
Processing triggers for rsyslog (8.2208.0-1ubuntu2) ...
Processing triggers for ufw (0.36.1-4build1) ...
Processing triggers for man-db (2.10.2-2) ...
Processing triggers for dbus (1.14.0-2ubuntu3) ...
Processing triggers for initramfs-tools (0.140ubuntu17) ...
update-initramfs: Generating /boot/initrd.img-5.19.0-46-generic
Processing triggers for ca-certificates (20230311ubuntu0.22.10.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Errors were encountered while processing:
openssh-server
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
System status after that was:
ubuntu@node-horsea:~$ systemctl status -l
Failed to read server status: Connection timed out
ubuntu@node-horsea:~$ systemctl status -l
● node-horsea
State: degraded
Units: 425 loaded (incl. loaded aliases)
Jobs: 1 queued
Failed: 7 units
Since: Thu 2023-07-27 10:48:27 UTC; 18min ago
systemd: 251.4-1ubuntu7.3
CGroup: /
├─init.scope
│ └─1 /lib/systemd/systemd --- --system --deserialize 22
├─system.slice
│ ├─cron.service
│ │ └─1138 /usr/sbin/cron -f -P
│ ├─dbus.service
│ │ ├─1139 @dbus-daemon --system --address=systemd: --nofork
--nopidfile --systemd-activation --syslog-only
│ │ └─6714 @dbus-daemon --system --address=systemd: --nofork
--nopidfile --systemd-activation --syslog-only
│ ├─irqbalance.service
│ │ ├─1145 /usr/sbin/irqbalance --foreground
│ │ └─6719 /usr/sbin/irqbalance --foreground
│ ├─multipathd.service
│ │ └─6654 /sbin/multipathd -d -s
│ ├─packagekit.service
│ │ └─2204 /usr/libexec/packagekitd
│ ├─polkit.service
│ │ └─2208 /usr/libexec/polkitd --no-debug
│ ├─rsyslog.service
│ │ └─12184 /usr/sbin/rsyslogd -n -iNONE
│ ├─snapd.service
│ │ ├─ 1150 /usr/lib/snapd/snapd
│ │ └─23845 /usr/lib/snapd/snapd
│ ├─ssh.service
│ │ ├─ 1849 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
│ │ ├─11762 "sshd: ubuntu [priv]"
│ │ ├─11841 "sshd: ubuntu@pts/3"
│ │ ├─11842 -bash
│ │ ├─23869 systemctl status -l
│ │ └─23870 less
│ ├─system-getty.slice
│ │ └─getty@tty1.service
│ │ └─6840 /sbin/agetty -o "-p -- \\u" --noclear - linux
│ ├─system-serial\x2dgetty.slice
│ │ └─serial-getty@ttyS1.service
│ │ └─6834 /sbin/agetty -o "-p -- \\u" --keep-baud
115200,57600,38400,9600 - vt220
│ ├─systemd-journald.service
│ │ └─3834 /lib/systemd/systemd-journald
│ ├─systemd-logind.service
│ │ └─11626 /lib/systemd/systemd-logind
│ ├─systemd-networkd.service
│ │ ├─1086 /lib/systemd/systemd-networkd
│ │ └─4522 /lib/systemd/systemd-networkd
│ ├─systemd-resolved.service
│ │ └─6942 /lib/systemd/systemd-resolved
│ ├─systemd-timesyncd.service
│ │ └─5877 /lib/systemd/systemd-timesyncd
│ ├─systemd-udevd.service
│ │ └─udev
│ │ └─5921 /lib/systemd/systemd-udevd
│ └─unattended-upgrades.service
│ ├─ 1227 /usr/bin/python3
/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
│ └─11628 /usr/bin/python3
/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
└─user.slice
└─user-1000.slice
└─session-1.scope
├─ 1856 "sshd: ubuntu [priv]"
├─ 1937 "sshd: ubuntu@pts/0"
├─ 1941 -bash
├─23835 sudo su
├─23836 sudo su
├─23837 su
└─23838 bash
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551
Title:
upgrade to lunar fails due to rescue-ssh.target or port 22 takeover
Status in openssh package in Ubuntu:
New
Bug description:
Hi,
I just upgraded a system from Jammy to Lunar and openssh-server refuses to
upgrade well.
Setting up openssh-server (1:9.0p1-1ubuntu8) ...
Replacing config file /etc/ssh/sshd_config with new version
Replacing config file /etc/ssh/sshd_config with new version
Synchronizing state of ssh.service with SysV service script with
/lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ssh
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess
returned error exit status 1
Processing triggers for man-db (2.11.2-1) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
Errors were encountered while processing:
openssh-server
Error: Timeout was reached
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
I'm not sure what exactly it is.
This output complains about rescue-ssh.target and indeed that can not be
started even directly.
$ sudo systemctl start rescue-ssh.target
A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for
details.
And in postinst is a try to start it:
$ grep rescue /var/lib/dpkg/info/openssh-server.postinst
deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null
|| true
But I think the underlying issue is that ssh is already on, and I'm logged in
via it.
And that makes the service restart of the ssh socket which was added break.
Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create
listening socket ([::]:22): Address already in use
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive
listening socket ([::]:22): Input/output error
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on
sockets: Input/output error
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result
'resources'.
Now, whichever it is, it is hard to resolve.
The only way to get the socket to own it would be rebooting so that sshd lets
go and systemd can take over.
I could reboot, but that is not the point.
What if I'd want to get the service and upgrade completed before reboot.
Because as of now dpkg considers the system unhappy, and that would usually
be a sign for "better not reboot before being resolved" to me.
One thing though, I have not upgraded with do-release-upgrade - would
we / do we have magic there to make the ssh socket activation
transition smoother?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
