Public bug reported: Hi,
Summary: ======== My system is not able to boot unattended anymore. Problem seems similar to https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/564360 History: ======== The history of this issue is the following: 1- On 2016.06.14, I "aptitude safe-upgrade" my system. The kernel upgraded from 3.13.0-85-generic to 3.13.0-88-generic. After the upgrade, the system won't boot anymore. After some troubleshooting, suspecting some kernel/low-level issue, I decided to reboot it on kernel 3.13.0-85-generic, and systems boots OK. 2- on 2016.07.26, I "aptitude safe-upgrade" my system. The kernel upgraded from 3.13.0-85-generic to 3.13.0-92-generic. After the upgrade, the system won't finish booting up. Fortunatelly, OpenSSH had started, and I could investigate the issue a little bit further. Hereunder are my findings about this issue. Findings: ========= On the console, the last boot message I can see is the following: * Starting configure virtual network devices [ OK ] [ 16.192284] bnx2 000:01:00.1 eth1 NIC Copper Link is Down my /var/log/boot.log file is the following: root@otrs-prod:/etc/init# cat /var/log/boot.log * Starting Read required files in advance [ OK ] * Starting Mount filesystems on boot [ OK ] * Starting Fix-up sensitive /proc filesystem entries [ OK ] * Starting Populate /dev filesystem [ OK ] * Starting Populate and link to /run filesystem [ OK ] * Stopping Populate /dev filesystem [ OK ] * Stopping Fix-up sensitive /proc filesystem entries [ OK ] * Stopping Populate and link to /run filesystem [ OK ] * Stopping Track if upstart is running in a container [ OK ] * Starting Signal sysvinit that the rootfs is mounted [ OK ] * Starting Clean /tmp directory [ OK ] * Stopping Read required files in advance (for other mountpoints) [ OK ] * Stopping Read required files in advance (for other mountpoints) [ OK ] * Stopping Clean /tmp directory [ OK ] * Starting Initialize or finalize resolvconf [ OK ] * Starting set console keymap [ OK ] * Starting Signal sysvinit that virtual filesystems are mounted [ OK ] * Starting Signal sysvinit that virtual filesystems are mounted [ OK ] * Starting set sysctls from /etc/sysctl.conf [ OK ] * Starting Bridge udev events into upstart [ OK ] * Starting Signal sysvinit that local filesystems are mounted [ OK ] * Starting Signal sysvinit that remote filesystems are mounted [ OK ] * Starting device node and kernel event manager [ OK ] * Starting GNU Screen Cleanup [ OK ] * Starting Flush boot log to disk [ OK ] * Starting flush early job output to logs [ OK ] * Stopping Mount filesystems on boot [ OK ] * Stopping set sysctls from /etc/sysctl.conf [ OK ] * Stopping set console keymap [ OK ] * Stopping Flush boot log to disk [ OK ] * Stopping flush early job output to logs [ OK ] * Starting load modules from /etc/modules [ OK ] * Starting cold plug devices [ OK ] * Starting log initial device creation [ OK ] * Stopping GNU Screen Cleanup [ OK ] * Starting Bridge file events into upstart [ OK ] * Stopping load modules from /etc/modules [ OK ] * Starting system logging daemon [ OK ] * Starting D-Bus system message bus [ OK ] * Starting SystemD login management service [ OK ] * Stopping cold plug devices [ OK ] * Stopping log initial device creation [ OK ] * Starting configure network device security [ OK ] * Starting configure network device security [ OK ] * Starting configure network device security [ OK ] * Starting configure network device security [ OK ] * Starting configure network device security [ OK ] * Starting load fallback graphics devices [ OK ] * Starting configure network device security [ OK ] * Stopping load fallback graphics devices [ OK ] * Starting save udev log and update rules [ OK ] * Starting set console font [ OK ] * Stopping save udev log and update rules [ OK ] * Stopping set console font [ OK ] * Starting userspace bootsplash [ OK ] * Starting configure network device [ OK ] * Starting Send an event to indicate plymouth is up [ OK ] * Stopping Send an event to indicate plymouth is up [ OK ] * Starting configure network device security [ OK ] * Stopping userspace bootsplash [ OK ] * Starting Mount network filesystems [ OK ] * Stopping Mount network filesystems [ OK ] * Starting Mount network filesystems [ OK ] * Starting configure network device [ OK ] * Stopping Mount network filesystems [ OK ] * Starting Mount network filesystems [ OK ] * Starting configure network device [ OK ] * Starting System V initialisation compatibility [ OK ] * Stopping Mount network filesystems [ OK ] * Starting Uncomplicated firewall [ OK ] Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd * Starting configure network device [ OK ] * Starting Mount network filesystems [ OK ] * Starting Failsafe Boot Delay [ OK ] * Stopping Mount network filesystems [ OK ] * Starting Bridge socket events into upstart [ OK ] * Starting AppArmor profiles [ OK ] * Setting up X socket directories... [ OK ] * Stopping System V initialisation compatibility [ OK ] * Starting System V runlevel compatibility [ OK ] * Starting save kernel messages [ OK ] * Starting regular background program processing daemon [ OK ] * Starting deferred execution scheduler [ OK ] * Starting ACPI daemon [ OK ] * Starting automatic crash report generation [ OK ] * Stopping save kernel messages [ OK ] * Starting CPU interrupts balancing daemon [ OK ] * Starting OpenSSH server [ OK ] Starting Systems Management Device Drivers: Starting dell_rbu: * Starting ipmi driver: * Already started Starting Systems Management Data Engine: Starting dsm_sa_datamgrd: * Starting MySQL Server [ OK ] * Starting dsm_sa_eventmgrd: * Starting dsm_sa_snmpd: * /etc/rc2.d/S20dataeng: 566: /etc/rc2.d/S20dataeng: /opt/dell/srvadmin//lib64/srvadmin-deng/dcfwsnmp.sh: Permission denied WARNING: The SNMP port appears to be closed by the firewall. You may not be able to manage this system remotely using SNMP. See Firewall Configuration in the product user's guide. * Starting Postfix Mail Transport Agent postfix [ OK ] * Starting ipmi drivers [ OK ] * Stopping OpenSSH server [ OK ] * Starting OpenSSH server [ OK ] * Starting Mount network filesystems [ OK ] * Stopping Mount network filesystems [ OK ] * Starting configure network device [ OK ] * Stopping OpenSSH server [ OK ] * Starting OpenSSH server [ OK ] * Starting Mount network filesystems [ OK ] * Starting configure network device [ OK ] * Stopping Mount network filesystems [ OK ] * Starting configure virtual network devices [ OK ] root@otrs-prod:/etc/init# When listing the processes running, I see some plymouth-related processes which seem hung: ... root 2944 2715 0 08:34 ? 00:00:00 sshd: support [priv] support 3021 2944 0 08:34 ? 00:00:00 sshd: support@pts/3 support 3022 3021 0 08:34 pts/3 00:00:00 -bash otrs 3147 1 0 08:35 ? 00:00:01 /usr/bin/perl /opt/otrs//bin/otrs.Scheduler.pl -w 1 root 3153 1 0 08:35 ? 00:00:00 plymouth --ping root 3406 2 0 08:39 ? 00:00:00 [kworker/u66:0] root 3506 3022 0 08:42 pts/3 00:00:00 sudo su root 3507 3506 0 08:42 pts/3 00:00:00 su root 3508 3507 0 08:42 pts/3 00:00:00 bash root 3540 1679 0 08:43 ? 00:00:00 /bin/sh /etc/rc2.d/S70dns-clean start root 3651 3540 0 08:48 ? 00:00:00 plymouth --ping postfix 3667 2458 0 08:49 ? 00:00:00 smtpd -n smtp -t inet -u -c -o stress= -s 2 postfix 3668 2458 0 08:49 ? 00:00:00 proxymap -t unix -u postfix 3669 2458 0 08:49 ? 00:00:00 anvil -l -t unix -u -c root 3673 1584 0 08:49 ? 00:00:00 /bin/plymouth message --text=Booting system without full network configuration... root 3681 3508 0 08:49 pts/3 00:00:00 ps -eaf root@otrs-prod:/home/support# root@otrs-prod:/home/support# ps -eaf | grep ply root 367 1 0 08:33 ? 00:00:00 @sbin/plymouthd --mode=boot --attach-to-session root 368 1 0 08:33 ? 00:00:00 plymouth-upstart-bridge root 3153 1 0 08:35 ? 00:00:00 plymouth --ping root 3651 1 0 08:48 ? 00:00:00 plymouth --ping root 3822 3815 0 08:52 ? 00:00:00 plymouth --ping root 4042 4022 0 08:56 ? 00:00:00 plymouth --ping root 4050 3508 0 08:56 pts/3 00:00:00 grep --color=auto ply root@otrs-prod:/home/support root@otrs-prod:/home/support# strace -p 367 Process 367 attached write(12, "Waiting for network configuratio"..., 36 Workaround: =========== After a few more troubleshooting, failing to understand what the problem is, I decided to kill a "plymouth --ping" process. Eventually, it allowed the system to boot a little further (ie: I can see in the list of processes that initd/systemd/whatever launched a new startup script. For example, in the processes list here above, the S70dns-clean startup script would have vanished and a new one, eg. S70ppd-dns, would have shown up), before it hung again on a "plymouth --ping" process. I thus continued to kill those "plymouth --ping" each time allowing the system to boot a little further, until no "plymouth --ping" process would show up anymore and the system appeared to be completly started. A that time, I also killed "plymouthd" for the sake of completeness. Expected result: ================ The system should boot without human assistance. Supportive information: ======================= My system is a physicall server Dell PowerEdge R710 running Ubuntu 14.04 now running linux kernel 3.13.0-92-generic: root@otrs-prod:/etc/init# lsb_release -rd Description: Ubuntu 14.04.4 LTS Release: 14.04 root@otrs-prod:/etc/init# uname -a Linux otrs-prod 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux root@otrs-prod:/etc/init# It has 3 physical ethernet interfaces, with 2 bounded together. Here's my (edited) interfaces file: root@otrs-prod:/etc/init# cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto bond0 iface bond0 inet static address a.b.c.d netmask 255.255.255.224 gateway a.b.c.1 bond-slaves none bond_mode 1 bond_miimon 100 iface bond0 inet6 static address a:b:c:d::e netmask 64 gateway a:b:c:d::1 # Impletement split access up ip -f inet6 rule add from a:b:c:d::e table local_v6 up ip -f inet6 route add default via a:b:c:d::1 dev bond0 table local_v6 auto eth0 iface eth0 inet manual bond-master bond0 bond_mode 1 bond_miimon 100 auto eth1 iface eth1 inet manual bond-master bond0 bond_mode 1 bond_miimon 100 # The management interface auto eth2 iface eth2 inet static address f.g.h.i netmask 255.255.0.0 network f.g.0.0 broadcast f.g.255.255 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers s.t.u.v w.x.y.z dns-search fw.belnet.be up route add -net f.0.0.0/8 gw f.g.0.1 dev eth2 iface eth2 inet6 static address a:b:c:f::g netmask 64 up route -A inet6 add a:b:c:h::/56 gw a:b:c:f::1 dev eth2 root@otrs-prod:/etc/init# Finally, the system runs the OpenManage stuff: root@otrs-prod:/etc/init# dpkg -l "srvadmin*" Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-===============================-====================-====================-=================================================================== ii srvadmin-base 8.3.0 amd64 Meta package for installing the Server Agent ii srvadmin-deng 8.3.0-1 amd64 Dell OpenManage Data Engine ii srvadmin-deng-snmp 8.3.0-1 amd64 Dell OpenManage Data Engine SNMP ii srvadmin-hapi 8.3.0-1 amd64 Dell OpenManage Hardware Abstraction Programming Interface ii srvadmin-idrac-snmp 8.2.0-1 amd64 CLI utils from the management station to the iDRAC ii srvadmin-isvc 8.3.0-1 amd64 Dell OpenManage Instrumentation Services ii srvadmin-isvc-snmp 8.3.0-1 amd64 Dell OpenManage Instrumentation Services SNMP ii srvadmin-nvme 8.3.0 amd64 This package provides the libaries to manage NVMe devices. ii srvadmin-omacore 8.3.0-1 amd64 Server Administrator CLI ii srvadmin-omacs 8.3.0-2 amd64 Dell OpenManage Server Administrator OMACS ii srvadmin-omcommon 8.3.0-2 amd64 Dell OpenManage Server Administrator Common Framework ii srvadmin-omilcore 8.3.0 all Dell OpenManage Server Administrator Install Core ii srvadmin-ominst 8.3.0-1 amd64 OMINST ii srvadmin-realssd 7.4.0-1 amd64 RealSSD package for storage management ii srvadmin-rnasoap 7.4.0-1 amd64 Fluid Cache Management ii srvadmin-server-cli 8.3.0 amd64 Meta package for installing the Server Administrator server cli com ii srvadmin-server-snmp 8.3.0 amd64 Meta package for installing the Server Administrator snmp component ii srvadmin-smcommon 8.3.0 amd64 Storage Management common files for GUI and CLI ii srvadmin-storage 8.3.0 amd64 Storage Management accessors package ii srvadmin-storage-cli 8.3.0 amd64 Storage Management cli component un srvadmin-storage-populator <none> <none> (no description available) ii srvadmin-storage-snmp 8.3.0 amd64 Storage Management SNMP component ii srvadmin-storageservices 7.4.0 amd64 Meta package for installing the Server Administrator Storage Servic ii srvadmin-storageservices-cli 7.4.0 amd64 Meta package for storageservices-cli ii srvadmin-storageservices-snmp 7.4.0 amd64 Meta package for storageservices-snmp ii srvadmin-storelib 8.3.0 amd64 StoreLib package for storage management ii srvadmin-storelib-sysfs 8.3.0 amd64 Metapackage for libsysfs2 ii srvadmin-xmlsup 8.3.0-1 amd64 Dell OpenManage XML Support SDK root@otrs-prod:/etc/init# root@otrs-prod:/etc/init# cat /etc/apt/sources.list.d/openmanage.ubuntu.com.sources.list deb http://linux.dell.com/repo/community/ubuntu trusty openmanage root@otrs-prod:/etc/init# Plymouth is at latest available version: root@otrs-prod:/etc/init# dpkg -l "plymouth*" Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-===============================-====================-====================-=================================================================== ii plymouth 0.8.8-0ubuntu17.1 amd64 graphical boot animation and logger - main package un plymouth-theme <none> <none> (no description available) ii plymouth-theme-ubuntu-text 0.8.8-0ubuntu17.1 amd64 graphical boot animation and logger - ubuntu-logo theme root@otrs-prod:/etc/init# ** Affects: plymouth (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1606472 Title: boot scripts hang in plymouth --ping (aka: Plymouth's bug returns) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/1606472/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
