Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
On Wed, May 22, 2019 at 1:41 PM Jon LaBadie wrote: > But we can blame systemd for the cryptic message > > A stop job is running > > Surely systemd knows what service it is waiting for, > why doesn't it tell us? > > The stop job XYZ is running The message reported by the OP and the message I see is 'A stop job is running for ...' where the ellipses stand in for the unit that systemd is waiting for. It seems pretty clear IMHO. Actually debugging it is harder since the system is not available during shutdown, but that's a generic problem and the systemd docs do provide debugging tips. Jim ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
Scott Robbins wrote: > ** WARNING: This mail is from an external source ** > > > On Wed, May 22, 2019 at 03:41:24PM -0400, Jon LaBadie wrote: >> On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger wrote: >>> On Wed, May 22, 2019 at 7:44 AM mark wrote: The joys of systemd >>> >>> I'm not sure it's right to blame systemd. Systemd asked nicely for >>> the service to shutdown. >> >> But we can blame systemd for the cryptic message >> >>A stop job is running > > I didn't read this thread all that carefully, but has anyone mentioned > editing /etc/systemd/system.conf and changing DefaultTimeoutStartSec and > DefaultTimeoutStopSec to a lower value? All the entries in /etc/systemd/system.conf are commented out - and as the systemd-system.conf man page states: By default the configuration file in /etc/systemd/ contains commented out entries showing the defaults as a guide to the administrator The DefaultTimeoutStopSec line in /etc/systemd/system.conf is: #DefaultTimeoutStartSec=90s In my case, the 'timeout' on whatever it was trying to do was 90 seconds - but it kept being increased by ~90 secs until it finally gave up at 30 minutes ... So I don't think changing DefaultTimeoutStartSec will help here ? There is no mention of 30 minutes in /etc/systemd/system.conf, that is why I'm guessing it has something to do with the JobTimeoutSec in /usr/lib/systemd/system/reboot.target A bit of (more) googling seems to suggest that this is the setting that needs to be changed. I believe this means something like 'after 30 minutes from starting the reboot process, force a reboot regardless' Personally, I think 30 minutes is far too long to wait, especially in the case of servers where no console access is available ... James Pearson ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
On Wed, May 22, 2019 at 03:41:24PM -0400, Jon LaBadie wrote: > On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger wrote: > > On Wed, May 22, 2019 at 7:44 AM mark wrote: > > > > > > The joys of systemd > > > > I'm not sure it's right to blame systemd. Systemd asked nicely for > > the service to shutdown. > > But we can blame systemd for the cryptic message > > A stop job is running I didn't read this thread all that carefully, but has anyone mentioned editing /etc/systemd/system.conf and changing DefaultTimeoutStartSec and DefaultTimeoutStopSec to a lower value? -- Scott Robbins PGP keyID EB3467D6 ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 ) gpg --keyserver pgp.mit.edu --recv-keys EB3467D6 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger wrote: > On Wed, May 22, 2019 at 7:44 AM mark wrote: > > > > The joys of systemd > > I'm not sure it's right to blame systemd. Systemd asked nicely for > the service to shutdown. But we can blame systemd for the cryptic message A stop job is running Surely systemd knows what service it is waiting for, why doesn't it tell us? The stop job XYZ is running jon -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C) ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
Once upon a time, James Szinger said: > On Wed, May 22, 2019 at 7:44 AM mark wrote: > > The joys of systemd > > I'm not sure it's right to blame systemd. Systemd asked nicely for > the service to shutdown. The service didn't, probably because the > update change something and pulled the rug out from beneath it. Right - before systemd, any old init script could also block shutdown. > This hasn't happened to me recently, but I think I've tried Ctl-C and > Ctl-Alt-Del without much success. That leaves the Big Red Switch > (which is mostly small and black these days). There's a "magic" thing systemd does now - hit C-A-D seven times in two seconds and it'll stop what it is waiting for and just go ahead and reboot. Will kill anything not shut down, but at least it'll still try to cleanly unmount filesystems and such I believe. -- Chris Adams ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
On Wed, May 22, 2019 at 7:44 AM mark wrote: > > The joys of systemd I'm not sure it's right to blame systemd. Systemd asked nicely for the service to shutdown. The service didn't, probably because the update change something and pulled the rug out from beneath it. Systemd then waited a bit to make sure the service wasn't just being slow, and finally gave up and forcibly killed it. I think this is a reasonable approach to killing a misbehaving service while trying to minimize data-loss, and the timeout can be configured. This hasn't happened to me recently, but I think I've tried Ctl-C and Ctl-Alt-Del without much success. That leaves the Big Red Switch (which is mostly small and black these days). Jim ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
mark wrote: > James Pearson wrote: >> James Pearson wrote: >> >>> >>> I'm currently trying to reboot a CentOS 7.5 workstation (to complete an >>> upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop >>> job is running for ...' - the counter initially gave a limit of '1min >>> 30s' - >>> but each time it reaches that limit, it just adds on ~90 seconds to the >>> limit ... >>> >>> Currently the limit is '25min 33s' >>> >>> >>> I'm in no hurry to have this workstation operational, but I guess at >>> some point I will have to power cycle it ... >>> >>> Does anyone know how to bypass this? - or at least stop it increasing >>> the limit each time it is reached? >>> >>> It does seems rather pointless to keep increasing the limit like this >>> ... >>> >> It _finally_ gave up at 30 mins and rebooted > > One question: did it have a mounted nfs filesystem? All our boxes have NFS mounted files systems - and usually this isn't a problem - reboots work without an issue In this case, it appeared to be 'stuck' on a local file bind mounted over a file on an NFS mounted file system But that isn't really the point - I don't really want to have to wait a maximum of 30 minutes for the reboot to give up waiting for 'whatever' Poking about a bit, I see that /usr/lib/systemd/system/reboot.target has the line: JobTimeoutSec=30min (there is a similar JobTimeoutSec=30min in poweroff.target) I'm guessing I could create something like /etc/systemd/system/reboot.target.d/override.conf containing something like: [Unit] JobTimeoutSec=3min Now I need to see if I can reproduce the issue and see if this setting works ... James Pearson ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
On 22/05/2019 14:43, mark wrote: > James Pearson wrote: >> James Pearson wrote: >> >>> >>> I'm currently trying to reboot a CentOS 7.5 workstation (to complete an >>> upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop >>> job is running for ...' - the counter initially gave a limit of '1min >>> 30s' - >>> but each time it reaches that limit, it just adds on ~90 seconds to the >>> limit ... >>> >>> Currently the limit is '25min 33s' >>> >>> >>> I'm in no hurry to have this workstation operational, but I guess at >>> some point I will have to power cycle it ... >>> >>> Does anyone know how to bypass this? - or at least stop it increasing >>> the limit each time it is reached? >>> >>> It does seems rather pointless to keep increasing the limit like this >>> ... >>> >> It _finally_ gave up at 30 mins and rebooted > > One question: did it have a mounted nfs filesystem? > > The joys of systemd > > mark > > mark > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos > "Anything Windows can do, systemd can do better" (with apologies to Irving Berlin). -- J Martin Rushton MBCS signature.asc Description: OpenPGP digital signature ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
> James Pearson wrote: >> James Pearson wrote: >> >>> >>> I'm currently trying to reboot a CentOS 7.5 workstation (to complete an >>> upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop >>> job is running for ...' - the counter initially gave a limit of '1min >>> 30s' - >>> but each time it reaches that limit, it just adds on ~90 seconds to the >>> limit ... >>> >>> Currently the limit is '25min 33s' >>> >>> >>> I'm in no hurry to have this workstation operational, but I guess at >>> some point I will have to power cycle it ... >>> >>> Does anyone know how to bypass this? - or at least stop it increasing >>> the limit each time it is reached? >>> >>> It does seems rather pointless to keep increasing the limit like this >>> ... >>> >> It _finally_ gave up at 30 mins and rebooted > > One question: did it have a mounted nfs filesystem? > > The joys of systemd Yes, NFS integration with systemd is broken by default, at least it was still the case when I last checked. If you want it to work correctly, you have to add 'x-systemd.requires=network-online.target' as NFS mount option. Clearly, how should systemd know that NFS won't work without network? I knew you agree :-) Regards, Simon ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
James Pearson wrote: > James Pearson wrote: > >> >> I'm currently trying to reboot a CentOS 7.5 workstation (to complete an >> upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop >> job is running for ...' - the counter initially gave a limit of '1min >> 30s' - >> but each time it reaches that limit, it just adds on ~90 seconds to the >> limit ... >> >> Currently the limit is '25min 33s' >> >> >> I'm in no hurry to have this workstation operational, but I guess at >> some point I will have to power cycle it ... >> >> Does anyone know how to bypass this? - or at least stop it increasing >> the limit each time it is reached? >> >> It does seems rather pointless to keep increasing the limit like this >> ... >> > It _finally_ gave up at 30 mins and rebooted One question: did it have a mounted nfs filesystem? The joys of systemd mark mark ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
James Pearson wrote: > > I'm currently trying to reboot a CentOS 7.5 workstation (to complete an > upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job > is running for ...' - the counter initially gave a limit of '1min 30s' - > but each time it reaches that limit, it just adds on ~90 seconds to the > limit ... > > Currently the limit is '25min 33s' > > I'm in no hurry to have this workstation operational, but I guess at > some point I will have to power cycle it ... > > Does anyone know how to bypass this? - or at least stop it increasing > the limit each time it is reached? > > It does seems rather pointless to keep increasing the limit like this ... It _finally_ gave up at 30 mins and rebooted James Pearson ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] Bypassing 'A stop job is running' when rebooting CentOS 7
I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ... Currently the limit is '25min 33s' I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ... Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached? It does seems rather pointless to keep increasing the limit like this ... Thanks James Pearson ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
