Hi all, Fixed it, thanks for the tip Rob :-)! Certmonger was to blame or my rather slow Udooboard Celeron processor. Anyway, instead of hacking the upgrade script, I modified the certmonger.serivce file by adding a 180 secs (!!) sleep and extra Timeout: (The modified certmonger.service was removed after the upgrade)
[Unit] Description=Certificate monitoring and PKI enrollment After=syslog.target network.target dbus.service [Service] Type=dbus PIDFile=/var/run/certmonger.pid EnvironmentFile=-/etc/sysconfig/certmonger ExecStart=/usr/sbin/certmonger -S -p /var/run/certmonger.pid -n $OPTS ExecStartPost=/bin/sleep 180 TimeoutSec=240 BusName=org.fedorahosted.certmonger Runing "ipa-server-upgrade" finished OK now. Certmonger takes itś time when it's (restarted, some dogtag-ipa-ca-r(enew ?) processes eating most of the cpu: top - 16:00:24 up 18:51, 3 users, load average: 2.41, 1.87, 1.37 Tasks: 261 total, 6 running, 221 sleeping, 0 stopped, 34 zombie %Cpu0 : 90.2 us, 7.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.6 hi, 0.3 si, 0.0 st %Cpu1 : 92.4 us, 6.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.0 hi, 0.0 si, 0.0 st %Cpu2 : 95.1 us, 3.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 0.0 si, 0.0 st %Cpu3 : 88.6 us, 9.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 1.0 si, 0.0 st MiB Mem : 3847.2 total, 335.4 free, 2154.9 used, 1356.9 buff/cache MiB Swap: 3968.0 total, 3968.0 free, 0.0 used. 1452.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21750 root 20 0 401244 85296 22612 R 85.9 2.2 0:13.36 dogtag-ipa-ca-r 21764 root 20 0 386700 72880 22508 R 78.4 1.8 0:06.93 dogtag-ipa-ca-r 21771 root 20 0 161788 27332 10812 R 74.5 0.7 0:03.65 dogtag-ipa-ca-r 21758 root 20 0 394512 78340 22436 R 67.3 2.0 0:10.65 dogtag-ipa-ca-r 21746 root 20 0 0 0 0 Z 51.6 0.0 0:15.36 dogtag-ipa-ca-r 21778 root 20 0 106004 1220 0 R 24.8 0.0 0:00.76 certmonger This seems like a new issue for me... Certainly, the Udoo x86 isn't the fasted in the world, but was running IPA bravely the last year... Am I hitting the bug Rob mentioned? Is there a bug report somewhere to track... I'll like to see it fixed in CentOS 8. "getcert list" showed "/var/lib/ipa/private/httpd.key" and "/var/lib/ipa/certs/httpd.crt" wating for PIN. Running "ipa-getcert resubmit -i 20200126151811 -p /var/lib/ipa/passwds/ipa.xxx-443-RSA" fixed it. Winfried -----Oorspronkelijk bericht----- Van: Rob Crittenden <rcrit...@redhat.com> Aan: FreeIPA users list <freeipa-users@lists.fedorahosted.org> Cc: Winfried de Heiden <w...@dds.nl> Onderwerp: Re: [Freeipa-users] Re: ipa-server-upgrade failed Datum: Sat, 25 Jan 2020 17:04:39 -0500 Winfried de Heiden via FreeIPA-users wrote: > Hi all, > /var/lib/ipa/private/httpd.key was in a status "waiting for PIN", but > Idid brong is back to life using "ipa-getcert resubmit -i > 20200117075404-p /var/lib/ipa/passwds/xxxx-443-RSA. All certss look > fine now. "getcert list" works, although it's a bit slow the first > time (runningon a Udoo x86 board with a celeron....) > Just to be shure about dbus, I restarted the entire machine; no > success. :-( > Timing issue and/or casued by my rather slow Udoo board.....? It is very possible. I fixed an issue in certmonger where every time itforked (and it forks a LOT) it closed ALL the fds it knew about. Oncontainers this was 1M. It took a LONG time. The default is a moremodest 1k but can still take a while given the amount of forks thatcertmonger does. This is fixed upstream, and I don't know of aworkaround, but this can definitely lead to timeout issues if certmongeris being restarted immediately before this failure. To diagnose it see what the load on the system is and what processes arerunning. If you see dozens of certmonger processes with high load thenthat's probably it. You'd have to hack the update script to do a sleepto give things a chance to settle down. rob > Winfried > > > > > Rob Crittenden schreef op za 25-01-2020 om 14:53 [-0500]: > > Winfried de Heiden via FreeIPA-users wrote: > > > Hi all, > > > Using CentOS Linux release 8.1.1911 and the Stream > > > repositories,upgrading IPA fails: > > > ( Upgrade > > > ipa-server-common-4.8.0-13.module_el8.1.0+265+e1e65be4.noarch@AppStream > > > Upgradedipa-server-common-4.8.0- > > > 11.module_el8.1.0+253+3b90c921.noarch @@System ) > > > Running ipa-server-upgrade manually will result in: > > > [Upgrading CA schema]CA schema update complete (no > > > changes)[Verifying that CA audit signing cert has 2 year > > > validity][Update certmonger certificate renewal > > > configuration]Introspect error on > > > :1.417:/org/fedorahosted/certmonger:dbus.exceptions.DBusException > > > : org.freedesktop.DBus.Error.NoReply: Didnot receive a reply. > > > Possible causes include: the remote application didnot send a > > > reply, the message bus security policy blocked the reply, > > > thereply timeout expired, or the network connection was broken. > > > > I assume certmonger and dbus services are running? > > Does `getcert list` work? > > The dbus service sometimes isn't too fond of being restarted but > > youcould try that. > > rob > > _______________________________________________FreeIPA-users mailing > list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to > freeipa-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
