Hi again, No joy yet with spotting CA anomalies. Any additional tips there Rob?
Gentle bump Simon, are you confident that building a new replica won't fall foul of the below from the upgrade page (the schema part): Words of caution - Note that the server is in a *maintenance mode* during upgrade and does not respond to requests! - Schema or Directory Server <https://www.freeipa.org/page/Directory_Server> database object changes done during the upgrade are replicated to *all FreeIPA masters* Thanks again for the support, David On 15 November 2017 at 16:52, David Harvey <davidchar...@googlemail.com> wrote: > Thanks Rob, Simon, > > Rob, will check, but thought my cert system was healthy before. It's > relatively new (6months or less), and no sub-ca's involved.. Any specifics > on how to invoke the selftests in some manner that might provide digestible > output? Or could it be my dirty hack of cloning and isolation and I should > do as Simon suggested :)? > > Simon. WRT spinning up a replica. I was under the impression that all > running servers had to be of the same version, am I mistaken with that? > I had avoided what you were suggesting as I feared the new server might > update the schema on the existing ones! > > Thanks again, appreciate the steering! > > > On 15 Nov 2017 14:34, "Rob Crittenden" <rcrit...@redhat.com> wrote: > > David Harvey via FreeIPA-users wrote: > > Sorry for the dump size, but not sure if the below from > > /var/log/pki/pki-tomcat/localhost.date.log helps: > > Looks like the selftests are failing. I'd check that your CA subsystem > certificates are not expired, etc. > > rob > > > > > 15-Nov-2017 12:14:50.557 SEVERE [localhost-startStop-1] > > org.apache.catalina.core.ApplicationContext.log > StandardWrapper.Throwable > > java.lang.NullPointerException > > at > > com.netscape.cmscore.selftests.SelfTestSubsystem.shutdown( > SelfTestSubsystem.java:1886) > > at > > com.netscape.cmscore.apps.CMSEngine.shutdownSubsystems(CMSEn > gine.java:2118) > > at com.netscape.cmscore.apps.CMSEngine.shutdown(CMSEngine.java:2013) > > at com.netscape.certsrv.apps.CMS.shutdown(CMS.java:234) > > at com.netscape.certsrv.apps.CMS.start(CMS.java:1630) > > at > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartS > ervlet.java:114) > > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > > at > > org.apache.catalina.core.StandardWrapper.initServlet(Standar > dWrapper.java:1227) > > at > > org.apache.catalina.core.StandardWrapper.loadServlet(Standar > dWrapper.java:1140) > > at org.apache.catalina.core.StandardWrapper.load(StandardWrappe > r.java:1027) > > at > > org.apache.catalina.core.StandardContext.loadOnStartup(Stand > ardContext.java:5038) > > at > > org.apache.catalina.core.StandardContext.startInternal(Stand > ardContext.java:5348) > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > > at > > org.apache.catalina.core.ContainerBase.addChildInternal(Cont > ainerBase.java:753) > > at org.apache.catalina.core.ContainerBase.addChild(ContainerBas > e.java:729) > > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) > > at > > org.apache.catalina.startup.HostConfig.deployDescriptor(Host > Config.java:621) > > at > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run( > HostConfig.java:1835) > > at java.util.concurrent.Executors$RunnableAdapter.call( > Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 15-Nov-2017 12:14:50.558 SEVERE [localhost-startStop-1] > > org.apache.catalina.core.StandardContext.loadOnStartup Servlet [castart] > > in web application [/ca] threw load() exception > > java.lang.NullPointerException > > at > > com.netscape.cmscore.selftests.SelfTestSubsystem.shutdown( > SelfTestSubsystem.java:1886) > > at > > com.netscape.cmscore.apps.CMSEngine.shutdownSubsystems(CMSEn > gine.java:2118) > > at com.netscape.cmscore.apps.CMSEngine.shutdown(CMSEngine.java:2013) > > at com.netscape.certsrv.apps.CMS.shutdown(CMS.java:234) > > at com.netscape.certsrv.apps.CMS.start(CMS.java:1630) > > at > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartS > ervlet.java:114) > > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > > at > > org.apache.catalina.core.StandardWrapper.initServlet(Standar > dWrapper.java:1227) > > at > > org.apache.catalina.core.StandardWrapper.loadServlet(Standar > dWrapper.java:1140) > > at org.apache.catalina.core.StandardWrapper.load(StandardWrappe > r.java:1027) > > at > > org.apache.catalina.core.StandardContext.loadOnStartup(Stand > ardContext.java:5038) > > at > > org.apache.catalina.core.StandardContext.startInternal(Stand > ardContext.java:5348) > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > > at > > org.apache.catalina.core.ContainerBase.addChildInternal(Cont > ainerBase.java:753) > > at org.apache.catalina.core.ContainerBase.addChild(ContainerBas > e.java:729) > > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) > > at > > org.apache.catalina.startup.HostConfig.deployDescriptor(Host > Config.java:621) > > at > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run( > HostConfig.java:1835) > > at java.util.concurrent.Executors$RunnableAdapter.call( > Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 15-Nov-2017 12:14:54.509 SEVERE [http-bio-8443-exec-1] > > org.apache.catalina.core.StandardHostValve.invoke Exception Processing > > /ca/rest/account/login > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > at > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(P > roxyRealm.java:138) > > at > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(A > uthenticatorBase.java:498) > > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHo > stValve.java:141) > > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorRepo > rtValve.java:79) > > at > > org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abs > tractAccessLogValve.java:620) > > at > > org.apache.catalina.core.StandardEngineValve.invoke(Standard > EngineValve.java:88) > > at > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAd > apter.java:502) > > at > > org.apache.coyote.http11.AbstractHttp11Processor.process(Abs > tractHttp11Processor.java:1132) > > at > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler > .process(AbstractProtocol.java:684) > > at > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run( > JIoEndpoint.java:283) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:624) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable. > run(TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:748) > > > > 15-Nov-2017 13:05:55.874 SEVERE [localhost-startStop-1] > > org.apache.catalina.core.ApplicationContext.log > StandardWrapper.Throwable > > java.lang.NullPointerException > > at > > com.netscape.cmscore.selftests.SelfTestSubsystem.shutdown( > SelfTestSubsystem.java:1886) > > at > > com.netscape.cmscore.apps.CMSEngine.shutdownSubsystems(CMSEn > gine.java:2118) > > at com.netscape.cmscore.apps.CMSEngine.shutdown(CMSEngine.java:2013) > > at com.netscape.certsrv.apps.CMS.shutdown(CMS.java:234) > > at com.netscape.certsrv.apps.CMS.start(CMS.java:1630) > > at > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartS > ervlet.java:114) > > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > > at > > org.apache.catalina.core.StandardWrapper.initServlet(Standar > dWrapper.java:1227) > > at > > org.apache.catalina.core.StandardWrapper.loadServlet(Standar > dWrapper.java:1140) > > at org.apache.catalina.core.StandardWrapper.load(StandardWrappe > r.java:1027) > > at > > org.apache.catalina.core.StandardContext.loadOnStartup(Stand > ardContext.java:5038) > > at > > org.apache.catalina.core.StandardContext.startInternal(Stand > ardContext.java:5348) > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > > at > > org.apache.catalina.core.ContainerBase.addChildInternal(Cont > ainerBase.java:753) > > at org.apache.catalina.core.ContainerBase.addChild(ContainerBas > e.java:729) > > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) > > at > > org.apache.catalina.startup.HostConfig.deployDescriptor(Host > Config.java:621) > > at > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run( > HostConfig.java:1835) > > at java.util.concurrent.Executors$RunnableAdapter.call( > Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 15-Nov-2017 13:05:55.875 SEVERE [localhost-startStop-1] > > org.apache.catalina.core.StandardContext.loadOnStartup Servlet [castart] > > in web application [/ca] threw load() exception > > java.lang.NullPointerException > > at > > com.netscape.cmscore.selftests.SelfTestSubsystem.shutdown( > SelfTestSubsystem.java:1886) > > at > > com.netscape.cmscore.apps.CMSEngine.shutdownSubsystems(CMSEn > gine.java:2118) > > at com.netscape.cmscore.apps.CMSEngine.shutdown(CMSEngine.java:2013) > > at com.netscape.certsrv.apps.CMS.shutdown(CMS.java:234) > > at com.netscape.certsrv.apps.CMS.start(CMS.java:1630) > > at > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartS > ervlet.java:114) > > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > > at > > org.apache.catalina.core.StandardWrapper.initServlet(Standar > dWrapper.java:1227) > > at > > org.apache.catalina.core.StandardWrapper.loadServlet(Standar > dWrapper.java:1140) > > at org.apache.catalina.core.StandardWrapper.load(StandardWrappe > r.java:1027) > > at > > org.apache.catalina.core.StandardContext.loadOnStartup(Stand > ardContext.java:5038) > > at > > org.apache.catalina.core.StandardContext.startInternal(Stand > ardContext.java:5348) > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > > at > > org.apache.catalina.core.ContainerBase.addChildInternal(Cont > ainerBase.java:753) > > at org.apache.catalina.core.ContainerBase.addChild(ContainerBas > e.java:729) > > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) > > at > > org.apache.catalina.startup.HostConfig.deployDescriptor(Host > Config.java:621) > > at > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run( > HostConfig.java:1835) > > at java.util.concurrent.Executors$RunnableAdapter.call( > Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 15-Nov-2017 13:05:59.706 SEVERE [http-bio-8443-exec-1] > > org.apache.catalina.core.StandardHostValve.invoke Exception Processing > > /ca/rest/account/login > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > at > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(P > roxyRealm.java:138) > > at > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(A > uthenticatorBase.java:498) > > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHo > stValve.java:141) > > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorRepo > rtValve.java:79) > > at > > org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abs > tractAccessLogValve.java:620) > > at > > org.apache.catalina.core.StandardEngineValve.invoke(Standard > EngineValve.java:88) > > at > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAd > apter.java:502) > > at > > org.apache.coyote.http11.AbstractHttp11Processor.process(Abs > tractHttp11Processor.java:1132) > > at > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler > .process(AbstractProtocol.java:684) > > at > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run( > JIoEndpoint.java:283) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:624) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable. > run(TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:748) > > > > > > On 15 November 2017 at 13:23, David Harvey <davidchar...@googlemail.com > > <mailto:davidchar...@googlemail.com>> wrote: > > > > Hi wisdom of the list, > > > > I know I am an edge case with running on ubuntu, but hoped someone > > might be able to shed some light. > > > > A bit of background. I'm trying to test upgrades without > > potentially hosing my existing services, so I have cloned the VM, > > given it a new IP address, updated hosts file and pointed DNS > > somewhere that doesn't know about the real IPA services (8.8.8.8) so > > it won't try and sync or replicate. > > > > Attempting to upgrade hits a snags or two, some described in bugs > > already like the pki version number confusing the apt > > scripts https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/17030 > 51 > > <https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1703051> ). > > The one I can't work around however is below. > > > > It seems deeply unhappy, and restarting the services result in the > > dogtag-pki web page being available until a login attempt is made > > (as occurs during the ipa-server-upgrade) after which point it bombs > > with a 500 error. > > > > Could the below caused > > by https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1716842 > > <https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1716842> ? > > > > Any advice appreciated, as I think even when 18.04 hits with the > > proposed updates to rely on to tomcat 8.5, I'll still need to > > upgrade via 17.10 which seems currently fraught! If it relates to > > my method of cloning the VM, is there a better way of testing > > upgrades without potentially hosing the existing live systems? > > > > > > Thanks in advance, > > > > David > > > > 2017-11-15T13:05:59Z DEBUG approved_usage = SSL Server > > intended_usage = SSL Server > > 2017-11-15T13:05:59Z DEBUG cert valid True for "CN=ipa1.my.net > > <http://ipa1.my.net>,O=THOMAC.NET <http://THOMAC.NET>" > > 2017-11-15T13:05:59Z DEBUG handshake complete, peer = IPADDRESS > > 2017-11-15T13:05:59Z DEBUG Protocol: TLS1.2 > > 2017-11-15T13:05:59Z DEBUG Cipher: TLS_RSA_WITH_AES_128_CBC_SHA > > 2017-11-15T13:05:59Z DEBUG response status 500 > > 2017-11-15T13:05:59Z DEBUG response headers {'content-length': > > '2292', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', > > 'connection': 'close', 'date': 'Wed, 15 Nov 2017 13:05:59 GMT', > > 'content-type': 'text/html;charset=utf-8'} > > 2017-11-15T13:05:59Z DEBUG response body '<!DOCTYPE > > html><html><head><title>Apache Tomcat/8.0.46 (Ubuntu) - Error > > report</title><style type="text/css">H1 > > {font-family:Tahoma,Arial,sans-serif;color:white;background > -color:#525D76;font-size:22px;} > > H2 > > {font-family:Tahoma,Arial,sans-serif;color:white;background > -color:#525D76;font-size:16px;} > > H3 > > {font-family:Tahoma,Arial,sans-serif;color:white;background > -color:#525D76;font-size:14px;} > > BODY > > {font-family:Tahoma,Arial,sans-serif;color:black;background > -color:white;} > > B > > {font-family:Tahoma,Arial,sans-serif;color:white;background > -color:#525D76;} > > P > > {font-family:Tahoma,Arial,sans-serif;background:white;color > :black;font-size:12px;}A > > {color : black;}A.name {color : black;}.line {height: 1px; > > background-color: #525D76; border: none;}</style> > > </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><div > > class="line"></div><p><b>type</b> Exception > > report</p><p><b>message</b> <u>Subsystem > > unavailable</u></p><p><b>description</b> <u>The server encountered > > an internal error that prevented it from fulfilling this > > request.</u></p><p><b>exception</b></p><pre>javax.ws.rs > > <http://ws.rs>.ServiceUnavailableException: Subsystem > > unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecur > ityConstraints(ProxyRealm.java:138)\n\torg.apache. > catalina.authenticator.AuthenticatorBase.invoke(Authenticato > rBase.java:498)\n\torg.apache.catalina.valves.ErrorReportVal > ve.invoke(ErrorReportValve.java:79)\n\torg.apache. > catalina.valves.AbstractAccessLogValve.invoke(AbstractAccess > LogValve.java:620)\n\torg.apache.catalina.connector.CoyoteAdapter.service( > CoyoteAdapter.java:502)\n\torg.apache.coyote.http11.Abst > ractHttp11Processor.process(AbstractHttp11Processor.java:113 > 2)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionH > andler.process(AbstractProtocol.java:684)\n\torg.apache.tomcat.util.net. > JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:283)\n\ > tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo > lExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExec > utor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache. > tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\ > tjava.lang.Thread.run(Thread.java:748)\n</pre><p><b>note</b> > > <u>The full stack trace of the root cause is available in the Apache > > Tomcat/8.0.46 (Ubuntu) logs.</u></p><hr class="line"><h3>Apache > > Tomcat/8.0.46 (Ubuntu)</h3></body></html>' > > 2017-11-15T13:05:59Z ERROR IPA server upgrade failed: Inspect > > /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. > > 2017-11-15T13:05:59Z DEBUG File > > "/usr/lib/python2.7/dist-packages/ipapython/admintool.py", line 172, > > in execute > > return_value = self.run() > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/install/ipa_ > server_upgrade.py", > > line 46, in run > > server.upgrade() > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/install/server/ > upgrade.py", > > line 1878, in upgrade > > upgrade_configuration() > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/install/server/ > upgrade.py", > > line 1797, in upgrade_configuration > > ca_enable_ldap_profile_subsystem(ca) > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/install/server/ > upgrade.py", > > line 347, in ca_enable_ldap_profile_subsystem > > cainstance.migrate_profiles_to_ldap() > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/install/cainstance.py", > > line 1981, in migrate_profiles_to_ldap > > _create_dogtag_profile(profile_id, profile_data, > overwrite=False) > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/install/cainstance.py", > > line 1987, in _create_dogtag_profile > > with api.Backend.ra_certprofile as profile_api: > > File > > "/usr/lib/python2.7/dist-packages/ipaserver/plugins/dogtag.py", line > > 1294, in __enter__ > > raise errors.RemoteRetrieveError(reason=_('Failed to > > authenticate to CA REST API')) > > > > 2017-11-15T13:05:59Z DEBUG The ipa-server-upgrade command failed, > > exception: RemoteRetrieveError: Failed to authenticate to CA REST API > > 2017-11-15T13:05:59Z ERROR Unexpected error - see > > /var/log/ipaupgrade.log for details: > > RemoteRetrieveError: Failed to authenticate to CA REST API > > > > > > > > > > _______________________________________________ > > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > > To unsubscribe send an email to freeipa-users-le...@lists.fedo > rahosted.org > > > > >
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
