Ade Lee wrote:
On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote:Martin Kosek wrote:On 08/31/2012 04:53 PM, Petr Viktorin wrote:On 08/28/2012 03:40 PM, Petr Viktorin wrote:On 08/17/2012 06:04 PM, Ade Lee wrote:On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote:On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote:On 08/16/2012 01:28 PM, Ade Lee wrote:Patch attached this time. I should know better than to do this in the middle of the night ..On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote:On 08/16/2012 07:53 AM, Ade Lee wrote:On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote:On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote:On 08/15/2012 03:54 PM, Ade Lee wrote:On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote:On 08/08/2012 10:05 PM, Ade Lee wrote:Hi, Dogtag 10 is being released on f18, and has a number of changes that will affect IPA. In particular, the following changes will affect current IPA code. * The directory layout of the dogtag instance has changed. Instead of using separate tomcat instances to host different subsystems, the standard dogtag installation will allow one to install a CA. KRA, OCSP and TKS within the same instance. There have been corresponding changes in the directory layout, as well as the default instance name (pki-tomcat instead of pki-ca), and startup daemon (pki-tomcatd, instead of pki-cad, pki-krad etc.) * The default instance will use only four ports (HTTPS, HTTP, AJP and tomcat shutdown port) rather than the 6 previously used. The default ports will be changed to the standard tomcat ports. As these ports are local to the ipa server machine, this should not cause too much disruption. * There is a new single step installer written in python. (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. * Dogtag 10 runs on tomcat7 - with a new corresponding version of tomcatjss. The attached patch integrates all the above changes in IPA installation and maintenance code. Once the patch is applied, users will be able to: 1. run ipa-server-install to completion on f18 with dogtag 10. 2. install a new replica on f18 on dogtag 10. 3. upgrade an f17 machine with an existing IPA instance to f18/ dogtag 10 - and have that old-style dogtag instance continue to run correctly. This will require the installation of the latest version of tomcatjss as well as the installation of tomcat6. The old-style instance will continue to use tomcat6. 4. in addition, the new cert renewal code has been patched and should continue to work. What is not yet completed / supported: 1. Installation with an external CA is not yet completed in the new installer. We plan to complete this soon. 2. There is some IPA upgrade code that has not yet been touched (install/tools/ipa-upgradeconfig). 3. A script needs to be written to allow admins to convert their old-style dogtag instances to new style instances, as well as code to periodically prompt admins to do this. 4. Installation of old-style instances using pkicreate/pkisilent on dogtag 10 will no longer be supported, and will be disabled soon. 5. The pki-selinux policy has been updated to reflect these changes, but is still in flux. In fact, it is our intention to place the dogtag selinux policy in the base selinux policy for f18. In the meantime, it may be necessary to run installs in permissive mode. The dogtag 10 code will be released shortly into f18. Prior to that though, we have placed the new dogtag 10 and tomcatjss code in a developer repo that is located at http://nkinder.fedorapeople.org/dogtag-devel/ Testing can be done on both f18 and f17 - although the target platform - and the only platform for which official builds will be created is f18. Thanks, AdeHi Ade, Thanks for the patch, I started with review and integration tests (currently running on Fedora 17 with Nathan's repo). Installation on single master was smooth, it worked just fine, even with enforced SELinux, without any error - kudos to you and the whole dogtag team. The resulting logs and the structure of your log directory seems improved. I believe that the brand new Python installers will make it easier to debug issues with dogtag installation when they come. When I tried our unit tests or some simple cert operation, it worked fine as well. Now the bad news, or rather few issues and suggestions I found: 1) As we already discussed on IRC, tomcat 7 was not pulled automatically on Fedora 17 when I updated pki-ca, you somewhere miss a Requires.We have a dogtag patch that is currently in review that will address this. Once this is in, tomcatjss >=7.0.0 will be required for f17+, rather than f18+2) I had installed IPA with dogtag10 on master. However, CA installation on a replica (ipa-ca-install) with dogtag9 failed - is this expectable?Yes. The current IPA patch is designed to work with dogtag 10 only, which will be officially available on f18+. So if you update to dogtag 10, you must have this patch and visa versa. We probably need to add the relevant requires to IPA to enforce this. If you have an existing dogtag 9 instance, and you upgrade to the new dogtag 10 and patched IPA, then that instance will continue to work. But any new instances would be created using dogtag 10.3) I had installed IPA with dogtag10 on master. Replica had dogtag10 as well and I got the following error: # ipa-ca-install /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg ... Configuring certificate server: Estimated time 3 minutes 30 seconds [1/14]: creating certificate server user [2/14]: configuring certificate server instance Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Unexpected error - see /var/log/ipareplica-ca-install.log for details: IOError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' Root cause: ... File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 625, in __spawn_instance "/root/cacert.p12") ...I need to look into this. I had fixed ipa-replica-install, rather than ipa-ca-install to create replicas. I didn't know ipa-ca-install existed! Should not be too bad to fix though - most likely just need to move files to the right place.Ok, thanks! Btw. CA on replica can be either installed during ipa-replica-install time (when --setup-ca option is passed, you probably used that one) and the aforementioned ipa-ca-install run after ipa-replica-install.I will be testing this out again. As ipa-ca-install uses the same code as ipa-replica-install, I would have expected it to work. Did you try it with selinux in permissive mode?OK -- I tested this out and realized that between the time I initially tested this and your tests, we had changed a URL from /ca/pki/installer/installToken to /ca/rest/installer/installToken, but I forgot to update ipa-pki-proxy.conf. The new patch has been rebased and changes the relevant patch. With this, the CA replica installs correctly.Umh, where can I find the new rebased patch? :-)There was one issue that I encountered. I have seen this once before and will need to investigate further. IPA sometimes restarts the dogtag instance by doing systemctl stop pki-tomcatd.target. and then systemctl start pki-tomcatd.target. I have noticed that occasionally the start invocation fails, while the stop and restart invocations succeed just fine. This makes absolutely no sense because restart is essentially just stop and then start. I think this is actually a bug in systemd. If I reboot the machine, it all works perfectly. If we can't figure out whats going on with systemd, we can always replace this start/stop with starting/stopping the service directly. (pki-tomcatd@pki-tomcat.service). That seems to work all the time with no problems. I suggest we leave this fix (if needed) for a separate patch.Ok, I will see if I hit this issue again. Btw. is it recommended in systemd to use target units this way? I thought that only service files are meant to be used for manipulation with a service. As you said yourself, using service unit file for restart worked every time. MartinNow, I tested the rebased patch with VMs with permissive SELinux. IPA server install was OK, as always, but replica installation ended up with error. Although it got much further than before: # ipa-ca-install /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg ... [3/3]: restarting directory server done configuring pkids. Configuring certificate server: Estimated time 3 minutes 30 seconds [1/14]: creating certificate server user [2/14]: configuring certificate server instance [3/14]: disabling nonces [4/14]: importing CA chain to RA certificate database [5/14]: fixing RA database permissions [6/14]: setting up signing cert profile [7/14]: set up CRL publishing [8/14]: set certificate subject base [9/14]: enabling Subject Key Identifier [10/14]: configuring certificate server to start on boot [11/14]: configure certmonger for renewals [12/14]: configure clone certificate renewals [13/14]: configure Server-Cert certificate renewal [14/14]: Configure HTTP to proxy connections done configuring pki-tomcatd. Restarting the directory and certificate servers Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. It seems like that CA on a replica did not start and so a check for port 8080 failed. Attached logs (pki-ca-logs.tgz) may provide more information to this issue.This is the systemd issue I mentioned before. I will submit a patch which will change the restart mechanism to restart the service and not the target.Patch attached. This patch should be applied on top of the large patch (being reviewed).Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10 (permissive SELinux). Now, the service was upgraded smoothly, CA service could be restarted without failure. However, certificate operations now did not work: # ipactl restart Restarting Directory Service Restarting KDC Service Restarting KPASSWD Service Restarting MEMCACHE Service Restarting HTTP Service Restarting CA Service # ipactl status Directory Service: RUNNING KDC Service: RUNNING KPASSWD Service: RUNNING MEMCACHE Service: RUNNING HTTP Service: RUNNING CA Service: RUNNING # ipa cert-show 1 ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz).The reason for this is that the old dogtag instances are missing: 1. a new parameter in CS.cfg 2. a couple of symlinks I plan to fix this in the dogtag code. Specifically, 1. I will modify the dogtag code to provide a default value in the case where the parameter does not exist. 2. I plan to add a function to the dogtag startup scripts to check and remake any required symlinks. Both of these changes are in the dogtag code, and do not affect this patch. As a workaround, to confirm that the upgrade actually works, do the following manual steps on your upgraded instance: 1. Add the following parameter to CS.cfg pidDir=/var/run 2. Add the following links; cd /var/lib/pki-ca/common/lib ln -s /usr/share/java/apache-commons-codec.jar apache-commons-codec.jar ln -s /usr/share/java/log4j.jar log4j.jar 3 Restart the dogtag instanceHTH, MartinThe attached patch conditionalizes the changes, so that IPA supports both Dogtag versions. I didn't put it in platform code for two reasons - One platform (f17) can have either Dogtag version installed - I didn't want to conflict with Timo Aaltonen's "platformizing" work. According to his WIP patches, he plans to move the whole dogtag module into platform code. I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If you know a better way, please comment. The dogtag_version option is now always added to I've reverted the changes to install/ui/test/data/ipa_init.json, so you'll have to change the ports manually when testing the UI with Dogtag 10.Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC.This approach looks good. I did not hit any regression on F17 with dogtag9, or clean installs of IPA+dogtag10. I think we are getting close to pushing this code. The only issues I hit so far are for upgraded dogtag9 instances (on F17): 1) The service did not start for me. There were some SELinux AVCs and even when I disabled SELinux the instance still did not start (logs attached). 2) Uninstall was also not clean, we leave some dogtag installation states for upgraded dogtag9 instance: # ipa-server-install --uninstall --unattended Shutting down all IPA services Removing IPA client configuration Unconfiguring ntpd Unconfiguring CA directory server Unconfiguring web server Unconfiguring krb5kdc Unconfiguring kadmin Unconfiguring directory server Unconfiguring ipa_memcached ipa : ERROR Some installation state for pki-cad has not been restored, see /var/lib/ipa/sysrestore/sysrestore.state /var/lib/ipa/sysrestore/sysrestore.state: [pki-cad] enabled = FalseAde is working on a new build to address the dogtag upgrade issues. The left over state should be removed when we drop the separate instance in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch who actually does this part, is this handled by dogtag? robThe build is available on the developer repo. Just do a yum update. As for this leftover state, this is an ipa file and such is never handled by dogtag. It must be handled somewhere within the ipa code.
Well, the question is, what handles the dogtag 9 -> dogtag 10 upgrade? If IPA isn't involved then it has no chance to remove this state. rob _______________________________________________ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
