On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote: > Martin Kosek wrote: > > On 08/31/2012 04:53 PM, Petr Viktorin wrote: > >> On 08/28/2012 03:40 PM, Petr Viktorin wrote: > >>> On 08/17/2012 06:04 PM, Ade Lee wrote: > >>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: > >>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: > >>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: > >>>>>>> Patch attached this time. I should know better than to do this in the > >>>>>>> middle of the night .. > >>>>>>> > >>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: > >>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: > >>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: > >>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: > >>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: > >>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: > >>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of > >>>>>>>>>>>>>> changes that > >>>>>>>>>>>>>> will affect IPA. In particular, the following changes will > >>>>>>>>>>>>>> affect > >>>>>>>>>>>>>> current IPA code. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed. > >>>>>>>>>>>>>> Instead of > >>>>>>>>>>>>>> using separate tomcat instances to host different > >>>>>>>>>>>>>> subsystems, the > >>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA. > >>>>>>>>>>>>>> KRA, OCSP > >>>>>>>>>>>>>> and TKS within the same instance. There have been > >>>>>>>>>>>>>> corresponding changes > >>>>>>>>>>>>>> in the directory layout, as well as the default instance name > >>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon > >>>>>>>>>>>>>> (pki-tomcatd, instead > >>>>>>>>>>>>>> of pki-cad, pki-krad etc.) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, > >>>>>>>>>>>>>> HTTP, AJP and > >>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. > >>>>>>>>>>>>>> The default > >>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As > >>>>>>>>>>>>>> these ports are > >>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much > >>>>>>>>>>>>>> disruption. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * There is a new single step installer written in python. > >>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding > >>>>>>>>>>>>>> version of > >>>>>>>>>>>>>> tomcatjss. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA > >>>>>>>>>>>>>> installation > >>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users will > >>>>>>>>>>>>>> be able to: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10. > >>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. > >>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to > >>>>>>>>>>>>>> f18/ dogtag > >>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run > >>>>>>>>>>>>>> correctly. > >>>>>>>>>>>>>> This will require the installation of the latest version of > >>>>>>>>>>>>>> tomcatjss as > >>>>>>>>>>>>>> well as the installation of tomcat6. The old-style instance > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>> continue to use tomcat6. > >>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched > >>>>>>>>>>>>>> and should > >>>>>>>>>>>>>> continue to work. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> What is not yet completed / supported: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in > >>>>>>>>>>>>>> the new > >>>>>>>>>>>>>> installer. We plan to complete this soon. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched > >>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert > >>>>>>>>>>>>>> their > >>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well > >>>>>>>>>>>>>> as code to > >>>>>>>>>>>>>> periodically prompt admins to do this. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 4. Installation of old-style instances using > >>>>>>>>>>>>>> pkicreate/pkisilent on > >>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled > >>>>>>>>>>>>>> soon. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect these > >>>>>>>>>>>>>> changes, > >>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to place > >>>>>>>>>>>>>> the dogtag > >>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the > >>>>>>>>>>>>>> meantime, it > >>>>>>>>>>>>>> may be necessary to run installs in permissive mode. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. Prior > >>>>>>>>>>>>>> to that > >>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code > >>>>>>>>>>>>>> in a > >>>>>>>>>>>>>> developer repo that is located at > >>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the > >>>>>>>>>>>>>> target platform - > >>>>>>>>>>>>>> and the only platform for which official builds will be > >>>>>>>>>>>>>> created is f18. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> Ade > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Ade, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for the patch, I started with review and integration > >>>>>>>>>>>>> tests (currently > >>>>>>>>>>>>> running on Fedora 17 with Nathan's repo). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Installation on single master was smooth, it worked just > >>>>>>>>>>>>> fine, even with > >>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the > >>>>>>>>>>>>> whole dogtag team. > >>>>>>>>>>>>> The resulting logs and the structure of your log directory > >>>>>>>>>>>>> seems improved. I > >>>>>>>>>>>>> believe that the brand new Python installers will make it > >>>>>>>>>>>>> easier to debug > >>>>>>>>>>>>> issues with dogtag installation when they come. When I tried > >>>>>>>>>>>>> our unit tests or > >>>>>>>>>>>>> some simple cert operation, it worked fine as well. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled > >>>>>>>>>>>>> automatically on > >>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires. > >>>>>>>>>>>>> > >>>>>>>>>>>> We have a dogtag patch that is currently in review that will > >>>>>>>>>>>> address > >>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be required for > >>>>>>>>>>>> f17+, > >>>>>>>>>>>> rather than f18+ > >>>>>>>>>>>> > >>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA > >>>>>>>>>>>>> installation on a > >>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this > >>>>>>>>>>>>> expectable? > >>>>>>>>>>>>> > >>>>>>>>>>>> Yes. The current IPA patch is designed to work with dogtag 10 > >>>>>>>>>>>> only, > >>>>>>>>>>>> which will be officially available on f18+. So if you update > >>>>>>>>>>>> to dogtag > >>>>>>>>>>>> 10, you must have this patch and visa versa. We probably need > >>>>>>>>>>>> to add > >>>>>>>>>>>> the relevant requires to IPA to enforce this. > >>>>>>>>>>>> > >>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to > >>>>>>>>>>>> the new > >>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to > >>>>>>>>>>>> work. > >>>>>>>>>>>> But any new instances would be created using dogtag 10. > >>>>>>>>>>>> > >>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had > >>>>>>>>>>>>> dogtag10 as well > >>>>>>>>>>>>> and I got the following error: > >>>>>>>>>>>>> > >>>>>>>>>>>>> # ipa-ca-install > >>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 > >>>>>>>>>>>>> seconds > >>>>>>>>>>>>> [1/14]: creating certificate server user > >>>>>>>>>>>>> [2/14]: configuring certificate server instance > >>>>>>>>>>>>> > >>>>>>>>>>>>> Your system may be partly configured. > >>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for > >>>>>>>>>>>>> details: > >>>>>>>>>>>>> IOError: [Errno 2] No such file or directory: > >>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' > >>>>>>>>>>>>> > >>>>>>>>>>>>> Root cause: > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> File > >>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", > >>>>>>>>>>>>> line > >>>>>>>>>>>>> 625, in __spawn_instance > >>>>>>>>>>>>> "/root/cacert.p12") > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> > >>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, > >>>>>>>>>>>> rather than > >>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know ipa-ca-install > >>>>>>>>>>>> existed! Should not be too bad to fix though - most likely > >>>>>>>>>>>> just need to > >>>>>>>>>>>> move files to the right place. > >>>>>>>>>>> > >>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during > >>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you > >>>>>>>>>>> probably used > >>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after > >>>>>>>>>>> ipa-replica-install. > >>>>>>>>>>> > >>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the > >>>>>>>>>> same code > >>>>>>>>>> as ipa-replica-install, I would have expected it to work. Did > >>>>>>>>>> you try > >>>>>>>>>> it with selinux in permissive mode? > >>>>>>>>>> > >>>>>>>>> OK -- I tested this out and realized that between the time I > >>>>>>>>> initially > >>>>>>>>> tested this and your tests, we had changed a URL > >>>>>>>>> from /ca/pki/installer/installToken to > >>>>>>>>> /ca/rest/installer/installToken, > >>>>>>>>> but I forgot to update ipa-pki-proxy.conf. > >>>>>>>>> > >>>>>>>>> The new patch has been rebased and changes the relevant patch. With > >>>>>>>>> this, the CA replica installs correctly. > >>>>>>>> > >>>>>>>> Umh, where can I find the new rebased patch? :-) > >>>>>>>> > >>>>>>>>> > >>>>>>>>> There was one issue that I encountered. I have seen this once > >>>>>>>>> before > >>>>>>>>> and will need to investigate further. IPA sometimes restarts the > >>>>>>>>> dogtag > >>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then > >>>>>>>>> systemctl > >>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally the > >>>>>>>>> start > >>>>>>>>> invocation fails, while the stop and restart invocations succeed > >>>>>>>>> just > >>>>>>>>> fine. This makes absolutely no sense because restart is essentially > >>>>>>>>> just stop and then start. > >>>>>>>>> > >>>>>>>>> I think this is actually a bug in systemd. If I reboot the > >>>>>>>>> machine, it > >>>>>>>>> all works perfectly. If we can't figure out whats going on with > >>>>>>>>> systemd, we can always replace this start/stop with > >>>>>>>>> starting/stopping > >>>>>>>>> the service directly. (pki-tomcatd@pki-tomcat.service). That > >>>>>>>>> seems to > >>>>>>>>> work all the time with no problems. > >>>>>>>>> > >>>>>>>>> I suggest we leave this fix (if needed) for a separate patch. > >>>>>>>> > >>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended > >>>>>>>> in systemd to > >>>>>>>> use target units this way? I thought that only service files are > >>>>>>>> meant to be > >>>>>>>> used for manipulation with a service. As you said yourself, using > >>>>>>>> service unit > >>>>>>>> file for restart worked every time. > >>>>>>>> > >>>>>>>> Martin > >>>>>>> > >>>>>> > >>>>>> Now, I tested the rebased patch with VMs with permissive SELinux. > >>>>>> IPA server > >>>>>> install was OK, as always, but replica installation ended up with > >>>>>> error. > >>>>>> Although it got much further than before: > >>>>>> > >>>>>> # ipa-ca-install > >>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg > >>>>>> ... > >>>>>> [3/3]: restarting directory server > >>>>>> done configuring pkids. > >>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds > >>>>>> [1/14]: creating certificate server user > >>>>>> [2/14]: configuring certificate server instance > >>>>>> [3/14]: disabling nonces > >>>>>> [4/14]: importing CA chain to RA certificate database > >>>>>> [5/14]: fixing RA database permissions > >>>>>> [6/14]: setting up signing cert profile > >>>>>> [7/14]: set up CRL publishing > >>>>>> [8/14]: set certificate subject base > >>>>>> [9/14]: enabling Subject Key Identifier > >>>>>> [10/14]: configuring certificate server to start on boot > >>>>>> [11/14]: configure certmonger for renewals > >>>>>> [12/14]: configure clone certificate renewals > >>>>>> [13/14]: configure Server-Cert certificate renewal > >>>>>> [14/14]: Configure HTTP to proxy connections > >>>>>> done configuring pki-tomcatd. > >>>>>> Restarting the directory and certificate servers > >>>>>> > >>>>>> Your system may be partly configured. > >>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. > >>>>>> > >>>>>> It seems like that CA on a replica did not start and so a check for > >>>>>> port 8080 > >>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information > >>>>>> to this issue. > >>>>>> > >>>>> This is the systemd issue I mentioned before. I will submit a patch > >>>>> which will change the restart mechanism to restart the service and not > >>>>> the target. > >>>>>> > >>>> > >>>> Patch attached. This patch should be applied on top of the large patch > >>>> (being reviewed). > >>>> > >>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10 > >>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA > >>>>>> service > >>>>>> could be restarted without failure. However, certificate operations now > >>>>>> did not work: > >>>>>> > >>>>>> # ipactl restart > >>>>>> Restarting Directory Service > >>>>>> Restarting KDC Service > >>>>>> Restarting KPASSWD Service > >>>>>> Restarting MEMCACHE Service > >>>>>> Restarting HTTP Service > >>>>>> Restarting CA Service > >>>>>> # ipactl status > >>>>>> Directory Service: RUNNING > >>>>>> KDC Service: RUNNING > >>>>>> KPASSWD Service: RUNNING > >>>>>> MEMCACHE Service: RUNNING > >>>>>> HTTP Service: RUNNING > >>>>>> CA Service: RUNNING > >>>>>> # ipa cert-show 1 > >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>>> communicate with CMS (Internal Server Error) > >>>>>> > >>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). > >>>>>> > >>>>> The reason for this is that the old dogtag instances are missing: > >>>>> 1. a new parameter in CS.cfg > >>>>> 2. a couple of symlinks > >>>>> > >>>>> I plan to fix this in the dogtag code. Specifically, > >>>>> 1. I will modify the dogtag code to provide a default value in the case > >>>>> where the parameter does not exist. > >>>>> 2. I plan to add a function to the dogtag startup scripts to check and > >>>>> remake any required symlinks. > >>>>> > >>>>> Both of these changes are in the dogtag code, and do not affect this > >>>>> patch. As a workaround, to confirm that the upgrade actually works, do > >>>>> the following manual steps on your upgraded instance: > >>>>> > >>>>> 1. Add the following parameter to CS.cfg > >>>>> pidDir=/var/run > >>>>> > >>>>> 2. Add the following links; > >>>>> cd /var/lib/pki-ca/common/lib > >>>>> ln -s /usr/share/java/apache-commons-codec.jar > >>>>> apache-commons-codec.jar > >>>>> ln -s /usr/share/java/log4j.jar log4j.jar > >>>>> > >>>>> 3 Restart the dogtag instance > >>>>> > >>>>>> HTH, > >>>>>> Martin > >>>>> > >>>>> > >>> > >>> The attached patch conditionalizes the changes, so that IPA supports > >>> both Dogtag versions. > >>> > >>> I didn't put it in platform code for two reasons > >>> - One platform (f17) can have either Dogtag version installed > >>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work. > >>> According to his WIP patches, he plans to move the whole dogtag module > >>> into platform code. > >>> > >>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If > >>> you know a better way, please comment. > >>> > >>> The dogtag_version option is now always added to > >>> > >>> I've reverted the changes to install/ui/test/data/ipa_init.json, so > >>> you'll have to change the ports manually when testing the UI with Dogtag > >>> 10. > >>> > >>> > >>> > >> > >> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. > >> > > > > This approach looks good. I did not hit any regression on F17 with dogtag9, > > or > > clean installs of IPA+dogtag10. I think we are getting close to pushing > > this code. > > > > The only issues I hit so far are for upgraded dogtag9 instances (on F17): > > > > 1) The service did not start for me. There were some SELinux AVCs and even > > when > > I disabled SELinux the instance still did not start (logs attached). > > > > 2) Uninstall was also not clean, we leave some dogtag installation states > > for > > upgraded dogtag9 instance: > > > > # ipa-server-install --uninstall --unattended > > Shutting down all IPA services > > Removing IPA client configuration > > Unconfiguring ntpd > > Unconfiguring CA directory server > > Unconfiguring web server > > Unconfiguring krb5kdc > > Unconfiguring kadmin > > Unconfiguring directory server > > Unconfiguring ipa_memcached > > ipa : ERROR Some installation state for pki-cad has not been > > restored, see /var/lib/ipa/sysrestore/sysrestore.state > > > > /var/lib/ipa/sysrestore/sysrestore.state: > > [pki-cad] > > enabled = False > > Ade is working on a new build to address the dogtag upgrade issues. > > The left over state should be removed when we drop the separate instance > in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch > who actually does this part, is this handled by dogtag? > > rob >
The build is available on the developer repo. Just do a yum update. As for this leftover state, this is an ipa file and such is never handled by dogtag. It must be handled somewhere within the ipa code. Ade _______________________________________________ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
