[Spacewalk-list] After Spacewalk serverrename/selfsigned SSL-issues on yum-updates for CentOS
Dear all, once again I'm puzzling about some strange issue and am hoping for some valued input I experienced in the past on this list: The gist (for the hasty reader:): CentOS yum/curl rejects my Spacewalk' servers certificate as Bad certificate received while a check with openssl s_client works. Fedoras don't show the issue at all. Any ideas? Best -Jonathan # Here is a bit more length explanation, but I want to show what works to exclude some point-of-errors So I renamed my Spacewalk-Server some while ago, and in/after the renaming (DNS etc reports clean name, the server has the name, all fine), I updated the Spacewalk servers' Certs, by using my own self-signed root-CA (which now not only exists in /etc/pki/tls/certs/ca-bundle* but of course also in /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT ). Everything works fine (I of course had to deploy the rootCA to my clients, update the server-name in up2date etc), except for yum update - commands on CentOS boxes. And this is the issue. Fedora (16,17,18) works fine, with identical steps of change-deployment. Debugging the yum-commands shows (domain+email slightly modified) (yum debug=10, on CentOS 5/6) [...] 2013-06-13 09:19:33,487 attempt 1/1: https://ibis-spacewalk.xx-muenchen.de/XMLRPC/GET-REQ/centos6-x86_64/repodata/repomd.xml INFO:urlgrabber:attempt 1/1: https://ibis-spacewalk.xx-muenchen.de/XMLRPC/GET-REQ/centos6-x86_64/repodata/repomd.xml 2013-06-13 09:19:33,487 opening local file /var/cache/yum/x86_64/6/centos6-x86_64/repomdKZp7dWtmp.xml with mode wb INFO:urlgrabber:opening local file /var/cache/yum/x86_64/6/centos6-x86_64/repomdKZp7dWtmp.xml with mode wb * About to connect() to ibis-spacewalk.xx-muenchen.de port 443 (#0) * Trying xx.107.218.92... * connected * Connected to ibis-spacewalk.xx-muenchen.de (xx.107.218.92) port 443 (#0) * warning: CURLOPT_CAPATH not a directory (/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT) * CAfile: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT CApath: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT * Bad certificate received. Subject = 'E=m...@xx-muenchen.de,CN=ibis-spacewalk.xx-muenchen.de,OU=IBIS,O=Helmholtz-Zentrum Muenchen GmbH,L=Munich,ST=Bavaria,C=DE', Issuer = 'E=m...@xx-muenchen.de,CN=IBIS-Root-CA,OU=IBIS,O=Helmholtz-Zentrum Muenchen GmbH,L=Munich,ST=Bavaria,C=DE' * NSS error -8182 * Closing connection #0 * Peer certificate cannot be authenticated with known CA certificates 2013-06-13 09:19:33,689 exception: [Errno 14] Peer cert cannot be verified or peer cert invalid INFO:urlgrabber:exception: [Errno 14] Peer cert cannot be verified or peer cert invalid 2013-06-13 09:19:33,690 retries exceeded, re-raising INFO:urlgrabber:retries exceeded, re-raising Error: Cannot retrieve repository metadata (repomd.xml) for repository: centos6-x86_64. Please verify its path and try again So I googled the messages and re-discovered it uses curl. Trying curl 'naked' on my server-url: curl --cacert /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT https://ibis-spacewalk.xx-muenchen.de -v * About to connect() to ibis-spacewalk.xx-muenchen.de port 443 (#0) * Trying xx.107.218.92... connected * Connected to ibis-spacewalk.xx-muenchen.de (146.107.218.92) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT CApath: none * Bad certificate received. Subject = 'E=m...@xx-muenchen.de,CN=ibis-spacewalk.xx-muenchen.de,OU=IBIS,O=Helmholtz-Zentrum Muenchen GmbH,L=Munich,ST=Bavaria,C=DE', Issuer = 'E=m...@xx-muenchen.de,CN=IBIS-Root-CA,OU=IBIS,O=Helmholtz-Zentrum Muenchen GmbH,L=Munich,ST=Bavaria,C=DE' * NSS error -8182 * Closing connection #0 * Peer certificate cannot be authenticated with known CA certificates curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a bundle of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. With the -k version it works, no questions asked. openssl s_client -connect ibis-spacewalk.xx-muenchen.de:443 [...finally returns...] Verify return code: 0 (ok) and thus works, and is happy with the results/the ca-certs. The crazy thing is - as I mentioned - that the identical requests (from yum and curl) work on Fedoras (16-18). The only real difference are the libcurl/curl versions: Fedora17 uses 7.24.0 (for both) CentOS 6.4 uses 7.19.7 I am a bit out of ideas here - I also added the rootCA to the NSS store (since this is also loaded in curl-Standalone) but that
Re: [Spacewalk-list] Keepalive for Database Connections?
Hi Stefan, you can create /etc/tnsname.ora with connect string you need. Then just change db_name in /etc/rhn/rhn.conf to the tnsname you've created (instead of default //host:port/name) and that's it :). Thanks a lot Michael! That did the trick. The file is actually called /etc/tnsnames.ora and not /etc/tnsname.ora Small difference, with big effects ;) I wrote a small tutorial - just in case anyone else is interested: How to implement Keepalive for Oracle Instant Client (for use in Spacewalk). Oracle Client will use the TCP Keepalive Feature of the Operating System (in my case Linux). 1) create /etc/tnsnames.ora by closely following the example in the ENABLE Section of the Oracle Docs: http://docs.oracle.com/cd/B28359_01/network.111/b28317/tnsnames.htm#CHDCDGCE 2) use Michael's magic and edit Spacewalk's /etc/rhn/rhn.conf. Change db_name = //your_db_server_name/your_db into db_name = net_service_name (use the same net_service_name you used in your /etc/tnsnames.ora) 3) check the kernel's current keepalive values: # sysctl -a | grep keepalive net.ipv4.tcp_keepalive_time = 7200 # send the first keepalive after 3600 seconds (2h) net.ipv4.tcp_keepalive_intvl = 75 # after the first keepalive, send keepalives every 75 seconds net.ipv4.tcp_keepalive_probes = 9 # kill connection after 9 failed keepalives 4) Adjust the kernels Keepalive values to your needs: # sysctl –w net.ipv4.tcp_keepalive_time=1800 # echo “net.ipv4.tcp_keepalive_time = 1800” /etc/sysctl.conf 5) restart Spacewalk # /usr/sbin/spacewalk-service stop /usr/sbin/spacewalk-service start 6) check if keepalive is now enabled for your database connections (use netstat -tpno and check the last two columns for the word keepalive and the corresponding timer values) # netstat -tnpo tcp0 0 192.168.1.1:54729 192.168.2.2:1521 ESTABLISHED 26988/java keepalive (1530.03/0/0) tcp0 0 192.168.1.1:54718 192.168.2.2:1521 ESTABLISHED 26944/httpd keepalive (1522.62/0/0) 7) enjoy your space walk ;) __ regards, Stefan ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] Keepalive for Database Connections?
Stefan Lasche wrote: % Hi Stefan, % % you can create /etc/tnsname.ora with connect string you need. % Then just change db_name in /etc/rhn/rhn.conf to the tnsname you've created % (instead of default //host:port/name) and that's it :). % % % Thanks a lot Michael! That did the trick. The file is actually called /etc/tnsnames.ora and not /etc/tnsname.ora % Small difference, with big effects ;) Oops, sorry for the typo. I hope you didn't spend ages to find it out ;). % I wrote a small tutorial - just in case anyone else is interested: ... Thanks both for success report and tutorial. Regards, -- Michael Mráka Satellite Engineering, Red Hat ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] libxml2-python and kickstart
One thing I found is that you need to use the image files from the Everything install disk when creating the kickstart distribution or you will run into problems like the one you described with the update channel. On Mon, Jun 10, 2013 at 8:08 AM, Jan Hutař jhu...@redhat.com wrote: On Mon, 3 Jun 2013 23:41:14 + Faye Salwin faye.sal...@betfair.com wrote: Centos6 updates contains libxml2-python and Spacewalk pulls THAT version (as it is the latest available version within any child channels of the selected distribution) of the RPM when building kickstart files instead of a package from a channel enabled for kickstart (The Java code pulls these files using the file download url) This works: http://x.x.x.x/download/package/2ca68cda56a77523c946e69ca5dc8d34e0f45f01/0/1/885/libxml2-python-2.7.6-8.el6_3.4.x86_64.rpm http://x.x.x.x/download/package/1e2379d4dadc3fef16c422d3aa705ff83264b469/0/1/5811/pyOpenSSL-0.10-2.el6.x86_64.rpm http://x.x.x.x/download/package/704fdbe78c79726bab6d1e5f8461045ef7f0a194/0/1/11132/rhnlib-2.5.55-1.el6.noarch.rpm rpm -Uvh --replacepkgs --replacefiles /tmp/rhn_rpms/optional/pyOpenSSL* /tmp/rhn_rpms/optional/rhnlib* /tmp/rhn_rpms/optional/libxml2-python* If the Updates repo is disabled in the kickstarter, but available as a child channel of the install tree, then libxml2-python fails to install as it cannot supply the dependencies to libxml2-python (*12-el6_4*) which are in the Updates repo too. I have read that one shouldn't enable the Updates Repo during Kickstart, but if I do not delete libxml2-python from the updates repo then a kickstart which does not include the updates repo will fail. If I do install the updates repo to kickstart then it will lag the install for a long time while it tries and retries to access a comps file which does not exist. (there are comps files for EPEL and Spacewalk which I have to include during kickstart to install the rhn_client, rhncfg-client etc. I have restricted EPEL to only those packages which the Spacewalk repo depends upon. I believe that the kickstart will continue to function from the time that libxml2-python from updates is added until the kickstart profile is updated at which point it will re-evaluate the includes. It doesn't seem like a good solution to delete libxml2-python every time you update kickstart and then re-sync. What is the best practice here? Should I kickstart from a tree with updates present? What happens about the Comps (groups file) in this case? I've also noticed that cloning child channels (using the manage channel lifecycle script at least) does not clone the RHNCHANNELCOMPS row that is relevant which caused me a few frustrations early on. Is it not possible to kickstart just one distribution channel selection and then switch parent channels at activation time? That would simplify a lot of the workflow (perhaps allow you to have a single kickstart for all your lifecycle locations and validate that the parent channel is a clone of the one you are activating?) Why does the Kickstart file create java code include files that are from repos not included in the kickstart channels? I am guessing it is so that you don't have to include the spacewalk repo as a child, but if you don't do that, it can't find rhnreg_ks for example (unless you use one of the many workarounds that have been posted in this list eg. tar ball of spacewalk files) It is entirely possible that I am just doing it wrong in which case a pointer to best practice would be appreciated. Thanks Faye Hello, you should kickstart with only base channel enabled (ideally no additional child channels/repos). After install you can assign additional child channels and update your system (you can assign these channels using activation key in your kickstart profile). If you have kickstart profile with base channel only and satellite generated these http://...; download links pointing to packages which are not in the base channel, that is a bug and please report it to the Bugzilla or here. Regards, Jan -- Jan Hutar Systems Management QA jhu...@redhat.com Red Hat, Inc. ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list ___ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list
Re: [Spacewalk-list] libxml2-python and kickstart
(To Jan) Question. Given that this is discussed in http://bugs.centos.org/view.php?id=4978 (going back almost 2 years now), and is still an active issue, I'm wondering if the official answer is to not use the updates repo during installation or not. But if this is just a metadata bug, as described above and in the referenced bugzilla entry, can someone please elaborate on what we, the users, can do to help get this resolved in a way that we can specify additional repositories during installation safely, as described in https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s1-pkgselection-x86.html (section 9.18.1) and https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s1-kickstart2-options.html (via the repo keyword)? Or are we just SOL? (To Paul) Can you please elaborate on this? As noted in the above-referenced bugzilla entry, people are still fighting with this issue 2 years later, but if you've got a working solution to this problem, I'm sure that many folks would like to know more about it... Thanks, -Ian On Thu, Jun 13, 2013 at 8:02 AM, Paul Robert Marino prmari...@gmail.comwrote: One thing I found is that you need to use the image files from the Everything install disk when creating the kickstart distribution or you will run into problems like the one you described with the update channel. On Mon, Jun 10, 2013 at 8:08 AM, Jan Hutař jhu...@redhat.com wrote: On Mon, 3 Jun 2013 23:41:14 + Faye Salwin faye.sal...@betfair.com wrote: Centos6 updates contains libxml2-python and Spacewalk pulls THAT version (as it is the latest available version within any child channels of the selected distribution) of the RPM when building kickstart files instead of a package from a channel enabled for kickstart (The Java code pulls these files using the file download url) This works: http://x.x.x.x/download/package/2ca68cda56a77523c946e69ca5dc8d34e0f45f01/0/1/885/libxml2-python-2.7.6-8.el6_3.4.x86_64.rpm http://x.x.x.x/download/package/1e2379d4dadc3fef16c422d3aa705ff83264b469/0/1/5811/pyOpenSSL-0.10-2.el6.x86_64.rpm http://x.x.x.x/download/package/704fdbe78c79726bab6d1e5f8461045ef7f0a194/0/1/11132/rhnlib-2.5.55-1.el6.noarch.rpm rpm -Uvh --replacepkgs --replacefiles /tmp/rhn_rpms/optional/pyOpenSSL* /tmp/rhn_rpms/optional/rhnlib* /tmp/rhn_rpms/optional/libxml2-python* If the Updates repo is disabled in the kickstarter, but available as a child channel of the install tree, then libxml2-python fails to install as it cannot supply the dependencies to libxml2-python (*12-el6_4*) which are in the Updates repo too. I have read that one shouldn't enable the Updates Repo during Kickstart, but if I do not delete libxml2-python from the updates repo then a kickstart which does not include the updates repo will fail. If I do install the updates repo to kickstart then it will lag the install for a long time while it tries and retries to access a comps file which does not exist. (there are comps files for EPEL and Spacewalk which I have to include during kickstart to install the rhn_client, rhncfg-client etc. I have restricted EPEL to only those packages which the Spacewalk repo depends upon. I believe that the kickstart will continue to function from the time that libxml2-python from updates is added until the kickstart profile is updated at which point it will re-evaluate the includes. It doesn't seem like a good solution to delete libxml2-python every time you update kickstart and then re-sync. What is the best practice here? Should I kickstart from a tree with updates present? What happens about the Comps (groups file) in this case? I've also noticed that cloning child channels (using the manage channel lifecycle script at least) does not clone the RHNCHANNELCOMPS row that is relevant which caused me a few frustrations early on. Is it not possible to kickstart just one distribution channel selection and then switch parent channels at activation time? That would simplify a lot of the workflow (perhaps allow you to have a single kickstart for all your lifecycle locations and validate that the parent channel is a clone of the one you are activating?) Why does the Kickstart file create java code include files that are from repos not included in the kickstart channels? I am guessing it is so that you don't have to include the spacewalk repo as a child, but if you don't do that, it can't find rhnreg_ks for example (unless you use one of the many workarounds that have been posted in this list eg. tar ball of spacewalk files) It is entirely possible that I am just doing it wrong in which case a pointer to best practice would be appreciated. Thanks Faye Hello, you should kickstart with only base channel enabled (ideally no additional child channels/repos). After
