I believe I have solved the problem which was caused by some directory permissions. That is why when I ran openssl verify by hand, everything seemed to work. What threw me was Apache said it was reading the CA certs in the log, but apparently wasn't really.
I spent 3 days working on this before posting to this mailing list, and a few hours after doing so, seem to have solved the problem. Sorry to have bothered everyone but maybe if someone else has the same problem in the future, they will find this and give their directory permissions a check. Quoting Joseph Felten <[EMAIL PROTECTED]>: > I'm stumped so I thought I would give this list a try as I believe my problem > is > an openssl issue. > > Background: Building an SSL enabled Apache web server on a closed network. > Apache under Solaris 8 OS. Need to restrict access to users with ID > certificates issued by particular CA's (issued by particular Root issuers) > read > from a smart card. I can make everything work except restricting access to > particular CA's. Whenever I enable SSLVerifyClient and SSLVerifyDepth in > Apache it denies all access even though I present a cert that was issued by > one > of the CA's under SSLCACertificatePath. Even though I have those CA's certs > loaded on the server and can dump and verify them with openssl. I get errors > in the Apache log such as.: > > "Certificate Verification: Error (20): unable to get local issuer > certificate" > > and > > "SSL Library Error: 336105650 error:140890B2:SSL > routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned" > > I'm not sure which certificate is not being returned. From the browser/smart > card? It seems to be presenting the cert to the server. I suspect that > error > is misleading. > > I know the browser is reading the cert from the smart card as the browser > security module kicks in and asks which cert from the smart card to present > to > the server. I can't just install the user ID cert directly in the browser as > they are flagged non-exportable for security reasons, plus the smart cards > are > a requirement. > > Software: Apache/2.2.4 (Unix) mod_jk/1.2.21 DAV/2 mod_ssl/2.2.4 > OpenSSL/0.9.8e > mod_perl/2.0.3 Perl/v5.8.8 > > I tried some tests with openssl verify, s_client, s_server etc. openssl > s_server seems happy with everything. For example.: > > openssl s_server -key conf/euukmoappd003n.dev.local.server.key -cert > conf/cert.euukmoappd003n.dev.local.server.crt -CApath conf/ssl.crt -state > -Verify 10 > > verify depth is 10, must return a certificate > Enter pass phrase for conf/disa.euukmoappd003n.dev.local.server.key: > Using default temp DH parameters > Using default temp ECDH parameters > ACCEPT > > And I can connect with s_client. > > Below is the debug log from starting the SSL server and trying and failing to > view a test page with a certificate issued by a root/CA chain the server has > loaded. When I try to load a test page, it grinds a bit, asks me to insert > my > smart card, grinds a bit, asks for my smart card PIN, grinds a bit more, then > the browser displays an error page that "The page cannot be displayed". This > is with microsoft internet explorer (unfortunately that is the browser the > users have). Sorry I can't post the actual certs here as we have pretty > tight > security rules. Thanks in advance. > > [Fri Dec 07 19:11:40 2007] [info] Loading certificate & private key of > SSL-aware > server > [Fri Dec 07 19:11:40 2007] [debug] ssl_engine_pphrase.c(481): encrypted RSA > private key - pass phrase reused > [Fri Dec 07 19:11:41 2007] [info] Configuring server for SSL protocol > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(405): Creating new SSL > context (protocols: SSLv3, TLSv1) > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(538): Configuring client > authentication > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CLASS 3 Root CA > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=ECA/CN=ECA Root CA > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/ST=Cambs/L=Mole/O=USG/OU=USA OU PKI DD/CN=euukmoappd003n.dev.local > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CLASS 3 Root CA > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-13 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=ECA/CN=ECA Root CA > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-13 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CLASS 3 Root CA > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=ECA/CN=ECA Root CA > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-15 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-13 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-15 > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(601): Configuring > permitted > SSL ciphers [ALL:!ADH:!EXPORT56:!EXP:RC4+RSA:+HIGH:-MEDIUM:!LOW:+SSLv2: > > -eNULL] > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(626): Configuring > certificate revocation facility > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(729): Configuring RSA > server certificate > [Fri Dec 07 19:11:41 2007] [debug] ssl_engine_init.c(768): Configuring RSA > server private key > [Fri Dec 07 19:11:43 2007] [info] Configuring server for SSL protocol > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(405): Creating new SSL > context (protocols: SSLv3, TLSv1) > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(538): Configuring client > authentication > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CLASS 3 Root CA > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=ECA/CN=ECA Root CA > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/ST=Cambs/L=Mole/O=USG/OU=USA OU PKI DD/CN=euukmoappd003n.dev.local > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CLASS 3 Root CA > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-13 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=ECA/CN=ECA Root CA > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-13 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CLASS 3 Root CA > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=ECA/CN=ECA Root CA > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-15 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-13 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(1113): CA certificate: > /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-15 > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(601): Configuring > permitted > SSL ciphers [ALL:!ADH:!EXPORT56:!EXP:RC4+RSA:+HIGH:-MEDIUM:!LOW:+SSLv2: > > -eNULL] > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(626): Configuring > certificate revocation facility > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(729): Configuring RSA > server certificate > [Fri Dec 07 19:11:43 2007] [debug] ssl_engine_init.c(768): Configuring RSA > server private key > [Fri Dec 07 19:11:49 2007] [info] [client 131.58.59.198] Connection to child > 0 > established (server euukmoappd003n.dev.local:443) > [Fri Dec 07 19:11:49 2007] [info] Seeding PRNG with 512 bytes of entropy > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1752): OpenSSL: > Handshake: start > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > before/accept initialization > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1775): OpenSSL: read 11/11 > bytes from BIO#100629330 [mem: 1007677e0] (BIO dump follows) > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1722): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1747): | 0000: 80 4c 01 03 > 01 > 00 33 00-00 00 10 .L....3.... | > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1753): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1775): OpenSSL: read 67/67 > bytes from BIO#100629330 [mem: 1007677eb] (BIO dump follows) > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1722): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1747): | 0000: 00 00 04 00 > 00 > 05 00 00-0a 01 00 80 07 00 c0 03 ................ | > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1747): | 0010: 00 80 00 00 > 09 > 06 00 40-00 00 64 00 00 62 00 00 [EMAIL PROTECTED] | > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1747): | 0020: 03 00 00 06 > 02 > 00 80 04-00 80 00 00 13 00 00 12 ................ | > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1747): | 0030: 00 00 63 58 > 73 > 4d 82 58-2f cf 3e 3f 17 85 78 27 ..cXsM.X/.>?..x' | > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1747): | 0040: c1 b5 bb > ... | > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_io.c(1753): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 read client hello A > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 write server hello A > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 write certificate A > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 write certificate request A > [Fri Dec 07 19:11:49 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 flush data > [Fri Dec 07 19:12:03 2007] [debug] ssl_engine_io.c(1786): OpenSSL: I/O error, > 5 > bytes expected to read on BIO#100629330 [mem: 1007677e0] > [Fri Dec 07 19:12:03 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: > error in SSLv3 read client certificate A > [Fri Dec 07 19:12:03 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: > error in SSLv3 read client certificate A > [Fri Dec 07 19:12:03 2007] [info] [client 131.58.59.198] (70014)End of file > found: SSL handshake interrupted by system [Hint: Stop button pressed in > > browser?!] > [Fri Dec 07 19:12:03 2007] [info] [client 131.58.59.198] Connection closed to > child 0 with abortive shutdown (server euukmoappd003n.dev.local:443) > [Fri Dec 07 19:12:13 2007] [info] [client 131.58.59.198] Connection to child > 1 > established (server euukmoappd003n.dev.local:443) > [Fri Dec 07 19:12:13 2007] [info] Seeding PRNG with 512 bytes of entropy > [Fri Dec 07 19:12:13 2007] [debug] ssl_engine_kernel.c(1752): OpenSSL: > Handshake: start > [Fri Dec 07 19:12:13 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > before/accept initialization > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1775): OpenSSL: read 11/11 > bytes from BIO#100629330 [mem: 1007677e0] (BIO dump follows) > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1722): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1747): | 0000: 80 4c 01 03 > 01 > 00 33 00-00 00 10 .L....3.... | > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1753): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1775): OpenSSL: read 67/67 > bytes from BIO#100629330 [mem: 1007677eb] (BIO dump follows) > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1722): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1747): | 0000: 00 00 04 00 > 00 > 05 00 00-0a 01 00 80 07 00 c0 03 ................ | > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1747): | 0010: 00 80 00 00 > 09 > 06 00 40-00 00 64 00 00 62 00 00 [EMAIL PROTECTED] | > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1747): | 0020: 03 00 00 06 > 02 > 00 80 04-00 80 00 00 13 00 00 12 ................ | > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1747): | 0030: 00 00 63 bb > 75 > 33 36 bc-e7 29 6d 0a 05 49 dc 04 ..c.u36..)m..I.. | > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1747): | 0040: 35 16 bc > 5.. | > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_io.c(1753): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 read client hello A > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 write server hello A > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 write certificate A > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 write certificate request A > [Fri Dec 07 19:12:34 2007] [debug] ssl_engine_kernel.c(1760): OpenSSL: Loop: > SSLv3 flush data > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1775): OpenSSL: read 5/5 > bytes from BIO#100629330 [mem: 1007677e0] (BIO dump follows) > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1722): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1747): | 0000: 16 03 01 09 > 50 > ....P | > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1753): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1775): OpenSSL: read > 2384/2384 bytes from BIO#100629330 [mem: 1007677e5] (BIO dump follows) > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1722): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1747): | 0000: 0b 00 08 40 > 00 > 08 3d 00-03 ff 30 82 03 fb 30 82 [EMAIL PROTECTED] | > ** SNIPPED A BUNCH OF THIS HEX DUMP ** > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1747): | 0940: 66 8f 49 f1 > e4 > a6 88 c5-db 06 cd 35 a4 f5 a2 13 f.I........5.... | > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_io.c(1753): > +-------------------------------------------------------------------------+ > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_kernel.c(1190): Certificate > Verification: depth: 1, subject: /C=US/O=USG/OU=DD/OU=PKI/CN=DD CA-12, > issuer: > > /C=US/O=USG/OU=DD/OU=PKI/CN=DD Root CA 2 > [Fri Dec 07 19:12:43 2007] [error] Certificate Verification: Error (20): > unable > to get local issuer certificate > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_kernel.c(1770): OpenSSL: Write: > SSLv3 read client certificate B > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: > error in SSLv3 read client certificate B > [Fri Dec 07 19:12:43 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: > error in SSLv3 read client certificate B > [Fri Dec 07 19:12:43 2007] [info] [client 131.58.59.198] SSL library error 1 > in > handshake (server euukmoappd003n.dev.local:443) > [Fri Dec 07 19:12:43 2007] [info] SSL Library Error: 336105650 > error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate > returned > [Fri Dec 07 19:12:43 2007] [info] [client 131.58.59.198] Connection closed to > child 1 with abortive shutdown (server euukmoappd003n.dev.local:443) > > ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
