JSSE and OpenSSL question

[Rose, Daniel]

I am having protocol difficulties connecting a JSSE client to an OpenSSL
server.  Here is what I have tried so far with the accompanying results:

 

JSSE client code:

KeyStore keyStore = KeyStore.getInstance("JKS");

                  

keyStore.load(new FileInputStream(fileName), new
String("XXXX").toCharArray());

 

TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");

tmf.init(keyStore);

 

SSLContext sslContext = SSLContext.getInstance("SSLv3"); 

 

TrustManager [] trustManagers = tmf.getTrustManagers();

 

sslContext.init(null, trustManagers, null);

                  

SSLSocketFactory  ssLSocketFactory = sslContext.getSocketFactory();

m_sockConn = (SSLSocket)ssLSocketFactory.createSocket(sPrimaryServer,
3508);

 

String [] ciphers = m_sockConn.getSupportedCipherSuites();

m_sockConn.setEnabledCipherSuites(ciphers);

 

String [] protocolsToUse = {"TLSv1", "SSLv3", "SSLv2Hello"};

m_sockConn.setEnabledProtocols(protocolsToUse);

                  

 

OpenSSL server code:

 

static sslProtocolVer = ACE_SSL_Context::SSLv23_server;

      

OpenSSL_add_ssl_algorithms();

m_pSSLContext = ACE_SSL_Context::instance();

 

if(0 != m_pSSLContext->set_mode(sslProtocolVer))

{

      return -1;

}

 

 

            

if( (0 > m_pSSLContext->certificate(certificateFile, SSL_FILETYPE_PEM))
||(0 > m_pSSLContext->private_key(privateKeyFile, SSL_FILETYPE_PEM))  )

{

      SSL_CTX_free(m_pSSLContext->context());

      m_pSSLContext = 0;

      return -1;

}

 

long sslCTXOptions = SSL_CTX_get_options(m_pSSLContext->context());

SSL_CTX_set_options(m_pSSLContext->context(),  sslCTXOptions 

m_pSSLContext->set_verify_peer(0);

      

    // ACE has a bug where the SSL_CTX is not updated with the mode,
using SSL method to do it explicitly

SSL_CTX_set_verify( m_pSSLContext->context(),
m_pSSLContext->default_verify_mode(), 0 );

 

 

Results:

 

Using the above client code I tested with combinations of the context
JSSE settings of "SSL", "TLS", "SSLv3", and "TLSv1" with comboniations
of setting enabled protocols of "TLSv1", "SSLv3" and "SSLv2Hello".
These were tested in connection with OpenSSL server side combinations
including "SSL23_server", "SSL3_server", "SSL2_server", and
"TLSv1_server" (along with non server specific version of these) with
combinations of the ctx_options of no_SSLv2, no_SSLv3, no_TLSv1,
DONT_INSERT_EMPTY_FRAGMENTS, and TLS_ROLLBACK_BUG.

 

It didn't seem to make any difference.  I consistently got

these openSSL responses:

 

ServerSide Context                      

 

ACE_SSL_Context::SSLv23_server   = > SSL23_GET_CLIENT_HELLO:unknown
protocol

 

 

ACE_SSL_Context::SSLv3_server   = > SSL3_GET_RECORD:wrong version number

 

ACE_SSL_Context::SSLv2_server   = > SSL2_READ_INTERNAL:non sslv2 initial
packet

 

ACE_SSL_Context::TLSv1_server   = > SSL3_GET_RECORD:wrong version number

 

One odd thing is that using TLSv1 seems to still use SSL3 calls as you
can see above (even when forced not to with ctx options and
setenabledprotocols). I believe I have tested all combinations of client
side JSSE protocol settings against all combinations of the OpenSSL
context setting. I must be missing something.  I'm new to SSL, but have
spent a lot of time in the past couple weeks trying to resolve this.
Any help would be greatly appreciated.  

 

Thanks

 

Dan