I have been testing some projects that I know of, with Java 11 RC. There's one specific test that has been failing for me, for a while now, during SSL handshake. Took me a while to narrow it down given the size of the application and the nature of the exception. The exception occurs during SSL handshake between a client and a server (both Java and both using Java 11 RC) and it kept throwing:
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:255) at java.base/sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:329) at java.base/sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeConsumer.consume(ECDHServerKeyExchange.java:535) at java.base/sun.security.ssl.ServerKeyExchange$ServerKeyExchangeConsumer.consume(ServerKeyExchange.java:103) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1581) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:245) ... This is consistently reproducible if, in the scheme of things: 1. the cipher suite selected is a ECDHE one. For example TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. The TLS version itself is TLSv1.2 (both server and client). 2. One side of the client/server, is backed by BouncyCastle as the security provider (very specifically for SignatureSpi). Assuming the server side is using BouncyCastle provider, what's happening is that during the handshake, the server side uses the RSASSA-PSS algorithm, backed by this provider and writes out the signature. The client side uses SunRsaSign (default provider) and tries to verify this RSASSA-PSS signature and fails with that above exception. FWIW, I don't believe the signature algorithm matters as long as the cipher is ECDHE backed and the client and server side use a different security provider. All this works perfectly fine when both the sides use the default provider and bouncycastle isn't involved. I was able to get this reproducible in a very simple server/client standalone program. I think this can even be demonstrated in a jtreg test but I don't have enough experience with it to see how to trigger a server in a separate JVM and then use a client for testing. The reproducer code (Server.java and Client.java) is at the end of this mail along with instructions on how to reproduce it. I was also able to narrow down this issue down to a specific part of the JDK code. sun.security.ssl.SignatureScheme#getSignature[1] inits the Signature instance for either signing or verifying. However it sets up the signature parameters _after_ the init is done and in fact, there's an explicit note[2] stating what/why it's doing it there. I admit, I don't have much knowledge of the Java SSL parts and none in these internal details and don't understand the details of that implementation notes. However, just to try it out, I switched the order of it by using this local patch: diff -r fbb71a7edc1a src/java.base/share/classes/sun/security/ssl/SignatureScheme.java --- a/src/java.base/share/classes/sun/security/ssl/SignatureScheme.java Sat Aug 25 20:16:43 2018 +0530 +++ b/src/java.base/share/classes/sun/security/ssl/SignatureScheme.java Tue Sep 18 18:47:52 2018 +0530 @@ -467,18 +467,16 @@ } Signature signer = JsseJce.getSignature(algorithm); + if (signAlgParameter != null) { + signer.setParameter(signAlgParameter); + } + if (key instanceof PublicKey) { signer.initVerify((PublicKey)(key)); } else { signer.initSign((PrivateKey)key); } - // Important note: Please don't set the parameters before signature - // or verification initialization, so that the crypto provider can - // be selected properly. - if (signAlgParameter != null) { - signer.setParameter(signAlgParameter); - } return signer; } Built this version and gave it a try with the sample code below (and also against the actual application). Both worked fine. I tried cases: - where one side had default provider and other side had bouncycastle. - where both sides were default provider Any thoughts on this issue? Here's the code to reproduce it: Server.java ----------- import java.io.*; import java.net.*; import javax.net.ssl.*; import java.util.*; import java.util.concurrent.*; import java.security.*; import com.sun.net.httpserver.*; import java.security.cert.*; public class Server { private static final String keyFilename = "keystore"; private static final String keystorePass = "passphrase"; private static final String WEB_APP_CONTEXT = "/test"; public static void main(final String[] args) throws Exception { if (args.length == 1 && args[0].equals("--use-bouncy-castle")) { // enable bouncycastle Security.insertProviderAt(new org.bouncycastle.jce.provider.BouncyCastleProvider(), 2); System.out.println("Using bouncycastle provider"); } else { System.out.println("Using JRE security provider"); } final int port = 12345; // start the server final HttpsServer server = startServer("localhost", port); // stop server on shutdown Runtime.getRuntime().addShutdownHook(new Thread(() -> { if (server != null) { server.stop(0); System.out.println("Stopped server"); } })); } private static HttpsServer startServer(final String host, final int port) throws Exception { final HttpsServer server = HttpsServer.create(new InetSocketAddress(host, port), 0); final Thread t = new Thread(() -> { try { final SSLContext sslctx = SSLContext.getInstance("TLSv1.2"); final KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509"); final KeyStore ks = KeyStore.getInstance("JKS"); try (final FileInputStream fis = new FileInputStream(keyFilename)) { ks.load(fis, keystorePass.toCharArray()); } kmf.init(ks, keystorePass.toCharArray()); sslctx.init(kmf.getKeyManagers(), null, null); final SSLParameters sslParameters = sslctx.getDefaultSSLParameters(); sslParameters.setProtocols(new String[] { "TLSv1.2" }); // use ECDHE specific ciphersuite sslParameters.setCipherSuites(new String[] { "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"}); server.setHttpsConfigurator(new HttpsConfigurator(sslctx) { @Override public void configure(final HttpsParameters params) { params.setSSLParameters(sslParameters); } }); server.createContext(WEB_APP_CONTEXT, (exchange)-> exchange.sendResponseHeaders(200, -1)); server.start(); System.out.println("Started server at " + server.getAddress()); } catch(Exception e) { throw new RuntimeException(e); } }); t.start(); return server; } } To run this: (you'll need bouncycastle jar in your classpath. you can get it from[3]) java -cp bcprov-jdk15on-1.58.jar Server.java --use-bouncy-castle You should see output like: Using bouncycastle provider Started server at /127.0.0.1:12345 (not passing --use-bouncy-castle will start the server with the regular default JRE provided security provider). The server is now up and running and ready to accept the request. See how to run the client below. This server code uses a keystore file which is part of the OpenJDK repo and can be obtained from [4] and stored in the current working directory. Client.java ------------ import java.io.*; import java.net.*; import javax.net.ssl.*; import java.util.*; import java.util.concurrent.*; import java.security.*; import com.sun.net.httpserver.*; import java.security.cert.*; public class Client { private static final String WEB_APP_CONTEXT = "/test"; public static void main(final String[] args) throws Exception { HttpsURLConnection.setDefaultHostnameVerifier((h, s) -> {return true;}); final int port = 12345; final URL targetURL = new URL("https://localhost:" + port + WEB_APP_CONTEXT); final HttpsURLConnection conn = (HttpsURLConnection) targetURL.openConnection(); // use a SSLSocketFactory which "trusts all" final SSLContext sslctx = SSLContext.getInstance("TLSv1.2"); sslctx.init(null, new TrustManager[] {new TrustAll()}, null); conn.setSSLSocketFactory(sslctx.getSocketFactory()); // read try (final InputStream is = conn.getInputStream()) { is.read(); } System.out.println("Received status code " + conn.getResponseCode()); } private static class TrustAll implements X509TrustManager { @Override public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException { return; } @Override public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException { return; } @Override public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; } } } To run the client: java Client.java A successful execution will show: Received status code 200 whereas a failed execution should throw the exception shown previously in the mail. [1] http://hg.openjdk.java.net/jdk/jdk/file/fbec908e2783/src/java.base/share/classes/sun/security/ssl/SignatureScheme.java#l463 [2] http://hg.openjdk.java.net/jdk/jdk/file/fbec908e2783/src/java.base/share/classes/sun/security/ssl/SignatureScheme.java#l476 [3] http://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.58/bcprov-jdk15on-1.58.jar [4] http://hg.openjdk.java.net/jdk/jdk/file/fbec908e2783/test/jdk/javax/net/ssl/etc/keystore -Jaikiran