[
https://issues.apache.org/jira/browse/KAFKA-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902896#comment-14902896
]
Ismael Juma edited comment on KAFKA-2561 at 9/24/15 2:37 PM:
-------------------------------------------------------------
In order to implement this properly (as opposed to a simple test), the
following steps are needed:
1. Add an optional build dependency on netty-tcnative. This library contains a
fork of tomcat native that is available in Maven and is available for major
platforms (Linux, OS X, Windows). It also handles extracting the
platform-specific JNI code at runtime (similar to snappy-java). apr and openssl
need to be installed separately.
2. Provide an implementation of `SSLEngine` based on OpenSSL. The easy option
would be to add an optional dependency on `netty-handler`, which includes this.
If this is not acceptable, there are some alternatives like extracting the code
into a separate library or copying it into Kafka.
3. Add a way to configure the `SSLEngine` implementation (OpenSSL or JDK).
4. Change `SSLFactory` to build the appropriate `SSLEngine` based on the
configuration added in `3`.
5. Potentially introduce a runtime mechanism to select `OpenSslEngine` by
default if the required libraries are present (since it's much faster)
6. Potentially update `SSLTransportLayer` to handle differences in behaviour
between the different `SSLEngine` implementations (the need for this depends on
whether we the issues reported to Netty are fixed or not). In addition to the
two issues mentioned in the description, there is also an issue related to the
size of the `applicationReadBuffer`.
7. Use `allocateDirect` to allocate the buffers in `SSLTransportLayer` when
using `OpenSslEngine` to avoid copies on each `wrap` and `unwrap` call.
8. Design and implement the story around the formats for keys, certificates,
key chains and certificate chains supported. OpenSSL doesn't understand the JKS
format since it's Java-specific. Netty uses the `PKCS#8` format for keys and
PEM format for chains when the OpenSSL engine is used.
9. Update tests to test all `SSLEngine` implementations.
Testing of this is more complicated than usual due to the native code aspect
and we would have to test it in all of our supported platforms.
Given the work that I've already done, it would probably take a couple of weeks
to agree on the details and implement the code (including unit tests). Maybe
another week for testing on the various platforms.
was (Author: ijuma):
In order to implement this properly (as opposed to a simple test), the
following steps are needed:
1. Add an optional build dependency on netty-tcnative. This library contains a
fork of tomcat native that is available in Maven and is available for major
platforms (Linux, OS X, Windows). It also handles extracting the
platform-specific JNI code at runtime (similar to snappy-java). apr and openssl
need to be installed separately.
2. Provide an implementation of `SSLEngine` based on OpenSSL. The easy option
would be to add an optional dependency on `netty-handler`, which includes this.
If this is not acceptable, there are some alternatives like extracting the code
into a separate library or copying it into Kafka.
3. Add a way to configure the `SSLEngine` implementation (OpenSSL or JDK).
4. Change `SSLFactory` to build the appropriate `SSLEngine` based on the
configuration added in `3`.
5. Potentially introduce a runtime mechanism to select `OpenSslEngine` by
default if the required libraries are present (since it's much faster)
6. Potentially update `SSLTransportLayer` to handle differences in behaviour
between the different `SSLEngine` implementations (the need for this depends on
whether we the issues reported to Netty are fixed or not). In addition to the
two issues mentioned in the description, there is also an issue related to the
size of the `applicationReadBuffer`.
7. Design and implement the story around the formats for keys, certificates,
key chains and certificate chains supported. OpenSSL doesn't understand the JKS
format since it's Java-specific. Netty uses the `PKCS#8` format for keys and
PEM format for chains when the OpenSSL engine is used.
8. Update tests to test all `SSLEngine` implementations.
Testing of this is more complicated than usual due to the native code aspect
and we would have to test it in all of our supported platforms.
Given the work that I've already done, it would probably take a couple of weeks
to agree on the details and implement the code (including unit tests). Maybe
another week for testing on the various platforms.
> Optionally support OpenSSL for SSL/TLS
> ---------------------------------------
>
> Key: KAFKA-2561
> URL: https://issues.apache.org/jira/browse/KAFKA-2561
> Project: Kafka
> Issue Type: New Feature
> Components: security
> Affects Versions: 0.9.0.0
> Reporter: Ismael Juma
>
> JDK's `SSLEngine` is unfortunately a bit slow (KAFKA-2431 covers this in more
> detail). We should consider supporting OpenSSL for SSL/TLS. Initial
> experiments on my laptop show that it performs a lot better:
> {code}
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg,
> nMsg.sec, config
> 2015-09-21 14:41:58:245, 2015-09-21 14:47:02:583, 28610.2295, 94.0081,
> 30000000, 98574.6111, Java 8u60/server auth JDK
> SSLEngine/TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
> 2015-09-21 14:38:24:526, 2015-09-21 14:40:19:941, 28610.2295, 247.8900,
> 30000000, 259931.5514, Java 8u60/server auth
> OpenSslEngine/TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> 2015-09-21 14:49:03:062, 2015-09-21 14:50:27:764, 28610.2295, 337.7751,
> 30000000, 354182.9000, Java 8u60/plaintext
> {code}
> Extracting the throughput figures:
> * JDK SSLEngine: 94 MB/s
> * OpenSSL SSLEngine: 247 MB/s
> * Plaintext: 337 MB/s (code from trunk, so no zero-copy due to KAFKA-2517)
> In order to get these figures, I used Netty's `OpenSslEngine` by hacking
> `SSLFactory` to use Netty's `SslContextBuilder` and made a few changes to
> `SSLTransportLayer` in order to workaround differences in behaviour between
> `OpenSslEngine` and JDK's SSLEngine (filed
> https://github.com/netty/netty/issues/4235 and
> https://github.com/netty/netty/issues/4238 upstream).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
