Thanks for the link, Ismael. I had thought that the most recent kernels already implemented this, but I was probably confusing it with BSD. Most of my systems are stuck in the stone age right now anyway.
It would be nice to get KAFKA-2561 in, either way. First off, if you can take advantage of it it’s a good performance boost. Second, especially with the security landscape getting worse and worse, it would be good to have options as far as the TLS implementation goes. A zero-day exploit in the Java TLS implementation would be devastating, and more difficult to react to as it would require a new JRE (bringing with it who knows what problems). Swapping an underlying OpenSSL version would be much more palatable. -Todd On Mon, Mar 6, 2017 at 9:01 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Even though OpenSSL is much faster than the Java 8 TLS implementation (I > haven't tested against Java 9, which is much faster than Java 8, but > probably still slower than OpenSSL), all the tests were without zero copy > in the sense that is being discussed here (i.e. sendfile). To benefit from > sendfile with TLS, kernel-level changes/modules are required: > > https://github.com/ktls/af_ktls > http://www.phoronix.com/scan.php?page=news_item&px=FreeBSD-Faster-Sendfile > > Ismael > > On Mon, Mar 6, 2017 at 4:18 PM, Todd Palino <tpal...@gmail.com> wrote: > > > So that’s not quite true, Hans. First, as far as the performance hit > being > > not a big impact (25% is huge). Or that it’s to be expected. Part of the > > problem is that the Java TLS implementation does not support zero copy. > > OpenSSL does, and in fact there’s been a ticket open to allow Kafka to > > support using OpenSSL for a while now: > > > > https://issues.apache.org/jira/browse/KAFKA-2561 > > > > > > > > > > On Mon, Mar 6, 2017 at 6:30 AM, Hans Jespersen <h...@confluent.io> > wrote: > > > > > > > > Its not a single message at a time that is encrypted with TLS its the > > > entire network byte stream so a Kafka broker can’t even see the Kafka > > > Protocol tunneled inside TLS unless it’s terminated at the broker. > > > It is true that losing the zero copy optimization impacts performance > > > somewhat but it’s not what I would call a “big impact” because Kafka > > does > > > a lot of other things to get it’s performance (like using page cache > and > > > doing lots on sequential disk I/O). The difference should be something > in > > > the order of 25-30% slower with TLS enabled which is about what you > would > > > see with any other messaging protocol with TLS on vs off. > > > > > > If you wanted to encrypt each message independently before sending to > > > Kafka then zero copy would still be in effect and all the consumers > would > > > get the same encrypted message (and have to understand how to decrypt > > it). > > > > > > -hans > > > > > > > > > > > > > On Mar 6, 2017, at 5:38 AM, Nicolas Motte <lingusi...@gmail.com> > > wrote: > > > > > > > > Hi everyone, > > > > > > > > I understand one of the reasons why Kafka is performant is by using > > > > zero-copy. > > > > > > > > I often hear that when encryption is enabled, then Kafka has to copy > > the > > > > data in user space to decode the message, so it has a big impact on > > > > performance. > > > > > > > > If it is true, I don t get why the message has to be decoded by > Kafka. > > I > > > > would assume that whether the message is encrypted or not, Kafka > simply > > > > receives it, appends it to the file, and when a consumer wants to > read > > > it, > > > > it simply reads at the right offset... > > > > > > > > Also I m wondering if it s the case if we don t use keys (pure > queuing > > > > system with key=null). > > > > > > > > Cheers > > > > Nico > > > > > > > > > > > > -- > > *Todd Palino* > > Staff Site Reliability Engineer > > Data Infrastructure Streaming > > > > > > > > linkedin.com/in/toddpalino > > > -- *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
