So that’s not quite true, Hans. First, as far as the performance hit being not a big impact (25% is huge). Or that it’s to be expected. Part of the problem is that the Java TLS implementation does not support zero copy. OpenSSL does, and in fact there’s been a ticket open to allow Kafka to support using OpenSSL for a while now:
https://issues.apache.org/jira/browse/KAFKA-2561 On Mon, Mar 6, 2017 at 6:30 AM, Hans Jespersen <h...@confluent.io> wrote: > > Its not a single message at a time that is encrypted with TLS its the > entire network byte stream so a Kafka broker can’t even see the Kafka > Protocol tunneled inside TLS unless it’s terminated at the broker. > It is true that losing the zero copy optimization impacts performance > somewhat but it’s not what I would call a “big impact” because Kafka does > a lot of other things to get it’s performance (like using page cache and > doing lots on sequential disk I/O). The difference should be something in > the order of 25-30% slower with TLS enabled which is about what you would > see with any other messaging protocol with TLS on vs off. > > If you wanted to encrypt each message independently before sending to > Kafka then zero copy would still be in effect and all the consumers would > get the same encrypted message (and have to understand how to decrypt it). > > -hans > > > > > On Mar 6, 2017, at 5:38 AM, Nicolas Motte <lingusi...@gmail.com> wrote: > > > > Hi everyone, > > > > I understand one of the reasons why Kafka is performant is by using > > zero-copy. > > > > I often hear that when encryption is enabled, then Kafka has to copy the > > data in user space to decode the message, so it has a big impact on > > performance. > > > > If it is true, I don t get why the message has to be decoded by Kafka. I > > would assume that whether the message is encrypted or not, Kafka simply > > receives it, appends it to the file, and when a consumer wants to read > it, > > it simply reads at the right offset... > > > > Also I m wondering if it s the case if we don t use keys (pure queuing > > system with key=null). > > > > Cheers > > Nico > > -- *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
