We did a proof of concept on end-to-end encryption using an approach which
sounds similar to what you describe.  We blogged about it here:
  
http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p
roof-concept

You might want to review what is there to see how it differs from what you
did.  In our tests, the encryption didn't add as much overhead as we
thought it would.

-- Jim

-- 
Jim Hoagland, Ph.D.
Sr. Principal Software Engineer
Big Data Analytics Team
Cloud Platform Engineering



On 1/14/16, 2:23 PM, "Bruno Rassaerts" <[email protected]> wrote:

>Hello,
>
>In our project we have a very strong requirement to protect all data, all
>the time. Even when the data is “in-rest” on disk, it needs to be
>protected.
>We’ve been trying to figure out how to this with Kafka, and hit some
>obstacles.
>
>One thing we’ve tried to do is to encrypt every message we hand over to
>kafka. This results in the encrypted messages being written to disk on
>the brokers.
>However, the performance of performing encryption has serious performance
>implications, due to the CPU intensive operation which encryption is, and
>the fact that batch compression offered by Kafka is not nearly as
>efficient anymore after encrypting the data. Doing this message by
>message encryption gives us a performance penalty of about 75%, even if
>we compress the messages before encryption.
>
>What we are looking for is a way to plugin our encryption in two possible
>locations:
>
>1. As a custom compression algorithm, which would batch compress, and
>batch encrypt. And get the files stored as such.
>2. As a encryption plugin specifically designed for storing the kafka
>broker files.
>
>Is there any way that this can be done using Kafka (0.9), or can somebody
>point us to the place were we could add this in the Kafka codebase.
>
>Thanks,
>Bruno Rassaerts 

Reply via email to