We did a proof of concept on end-to-end encryption using an approach which sounds similar to what you describe. We blogged about it here: http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p roof-concept
You might want to review what is there to see how it differs from what you did. In our tests, the encryption didn't add as much overhead as we thought it would. -- Jim -- Jim Hoagland, Ph.D. Sr. Principal Software Engineer Big Data Analytics Team Cloud Platform Engineering On 1/14/16, 2:23 PM, "Bruno Rassaerts" <[email protected]> wrote: >Hello, > >In our project we have a very strong requirement to protect all data, all >the time. Even when the data is “in-rest” on disk, it needs to be >protected. >We’ve been trying to figure out how to this with Kafka, and hit some >obstacles. > >One thing we’ve tried to do is to encrypt every message we hand over to >kafka. This results in the encrypted messages being written to disk on >the brokers. >However, the performance of performing encryption has serious performance >implications, due to the CPU intensive operation which encryption is, and >the fact that batch compression offered by Kafka is not nearly as >efficient anymore after encrypting the data. Doing this message by >message encryption gives us a performance penalty of about 75%, even if >we compress the messages before encryption. > >What we are looking for is a way to plugin our encryption in two possible >locations: > >1. As a custom compression algorithm, which would batch compress, and >batch encrypt. And get the files stored as such. >2. As a encryption plugin specifically designed for storing the kafka >broker files. > >Is there any way that this can be done using Kafka (0.9), or can somebody >point us to the place were we could add this in the Kafka codebase. > >Thanks, >Bruno Rassaerts
