The questions we get from customers typically end up being general so we break out our answer into network level and on disk scenarios.
On disk/at rest scenario may just be use full disk encryption at the OS level and Kafka doesn't need to worry about it. But documenting any issues around it would be good. For example what sort of Kafka specific performance impacts does it have, ie budgeting for better processors. The security story right now is to run on a private network, but I believe some of our customers like to be told that within datacenter transmissions are encrypted on the wire. Based on https://cwiki.apache.org/confluence/display/KAFKA/Security that might mean waiting for TLS support, or using a VPN/ssh tunnel for the network connections. Since we're in hosted stream land we can't do either of the above and encrypt the messages themselves. For those enterprises that are like our customers but would run Kafka or use Confluent, having a story like the above so they don't give up the benefits of your schema management layers would be good. Since I didn't mention it before I did find your blog posts handy (though I'm already moving us towards stream centric land). Christian On Wed, Feb 25, 2015 at 3:57 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hey Christian, > > That makes sense. I agree that would be a good area to dive into. Are you > primarily interested in network level security or encryption on disk? > > -Jay > > On Wed, Feb 25, 2015 at 1:38 PM, Christian Csar <christ...@csar.us> wrote: > > > I wouldn't say no to some discussion of encryption. We're running on > Azure > > EventHubs (with preparations for Kinesis for EC2, and Kafka for > deployments > > in customer datacenters when needed) so can't just use disk level > > encryption (which would have its own overhead). We're putting all of our > > messages inside of encrypted envelopes before sending them to the stream > > which limits our opportunities for schema verification of the underlying > > messages to the declared type of the message. > > > > Encryption at rest mostly works out to a sales point for customers who > want > > assurances, and in a Kafka focused discussion might be dealt with by > > covering disk encryption and how the conversations between Kafka > instances > > are protected. > > > > Christian > > > > > > On Wed, Feb 25, 2015 at 11:51 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > Hey guys, > > > > > > One thing we tried to do along with the product release was start to > put > > > together a practical guide for using Kafka. I wrote this up here: > > > http://blog.confluent.io/2015/02/25/stream-data-platform-1/ > > > > > > I'd like to keep expanding on this as good practices emerge and we > learn > > > more stuff. So two questions: > > > 1. Anything you think other people should know about working with data > > > streams? What did you wish you knew when you got started? > > > 2. Anything you don't know about but would like to hear more about? > > > > > > -Jay > > > > > >
