I've seen interest in HDFS-like "encryption zones" in Kafka. This has the advantage of magically encrypting data at rest regardless of which client is used as a producer. Adding it on the client side limits the feature to the java client.
Gwen On Fri, Jul 31, 2015 at 1:20 PM, eugene miretsky <eugene.miret...@gmail.com> wrote: > I think that Hadoop and Cassandra do [1] (Transparent Encryption) > > We're doing [2] (on a side note, for [2] you still need authentication on > the producer side - you don't want an unauthorized user writing garbage). > Right now we have the 'user' doing the encryption and submitting raw bytes > to the producer. I was suggesting implementing an encryptor in the > producer itself - I think it's cleaner and can be reused by other users > (instead of having to do their own encryption) > > Cheers, > Eugene > > On Fri, Jul 31, 2015 at 4:04 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > >> I think the goal here is to make the actual message stored on broker to be >> encrypted, because after we have SSL, the transmission would be encrypted. >> >> In general there might be tow approaches: >> 1. Broker do the encryption/decryption >> 2. Client do the encryption/decryption >> >> From performance point of view, I would prefer [2]. It is just in that >> case, maybe user does not necessarily need to use SSL anymore because the >> data would be encrypted anyway. >> >> If we let client do the encryption, there are also two ways to do so - >> either we let producer take an encryptor or users can do >> serialization/encryption outside the producer and send raw bytes. The only >> difference between the two might be flexibility. For example, if someone >> wants to know the actual bytes of a message that got sent over the wire, >> doing it outside the producer would probably more preferable. >> >> Jiangjie (Becket) Qin >> >> On Thu, Jul 30, 2015 at 12:16 PM, eugene miretsky < >> eugene.miret...@gmail.com >> > wrote: >> >> > Hi, >> > >> > Based on the security wiki page >> > <https://cwiki.apache.org/confluence/display/KAFKA/Security> encryption >> of >> > data at rest is out of scope for the time being. However, we are >> > implementing encryption in Kafka and would like to see if there is >> > interest in submitting a patch got it. >> > >> > I suppose that one way to implement encryption would be to add an >> > 'encrypted key' field to the Message/MessageSet structures in the >> > wire protocole - however, this is a very big and fundamental change. >> > >> > A simpler way to add encryption support would be: >> > 1) Custom Serializer, but it wouldn't be compatible with other custom >> > serializers (Avro, etc. ) >> > 2) Add a step in KafkaProducer after serialization to encrypt the data >> > before it's being submitted to the accumulator (encryption is done in the >> > submitting thread, not in the producer io thread) >> > >> > Is there interest in adding #2 to Kafka? >> > >> > Cheers, >> > Eugene >> > >>
