Indeed to get proper performance, messages need to be batched before encryption. However, this is not that straightforward to implement and Kafka has already a very good batching algorithm. For example, when do you decide to no longer wait for additional messages and send a non-full batch ? Not that obvious.
Ideally, we would like kafka clients to encrypt/decrypt the compressed batches of kafka. That seems like the ideal place to do this. > On 03 Jun 2016, at 07:27, Jim Hoagland <jim_hoagl...@symantec.com> wrote: > > I'm hesitant to cite it because it wasn't really a proper benchmark, but > with the end-to-end encryption through Kafka proof of concept described at > http://symc.ly/1pC2CEG, doing the encryption added only 26% to the time > taken to send messages and only 6% to the time taken to consume messages. > This is with batching 200 300-byte messages together for encryption. More > details are in the blog post. > > Personally I think that encrypting sensitive data before handing it to > Kafka (or at least before it leaves the producing box) just makes sense to > do if the situation allows. The Kafka installation wouldn't be able > reveal the data even if its systems and networks are compromised because > it never sees the data in the clear and doesn't know how to decrypt it. > In the way we set it up, someone would need the recipient's RSA private > key to decrypt (or would need to have compromised a decrypting system). > > -- Jim > > > On 6/2/16, 2:56 AM, "Tom Crayford" <tcrayf...@heroku.com> wrote: > >> Filesystem encryption is transparent to Kafka. You don't need to use SSL, >> but your encryption requirements may cause you to need SSL as well. >> >> With regards to compression, without adding at rest encryption to Kafka >> (which is a very major piece of work, one that for sure requires a KIP and >> has many, many implications), there's not much to do there. I think it's >> worth examining your threat models that require encryption on disk without >> full disk encryption being suitable. Generally compromised broker machines >> means an attacker will be able to sniff in flight traffic anyway, if the >> goal is to never leak messages even if an attacker has full control of the >> broker machine, I'd suggest that that seems pretty impossible under >> current >> operating environments. >> >> If the issue is compliance, I'd recommend querying whichever compliance >> standard you're operating under about the suitability of full disk >> encryption, and careful thought about encrypting the most sensitive parts >> of messages. Whilst encryption in the producer and consumer does lead to >> performance issues and decrease the capability of compression to shrink a >> dataset, doing partial encryption of messages is easy enough. >> >> Generally we've found that the kinds of uses of Kafka that require in >> message encryption (alongside full disk encryption and SSL which we >> provide >> as standard) don't have such high throughput needs that they worry about >> compression etc. That clearly isn't true for all use cases though. >> >> Thanks >> >> Tom Crayford >> Heroku Kafka >> >> On Thursday, 2 June 2016, Gerard Klijs <gerard.kl...@dizzit.com> wrote: >> >>> You could add a header to every message, with information whether it's >>> encrypted or not, then you don't have to encrypt all the messages, or >>> you >>> only do it for some topics. >>> >>> On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts < >>> bruno.rassae...@novazone.be <javascript:;>> >>> wrote: >>> >>>> It works indeed but encrypting individual messages really influences >>> the >>>> batch compression done by Kafka. >>>> Performance drops to about 1/3 of what it is without (even if we >>> prepare >>>> the encrypted samples upfront). >>>> In the end what we going for is only encrypting what we really really >>> need >>>> to encrypt, not every message systematically. >>>> >>>>> On 31 May 2016, at 13:00, Gerard Klijs <gerard.kl...@dizzit.com >>> <javascript:;>> wrote: >>>>> >>>>> If you want system administrators not being able to see the data, >>> the >>>> only >>>>> option is encryption, with only the clients sharing the key (or >>> whatever >>>> is >>>>> used to (de)crypt the data). Like the example from eugene. I don't >>> know >>>> the >>>>> kind of messages you have, but you could always wrap something >>> around >>> any >>>>> (de)serializer your currently using. >>>>> >>>>> On Tue, May 31, 2016 at 12:21 PM Bruno Rassaerts < >>>>> bruno.rassae...@novazone.be <javascript:;>> wrote: >>>>> >>>>>> I’ve asked the same question in the past, and disk encryption was >>>>>> suggested as a solution as well. >>>>>> However, as far as I know, disk encryption will not prevent your >>> data >>> to >>>>>> be stolen when the machine is compromised. >>>>>> What we are looking for is even an additional barrier, so that even >>>> system >>>>>> administrators do not have access to the data. >>>>>> Any suggestions ? >>>>>> >>>>>>> On 24 May 2016, at 14:40, Tom Crayford <tcrayf...@heroku.com >>> <javascript:;>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> There's no encryption at rest. It's recommended to use filesystem >>>>>>> encryption, or encryption of each individual message before >>> producing >>>> it >>>>>>> for this. >>>>>>> >>>>>>> Only the new producer and consumers have SSL support. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Tom Crayford >>>>>>> Heroku Kafka >>>>>>> >>>>>>> On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje < >>>>>>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks for quick reply. >>>>>>>> >>>>>>>> Do you mean If I see messages in kafka, those will not be >>> readable? >>>>>>>> >>>>>>>> And also, we are using new producer but old consumer , does old >>>> consumer >>>>>>>> have ssl support? >>>>>>>> >>>>>>>> As mentioned in document, its not there. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Snehalata >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Mudit Kumar" <mudit.ku...@askme.in <javascript:;>> >>>>>>>> To: users@kafka.apache.org <javascript:;> >>>>>>>> Sent: Tuesday, May 24, 2016 3:53:26 PM >>>>>>>> Subject: Re: Kafka encryption >>>>>>>> >>>>>>>> Yes,it does that.What specifically you are looking for? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 5/24/16, 3:52 PM, "Snehalata Nagaje" < >>>>>>>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> >>>>>>>>> We have requirement of encryption in kafka. >>>>>>>>> >>>>>>>>> As per docs, we can configure kafka with ssl, for secured >>>>>> communication. >>>>>>>>> >>>>>>>>> But does kafka also stores data in encrypted format? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Snehalata >>>>>>>> >>>>>> >>>>>> >>>> >>>> >>> >
