Filesystem encryption is transparent to Kafka. You don't need to use SSL, but your encryption requirements may cause you to need SSL as well.
With regards to compression, without adding at rest encryption to Kafka (which is a very major piece of work, one that for sure requires a KIP and has many, many implications), there's not much to do there. I think it's worth examining your threat models that require encryption on disk without full disk encryption being suitable. Generally compromised broker machines means an attacker will be able to sniff in flight traffic anyway, if the goal is to never leak messages even if an attacker has full control of the broker machine, I'd suggest that that seems pretty impossible under current operating environments. If the issue is compliance, I'd recommend querying whichever compliance standard you're operating under about the suitability of full disk encryption, and careful thought about encrypting the most sensitive parts of messages. Whilst encryption in the producer and consumer does lead to performance issues and decrease the capability of compression to shrink a dataset, doing partial encryption of messages is easy enough. Generally we've found that the kinds of uses of Kafka that require in message encryption (alongside full disk encryption and SSL which we provide as standard) don't have such high throughput needs that they worry about compression etc. That clearly isn't true for all use cases though. Thanks Tom Crayford Heroku Kafka On Thursday, 2 June 2016, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > You could add a header to every message, with information whether it's > encrypted or not, then you don't have to encrypt all the messages, or you > only do it for some topics. > > On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts < > bruno.rassae...@novazone.be <javascript:;>> > wrote: > > > It works indeed but encrypting individual messages really influences the > > batch compression done by Kafka. > > Performance drops to about 1/3 of what it is without (even if we prepare > > the encrypted samples upfront). > > In the end what we going for is only encrypting what we really really > need > > to encrypt, not every message systematically. > > > > > On 31 May 2016, at 13:00, Gerard Klijs <gerard.kl...@dizzit.com > <javascript:;>> wrote: > > > > > > If you want system administrators not being able to see the data, the > > only > > > option is encryption, with only the clients sharing the key (or > whatever > > is > > > used to (de)crypt the data). Like the example from eugene. I don't know > > the > > > kind of messages you have, but you could always wrap something around > any > > > (de)serializer your currently using. > > > > > > On Tue, May 31, 2016 at 12:21 PM Bruno Rassaerts < > > > bruno.rassae...@novazone.be <javascript:;>> wrote: > > > > > >> I’ve asked the same question in the past, and disk encryption was > > >> suggested as a solution as well. > > >> However, as far as I know, disk encryption will not prevent your data > to > > >> be stolen when the machine is compromised. > > >> What we are looking for is even an additional barrier, so that even > > system > > >> administrators do not have access to the data. > > >> Any suggestions ? > > >> > > >>> On 24 May 2016, at 14:40, Tom Crayford <tcrayf...@heroku.com > <javascript:;>> wrote: > > >>> > > >>> Hi, > > >>> > > >>> There's no encryption at rest. It's recommended to use filesystem > > >>> encryption, or encryption of each individual message before producing > > it > > >>> for this. > > >>> > > >>> Only the new producer and consumers have SSL support. > > >>> > > >>> Thanks > > >>> > > >>> Tom Crayford > > >>> Heroku Kafka > > >>> > > >>> On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje < > > >>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: > > >>> > > >>>> > > >>>> > > >>>> Thanks for quick reply. > > >>>> > > >>>> Do you mean If I see messages in kafka, those will not be readable? > > >>>> > > >>>> And also, we are using new producer but old consumer , does old > > consumer > > >>>> have ssl support? > > >>>> > > >>>> As mentioned in document, its not there. > > >>>> > > >>>> > > >>>> Thanks, > > >>>> Snehalata > > >>>> > > >>>> ----- Original Message ----- > > >>>> From: "Mudit Kumar" <mudit.ku...@askme.in <javascript:;>> > > >>>> To: users@kafka.apache.org <javascript:;> > > >>>> Sent: Tuesday, May 24, 2016 3:53:26 PM > > >>>> Subject: Re: Kafka encryption > > >>>> > > >>>> Yes,it does that.What specifically you are looking for? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> On 5/24/16, 3:52 PM, "Snehalata Nagaje" < > > >>>> snehalata.nag...@harbingergroup.com <javascript:;>> wrote: > > >>>> > > >>>>> Hi All, > > >>>>> > > >>>>> > > >>>>> We have requirement of encryption in kafka. > > >>>>> > > >>>>> As per docs, we can configure kafka with ssl, for secured > > >> communication. > > >>>>> > > >>>>> But does kafka also stores data in encrypted format? > > >>>>> > > >>>>> > > >>>>> Thanks, > > >>>>> Snehalata > > >>>> > > >> > > >> > > > > >
