Re: GDPR compliance

2020-08-19 Thread Nemeth Sandor
Hey Christian, my understanding is that you have an upstream system publishing data via Kafka topic to a downstream system, and your goal is to delete the PII data both from Kafka and the downstream system via a message published through the same topic. Is my understanding correct? Does the coord

Re: GDPR compliance

2020-08-19 Thread Christopher Smith
Yup. The crypto-shredding approach tends to be the most practical. Basically do payload encryption of your PI and with a unique per-user key. Throw away the per user key, and the data is "deleted" from a CCPA perspective. The alternative is to have the relevant topic have tight retention SLAs, whi

Re: GDPR compliance

2020-08-19 Thread Apolloni, Christian
> Hi all,> > > there has been an interesting talk about this during a previous Kafka> > Summit. It talks about using crypto-shredding to 'forget' user information.> > I'm not sure if there are any slides, but it basically suggests that you'd> > encrypt user data on Kafka, and when you get a informa

Re: GDPR compliance

2020-08-19 Thread Patrick Plaatje
Hi all, there has been an interesting talk about this during a previous Kafka Summit. It talks about using crypto-shredding to 'forget' user information. I'm not sure if there are any slides, but it basically suggests that you'd encrypt user data on Kafka, and when you get a information removal re

Re: GDPR compliance

2020-08-19 Thread Apolloni, Christian
As alternative solution we also investigated encryption: encrypting all messages with an individual key and removing the key once the "deletion" needs to be performed. Has anyone experience with such a solution? --  Christian Apolloni Disclaimer: The contents of this email and any attachme

Re: GDPR compliance

2020-08-19 Thread Apolloni, Christian
Hi Sandor, thanks again for your reply. > If you have a non-log-compacted topic, after `retention.ms` the message> > (along with the PII) gets deleted from the Kafka message store without any> > further action, which should satisfy GDPR requirements:> > - you are handling PII in Kafka for a limite

Re: GDPR compliance

2020-08-19 Thread Nemeth Sandor
Hi Christian, disclaimer: IANAL, so take everything with a grain of salt from the legal perspective, I'm sharing the experience I have handling PII data with Kafka in an ecommerce system, so your requirements may differ. I'm not sure how your system is designed but in general from a data manageme

Re: GDPR compliance

2020-08-19 Thread Apolloni, Christian
On 2020/08/19 16:15:40, Nemeth Sandor wrote: > Hi Christian,> Hi, thanks for your reply. > depending on how your Kafka topics are configured, you have 2 different> > options:> > > a) if you have a non-log-compacted then you can set the message retention> > on the topic to the desired value. In t

Re: GDPR compliance

2020-08-19 Thread Nemeth Sandor
Hi Christian, depending on how your Kafka topics are configured, you have 2 different options: a) if you have a non-log-compacted then you can set the message retention on the topic to the desired value. In that case the message will be deleted by Kafka after the retention period expires. (the co

Re: GDPR compliance

2020-08-19 Thread Jörn Franke
Be aware that deleting personal data is already processing ! You will already need user consent to process it In Kafka - even if it is about deletion . Simply do not collect it. > Am 19.08.2020 um 16:53 schrieb Apolloni, Christian > : > > Hello, > > I have some questions about implementing