Hi all, there has been an interesting talk about this during a previous Kafka Summit. It talks about using crypto-shredding to 'forget' user information. I'm not sure if there are any slides, but it basically suggests that you'd encrypt user data on Kafka, and when you get a information removal request, the only thing you have to do is to delete the encryption key for that user.
Here's the announcement of the talk: https://kafka-summit.org/sessions/handling-gdpr-apache-kafka-comply-without-freaking/, but not sure where slides or a recording can be found unfortunately. Hope it helps. BR, Patrick On Wed, 19 Aug 2020 at 18:16, Nemeth Sandor <[email protected]> wrote: > Hi Christian, > > depending on how your Kafka topics are configured, you have 2 different > options: > > a) if you have a non-log-compacted then you can set the message retention > on the topic to the desired value. In that case the message will be deleted > by Kafka after the retention period expires. (the config value is ` > retention.ms` I think) > > b) if you use Kafka as a log store with topics having infinite retention, > then one common solution is to send a so-called tombstone record (a record > with the same key containing only GDPR compatible data with the sensitive > information removed), and let Kafka take care of the removal using log > compaction. > > Kind regards, > Sandor > > > On Wed, 19 Aug 2020 at 16:53, Apolloni, Christian < > [email protected]> wrote: > > > Hello, > > > > I have some questions about implementing GDPR compliance in Kafka. > > > > In our situation we have the requirement of removing personal data from > in > > coordination with multiple systems. The idea is having a central > > "coordinator system" which triggers the deletion process for the > individual > > systems in a specific, controlled sequence which takes into account the > > various system inter-dependencies and data flows. This means e.g. system > > nr. 2 will receive the delete order only after system nr. 1 has reported > > that it's done with the deletion on its side (and so forth). > > > > One of the systems in question publishes data in Kafka topics for > > consumption in other systems and part of the deletion process is to > remove > > the relevant personal data from these Kafka topics too. This has to > happen > > in a relatively short time after the deletion order is received, to > prevent > > a long delay before the systems further down the chain can start their > own > > deletion. Furthermore, we need to know when the operation is completed: > > only at that point we can give the "go" to the other systems. > > > > We are unsure how to satisfy those requirements in Kafka. If anyone has > > ideas or suggestions we would be very interested in your opinion. We are > > also interested in general about experiences in implementing GDPR > > compliance in Kafka, especially when dealing with multiple, > interconnected > > systems. > > > > Kind regards, > > > > -- > > Christian Apolloni > > > > Disclaimer: The contents of this email and any attachment thereto are > > intended exclusively for the attention of the addressee(s). The email and > > any such attachment(s) may contain information that is confidential and > > protected on the strength of professional, official or business secrecy > > laws and regulations or contractual obligations. Should you have received > > this email by mistake, you may neither make use of nor divulge the > contents > > of the email or of any attachment thereto. In such a case, please inform > > the email's sender and delete the message and all attachments without > delay > > from your systems. > > You can find our e-mail disclaimer statement in other languages under > > http://www.baloise.ch/email_disclaimer > > > -- Patrick Plaatje
