Hi Christian, depending on how your Kafka topics are configured, you have 2 different options:
a) if you have a non-log-compacted then you can set the message retention on the topic to the desired value. In that case the message will be deleted by Kafka after the retention period expires. (the config value is ` retention.ms` I think) b) if you use Kafka as a log store with topics having infinite retention, then one common solution is to send a so-called tombstone record (a record with the same key containing only GDPR compatible data with the sensitive information removed), and let Kafka take care of the removal using log compaction. Kind regards, Sandor On Wed, 19 Aug 2020 at 16:53, Apolloni, Christian < christian.apoll...@baloise.ch> wrote: > Hello, > > I have some questions about implementing GDPR compliance in Kafka. > > In our situation we have the requirement of removing personal data from in > coordination with multiple systems. The idea is having a central > "coordinator system" which triggers the deletion process for the individual > systems in a specific, controlled sequence which takes into account the > various system inter-dependencies and data flows. This means e.g. system > nr. 2 will receive the delete order only after system nr. 1 has reported > that it's done with the deletion on its side (and so forth). > > One of the systems in question publishes data in Kafka topics for > consumption in other systems and part of the deletion process is to remove > the relevant personal data from these Kafka topics too. This has to happen > in a relatively short time after the deletion order is received, to prevent > a long delay before the systems further down the chain can start their own > deletion. Furthermore, we need to know when the operation is completed: > only at that point we can give the "go" to the other systems. > > We are unsure how to satisfy those requirements in Kafka. If anyone has > ideas or suggestions we would be very interested in your opinion. We are > also interested in general about experiences in implementing GDPR > compliance in Kafka, especially when dealing with multiple, interconnected > systems. > > Kind regards, > > -- > Christian Apolloni > > Disclaimer: The contents of this email and any attachment thereto are > intended exclusively for the attention of the addressee(s). The email and > any such attachment(s) may contain information that is confidential and > protected on the strength of professional, official or business secrecy > laws and regulations or contractual obligations. Should you have received > this email by mistake, you may neither make use of nor divulge the contents > of the email or of any attachment thereto. In such a case, please inform > the email's sender and delete the message and all attachments without delay > from your systems. > You can find our e-mail disclaimer statement in other languages under > http://www.baloise.ch/email_disclaimer >
