On 2020/08/19 16:15:40, Nemeth Sandor <s...@gmail.com> wrote: > Hi Christian,>
Hi, thanks for your reply. > depending on how your Kafka topics are configured, you have 2 different> > options:> > > a) if you have a non-log-compacted then you can set the message retention> > on the topic to the desired value. In that case the message will be deleted> > by Kafka after the retention period expires. (the config value is `> > retention.ms` I think)> That's what we thought too at first as solution, but we likely cannot set the retention low enough. > b) if you use Kafka as a log store with topics having infinite retention,> > then one common solution is to send a so-called tombstone record (a record> > with the same key containing only GDPR compatible data with the sensitive> > information removed), and let Kafka take care of the removal using log> > compaction.> We also thought about this, but as far as we understood there is no real guarantee that the compaction completes in a given time for all messages in the topic. From what we understood compaction can be delayed by the messages still being in the active segment and/or the compaction thread pool being too busy. It's also unclear to us how we can know that the compaction has completed for all relevant messages and that we can safely report to our "coordinator system" that the next system can start its own deletion process safely. Kind Regards, -- Christian Apolloni Disclaimer: The contents of this email and any attachment thereto are intended exclusively for the attention of the addressee(s). The email and any such attachment(s) may contain information that is confidential and protected on the strength of professional, official or business secrecy laws and regulations or contractual obligations. Should you have received this email by mistake, you may neither make use of nor divulge the contents of the email or of any attachment thereto. In such a case, please inform the email's sender and delete the message and all attachments without delay from your systems. You can find our e-mail disclaimer statement in other languages under http://www.baloise.ch/email_disclaimer
