Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

For what it's worth, here is an example sketch that I came up with. Point is to show an alternative direction for the KStreams logo. https://ibb.co/bmZxDCg Thinking process: - It shows much more clearly (I hope) that KStreams is an official part of Kafka. - The Kafka logo is still front

Re: GDPR compliance

Hey Christian, my understanding is that you have an upstream system publishing data via Kafka topic to a downstream system, and your goal is to delete the PII data both from Kafka and the downstream system via a message published through the same topic. Is my understanding correct? Does the coord

Re: Mirror Maker 2.0 Queries

Any help here would be greatly appreciated. On Sat, Aug 8, 2020, 12:13 PM Ananya Sen wrote: > Thank you Ryanne for the quick response. > I further want to clarify a few points. > > The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka > connect we have multiple workers and each

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

Hey Ben, that otter was supposed to be a river-otter to connect to "streams". And of course, it's cute :) On Wed, Aug 19, 2020 at 12:41 PM Philip Schmitt wrote: > Hi, > > I’m with Robin and Michael here. > > What this decision needs is a good design brief. > This article seems decent: > https:/

Re: GDPR compliance

Yup. The crypto-shredding approach tends to be the most practical. Basically do payload encryption of your PI and with a unique per-user key. Throw away the per user key, and the data is "deleted" from a CCPA perspective. The alternative is to have the relevant topic have tight retention SLAs, whi

Re: MirrorMaker 2.0 - Translating offsets for remote topics and consumer groups

Josh, if you have two clusters with bidirectional replication, you only get two copies of each record. MM2 won't replicate the data "upstream", cuz it knows it's already there. In particular, MM2 knows not to create topics like B.A.topic1 on cluster A, as this would be an unnecessary cycle. > is

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

Hi, I’m with Robin and Michael here. What this decision needs is a good design brief. This article seems decent: https://yourcreativejunkie.com/logo-design-brief-the-ultimate-guide-for-designers/ Robin is right about the usage requirements. It goes a bit beyond resolution. How does the logo wor

Re: MirrorMaker 2.0 - Translating offsets for remote topics and consumer groups

Sorry, correction -- I am realizing now it would be 3 copies of the same topic data as A.topic1 has different data than B.topic1. However, that would still be 3 copies as opposed to just 2 with something like topic1 and A.topic1. As well, if I were to explicitly replicate the remote topic back to

Re: GDPR compliance

> Hi all,> > > there has been an interesting talk about this during a previous Kafka> > Summit. It talks about using crypto-shredding to 'forget' user information.> > I'm not sure if there are any slides, but it basically suggests that you'd> > encrypt user data on Kafka, and when you get a informa

Re: MirrorMaker 2.0 - Translating offsets for remote topics and consumer groups

Thanks for the clarification Ryanne. In the context of active/active clusters, does this mean there would be 6 copies of the same topic data? A topics: - topic1 - B.topic1 - B.A.topic1 B topics: - topic1 - A.topic1 - A.B.topic1 Out of curiosity, is there a reason for MM2 not emitting checkpoint

Re: GDPR compliance

Hi all, there has been an interesting talk about this during a previous Kafka Summit. It talks about using crypto-shredding to 'forget' user information. I'm not sure if there are any slides, but it basically suggests that you'd encrypt user data on Kafka, and when you get a information removal re

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

Purely out of curiosity, why an otter? Is there some reasoning behind it or is it just because it's cute? On Wed, 19 Aug 2020 at 17:57, Guozhang Wang wrote: > Hi Michael, > > Thanks for the feedback, but I'm not totally in agreement with your > proposed objectives of the logos. More specifically

Re: Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x

Sure Manoj! Really appreciate your quick response.. On 2020/08/19 17:40:54, wrote: > Great . > Share your finding to this group once you done upgrade Confluent Kafka 4.1x > to 5.3x successfully . > > I see many people having same question here . > > On 8/19/20, 10:38 AM, "Rijo Roy" wrot

Re: GDPR compliance

As alternative solution we also investigated encryption: encrypting all messages with an individual key and removing the key once the "deletion" needs to be performed. Has anyone experience with such a solution? --  Christian Apolloni Disclaimer: The contents of this email and any attachme

Re: GDPR compliance

Hi Sandor, thanks again for your reply. > If you have a non-log-compacted topic, after `retention.ms` the message> > (along with the PII) gets deleted from the Kafka message store without any> > further action, which should satisfy GDPR requirements:> > - you are handling PII in Kafka for a limite

Re: Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x

Great . Share your finding to this group once you done upgrade Confluent Kafka 4.1x to 5.3x successfully . I see many people having same question here . On 8/19/20, 10:38 AM, "Rijo Roy" wrote: [External] Thanks Manoj! Yeah, the plan is to start with non-prod and validate fir

Re: Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x

Thanks Manoj! Yeah, the plan is to start with non-prod and validate first before going to prod. Thanks & Regards, Rijo Roy On 2020/08/19 17:33:53, wrote: > I advise to do it non-prod for validation . > You can backup data log folder if you want but I have'nt see any issue . but > better to b

Re: Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x

I advise to do it non-prod for validation . You can backup data log folder if you want but I have'nt see any issue . but better to backup data if it small . Don’t change below value to latest until you done full validation , once you changed to latest then you can't rollback . inter.broker.pro

Re: GDPR compliance

Hi Christian, disclaimer: IANAL, so take everything with a grain of salt from the legal perspective, I'm sharing the experience I have handling PII data with Kafka in an ecommerce system, so your requirements may differ. I'm not sure how your system is designed but in general from a data manageme

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

Hi Michael, Thanks for the feedback, but I'm not totally in agreement with your proposed objectives of the logos. More specifically, though I agree Kafka's logo is a smartly designed one to convey some key ideas of the project such as "distributed" and "pipeline" in a techy manner, I'm not convinc

Re: Steps & best-practices to upgrade Confluent Kafka 4.1x to 5.3x

Thanks Manoj! Appreciate your help.. I will follow the steps you pointed out.. Do you think there is a need to : 1. backup the data before the rolling upgrade 2. some kind of datasync that should be considered here.. I don't think this is required as I am performing an in-place upgrade.. Thank

Re: GDPR compliance

On 2020/08/19 16:15:40, Nemeth Sandor wrote: > Hi Christian,> Hi, thanks for your reply. > depending on how your Kafka topics are configured, you have 2 different> > options:> > > a) if you have a non-log-compacted then you can set the message retention> > on the topic to the desired value. In t

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

I echo what Michael says here. Another consideration is that logos are often shrunk (when used on slides) and need to work at lower resolution (think: printing swag, stitching socks, etc) and so whatever logo we come up with needs to not be too fiddly in the level of detail - something that I thin

Re: GDPR compliance

Hi Christian, depending on how your Kafka topics are configured, you have 2 different options: a) if you have a non-log-compacted then you can set the message retention on the topic to the desired value. In that case the message will be deleted by Kafka after the retention period expires. (the co

Re: GDPR compliance

Be aware that deleting personal data is already processing ! You will already need user consent to process it In Kafka - even if it is about deletion . Simply do not collect it. > Am 19.08.2020 um 16:53 schrieb Apolloni, Christian > : > > Hello, > > I have some questions about implementing

GDPR compliance

Hello, I have some questions about implementing GDPR compliance in Kafka. In our situation we have the requirement of removing personal data from in coordination with multiple systems. The idea is having a central "coordinator system" which triggers the deletion process for the individual syste

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

Hi all! Great to see we are in the process of creating a cool logo for Kafka Streams. First, I apologize for sharing feedback so late -- I just learned about it today. :-) Here's my *personal, subjective* opinion on the currently two logo candidates for Kafka Streams. TL;DR: Sorry, but I really

Re: Kafka Streams Key-value store question

Hi Pirow, If I'm understanding your requirements correctly, I think using a global store

Re: MirrorMaker 2.0 - Translating offsets for remote topics and consumer groups

Josh, yes it's possible to migrate the consumer group back to the source topic, but you need to explicitly replicate the remote topic back to the source cluster -- otherwise no checkpoints will flow "upstream": A->B.topics=test1 B->A.topics=A.test1 After the first checkpoint is emitted upstream,

Kafka Streams Key-value store question

Hello, We're building a JSON decorator using Kafka Streams' processing API. The process is briefly that a piece of JSON should be consumed from an input topic (keys are null, value is the JSON). The JSON contains a field (e.g. "thisField") with a value (e.g. "someLink") . This value (and a time

Re: Kafka BrokerState Metric Value 3

Note that even when all partitions are in sync, leader election might have not happened yet and the broker isn't serving anything. Which might be OK, depending on your actual use case. On Wed, Aug 19, 2020 at 11:40 AM Dhirendra Singh wrote: > Thank you Peter ! > I intended to use broker state to

Re: Kafka BrokerState Metric Value 3

Thank you Peter ! I intended to use broker state to determine the health but i was not sure. I will use under replicated partition metric instead. --dsingh On Wed, Aug 19, 2020 at 1:40 PM Peter Bukowinski wrote: > The broker state metric just reports on the state of the broker itself, > not whe

MirrorMaker 2 WorkerSourceTask Failed to flush error messages

Hello, I'm seeing large lag sometimes on my MM2 clusters after restarting the cluster (it runs on k8s). I have 3 mm2 clusters, each one reads from 1 source and writes to the same destination. I am seeing these errors on one of my clusters right now. WorkerSourceTask{id=MirrorSourceConnector-33} Fa

Re: Kafka BrokerState Metric Value 3

The broker state metric just reports on the state of the broker itself, not whether it is in sync. A replacement broker will quickly reach a broker state of 3 on startup even though it has to catch up on many replicas. Don’t rely on it for checking if a cluster/broker is healthy with no under-re

Re: Kafka BrokerState Metric Value 3

So is this metric just gives information that broker process up and running ? or does it indicate something more of broker state or partitions it hold ? On Mon, Aug 17, 2020 at 6:17 PM Karolis Pocius wrote: > I tried using this metric for determining when the broker is back in the > cluster an