Re: KIP-382 + Kafka Streams Question

2019-07-24 Thread Adam Bellemare
Hi Ryanne > Lemme know if I haven't answered this clearly. Nope, this was very helpful. Thank you! > A single "stream" can come from multiple input topics I overlooked that - I was thinking of simply using the StreamBuilder.table() functionality instead, but that function doesn't support a

Re: KIP-382 + Kafka Streams Question

2019-07-23 Thread Ryanne Dolan
Adam, I think we are converging :) > "userEntity"...where I only want the latest emailAddress (basic materialization) to send an email on account password update. Yes, you want all "userEntity" data on both clusters. Each cluster will have "userEntity" and the remote counterpart

Re: KIP-382 + Kafka Streams Question

2019-07-23 Thread Adam Bellemare
Hi Ryanne Thanks for the clarifications! Here is one of my own, as I think it's the biggest stumbling block in my description: *> What is "table" exactly? I am interpreting this as a KTable changelog topic* "table" is not a KTable changelog topic, but simply entity data that is to be

Re: KIP-382 + Kafka Streams Question

2019-07-23 Thread Ryanne Dolan
Adam, > I think we have inconsistent definitions of Active-Active Yes, this terminology gets thrown around a lot. IMO "active" means both producers and consumers are using a cluster under normal operation -- not just during outages, and not just by something like MM2. (Obviously, MM2 has

Re: KIP-382 + Kafka Streams Question

2019-07-23 Thread Adam Bellemare
Hi Ryanne I think we have inconsistent definitions of Active-Active. The producer is only producing to one cluster (primary) and one topic (topic "table"), and the other cluster (secondary) contains only a replication of the data via MM2 ("primary.table"). What you seemed to be proposing is that

Re: KIP-382 + Kafka Streams Question

2019-07-22 Thread Ryanne Dolan
Hello Adam, thanks for the questions. Yes my organization uses Streams, and yes you can use Streams with MM2/KIP-382, though perhaps not in the way you are describing. The architecture you mention is more "active/standby" than "active/active" IMO. The "secondary" cluster is not being used until a

KIP-382 + Kafka Streams Question

2019-07-22 Thread Adam Bellemare
Hi Ryanne I have a quick question for you about Active+Active replication and Kafka Streams. First, does your org /do you use Kafka Streams? If not then I think this conversation can end here. ;) Secondly, and for the broader Kafka Dev group - what happens if I want to use Active+Active