Re: KIP-382 + Kafka Streams Question

Hi Ryanne > Lemme know if I haven't answered this clearly. Nope, this was very helpful. Thank you! > A single "stream" can come from multiple input topics I overlooked that - I was thinking of simply using the StreamBuilder.table() functionality instead, but that function doesn't support a

Re: KIP-382 + Kafka Streams Question

Adam, I think we are converging :) > "userEntity"...where I only want the latest emailAddress (basic materialization) to send an email on account password update. Yes, you want all "userEntity" data on both clusters. Each cluster will have "userEntity" and the remote counterpart

Re: KIP-382 + Kafka Streams Question

Hi Ryanne Thanks for the clarifications! Here is one of my own, as I think it's the biggest stumbling block in my description: *> What is "table" exactly? I am interpreting this as a KTable changelog topic* "table" is not a KTable changelog topic, but simply entity data that is to be

Re: KIP-382 + Kafka Streams Question

Adam, > I think we have inconsistent definitions of Active-Active Yes, this terminology gets thrown around a lot. IMO "active" means both producers and consumers are using a cluster under normal operation -- not just during outages, and not just by something like MM2. (Obviously, MM2 has

Re: KIP-382 + Kafka Streams Question

Hi Ryanne I think we have inconsistent definitions of Active-Active. The producer is only producing to one cluster (primary) and one topic (topic "table"), and the other cluster (secondary) contains only a replication of the data via MM2 ("primary.table"). What you seemed to be proposing is that

Re: KIP-382 + Kafka Streams Question

Hello Adam, thanks for the questions. Yes my organization uses Streams, and yes you can use Streams with MM2/KIP-382, though perhaps not in the way you are describing. The architecture you mention is more "active/standby" than "active/active" IMO. The "secondary" cluster is not being used until a

KIP-382 + Kafka Streams Question

Hi Ryanne I have a quick question for you about Active+Active replication and Kafka Streams. First, does your org /do you use Kafka Streams? If not then I think this conversation can end here. ;) Secondly, and for the broader Kafka Dev group - what happens if I want to use Active+Active

Re: Kafka Streams question

Also, in the next version of Kafka / Kafka Streams such "intermediate" topics will automatically be created for you when you do joins or aggregations: https://issues.apache.org/jira/browse/KAFKA-3561 So my previous message explained your options today when using the current release of Kafka

Re: Kafka Streams question

Poul, to add to what Matthias said: If you are wondering how to manually create a topic, you have basically two options. A. Use Kafka's CLI tools to create the topic "from the outside". # Example $ kafka-topics.sh --create --topic my-custom-toipc --zookeeper localhost:2181 --partitions

Re: Kafka Streams question

Hi, you can manually create a topic with the number of partitions you want to have and use this topic via through() KStream input = ... input.map().through("manually-created-topic").join(...) However, both KStream and KTable need to have the same number of partitions for perform the join.

Kafka Streams question

Hi! I am prototyping some code using Kafka Streams, and have a question. I need to map a stream into another (with different partition key) and join it with a table. How do I control number of partitions of the mapped stream? Thanks! Poul Costinsky Chief Architect