Henry, you might want to check out the docs, that give an overview of the architecture: http://docs.confluent.io/current/streams/architecture.html#example
Also, I am wondering why your application did not crash: I would expect an exception like java.lang.IllegalArgumentException: Assigned partition foo-2 for non-subscribed topic regex pattern; subscription pattern is bar Maybe you just don't hit it, because both topics have a single partition and not multiple. >> Out of interest though, had I subscribed for both topics in one subscriber >> - I would have expected records for both topics interleaved Yes. That should happen. > why when >> running this in two separate processes do I not observe the same? Not sure what you mean by this? >> If I fix this by changing the application ID for each streaming process - >> does this mean I lose the ability to share state stores between the >> applications? Yes. If both your topics are single partitioned, and you want to share state, you will not be able to run with more then one thread in your Streams app. The only way to work around this, would be to copy the data into another topic with more partitions before you process them -- of course, this would mean data duplication. -Matthias On 4/28/17 12:45 PM, Henry Thacker wrote: > Thanks Michael and Eno for your help - I always thought the unit of > parallelism was a combination of topic & partition rather than just > partition. > > Out of interest though, had I subscribed for both topics in one subscriber > - I would have expected records for both topics interleaved, why when > running this in two separate processes do I not observe the same? Just > wanting to try and form a mental model of how this is all working - I will > try and look through some code over the weekend. > > If I fix this by changing the application ID for each streaming process - > does this mean I lose the ability to share state stores between the > applications? > > Unfortunately the data on the input topics are provided by a third party > component which sends these keyless messages on a single partition per > topic, so I have little ability to fix this at source :-( > > Thanks, > Henry >
signature.asc
Description: OpenPGP digital signature
