Re: Partition assignment with multiple topics

2017-05-23 Thread Mike Gould
Hi No joins - they're all separate data flows. Having separate stream instances for subsets of topics would probably work. Is doesn't seem as clean. It's also slightly more tricky to distribute the load across separate processes. We'd have to have only one stream thread allocated to each topic in

Re: Partition assignment with multiple topics

2017-05-23 Thread Michal Borowiecki
Hi Mike, Are you performing any operations (e.g. joins) across all topics? If so, I'd think increasing the number of partitions is indeed the way to go. Partition is the unit of parallelism per topic and all topics are bound together in your app in that case. If not, your other option is to

Partition assignment with multiple topics

2017-05-23 Thread Mike Gould
Hi We have a couple of hundred topics - each carrying a similar but distinct message type but to keep the total partition count down each only has 3 partitions. If I start Kafka-streams consuming all topics only 3 threads ever get assigned any partitions. I think the first thread to start gets