Hi Mike,
Are you performing any operations (e.g. joins) across all topics?
If so, I'd think increasing the number of partitions is indeed the way
to go. Partition is the unit of parallelism per topic and all topics are
bound together in your app in that case.
If not, your other option is to break up your application into a number
of KafkaStreams instances, each dealing with a subset of topics.
Hope that helps.
Michał
On 23/05/17 08:47, Mike Gould wrote:
Hi
We have a couple of hundred topics - each carrying a similar but distinct
message type but to keep the total partition count down each only has 3
partitions.
If I start Kafka-streams consuming all topics only 3 threads ever get
assigned any partitions.
I think the first thread to start gets the first partition of each topic,
and so on until the 3rd thread, after that all the partitions are assigned
- any further threads are just left idle.
Is there any way to make the partition assignment smarter and either add a
random element that moves partitions when further consumers start or
considers all the partitions of subscribed topics together when assigning
them?
Our only alternative is creating many more partitions for each topic - and
we're worried about how far this will scale.
Thank you
Mike G
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowie...@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmas...@openbet.com <mailto:postmas...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612
