Hi all, I’m running Kafka 0.10.0.1 on Java 8 on Linux — same for brokers and streams nodes.
I’m attempting to scale up a Streams app that does a lot of I/O — in particular, I’m hoping to isolate each partition into its own thread — and I’m confused: * I recently created 11 new topics and partitions and modified this app to consume from the 11 new topics * The app is now consuming from 130 partitions across those 11 topics * I’ve set the config value num.stream.threads to 130 * In the logs from the startup routine, I can see that Streams is creating 130 threads * When I use kafka-consumer-groups to see the status of the app, in particular lag, I can see that it’s only using 20 of those threads to consume from its topics * This means, of course, that when processing for one partition blocks, it blocks processing for any other partition assigned to the same thread. So: * Why is this happening? * How can I achieve my goal? Thank you, Avi ———— Software Architect @ Park Assist We’re hiring! http://tech.parkassist.com/jobs/
