Hello, We have 4 topics deployed on 4 node kafka cluster. For one of the topic we are trying to read data from beginning, using the kafka high level consumer.
the topic has 32 partitions and we create 32 streams using high level consumer so that one partition per stream is used, we then have 32 threads each working with one stream that we create. we have set the "consumer.timeout.ms" to 10000(10 sec ). We are seeing issues where most of threads end prematurely by not reading the entire partition and timing out. The way i have interpreted "consumer.timeout.ms" -- the consumer will wait for the configured milliseconds once it reaches the end of a partition for a new message before it throws ConsumertimeoutException. is this correct ? if yes then the only reason i see the exceptions before we reach the end of partition is if the leader for that partition is blocked doing something else and hence is not serving the request to read. is this even possible ? given kafka is such high throughput system. Thanks in advance for helping ! anishek
