NotCoordinatorForGroupException in a long loop

2018-07-24 Thread Jamie Park
Hi, org.apache.kafka.common.errors.NotCoordinatorForGroupException: This is not the correct coordinator for this group. When there is an under-replication partition, it causes our stream-application to stall with NotCoorindatorForGroupException. This retriableException goes on in a loop for a l

dev and prod cluster consumption rate are different

2018-07-24 Thread Jamie Park
Hi, I have a kafka stream application that processes 250K events per minute in our dev kafka cluster but in production using the same binary, the performance degrades dramatically to a few thousands per minute. What could possibly be wrong? 1. server.properties in dev and prod, kafka brokers are

Consumer group record fetch rate

2018-04-22 Thread Jamie Park
Hi I am puzzled on this consumer group behavior. I have 40 consumers within a single consumer group with large lags. We are looking to increase partitions and number of consumers as lags indicate our records are produced faster than this consumer group can consume. However I observed thi

consistent consumption rate

2018-03-24 Thread Jamie Park
Hi, I am developing Kafka stream-application using low level processor API. I am consuming records from multiple topics, applying some joining logics, and then sending them off another topics. 1) How can I make sure my application consume records at consistent rate across different topics? If