Yeah, I've seen the same issue. It appears that the storm-kafka-client in versions < 1.1 has significant throughput problems. We saw a 10x speedup in moving to the 1.1 version. There is a PR out for this currently: https://github.com/apache/metron/pull/584
Casey On Tue, May 16, 2017 at 4:26 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > I am still facing this issue and couldn't manage to fix it. I would be > really grateful If somebody can help me. > > Thanks, > Ali > > On Sun, May 14, 2017 at 1:58 PM, Ali Nazemian <alinazem...@gmail.com> > wrote: > >> I was wrong. I think I couldn't increase the timeout value for Kafka >> spout properly. Therefore, I was wondering how I can increase the timeout >> value for Kafka spout? What is the right "-esc" property name I need to set >> in this case? Also, what has changed in the newer version since I didn't >> have this issue with the previous version? >> >> >> >> On Sun, May 14, 2017 at 3:00 AM, Ali Nazemian <alinazem...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I have installed the new version of HCP recently. I can see that the >>> following error has appeared in Storm UI at Kafka spout section related to >>> Parser topologies: >>> >>> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot >>> be completed since the group has already rebalanced and assigned the >>> partitions to another member. This means that the time between subsequent >>> calls to poll() was longer than the configured session.timeout.ms, >>> which typically implies that the poll loop is spending too much time >>> message processing. You can address this either by increasing the session >>> timeout or by reducing the maximum size of batches returned in poll() with >>> max.poll.records. at org.apache.kafka.clients.consu >>> mer.internals.ConsumerCoordinator$OffsetCommitResponseHandle >>> r.handle(ConsumerCoordinator.java:600) at org.apache.kafka.clients.consu >>> mer.internals.ConsumerCoordinator$OffsetCommitResponseHandle >>> r.handle(ConsumerCoordinator.java:541) at org.apache.kafka.clients.consu >>> mer.internals.AbstractCoordinator$CoordinatorResponseHandler >>> .onSuccess(AbstractCoordinator.java:679) at >>> org.apache.kafka.clients.consumer.internals.AbstractCoordina >>> tor$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658) >>> at org.apache.kafka.clients.consumer.internals.RequestFuture$1. >>> onSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consu >>> mer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at >>> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) >>> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkC >>> lient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426) >>> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278) >>> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkC >>> lient.clientPoll(ConsumerNetworkClient.java:360) at >>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkC >>> lient.poll(ConsumerNetworkClient.java:224) at >>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkC >>> lient.poll(ConsumerNetworkClient.java:192) at >>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkC >>> lient.poll(ConsumerNetworkClient.java:163) at >>> org.apache.kafka.clients.consumer.internals.ConsumerCoordina >>> tor.commitOffsetsSync(ConsumerCoordinator.java:426) at >>> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1059) >>> at >>> org.apache.storm.kafka.spout.KafkaSpout.commitOffsetsForAckedTuples(KafkaSpout.java:302) >>> at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:204) >>> at >>> org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) >>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at >>> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.ja >>> va:748) >>> >>> >>> This error has affected the Parsers throughput significantly! >>> >>> I have tried to increase the session timeout, but It didn't affect my >>> situation. I would be grateful if you can help me to find the source of >>> this issue. Please be advised that I haven't had this issue with the >>> previous version of Metron (0.3.1). >>> >>> Regards, >>> Ali >>> >>> >> >> >> -- >> A.Nazemian >> > > > > -- > A.Nazemian >
