[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15793692#comment-15793692 ] Ismael Juma commented on KAFKA-3172: Thanks [~oleg_gorobets]. JDK 8u112 includes the fix for the referenced bug. Has anyone else been able to reproduce the issue with JDK 8u112? I am clearing the "Fix version" field as the issue doesn't seem to be in Kafka. > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436624#comment-15436624 ] Oleg Gorobets commented on KAFKA-3172: -- Looks like JDK bug: https://bugs.openjdk.java.net/browse/JDK-8153192 > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425986#comment-15425986 ] Oleg Gorobets commented on KAFKA-3172: -- Couple more observations: (1) this happens only when I am running consumer out of topic with single partition, if I increase number of partitions per topic the problem goes away. (2) the number of messages consumed after which consumer blocks is always close or equal to 1 messages, then it blocks, which gives me the idea that there might be some limit per partition, queue or buffer size? (3) I tried the old scala api and it also seem to have the same problem, so looks like not connected with api version. > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424715#comment-15424715 ] Oleg Gorobets commented on KAFKA-3172: -- I experience the same problem with new consumer API (kafka_2.10 0.9.0.1) when number of messages gets higher (around 2000/sec). The thread is just blocked on poll(), no error messages whatsoever. I didn't have this issue with old (scala) API. ConsumerRecords records = consumer.poll(200); "CoreKafkaConsumer" prio=10 tid=0x7fb3e87da000 nid=0x2666 runnable [0x7fb3dc56a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000707d943a0> (a sun.nio.ch.Util$2) - locked <0x000707d94390> (a java.util.Collections$UnmodifiableSet) - locked <0x000707d93878> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.kafka.common.network.Selector.select(Selector.java:425) at org.apache.kafka.common.network.Selector.poll(Selector.java:254) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853) > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(S
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203906#comment-15203906 ] Ismael Juma commented on KAFKA-3172: Please make sure you are using 0.9.0.1 as it includes a number of important fixes. cc [~hachikuji] > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203901#comment-15203901 ] Mikael Sundberg commented on KAFKA-3172: Im seeing the same thing. A machine can suddenly stop consuming one or several partitions, no errors shown in log. Threaddump gives foe ex: "pool-4-thread-1" #29 prio=5 os_prio=0 tid=0x7f8d35a5e000 nid=0x21 runnable [0x7f8d022cc000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xeeaebf50> (a sun.nio.ch.Util$2) - locked <0xeeaebf60> (a java.util.Collections$UnmodifiableSet) - locked <0xeeaebf08> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.kafka.common.network.Selector.select(Selector.java:425) at org.apache.kafka.common.network.Selector.poll(Selector.java:254) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:877) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829) at com.klarna.ordermanagement.messaging.kafka.KafkaMessageConsumer.consumeRecords(KafkaMessageConsumer.java:63) at com.klarna.ordermanagement.messaging.kafka.KafkaMessageConsumer$$Lambda$87/2015198349.run(Unknown Source) at com.klarna.ordermanagement.commons.LambdaUtils.repeat(LambdaUtils.java:81) at com.klarna.ordermanagement.messaging.kafka.KafkaMessageConsumer.run(KafkaMessageConsumer.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) >
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15147563#comment-15147563 ] Dany Benjamin commented on KAFKA-3172: -- Yes. This is the jstack dump I am getting. I started my multi-threaded consumer with a jmx port in it and I am seeing the same 'kafka-producer-network-thread | producer n' via Jrockit. I am running the ConsumerGroupExample modified to read my topic. > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126875#comment-15126875 ] Gwen Shapira commented on KAFKA-3172: - Your description says "consumer" but your jstack dump says "producer"... can you double check if you are dumping the right process? > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: jmx_info.png, jstack.png, lagSample.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method
[ https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123994#comment-15123994 ] Dany Benjamin commented on KAFKA-3172: -- Even after the consumers are stopped (400) and new consumers in same group are added (100) - there is still no movement on threads. Why does poll method block even when no messages despite lag are returned? > Consumer threads stay in 'Watiting' status and are blocked at consumer poll > method > -- > > Key: KAFKA-3172 > URL: https://issues.apache.org/jira/browse/KAFKA-3172 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.0 > Environment: linux >Reporter: Dany Benjamin >Assignee: Neha Narkhede > Fix For: 0.9.0.0 > > Attachments: jstack.png > > > When running multiple consumers on same group (400 - for a 400 partitioned > topic), the application for all threads blocks at consumer.poll() method. The > timeout parameter sent in is 1. > Stack dump: > "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 > waiting on condition [0x7f63df8f7000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605812710> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 > tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000] >java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x0006058283e8> (a sun.nio.ch.Util$2) > - locked <0x0006058283d8> (a > java.util.Collections$UnmodifiableSet) > - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at org.apache.kafka.common.network.Selector.select(Selector.java:425) > at org.apache.kafka.common.network.Selector.poll(Selector.java:254) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
