Hi all, I have a kafka 0.9.0.0 cluster with 11 nodes. First,I found server logs as below, server.log.2016-10-17-22:[2016-10-17 22:22:13,885] WARN [ReplicaFetcherThread-0-4], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@367c9f98. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1786735, only 2389 bytes available (kafka.server.ReplicaFetcherThread) server.log.2016-10-17-22:[2016-10-17 22:22:15,456] WARN [ReplicaFetcherThread-0-5], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@12088f91. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1338722, only 5662 bytes available (kafka.server.ReplicaFetcherThread) server.log.2016-10-17-22:[2016-10-17 22:22:15,888] WARN [ReplicaFetcherThread-0-4], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@60069db2. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1786735, only 2389 bytes available (kafka.server.ReplicaFetcherThread) server.log.2016-10-17-22:[2016-10-17 22:22:17,460] WARN [ReplicaFetcherThread-0-5], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4a5991cb. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1338722, only 5662 bytes available (kafka.server.ReplicaFetcherThread)
Then I jstack pid,and I see "ReplicaFetcherThread-0-3" prio=10 tid=0x00007f1254319800 nid=0xfdb runnable [0x00007f0ee36d7000] "ReplicaFetcherThread-0-8" prio=10 tid=0x00007f1278141800 nid=0x66f runnable [0x00007f0ee2ecf000] "ReplicaFetcherThread-0-9" prio=10 tid=0x00007f1278127000 nid=0x66e runnable [0x00007f0ee2fd0000] "ReplicaFetcherThread-0-4" prio=10 tid=0x00007f127810c800 nid=0x66d waiting on condition [0x00007f0ee30d1000] "ReplicaFetcherThread-0-1" prio=10 tid=0x00007f12780ef800 nid=0x66c runnable [0x00007f0ee31d2000] "ReplicaFetcherThread-0-7" prio=10 tid=0x00007f12780d4800 nid=0x66b runnable [0x00007f0ee32d3000] "ReplicaFetcherThread-0-5" prio=10 tid=0x00007f12780b9800 nid=0x66a waiting on condition [0x00007f0ee33d4000] "ReplicaFetcherThread-0-6" prio=10 tid=0x00007f127809f000 nid=0x669 runnable [0x00007f0ee34d5000] "ReplicaFetcherThread-0-2" prio=10 tid=0x00007f1278084800 nid=0x668 runnable [0x00007f0ee35d6000] "ReplicaFetcherThread-0-10" prio=10 tid=0x00007f127804c800 nid=0x666 runnable [0x00007f0ee37d8000] the log shows that there are 2 replicaFetcherThreads waiting on condition. my cluster have no broker version compatible problem. From log, I thought there are some exceptions in broker 4 and broker 5, so I restart then, and everything goes right. what does this log means and how can it occurs? Will appreciate if anyone has any insight on what's happening here. Thanks.