[
https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Rosenberg updated KAFKA-2127:
-----------------------------------
Description:
I am using 0.8.2.1. I've been noticing that any time I use TopicCommand to
alter a topic (e.g. add partitions) or delete a topic, the broker logs show a
bunch of closed connections, and usually 2 or 3 Connection reset exceptions.
It logs these with ERROR status.
I recently used the kafka.admin.TopicCommand tool to increase the partitions
for a topic from 1 to 4. So I ran:
{code}
java -cp kafka.jar kafka.admin.TopicCommand --zookeeper myzkserver:12345
--topic mytopic --alter --partitions 4
{code}
This resulted in the following sequence in the broker log (repeated pretty much
in the logs of each broker):
{code}
2015-04-16 03:51:26,156 INFO [kafka-network-thread-27330-1] network.Processor
- Closing socket connection to /1.2.3.12.
2015-04-16 03:51:26,169 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.3.89.
2015-04-16 03:51:26,169 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.3.95.
2015-04-16 03:51:26,176 ERROR [kafka-network-thread-27330-2] network.Processor
- Closing socket for /1.2.4.34 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
2015-04-16 03:51:26,178 ERROR [kafka-network-thread-27330-1] network.Processor
- Closing socket for /1.2.4.59 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
2015-04-16 03:51:26,192 ERROR [kafka-network-thread-27330-1] network.Processor
- Closing socket for /1.2.3.11 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
2015-04-16 03:51:26,451 INFO [kafka-request-handler-3]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 45] Removed
fetcher for partitions [mytopic,2]
2015-04-16 03:51:26,453 INFO [kafka-request-handler-3] log.Log - Completed
load of log mytopic-2 with log end offset 0
2015-04-16 03:51:26,454 INFO [kafka-request-handler-3] log.LogManager -
Created log for partition [mytopic,2] in
/data_sdd/app/samsa-kafkaserver-ng/kafka_logs_ng with properties
{segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, segment.bytes
-> 1073741824, flush.ms -> 9223372036854775807, delete.retention.ms ->
86400000, index.interval.bytes -> 4096, retention.bytes -> 50000000000,
min.insync.replicas -> 1, cleanup.policy -> delete,
unclean.leader.election.enable -> true, segment.ms -> 604800000,
max.message.bytes -> 1000012, flush.messages -> 9223372036854775807,
min.cleanable.dirty.ratio -> 0.5, retention.ms -> 86400000, segment.jitter.ms
-> 0}.
2015-04-16 03:51:26,454 WARN [kafka-request-handler-3] cluster.Partition -
Partition [mytopic,2] on broker 45: No checkpointed highwatermark is found for
partition [mytopic,2]
2015-04-16 03:51:26,558 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.4.34.
2015-04-16 03:51:26,658 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.3.95.
2015-04-16 03:51:26,920 INFO [kafka-network-thread-27330-2] network.Processor
- Closing socket connection to /1.2.4.246.
2015-04-16 03:51:26,921 INFO [kafka-network-thread-27330-2] network.Processor
- Closing socket connection to /1.2.5.4.
{code}
was:
I am using 0.8.2.1. I've been noticing that any time I use TopicCommand to
alter a topic (e.g. add partitions) or delete a topic, the broker logs show a
bunch of closed connections, and usually 2 or 3 Connection reset exceptions.
It logs these with ERROR status.
I recently used the kafka.admin.TopicCommand tool to increase the partitions
for a topic from 1 to 4. So I ran:
{code}
java -cp kafka.jar kafka.admin.TopicCommand --zookeeper myzkserver:12345
--topic mytopic --alter --partitions 4
{code}
{code}
2015-04-16 03:51:26,156 INFO [kafka-network-thread-27330-1] network.Processor
- Closing socket connection to /1.2.3.12.
2015-04-16 03:51:26,169 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.3.89.
2015-04-16 03:51:26,169 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.3.95.
2015-04-16 03:51:26,176 ERROR [kafka-network-thread-27330-2] network.Processor
- Closing socket for /1.2.4.34 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
2015-04-16 03:51:26,178 ERROR [kafka-network-thread-27330-1] network.Processor
- Closing socket for /1.2.4.59 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
2015-04-16 03:51:26,192 ERROR [kafka-network-thread-27330-1] network.Processor
- Closing socket for /1.2.3.11 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
2015-04-16 03:51:26,451 INFO [kafka-request-handler-3]
server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 45] Removed
fetcher for partitions [mytopic,2]
2015-04-16 03:51:26,453 INFO [kafka-request-handler-3] log.Log - Completed
load of log mytopic-2 with log end offset 0
2015-04-16 03:51:26,454 INFO [kafka-request-handler-3] log.LogManager -
Created log for partition [mytopic,2] in
/data_sdd/app/samsa-kafkaserver-ng/kafka_logs_ng with properties
{segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, segment.bytes
-> 1073741824, flush.ms -> 9223372036854775807, delete.retention.ms ->
86400000, index.interval.bytes -> 4096, retention.bytes -> 50000000000,
min.insync.replicas -> 1, cleanup.policy -> delete,
unclean.leader.election.enable -> true, segment.ms -> 604800000,
max.message.bytes -> 1000012, flush.messages -> 9223372036854775807,
min.cleanable.dirty.ratio -> 0.5, retention.ms -> 86400000, segment.jitter.ms
-> 0}.
2015-04-16 03:51:26,454 WARN [kafka-request-handler-3] cluster.Partition -
Partition [mytopic,2] on broker 45: No checkpointed highwatermark is found for
partition [mytopic,2]
2015-04-16 03:51:26,558 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.4.34.
2015-04-16 03:51:26,658 INFO [kafka-network-thread-27330-0] network.Processor
- Closing socket connection to /1.2.3.95.
2015-04-16 03:51:26,920 INFO [kafka-network-thread-27330-2] network.Processor
- Closing socket connection to /1.2.4.246.
2015-04-16 03:51:26,921 INFO [kafka-network-thread-27330-2] network.Processor
- Closing socket connection to /1.2.5.4.
{code}
> Running TopicCommand --alter causes connection close/reset errors in kafka
> logs
> -------------------------------------------------------------------------------
>
> Key: KAFKA-2127
> URL: https://issues.apache.org/jira/browse/KAFKA-2127
> Project: Kafka
> Issue Type: Bug
> Reporter: Jason Rosenberg
> Priority: Minor
>
> I am using 0.8.2.1. I've been noticing that any time I use TopicCommand to
> alter a topic (e.g. add partitions) or delete a topic, the broker logs show a
> bunch of closed connections, and usually 2 or 3 Connection reset exceptions.
> It logs these with ERROR status.
> I recently used the kafka.admin.TopicCommand tool to increase the partitions
> for a topic from 1 to 4. So I ran:
> {code}
> java -cp kafka.jar kafka.admin.TopicCommand --zookeeper myzkserver:12345
> --topic mytopic --alter --partitions 4
> {code}
> This resulted in the following sequence in the broker log (repeated pretty
> much in the logs of each broker):
> {code}
> 2015-04-16 03:51:26,156 INFO [kafka-network-thread-27330-1]
> network.Processor - Closing socket connection to /1.2.3.12.
> 2015-04-16 03:51:26,169 INFO [kafka-network-thread-27330-0]
> network.Processor - Closing socket connection to /1.2.3.89.
> 2015-04-16 03:51:26,169 INFO [kafka-network-thread-27330-0]
> network.Processor - Closing socket connection to /1.2.3.95.
> 2015-04-16 03:51:26,176 ERROR [kafka-network-thread-27330-2]
> network.Processor - Closing socket for /1.2.4.34 because of error
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at kafka.utils.Utils$.read(Utils.scala:380)
> at
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Processor.read(SocketServer.scala:444)
> at kafka.network.Processor.run(SocketServer.scala:340)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-16 03:51:26,178 ERROR [kafka-network-thread-27330-1]
> network.Processor - Closing socket for /1.2.4.59 because of error
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at kafka.utils.Utils$.read(Utils.scala:380)
> at
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Processor.read(SocketServer.scala:444)
> at kafka.network.Processor.run(SocketServer.scala:340)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-16 03:51:26,192 ERROR [kafka-network-thread-27330-1]
> network.Processor - Closing socket for /1.2.3.11 because of error
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at kafka.utils.Utils$.read(Utils.scala:380)
> at
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Processor.read(SocketServer.scala:444)
> at kafka.network.Processor.run(SocketServer.scala:340)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-16 03:51:26,451 INFO [kafka-request-handler-3]
> server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 45] Removed
> fetcher for partitions [mytopic,2]
> 2015-04-16 03:51:26,453 INFO [kafka-request-handler-3] log.Log - Completed
> load of log mytopic-2 with log end offset 0
> 2015-04-16 03:51:26,454 INFO [kafka-request-handler-3] log.LogManager -
> Created log for partition [mytopic,2] in
> /data_sdd/app/samsa-kafkaserver-ng/kafka_logs_ng with properties
> {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000,
> segment.bytes -> 1073741824, flush.ms -> 9223372036854775807,
> delete.retention.ms -> 86400000, index.interval.bytes -> 4096,
> retention.bytes -> 50000000000, min.insync.replicas -> 1, cleanup.policy ->
> delete, unclean.leader.election.enable -> true, segment.ms -> 604800000,
> max.message.bytes -> 1000012, flush.messages -> 9223372036854775807,
> min.cleanable.dirty.ratio -> 0.5, retention.ms -> 86400000, segment.jitter.ms
> -> 0}.
> 2015-04-16 03:51:26,454 WARN [kafka-request-handler-3] cluster.Partition -
> Partition [mytopic,2] on broker 45: No checkpointed highwatermark is found
> for partition [mytopic,2]
> 2015-04-16 03:51:26,558 INFO [kafka-network-thread-27330-0]
> network.Processor - Closing socket connection to /1.2.4.34.
> 2015-04-16 03:51:26,658 INFO [kafka-network-thread-27330-0]
> network.Processor - Closing socket connection to /1.2.3.95.
> 2015-04-16 03:51:26,920 INFO [kafka-network-thread-27330-2]
> network.Processor - Closing socket connection to /1.2.4.246.
> 2015-04-16 03:51:26,921 INFO [kafka-network-thread-27330-2]
> network.Processor - Closing socket connection to /1.2.5.4.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
