[ https://issues.apache.org/jira/browse/KAFKA-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284675#comment-16284675 ]
Abhi edited comment on KAFKA-6337 at 12/11/17 10:01 AM: -------------------------------------------------------- Server.log when the error started coming // Server.log [2017-12-09 03:10:49,947] ERROR [KafkaApi-1] Error when handling request {controller_id=0,controller_epoch=1,partition_states=[{topic=LIVE,partition=31,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[0,4,5]},{topic=LIVE,partition=9,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[6,0,1]},{topic=__consumer_offsets,partition=27,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[0,5,6]},{topic=__consumer_offsets,partition=19,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[6,3,4]},{topic=LIVEOLD,partition=10,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[6,2,3]},{topic=LIVEOLD,partition=32,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=__consumer_offsets,partition=13,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[0,3,4]},{topic=LIVE,partition=17,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=__consumer_offsets,partition=5,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVEOLD,partition=18,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[0,4,5]},{topic=LIVEOLD,partition=45,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVE,partition=3,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=LIVE,partition=30,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[6,3,4]},{topic=LIVEOLD,partition=4,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=__consumer_offsets,partition=48,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=LIVE,partition=44,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=__consumer_offsets,partition=40,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[6,0,1]},{topic=LIVEOLD,partition=31,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=LIVE,partition=16,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVE,partition=38,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[0,5,6]},{topic=__consumer_offsets,partition=34,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=LIVEOLD,partition=17,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[6,3,4]},{topic=LIVEOLD,partition=39,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[0,1,2]},{topic=__consumer_offsets,partition=26,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[6,4,5]},{topic=LIVE,partition=24,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[0,3,4]},{topic=LIVE,partition=2,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=__consumer_offsets,partition=20,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[0,4,5]},{topic=__consumer_offsets,partition=12,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[6,2,3]},{topic=LIVEOLD,partition=3,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVEOLD,partition=25,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[0,5,6]},{topic=LIVE,partition=10,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[0,1,2]},{topic=__consumer_offsets,partition=6,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=LIVEOLD,partition=11,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[0,3,4]},{topic=__consumer_offsets,partition=47,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVEOLD,partition=38,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[6,0,1]},{topic=__consumer_offsets,partition=41,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[0,1,2]},{topic=LIVE,partition=23,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[6,2,3]},{topic=LIVE,partition=45,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=__consumer_offsets,partition=33,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=LIVEOLD,partition=24,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[6,4,5]},{topic=LIVEOLD,partition=46,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=LIVE,partition=37,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[6,4,5]}],live_brokers=[{id=2,end_points=[{port=9095,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9098,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9096,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9097,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9094,host=1.1.1.2,security_protocol_type=0}],rack=null}]} (kafka.server.KafkaApis) org.apache.kafka.common.errors.ControllerMovedException: Broker 1 received update metadata request with correlation id 13 from an old controller 0 with epoch 1. Latest known controller epoch is 2 state-change.log [2017-12-08 17:57:56,154] TRACE Controller 0 epoch 1 received response {error_code=0} for a request sent to broker 1.1.1.2:9093 (id: 0 rack: null) (state.change.logger) was (Author: abhit011): {{ {controller_id=0,controller_epoch=1,partition_states=[{topic=LIVETOPIC,partition=31,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[0,4,5]},{topic=LIVETOPIC,partition=9,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[6,0,1]},{topic=__consumer_offsets,partition=27,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[0,5,6]},{topic=__consumer_offsets,partition=19,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[6,3,4]},{topic=LIVETOPICOLD,partition=10,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[6,2,3]},{topic=LIVETOPICOLD,partition=32,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=__consumer_offsets,partition=13,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[0,3,4]},{topic=LIVETOPIC,partition=17,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=__consumer_offsets,partition=5,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVETOPICOLD,partition=18,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[0,4,5]},{topic=LIVETOPICOLD,partition=45,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVETOPIC,partition=3,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=LIVETOPIC,partition=30,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[6,3,4]},{topic=LIVETOPICOLD,partition=4,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=__consumer_offsets,partition=48,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=LIVETOPIC,partition=44,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=__consumer_offsets,partition=40,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[6,0,1]},{topic=LIVETOPICOLD,partition=31,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=LIVETOPIC,partition=16,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVETOPIC,partition=38,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[0,5,6]},{topic=__consumer_offsets,partition=34,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=LIVETOPICOLD,partition=17,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[6,3,4]},{topic=LIVETOPICOLD,partition=39,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[0,1,2]},{topic=__consumer_offsets,partition=26,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[6,4,5]},{topic=LIVETOPIC,partition=24,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[0,3,4]},{topic=LIVETOPIC,partition=2,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=__consumer_offsets,partition=20,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[0,4,5]},{topic=__consumer_offsets,partition=12,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[6,2,3]},{topic=LIVETOPICOLD,partition=3,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVETOPICOLD,partition=25,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[0,5,6]},{topic=LIVETOPIC,partition=10,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[0,1,2]},{topic=__consumer_offsets,partition=6,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=LIVETOPICOLD,partition=11,controller_epoch=1,leader=3,leader_epoch=1,isr=[3,4],zk_version=1,replicas=[0,3,4]},{topic=__consumer_offsets,partition=47,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[6,1,2]},{topic=LIVETOPICOLD,partition=38,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[6,0,1]},{topic=__consumer_offsets,partition=41,controller_epoch=1,leader=1,leader_epoch=1,isr=[1,2],zk_version=1,replicas=[0,1,2]},{topic=LIVETOPIC,partition=23,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[6,2,3]},{topic=LIVETOPIC,partition=45,controller_epoch=1,leader=1,leader_epoch=1,isr=[1],zk_version=1,replicas=[0,6,1]},{topic=__consumer_offsets,partition=33,controller_epoch=1,leader=5,leader_epoch=1,isr=[5],zk_version=1,replicas=[6,5,0]},{topic=LIVETOPICOLD,partition=24,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[6,4,5]},{topic=LIVETOPICOLD,partition=46,controller_epoch=1,leader=2,leader_epoch=1,isr=[2,3],zk_version=1,replicas=[0,2,3]},{topic=LIVETOPIC,partition=37,controller_epoch=1,leader=4,leader_epoch=1,isr=[4,5],zk_version=1,replicas=[6,4,5]}],live_brokers=[{id=2,end_points=[{port=9095,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9098,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9096,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9097,host=1.1.1.2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9094,host=1.1.1.2,security_protocol_type=0}],rack=null}]}}} > Error for partition [__consumer_offsets,15] to broker > ----------------------------------------------------- > > Key: KAFKA-6337 > URL: https://issues.apache.org/jira/browse/KAFKA-6337 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.10.2.0 > Environment: Windows running Kafka(0.10.2.0) > 3 ZK Instances running on 3 different Windows Servers, 7 Kafka Broker nodes > running on single windows machine with different disk for logs directory. > Reporter: Abhi > Priority: Blocker > Labels: windows > > Hello * > I am running Kafka(0.10.2.0) on windows from the past one year ... > But off late there has been unique Broker issues that I have observed 4-5 > times in > last 4 months. > Kafka setup cofig... > 3 ZK Instances running on 3 different Windows Servers, 7 Kafka Broker nodes > running on single windows machine with different disk for logs directory.... > My Kafka has 2 Topics with partition size 50 each , and replication factor of > 3. > My partition logic selection: Each message has a unique ID and logic of > selecting partition is ( unique ID % 50), and then calling Kafka producer API > to route a specific message to a particular topic partition . > My Each Broker Properties look like this > {{broker.id=0 > port:9093 > num.network.threads=3 > num.io.threads=8 > socket.send.buffer.bytes=102400 > socket.receive.buffer.bytes=102400 > socket.request.max.bytes=104857600 > offsets.retention.minutes=360 > advertised.host.name=1.1.1.2 > advertised.port:9093 > ctories under which to store log files > log.dirs=C:\\kafka_2.10-0.10.2.0-SNAPSHOT\\data\\kafka-logs > num.partitions=1 > num.recovery.threads.per.data.dir=1 > log.retention.minutes=360 > log.segment.bytes=52428800 > log.retention.check.interval.ms=300000 > log.cleaner.enable=true > log.cleanup.policy=delete > log.cleaner.min.cleanable.ratio=0.5 > log.cleaner.backoff.ms=15000 > log.segment.delete.delay.ms=6000 > auto.create.topics.enable=false > zookeeper.connect=1.1.1.2:2181,1.1.1.3:2182,1.1.1.4:2183 > zookeeper.connection.timeout.ms=6000 > }} > But of-late there has been a unique case that's cropping out in Kafka broker > nodes, > _[2017-12-02 02:47:40,024] ERROR [ReplicaFetcherThread-0-4], Error for > partition [__consumer_offsets,15] to broker > 4:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server > is not the leader for that topic-partition. > (kafka.server.ReplicaFetcherThread)_ > The entire server.log is filled with these logs, and its very huge too , > please help me in understanding under what circumstances these can occur, and > what measures I need to take.. > Please help me this is the third time in last three Saturdays i faced the > similar issue. > Courtesy > Abhi > !wq -- This message was sent by Atlassian JIRA (v6.4.14#64029)