Please find the log attached.
From: Seshadri, Balaji [mailto:[email protected]]
Sent: Wednesday, September 24, 2014 11:48 AM
To: '[email protected]'
Subject: BadVersion state in Kafka Logs
Hi,
We got the below error in our logs and our consumers stopped consuming any data
?.It worked only after restart.
We would like to confirm that it's because we are running with 0.8-beta version
and not 0.8 release version to convince "THE MGMT" guys.
Please let me know if it's this KAFKA-1382 causing the issue.
Thanks,
Balaji
From: Gulia, Vikram
Sent: Wednesday, September 24, 2014 8:43 AM
To: Sharma, Navdeep; #IT-MAD DES; #IT-MAA
Cc: Alam, Mohammad Shah
Subject: RE: 9/23 prod issue - offline kafka partitions.
Adding full MAA distro.
DES Offshore looked in to the logs on kafka servers and seems like the issue we
encountered yesterday may be described in these threads, please have a look -
http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the fix/patch
which is available in 0.8.1.2/0.8.2)
Thank You,
Vikram Gulia
From: Sharma, Navdeep
Sent: Wednesday, September 24, 2014 6:53 AM
To: Gulia, Vikram; #IT-MAD DES
Cc: #IT-MAA Offshore; Alam, Mohammad Shah
Subject: RE: 9/23 prod issue - offline kafka partitions.
Hi Vikram,
We analyzed below mentioned issue with MAA-Offshore (Abhishek) and found that
the error occurred only on 23 Sept. This is not historical as we checked last
4 days logs.
It looks like that consumer got stopped on September 22 2014 for Linux patching
activty.MAA started consumer September 23 2014 at 1:00 AM.
Issue in server log " BadVersion for
/brokers/topics/rain-burn-in/partitions/121/state" but it is not present in
previous 4 days logs.
More detail of this error can be found at-
http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904
We are not sure about data loss in this scenario and working on this.
[cid:[email protected]]
[cid:[email protected]]
Let us know if any concerns.
[cid:[email protected]]
Navdeep Sharma
Developer - offshore, Middleware Applications & Development
o India: 0120-4532000 - 2234
c: +91-9911698102
From: Gulia, Vikram
Sent: Tuesday, September 23, 2014 6:17 PM
To: #IT-MAD DES
Subject: FW: 9/23 prod issue - offline kafka partitions.
DES Offshore dev,
Please work with MAA offshore to monitor the kafka broker as we had this
incident where lot of partitions went offline around 1.45 PM MST and MAA has to
restart the kafka servers. We may have lost messages and we need to see if
there is a way to figure out what was the impact.
Also, check the logs for kafka servers and see if we can figure out why did
partitions go offline or are un-available? Let us know if you find anything
relevant.
Thank You,
Vikram Gulia
From: Nielsen, Andy
Sent: Tuesday, September 23, 2014 5:04 PM
To: #IT-MAD DES; Gulia, Vikram
Cc: #IT-MAA
Subject: 9/23 prod issue - offline kafka partitions.
desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions
topic: account-access partition: 21 leader: -1 replicas: 4,6,1 isr: 1
topic: account-access partition: 51 leader: -1 replicas: 4,6,1 isr:
topic: account-access partition: 81 leader: -1 replicas: 4,6,1 isr: 1
topic: account-access partition: 111 leader: -1 replicas: 4,6,1 isr: 1
topic: account-activated partition: 13 leader: -1 replicas: 4,6,1
isr:
topic: account-activated partition: 43 leader: -1 replicas: 4,6,1
isr:
topic: account-activated partition: 73 leader: -1 replicas: 4,6,1
isr:
topic: account-activated partition: 103 leader: -1 replicas: 4,6,1
isr: 1
topic: account-adjustment-issued partition: 27 leader: -1
replicas: 4,6,1 isr:
topic: account-adjustment-issued partition: 57 leader: -1
replicas: 4,6,1 isr:
topic: account-adjustment-issued partition: 87 leader: -1
replicas: 4,6,1 isr: 1
topic: account-adjustment-issued partition: 117 leader: -1
replicas: 4,6,1 isr:
topic: account-created partition: 11 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 41 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 71 leader: -1 replicas: 4,6,1 isr:
topic: account-created partition: 101 leader: -1 replicas: 4,6,1 isr:
topic: account-info-updated partition: 7 leader: -1 replicas: 4,6,1
isr: 1
topic: account-info-updated partition: 37 leader: -1 replicas: 4,6,1
isr: 1
topic: account-info-updated partition: 67 leader: -1 replicas: 4,6,1
isr:
topic: account-info-updated partition: 97 leader: -1 replicas: 4,6,1
isr: 1
topic: account-info-updated partition: 127 leader: -1 replicas: 4,6,1
isr: 1
topic: application-access partition: 21 leader: -1 replicas: 4,6,1
isr: 1
topic: application-access partition: 51 leader: -1 replicas: 4,6,1
isr: 1
topic: application-access partition: 81 leader: -1 replicas: 4,6,1
isr: 1
topic: application-access partition: 111 leader: -1 replicas: 4,6,1
isr: 1
topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1 isr:
topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1 isr: 1
topic: collected-event partition: 29 leader: -1 replicas: 4,6,1 isr: 1
topic: collected-event partition: 59 leader: -1 replicas: 4,6,1 isr:
topic: collected-event partition: 89 leader: -1 replicas: 4,6,1 isr:
topic: collected-event partition: 119 leader: -1 replicas: 4,6,1 isr: 1
topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1 isr:
topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1 isr: 1
topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 23 leader: -1
replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 53 leader: -1
replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 83 leader: -1
replicas: 4,6,1 isr:
topic: dish-promo-application-access partition: 113 leader: -1
replicas: 4,6,1 isr:
topic: event-response partition: 2 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 32 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 62 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 92 leader: -1 replicas: 4,6,1 isr:
topic: event-response partition: 122 leader: -1 replicas: 4,6,1 isr: 1
topic: leads-service partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 84 leader: -1 replicas: 4,6,1 isr:
topic: leads-service partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1 isr: 1
topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1 isr:
topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1 isr: 1
topic: online-account-registration-attempted partition: 21 leader: -1
replicas: 4,6,1 isr:
topic: online-account-registration-attempted partition: 51 leader: -1
replicas: 4,6,1 isr: 1
topic: online-account-registration-attempted partition: 81 leader: -1
replicas: 4,6,1 isr:
topic: online-account-registration-attempted partition: 111 leader: -1
replicas: 4,6,1 isr:
topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1 isr:
topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1 isr: 1
topic: order-completed partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: order-completed partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: order-completed partition: 84 leader: -1 replicas: 4,6,1 isr: 1
topic: order-completed partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 25 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 55 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 85 leader: -1 replicas: 4,6,1 isr:
topic: order-created partition: 115 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 8 leader: -1 replicas: 4,6,1 isr: 1
topic: order-modified partition: 38 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 68 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 98 leader: -1 replicas: 4,6,1 isr:
topic: order-modified partition: 128 leader: -1 replicas: 4,6,1 isr: 1
topic: order-request partition: 24 leader: -1 replicas: 4,6,1 isr:
topic: order-request partition: 54 leader: -1 replicas: 4,6,1 isr:
topic: order-request partition: 84 leader: -1 replicas: 4,6,1 isr: 1
topic: order-request partition: 114 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 27 leader: -1 replicas: 4,6,1 isr: 1
topic: order-response partition: 57 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 87 leader: -1 replicas: 4,6,1 isr:
topic: order-response partition: 117 leader: -1 replicas: 4,6,1 isr:
topic: outbound-call-attempted partition: 13 leader: -1 replicas: 4,6,1
isr:
topic: outbound-call-attempted partition: 43 leader: -1 replicas: 4,6,1
isr: 1
topic: outbound-call-attempted partition: 73 leader: -1 replicas: 4,6,1
isr: 1
topic: outbound-call-attempted partition: 103 leader: -1 replicas: 4,6,1
isr:
topic: outbound-communications partition: 4 leader: -1 replicas: 4,6,1
isr:
topic: outbound-communications partition: 34 leader: -1 replicas: 4,6,1
isr:
topic: outbound-communications partition: 64 leader: -1 replicas: 4,6,1
isr:
topic: outbound-communications partition: 94 leader: -1 replicas: 4,6,1
isr: 1
topic: outbound-communications partition: 124 leader: -1 replicas: 4,6,1
isr: 1
topic: postal-mail-undeliverable partition: 15 leader: -1
replicas: 4,6,1 isr: 1
topic: postal-mail-undeliverable partition: 45 leader: -1
replicas: 4,6,1 isr:
topic: postal-mail-undeliverable partition: 75 leader: -1
replicas: 4,6,1 isr:
topic: postal-mail-undeliverable partition: 105 leader: -1
replicas: 4,6,1 isr:
topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1 isr:
topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1 isr:
topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1 isr:
topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1 isr:
topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1 isr:
topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1 isr:
topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1 isr:
topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1 isr: 1
topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1 isr: 1
topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1 isr:
topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1 isr:
topic: threshold-exceeded partition: 14 leader: -1 replicas: 4,6,1
isr:
topic: threshold-exceeded partition: 44 leader: -1 replicas: 4,6,1
isr:
topic: threshold-exceeded partition: 74 leader: -1 replicas: 4,6,1
isr:
topic: threshold-exceeded partition: 104 leader: -1 replicas: 4,6,1
isr: 1
Andy Nielsen
Middleware Application Admin
303-723-2347
cell:720-971-2856
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/dish-promo-application-access/partitions/128/state with data {
"controller_epoch":40, "isr":[ 6, 1 ], "leader":1, "leader_epoch":99,
"version":1 } and expected version 150 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/dish-promo-application-access/partitions/128/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/order-completed/partitions/57/state with data {
"controller_epoch":40, "isr":[ 4, 1 ], "leader":1, "leader_epoch":264,
"version":1 } and expected version 402 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/order-completed/partitions/57/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/order-completed/partitions/57/state with data {
"controller_epoch":40, "isr":[ 4, 1 ], "leader":1, "leader_epoch":264,
"version":1 } and expected version 402 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/order-completed/partitions/57/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/online-account-registration-attempted/partitions/126/state with
data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, "leader_epoch":99,
"version":1 } and expected version 150 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/online-account-registration-attempted/partitions/126/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/online-account-registration-attempted/partitions/126/state with
data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, "leader_epoch":99,
"version":1 } and expected version 150 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/online-account-registration-attempted/partitions/126/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/application-access/partitions/12/state with data
{ "controller_epoch":40, "isr":[ 1 ], "leader":1, "leader_epoch":98,
"version":1 } and expected version 148 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/application-access/partitions/12/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/application-access/partitions/12/state with data
{ "controller_epoch":40, "isr":[ 1 ], "leader":1, "leader_epoch":98,
"version":1 } and expected version 148 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/application-access/partitions/12/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/account-adjustment-issued/partitions/96/state
with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
"leader_epoch":566, "version":1 } and expected version 888 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/account-adjustment-issued/partitions/96/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/account-adjustment-issued/partitions/96/state
with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
"leader_epoch":566, "version":1 } and expected version 888 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/account-adjustment-issued/partitions/96/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/rain-enhanced/partitions/65/state with data {
"controller_epoch":40, "isr":[ 4, 1 ], "leader":1, "leader_epoch":615,
"version":1 } and expected version 932 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/rain-enhanced/partitions/65/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/rain-enhanced/partitions/65/state with data {
"controller_epoch":40, "isr":[ 4, 1 ], "leader":1, "leader_epoch":615,
"version":1 } and expected version 932 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/rain-enhanced/partitions/65/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/online-account-registration-attempted/partitions/72/state with
data { "controller_epoch":40, "isr":[ 1 ], "leader":1, "leader_epoch":98,
"version":1 } and expected version 148 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/online-account-registration-attempted/partitions/72/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/online-account-registration-attempted/partitions/72/state with
data { "controller_epoch":40, "isr":[ 1 ], "leader":1, "leader_epoch":98,
"version":1 } and expected version 148 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/online-account-registration-attempted/partitions/72/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/account-adjustment-issued/partitions/24/state
with data { "controller_epoch":40, "isr":[ 4, 1 ], "leader":1,
"leader_epoch":569, "version":1 } and expected version 905 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/account-adjustment-issued/partitions/24/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/account-adjustment-issued/partitions/24/state
with data { "controller_epoch":40, "isr":[ 4, 1 ], "leader":1,
"leader_epoch":569, "version":1 } and expected version 905 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/account-adjustment-issued/partitions/24/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/dish-promo-application-access/partitions/2/state
with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
"leader_epoch":98, "version":1 } and expected version 148 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/dish-promo-application-access/partitions/2/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/dish-promo-application-access/partitions/2/state
with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1,
"leader_epoch":98, "version":1 } and expected version 148 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/dish-promo-application-access/partitions/2/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/online-account-registration-attempted/partitions/36/state with
data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, "leader_epoch":99,
"version":1 } and expected version 150 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/online-account-registration-attempted/partitions/36/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path
/brokers/topics/online-account-registration-attempted/partitions/36/state with
data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, "leader_epoch":99,
"version":1 } and expected version 150 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/brokers/topics/online-account-registration-attempted/partitions/36/state
server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - Conditional
update of path /brokers/topics/outbound-communications/partitions/91/state with
data { "controller_epoch":40, "isr":[ 4, 1 ], "leader":1, "leader_epoch":598,
"version":1 } and expected version 939 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/outbound-communications/par
