[jira] [Updated] (KAFKA-1554) Corrupt index found on clean startup

2016-10-07 Thread Jason Gustafson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1554:
---
Fix Version/s: (was: 0.10.0.0)
   (was: 0.10.1.0)

> Corrupt index found on clean startup
> 
>
> Key: KAFKA-1554
> URL: https://issues.apache.org/jira/browse/KAFKA-1554
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1
> Environment: ubuntu 12.04, oracle jdk 1.7
>Reporter: Alexis Midon
>Assignee: Mayuresh Gharat
>Priority: Critical
> Attachments: KAFKA-1554.patch
>
>
> On a clean start up, corrupted index files are found.
> After investigations, it appears that some pre-allocated index files are not 
> "compacted" correctly and the end of the file is full of zeroes.
> As a result, on start up, the last relative offset is zero which yields an 
> offset equal to the base offset.
> The workaround is to delete all index files of size 10MB (the size of the 
> pre-allocated files), and restart. Index files will be re-created.
> {code}
> find $your_data_directory -size 10485760c -name *.index #-delete
> {code}
> This is issue might be related/similar to 
> https://issues.apache.org/jira/browse/KAFKA-1112
> {code}
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], 
> Connecting to zookeeper on 
> zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 
> INFO 
> ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
>  org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:host.name=i-6b948138.inst.aws.airbnb.com
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.version=1.7.0_55
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.vendor=Oracle Corporation
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.io.tmpdir=/tmp
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.compiler=
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.name=Linux
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.arch=amd64
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.version=3.2.0-61-virtual
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:user.name=kafka
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:user.home=/srv/kafka
>

[jira] [Updated] (KAFKA-1554) Corrupt index found on clean startup

2016-04-29 Thread Gwen Shapira (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-1554:

Fix Version/s: 0.10.0.0

> Corrupt index found on clean startup
> 
>
> Key: KAFKA-1554
> URL: https://issues.apache.org/jira/browse/KAFKA-1554
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1
> Environment: ubuntu 12.04, oracle jdk 1.7
>Reporter: Alexis Midon
>Assignee: Mayuresh Gharat
>Priority: Critical
> Fix For: 0.10.1.0, 0.10.0.0
>
> Attachments: KAFKA-1554.patch
>
>
> On a clean start up, corrupted index files are found.
> After investigations, it appears that some pre-allocated index files are not 
> "compacted" correctly and the end of the file is full of zeroes.
> As a result, on start up, the last relative offset is zero which yields an 
> offset equal to the base offset.
> The workaround is to delete all index files of size 10MB (the size of the 
> pre-allocated files), and restart. Index files will be re-created.
> {code}
> find $your_data_directory -size 10485760c -name *.index #-delete
> {code}
> This is issue might be related/similar to 
> https://issues.apache.org/jira/browse/KAFKA-1112
> {code}
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], 
> Connecting to zookeeper on 
> zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 
> INFO 
> ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
>  org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:host.name=i-6b948138.inst.aws.airbnb.com
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.version=1.7.0_55
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.vendor=Oracle Corporation
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.io.tmpdir=/tmp
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.compiler=
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.name=Linux
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.arch=amd64
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.version=3.2.0-61-virtual
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:user.name=kafka
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:user.home=/srv/kafka
> 2014-07-11T00:53:17+00:00

[jira] [Updated] (KAFKA-1554) Corrupt index found on clean startup

2015-03-23 Thread Mayuresh Gharat (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-1554:
---
Status: Patch Available  (was: Open)

 Corrupt index found on clean startup
 

 Key: KAFKA-1554
 URL: https://issues.apache.org/jira/browse/KAFKA-1554
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
 Environment: ubuntu 12.04, oracle jdk 1.7
Reporter: Alexis Midon
Assignee: Mayuresh Gharat
Priority: Critical
 Fix For: 0.9.0

 Attachments: KAFKA-1554.patch


 On a clean start up, corrupted index files are found.
 After investigations, it appears that some pre-allocated index files are not 
 compacted correctly and the end of the file is full of zeroes.
 As a result, on start up, the last relative offset is zero which yields an 
 offset equal to the base offset.
 The workaround is to delete all index files of size 10MB (the size of the 
 pre-allocated files), and restart. Index files will be re-created.
 {code}
 find $your_data_directory -size 10485760c -name *.index #-delete
 {code}
 This is issue might be related/similar to 
 https://issues.apache.org/jira/browse/KAFKA-1112
 {code}
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 
 INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 
 INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], 
 Connecting to zookeeper on 
 zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 
 INFO 
 ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
  org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:host.name=i-6b948138.inst.aws.airbnb.com
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.version=1.7.0_55
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.vendor=Oracle Corporation
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.io.tmpdir=/tmp
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.compiler=NA
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.name=Linux
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.arch=amd64
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.version=3.2.0-61-virtual
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.name=kafka
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.home=/srv/kafka
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main

[jira] [Updated] (KAFKA-1554) Corrupt index found on clean startup

2015-03-23 Thread Mayuresh Gharat (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-1554:
---
Attachment: KAFKA-1554.patch

 Corrupt index found on clean startup
 

 Key: KAFKA-1554
 URL: https://issues.apache.org/jira/browse/KAFKA-1554
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
 Environment: ubuntu 12.04, oracle jdk 1.7
Reporter: Alexis Midon
Assignee: Mayuresh Gharat
Priority: Critical
 Fix For: 0.9.0

 Attachments: KAFKA-1554.patch


 On a clean start up, corrupted index files are found.
 After investigations, it appears that some pre-allocated index files are not 
 compacted correctly and the end of the file is full of zeroes.
 As a result, on start up, the last relative offset is zero which yields an 
 offset equal to the base offset.
 The workaround is to delete all index files of size 10MB (the size of the 
 pre-allocated files), and restart. Index files will be re-created.
 {code}
 find $your_data_directory -size 10485760c -name *.index #-delete
 {code}
 This is issue might be related/similar to 
 https://issues.apache.org/jira/browse/KAFKA-1112
 {code}
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 
 INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 
 INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], 
 Connecting to zookeeper on 
 zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 
 INFO 
 ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
  org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:host.name=i-6b948138.inst.aws.airbnb.com
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.version=1.7.0_55
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.vendor=Oracle Corporation
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.io.tmpdir=/tmp
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.compiler=NA
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.name=Linux
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.arch=amd64
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.version=3.2.0-61-virtual
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.name=kafka
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.home=/srv/kafka
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main

[jira] [Updated] (KAFKA-1554) Corrupt index found on clean startup

2014-09-14 Thread Neha Narkhede (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1554:
-
Fix Version/s: 0.9.0

 Corrupt index found on clean startup
 

 Key: KAFKA-1554
 URL: https://issues.apache.org/jira/browse/KAFKA-1554
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
 Environment: ubuntu 12.04, oracle jdk 1.7
Reporter: Alexis Midon
Priority: Critical
 Fix For: 0.9.0


 On a clean start up, corrupted index files are found.
 After investigations, it appears that some pre-allocated index files are not 
 compacted correctly and the end of the file is full of zeroes.
 As a result, on start up, the last relative offset is zero which yields an 
 offset equal to the base offset.
 The workaround is to delete all index files of size 10MB (the size of the 
 pre-allocated files), and restart. Index files will be re-created.
 {code}
 find $your_data_directory -size 10485760c -name *.index #-delete
 {code}
 This is issue might be related/similar to 
 https://issues.apache.org/jira/browse/KAFKA-1112
 {code}
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 
 INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 
 INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], 
 Connecting to zookeeper on 
 zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 
 INFO 
 ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
  org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:host.name=i-6b948138.inst.aws.airbnb.com
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.version=1.7.0_55
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.vendor=Oracle Corporation
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.io.tmpdir=/tmp
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:java.compiler=NA
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.name=Linux
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.arch=amd64
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:os.version=3.2.0-61-virtual
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.name=kafka
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.home=/srv/kafka
 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
 INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
 environment:user.dir=/srv/kafka/kafka_2.10-0.8.1