[ https://issues.apache.org/jira/browse/KAFKA-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Schiff updated KAFKA-4451: ---------------------------------- Description: Bringing up an empty broker. the partition for a compact topic is not split into multiple log files. All data is written into a single log file, causing offsets to overflow. A dump of the affected broker shortly after it started replicating: {code} michael.schiff@stats-kafka09:~$ sudo /opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /kafka/attainment_event-0/00000000000000000000.index | head -n 10 Dumping /kafka/attainment_event-0/00000000000000000000.index offset: 1022071124 position: 1037612 offset: -1713432120 position: 1348740 offset: -886291423 position: 2397130 offset: -644750126 position: 3445630 offset: -57889876 position: 4493972 offset: 433950099 position: 5388461 offset: 1071769472 position: 6436837 offset: 1746859069 position: 7485367 offset: 2090359736 position: 8533822 ... {code} and the dump of the same log file from the leader of this partition {code} michael.schiff@stats-kafka12:~$ sudo /opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /kafka/attainment_event-0/00000000000000000000.index [sudo] password for michael.schiff: Dumping /kafka/attainment_event-0/00000000000000000000.index offset: 353690666 position: 262054 offset: 633140428 position: 523785 offset: 756537951 position: 785815 {code} was: Bringing up an empty broker. the partition for compact topic is not split into multiple log files. All data is written into a single log file, causing offsets to overflow. A dump of the affected broker shortly after it started replicating: {code} michael.schiff@stats-kafka09:~$ sudo /opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /kafka/attainment_event-0/00000000000000000000.index | head -n 10 Dumping /kafka/attainment_event-0/00000000000000000000.index offset: 1022071124 position: 1037612 offset: -1713432120 position: 1348740 offset: -886291423 position: 2397130 offset: -644750126 position: 3445630 offset: -57889876 position: 4493972 offset: 433950099 position: 5388461 offset: 1071769472 position: 6436837 offset: 1746859069 position: 7485367 offset: 2090359736 position: 8533822 ... {code} and the dump of the same log file from the leader of this partition {code} michael.schiff@stats-kafka12:~$ sudo /opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /kafka/attainment_event-0/00000000000000000000.index [sudo] password for michael.schiff: Dumping /kafka/attainment_event-0/00000000000000000000.index offset: 353690666 position: 262054 offset: 633140428 position: 523785 offset: 756537951 position: 785815 {code} > Recovering empty replica yields negative offsets in index of compact > partitions > ------------------------------------------------------------------------------- > > Key: KAFKA-4451 > URL: https://issues.apache.org/jira/browse/KAFKA-4451 > Project: Kafka > Issue Type: Bug > Reporter: Michael Schiff > Fix For: 0.9.0.1 > > > Bringing up an empty broker. > the partition for a compact topic is not split into multiple log files. All > data is written into a single log file, causing offsets to overflow. > A dump of the affected broker shortly after it started replicating: > {code} > michael.schiff@stats-kafka09:~$ sudo /opt/kafka/bin/kafka-run-class.sh > kafka.tools.DumpLogSegments --files > /kafka/attainment_event-0/00000000000000000000.index | head -n 10 > Dumping /kafka/attainment_event-0/00000000000000000000.index > offset: 1022071124 position: 1037612 > offset: -1713432120 position: 1348740 > offset: -886291423 position: 2397130 > offset: -644750126 position: 3445630 > offset: -57889876 position: 4493972 > offset: 433950099 position: 5388461 > offset: 1071769472 position: 6436837 > offset: 1746859069 position: 7485367 > offset: 2090359736 position: 8533822 > ... > {code} > and the dump of the same log file from the leader of this partition > {code} > michael.schiff@stats-kafka12:~$ sudo /opt/kafka/bin/kafka-run-class.sh > kafka.tools.DumpLogSegments --files > /kafka/attainment_event-0/00000000000000000000.index > [sudo] password for michael.schiff: > Dumping /kafka/attainment_event-0/00000000000000000000.index > offset: 353690666 position: 262054 > offset: 633140428 position: 523785 > offset: 756537951 position: 785815 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)