[
https://issues.apache.org/jira/browse/KAFKA-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161185#comment-14161185
]
Jay Kreps commented on KAFKA-1670:
----------------------------------
Actually the reason for limiting the segments to 4GB (or possibly 2GB since we
are using java ints) is to keep the index pointers into the file limited to 4
bytes which keeps the index files small (4 byte relative offset and 4 byte file
position). Index file size and density is important since we hope to keep those
cached to make lookups cheap. We should fix the variable to avoid overflow and
even extend to unsigned ints but we probably can't allow arbitrarily large
segment files very easily.
The reasoning at the time was basically that there is no particular reason to
want very large segment files, and since we always do recovery from the
beginning of the file having 10GB segments would cause other problems when you
crashed and had to do recovery on hundreds of 10GB files.
> Corrupt log files for segment.bytes values close to Int.MaxInt
> --------------------------------------------------------------
>
> Key: KAFKA-1670
> URL: https://issues.apache.org/jira/browse/KAFKA-1670
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.1.1
> Reporter: Ryan Berdeen
> Assignee: Sriharsha Chintalapani
> Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1670.patch, KAFKA-1670_2014-10-04_20:17:46.patch,
> KAFKA-1670_2014-10-06_09:48:25.patch
>
>
> The maximum value for the topic-level config {{segment.bytes}} is
> {{Int.MaxInt}} (2147483647). *Using this value causes brokers to corrupt
> their log files, leaving them unreadable.*
> We set {{segment.bytes}} to {{2122317824}} which is well below the maximum.
> One by one, the ISR of all partitions shrunk to 1. Brokers would crash when
> restarted, attempting to read from a negative offset in a log file. After
> discovering that many segment files had grown to 4GB or more, we were forced
> to shut down our *entire production Kafka cluster* for several hours while we
> split all segment files into 1GB chunks.
> Looking into the {{kafka.log}} code, the {{segment.bytes}} parameter is used
> inconsistently. It is treated as a *soft* maximum for the size of the segment
> file
> (https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/log/LogConfig.scala#L26)
> with logs rolled only after
> (https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/log/Log.scala#L246)
> they exceed this value. However, much of the code that deals with log files
> uses *ints* to store the size of the file and the position in the file.
> Overflow of these ints leads the broker to append to the segments
> indefinitely, and to fail to read these segments for consuming or recovery.
> This is trivial to reproduce:
> {code}
> $ bin/kafka-topics.sh --topic segment-bytes-test --create
> --replication-factor 2 --partitions 1 --zookeeper zkhost:2181
> $ bin/kafka-topics.sh --topic segment-bytes-test --alter --config
> segment.bytes=2147483647 --zookeeper zkhost:2181
> $ yes "Int.MaxValue is a ridiculous bound on file size in 2014" |
> bin/kafka-console-producer.sh --broker-list localhost:6667 zkhost:2181
> --topic segment-bytes-test
> {code}
> After running for a few minutes, the log file is corrupt:
> {code}
> $ ls -lh data/segment-bytes-test-0/
> total 9.7G
> -rw-r--r-- 1 root root 10M Oct 3 19:39 00000000000000000000.index
> -rw-r--r-- 1 root root 9.7G Oct 3 19:39 00000000000000000000.log
> {code}
> We recovered the data from the log files using a simple Python script:
> https://gist.github.com/also/9f823d9eb9dc0a410796
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
