inter.broker.protocol.version = 0.10.1-IV2 log.message.format.version = 0.10.1-IV2
It will take me longer to check the producer/consumer versions, but I believe they're all *at least* 0.10 -Gwilym On 20 April 2017 at 06:42, Manikumar <[email protected]> wrote: > You may be producing in the old message format. Check the > "log.message.format.version" config. > What is the version of the Producer/Consumer clients? > > > On Thu, Apr 20, 2017 at 11:39 AM, Gwilym Evans < > [email protected] > > wrote: > > > I am running 0.10.1.0 so, if that's true, it might not be a default. If > you > > know of a config value to change that would be very helpful. > > > > -Gwilym > > > > On 20 April 2017 at 06:07, Manikumar <[email protected]> wrote: > > > > > AFAIK, this behavior is changed in 0.10.1.0 release. Now retention is > > based > > > on the largest > > > timestamp of the messages in a log segment. > > > > > > On Thu, Apr 20, 2017 at 11:19 AM, Gwilym Evans < > > > [email protected] > > > > wrote: > > > > > > > Hello, > > > > > > > > Yesterday, I had to replace a faulty Kafka broker node, and the > method > > of > > > > replacement involved bringing up a blank replacement using the old > > > broker's > > > > ID, thus triggering a replication of all its old partitions. > > > > > > > > Today I was dealing with disk usage alerts for only that broker: it > > > turned > > > > out that the broker was not deleting old logs like the rest of the > > nodes. > > > > > > > > I haven't checked the code, but eventually I came to the conclusion > > that > > > > Kafka log file deletion is based on file create or modified time, > > rather > > > > than the max produce time of the messages within the log file itself. > > > > > > > > This makes the method I use of replacing a faulty node with a blank > > slate > > > > problematic, since five day old messages will be stored in a file > with > > a > > > > recent c/mtime, thus won't be deleted and will soon cause disk space > > > > exhaustion. > > > > > > > > My temporary workaround was to reduce retention of the largest topic > to > > > 24 > > > > hours but I'd prefer not doing that since it's more manual work and > it > > > > breaks my SLA. > > > > > > > > Can this behaviour of Kafka be changed via configs at all? > > > > > > > > Has anyone faced a similar problem and have suggestions? > > > > > > > > Thanks, > > > > Gwilym > > > > > > > > > >
