[
https://issues.apache.org/jira/browse/KAFKA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109447#comment-14109447
]
Guozhang Wang commented on KAFKA-1489:
--------------------------------------
One concern I have about a global threshold based on percentage is about
multi-tenancy. Say we have two applications using the same Kafka server, one
with huge traffic on topic A and another with small traffic on topic B, and
hence because of the huge traffic of the first application causing disk to be
full, we will drop some messages of another application's topic B without the
application even being aware of that happening..
The per-topic config right now basically provides guarantees for each topic /
tenant using Kafka, although it may be tedious, it sounds like the right
approach to me for hosting multiple applications. I agree that it ignores the
global disk capacity, which will be caused usually when 1) application traffic
has naturally increased, in this case we probably should drop its data more
aggressively but instead in order to still make the data retention guarantee
(e.g. 7 days) we need to expand disk space or event the cluster size; 2) some
testing / temporary data get accumulated but not cleaned up in time, in this
case we would probably just delete the temporary topic and free up the space;
to automate this case we probably can tag the topic when created it (e.g.
setting log.retention.size to "-1" indicating we do not want to retain any of
its data), and when the disk full is alerted, the system can firstly try to
clean up those data.
Another issue is that, like Jay said, there is a risk that different nodes may
have different sized logs retained for the same partition due to the retention
policy, and hence when there is a leader change the consumer clients may get
"out-of-range" exception. We also need to be careful handling that case.
> Global threshold on data retention size
> ---------------------------------------
>
> Key: KAFKA-1489
> URL: https://issues.apache.org/jira/browse/KAFKA-1489
> Project: Kafka
> Issue Type: New Feature
> Components: log
> Affects Versions: 0.8.1.1
> Reporter: Andras Sereny
> Assignee: Jay Kreps
> Labels: newbie
>
> Currently, Kafka has per topic settings to control the size of one single log
> (log.retention.bytes). With lots of topics of different volume and as they
> grow in number, it could become tedious to maintain topic level settings
> applying to a single log.
> Often, a chunk of disk space is dedicated to Kafka that hosts all logs
> stored, so it'd make sense to have a configurable threshold to control how
> much space *all* data in one Kafka log data directory can take up.
> See also:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201406.mbox/browser
> http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3c20131107015125.gc9...@jkoshy-ld.linkedin.biz%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)
