[ https://issues.apache.org/jira/browse/KAFKA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109447#comment-14109447 ]
Guozhang Wang commented on KAFKA-1489: -------------------------------------- One concern I have about a global threshold based on percentage is about multi-tenancy. Say we have two applications using the same Kafka server, one with huge traffic on topic A and another with small traffic on topic B, and hence because of the huge traffic of the first application causing disk to be full, we will drop some messages of another application's topic B without the application even being aware of that happening.. The per-topic config right now basically provides guarantees for each topic / tenant using Kafka, although it may be tedious, it sounds like the right approach to me for hosting multiple applications. I agree that it ignores the global disk capacity, which will be caused usually when 1) application traffic has naturally increased, in this case we probably should drop its data more aggressively but instead in order to still make the data retention guarantee (e.g. 7 days) we need to expand disk space or event the cluster size; 2) some testing / temporary data get accumulated but not cleaned up in time, in this case we would probably just delete the temporary topic and free up the space; to automate this case we probably can tag the topic when created it (e.g. setting log.retention.size to "-1" indicating we do not want to retain any of its data), and when the disk full is alerted, the system can firstly try to clean up those data. Another issue is that, like Jay said, there is a risk that different nodes may have different sized logs retained for the same partition due to the retention policy, and hence when there is a leader change the consumer clients may get "out-of-range" exception. We also need to be careful handling that case. > Global threshold on data retention size > --------------------------------------- > > Key: KAFKA-1489 > URL: https://issues.apache.org/jira/browse/KAFKA-1489 > Project: Kafka > Issue Type: New Feature > Components: log > Affects Versions: 0.8.1.1 > Reporter: Andras Sereny > Assignee: Jay Kreps > Labels: newbie > > Currently, Kafka has per topic settings to control the size of one single log > (log.retention.bytes). With lots of topics of different volume and as they > grow in number, it could become tedious to maintain topic level settings > applying to a single log. > Often, a chunk of disk space is dedicated to Kafka that hosts all logs > stored, so it'd make sense to have a configurable threshold to control how > much space *all* data in one Kafka log data directory can take up. > See also: > http://mail-archives.apache.org/mod_mbox/kafka-users/201406.mbox/browser > http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3c20131107015125.gc9...@jkoshy-ld.linkedin.biz%3E -- This message was sent by Atlassian JIRA (v6.2#6252)