[
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941910#comment-14941910
]
Vinoth Chandar commented on KAFKA-2580:
---------------------------------------
A LRU file handle is something very commonly employed in databases, which works
pretty well in practice. (considering that it involves random access). So +1
on that path.
[~granthenke] would you have cycles for this? If no one is working on this
currently, we (uber) can take a stab at this, later this quarter.
> Kafka Broker keeps file handles open for all log files (even if its not
> written to/read from)
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: Vinoth Chandar
>
> We noticed this in one of our clusters where we stage logs for a longer
> amount of time. It appears that the Kafka broker keeps file handles open even
> for non active (not written to or read from) files. (in fact, there are some
> threads going back to 2013
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever)
> Needless to say, this is a problem and forces us to either artificially bump
> up ulimit (its already at 100K) or expand the cluster (even if we have
> sufficient IO and everything).
> Filing this ticket, since I could find anything similar. Very interested to
> know if there are plans to address this (given how Samza's changelog topic is
> meant to be a persistent large state use case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
