Zhuo Liu created STORM-1206:
-------------------------------
Summary: Reduce logviewer memory usage
Key: STORM-1206
URL: https://issues.apache.org/jira/browse/STORM-1206
Project: Apache Storm
Issue Type: Improvement
Components: storm-core
Reporter: Zhuo Liu
Assignee: Zhuo Liu
In production, we ran into an issue with logviewers bouncing with out of memory
errors. Note that this happens very rarely, we met this in some extreme case
when super frequently restarting of workers generates a huge number of gc files
(~1M files).
What was happening is that if there are lots of log files (~1 M files) for a
particular headless user, we would have so many strings resident in memory that
logviewer would run out of heap space.
We were able to work around this by increasing the heap space, but we should
consider putting some sort of an upper bound on the number of files so that we
don't run in to this issue, even with the bigger heap.
Using the java DirectoryStream can avoid holding all file names in memory
during file listing. Also, a multi-round directory cleaner can be introduced to
delete files while disk quota is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
