More than 2G memory required for jruby -e 'buf = IO.read("/tmp/1GB.txt"); p buf.size' -------------------------------------------------------------------------------------
Key: JRUBY-3784 URL: http://jira.codehaus.org/browse/JRUBY-3784 Project: JRuby Issue Type: Bug Affects Versions: JRuby 1.3.1 Reporter: Wayne Meissner Assignee: Thomas E Enebo Fix For: JRuby 1.4 Leaving aside the wisdom or otherwise of trying to read 1G of data in one go, JRuby fails to load a 1G file even when the jvm memory is set to 1G. e.g. ./bin/jruby -J-Xmx2048m -e 'buf = IO.read("/tmp/1GB.txt"); p buf.size' Error: Your application used more memory than the safety cap of 2048m. Specify -J-Xmx####m to increase it (#### = cap size in MB). Specify -w for full OutOfMemoryError stack trace Part of this is due to the way the jvm does file I/O. When doing a 1G read into a heap buffer, the jvm will allocate a 1G direct ByteBuffer, do the read into the direct buffer, and then copy from the direct buffer to the heap buffer. Ergo, for a 1G read, it will allocate 2G of memory (1G heap, 1G direct). Splitting reads larger than 1M up into 1M sized chunks, seems to alleviate this situation. {format} diff --git a/src/org/jruby/util/io/ChannelStream.java b/src/org/jruby/util/io/ChannelStream.java index 4582eca..44b8614 100644 --- a/src/org/jruby/util/io/ChannelStream.java +++ b/src/org/jruby/util/io/ChannelStream.java @@ -362,10 +362,22 @@ public class ChannelStream implements Stream, Finalizable { // Now read unbuffered directly from the file // while (buf.hasRemaining()) { - int n = channel.read(buf); + final int MAX_READ_CHUNK = 1 * 1024 * 1024; + // + // When reading into a heap buffer, the jvm allocates a temporary + // direct ByteBuffer of the requested size. To avoid allocating + // a huge direct buffer when doing ludicrous reads (e.g. 1G or more) + // we split the read up into chunks of no more than 1M + // + ByteBuffer tmp = buf.duplicate(); + if (tmp.remaining() > MAX_READ_CHUNK) { + tmp.limit(tmp.position() + MAX_READ_CHUNK); + } + int n = channel.read(tmp); if (n <= 0) { break; } + buf.position(tmp.position()); } eof = true; result.length(buf.position()); {format} -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email