Nate Denning encountered the following error when trying to load a
large (greater than 2147483647 bytes) index into a RAMDirectory. The
server has 12GB of memory, so loading it into memory should not be a
problem.
Exception in thread "main" java.lang.NegativeArraySizeException
at
org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:63)
at
org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:89)
at org.search.lucene.SearchEngine.search(SearchEngine.java:105)
at org.search.lucene.SearchEngine.main(SearchEngine.java:52)
I helped modify the RAMDirectory the current dev branch as follows to
fix the error:
[Added]
private static final int MAX_BUFFER_SIZE = Integer.MAX_VALUE;
[Replaced]
int len = (int) is.length();
byte[] buf = new byte[len];
is.readBytes(buf, 0, len);
os.writeBytes(buf, len);
[with]
long len = is.length();
byte[] buf = len < MAX_BUFFER_SIZE ? new byte[(int) len] : new
byte[MAX_BUFFER_SIZE];
long bytesRemaining = len;
while (bytesRemaining > 0) {
int bytesToRead = bytesRemaining < buf.length ? (int)
bytesRemaining : buf.length;
is.readBytes(buf, 0, bytesToRead);
os.writeBytes(buf, bytesToRead);
bytesRemaining -= bytesToRead;
}
I first tested it with a smaller MAX_BUFFER_SIZE to make sure
that it will work. It still loaded a small index.
Nate is still trying to get it to load the large index, but is now
encountering memory allocation issues when trying to allocate a heap
larger than 2580MB. I'll keep you updated, if we make any more
progress using lucene and a RAMDirectory for large indexes.
Jonathan
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
