Hi Joseph,

The way the CSV importer works, for safety, is to process the whole CSV at 
once, and hold it all in memory (rather than reading and processing it 
line-by-line).  Therefore is probably safer to just import one or two thousand 
records at a time.  THe reason it is probably running out of memory half way 
through, is that as it goes through each line, it is building up its records of 
what has changed, so that it can display this information at the end.  So it 
has enough memory to read your long CSV file, but not enough to hold that, and 
the details of all the changes it has made.

Unfortunately no extra information (e.g. about checking the write lock) can be 
given, as the error you are encountering is an 'out of memory error', and these 
are provided directly by Java, not DSpace, so we have no control over them.

Thanks,


Stuart Lewis
Digital Development Manager
Te Tumu Herenga The University of Auckland Library
Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
Ph: +64 (0)9 373 7599 x81928



On 5/02/2011, at 6:47 AM, Joseph Rhoads wrote:

> Dear Dspace-tech community,
> 
> I'm curious if anyone else has had problems using the command line 
> metadata-import?
> 
> I'm trying to import 12861 items and it dies about half-way through.(7582 
> items).
> Before I upped the JAVA_OPTS memory max it would die between 2k and 3k items.
> 
> The error message is below, but an additional problem that the crash creates 
> is that  it leaves a "write.lock" file in "[dspace home]/search/"
> Which will create problems if you don't realize that it's there.
> Perhaps something should be mentioned in the error message when that happens.
> 
> Thank You,
> Joseph
> 
> Server Setup:
> -------------------
> Using Dspace 1.7
> Ubuntu 10.04 Server
> JAVA_OPTS=-Xmx512m
> Postgresql-8.4
> 
> Error Message
> --------------------------
> 12861 item(s) will be changed
> 
> Do you want to make these changes? [y/n] y
> Exception: Java heap space
> java.lang.OutOfMemoryError: Java heap space
>         at 
> org.apache.lucene.store.BufferedIndexOutput.<init>(BufferedIndexOutput.java:26)
>         at 
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:185)
>         at 
> org.apache.lucene.store.SimpleFSDirectory.createOutput(SimpleFSDirectory.java:59)
>         at 
> org.apache.lucene.index.FormatPostingsPositionsWriter.<init>(FormatPostingsPositionsWriter.java:42)
>         at 
> org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:55)
>         at 
> org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33)
>         at 
> org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59)
>         at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:96)
>         at org.apache.lucene.index.TermsHash.flush(TermsHash.java:154)
>         at org.apache.lucene.index.DocInverter.flush(DocInverter.java:76)
>         at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:63)
>         at 
> org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:634)
>         at 
> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4374)
>         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
>         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
>         at 
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2244)
>         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2207)
>         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2171)
>         at org.dspace.search.DSIndexer.processIndexingTask(DSIndexer.java:714)
>         at org.dspace.search.DSIndexer.indexContent(DSIndexer.java:271)
>         at org.dspace.search.SearchConsumer.end(SearchConsumer.java:181)
>         at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:149)
>         at org.dspace.core.Context.commit(Context.java:336)
>         at 
> org.dspace.app.bulkedit.MetadataImport.runImport(MetadataImport.java:254)
>         at 
> org.dspace.app.bulkedit.MetadataImport.main(MetadataImport.java:1060)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at 
> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:183)
> ---------------------
> 
> 
> 
> <ATT00001..txt><ATT00002..txt>



------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Reply via email to