From: Noble Paul നോബിള് नोब्ळ्
<noble.p...@corp.aol.com>
Subject: Re: How to index large set data
To: solr-user@lucene.apache.org
Date: Thursday, May 21, 2009, 10:39 PM
what is the total no:of docs created
? I guess it may not be memory
bound. indexing is mostly amn IO bound operation. You may
be able to
get a better perf if a SSD is used (solid state disk)
On Fri, May 22, 2009 at 10:46 AM, Jianbin Dai <djian...@yahoo.com>
wrote:
Hi Paul,
Thank you so much for answering my questions. It
really helped.
After some adjustment, basically setting mergeFactor
to 1000 from the default value of 10, I can finished the
whole job in 2.5 hours. I checked that during running time,
only around 18% of memory is being used, and VIRT is always
1418m. I am thinking it may be restricted by JVM memory
setting. But I run the data import command through web,
i.e.,
http://<host>:<port>/solr/dataimport?command=full-import,
how can I set the memory allocation for JVM?
Thanks again!
JB
--- On Thu, 5/21/09, Noble Paul നോബിള്
नोब्ळ् <noble.p...@corp.aol.com>
wrote:
From: Noble Paul നോബിള്
नोब्ळ् <noble.p...@corp.aol.com>
Subject: Re: How to index large set data
To: solr-user@lucene.apache.org
Date: Thursday, May 21, 2009, 9:57 PM
check the status page of DIH and see
if it is working properly. and
if, yes what is the rate of indexing
On Thu, May 21, 2009 at 11:48 AM, Jianbin Dai
<djian...@yahoo.com>
wrote:
Hi,
I have about 45GB xml files to be indexed. I
am using
DataImportHandler. I started the full import 4
hours ago,
and it's still running....
My computer has 4GB memory. Any suggestion on
the
solutions?
Thanks!
JB
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com