On Thursday 10 September 2009 01:47:38 pm Walter Underwood wrote: > What kind of storage is used for the Solr index files? When I tested it, NFS > was 100X slower than local disk.
I'm sorry - I misunderstood your question. The Solr indexes themselves are stored on local disk. The documents are retrievable (for DIH) from NFS. And, I started looking closer into this problem... both the box doing the posts, and the solr box are around 90% idle while the indexing process is running. And there is no I/O wait time. I'm now looking into possible network slowness... -Dan > > wunder > > -----Original Message----- > From: Dan A. Dickey [mailto:dan.dic...@savvis.net] > Sent: Thursday, September 10, 2009 11:15 AM > To: solr-user@lucene.apache.org > Cc: Walter Underwood > Subject: Re: Solr http post performance seems slow - help? > > On Thursday 10 September 2009 09:10:27 am Walter Underwood wrote: > > How big are your documents? > > For the most part, I'm just indexing metadata that has been pulled from > the documents. I think I have currently about 40 or so fields that I'm > setting. > When the document is an actual document - pdf, doc, etc... I use the DIH > to extract stuff and also set the metadata then. > > > Is your index on local disk or network- > > mounted disk? > > I'm basically pulling the metadata info from a database and the documents > themselves are shared via NFS to the Solr indexer. > -Dan > > > > > wunder > > > > On Sep 10, 2009, at 6:39 AM, Yonik Seeley wrote: > > > > > On Thu, Sep 10, 2009 at 9:13 AM, Dan A. Dickey > > > <dan.dic...@savvis.net> wrote: > > >> I'm posting documents to Solr using http (curl) from > > >> C++/C code and am seeing approximately 3.3 - 3.4 > > >> documents per second being posted. Is this to be expected? > > > > > > No, that's very slow. > > > Are you using libcurl, or actually forking a new process for every > > > document? > > > Are you committing on every document? > > > > > > If you can, using Java would make your life much easier since you > > > could use the SolrJ client and it's binary protocol for indexing. > > > > > > -Yonik > > > http://www.lucidimagination.com > > > > > > > > > -- Dan A. Dickey | Senior Software Engineer Savvis 10900 Hampshire Ave. S., Bloomington, MN 55438 Office: 952.852.4803 | Fax: 952.852.4951 E-mail: dan.dic...@savvis.net