Re: SOLR Index Speed

2011-09-30 Thread Lord Khan Han
;> Lucene ecosystem search :: http://search-lucene.com/ >> >> >> -- >> *From:* Otis Gospodnetic >> *To:* "solr-user@lucene.apache.org" >> *Sent:* Tuesday, September 27, 2011 1:37 PM >> >> *Subject:* Re: SOLR Index Sp

Re: SOLR Index Speed

2011-09-29 Thread Lord Khan Han
: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase > Lucene ecosystem search :: http://search-lucene.com/ > > > -- > *From:* Otis Gospodnetic > *To:* "solr-user@lucene.apache.org" > *Sent:* Tuesday, September 27, 2011 1:37 PM > >

Re: SOLR Index Speed

2011-09-27 Thread Otis Gospodnetic
t.com/ :: Solr - Lucene - Hadoop - HBase Lucene ecosystem search :: http://search-lucene.com/ > >From: Lord Khan Han >To: solr-user@lucene.apache.org; Otis Gospodnetic >Sent: Tuesday, September 27, 2011 4:42 AM >Subject: Re: SOLR Index Speed >

Re: SOLR Index Speed

2011-09-27 Thread Lord Khan Han
For SUSS https://issues.apache.org/jira/browse/SOLR-1565 it says no binary support... When we try to use from solrj binary through SUSS adding document took thousands of milliseconds to million per doc.!! so we turn back normal submit. On Tue, Sep 27, 2011 at 7:14 AM, Otis Gospodnetic < ot

Re: SOLR Index Speed

2011-09-27 Thread Lord Khan Han
1- each document around 50 kb - 150 kb (web document) 2-final index is 40 gig 3-jre memory carefully given. On Mon, Sep 26, 2011 at 9:57 PM, Jaeger, Jay - DOT wrote: > 500 / second would be 1,800,000 per hour (much more than 500K documents). > > 1) how big is each document? > 2) how big are

Re: SOLR Index Speed

2011-09-27 Thread Lord Khan Han
Our producer (hadoop mapper prepare the docs for submitting and the reducer diriectly submit from solrj http submit..) now 32 reducer but still the indexing speed 500 - 700 doc per minute. submission coming from a hadoop cluster so submit speed is not a problem. I couldnt use the full solr inde

Re: SOLR Index Speed

2011-09-27 Thread Lord Khan Han
Sorry :) it is not 500 doc per sec. ( It is what i wish I think) It is 500 doc per MINUTE.. On Tue, Sep 27, 2011 at 7:14 AM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > Hello, > > > PS: solr streamindex is not option because we need to submit javabin... > > > If you are referrin

Re: SOLR Index Speed

2011-09-26 Thread Otis Gospodnetic
Hello, > PS: solr streamindex  is not option because we need to submit javabin... If you are referring to StreamingUpdateSolrServer, then the above statement makes no sense and you should give SUSS a try. Are you sure your 16 reducers produce more than 500 docs/second? I think somebody already

RE: SOLR Index Speed

2011-09-26 Thread Lan
Are you batching the documents before sending them to the solr server? Are you doing a commit only at the end? Also since you have 32 cores, you can try upping the number of concurrent updaters from 16 to 32. Jaeger, Jay - DOT wrote: > > 500 / second would be 1,800,000 per hour (much more than

RE: SOLR Index Speed

2011-09-26 Thread Jaeger, Jay - DOT
500 / second would be 1,800,000 per hour (much more than 500K documents). 1) how big is each document? 2) how big are your index files? 3) as others have recently written, make sure you don't give your JRE so much memory that your OS is starved for memory to use for file system cache. JRJ --