;> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>> --
>> *From:* Otis Gospodnetic
>> *To:* "solr-user@lucene.apache.org"
>> *Sent:* Tuesday, September 27, 2011 1:37 PM
>>
>> *Subject:* Re: SOLR Index Sp
: http://sematext.com/ :: Solr - Lucene - Hadoop - HBase
> Lucene ecosystem search :: http://search-lucene.com/
>
>
> --
> *From:* Otis Gospodnetic
> *To:* "solr-user@lucene.apache.org"
> *Sent:* Tuesday, September 27, 2011 1:37 PM
>
>
t.com/ :: Solr - Lucene - Hadoop - HBase
Lucene ecosystem search :: http://search-lucene.com/
>
>From: Lord Khan Han
>To: solr-user@lucene.apache.org; Otis Gospodnetic
>Sent: Tuesday, September 27, 2011 4:42 AM
>Subject: Re: SOLR Index Speed
>
For SUSS https://issues.apache.org/jira/browse/SOLR-1565 it says no
binary support...
When we try to use from solrj binary through SUSS adding document took
thousands of milliseconds to million per doc.!! so we turn back normal
submit.
On Tue, Sep 27, 2011 at 7:14 AM, Otis Gospodnetic <
ot
1- each document around 50 kb - 150 kb (web document)
2-final index is 40 gig
3-jre memory carefully given.
On Mon, Sep 26, 2011 at 9:57 PM, Jaeger, Jay - DOT wrote:
> 500 / second would be 1,800,000 per hour (much more than 500K documents).
>
> 1) how big is each document?
> 2) how big are
Our producer (hadoop mapper prepare the docs for submitting and the reducer
diriectly submit from solrj http submit..) now 32 reducer but still the
indexing speed 500 - 700 doc per minute. submission coming from a hadoop
cluster so submit speed is not a problem. I couldnt use the full solr inde
Sorry :) it is not 500 doc per sec. ( It is what i wish I think) It is
500 doc per MINUTE..
On Tue, Sep 27, 2011 at 7:14 AM, Otis Gospodnetic <
otis_gospodne...@yahoo.com> wrote:
> Hello,
>
> > PS: solr streamindex is not option because we need to submit javabin...
>
>
> If you are referrin
Hello,
> PS: solr streamindex is not option because we need to submit javabin...
If you are referring to StreamingUpdateSolrServer, then the above statement
makes no sense and you should give SUSS a try.
Are you sure your 16 reducers produce more than 500 docs/second?
I think somebody already
Are you batching the documents before sending them to the solr server? Are
you doing a commit only at the end? Also since you have 32 cores, you can
try upping the number of concurrent updaters from 16 to 32.
Jaeger, Jay - DOT wrote:
>
> 500 / second would be 1,800,000 per hour (much more than
500 / second would be 1,800,000 per hour (much more than 500K documents).
1) how big is each document?
2) how big are your index files?
3) as others have recently written, make sure you don't give your JRE so much
memory that your OS is starved for memory to use for file system cache.
JRJ
--
10 matches
Mail list logo