Thank you Erick for your valuable inputs. Yes, we have to re-index data again &
again. I'll look into possibility of tuning db access.
On SolrJ and automating the indexing (incremental as well as one time) I want
to get your opinion on below two points. We will be indexing separate sets of
tables with similar data structure
- Should we use SolrJ and write Java programs that can be scheduled to trigger
indexing on demand/schedule based.
- Is using SolrJ a better idea even for searching than using SolrNet? As our
frontend is in .Net so we started using SolrNet but I am afraid down the road
when we scale/support SolrClod using SolrJ is better?
Thanks
Susheel
-----Original Message-----
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Sunday, January 26, 2014 8:37 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr server requirements for 100+ million documents
Dumping the raw data would probably be a good idea. I guarantee you'll be
re-indexing the data several times as you change the schema to accommodate
different requirements...
But it may also be worth spending some time figuring out why the DB access is
slow. Sometimes one can tune that.
If you go the SolrJ route, you also have the possibility of setting up N
clients to work simultaneously, sometimes that'll help.
FWIW,
Erick
On Sat, Jan 25, 2014 at 11:06 PM, Susheel Kumar
<susheel.ku...@thedigitalgroup.net> wrote:
> Hi Kranti,
>
> Attach are the solrconfig & schema xml for review. I did run indexing with
> just few fields (5-6 fields) in schema.xml & keeping the same db config but
> Indexing almost still taking similar time (average 1 million records 1 hr)
> which confirms that the bottleneck is in the data acquisition which in our
> case is oracle database. I am thinking to not use dataimporthandler / jdbc to
> get data from Oracle but to rather dump data somehow from oracle using SQL
> loader and then index it. Any thoughts?
>
> Thnx
>
> -----Original Message-----
> From: Kranti Parisa [mailto:kranti.par...@gmail.com]
> Sent: Saturday, January 25, 2014 12:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr server requirements for 100+ million documents
>
> can you post the complete solrconfig.xml file and schema.xml files to review
> all of your settings that would impact your indexing performance.
>
> Thanks,
> Kranti K. Parisa
> http://www.linkedin.com/in/krantiparisa
>
>
>
> On Sat, Jan 25, 2014 at 12:56 AM, Susheel Kumar <
> susheel.ku...@thedigitalgroup.net> wrote:
>
>> Thanks, Svante. Your indexing speed using db seems to really fast.
>> Can you please provide some more detail on how you are indexing db
>> records. Is it thru DataImportHandler? And what database? Is that
>> local db? We are indexing around 70 fields (60 multivalued) but data
>> is not populated always in all fields. The average size of document is in
>> 5-10 kbs.
>>
>> -----Original Message-----
>> From: saka.csi...@gmail.com [mailto:saka.csi...@gmail.com] On Behalf
>> Of svante karlsson
>> Sent: Friday, January 24, 2014 5:05 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr server requirements for 100+ million documents
>>
>> I just indexed 100 million db docs (records) with 22 fields (4
>> multivalued) in 9524 sec using libcurl.
>> 11 million took 763 seconds so the speed drops somewhat with
>> increasing dbsize.
>>
>> We write 1000 docs (just an arbitrary number) in each request from
>> two threads. If you will be using solrcloud you will want more writer
>> threads.
>>
>> The hardware is a single cheap hp DL320E GEN8 V2 1P E3-1220V3 with
>> one SSD and 32GB and the solr runs on ubuntu 13.10 inside a esxi virtual
>> machine.
>>
>> /svante
>>
>>
>>
>>
>> 2014/1/24 Susheel Kumar <susheel.ku...@thedigitalgroup.net>
>>
>> > Thanks, Erick for the info.
>> >
>> > For indexing I agree the more time is consumed in data acquisition
>> > which in our case from Database. For indexing currently we are
>> > using the manual process i.e. Solr dashboard Data Import but now
>> > looking to automate. How do you suggest to automate the index part.
>> > Do you recommend to use SolrJ or should we try to automate using Curl?
>> >
>> >
>> > -----Original Message-----
>> > From: Erick Erickson [mailto:erickerick...@gmail.com]
>> > Sent: Friday, January 24, 2014 2:59 PM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: Solr server requirements for 100+ million documents
>> >
>> > Can't be done with the information you provided, and can only be
>> > guessed at even with more comprehensive information.
>> >
>> > Here's why:
>> >
>> >
>> > http://searchhub.org/2012/07/23/sizing-hardware-in-the-abstract-why
>> > -
>> > we
>> > -dont-have-a-definitive-answer/
>> >
>> > Also, at a guess, your indexing speed is so slow due to data
>> > acquisition; I rather doubt you're being limited by raw Solr indexing.
>> > If you're using SolrJ, try commenting out the
>> > server.add() bit and running again. My guess is that your indexing
>> > speed will be almost unchanged, in which case it's the data
>> > acquisition process is where you should concentrate efforts. As a
>> > comparison, I can index 11M Wikipedia docs on my laptop in 45
>> > minutes without any attempts at parallelization.
>> >
>> >
>> > Best,
>> > Erick
>> >
>> > On Fri, Jan 24, 2014 at 12:10 PM, Susheel Kumar <
>> > susheel.ku...@thedigitalgroup.net> wrote:
>> > > Hi,
>> > >
>> > > Currently we are indexing 10 million document from database (10
>> > > db data
>> > entities) & index size is around 8 GB on windows virtual box.
>> > Indexing in one shot taking 12+ hours while indexing parallel in
>> > separate cores & merging them together taking 4+ hours.
>> > >
>> > > We are looking to scale to 100+ million documents and looking for
>> > recommendation on servers requirements on below parameters for a
>> > Production environment. There can be 200+ users performing search
>> > same
>> time.
>> > >
>> > > No of physical servers (considering solr cloud) Memory
>> > > requirement Processor requirement (# cores) Linux as OS oppose to
>> > > windows
>> > >
>> > > Thanks in advance.
>> > > Susheel
>> > >
>> >
>>
