Hi, Yes, you can write to the same index from multiple threads. You still need to keep track of the index size manually, whether you create 1 or N indices/cores. I'd go with a single index first.
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: vivek sar <vivex...@gmail.com> > To: solr-user@lucene.apache.org > Sent: Wednesday, April 1, 2009 4:26:04 AM > Subject: Re: Merging Solr Indexes > > Thanks Otis. Could you write to same core (same index) from multiple > threads at the same time? I thought each writer would lock the index > so other can not write at the same time. I'll try it though. > > Another reason of putting indexes in separate core was to limit the > index size. Our index can grow up to 50G a day, so I was hoping > writing to smaller indexes would be faster in separate cores and if > needed I can merge them at later point (like end of day). I want to > keep daily cores. Isn't this a good idea? How else can I limit the > index size (beside multiple instances or separate boxes). > > Thanks, > -vivek > > > On Tue, Mar 31, 2009 at 8:28 PM, Otis Gospodnetic > wrote: > > > > Let me start with 4) > > Have you tried simply using multiple threads to send your docs to a single > Solr instance/core? You should get about the same performance as what you > are > trying with your approach below, but without the headache of managing > multiple > cores and index merging (not yet possible to do programatically). > > > > Otis > > -- > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > > > > > ----- Original Message ---- > >> From: vivek sar > >> To: solr-user@lucene.apache.org > >> Sent: Tuesday, March 31, 2009 1:59:01 PM > >> Subject: Merging Solr Indexes > >> > >> Hi, > >> > >> As part of speeding up the index process I'm thinking of spawning > >> multiple threads which will write to different temporary SolrCores. > >> Once the index process is done I want to merge all the indexes in > >> temporary cores to a master core. For ex., if I want one SolrCore per > >> day then every index cycle I'll spawn 4 threads which will index into > >> some temporary index and once they are done I want to merge all these > >> into the day core. My questions, > >> > >> 1) I want to use the same schema and solrconfig.xml for all cores > >> without duplicating them - how do I do that? > >> 2) How do I merge the temporary Solr cores into one master core > >> programmatically? I've read the wiki on "MergingSolrIndexes", but I > >> want to do it programmatically (like in Lucene - > >> writer.addIndexes(..)) once the temporary indices are done. > >> 3) Can I remove the temporary indices once the merge process is done? > >> 4) Is this the right strategy to speed up indexing? > >> > >> Thanks, > >> -vivek > > > >
