Re: Load existing Lucene sharded indexes onto single Solr collection

2015-01-14 Thread Jaikit Savla
Yes, I wanted to get rid of merge step. But looks like merge is not that cumbersome either. Thanks Mikhail and Erick for pointers, that helped. Jaikit On Wednesday, January 14, 2015 8:24 AM, Erick Erickson wrote: You certainly can't do this into a single directory, there would be zi

Re: Load existing Lucene sharded indexes onto single Solr collection

2015-01-14 Thread Erick Erickson
You certainly can't do this into a single directory, there would be zillions of name conflicts. I believe I saw Uwe make a comment on the Lucene list about using MultiReaders and keeping the sub-indexes in different directories, but that's lower-level than Solr has access to Plus, you'd have to co

Re: Load existing Lucene sharded indexes onto single Solr collection

2015-01-14 Thread Jaikit Savla
This solution will merge the index as well. I want to find out if merge is "required" before loading indexes onto Solr ?  If that is possible than I can just point solrconfig.xml to directory where I have all the shards. Jaikit On Wednesday, January 14, 2015 1:11 AM, Mikhail Khludnev wro

Re: Load existing Lucene sharded indexes onto single Solr collection

2015-01-14 Thread Mikhail Khludnev
On Wed, Jan 14, 2015 at 11:42 AM, Jaikit Savla < jaikit.sa...@yahoo.com.invalid> wrote: > Now to load this index, I am currently using Lucene IndexMergeTool to > merge all the shards into one giant index. My question is, is there a way > to load shared index without merging into one giant index on

Load existing Lucene sharded indexes onto single Solr collection

2015-01-14 Thread Jaikit Savla
Folks, I have generated multiple (count of 100) sharded Lucene indexes on Hadoop and they are of format. The total indexed data (sum of all the index-*) is of size 500GB and hence the number of shards.drwxr-x--- 2 index-66drwxr-x--- 2 index-68drwxr-x--- 2 index-9 and each index directory