Distributing lucene segments across multiple disks.

2013-09-11 Thread Deepak Konidena
Hi, I know that SolrCloud allows you to have multiple shards on different machines (or a single machine). But it requires a zookeeper installation for doing things like leader election, leader availability, etc While SolrCloud may be the ideal solution for my usecase eventually, I'd like to know

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Upayavira
I think you'll find it hard to distribute different segments between disks, as they are typically stored in the same directory. However, instantiating separate cores on different disks should be straight-forward enough, and would give you a performance benefit. I've certainly heard of that done

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Deepak Konidena
...@gmail.com] Sent: Wednesday, September 11, 2013 2:07 PM To: solr-user@lucene.apache.org Subject: Re: Distributing lucene segments across multiple disks. Are you suggesting a multi-core setup, where all the cores share the same schema, and the cores lie on different disks? Basically, I'd

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Deepak Konidena
Are you suggesting a multi-core setup, where all the cores share the same schema, and the cores lie on different disks? Basically, I'd like to know if I can distribute shards/segments on a single machine (with multiple disks) without the use of zookeeper. -Deepak On Wed, Sep 11, 2013 at

RE: Distributing lucene segments across multiple disks.

2013-09-11 Thread Greg Walters
To: solr-user@lucene.apache.org Subject: Re: Distributing lucene segments across multiple disks. Are you suggesting a multi-core setup, where all the cores share the same schema, and the cores lie on different disks? Basically, I'd like to know if I can distribute shards/segments on a single

RE: Distributing lucene segments across multiple disks.

2013-09-11 Thread Greg Walters
cores/collections. Thanks, Greg -Original Message- From: Deepak Konidena [mailto:deepakk...@gmail.com] Sent: Wednesday, September 11, 2013 2:26 PM To: solr-user@lucene.apache.org Subject: Re: Distributing lucene segments across multiple disks. @Greg - Are you suggesting RAID

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Shawn Heisey
On 9/11/2013 1:07 PM, Deepak Konidena wrote: Are you suggesting a multi-core setup, where all the cores share the same schema, and the cores lie on different disks? Basically, I'd like to know if I can distribute shards/segments on a single machine (with multiple disks) without the use of

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Deepak Konidena
I guess at this point in the discussion, I should probably give some more background on why I am doing what I am doing. Having a single Solr shard (multiple segments) on the same disk is posing severe performance problems under load,in that, calls to Solr cause a lot of connection timeouts. When

RE: Distributing lucene segments across multiple disks.

2013-09-11 Thread Greg Walters
, September 11, 2013 3:57 PM To: solr-user@lucene.apache.org Subject: Re: Distributing lucene segments across multiple disks. I guess at this point in the discussion, I should probably give some more background on why I am doing what I am doing. Having a single Solr shard (multiple segments

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Shawn Heisey
On 9/11/2013 2:57 PM, Deepak Konidena wrote: I guess at this point in the discussion, I should probably give some more background on why I am doing what I am doing. Having a single Solr shard (multiple segments) on the same disk is posing severe performance problems under load,in that, calls to

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Deepak Konidena
@Greg - Thanks for the suggestion. Will pass it along to my folks. @Shawn - That's the link I was looking for 'non-SolrCloud approach to distributed search'. Thanks for passing that along. Will give it a try. As far as RAM usage goes, I believe we set the heap size to about 40% of the RAM and

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Shawn Heisey
On 9/11/2013 4:16 PM, Deepak Konidena wrote: As far as RAM usage goes, I believe we set the heap size to about 40% of the RAM and less than 10% is available for OS caching ( since replica takes another 40%). Why does unallocated RAM help? How does it impact performance under load? Because once

Re: Distributing lucene segments across multiple disks.

2013-09-11 Thread Deepak Konidena
Very helpful link. Thanks for sharing that. -Deepak On Wed, Sep 11, 2013 at 4:34 PM, Shawn Heisey s...@elyograg.org wrote: On 9/11/2013 4:16 PM, Deepak Konidena wrote: As far as RAM usage goes, I believe we set the heap size to about 40% of the RAM and less than 10% is available for OS