Re: Distributing lucene segments across multiple disks.

[Deepak Konidena]

Very helpful link. Thanks for sharing that.


-Deepak



On Wed, Sep 11, 2013 at 4:34 PM, Shawn Heisey  wrote:

> On 9/11/2013 4:16 PM, Deepak Konidena wrote:
>
>> As far as RAM usage goes, I believe we set the heap size to about 40% of
>> the RAM and less than 10% is available for OS caching ( since replica
>> takes
>> another 40%). Why does unallocated RAM help? How does it impact
>> performance
>> under load?
>>
>
> Because once the data is in the OS disk cache, reading it becomes
> instantaneous, it doesn't need to go out to the disk.  Disks are glacial
> compared to RAM.  Even SSD has a far slower response time.  Any recent
> operating system does this automatically, including the one from Redmond
> that we all love to hate.
>
> http://blog.thetaphi.de/2012/**07/use-lucenes-mmapdirectory-**
> on-64bit.html
>
> Thanks,
> Shawn
>
>

Re: Distributing lucene segments across multiple disks.

[Shawn Heisey]

On 9/11/2013 4:16 PM, Deepak Konidena wrote:

As far as RAM usage goes, I believe we set the heap size to about 40% of
the RAM and less than 10% is available for OS caching ( since replica takes
another 40%). Why does unallocated RAM help? How does it impact performance
under load?


Because once the data is in the OS disk cache, reading it becomes 
instantaneous, it doesn't need to go out to the disk.  Disks are glacial 
compared to RAM.  Even SSD has a far slower response time.  Any recent 
operating system does this automatically, including the one from Redmond 
that we all love to hate.


http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Thanks,
Shawn

Re: Distributing lucene segments across multiple disks.

[Deepak Konidena]

@Greg - Thanks for the suggestion. Will pass it along to my folks.

@Shawn - That's the link I was looking for 'non-SolrCloud approach to
distributed search'. Thanks for passing that along. Will give it a try.

As far as RAM usage goes, I believe we set the heap size to about 40% of
the RAM and less than 10% is available for OS caching ( since replica takes
another 40%). Why does unallocated RAM help? How does it impact performance
under load?


-Deepak



On Wed, Sep 11, 2013 at 2:50 PM, Shawn Heisey  wrote:

> On 9/11/2013 2:57 PM, Deepak Konidena wrote:
>
>> I guess at this point in the discussion, I should probably give some more
>> background on why I am doing what I am doing. Having a single Solr shard
>> (multiple segments) on the same disk is posing severe performance problems
>> under load,in that, calls to Solr cause a lot of connection timeouts. When
>> we looked at the ganglia stats for the Solr box, we saw that while memory,
>> cpu and network usage were quite normal, the i/o wait spiked. We are
>> unsure
>> on what caused the i/o wait and why there were no spikes in the cpu/memory
>> usage. Since the Solr box is a beefy box (multi-core setup, huge ram,
>> SSD),
>> we'd like to distribute the segments to multiple locations (disks) and see
>> whether this improves performance under load.
>>
>> @Greg - Thanks for clarifying that.  I just learnt that I can't set them
>> up
>> using RAID as some of them are SSDs and some others are SATA (spinning
>> disks).
>>
>> @Shawn Heisey - Could you elaborate more about the "broker" core and
>> delegating the requests to other cores?
>>
>
> On the broker core - I have a core on my servers that has no index of its
> own.  In the /select handler (and others) I have placed a shards parameter,
> and many of them also have a shards.qt parameter.  The shards paramter is
> how a non-cloud distributed search is done.
>
> http://wiki.apache.org/solr/**DistributedSearch
>
> Addressing your first paragraph: You say that you have lots of RAM ... but
> is there a lot of unallocated RAM that the OS can use for caching, or is it
> mostly allocated to processes, such as the java heap for Solr?
>
> Depending on exactly how your indexes are composed, you need up to 100% of
> the total index size available as unallocated RAM.  With SSD, the
> requirement is less, but cannot be ignored.  I personally wouldn't go below
> about 25-50% even with SSD, and I'd plan on 50-100% for regular disks.
>
> There is some evidence to suggest that you only need unallocated RAM equal
> to 10% of your index size for caching with SSD, but that is only likely to
> work if you have a lot of stored (as opposed to indexed) data.  If most of
> your index is unstored, then more would be required.
>
> Thanks,
> Shawn
>
>

Re: Distributing lucene segments across multiple disks.

[Shawn Heisey]

On 9/11/2013 2:57 PM, Deepak Konidena wrote:

I guess at this point in the discussion, I should probably give some more
background on why I am doing what I am doing. Having a single Solr shard
(multiple segments) on the same disk is posing severe performance problems
under load,in that, calls to Solr cause a lot of connection timeouts. When
we looked at the ganglia stats for the Solr box, we saw that while memory,
cpu and network usage were quite normal, the i/o wait spiked. We are unsure
on what caused the i/o wait and why there were no spikes in the cpu/memory
usage. Since the Solr box is a beefy box (multi-core setup, huge ram, SSD),
we'd like to distribute the segments to multiple locations (disks) and see
whether this improves performance under load.

@Greg - Thanks for clarifying that.  I just learnt that I can't set them up
using RAID as some of them are SSDs and some others are SATA (spinning
disks).

@Shawn Heisey - Could you elaborate more about the "broker" core and
delegating the requests to other cores?


On the broker core - I have a core on my servers that has no index of 
its own.  In the /select handler (and others) I have placed a shards 
parameter, and many of them also have a shards.qt parameter.  The shards 
paramter is how a non-cloud distributed search is done.


http://wiki.apache.org/solr/DistributedSearch

Addressing your first paragraph: You say that you have lots of RAM ... 
but is there a lot of unallocated RAM that the OS can use for caching, 
or is it mostly allocated to processes, such as the java heap for Solr?


Depending on exactly how your indexes are composed, you need up to 100% 
of the total index size available as unallocated RAM.  With SSD, the 
requirement is less, but cannot be ignored.  I personally wouldn't go 
below about 25-50% even with SSD, and I'd plan on 50-100% for regular disks.


There is some evidence to suggest that you only need unallocated RAM 
equal to 10% of your index size for caching with SSD, but that is only 
likely to work if you have a lot of stored (as opposed to indexed) data. 
 If most of your index is unstored, then more would be required.


Thanks,
Shawn

RE: Distributing lucene segments across multiple disks.

[Greg Walters]

Deepak,

It might be a bit outside what you're willing to consider but you can make a 
raid out of your spinning disks then use your SSD(s) as a dm-cache device to 
accelerate reads and writes to the raid device. If you're putting lucene 
indexes on a mixed bag of disks and ssd's without any type of control for what 
goes where you'd want to use the ssd to accelerate the spinning disks anyway. 
Check out http://lwn.net/Articles/540996/ for more information on the dm-cache 
device.

Thanks,
Greg

-Original Message-
From: Deepak Konidena [mailto:deepakk...@gmail.com] 
Sent: Wednesday, September 11, 2013 3:57 PM
To: solr-user@lucene.apache.org
Subject: Re: Distributing lucene segments across multiple disks.

I guess at this point in the discussion, I should probably give some more 
background on why I am doing what I am doing. Having a single Solr shard 
(multiple segments) on the same disk is posing severe performance problems 
under load,in that, calls to Solr cause a lot of connection timeouts. When we 
looked at the ganglia stats for the Solr box, we saw that while memory, cpu and 
network usage were quite normal, the i/o wait spiked. We are unsure on what 
caused the i/o wait and why there were no spikes in the cpu/memory usage. Since 
the Solr box is a beefy box (multi-core setup, huge ram, SSD), we'd like to 
distribute the segments to multiple locations (disks) and see whether this 
improves performance under load.

@Greg - Thanks for clarifying that.  I just learnt that I can't set them up 
using RAID as some of them are SSDs and some others are SATA (spinning disks).

@Shawn Heisey - Could you elaborate more about the "broker" core and delegating 
the requests to other cores?


-Deepak



On Wed, Sep 11, 2013 at 1:10 PM, Shawn Heisey  wrote:

> On 9/11/2013 1:07 PM, Deepak Konidena wrote:
>
>> Are you suggesting a multi-core setup, where all the cores share the 
>> same schema, and the cores lie on different disks?
>>
>> Basically, I'd like to know if I can distribute shards/segments on a 
>> single machine (with multiple disks) without the use of zookeeper.
>>
>
> Sure, you can do it all manually.  At that point you would not be using
> SolrCloud at all, because the way to enable SolrCloud is to tell Solr where
> zookeeper lives.
>
> Without SolrCloud, there is no cluster automation at all.  There is no
> "collection" paradigm, you just have cores.  You have to send updates to
> the correct core; they not be redirected for you.  Similarly, queries will
> not be load balanced automatically.  For Java clients, the CloudSolrServer
> object can work seamlessly when servers go down.  If you're not using
> SolrCloud, you can't use CloudSolrServer.
>
> You would be in charge of creating the shards parameter yourself.  The way
> that I do this on my index is that I have a "broker" core that has no index
> of its own, but its solrconfig.xml has the shards and shards.qt parameters
> in all the request handler definitions.  You can also include the parameter
> with the query.
>
> You would also have to handle redundancy yourself, either with replication
> or with independently updated indexes.  I use the latter method, because it
> offers a lot more flexibility than replication.
>
> As mentioned in another reply, setting up RAID with a lot of disks may be
> better than trying to split your index up on different filesystems that
> each reside on different disks.  I would recommend RAID10 for Solr, and it
> works best if it's hardware RAID and the controller has battery-backed (or
> NVRAM) cache.
>
> Thanks,
> Shawn
>
>

Re: Distributing lucene segments across multiple disks.

[Deepak Konidena]

I guess at this point in the discussion, I should probably give some more
background on why I am doing what I am doing. Having a single Solr shard
(multiple segments) on the same disk is posing severe performance problems
under load,in that, calls to Solr cause a lot of connection timeouts. When
we looked at the ganglia stats for the Solr box, we saw that while memory,
cpu and network usage were quite normal, the i/o wait spiked. We are unsure
on what caused the i/o wait and why there were no spikes in the cpu/memory
usage. Since the Solr box is a beefy box (multi-core setup, huge ram, SSD),
we'd like to distribute the segments to multiple locations (disks) and see
whether this improves performance under load.

@Greg - Thanks for clarifying that.  I just learnt that I can't set them up
using RAID as some of them are SSDs and some others are SATA (spinning
disks).

@Shawn Heisey - Could you elaborate more about the "broker" core and
delegating the requests to other cores?


-Deepak



On Wed, Sep 11, 2013 at 1:10 PM, Shawn Heisey  wrote:

> On 9/11/2013 1:07 PM, Deepak Konidena wrote:
>
>> Are you suggesting a multi-core setup, where all the cores share the same
>> schema, and the cores lie on different disks?
>>
>> Basically, I'd like to know if I can distribute shards/segments on a
>> single
>> machine (with multiple disks) without the use of zookeeper.
>>
>
> Sure, you can do it all manually.  At that point you would not be using
> SolrCloud at all, because the way to enable SolrCloud is to tell Solr where
> zookeeper lives.
>
> Without SolrCloud, there is no cluster automation at all.  There is no
> "collection" paradigm, you just have cores.  You have to send updates to
> the correct core; they not be redirected for you.  Similarly, queries will
> not be load balanced automatically.  For Java clients, the CloudSolrServer
> object can work seamlessly when servers go down.  If you're not using
> SolrCloud, you can't use CloudSolrServer.
>
> You would be in charge of creating the shards parameter yourself.  The way
> that I do this on my index is that I have a "broker" core that has no index
> of its own, but its solrconfig.xml has the shards and shards.qt parameters
> in all the request handler definitions.  You can also include the parameter
> with the query.
>
> You would also have to handle redundancy yourself, either with replication
> or with independently updated indexes.  I use the latter method, because it
> offers a lot more flexibility than replication.
>
> As mentioned in another reply, setting up RAID with a lot of disks may be
> better than trying to split your index up on different filesystems that
> each reside on different disks.  I would recommend RAID10 for Solr, and it
> works best if it's hardware RAID and the controller has battery-backed (or
> NVRAM) cache.
>
> Thanks,
> Shawn
>
>

Re: Distributing lucene segments across multiple disks.

[Shawn Heisey]

On 9/11/2013 1:07 PM, Deepak Konidena wrote:

Are you suggesting a multi-core setup, where all the cores share the same
schema, and the cores lie on different disks?

Basically, I'd like to know if I can distribute shards/segments on a single
machine (with multiple disks) without the use of zookeeper.


Sure, you can do it all manually.  At that point you would not be using 
SolrCloud at all, because the way to enable SolrCloud is to tell Solr 
where zookeeper lives.


Without SolrCloud, there is no cluster automation at all.  There is no 
"collection" paradigm, you just have cores.  You have to send updates to 
the correct core; they not be redirected for you.  Similarly, queries 
will not be load balanced automatically.  For Java clients, the 
CloudSolrServer object can work seamlessly when servers go down.  If 
you're not using SolrCloud, you can't use CloudSolrServer.


You would be in charge of creating the shards parameter yourself.  The 
way that I do this on my index is that I have a "broker" core that has 
no index of its own, but its solrconfig.xml has the shards and shards.qt 
parameters in all the request handler definitions.  You can also include 
the parameter with the query.


You would also have to handle redundancy yourself, either with 
replication or with independently updated indexes.  I use the latter 
method, because it offers a lot more flexibility than replication.


As mentioned in another reply, setting up RAID with a lot of disks may 
be better than trying to split your index up on different filesystems 
that each reside on different disks.  I would recommend RAID10 for Solr, 
and it works best if it's hardware RAID and the controller has 
battery-backed (or NVRAM) cache.


Thanks,
Shawn

RE: Distributing lucene segments across multiple disks.

[Greg Walters]

Deepak,

Sorry for not being more verbose in my previous suggestion. As I take your 
question, you'd like to spread your index files across multiple disks (for 
performance or space reasons I assume). If you used even a basic md-raid setup 
you could then format the raid device and thus your entire set of disks with 
your favorite filesystem, mount it in one directory in the directory tree then 
configure it as the data directory that solr uses.

This setup would accomplish your goal of having the lucene indexes spread 
across multiple disks without the complexity of using multiple solr 
cores/collections.

Thanks,
Greg


-Original Message-
From: Deepak Konidena [mailto:deepakk...@gmail.com] 
Sent: Wednesday, September 11, 2013 2:26 PM
To: solr-user@lucene.apache.org
Subject: Re: Distributing lucene segments across multiple disks.

@Greg - Are you suggesting RAID as a replacement for Solr or making Solr work 
with RAID? Could you elaborate more on the latter, if that's you meant?
We make use of solr's advanced text processing features which would be hard to 
replicate just using RAID.


-Deepak



On Wed, Sep 11, 2013 at 12:11 PM, Greg Walters  wrote:

> Why not use some form of RAID for your index store? You'd get the 
> performance benefit of multiple disks without the complexity of 
> managing them via solr.
>
> Thanks,
> Greg
>
>
>
> -Original Message-
> From: Deepak Konidena [mailto:deepakk...@gmail.com]
> Sent: Wednesday, September 11, 2013 2:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Distributing lucene segments across multiple disks.
>
> Are you suggesting a multi-core setup, where all the cores share the 
> same schema, and the cores lie on different disks?
>
> Basically, I'd like to know if I can distribute shards/segments on a 
> single machine (with multiple disks) without the use of zookeeper.
>
>
>
>
>
> -Deepak
>
>
>
> On Wed, Sep 11, 2013 at 11:55 AM, Upayavira  wrote:
>
> > I think you'll find it hard to distribute different segments between 
> > disks, as they are typically stored in the same directory.
> >
> > However, instantiating separate cores on different disks should be 
> > straight-forward enough, and would give you a performance benefit.
> >
> > I've certainly heard of that done at Amazon, with a separate EBS 
> > volume per core giving some performance improvement.
> >
> > Upayavira
> >
> > On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > > Hi,
> > >
> > > I know that SolrCloud allows you to have multiple shards on 
> > > different machines (or a single machine). But it requires a 
> > > zookeeper installation for doing things like leader election, 
> > > leader availability, etc
> > >
> > > While SolrCloud may be the ideal solution for my usecase 
> > > eventually, I'd like to know if there's a way I can point my Solr 
> > > instance to read lucene segments distributed across different 
> > > disks attached to
> the same machine.
> > >
> > > Thanks!
> > >
> > > -Deepak
> >
>

RE: Distributing lucene segments across multiple disks.

[Greg Walters]

Why not use some form of RAID for your index store? You'd get the performance 
benefit of multiple disks without the complexity of managing them via solr.

Thanks,
Greg



-Original Message-
From: Deepak Konidena [mailto:deepakk...@gmail.com] 
Sent: Wednesday, September 11, 2013 2:07 PM
To: solr-user@lucene.apache.org
Subject: Re: Distributing lucene segments across multiple disks.

Are you suggesting a multi-core setup, where all the cores share the same 
schema, and the cores lie on different disks?

Basically, I'd like to know if I can distribute shards/segments on a single 
machine (with multiple disks) without the use of zookeeper.





-Deepak



On Wed, Sep 11, 2013 at 11:55 AM, Upayavira  wrote:

> I think you'll find it hard to distribute different segments between 
> disks, as they are typically stored in the same directory.
>
> However, instantiating separate cores on different disks should be 
> straight-forward enough, and would give you a performance benefit.
>
> I've certainly heard of that done at Amazon, with a separate EBS 
> volume per core giving some performance improvement.
>
> Upayavira
>
> On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > Hi,
> >
> > I know that SolrCloud allows you to have multiple shards on 
> > different machines (or a single machine). But it requires a 
> > zookeeper installation for doing things like leader election, leader 
> > availability, etc
> >
> > While SolrCloud may be the ideal solution for my usecase eventually, 
> > I'd like to know if there's a way I can point my Solr instance to 
> > read lucene segments distributed across different disks attached to the 
> > same machine.
> >
> > Thanks!
> >
> > -Deepak
>

Re: Distributing lucene segments across multiple disks.

[Deepak Konidena]

Are you suggesting a multi-core setup, where all the cores share the same
schema, and the cores lie on different disks?

Basically, I'd like to know if I can distribute shards/segments on a single
machine (with multiple disks) without the use of zookeeper.





-Deepak



On Wed, Sep 11, 2013 at 11:55 AM, Upayavira  wrote:

> I think you'll find it hard to distribute different segments between
> disks, as they are typically stored in the same directory.
>
> However, instantiating separate cores on different disks should be
> straight-forward enough, and would give you a performance benefit.
>
> I've certainly heard of that done at Amazon, with a separate EBS volume
> per core giving some performance improvement.
>
> Upayavira
>
> On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > Hi,
> >
> > I know that SolrCloud allows you to have multiple shards on different
> > machines (or a single machine). But it requires a zookeeper installation
> > for doing things like leader election, leader availability, etc
> >
> > While SolrCloud may be the ideal solution for my usecase eventually, I'd
> > like to know if there's a way I can point my Solr instance to read lucene
> > segments distributed across different disks attached to the same machine.
> >
> > Thanks!
> >
> > -Deepak
>

Re: Distributing lucene segments across multiple disks.

[Deepak Konidena]

@Greg - Are you suggesting RAID as a replacement for Solr or making Solr
work with RAID? Could you elaborate more on the latter, if that's you
meant?
We make use of solr's advanced text processing features which would be hard
to replicate just using RAID.


-Deepak



On Wed, Sep 11, 2013 at 12:11 PM, Greg Walters  wrote:

> Why not use some form of RAID for your index store? You'd get the
> performance benefit of multiple disks without the complexity of managing
> them via solr.
>
> Thanks,
> Greg
>
>
>
> -Original Message-
> From: Deepak Konidena [mailto:deepakk...@gmail.com]
> Sent: Wednesday, September 11, 2013 2:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Distributing lucene segments across multiple disks.
>
> Are you suggesting a multi-core setup, where all the cores share the same
> schema, and the cores lie on different disks?
>
> Basically, I'd like to know if I can distribute shards/segments on a
> single machine (with multiple disks) without the use of zookeeper.
>
>
>
>
>
> -Deepak
>
>
>
> On Wed, Sep 11, 2013 at 11:55 AM, Upayavira  wrote:
>
> > I think you'll find it hard to distribute different segments between
> > disks, as they are typically stored in the same directory.
> >
> > However, instantiating separate cores on different disks should be
> > straight-forward enough, and would give you a performance benefit.
> >
> > I've certainly heard of that done at Amazon, with a separate EBS
> > volume per core giving some performance improvement.
> >
> > Upayavira
> >
> > On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> > > Hi,
> > >
> > > I know that SolrCloud allows you to have multiple shards on
> > > different machines (or a single machine). But it requires a
> > > zookeeper installation for doing things like leader election, leader
> > > availability, etc
> > >
> > > While SolrCloud may be the ideal solution for my usecase eventually,
> > > I'd like to know if there's a way I can point my Solr instance to
> > > read lucene segments distributed across different disks attached to
> the same machine.
> > >
> > > Thanks!
> > >
> > > -Deepak
> >
>

Re: Distributing lucene segments across multiple disks.

[Upayavira]

I think you'll find it hard to distribute different segments between
disks, as they are typically stored in the same directory.

However, instantiating separate cores on different disks should be
straight-forward enough, and would give you a performance benefit.

I've certainly heard of that done at Amazon, with a separate EBS volume
per core giving some performance improvement.

Upayavira

On Wed, Sep 11, 2013, at 07:35 PM, Deepak Konidena wrote:
> Hi,
> 
> I know that SolrCloud allows you to have multiple shards on different
> machines (or a single machine). But it requires a zookeeper installation
> for doing things like leader election, leader availability, etc
> 
> While SolrCloud may be the ideal solution for my usecase eventually, I'd
> like to know if there's a way I can point my Solr instance to read lucene
> segments distributed across different disks attached to the same machine.
> 
> Thanks!
> 
> -Deepak

Distributing lucene segments across multiple disks.

[Deepak Konidena]

Hi,

I know that SolrCloud allows you to have multiple shards on different
machines (or a single machine). But it requires a zookeeper installation
for doing things like leader election, leader availability, etc

While SolrCloud may be the ideal solution for my usecase eventually, I'd
like to know if there's a way I can point my Solr instance to read lucene
segments distributed across different disks attached to the same machine.

Thanks!

-Deepak