[Gluster-users] Gluster VMs Paused State then not resumable

[Andrew Lau]

Hi all,

I'm wondering if anyone's had any luck with running VMs on top of a gluster VM?
I have a bug open here,
https://bugzilla.redhat.com/show_bug.cgi?id=1058300 about VMs going
into a paused state and not being able to resume.

It's gone a little quiet, so I was wondering has anyone else had any
luck with this style of setup? I have my gluster and virt on the same
boxes, and currently 5 in the cluster (replica 2)

A few cases this can be reproduced:

- switch dies
- cable unplug
- gluster volume stop
- gluster brick dies

I recently tried a suggested volume option:
gluster volume set vm-data network.ping-timeout 10

The qemu logs now report on virsh resume vmname:
block I/O error in device 'drive-virtio-disk0': Transport endpoint is
not connected (107)

Thanks,
Andrew
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Brick on just one host constantly going offline

[Andrew Lau]

On Tue, Jun 3, 2014 at 11:35 PM, Justin Clift  wrote:
> On 03/06/2014, at 3:14 AM, Pranith Kumar Karampuri wrote:
>>> From: "Andrew Lau" 
>>> Sent: Tuesday, June 3, 2014 6:42:44 AM
> 
>>> Ah, that makes sense as it was the only volume which had that ping
>>> timeout setting. I also did see the timeout messages in the logs when
>>> I was checking. So is this merged in 3.5.1 ?
>>
>> Yes! http://review.gluster.org/7570
>
>
> Would you have time/inclination to test the 3.5.1 beta?
>
>   http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.1beta1/
>
> We're really needing people to test it and report back. :)
>

I'll take it for a spin, will it be an easy process to upgrade to the
GA release from beta? Contemplating whether it's worth living life on
the edge and deploying with beta :)

> Regards and best wishes,
>
> Justin Clift
>
> --
> Open Source and Standards @ Red Hat
>
> twitter.com/realjustinclift
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Brick on just one host constantly going offline

[Andrew Lau]

Hi Pranith,

On Tue, Jun 3, 2014 at 10:56 AM, Pranith Kumar Karampuri
 wrote:
>
>
> - Original Message -
>> From: "Andrew Lau" 
>> To: "gluster-users@gluster.org List" 
>> Sent: Tuesday, June 3, 2014 4:10:25 AM
>> Subject: [Gluster-users] Brick on just one host constantly going offline
>>
>> Hi,
>>
>> Just a short post as I've since nuked the test environment.
>>
>> I've had this case where in a 2 node gluster replica, the brick of the
>> first host is constantly going offline.
>>
>> gluster volume status
>>
>> would report host 1's brick is offline. The quorum would kick in,
>> putting the whole cluster into a read only state. This has only
>> recently been happening w/ gluster 3.5 and it normally happens after
>> about 3-4 days of 500GB or so data transfer.
>
> Could you check mount logs to see if there are ping timer expiry messages for 
> disconnects?
> If you see them, then it is very likely that you are hitting throttling 
> problem fixed by http://review.gluster.org/7531
>

Ah, that makes sense as it was the only volume which had that ping
timeout setting. I also did see the timeout messages in the logs when
I was checking. So is this merged in 3.5.1 ?

> Pranith
>
>>
>> Has anyone noticed this before? The only way to bring it back was to:
>>
>> killall glusterfsd ; killall -9 glusterfsd ; killall glusterd ; glusterd
>>
>>
>> Thanks,
>> Andrew
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Brick on just one host constantly going offline

[Andrew Lau]

Hi,

Just a short post as I've since nuked the test environment.

I've had this case where in a 2 node gluster replica, the brick of the
first host is constantly going offline.

gluster volume status

would report host 1's brick is offline. The quorum would kick in,
putting the whole cluster into a read only state. This has only
recently been happening w/ gluster 3.5 and it normally happens after
about 3-4 days of 500GB or so data transfer.

Has anyone noticed this before? The only way to bring it back was to:

killall glusterfsd ; killall -9 glusterfsd ; killall glusterd ; glusterd


Thanks,
Andrew
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] cgroup config for glusterfsd in gluster 3.5?

[Andrew Lau]

Hi,

I had recent comment from Giuseppe on my post about gluster cgroups [1],
and also I heard some interesting things at a recent meetup around recent
progressions with systemd along with gluster etc.

Comment:
" I've noted that latest GlusterFS 3.5.0 nightly packages do not include
(nor use) the /etc/sysconfig/glusterfsd file anymore.
Should we deduce that the glusterd hierarchy/settings now controls both? "

I haven't had the time to look into 3.5, so now does the glusterd process
control glusterfsd as well? Or would this cgroup method no longer work.

[1]
http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Optimizing Gluster (gfapi) for high IOPS

[Andrew Lau]

On Sun, Mar 23, 2014 at 6:10 AM, Josh Boon  wrote:

> Thanks for those options. My machines tend to be self-healing rather
> frequently.  Doing a gluster volume heal VMARRAY info the file list cycles
> through most of my high IOPS machines
>
> Also what's the best way to apply those options with out bricking the
> running VM's?  I just made a rough stab and took the cluster down.
>
>
The CPU problem thing sounds a lot like what I ran into with my ovirt on
gluster deployment (same boxes). What I did to solve that was use cgroups
to limit the CPU usage glusterd and glusterfsd is allowed to use. [1]

I'm not completely sure if libgfapi uses the glusterd process to access the
storage, could someone else comment? However, I know by limiting glusterfsd
we can slow down the replication process by limiting the CPU it sees, thus
not bringing the entire system to a halt.

[1]
http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/



 - Original Message -
> From: "Vijay Bellur" 
> To: "Josh Boon" , "Nick Majeran"  >
> Cc: "Gluster-users@gluster.org List" 
> Sent: Saturday, March 22, 2014 1:36:09 PM
> Subject: Re: [Gluster-users] Optimizing Gluster (gfapi) for high IOPS
>
> On 03/21/2014 09:50 PM, Josh Boon wrote:
> > Hardware RAID 5 on SSD's using LVM formatted with XFS default options
> > mounted with noatime
> >
> > Also I don't a lot of history for this current troubled machine but the
> > sysctl additions don't appear to have made a significant difference
>
> Performance tunables in [1] are normally recommended for qemu -
> libgfapi. The last two options are related to quorum and the remaining
> tunables are related to performance. It might be worth a check to see if
> these options help provide better performance.
>
> Do you happen to know if self-healing was in progress when the machines
> stall?
>
> -Vijay
>
> [1]
> https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Clarification on cluster quorum

[Andrew Lau]

lume to
>> participate in quorum.
>> cluster.server-quorum-type
>>
>> # If quorum-type is "fixed" only allow writes if this many
>> bricks or present. Other quorum types will OVERWRITE this value.
>> cluster.quorum-count
>>
>> # If value is "fixed" only allow writes if quorum-count bricks
>> are present. If value is "auto" only allow writes if more than half of
>> bricks, or exactly half including the first, are present.
>> cluster.quorum-type
>>
>> I took these from my previous "notes" (code) in:
>>
>> https://github.com/purpleidea/puppet-gluster/blob/master/manifests/volume/property/data.pp#L18
>>
>> You can get newer values or appropriate values for your version by
>> running something like:
>>
>> gluster volume set help ( i think )
>>
>> Cheers,
>> James
>>
>>
>> On Mon, Mar 10, 2014 at 2:43 AM, Andrew Lau 
>> wrote:
>> > Thanks again James!
>> >
>> > So far so good, I plan to test this a little more in a few days but so
>> far
>> > it seems the only volume setting I need is:
>> > cluster.server-quorum-type: server
>> >
>> > Default cluster.server-quorum-ratio >50%
>> > So 2 is greater than 1.5.. which should allow writes.
>> >
>> > On Thu, Mar 6, 2014 at 5:00 PM, James  wrote:
>> >>
>> >> Top posting sorry:
>> >>
>> >> Yes, you can add a third "arbiter" node, that exists to help with the
>> >> quorum issues. AFAICT, all you do is peer is with the cluster (as you
>> >> did with the other hosts) but don't add any storage for example.
>> >>
>> >> Then you set the cluster.quorum* style volume settings that you're
>> >> interested. I don't have a list of exactly which ones off the top of
>> >> my head, but if you make a list, let me know!
>> >>
>> >> Cheers,
>> >> James
>> >>
>> >>
>> >> On Wed, Mar 5, 2014 at 10:51 PM, Andrew Lau 
>> wrote:
>> >> > Hi,
>> >> >
>> >> > I'm looking for an option to add an arbiter node to the gluster
>> >> > cluster, but the leads I've been following seem to lead to
>> >> > inconclusive results.
>> >> >
>> >> > The scenario is, a 2 node replicated cluster. What I want to do is
>> >> > introduce a fake host/arbiter node which would set the cluster to a 3
>> >> > node meaning, we can meet the conditions of allow over 50% to write
>> >> > (ie. 2 can write, 1 can not).
>> >> >
>> >> > elyograg from IRC gave me a few links [1], [2]
>> >> > But these appear to be over a year old, and still under review.
>> >> >
>> >> > Gluster 3.2 volume options (I'm running 3.4, but there doesn't seem
>> to
>> >> > be an updated page) [3]
>> >> > seem to state the that the cluster quorum is identified by active
>> >> > peers. This also backs up the statement in [2] in regards to a patch
>> >> > for active volumes rather than cluster peers.
>> >> >
>> >> > Has anyone gone down this path, or could they confirm any of these
>> >> > leads? (ie. does a host w/o any volumes get considered as a peer
>> >> > within the cluster)
>> >> >
>> >> > Thanks,
>> >> > Andrew
>> >> >
>> >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=914804
>> >> > [2] http://review.gluster.org/#/c/4363/
>> >> > [3]
>> >> >
>> http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type
>> >> > ___
>> >> > Gluster-users mailing list
>> >> > Gluster-users@gluster.org
>> >> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Clarification on cluster quorum

[Andrew Lau]

Thanks again James!

So far so good, I plan to test this a little more in a few days but so far
it seems the only volume setting I need is:
cluster.server-quorum-type: server

Default cluster.server-quorum-ratio >50%
So 2 is greater than 1.5.. which should allow writes.

On Thu, Mar 6, 2014 at 5:00 PM, James  wrote:

> Top posting sorry:
>
> Yes, you can add a third "arbiter" node, that exists to help with the
> quorum issues. AFAICT, all you do is peer is with the cluster (as you
> did with the other hosts) but don't add any storage for example.
>
> Then you set the cluster.quorum* style volume settings that you're
> interested. I don't have a list of exactly which ones off the top of
> my head, but if you make a list, let me know!
>
> Cheers,
> James
>
>
> On Wed, Mar 5, 2014 at 10:51 PM, Andrew Lau  wrote:
> > Hi,
> >
> > I'm looking for an option to add an arbiter node to the gluster
> > cluster, but the leads I've been following seem to lead to
> > inconclusive results.
> >
> > The scenario is, a 2 node replicated cluster. What I want to do is
> > introduce a fake host/arbiter node which would set the cluster to a 3
> > node meaning, we can meet the conditions of allow over 50% to write
> > (ie. 2 can write, 1 can not).
> >
> > elyograg from IRC gave me a few links [1], [2]
> > But these appear to be over a year old, and still under review.
> >
> > Gluster 3.2 volume options (I'm running 3.4, but there doesn't seem to
> > be an updated page) [3]
> > seem to state the that the cluster quorum is identified by active
> > peers. This also backs up the statement in [2] in regards to a patch
> > for active volumes rather than cluster peers.
> >
> > Has anyone gone down this path, or could they confirm any of these
> > leads? (ie. does a host w/o any volumes get considered as a peer
> > within the cluster)
> >
> > Thanks,
> > Andrew
> >
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=914804
> > [2] http://review.gluster.org/#/c/4363/
> > [3]
> http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Clarification on cluster quorum

[Andrew Lau]

Hi,

I'm looking for an option to add an arbiter node to the gluster
cluster, but the leads I've been following seem to lead to
inconclusive results.

The scenario is, a 2 node replicated cluster. What I want to do is
introduce a fake host/arbiter node which would set the cluster to a 3
node meaning, we can meet the conditions of allow over 50% to write
(ie. 2 can write, 1 can not).

elyograg from IRC gave me a few links [1], [2]
But these appear to be over a year old, and still under review.

Gluster 3.2 volume options (I'm running 3.4, but there doesn't seem to
be an updated page) [3]
seem to state the that the cluster quorum is identified by active
peers. This also backs up the statement in [2] in regards to a patch
for active volumes rather than cluster peers.

Has anyone gone down this path, or could they confirm any of these
leads? (ie. does a host w/o any volumes get considered as a peer
within the cluster)

Thanks,
Andrew

[1] https://bugzilla.redhat.com/show_bug.cgi?id=914804
[2] http://review.gluster.org/#/c/4363/
[3] 
http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to put some sort of qos/rate-limit on glusterfsd?

[Andrew Lau]

On Tue, Feb 4, 2014 at 1:45 PM, James  wrote:

> On Mon, Feb 3, 2014 at 6:57 PM, Andrew Lau  wrote:
> > Hi all,
> >
> > Here's a quick write up I put together, it's very rough and may not be
> for
> > everyone but I hope this helps some people:
> >
> http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/
>
> Sweet... Thanks for the nod. Glad it worked out for you. I think this
> is only going to get cleaner with systemd/cgroups integration. I think
> systemd is going to be in RHEL7, but I have no idea :P
>

Yeah - systemd is in Fedora 19 which is base for RHEL7. Even then, after
getting a proper grasp of cgroups the sysconfig method felt very easy.


> >
> > Please leave your comments and feedback.
> >
> > Cheers,
> > Andrew.
> If anyone feels strongly about getting cgroups integration into
> puppet-gluster, let me know.


If you do, I'd love to hear about how the implementation goes and whether
you've had any better result with different cgroup settings. I only have a
very moderate understanding of how the gluster design works so I'm not 100%
cgroup solution may apply to all case scenarios.


>
> Cheers,
>
> James
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to put some sort of qos/rate-limit on glusterfsd?

[Andrew Lau]

Hi all,

Here's a quick write up I put together, it's very rough and may not be for
everyone but I hope this helps some people:
http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/

Please leave your comments and feedback.

Cheers,
Andrew.


On Tue, Feb 4, 2014 at 2:19 AM, John Mark Walker wrote:

> This sounds exciting! I'm looking forward to the writeup :)
>
> -JM
>
>
> --
>
> Quick update, I've successfully got glusterfs running with cgroups and I'm
> very impressed with the results.
>
> I'll write something up in the coming days - thanks again for the
> suggestion.
>
> On Mon, Feb 3, 2014 at 11:48 AM, Andrew Lau  wrote:
>
>> Thanks for the suggestions - I'm seeing some promising results with
>> cgroups.
>>
>> Just for confirmation - am I right in saying glusterd is just the
>> management daemon? and glusterfsd is the actual process which does the
>> checksums, replication, healing etc?
>>
>> On Mon, Feb 3, 2014 at 10:18 AM, Dan Mons wrote:
>>
>>> Try experimenting with performance.io-thread-count to see if that has an
>>> impact.
>>>
>>> -Dan
>>> 
>>> Dan Mons
>>> Skunk Works
>>> Cutting Edge
>>> http://cuttingedge.com.au
>>>
>>>
>>> On 2 February 2014 15:46, Andrew Lau  wrote:
>>> > Hi all,
>>> >
>>> > Sadly my google skills aren't finding me any results - is there an
>>> option to
>>> > limit the CPU usage and/or the disk IO intensity of glusterfsd.
>>> >
>>> > Example scenario, oVirt + gluster on the same host when it comes to
>>> adding
>>> > an extra host + replicated brick the original host with the brick goes
>>> crazy
>>> > with 500% cpu as it copies just under 1TB of data across to the new
>>> > replicated brick. Going crazy I mean everything else will hang, simple
>>> "ls"
>>> > command will take 30+ seconds.
>>> >
>>> > Limiting the network bandwidth to 200Mbps seems to solve this issue,
>>> I'm
>>> > quite sure this is a CPU issue rather than IO so I was wondering if
>>> there's
>>> > any possibility to limit this down so the nic's itself don't have to
>>> get
>>> > rate limited.
>>> >
>>> > Thanks,
>>> > Andrew
>>> >
>>> > ___
>>> > Gluster-users mailing list
>>> > Gluster-users@gluster.org
>>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster NFS server not starting on boot

[Andrew Lau]

On Mon, Feb 3, 2014 at 9:40 PM, Vijay Bellur  wrote:

> On 02/03/2014 04:03 PM, Andrew Lau wrote:
>
>> Hi,
>>
>> Is there a config to have the gluster NFS server start on boot?
>>
>> I'm having this weird issue where in a two brick replica (one offline
>> temporarily) the NFS server will show as Online: N. The NFS server will
>> only startup if I make a change to an option (such as change the cache
>> size)  or the second host comes up.
>>
>> The volume is however still mountable through glusterfs.
>>
>> Here are my configs:
>> performance.cache-size: 1GB
>> server.allow-insecure: on
>> performance.io-thread-count: 8
>> auth.allow: 172.16.*.*
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> nfs.disable: no
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> cluster.quorum-type: none
>> cluster.server-quorum-type: none
>>
>> nfs.register-with-portmap: on
>>
>> Any suggestions?
>>
>>
>
> Looking up the nfs log file in /var/log/glusterfs might provide some clue.
>
> -Vijay
>
>
I nulled all the log files and did a reboot. nfs.log remains empty but
etc-glusterfs-glusterd.vol.log has a couple errors about the other brick
not being online eg.
E [socket.c:2157:socket_connect_finish] 0-management: connection to
172.16.1.11:24007 failed (No route to host)
W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster NFS server not starting on boot

[Andrew Lau]

Hi,

Is there a config to have the gluster NFS server start on boot?

I'm having this weird issue where in a two brick replica (one offline
temporarily) the NFS server will show as Online: N. The NFS server will
only startup if I make a change to an option (such as change the cache
size)  or the second host comes up.

The volume is however still mountable through glusterfs.

Here are my configs:
performance.cache-size: 1GB
server.allow-insecure: on
performance.io-thread-count: 8
auth.allow: 172.16.*.*
storage.owner-uid: 36
storage.owner-gid: 36
nfs.disable: no
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: none
cluster.server-quorum-type: none
nfs.register-with-portmap: on

Any suggestions?

Thanks,
Andrew
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to put some sort of qos/rate-limit on glusterfsd?

[Andrew Lau]

Quick update, I've successfully got glusterfs running with cgroups and I'm
very impressed with the results.

I'll write something up in the coming days - thanks again for the
suggestion.

On Mon, Feb 3, 2014 at 11:48 AM, Andrew Lau  wrote:

> Thanks for the suggestions - I'm seeing some promising results with
> cgroups.
>
> Just for confirmation - am I right in saying glusterd is just the
> management daemon? and glusterfsd is the actual process which does the
> checksums, replication, healing etc?
>
> On Mon, Feb 3, 2014 at 10:18 AM, Dan Mons wrote:
>
>> Try experimenting with performance.io-thread-count to see if that has an
>> impact.
>>
>> -Dan
>> 
>> Dan Mons
>> Skunk Works
>> Cutting Edge
>> http://cuttingedge.com.au
>>
>>
>> On 2 February 2014 15:46, Andrew Lau  wrote:
>> > Hi all,
>> >
>> > Sadly my google skills aren't finding me any results - is there an
>> option to
>> > limit the CPU usage and/or the disk IO intensity of glusterfsd.
>> >
>> > Example scenario, oVirt + gluster on the same host when it comes to
>> adding
>> > an extra host + replicated brick the original host with the brick goes
>> crazy
>> > with 500% cpu as it copies just under 1TB of data across to the new
>> > replicated brick. Going crazy I mean everything else will hang, simple
>> "ls"
>> > command will take 30+ seconds.
>> >
>> > Limiting the network bandwidth to 200Mbps seems to solve this issue, I'm
>> > quite sure this is a CPU issue rather than IO so I was wondering if
>> there's
>> > any possibility to limit this down so the nic's itself don't have to get
>> > rate limited.
>> >
>> > Thanks,
>> > Andrew
>> >
>> > ___
>> > Gluster-users mailing list
>> > Gluster-users@gluster.org
>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to put some sort of qos/rate-limit on glusterfsd?

[Andrew Lau]

Thanks for the suggestions - I'm seeing some promising results with
cgroups.

Just for confirmation - am I right in saying glusterd is just the
management daemon? and glusterfsd is the actual process which does the
checksums, replication, healing etc?

On Mon, Feb 3, 2014 at 10:18 AM, Dan Mons  wrote:

> Try experimenting with performance.io-thread-count to see if that has an
> impact.
>
> -Dan
> 
> Dan Mons
> Skunk Works
> Cutting Edge
> http://cuttingedge.com.au
>
>
> On 2 February 2014 15:46, Andrew Lau  wrote:
> > Hi all,
> >
> > Sadly my google skills aren't finding me any results - is there an
> option to
> > limit the CPU usage and/or the disk IO intensity of glusterfsd.
> >
> > Example scenario, oVirt + gluster on the same host when it comes to
> adding
> > an extra host + replicated brick the original host with the brick goes
> crazy
> > with 500% cpu as it copies just under 1TB of data across to the new
> > replicated brick. Going crazy I mean everything else will hang, simple
> "ls"
> > command will take 30+ seconds.
> >
> > Limiting the network bandwidth to 200Mbps seems to solve this issue, I'm
> > quite sure this is a CPU issue rather than IO so I was wondering if
> there's
> > any possibility to limit this down so the nic's itself don't have to get
> > rate limited.
> >
> > Thanks,
> > Andrew
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible to put some sort of qos/rate-limit on glusterfsd?

[Andrew Lau]

On Sun, Feb 2, 2014 at 7:05 PM, James  wrote:

> On Sun, Feb 2, 2014 at 12:46 AM, Andrew Lau  wrote:
> > Hi all,
> Hi,
>
> >
> > Sadly my google skills aren't finding me any results - is there an
> option to
> > limit the CPU usage and/or the disk IO intensity of glusterfsd.
> This is what cgroups are for. Looking forward to you setting this up
> and telling us how it goes and what the optimized values are. AFAIK,
> gluster doesn't have anything built in to limit this. I actually don't
> think that it makes sense for it to either.
>

Thanks for the suggestions on cgroups, I actually did not think about that.
I'll try that and report my findings.



>
> >
> > Example scenario, oVirt + gluster on the same host when it comes to
> adding
> > an extra host + replicated brick the original host with the brick goes
> crazy
> > with 500% cpu as it copies just under 1TB of data across to the new
> > replicated brick. Going crazy I mean everything else will hang, simple
> "ls"
> > command will take 30+ seconds.
> Not sure what the correct solution is here. Maybe other gluster dev's
> can comment. Although it's really not detailed and you comment is
> unclear, so i'd elaborate before expecting a good response.
>
>
Sorry for the poor choice of wordings - here's attempt 2:

Host 1 (1TB brick w/ no replica)
Host 2 (NEW - 1TB brick to join Host 1 as replica 2)

Top will report glusterfsd with up to 500% CPU usage (combined processes).
Causing the system to grind to a halt very quickly.



>  >
> > Limiting the network bandwidth to 200Mbps seems to solve this issue, I'm
> > quite sure this is a CPU issue rather than IO so I was wondering if
> there's
> > any possibility to limit this down so the nic's itself don't have to get
> > rate limited.
> I occasionally use shorewall+puppet-gluster to rate limit the network
> interfaces. Puppet-Gluster could do per interface and per port/gluster
> connection throttling.
>

Rate limiting seems to be only solution right now (until I take a look into
cgoups). It seems the CPU cannot handle the high cpu usage that glusterfsd
will create.


>
> >
> > Thanks,
> > Andrew
>
> HTH,
> James
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Possible to put some sort of qos/rate-limit on glusterfsd?

[Andrew Lau]

Hi all,

Sadly my google skills aren't finding me any results - is there an option
to limit the CPU usage and/or the disk IO intensity of glusterfsd.

Example scenario, oVirt + gluster on the same host when it comes to adding
an extra host + replicated brick the original host with the brick goes
crazy with 500% cpu as it copies just under 1TB of data across to the new
replicated brick. Going crazy I mean everything else will hang, simple "ls"
command will take 30+ seconds.

Limiting the network bandwidth to 200Mbps seems to solve this issue, I'm
quite sure this is a CPU issue rather than IO so I was wondering if there's
any possibility to limit this down so the nic's itself don't have to get
rate limited.

Thanks,
Andrew
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster and kvm livemigration

[Andrew Lau]

Have you tried setting the uid/guid as part of the gluster volume? For
oVirt it uses 36:36 for virt

gluster volume set DATA storage.owner-uid 36
gluster volume set DATA storage.owner-gid 36

I'm assuming, setting this will enforce these ownership permissions on all
files.

On Sun, Jan 26, 2014 at 10:42 PM, Paul Boven  wrote:

> Hi Bernhard,
>
> Indeed I see the same behaviour:
> When a guest is running, it is owned by libvirt:kvm on both servers.
> When a guest is stopped, it is owned by root:root on both servers.
> In a failed migration, the ownership changes to root:root.
>
> I'm not convinced though that it is a simple unix permission problem,
> because after a failed migration, the guest.raw image is completely
> unreadable on the destination machine, even for root (permission denied),
> whereas I can still read it fine (e.g. dd or md5sum) on the originating
> server.
>
> Regards, Paul Boven.
>
>
> On 01/23/2014 08:10 PM, BGM wrote:
>
>> Hi Paul,
>> thnx, nice report,
>> u file(d) the bug?
>> can u do a
>> watch tree - pfungiA 
>> on both hosts
>> some vm running, some stopped.
>> start a machine
>> trigger the migration
>> at some point, the ownership of the vmimage.file flips from
>> libvirtd (running machnie) to root (normal permission, but only when
>> stopped).
>> If the ownership/permission flips that way,
>> libvirtd on the reciving side
>> can't write that file ...
>> does group/acl permission flip likewise?
>> Regards
>> Bernhard
>>
>> On 23.01.2014, at 16:49, Paul Boven  wrote:
>>
>>  Hi Bernhard,
>>>
>>> I'm having exactly the same problem on Ubuntu 13.04 with the 3.4.1
>>> packages from semiosis. It worked fine with glusterfs-3.4.0.
>>>
>>> We've been trying to debug this on the list, but haven't found the
>>> smoking gun yet.
>>>
>>> Please have a look at the URL below, and see if it matches what you are
>>> experiencing?
>>>
>>> http://epboven.home.xs4all.nl/gluster-migrate.html
>>>
>>> Regards, Paul Boven.
>>>
>>> On 01/23/2014 04:27 PM, Bernhard Glomm wrote:
>>>

 I had/have problems with live-migrating a virtual machine on a 2sided
 replica volume.

 I run ubuntu 13.04 and gluster 3.4.2 from semiosis


 with network.remote-dio to enable I can use "cache mode = none" as
 performance option for the virtual disks,

 so live migration works without "--unsafe"

 I'm triggering the migration now through the "Virtual Machine Manager"
 as an

 unprivileged user which is group member of libvirtd.


 After migration the disks become read-only because

 on migration the disk files changes ownership from

 libvirt-qemu to root


 What am I missing?


 TIA


 Bernhard



 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-users

>>>
>>>
>>> --
>>> Paul Boven  +31 (0)521-596547
>>> Unix/Linux/Networking specialist
>>> Joint Institute for VLBI in Europe - www.jive.nl
>>> VLBI - It's a fringe science
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>
> --
> Paul Boven  +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.nl
> VLBI - It's a fringe science
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster and kvm livemigration

[Andrew Lau]

Have you tried this?

Work Around until libvirtd fixes the port conflict (
http://review.gluster.org/#/c/6147/)

Modify /etc/glusterfs/glusterd.vol
option base-port 50152

service glusterd restart

On Sat, Jan 25, 2014 at 1:44 AM, Bernhard Glomm
wrote:

> I submitted
> Bug 1057645
> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>
> Bernhard
>
> On 24.01.2014 11:07:49, Bernhard Glomm wrote:
>
> samuli wrote:
>
> Can you try to set storage.owner-uid and storage.owner-gid to
> libvirt-qemu? To do that you have to stop volume.
>
> hi samuli, hi all
>
>
> I tried setting storage.owner-uid and storage.owner-gid to
>
> libvirt-qemu, as suggested, but with the same effect,
>
> during livemigration the ownership of the imagefile changes from
> libvirt-qemu/kvm to root/root
>
>
> root@pong[/5]:~ # gluster volume info glfs_atom01
>
>
>
> Volume Name: glfs_atom01
>
> Type: Replicate
>
> Volume ID: f28f0f62-37b3-4b10-8e86-9b373f4c0e75
>
> Status: Started
>
> Number of Bricks: 1 x 2 = 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: 172.24.1.11:/ecopool/fs_atom01
>
> Brick2: 172.24.1.13:/ecopool/fs_atom01
>
> Options Reconfigured:
>
> storage.owner-gid: 104
>
> storage.owner-uid: 107
>
> network.remote-dio: enable
>
>
> this is tree -pfungiA  : atom01 is running
>
>
> [-rw--- libvirt- kvm ]  /srv/vms/mnt_atom01/atom01.img
>
> [drwxr-xr-x libvirt- kvm ]  /srv/vms/mnt_atom02
>
> [-rw--- root root]  /srv/vms/mnt_atom02/atom02.img
>
> [drwxr-xr-x libvirt- kvm ]  /srv/vms/mnt_atom03
>
>
> Now I migrate through "VirtualMachineManager" and watching tree
>
> I see the permission changing to:
>
>
> [drwxr-xr-x libvirt- kvm ]  /srv/vms/mnt_atom01
>
> [-rw--- root root]  /srv/vms/mnt_atom01/atom01.img
>
> [drwxr-xr-x libvirt- kvm ]  /srv/vms/mnt_atom02
>
> [-rw--- root root]  /srv/vms/mnt_atom02/atom02.img
>
>
> From inside the atom01 (the VM) the filesystem becomes readonly.
>
> But in contrast to
> http://epboven.home.xs4all.nl/gluster-migrate.html
>
> I can still read all file, can checksum them, just no write access
>
> from outside the image file behaves as Paul described,
>
> as long as the machine is running I can't read the file
>
>
> root@pong[/5]:~ # virsh list
>
>  IdName   State
>
> 
>
>  6 atom01 running
>
>
> root@pong[/5]:~ # l /srv/vms/mnt_atom01/atom01.img
>
> -rw--- 1 root root 10G Jan 24 10:20 /srv/vms/mnt_atom01/atom01.img
>
> root@pong[/5]:~ # file /srv/vms/mnt_atom01/atom01.img
>
> /srv/vms/mnt_atom01/atom01.img: writable, regular file, no read permission
>
> root@pong[/5]:~ # md5sum /srv/vms/mnt_atom01/atom01.img
>
> md5sum: /srv/vms/mnt_atom01/atom01.img: Permission denied
>
> root@pong[/5]:~ # virsh destroy atom01
>
> Domain atom01 destroyed
>
>
> root@pong[/5]:~ # l /srv/vms/mnt_atom01/atom01.img
>
> -rw--- 1 root root 10G Jan 24 10:20 /srv/vms/mnt_atom01/atom01.img
>
> root@pong[/5]:~ # file /srv/vms/mnt_atom01/atom01.img
>
> /srv/vms/mnt_atom01/atom01.img: x86 boot sector; partition 1: ID=0x83,
> starthead 1, startsector 63, 16777165 sectors; partition 2: ID=0xf,
> starthead 254, startsector 16777228, 1677718 sectors, code offset 0x63
>
> root@pong[/5]:~ # md5sum /srv/vms/mnt_atom01/atom01.img
>
> 9d048558deb46fef7b24e8895711c554  /srv/vms/mnt_atom01/atom01.img
> root@pong[/5]:~ #
>
>
> But interestingly the source of the migration can access the file after
> migration completed
>
> like so: start atom01 on host "ping", migrate it to "pong"
>
>
> root@pong[/8]:~ # file /srv/vms/mnt_atom01/atom01.img
>
> /srv/vms/mnt_atom01/atom01.img: writable, regular file, no read permission
>
> root@ping[/5]:~ # file /srv/vms/mnt_atom01/atom01.img
>
> /srv/vms/mnt_atom01/atom01.img: x86 boot sector; partition 1: ID=0x83,
> starthead 1, startsector 63, 16777165 sectors; partition 2: ID=0xf,
> starthead 254, startsector 16777228, 1677718 sectors, code offset 0x63
>
> 100% reproducible
>
>
> Regards
>
>
> Bernhard
> ___
>
> Gluster-users mailing list
> Gluster-users@gluster.org <#143c4b5e56a77d0b_>
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
>   --
>   [image: *Ecologic Institute*]   *Bernhard Glomm*
> IT Administration
>
>Phone:  +49 (30) 86880 134   Fax:  +49 (30) 86880 100   Skype: 
> bernhard.glomm.ecologic [image:
> Website:]  [image: | 
> Video:] [image:
> | Newsletter:]  [image: |
> Facebook:]  [image: |
> Linkedin:] 
> [image:
> | Twitter:]  [image: | 
> YouTube:] [image:
> | Google+:] 

Re: [Gluster-users] Gluster infrastructure question

[Andrew Lau]

Hi Ben,

For glusterfs would you recommend the enterprise-storage
or throughput-performance tuned profile?

Thanks,
Andrew


On Tue, Dec 10, 2013 at 6:31 AM, Ben Turner  wrote:

> - Original Message -
> > From: "Ben Turner" 
> > To: "Heiko Krämer" 
> > Cc: "gluster-users@gluster.org List" 
> > Sent: Monday, December 9, 2013 2:26:45 PM
> > Subject: Re: [Gluster-users] Gluster infrastructure question
> >
> > - Original Message -
> > > From: "Heiko Krämer" 
> > > To: "gluster-users@gluster.org List" 
> > > Sent: Monday, December 9, 2013 8:18:28 AM
> > > Subject: [Gluster-users] Gluster infrastructure question
> > >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA1
> > >
> > > Heyho guys,
> > >
> > > I'm running since years glusterfs in a small environment without big
> > > problems.
> > >
> > > Now I'm going to use glusterFS for a bigger cluster but I've some
> > > questions :)
> > >
> > > Environment:
> > > * 4 Servers
> > > * 20 x 2TB HDD, each
> > > * Raidcontroller
> > > * Raid 10
> > > * 4x bricks => Replicated, Distributed volume
> > > * Gluster 3.4
> > >
> > > 1)
> > > I'm asking me, if I can delete the raid10 on each server and create
> > > for each HDD a separate brick.
> > > In this case have a volume 80 Bricks so 4 Server x 20 HDD's. Is there
> > > any experience about the write throughput in a production system with
> > > many of bricks like in this case? In addition i'll get double of HDD
> > > capacity.
> >
> > Have a look at:
> >
> > http://rhsummit.files.wordpress.com/2012/03/england-rhs-performance.pdf
>
> That one was from 2012, here is the latest:
>
>
> http://rhsummit.files.wordpress.com/2013/07/england_th_0450_rhs_perf_practices-4_neependra.pdf
>
> -b
>
> > Specifically:
> >
> > ● RAID arrays
> > ● More RAID LUNs for better concurrency
> > ● For RAID6, 256-KB stripe size
> >
> > I use a single RAID 6 that is divided into several LUNs for my bricks.
>  For
> > example, on my Dell servers(with PERC6 RAID controllers) each server has
> 12
> > disks that I put into raid 6.  Then I break the RAID 6 into 6 LUNs and
> > create a new PV/VG/LV for each brick.  From there I follow the
> > recommendations listed in the presentation.
> >
> > HTH!
> >
> > -b
> >
> > > 2)
> > > I've heard a talk about glusterFS and out scaling. The main point was
> > > if more bricks are in use, the scale out process will take a long
> > > time. The problem was/is the Hash-Algo. So I'm asking me how is it if
> > > I've one very big brick (Raid10 20TB on each server) or I've much more
> > > bricks, what's faster and is there any issues?
> > > Is there any experiences ?
> > >
> > > 3)
> > > Failover of a HDD is for a raid controller with HotSpare HDD not a big
> > > deal. Glusterfs will rebuild automatically if a brick fails and there
> > > are no data present, this action will perform a lot of network traffic
> > > between the mirror bricks but it will handle it equal as the raid
> > > controller right ?
> > >
> > >
> > >
> > > Thanks and cheers
> > > Heiko
> > >
> > >
> > >
> > > - --
> > > Anynines.com
> > >
> > > Avarteq GmbH
> > > B.Sc. Informatik
> > > Heiko Krämer
> > > CIO
> > > Twitter: @anynines
> > >
> > > - 
> > > Geschäftsführer: Alexander Faißt, Dipl.-Inf.(FH) Julian Fischer
> > > Handelsregister: AG Saarbrücken HRB 17413, Ust-IdNr.: DE262633168
> > > Sitz: Saarbrücken
> > > -BEGIN PGP SIGNATURE-
> > > Version: GnuPG v1.4.14 (GNU/Linux)
> > > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> > >
> > > iQEcBAEBAgAGBQJSpcMfAAoJELxFogM4ixOF/ncH/3L9DvOWHrF0XBqCgeT6QQ6B
> > > lDwtXiD9xoznht0Zs2S9LA9Z7r2l5/fzMOUSOawEMv6M16Guwq3gQ1lClUi4Iwj0
> > > GKKtYQ6F4aG4KXHY4dlu1QKT5OaLk8ljCQ47Tc9aAiJMhfC1/IgQXOslFv26utdJ
> > > N9jxiCl2+r/tQvQRw6mA4KAuPYPwOV+hMtkwfrM4UsIYGGbkNPnz1oqmBsfGdSOs
> > > TJh6+lQRD9KYw72q3I9G6ZYlI7ylL9Q7vjTroVKH232pLo4G58NLxyvWvcOB9yK6
> > > Bpf/gRMxFNKA75eW5EJYeZ6EovwcyCAv7iAm+xNKhzsoZqbBbTOJxS5zKm4YWoY=
> > > =bDly
> > > -END PGP SIGNATURE-
> > >
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster NFS Replicate bricks different size

[Andrew Lau]

I've mounted a gluster 1x2 replica through NFS in oVirt. The NFS share
holds the qcow images of the VMs.

I recently nuked a whole replica brick in an 1x2 array (for numerous other
reasons including split-brain), the brick self healed and restored back to
the same state as its partner.

4 days later, they've become inbalanced. The direct `du` of the /brick are
showing different sizes by around 20GB. I can see at brick level how some
images are not the same size. I don't think this is normal, but I can't see
anything to point what could be the issue.

gluster volume heal STORAGE info
gluster volume heal STORAGE info split-brain

Shows no issues.

Any suggestions?

Cheers,
Andrew.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Self-Heal Daemon not Running

[Andrew Lau]

On Wed, Sep 25, 2013 at 2:28 PM, Ravishankar N wrote:

>  On 09/25/2013 06:16 AM, Andrew Lau wrote:
>
>  That's where I found the 200+ entries
>
>  [ root@hv01 ]gluster volume heal STORAGE info split-brain
> Gathering Heal info on volume STORAGE has been successful
>
>  Brick hv01:/data1
> Number of entries: 271
> atpath on brick
>
>   2013-09-25 00:04:29 /6682d31f-39ce-4896-99ef-14e1c9682585/dom_md/ids
> 2013-09-25 00:04:29
> /6682d31f-39ce-4896-99ef-14e1c9682585/images/5599c7c7-0c25-459a-9d7d-80190a7c739b/0593d351-2ab1-49cd-a9b6-c94c897ebcc7
>  2013-09-24 23:54:29 
> 2013-09-24 23:54:29 
> 
>
>   Brick hv02:/data1
> Number of entries: 0
>
>  When I run the same command on hv02, it will show the reverse (the other
> node having 0 entries).
>
>  I remember last time having to delete these files individually on
> another split-brain case, but I was hoping there was a better solution than
> going through 200+ entries.
>
>   While I haven't tried it out myself, Jeff Darcy has written a script (
> https://github.com/jdarcy/glusterfs/tree/heal-script/extras/heal_script)
> which helps in automating the process. He has detailed it's usage in his
> blog post http://hekafs.org/index.php/2012/06/healing-split-brain/
>
> Hope this helps.
> -Ravi
>

That didn't end up working, ImportError: No module named volfilter

But I didn't end up spending much time with it as the number of entries
magically reduced to 10, I removed the files and the split-brain info
reports 0 entries. Still wondering why there's different file sizes on the
two bricks.

>
>   Cheers.
>
>
>  On Wed, Sep 25, 2013 at 10:39 AM, Mohit Anchlia 
> wrote:
>
>> What's the output of
>>
>>  gluster volume heal $VOLUME info split-brain
>>
>>
>>   On Tue, Sep 24, 2013 at 5:33 PM, Andrew Lau wrote:
>>
>>>   Found the BZ https://bugzilla.redhat.com/show_bug.cgi?id=960190 - so
>>> I restarted one of the volumes and it seems to have restarted the all
>>> daemons again.
>>>
>>>  Self heal started again, but I seem to have split-brain issues
>>> everywhere. There's over 100 different entries on each node, what's the
>>> best way to restore this now? Short of having to manually go through and
>>> delete 200+ files. It looks like a full split brain as the file sizes on
>>> the different nodes are out of balance by about 100GB or so.
>>>
>>>  Any suggestions would be much appreciated!
>>>
>>>  Cheers.
>>>
>>> On Tue, Sep 24, 2013 at 10:32 PM, Andrew Lau wrote:
>>>
>>>>  Hi,
>>>>
>>>>  Right now, I have a 2x1 replica. Ever since I had to reinstall one of
>>>> the gluster servers, there's been issues with split-brain. The self-heal
>>>> daemon doesn't seem to be running on either of the nodes.
>>>>
>>>>  To reinstall the gluster server (the original brick data was intact
>>>> but the OS had to be reinstalled)
>>>>  - Reinstalled gluster
>>>> - Copied over the old uuid from backup
>>>> - gluster peer probe
>>>> - gluster volume sync $othernode all
>>>> - mount -t glusterfs localhost:STORAGE /mnt
>>>> - find /mnt -noleaf -print0 | xargs --null stat >/dev/null
>>>> 2>/var/log/glusterfs/mnt-selfheal.log
>>>>
>>>>  I let it resync and it was working fine, atleast so I thought. I just
>>>> came back a few days later to see there's a miss match in the brick
>>>> volumes. One is 50GB ahead of the other.
>>>>
>>>>  # gluster volume heal STORAGE info
>>>> Status: self-heal-daemon is not running on
>>>> 966456a1-b8a6-4ca8-9da7-d0eb96997cbe
>>>>
>>>>  /var/log/gluster/glustershd.log doesn't seem to have any recent logs,
>>>> only those from when the two original gluster servers were running.
>>>>
>>>>  # gluster volume status
>>>>
>>>>  Self-heal Daemon on localhost N/A N N/A
>>>>
>>>>  Any suggestions would be much appreciated!
>>>>
>>>>  Cheers
>>>>  Andrew.
>>>>
>>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Self-Heal Daemon not Running

[Andrew Lau]

That's where I found the 200+ entries

[ root@hv01 ]gluster volume heal STORAGE info split-brain
Gathering Heal info on volume STORAGE has been successful

Brick hv01:/data1
Number of entries: 271
atpath on brick

2013-09-25 00:04:29 /6682d31f-39ce-4896-99ef-14e1c9682585/dom_md/ids
2013-09-25 00:04:29
/6682d31f-39ce-4896-99ef-14e1c9682585/images/5599c7c7-0c25-459a-9d7d-80190a7c739b/0593d351-2ab1-49cd-a9b6-c94c897ebcc7
2013-09-24 23:54:29 
2013-09-24 23:54:29 


Brick hv02:/data1
Number of entries: 0

When I run the same command on hv02, it will show the reverse (the other
node having 0 entries).

I remember last time having to delete these files individually on another
split-brain case, but I was hoping there was a better solution than going
through 200+ entries.

Cheers.


On Wed, Sep 25, 2013 at 10:39 AM, Mohit Anchlia wrote:

> What's the output of
>
> gluster volume heal $VOLUME info split-brain
>
>
> On Tue, Sep 24, 2013 at 5:33 PM, Andrew Lau  wrote:
>
>> Found the BZ https://bugzilla.redhat.com/show_bug.cgi?id=960190 - so I
>> restarted one of the volumes and it seems to have restarted the all daemons
>> again.
>>
>> Self heal started again, but I seem to have split-brain issues
>> everywhere. There's over 100 different entries on each node, what's the
>> best way to restore this now? Short of having to manually go through and
>> delete 200+ files. It looks like a full split brain as the file sizes on
>> the different nodes are out of balance by about 100GB or so.
>>
>> Any suggestions would be much appreciated!
>>
>> Cheers.
>>
>> On Tue, Sep 24, 2013 at 10:32 PM, Andrew Lau wrote:
>>
>>> Hi,
>>>
>>> Right now, I have a 2x1 replica. Ever since I had to reinstall one of
>>> the gluster servers, there's been issues with split-brain. The self-heal
>>> daemon doesn't seem to be running on either of the nodes.
>>>
>>> To reinstall the gluster server (the original brick data was intact but
>>> the OS had to be reinstalled)
>>> - Reinstalled gluster
>>> - Copied over the old uuid from backup
>>> - gluster peer probe
>>> - gluster volume sync $othernode all
>>> - mount -t glusterfs localhost:STORAGE /mnt
>>> - find /mnt -noleaf -print0 | xargs --null stat >/dev/null
>>> 2>/var/log/glusterfs/mnt-selfheal.log
>>>
>>> I let it resync and it was working fine, atleast so I thought. I just
>>> came back a few days later to see there's a miss match in the brick
>>> volumes. One is 50GB ahead of the other.
>>>
>>> # gluster volume heal STORAGE info
>>> Status: self-heal-daemon is not running on
>>> 966456a1-b8a6-4ca8-9da7-d0eb96997cbe
>>>
>>> /var/log/gluster/glustershd.log doesn't seem to have any recent logs,
>>> only those from when the two original gluster servers were running.
>>>
>>> # gluster volume status
>>>
>>> Self-heal Daemon on localhost N/A N N/A
>>>
>>> Any suggestions would be much appreciated!
>>>
>>> Cheers
>>> Andrew.
>>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Self-Heal Daemon not Running

[Andrew Lau]

Found the BZ https://bugzilla.redhat.com/show_bug.cgi?id=960190 - so I
restarted one of the volumes and it seems to have restarted the all daemons
again.

Self heal started again, but I seem to have split-brain issues everywhere.
There's over 100 different entries on each node, what's the best way to
restore this now? Short of having to manually go through and delete 200+
files. It looks like a full split brain as the file sizes on the different
nodes are out of balance by about 100GB or so.

Any suggestions would be much appreciated!

Cheers.

On Tue, Sep 24, 2013 at 10:32 PM, Andrew Lau  wrote:

> Hi,
>
> Right now, I have a 2x1 replica. Ever since I had to reinstall one of the
> gluster servers, there's been issues with split-brain. The self-heal daemon
> doesn't seem to be running on either of the nodes.
>
> To reinstall the gluster server (the original brick data was intact but
> the OS had to be reinstalled)
> - Reinstalled gluster
> - Copied over the old uuid from backup
> - gluster peer probe
> - gluster volume sync $othernode all
> - mount -t glusterfs localhost:STORAGE /mnt
> - find /mnt -noleaf -print0 | xargs --null stat >/dev/null
> 2>/var/log/glusterfs/mnt-selfheal.log
>
> I let it resync and it was working fine, atleast so I thought. I just came
> back a few days later to see there's a miss match in the brick volumes. One
> is 50GB ahead of the other.
>
> # gluster volume heal STORAGE info
> Status: self-heal-daemon is not running on
> 966456a1-b8a6-4ca8-9da7-d0eb96997cbe
>
> /var/log/gluster/glustershd.log doesn't seem to have any recent logs, only
> those from when the two original gluster servers were running.
>
> # gluster volume status
>
> Self-heal Daemon on localhost N/A N N/A
>
> Any suggestions would be much appreciated!
>
> Cheers
> Andrew.
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Self-Heal Daemon not Running

[Andrew Lau]

Hi,

Right now, I have a 2x1 replica. Ever since I had to reinstall one of the
gluster servers, there's been issues with split-brain. The self-heal daemon
doesn't seem to be running on either of the nodes.

To reinstall the gluster server (the original brick data was intact but the
OS had to be reinstalled)
- Reinstalled gluster
- Copied over the old uuid from backup
- gluster peer probe
- gluster volume sync $othernode all
- mount -t glusterfs localhost:STORAGE /mnt
- find /mnt -noleaf -print0 | xargs --null stat >/dev/null
2>/var/log/glusterfs/mnt-selfheal.log

I let it resync and it was working fine, atleast so I thought. I just came
back a few days later to see there's a miss match in the brick volumes. One
is 50GB ahead of the other.

# gluster volume heal STORAGE info
Status: self-heal-daemon is not running on
966456a1-b8a6-4ca8-9da7-d0eb96997cbe

/var/log/gluster/glustershd.log doesn't seem to have any recent logs, only
those from when the two original gluster servers were running.

# gluster volume status

Self-heal Daemon on localhost N/A N N/A

Any suggestions would be much appreciated!

Cheers
Andrew.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users