Re: [Gluster-users] Peer isolation while healing

[ML]

We have a lot of small files indeed.

I'll test the different values for cluster.data-self-heal-algorithm

Thanks!


Le 09/10/2017 à 15:38, lemonni...@ulrar.net a écrit :

On Mon, Oct 09, 2017 at 03:29:41PM +0200, ML wrote:

The server's load was huge during the healing (cpu at 100%), and the
disk latency increased a lot.

Depending on the file sizes, you might want to consider changing the
heal algortithm. Might be better to just re-download the whole file /
shard than to try and heal it, assuming you don't have big files. That
would free up the CPU


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Peer isolation while healing

[WK]

You have replica2 so you can't really take 50% of your cluster down 
without turning off quorum (and risking split brain). So detaching the 
rebuilding peer is really not an option.


If you had replica3 or an arbiter, you CAN detach or isolate the problem 
peer.  I've done things like change the Gluster network IP on the 'bad' 
peer to help speed up a RAID6 rebuild, that wasn't happy with the 
gluster heal process going on at the same time.


Your data will still be available and fully functional on the remaining 
peer (though you lost redundancy)


Then once the raid rebuild had caught up, you could return the peer to 
the cluster and do a final 'heal'.


-bill


On 10/9/2017 2:32 AM, ML wrote:


Hi everyone,

I've been using gluster for a few month now, on a simple 2 peers 
replicated infrastructure, 22Tb each.


One of the peers has been offline last week during 10 hours (raid 
resync after a disk crash), and while my gluster server was healing 
bricks, I could see some write errors on my gluster clients.


I couldn't find a way to isolate my healing peer, in the documentation 
or anywhere else.


Is there a way to avoid that ? Detach the peer while healing ? Some 
tunning on the client side maybe ?


I'm using gluster 3.9 on debian 8.

Thank you for your help.

Quentin

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Peer isolation while healing

[lemonnierk]

On Mon, Oct 09, 2017 at 03:29:41PM +0200, ML wrote:
> The server's load was huge during the healing (cpu at 100%), and the 
> disk latency increased a lot.

Depending on the file sizes, you might want to consider changing the
heal algortithm. Might be better to just re-download the whole file /
shard than to try and heal it, assuming you don't have big files. That
would free up the CPU


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Peer isolation while healing

[ML]

That make sense ^_^

Unfortunately I haven't kept the interresting data you need.

Basically I had some write errors on my gluster clients when my 
monitoring tool tested mkdir & create files.


The server's load was huge during the healing (cpu at 100%), and the 
disk latency increased a lot.
That may be the source of my write errors, we'll know for sure next 
time... I'll keep & post all the data you asked.


No way on the client side to force the gluster mount on 1 peer ?

Thanks for your help Karthik!

Quentin


Le 09/10/2017 à 12:10, Karthik Subrahmanya a écrit :

Hi,

There is no way to isolate the healing peer. Healing happens from the 
good brick to the bad brick.
I guess your replica bricks are on a different peers. If you try to 
isolate the healing peer, it will stop the healing process itself.


What is the error you are getting while writing? It would be helpful 
to debug the issue, if you can provide us the output of the following 
commands:

    gluster volume info 
    gluster volume heal  info
And also provide the client & heal logs.

Thanks & Regards,
Karthik

On Mon, Oct 9, 2017 at 3:02 PM, ML > wrote:


Hi everyone,

I've been using gluster for a few month now, on a simple 2 peers
replicated infrastructure, 22Tb each.

One of the peers has been offline last week during 10 hours (raid
resync after a disk crash), and while my gluster server was
healing bricks, I could see some write errors on my gluster clients.

I couldn't find a way to isolate my healing peer, in the
documentation or anywhere else.

Is there a way to avoid that ? Detach the peer while healing ?
Some tunning on the client side maybe ?

I'm using gluster 3.9 on debian 8.

Thank you for your help.

Quentin

___
Gluster-users mailing list
Gluster-users@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Peer isolation while healing

[Karthik Subrahmanya]

Hi,

There is no way to isolate the healing peer. Healing happens from the good
brick to the bad brick.
I guess your replica bricks are on a different peers. If you try to isolate
the healing peer, it will stop the healing process itself.

What is the error you are getting while writing? It would be helpful to
debug the issue, if you can provide us the output of the following commands:
gluster volume info 
gluster volume heal  info
And also provide the client & heal logs.

Thanks & Regards,
Karthik

On Mon, Oct 9, 2017 at 3:02 PM, ML  wrote:

> Hi everyone,
>
> I've been using gluster for a few month now, on a simple 2 peers
> replicated infrastructure, 22Tb each.
>
> One of the peers has been offline last week during 10 hours (raid resync
> after a disk crash), and while my gluster server was healing bricks, I
> could see some write errors on my gluster clients.
>
> I couldn't find a way to isolate my healing peer, in the documentation or
> anywhere else.
>
> Is there a way to avoid that ? Detach the peer while healing ? Some
> tunning on the client side maybe ?
>
> I'm using gluster 3.9 on debian 8.
>
> Thank you for your help.
>
> Quentin
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Peer isolation while healing

[ML]

Hi everyone,

I've been using gluster for a few month now, on a simple 2 peers 
replicated infrastructure, 22Tb each.


One of the peers has been offline last week during 10 hours (raid resync 
after a disk crash), and while my gluster server was healing bricks, I 
could see some write errors on my gluster clients.


I couldn't find a way to isolate my healing peer, in the documentation 
or anywhere else.


Is there a way to avoid that ? Detach the peer while healing ? Some 
tunning on the client side maybe ?


I'm using gluster 3.9 on debian 8.

Thank you for your help.

Quentin

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users