Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Pranith Kumar Karampuri]

On Fri, Apr 28, 2017 at 12:23 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2017-04-27 14:03 GMT+02:00 Pranith Kumar Karampuri :
> > The bugs are not in sharding. Sharding + VM workload is exposing bugs
> are in
> > DHT/rebalance. These bugs existed for years. They are coming to the fore
> > only now. It proves to be very difficult to recreate these bugs in other
> > environments. We are very transparent about this fact. Until these issues
> > are fixed please don't do rebalance in shard+VM environments.
>
> I apreciate this, but it's so critical that you should warn users in
> official docs, not only in mailing list.
> Not all users reads gluster ML, if someone is trying to use sharding
> (as wrote in docs) and then rebalance,
> will loose everything.
>
> Anyway , any plan to fix all of these in the upcoming release ?
>

First part is already fixed. I will update you once 2nd part is fixed as
well. If they find any more issues on the path, I will let you know about
that too.

-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

2017-04-27 14:03 GMT+02:00 Pranith Kumar Karampuri :
> The bugs are not in sharding. Sharding + VM workload is exposing bugs are in
> DHT/rebalance. These bugs existed for years. They are coming to the fore
> only now. It proves to be very difficult to recreate these bugs in other
> environments. We are very transparent about this fact. Until these issues
> are fixed please don't do rebalance in shard+VM environments.

I apreciate this, but it's so critical that you should warn users in
official docs, not only in mailing list.
Not all users reads gluster ML, if someone is trying to use sharding
(as wrote in docs) and then rebalance,
will loose everything.

Anyway , any plan to fix all of these in the upcoming release ?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Pranith Kumar Karampuri]

On Thu, Apr 27, 2017 at 5:21 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2017-04-27 13:31 GMT+02:00 Pranith Kumar Karampuri :
> > But even after that fix, it is still leading to pause. And these are the
> two
> > updates on what the developers are doing as per my understanding. So that
> > workflow is not stable yet IMO.
>
> So, even after that fix, two more critical bug leading to
> dataloss/corruption where found ?
>

yes


> I'm sorry but this is pure madness, plese put a "beta" label on
> sharding or gluster itself. 4 bugs in a row making loose of data in a
> software enginereed to keep data safe are unacceptable.
> Yes, all software has bug, but not with this frequency, for the same
> reason, with the same results: data loss.
>

The bugs are not in sharding. Sharding + VM workload is exposing bugs are
in DHT/rebalance. These bugs existed for years. They are coming to the fore
only now. It proves to be very difficult to recreate these bugs in other
environments. We are very transparent about this fact. Until these issues
are fixed please don't do rebalance in shard+VM environments.


>
> I'm really, really, really warried to put my company valuable data in
> a gluster storage. If I loose hundreds of VMs at once, i'm really
> fucked.
> Yes, backup exists, but try to say to your customers that you have to
> restore 20TB from backups (it takes days) and that they lost many
> ecommerce orders/transactions.
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

2017-04-27 13:31 GMT+02:00 Pranith Kumar Karampuri :
> But even after that fix, it is still leading to pause. And these are the two
> updates on what the developers are doing as per my understanding. So that
> workflow is not stable yet IMO.

So, even after that fix, two more critical bug leading to
dataloss/corruption where found ?

I'm sorry but this is pure madness, plese put a "beta" label on
sharding or gluster itself. 4 bugs in a row making loose of data in a
software enginereed to keep data safe are unacceptable.
Yes, all software has bug, but not with this frequency, for the same
reason, with the same results: data loss.

I'm really, really, really warried to put my company valuable data in
a gluster storage. If I loose hundreds of VMs at once, i'm really
fucked.
Yes, backup exists, but try to say to your customers that you have to
restore 20TB from backups (it takes days) and that they lost many
ecommerce orders/transactions.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

2017-04-27 13:21 GMT+02:00 Serkan Çoban :
> I think this is he fix Gandalf asking for:
> https://github.com/gluster/glusterfs/commit/6e3054b42f9aef1e35b493fbb002ec47e1ba27ce

Yes, i'm talking about this.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Pranith Kumar Karampuri]

;>> 4. The inode for "shard_file" is not present in itable after a
> >>>>> >>> graph
> >>>>> >>> switch and features/shard issues an mknod.
> >>>>> >>> 5. With new layout of .shard, lets say "shard_file" hashes to s3
> >>>>> >>> and
> >>>>> >>> mknod (shard_file) on s3 succeeds. But, the shard_file is already
> >>>>> >>> present on
> >>>>> >>> s2.
> >>>>> >>>
> >>>>> >>> So, we have two files on two different subvols of dht
> representing
> >>>>> >>> same
> >>>>> >>> shard and this will lead to corruption.
> >>>>> >>>
> >>>>> >>> 
> >>>>> >>>
> >>>>> >>> Raghavendra will be sending out a patch in DHT to fix this issue.
> >>>>> >>>
> >>>>> >>> -Krutika
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
> >>>>> >>>  wrote:
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan
> >>>>> >>>> 
> >>>>> >>>> wrote:
> >>>>> >>>>>
> >>>>> >>>>> Hi,
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>> Do you guys have any update regarding this issue ?
> >>>>> >>>>
> >>>>> >>>> I do not actively work on this issue so I do not have an
> accurate
> >>>>> >>>> update, but from what I heard from Krutika and Raghavendra(works
> >>>>> >>>> on DHT) is:
> >>>>> >>>> Krutika debugged initially and found that the issue seems more
> >>>>> >>>> likely to be
> >>>>> >>>> in DHT, Satheesaran who helped us recreate this issue in lab
> found
> >>>>> >>>> that just
> >>>>> >>>> fix-layout without rebalance also caused the corruption 1 out
> of 3
> >>>>> >>>> times.
> >>>>> >>>> Raghavendra came up with a possible RCA for why this can happen.
> >>>>> >>>> Raghavendra(CCed) would be the right person to provide accurate
> >>>>> >>>> update.
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>> --
> >>>>> >>>>>
> >>>>> >>>>> Respectfully
> >>>>> >>>>> Mahdi A. Mahdi
> >>>>> >>>>>
> >>>>> >>>>> 
> >>>>> >>>>> From: Krutika Dhananjay 
> >>>>> >>>>> Sent: Tuesday, March 21, 2017 3:02:55 PM
> >>>>> >>>>> To: Mahdi Adnan
> >>>>> >>>>> Cc: Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
> >>>>> >>>>> gluster-users@gluster.org List
> >>>>> >>>>>
> >>>>> >>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs
> >>>>> >>>>> corruption
> >>>>> >>>>>
> >>>>> >>>>> Hi,
> >>>>> >>>>>
> >>>>> >>>>> So it looks like Satheesaran managed to recreate this issue. We
> >>>>> >>>>> will be
> >>>>> >>>>> seeking his help in debugging this. It will be easier that way.
> >>>>> >>>>>
> >>>>> >>>>> -Krutika
> >>>>> >>>>>
> >>>>> >>>>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan
> >>>>> >>>>> 
> >>>>> >>>>> wrote:
> >>>>> >>>>>>
> >>>>> >

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Serkan Çoban]

> >>> same
>>>>> >>> shard and this will lead to corruption.
>>>>> >>>
>>>>> >>> 
>>>>> >>>
>>>>> >>> Raghavendra will be sending out a patch in DHT to fix this issue.
>>>>> >>>
>>>>> >>> -Krutika
>>>>> >>>
>>>>> >>>
>>>>> >>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
>>>>> >>>  wrote:
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan
>>>>> >>>> 
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi,
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> Do you guys have any update regarding this issue ?
>>>>> >>>>
>>>>> >>>> I do not actively work on this issue so I do not have an accurate
>>>>> >>>> update, but from what I heard from Krutika and Raghavendra(works
>>>>> >>>> on DHT) is:
>>>>> >>>> Krutika debugged initially and found that the issue seems more
>>>>> >>>> likely to be
>>>>> >>>> in DHT, Satheesaran who helped us recreate this issue in lab found
>>>>> >>>> that just
>>>>> >>>> fix-layout without rebalance also caused the corruption 1 out of 3
>>>>> >>>> times.
>>>>> >>>> Raghavendra came up with a possible RCA for why this can happen.
>>>>> >>>> Raghavendra(CCed) would be the right person to provide accurate
>>>>> >>>> update.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>>
>>>>> >>>>> Respectfully
>>>>> >>>>> Mahdi A. Mahdi
>>>>> >>>>>
>>>>> >>>>> 
>>>>> >>>>> From: Krutika Dhananjay 
>>>>> >>>>> Sent: Tuesday, March 21, 2017 3:02:55 PM
>>>>> >>>>> To: Mahdi Adnan
>>>>> >>>>> Cc: Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>>>>> >>>>> gluster-users@gluster.org List
>>>>> >>>>>
>>>>> >>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs
>>>>> >>>>> corruption
>>>>> >>>>>
>>>>> >>>>> Hi,
>>>>> >>>>>
>>>>> >>>>> So it looks like Satheesaran managed to recreate this issue. We
>>>>> >>>>> will be
>>>>> >>>>> seeking his help in debugging this. It will be easier that way.
>>>>> >>>>>
>>>>> >>>>> -Krutika
>>>>> >>>>>
>>>>> >>>>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan
>>>>> >>>>> 
>>>>> >>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Hello and thank you for your email.
>>>>> >>>>>> Actually no, i didn't check the gfid of the vms.
>>>>> >>>>>> If this will help, i can setup a new test cluster and get all
>>>>> >>>>>> the data
>>>>> >>>>>> you need.
>>>>> >>>>>>
>>>>> >>>>>> Get Outlook for Android
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> From: Nithya Balachandran
>>>>> >>>>>> Sent: Monday, March 20, 20:57
>>>>> >>>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs
>>>>> >>>>>> corruption
>>>>> >>>>>> To: Krutika Dhananjay
>>>>> >>>>>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>>>>> >>&g

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Pranith Kumar Karampuri]

I am very positive about the two things I told you. These are the latest
things that happened for VM corruption with rebalance.

On Thu, Apr 27, 2017 at 4:30 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> I think we are talking about a different bug.
>
> Il 27 apr 2017 12:58 PM, "Pranith Kumar Karampuri" 
> ha scritto:
>
>> I am not a DHT developer, so some of what I say could be a little wrong.
>> But this is what I gather.
>> I think they found 2 classes of bugs in dht
>> 1) Graceful fop failover when rebalance is in progress is missing for
>> some fops, that lead to VM pause.
>>
>> I see that https://review.gluster.org/17085 got merged on 24th on master
>> for this. I see patches are posted for 3.8.x for this one.
>>
>> 2) I think there is some work needs to be done for dht_[f]xattrop. I
>> believe this is the next step that is underway.
>>
>>
>> On Thu, Apr 27, 2017 at 12:13 PM, Gandalf Corvotempesta <
>> gandalf.corvotempe...@gmail.com> wrote:
>>
>>> Updates on this critical bug ?
>>>
>>> Il 18 apr 2017 8:24 PM, "Gandalf Corvotempesta" <
>>> gandalf.corvotempe...@gmail.com> ha scritto:
>>>
>>>> Any update ?
>>>> In addition, if this is a different bug but the "workflow" is the same
>>>> as the previous one, how is possible that fixing the previous bug
>>>> triggered this new one ?
>>>>
>>>> Is possible to have some details ?
>>>>
>>>> 2017-04-04 16:11 GMT+02:00 Krutika Dhananjay :
>>>> > Nope. This is a different bug.
>>>> >
>>>> > -Krutika
>>>> >
>>>> > On Mon, Apr 3, 2017 at 5:03 PM, Gandalf Corvotempesta
>>>> >  wrote:
>>>> >>
>>>> >> This is a good news
>>>> >> Is this related to the previously fixed bug?
>>>> >>
>>>> >> Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
>>>> >> scritto:
>>>> >>>
>>>> >>> So Raghavendra has an RCA for this issue.
>>>> >>>
>>>> >>> Copy-pasting his comment here:
>>>> >>>
>>>> >>> 
>>>> >>>
>>>> >>> Following is a rough algorithm of shard_writev:
>>>> >>>
>>>> >>> 1. Based on the offset, calculate the shards touched by current
>>>> write.
>>>> >>> 2. Look for inodes corresponding to these shard files in itable.
>>>> >>> 3. If one or more inodes are missing from itable, issue mknod for
>>>> >>> corresponding shard files and ignore EEXIST in cbk.
>>>> >>> 4. resume writes on respective shards.
>>>> >>>
>>>> >>> Now, imagine a write which falls to an existing "shard_file". For
>>>> the
>>>> >>> sake of discussion lets consider a distribute of three subvols -
>>>> s1, s2, s3
>>>> >>>
>>>> >>> 1. "shard_file" hashes to subvolume s2 and is present on s2
>>>> >>> 2. add a subvolume s4 and initiate a fix layout. The layout of
>>>> ".shard"
>>>> >>> is fixed to include s4 and hash ranges are changed.
>>>> >>> 3. write that touches "shard_file" is issued.
>>>> >>> 4. The inode for "shard_file" is not present in itable after a graph
>>>> >>> switch and features/shard issues an mknod.
>>>> >>> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and
>>>> >>> mknod (shard_file) on s3 succeeds. But, the shard_file is already
>>>> present on
>>>> >>> s2.
>>>> >>>
>>>> >>> So, we have two files on two different subvols of dht representing
>>>> same
>>>> >>> shard and this will lead to corruption.
>>>> >>>
>>>> >>> 
>>>> >>>
>>>> >>> Raghavendra will be sending out a patch in DHT to fix this issue.
>>>> >>>
>>>> >>> -Krutika
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
>>>> >>>  wrote:
>>>> >>>>
>>>> >>>&

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

I think we are talking about a different bug.

Il 27 apr 2017 12:58 PM, "Pranith Kumar Karampuri"  ha
scritto:

> I am not a DHT developer, so some of what I say could be a little wrong.
> But this is what I gather.
> I think they found 2 classes of bugs in dht
> 1) Graceful fop failover when rebalance is in progress is missing for some
> fops, that lead to VM pause.
>
> I see that https://review.gluster.org/17085 got merged on 24th on master
> for this. I see patches are posted for 3.8.x for this one.
>
> 2) I think there is some work needs to be done for dht_[f]xattrop. I
> believe this is the next step that is underway.
>
>
> On Thu, Apr 27, 2017 at 12:13 PM, Gandalf Corvotempesta <
> gandalf.corvotempe...@gmail.com> wrote:
>
>> Updates on this critical bug ?
>>
>> Il 18 apr 2017 8:24 PM, "Gandalf Corvotempesta" <
>> gandalf.corvotempe...@gmail.com> ha scritto:
>>
>>> Any update ?
>>> In addition, if this is a different bug but the "workflow" is the same
>>> as the previous one, how is possible that fixing the previous bug
>>> triggered this new one ?
>>>
>>> Is possible to have some details ?
>>>
>>> 2017-04-04 16:11 GMT+02:00 Krutika Dhananjay :
>>> > Nope. This is a different bug.
>>> >
>>> > -Krutika
>>> >
>>> > On Mon, Apr 3, 2017 at 5:03 PM, Gandalf Corvotempesta
>>> >  wrote:
>>> >>
>>> >> This is a good news
>>> >> Is this related to the previously fixed bug?
>>> >>
>>> >> Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
>>> >> scritto:
>>> >>>
>>> >>> So Raghavendra has an RCA for this issue.
>>> >>>
>>> >>> Copy-pasting his comment here:
>>> >>>
>>> >>> 
>>> >>>
>>> >>> Following is a rough algorithm of shard_writev:
>>> >>>
>>> >>> 1. Based on the offset, calculate the shards touched by current
>>> write.
>>> >>> 2. Look for inodes corresponding to these shard files in itable.
>>> >>> 3. If one or more inodes are missing from itable, issue mknod for
>>> >>> corresponding shard files and ignore EEXIST in cbk.
>>> >>> 4. resume writes on respective shards.
>>> >>>
>>> >>> Now, imagine a write which falls to an existing "shard_file". For the
>>> >>> sake of discussion lets consider a distribute of three subvols - s1,
>>> s2, s3
>>> >>>
>>> >>> 1. "shard_file" hashes to subvolume s2 and is present on s2
>>> >>> 2. add a subvolume s4 and initiate a fix layout. The layout of
>>> ".shard"
>>> >>> is fixed to include s4 and hash ranges are changed.
>>> >>> 3. write that touches "shard_file" is issued.
>>> >>> 4. The inode for "shard_file" is not present in itable after a graph
>>> >>> switch and features/shard issues an mknod.
>>> >>> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and
>>> >>> mknod (shard_file) on s3 succeeds. But, the shard_file is already
>>> present on
>>> >>> s2.
>>> >>>
>>> >>> So, we have two files on two different subvols of dht representing
>>> same
>>> >>> shard and this will lead to corruption.
>>> >>>
>>> >>> 
>>> >>>
>>> >>> Raghavendra will be sending out a patch in DHT to fix this issue.
>>> >>>
>>> >>> -Krutika
>>> >>>
>>> >>>
>>> >>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
>>> >>>  wrote:
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan <
>>> mahdi.ad...@outlook.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>>
>>> >>>>> Do you guys have any update regarding this issue ?
>>> >>>>
>>> >>>> I do not actively work on this issue so I do not have an accurate
>>> >>>> update, but from what I heard from Krutika an

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Pranith Kumar Karampuri]

I am not a DHT developer, so some of what I say could be a little wrong.
But this is what I gather.
I think they found 2 classes of bugs in dht
1) Graceful fop failover when rebalance is in progress is missing for some
fops, that lead to VM pause.

I see that https://review.gluster.org/17085 got merged on 24th on master
for this. I see patches are posted for 3.8.x for this one.

2) I think there is some work needs to be done for dht_[f]xattrop. I
believe this is the next step that is underway.


On Thu, Apr 27, 2017 at 12:13 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Updates on this critical bug ?
>
> Il 18 apr 2017 8:24 PM, "Gandalf Corvotempesta" <
> gandalf.corvotempe...@gmail.com> ha scritto:
>
>> Any update ?
>> In addition, if this is a different bug but the "workflow" is the same
>> as the previous one, how is possible that fixing the previous bug
>> triggered this new one ?
>>
>> Is possible to have some details ?
>>
>> 2017-04-04 16:11 GMT+02:00 Krutika Dhananjay :
>> > Nope. This is a different bug.
>> >
>> > -Krutika
>> >
>> > On Mon, Apr 3, 2017 at 5:03 PM, Gandalf Corvotempesta
>> >  wrote:
>> >>
>> >> This is a good news
>> >> Is this related to the previously fixed bug?
>> >>
>> >> Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
>> >> scritto:
>> >>>
>> >>> So Raghavendra has an RCA for this issue.
>> >>>
>> >>> Copy-pasting his comment here:
>> >>>
>> >>> 
>> >>>
>> >>> Following is a rough algorithm of shard_writev:
>> >>>
>> >>> 1. Based on the offset, calculate the shards touched by current write.
>> >>> 2. Look for inodes corresponding to these shard files in itable.
>> >>> 3. If one or more inodes are missing from itable, issue mknod for
>> >>> corresponding shard files and ignore EEXIST in cbk.
>> >>> 4. resume writes on respective shards.
>> >>>
>> >>> Now, imagine a write which falls to an existing "shard_file". For the
>> >>> sake of discussion lets consider a distribute of three subvols - s1,
>> s2, s3
>> >>>
>> >>> 1. "shard_file" hashes to subvolume s2 and is present on s2
>> >>> 2. add a subvolume s4 and initiate a fix layout. The layout of
>> ".shard"
>> >>> is fixed to include s4 and hash ranges are changed.
>> >>> 3. write that touches "shard_file" is issued.
>> >>> 4. The inode for "shard_file" is not present in itable after a graph
>> >>> switch and features/shard issues an mknod.
>> >>> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and
>> >>> mknod (shard_file) on s3 succeeds. But, the shard_file is already
>> present on
>> >>> s2.
>> >>>
>> >>> So, we have two files on two different subvols of dht representing
>> same
>> >>> shard and this will lead to corruption.
>> >>>
>> >>> 
>> >>>
>> >>> Raghavendra will be sending out a patch in DHT to fix this issue.
>> >>>
>> >>> -Krutika
>> >>>
>> >>>
>> >>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
>> >>>  wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan <
>> mahdi.ad...@outlook.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>>
>> >>>>> Do you guys have any update regarding this issue ?
>> >>>>
>> >>>> I do not actively work on this issue so I do not have an accurate
>> >>>> update, but from what I heard from Krutika and Raghavendra(works on
>> DHT) is:
>> >>>> Krutika debugged initially and found that the issue seems more
>> likely to be
>> >>>> in DHT, Satheesaran who helped us recreate this issue in lab found
>> that just
>> >>>> fix-layout without rebalance also caused the corruption 1 out of 3
>> times.
>> >>>> Raghavendra came up with a possible RCA for why this can happen.
>> >>>> Raghavendra(CCed) would be the right person to provide accu

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

Updates on this critical bug ?

Il 18 apr 2017 8:24 PM, "Gandalf Corvotempesta" <
gandalf.corvotempe...@gmail.com> ha scritto:

> Any update ?
> In addition, if this is a different bug but the "workflow" is the same
> as the previous one, how is possible that fixing the previous bug
> triggered this new one ?
>
> Is possible to have some details ?
>
> 2017-04-04 16:11 GMT+02:00 Krutika Dhananjay :
> > Nope. This is a different bug.
> >
> > -Krutika
> >
> > On Mon, Apr 3, 2017 at 5:03 PM, Gandalf Corvotempesta
> >  wrote:
> >>
> >> This is a good news
> >> Is this related to the previously fixed bug?
> >>
> >> Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
> >> scritto:
> >>>
> >>> So Raghavendra has an RCA for this issue.
> >>>
> >>> Copy-pasting his comment here:
> >>>
> >>> 
> >>>
> >>> Following is a rough algorithm of shard_writev:
> >>>
> >>> 1. Based on the offset, calculate the shards touched by current write.
> >>> 2. Look for inodes corresponding to these shard files in itable.
> >>> 3. If one or more inodes are missing from itable, issue mknod for
> >>> corresponding shard files and ignore EEXIST in cbk.
> >>> 4. resume writes on respective shards.
> >>>
> >>> Now, imagine a write which falls to an existing "shard_file". For the
> >>> sake of discussion lets consider a distribute of three subvols - s1,
> s2, s3
> >>>
> >>> 1. "shard_file" hashes to subvolume s2 and is present on s2
> >>> 2. add a subvolume s4 and initiate a fix layout. The layout of ".shard"
> >>> is fixed to include s4 and hash ranges are changed.
> >>> 3. write that touches "shard_file" is issued.
> >>> 4. The inode for "shard_file" is not present in itable after a graph
> >>> switch and features/shard issues an mknod.
> >>> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and
> >>> mknod (shard_file) on s3 succeeds. But, the shard_file is already
> present on
> >>> s2.
> >>>
> >>> So, we have two files on two different subvols of dht representing same
> >>> shard and this will lead to corruption.
> >>>
> >>> 
> >>>
> >>> Raghavendra will be sending out a patch in DHT to fix this issue.
> >>>
> >>> -Krutika
> >>>
> >>>
> >>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
> >>>  wrote:
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan <
> mahdi.ad...@outlook.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>>
> >>>>> Do you guys have any update regarding this issue ?
> >>>>
> >>>> I do not actively work on this issue so I do not have an accurate
> >>>> update, but from what I heard from Krutika and Raghavendra(works on
> DHT) is:
> >>>> Krutika debugged initially and found that the issue seems more likely
> to be
> >>>> in DHT, Satheesaran who helped us recreate this issue in lab found
> that just
> >>>> fix-layout without rebalance also caused the corruption 1 out of 3
> times.
> >>>> Raghavendra came up with a possible RCA for why this can happen.
> >>>> Raghavendra(CCed) would be the right person to provide accurate
> update.
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Respectfully
> >>>>> Mahdi A. Mahdi
> >>>>>
> >>>>> 
> >>>>> From: Krutika Dhananjay 
> >>>>> Sent: Tuesday, March 21, 2017 3:02:55 PM
> >>>>> To: Mahdi Adnan
> >>>>> Cc: Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
> >>>>> gluster-users@gluster.org List
> >>>>>
> >>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> So it looks like Satheesaran managed to recreate this issue. We will
> be
> >>>>> seeking his help in debugging this. It will

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

Any update ?
In addition, if this is a different bug but the "workflow" is the same
as the previous one, how is possible that fixing the previous bug
triggered this new one ?

Is possible to have some details ?

2017-04-04 16:11 GMT+02:00 Krutika Dhananjay :
> Nope. This is a different bug.
>
> -Krutika
>
> On Mon, Apr 3, 2017 at 5:03 PM, Gandalf Corvotempesta
>  wrote:
>>
>> This is a good news
>> Is this related to the previously fixed bug?
>>
>> Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
>> scritto:
>>>
>>> So Raghavendra has an RCA for this issue.
>>>
>>> Copy-pasting his comment here:
>>>
>>> 
>>>
>>> Following is a rough algorithm of shard_writev:
>>>
>>> 1. Based on the offset, calculate the shards touched by current write.
>>> 2. Look for inodes corresponding to these shard files in itable.
>>> 3. If one or more inodes are missing from itable, issue mknod for
>>> corresponding shard files and ignore EEXIST in cbk.
>>> 4. resume writes on respective shards.
>>>
>>> Now, imagine a write which falls to an existing "shard_file". For the
>>> sake of discussion lets consider a distribute of three subvols - s1, s2, s3
>>>
>>> 1. "shard_file" hashes to subvolume s2 and is present on s2
>>> 2. add a subvolume s4 and initiate a fix layout. The layout of ".shard"
>>> is fixed to include s4 and hash ranges are changed.
>>> 3. write that touches "shard_file" is issued.
>>> 4. The inode for "shard_file" is not present in itable after a graph
>>> switch and features/shard issues an mknod.
>>> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and
>>> mknod (shard_file) on s3 succeeds. But, the shard_file is already present on
>>> s2.
>>>
>>> So, we have two files on two different subvols of dht representing same
>>> shard and this will lead to corruption.
>>>
>>> 
>>>
>>> Raghavendra will be sending out a patch in DHT to fix this issue.
>>>
>>> -Krutika
>>>
>>>
>>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri
>>>  wrote:
>>>>
>>>>
>>>>
>>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> Do you guys have any update regarding this issue ?
>>>>
>>>> I do not actively work on this issue so I do not have an accurate
>>>> update, but from what I heard from Krutika and Raghavendra(works on DHT) 
>>>> is:
>>>> Krutika debugged initially and found that the issue seems more likely to be
>>>> in DHT, Satheesaran who helped us recreate this issue in lab found that 
>>>> just
>>>> fix-layout without rebalance also caused the corruption 1 out of 3 times.
>>>> Raghavendra came up with a possible RCA for why this can happen.
>>>> Raghavendra(CCed) would be the right person to provide accurate update.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Respectfully
>>>>> Mahdi A. Mahdi
>>>>>
>>>>> 
>>>>> From: Krutika Dhananjay 
>>>>> Sent: Tuesday, March 21, 2017 3:02:55 PM
>>>>> To: Mahdi Adnan
>>>>> Cc: Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>>>>> gluster-users@gluster.org List
>>>>>
>>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>>>
>>>>> Hi,
>>>>>
>>>>> So it looks like Satheesaran managed to recreate this issue. We will be
>>>>> seeking his help in debugging this. It will be easier that way.
>>>>>
>>>>> -Krutika
>>>>>
>>>>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
>>>>> wrote:
>>>>>>
>>>>>> Hello and thank you for your email.
>>>>>> Actually no, i didn't check the gfid of the vms.
>>>>>> If this will help, i can setup a new test cluster and get all the data
>>>>>> you need.
>>>>>>
>>>>>> Get Outlook for Android
>>>>>>
>>>>>>
>>>>>> From: Nithya Balachandran
>>>>>

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

Nope. This is a different bug.

-Krutika

On Mon, Apr 3, 2017 at 5:03 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> This is a good news
> Is this related to the previously fixed bug?
>
> Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
> scritto:
>
>> So Raghavendra has an RCA for this issue.
>>
>> Copy-pasting his comment here:
>>
>> 
>>
>> Following is a rough algorithm of shard_writev:
>>
>> 1. Based on the offset, calculate the shards touched by current write.
>> 2. Look for inodes corresponding to these shard files in itable.
>> 3. If one or more inodes are missing from itable, issue mknod for 
>> corresponding shard files and ignore EEXIST in cbk.
>> 4. resume writes on respective shards.
>>
>> Now, imagine a write which falls to an existing "shard_file". For the sake 
>> of discussion lets consider a distribute of three subvols - s1, s2, s3
>>
>> 1. "shard_file" hashes to subvolume s2 and is present on s2
>> 2. add a subvolume s4 and initiate a fix layout. The layout of ".shard" is 
>> fixed to include s4 and hash ranges are changed.
>> 3. write that touches "shard_file" is issued.
>> 4. The inode for "shard_file" is not present in itable after a graph switch 
>> and features/shard issues an mknod.
>> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and mknod 
>> (shard_file) on s3 succeeds. But, the shard_file is already present on s2.
>>
>> So, we have two files on two different subvols of dht representing same 
>> shard and this will lead to corruption.
>>
>> 
>>
>> Raghavendra will be sending out a patch in DHT to fix this issue.
>>
>> -Krutika
>>
>>
>> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> Do you guys have any update regarding this issue ?
>>>>
>>> I do not actively work on this issue so I do not have an accurate
>>> update, but from what I heard from Krutika and Raghavendra(works on DHT)
>>> is: Krutika debugged initially and found that the issue seems more likely
>>> to be in DHT, Satheesaran who helped us recreate this issue in lab found
>>> that just fix-layout without rebalance also caused the corruption 1 out of
>>> 3 times. Raghavendra came up with a possible RCA for why this can happen.
>>> Raghavendra(CCed) would be the right person to provide accurate update.
>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Respectfully
>>>> *Mahdi A. Mahdi*
>>>>
>>>> --
>>>> *From:* Krutika Dhananjay 
>>>> *Sent:* Tuesday, March 21, 2017 3:02:55 PM
>>>> *To:* Mahdi Adnan
>>>> *Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>>>> gluster-users@gluster.org List
>>>>
>>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>>
>>>> Hi,
>>>>
>>>> So it looks like Satheesaran managed to recreate this issue. We will be
>>>> seeking his help in debugging this. It will be easier that way.
>>>>
>>>> -Krutika
>>>>
>>>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
>>>> wrote:
>>>>
>>>>> Hello and thank you for your email.
>>>>> Actually no, i didn't check the gfid of the vms.
>>>>> If this will help, i can setup a new test cluster and get all the data
>>>>> you need.
>>>>>
>>>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>>>
>>>>> From: Nithya Balachandran
>>>>> Sent: Monday, March 20, 20:57
>>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>>> To: Krutika Dhananjay
>>>>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>>>>> gluster-users@gluster.org List
>>>>>
>>>>> Hi,
>>>>>
>>>>> Do you know the GFIDs of the VM images which were corrupted?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Nithya
>>>>>
>>>>> On 20 March 2017 at 20:37, Krutika Dhananjay 
>>>>> wrote:
>&

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Good to hear.
Eagerly waiting for the patch.

Thank you guys.

Get Outlook for Android<https://aka.ms/ghei36>



From: Krutika Dhananjay 
Sent: Monday, April 3, 2017 11:22:40 AM
To: Pranith Kumar Karampuri
Cc: Mahdi Adnan; gluster-users@gluster.org List; Gowdappa, Raghavendra
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

So Raghavendra has an RCA for this issue.

Copy-pasting his comment here:



Following is a rough algorithm of shard_writev:

1. Based on the offset, calculate the shards touched by current write.
2. Look for inodes corresponding to these shard files in itable.
3. If one or more inodes are missing from itable, issue mknod for corresponding 
shard files and ignore EEXIST in cbk.
4. resume writes on respective shards.

Now, imagine a write which falls to an existing "shard_file". For the sake of 
discussion lets consider a distribute of three subvols - s1, s2, s3

1. "shard_file" hashes to subvolume s2 and is present on s2
2. add a subvolume s4 and initiate a fix layout. The layout of ".shard" is 
fixed to include s4 and hash ranges are changed.
3. write that touches "shard_file" is issued.
4. The inode for "shard_file" is not present in itable after a graph switch and 
features/shard issues an mknod.
5. With new layout of .shard, lets say "shard_file" hashes to s3 and mknod 
(shard_file) on s3 succeeds. But, the shard_file is already present on s2.

So, we have two files on two different subvols of dht representing same shard 
and this will lead to corruption.




Raghavendra will be sending out a patch in DHT to fix this issue.

-Krutika


On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri 
mailto:pkara...@redhat.com>> wrote:


On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hi,


Do you guys have any update regarding this issue ?

I do not actively work on this issue so I do not have an accurate update, but 
from what I heard from Krutika and Raghavendra(works on DHT) is: Krutika 
debugged initially and found that the issue seems more likely to be in DHT, 
Satheesaran who helped us recreate this issue in lab found that just fix-layout 
without rebalance also caused the corruption 1 out of 3 times. Raghavendra came 
up with a possible RCA for why this can happen. Raghavendra(CCed) would be the 
right person to provide accurate update.


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Tuesday, March 21, 2017 3:02:55 PM
To: Mahdi Adnan
Cc: Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai; 
gluster-users@gluster.org<mailto:gluster-users@gluster.org> List

Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

Hi,

So it looks like Satheesaran managed to recreate this issue. We will be seeking 
his help in debugging this. It will be easier that way.

-Krutika

On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hello and thank you for your email.
Actually no, i didn't check the gfid of the vms.
If this will help, i can setup a new test cluster and get all the data you need.

Get Outlook for Android<https://aka.ms/ghei36>


From: Nithya Balachandran
Sent: Monday, March 20, 20:57
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
To: Krutika Dhananjay
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai, 
gluster-users@gluster.org<mailto:gluster-users@gluster.org> List

Hi,

Do you know the GFIDs of the VM images which were corrupted?

Regards,

Nithya

On 20 March 2017 at 20:37, Krutika Dhananjay 
mailto:kdhan...@redhat.com>> wrote:

I looked at the logs.

>From the time the new graph (since the add-brick command you shared where 
>bricks 41 through 44 are added) is switched to (line 3011 onwards in 
>nfs-gfapi.log), I see the following kinds of errors:

1. Lookups to a bunch of files failed with ENOENT on both replicas which 
protocol/client converts to ESTALE. I am guessing these entries got migrated to

other subvolumes leading to 'No such file or directory' errors.

DHT and thereafter shard get the same error code and log the following:

 0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] 
[dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht: 
: failed to lookup the file on 
vmware2-dht [Stale file handle]
  1 [2017-03-17 14:04:26.353528] E [MSGID: 133014] 
[shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed: 
a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]

which is fine.

2. The other kind are from AFR logging of possible split-brain which I suppose 
are harmless too.
[2017-03-17 14:23:36.968883] W [MSGID: 108008] 
[afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable subvolume 
-1 found with event generation 2 for gfid 
74d49288-8452-

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

This is a good news
Is this related to the previously fixed bug?

Il 3 apr 2017 10:22 AM, "Krutika Dhananjay"  ha
scritto:

> So Raghavendra has an RCA for this issue.
>
> Copy-pasting his comment here:
>
> 
>
> Following is a rough algorithm of shard_writev:
>
> 1. Based on the offset, calculate the shards touched by current write.
> 2. Look for inodes corresponding to these shard files in itable.
> 3. If one or more inodes are missing from itable, issue mknod for 
> corresponding shard files and ignore EEXIST in cbk.
> 4. resume writes on respective shards.
>
> Now, imagine a write which falls to an existing "shard_file". For the sake of 
> discussion lets consider a distribute of three subvols - s1, s2, s3
>
> 1. "shard_file" hashes to subvolume s2 and is present on s2
> 2. add a subvolume s4 and initiate a fix layout. The layout of ".shard" is 
> fixed to include s4 and hash ranges are changed.
> 3. write that touches "shard_file" is issued.
> 4. The inode for "shard_file" is not present in itable after a graph switch 
> and features/shard issues an mknod.
> 5. With new layout of .shard, lets say "shard_file" hashes to s3 and mknod 
> (shard_file) on s3 succeeds. But, the shard_file is already present on s2.
>
> So, we have two files on two different subvols of dht representing same shard 
> and this will lead to corruption.
>
> 
>
> Raghavendra will be sending out a patch in DHT to fix this issue.
>
> -Krutika
>
>
> On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> Do you guys have any update regarding this issue ?
>>>
>> I do not actively work on this issue so I do not have an accurate update,
>> but from what I heard from Krutika and Raghavendra(works on DHT) is:
>> Krutika debugged initially and found that the issue seems more likely to be
>> in DHT, Satheesaran who helped us recreate this issue in lab found that
>> just fix-layout without rebalance also caused the corruption 1 out of 3
>> times. Raghavendra came up with a possible RCA for why this can happen.
>> Raghavendra(CCed) would be the right person to provide accurate update.
>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Tuesday, March 21, 2017 3:02:55 PM
>>> *To:* Mahdi Adnan
>>> *Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>>> gluster-users@gluster.org List
>>>
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>> Hi,
>>>
>>> So it looks like Satheesaran managed to recreate this issue. We will be
>>> seeking his help in debugging this. It will be easier that way.
>>>
>>> -Krutika
>>>
>>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Hello and thank you for your email.
>>>> Actually no, i didn't check the gfid of the vms.
>>>> If this will help, i can setup a new test cluster and get all the data
>>>> you need.
>>>>
>>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>>
>>>> From: Nithya Balachandran
>>>> Sent: Monday, March 20, 20:57
>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>> To: Krutika Dhananjay
>>>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>>>> gluster-users@gluster.org List
>>>>
>>>> Hi,
>>>>
>>>> Do you know the GFIDs of the VM images which were corrupted?
>>>>
>>>> Regards,
>>>>
>>>> Nithya
>>>>
>>>> On 20 March 2017 at 20:37, Krutika Dhananjay 
>>>> wrote:
>>>>
>>>> I looked at the logs.
>>>>
>>>> From the time the new graph (since the add-brick command you shared
>>>> where bricks 41 through 44 are added) is switched to (line 3011 onwards in
>>>> nfs-gfapi.log), I see the following kinds of errors:
>>>>
>>>> 1. Lookups to a bunch of files failed with ENOENT on both replicas
>>>> which protocol/client converts to ESTALE. I am guessing these entries got
>>>> migrated to
>>>>
>>>> other subvolumes leading to '

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

So Raghavendra has an RCA for this issue.

Copy-pasting his comment here:



Following is a rough algorithm of shard_writev:

1. Based on the offset, calculate the shards touched by current write.
2. Look for inodes corresponding to these shard files in itable.
3. If one or more inodes are missing from itable, issue mknod for
corresponding shard files and ignore EEXIST in cbk.
4. resume writes on respective shards.

Now, imagine a write which falls to an existing "shard_file". For the
sake of discussion lets consider a distribute of three subvols - s1,
s2, s3

1. "shard_file" hashes to subvolume s2 and is present on s2
2. add a subvolume s4 and initiate a fix layout. The layout of
".shard" is fixed to include s4 and hash ranges are changed.
3. write that touches "shard_file" is issued.
4. The inode for "shard_file" is not present in itable after a graph
switch and features/shard issues an mknod.
5. With new layout of .shard, lets say "shard_file" hashes to s3 and
mknod (shard_file) on s3 succeeds. But, the shard_file is already
present on s2.

So, we have two files on two different subvols of dht representing
same shard and this will lead to corruption.



Raghavendra will be sending out a patch in DHT to fix this issue.

-Krutika


On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
> wrote:
>
>> Hi,
>>
>>
>> Do you guys have any update regarding this issue ?
>>
> I do not actively work on this issue so I do not have an accurate update,
> but from what I heard from Krutika and Raghavendra(works on DHT) is:
> Krutika debugged initially and found that the issue seems more likely to be
> in DHT, Satheesaran who helped us recreate this issue in lab found that
> just fix-layout without rebalance also caused the corruption 1 out of 3
> times. Raghavendra came up with a possible RCA for why this can happen.
> Raghavendra(CCed) would be the right person to provide accurate update.
>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Tuesday, March 21, 2017 3:02:55 PM
>> *To:* Mahdi Adnan
>> *Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>> gluster-users@gluster.org List
>>
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> Hi,
>>
>> So it looks like Satheesaran managed to recreate this issue. We will be
>> seeking his help in debugging this. It will be easier that way.
>>
>> -Krutika
>>
>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
>> wrote:
>>
>>> Hello and thank you for your email.
>>> Actually no, i didn't check the gfid of the vms.
>>> If this will help, i can setup a new test cluster and get all the data
>>> you need.
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>
>>> From: Nithya Balachandran
>>> Sent: Monday, March 20, 20:57
>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>> To: Krutika Dhananjay
>>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>>> gluster-users@gluster.org List
>>>
>>> Hi,
>>>
>>> Do you know the GFIDs of the VM images which were corrupted?
>>>
>>> Regards,
>>>
>>> Nithya
>>>
>>> On 20 March 2017 at 20:37, Krutika Dhananjay 
>>> wrote:
>>>
>>> I looked at the logs.
>>>
>>> From the time the new graph (since the add-brick command you shared
>>> where bricks 41 through 44 are added) is switched to (line 3011 onwards in
>>> nfs-gfapi.log), I see the following kinds of errors:
>>>
>>> 1. Lookups to a bunch of files failed with ENOENT on both replicas which
>>> protocol/client converts to ESTALE. I am guessing these entries got
>>> migrated to
>>>
>>> other subvolumes leading to 'No such file or directory' errors.
>>>
>>> DHT and thereafter shard get the same error code and log the following:
>>>
>>>  0 [2017-03-17 14:04:26.353444] E [MSGID: 109040]
>>> [dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht:
>>> : failed to lookup the
>>> file on vmware2-dht [Stale file handle]
>>>
>>>
>>>   1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
>>> [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
>>> a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]
>>&

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Amar Tumballi]

On Wed, Mar 29, 2017 at 12:32 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Is rebalance and fix layout needed when adding new bricks?
> Any workaround for extending a cluster without loose data?
>

If your existing nodes are not completely full, and you add more nodes to
create new directories and create files in these new directories,
fix-layout and rebalance is not always required. If the new files are
getting created in existing directories, then fix-layout is required.

Regards,
Amar


>
> Il 28 mar 2017 8:19 PM, "Pranith Kumar Karampuri" 
> ha scritto:
>
>>
>>
>> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> Do you guys have any update regarding this issue ?
>>>
>> I do not actively work on this issue so I do not have an accurate update,
>> but from what I heard from Krutika and Raghavendra(works on DHT) is:
>> Krutika debugged initially and found that the issue seems more likely to be
>> in DHT, Satheesaran who helped us recreate this issue in lab found that
>> just fix-layout without rebalance also caused the corruption 1 out of 3
>> times. Raghavendra came up with a possible RCA for why this can happen.
>> Raghavendra(CCed) would be the right person to provide accurate update.
>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Tuesday, March 21, 2017 3:02:55 PM
>>> *To:* Mahdi Adnan
>>> *Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>>> gluster-users@gluster.org List
>>>
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>> Hi,
>>>
>>> So it looks like Satheesaran managed to recreate this issue. We will be
>>> seeking his help in debugging this. It will be easier that way.
>>>
>>> -Krutika
>>>
>>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Hello and thank you for your email.
>>>> Actually no, i didn't check the gfid of the vms.
>>>> If this will help, i can setup a new test cluster and get all the data
>>>> you need.
>>>>
>>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>>
>>>> From: Nithya Balachandran
>>>> Sent: Monday, March 20, 20:57
>>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>> To: Krutika Dhananjay
>>>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>>>> gluster-users@gluster.org List
>>>>
>>>> Hi,
>>>>
>>>> Do you know the GFIDs of the VM images which were corrupted?
>>>>
>>>> Regards,
>>>>
>>>> Nithya
>>>>
>>>> On 20 March 2017 at 20:37, Krutika Dhananjay 
>>>> wrote:
>>>>
>>>> I looked at the logs.
>>>>
>>>> From the time the new graph (since the add-brick command you shared
>>>> where bricks 41 through 44 are added) is switched to (line 3011 onwards in
>>>> nfs-gfapi.log), I see the following kinds of errors:
>>>>
>>>> 1. Lookups to a bunch of files failed with ENOENT on both replicas
>>>> which protocol/client converts to ESTALE. I am guessing these entries got
>>>> migrated to
>>>>
>>>> other subvolumes leading to 'No such file or directory' errors.
>>>>
>>>> DHT and thereafter shard get the same error code and log the following:
>>>>
>>>>  0 [2017-03-17 14:04:26.353444] E [MSGID: 109040]
>>>> [dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht:
>>>> : failed to lookup the
>>>> file on vmware2-dht [Stale file handle]
>>>>
>>>>
>>>>   1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
>>>> [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
>>>> a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]
>>>>
>>>> which is fine.
>>>>
>>>> 2. The other kind are from AFR logging of possible split-brain which I
>>>> suppose are harmless too.
>>>> [2017-03-17 14:23:36.968883] W [MSGID: 108008]
>>>> [afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable
>>>> subvolume -1 found with event generation 2 for gfid
>>

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

Is rebalance and fix layout needed when adding new bricks?
Any workaround for extending a cluster without loose data?

Il 28 mar 2017 8:19 PM, "Pranith Kumar Karampuri"  ha
scritto:

>
>
> On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
> wrote:
>
>> Hi,
>>
>>
>> Do you guys have any update regarding this issue ?
>>
> I do not actively work on this issue so I do not have an accurate update,
> but from what I heard from Krutika and Raghavendra(works on DHT) is:
> Krutika debugged initially and found that the issue seems more likely to be
> in DHT, Satheesaran who helped us recreate this issue in lab found that
> just fix-layout without rebalance also caused the corruption 1 out of 3
> times. Raghavendra came up with a possible RCA for why this can happen.
> Raghavendra(CCed) would be the right person to provide accurate update.
>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Tuesday, March 21, 2017 3:02:55 PM
>> *To:* Mahdi Adnan
>> *Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
>> gluster-users@gluster.org List
>>
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> Hi,
>>
>> So it looks like Satheesaran managed to recreate this issue. We will be
>> seeking his help in debugging this. It will be easier that way.
>>
>> -Krutika
>>
>> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
>> wrote:
>>
>>> Hello and thank you for your email.
>>> Actually no, i didn't check the gfid of the vms.
>>> If this will help, i can setup a new test cluster and get all the data
>>> you need.
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>
>>> From: Nithya Balachandran
>>> Sent: Monday, March 20, 20:57
>>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>> To: Krutika Dhananjay
>>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>>> gluster-users@gluster.org List
>>>
>>> Hi,
>>>
>>> Do you know the GFIDs of the VM images which were corrupted?
>>>
>>> Regards,
>>>
>>> Nithya
>>>
>>> On 20 March 2017 at 20:37, Krutika Dhananjay 
>>> wrote:
>>>
>>> I looked at the logs.
>>>
>>> From the time the new graph (since the add-brick command you shared
>>> where bricks 41 through 44 are added) is switched to (line 3011 onwards in
>>> nfs-gfapi.log), I see the following kinds of errors:
>>>
>>> 1. Lookups to a bunch of files failed with ENOENT on both replicas which
>>> protocol/client converts to ESTALE. I am guessing these entries got
>>> migrated to
>>>
>>> other subvolumes leading to 'No such file or directory' errors.
>>>
>>> DHT and thereafter shard get the same error code and log the following:
>>>
>>>  0 [2017-03-17 14:04:26.353444] E [MSGID: 109040]
>>> [dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht:
>>> : failed to lookup the
>>> file on vmware2-dht [Stale file handle]
>>>
>>>
>>>   1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
>>> [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
>>> a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]
>>>
>>> which is fine.
>>>
>>> 2. The other kind are from AFR logging of possible split-brain which I
>>> suppose are harmless too.
>>> [2017-03-17 14:23:36.968883] W [MSGID: 108008]
>>> [afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable
>>> subvolume -1 found with event generation 2 for gfid
>>> 74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)
>>>
>>> Since you are saying the bug is hit only on VMs that are undergoing IO
>>> while rebalance is running (as opposed to those that remained powered off),
>>>
>>> rebalance + IO could be causing some issues.
>>>
>>> CC'ing DHT devs
>>>
>>> Raghavendra/Nithya/Susant,
>>>
>>> Could you take a look?
>>>
>>> -Krutika
>>>
>>>
>>> On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
>>> wrote:
>>>
>>> Thank you for your email mate.
>>>
>>> Yes, im aware of this but, to save costs i chose replica 2, this cluster
>>> is all flash.
>>>
>>> In version 3.

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gambit15]

On 19 March 2017 at 07:25, Mahdi Adnan  wrote:

> Thank you for your email mate.
>
> Yes, im aware of this but, to save costs i chose replica 2, this cluster
> is all flash.
>

For what it's worth, arbiters FTW!

https://gluster.readthedocs.io/en/latest/Administrator
Guide/arbiter-volumes-and-quorum/


> In version 3.7.x i had issues with ping timeout, if one hosts went down
> for few seconds the whole cluster hangs and become unavailable, to avoid
> this i adjusted the ping timeout to 5 seconds.
>
> As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE
> or gfapi) im stuck with NFS for this volume.
>
> The other volume is mounted using gfapi in oVirt cluster.
>
>
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 2:01:49 PM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> While I'm still going through the logs, just wanted to point out a couple
> of things:
>
> 1. It is recommended that you use 3-way replication (replica count 3) for
> VM store use case
> 2. network.ping-timeout at 5 seconds is way too low. Please change it to
> 30.
>
> Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?
>
> Will get back with anything else I might find or more questions if I have
> any.
>
> -Krutika
>
> On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
> wrote:
>
>> Thanks mate,
>>
>> Kindly, check the attachment.
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Sunday, March 19, 2017 10:00:22 AM
>>
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> In that case could you share the ganesha-gfapi logs?
>>
>> -Krutika
>>
>> On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
>> wrote:
>>
>>> I have two volumes, one is mounted using libgfapi for ovirt mount, the
>>> other one is exported via NFS-Ganesha for VMWare which is the one im
>>> testing now.
>>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Sunday, March 19, 2017 8:02:19 AM
>>>
>>> *To:* Mahdi Adnan
>>> *Cc:* gluster-users@gluster.org
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>>
>>>
>>> On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Kindly, check the attached new log file, i dont know if it's helpful or
>>>> not but, i couldn't find the log with the name you just described.
>>>>
>>> No. Are you using FUSE or libgfapi for accessing the volume? Or is it
>>> NFS?
>>>
>>> -Krutika
>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Respectfully
>>>> *Mahdi A. Mahdi*
>>>>
>>>> --
>>>> *From:* Krutika Dhananjay 
>>>> *Sent:* Saturday, March 18, 2017 6:10:40 PM
>>>>
>>>> *To:* Mahdi Adnan
>>>> *Cc:* gluster-users@gluster.org
>>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>>
>>>> mnt-disk11-vmware2.log seems like a brick log. Could you attach the
>>>> fuse mount logs? It should be right under /var/log/glusterfs/ directory
>>>> named after the mount point name, only hyphenated.
>>>>
>>>> -Krutika
>>>>
>>>> On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
>>>> wrote:
>>>>
>>>>> Hello Krutika,
>>>>>
>>>>>
>>>>> Kindly, check the attached logs.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Respectfully
>>>>> *Mahdi A. Mahdi*
>>>>>
>>>>> --
>>>>> *From:* Krutika Dhananjay 
>>>>> *Sent:* Saturday, March 18, 2017 3:29:03 PM
>>>>> *To:* Mahdi Adnan
>>>>> *Cc:* gluster-users@gluster.org
>>>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VM

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Pranith Kumar Karampuri]

On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan 
wrote:

> Hi,
>
>
> Do you guys have any update regarding this issue ?
>
I do not actively work on this issue so I do not have an accurate update,
but from what I heard from Krutika and Raghavendra(works on DHT) is:
Krutika debugged initially and found that the issue seems more likely to be
in DHT, Satheesaran who helped us recreate this issue in lab found that
just fix-layout without rebalance also caused the corruption 1 out of 3
times. Raghavendra came up with a possible RCA for why this can happen.
Raghavendra(CCed) would be the right person to provide accurate update.

>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Tuesday, March 21, 2017 3:02:55 PM
> *To:* Mahdi Adnan
> *Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai;
> gluster-users@gluster.org List
>
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> Hi,
>
> So it looks like Satheesaran managed to recreate this issue. We will be
> seeking his help in debugging this. It will be easier that way.
>
> -Krutika
>
> On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
> wrote:
>
>> Hello and thank you for your email.
>> Actually no, i didn't check the gfid of the vms.
>> If this will help, i can setup a new test cluster and get all the data
>> you need.
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> From: Nithya Balachandran
>> Sent: Monday, March 20, 20:57
>> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>> To: Krutika Dhananjay
>> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
>> gluster-users@gluster.org List
>>
>> Hi,
>>
>> Do you know the GFIDs of the VM images which were corrupted?
>>
>> Regards,
>>
>> Nithya
>>
>> On 20 March 2017 at 20:37, Krutika Dhananjay  wrote:
>>
>> I looked at the logs.
>>
>> From the time the new graph (since the add-brick command you shared where
>> bricks 41 through 44 are added) is switched to (line 3011 onwards in
>> nfs-gfapi.log), I see the following kinds of errors:
>>
>> 1. Lookups to a bunch of files failed with ENOENT on both replicas which
>> protocol/client converts to ESTALE. I am guessing these entries got
>> migrated to
>>
>> other subvolumes leading to 'No such file or directory' errors.
>>
>> DHT and thereafter shard get the same error code and log the following:
>>
>>  0 [2017-03-17 14:04:26.353444] E [MSGID: 109040]
>> [dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht:
>> : failed to lookup the
>> file on vmware2-dht [Stale file handle]
>>
>>
>>   1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
>> [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
>> a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]
>>
>> which is fine.
>>
>> 2. The other kind are from AFR logging of possible split-brain which I
>> suppose are harmless too.
>> [2017-03-17 14:23:36.968883] W [MSGID: 108008]
>> [afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable
>> subvolume -1 found with event generation 2 for gfid
>> 74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)
>>
>> Since you are saying the bug is hit only on VMs that are undergoing IO
>> while rebalance is running (as opposed to those that remained powered off),
>>
>> rebalance + IO could be causing some issues.
>>
>> CC'ing DHT devs
>>
>> Raghavendra/Nithya/Susant,
>>
>> Could you take a look?
>>
>> -Krutika
>>
>>
>> On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
>> wrote:
>>
>> Thank you for your email mate.
>>
>> Yes, im aware of this but, to save costs i chose replica 2, this cluster
>> is all flash.
>>
>> In version 3.7.x i had issues with ping timeout, if one hosts went down
>> for few seconds the whole cluster hangs and become unavailable, to avoid
>> this i adjusted the ping timeout to 5 seconds.
>>
>> As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE
>> or gfapi) im stuck with NFS for this volume.
>>
>> The other volume is mounted using gfapi in oVirt cluster.
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> *From:* Krutika Dhananjay 
>> *Sent:* Sunday, March 19, 2017 2:01:49 PM
>>
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Joe Julian]

Based on what I know of the workflow, there is no update. There is no 
bug report in bugzilla so there are no patches in review for it.



On 03/27/2017 10:59 AM, Mahdi Adnan wrote:


Hi,


Do you guys have any update regarding this issue ?



--

Respectfully*
**Mahdi A. Mahdi*


*From:* Krutika Dhananjay 
*Sent:* Tuesday, March 21, 2017 3:02:55 PM
*To:* Mahdi Adnan
*Cc:* Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai; 
gluster-users@gluster.org List

*Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
Hi,

So it looks like Satheesaran managed to recreate this issue. We will 
be seeking his help in debugging this. It will be easier that way.


-Krutika

On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan <mailto:mahdi.ad...@outlook.com>> wrote:


Hello and thank you for your email.
Actually no, i didn't check the gfid of the vms.
If this will help, i can setup a new test cluster and get all the
data you need.

Get Outlook for Android <https://aka.ms/ghei36>


From: Nithya Balachandran
Sent: Monday, March 20, 20:57
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
To: Krutika Dhananjay
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
gluster-users@gluster.org <mailto:gluster-users@gluster.org> List

Hi,

Do you know the GFIDs of the VM images which were corrupted?

Regards,

Nithya

On 20 March 2017 at 20:37, Krutika Dhananjay mailto:kdhan...@redhat.com>> wrote:


I looked at the logs.

From the time the new graph (since the add-brick command you
shared where bricks 41 through 44 are added) is switched to (line
3011 onwards in nfs-gfapi.log), I see the following kinds of errors:

1. Lookups to a bunch of files failed with ENOENT on both
replicas which protocol/client converts to ESTALE. I am guessing
these entries got migrated to

other subvolumes leading to 'No such file or directory' errors.

DHT and thereafter shard get the same error code and log the
following:

 0 [2017-03-17 14:04:26.353444] E [MSGID: 109040]
[dht-helper.c:1198:dht_migration_complete_check_task]
17-vmware2-dht: :
failed to lookup the file on vmware2-dht [Stale file handle]
  1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
[shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat
failed: a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]

which is fine.

2. The other kind are from AFR logging of possible split-brain
which I suppose are harmless too.
[2017-03-17 14:23:36.968883] W [MSGID: 108008]
[afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13:
Unreadable subvolume -1 found with event generation 2 for gfid
74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)

Since you are saying the bug is hit only on VMs that are
undergoing IO while rebalance is running (as opposed to those
that remained powered off),

rebalance + IO could be causing some issues.

CC'ing DHT devs

Raghavendra/Nithya/Susant,

Could you take a look?

-Krutika


On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan
mailto:mahdi.ad...@outlook.com>> wrote:


Thank you for your email mate.

Yes, im aware of this but, to save costs i chose replica 2, this
cluster is all flash.

In version 3.7.x i had issues with ping timeout, if one hosts
went down for few seconds the whole cluster hangs and become
unavailable, to avoid this i adjusted the ping timeout to 5 seconds.

As for choosing Ganesha over gfapi, VMWare does not support
Gluster (FUSE or gfapi) im stuck with NFS for this volume.

The other volume is mounted using gfapi in oVirt cluster.



-- 


Respectfully
*Mahdi A. Mahdi*

*From:*Krutika Dhananjay mailto:kdhan...@redhat.com>>
*Sent:*Sunday, March 19, 2017 2:01:49 PM

*To:* Mahdi Adnan
*Cc:* gluster-users@gluster.org <mailto:gluster-users@gluster.org>
    *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs
corruption

While I'm still going through the logs, just wanted to point out
a couple of things:

1. It is recommended that you use 3-way replication (replica
count 3) for VM store use case

2. network.ping-timeout at 5 seconds is way too low. Please
change it to 30.

Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?

Will get back with anything else I might find or more questions
if I have any.

-Krutika

On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan
mailto:mahdi.ad...@outlook.com>> wrote:


Thanks mate,

Kindly, check the attachment.

-- 


Respectfully
*Mahdi A. Mahdi*

*From:*Krutika Dhananjay mailto:kdhan...@redhat.com>>
*Sent:*Sunday, March 19, 2017 10:00:22 AM

*To:*

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Hi,


Do you guys have any update regarding this issue ?


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay 
Sent: Tuesday, March 21, 2017 3:02:55 PM
To: Mahdi Adnan
Cc: Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai; 
gluster-users@gluster.org List
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

Hi,

So it looks like Satheesaran managed to recreate this issue. We will be seeking 
his help in debugging this. It will be easier that way.

-Krutika

On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hello and thank you for your email.
Actually no, i didn't check the gfid of the vms.
If this will help, i can setup a new test cluster and get all the data you need.

Get Outlook for Android<https://aka.ms/ghei36>


From: Nithya Balachandran
Sent: Monday, March 20, 20:57
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
To: Krutika Dhananjay
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai, 
gluster-users@gluster.org<mailto:gluster-users@gluster.org> List

Hi,

Do you know the GFIDs of the VM images which were corrupted?

Regards,

Nithya

On 20 March 2017 at 20:37, Krutika Dhananjay 
mailto:kdhan...@redhat.com>> wrote:

I looked at the logs.

>From the time the new graph (since the add-brick command you shared where 
>bricks 41 through 44 are added) is switched to (line 3011 onwards in 
>nfs-gfapi.log), I see the following kinds of errors:

1. Lookups to a bunch of files failed with ENOENT on both replicas which 
protocol/client converts to ESTALE. I am guessing these entries got migrated to

other subvolumes leading to 'No such file or directory' errors.

DHT and thereafter shard get the same error code and log the following:

 0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] 
[dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht: 
: failed to lookup the file on 
vmware2-dht [Stale file handle]
  1 [2017-03-17 14:04:26.353528] E [MSGID: 133014] 
[shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed: 
a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]

which is fine.

2. The other kind are from AFR logging of possible split-brain which I suppose 
are harmless too.
[2017-03-17 14:23:36.968883] W [MSGID: 108008] 
[afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable subvolume 
-1 found with event generation 2 for gfid 
74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)

Since you are saying the bug is hit only on VMs that are undergoing IO while 
rebalance is running (as opposed to those that remained powered off),

rebalance + IO could be causing some issues.

CC'ing DHT devs

Raghavendra/Nithya/Susant,

Could you take a look?

-Krutika



On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Thank you for your email mate.

Yes, im aware of this but, to save costs i chose replica 2, this cluster is all 
flash.

In version 3.7.x i had issues with ping timeout, if one hosts went down for few 
seconds the whole cluster hangs and become unavailable, to avoid this i 
adjusted the ping timeout to 5 seconds.

As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE or 
gfapi) im stuck with NFS for this volume.

The other volume is mounted using gfapi in oVirt cluster.




--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 2:01:49 PM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption



While I'm still going through the logs, just wanted to point out a couple of 
things:

1. It is recommended that you use 3-way replication (replica count 3) for VM 
store use case

2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.

Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?

Will get back with anything else I might find or more questions if I have any.

-Krutika

On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Thanks mate,

Kindly, check the attachment.


--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 10:00:22 AM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption



In that case could you share the ganesha-gfapi logs?

-Krutika

On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

I have two volumes, one is mounted using libgfapi for ovirt mount, the other 
one is exported via NFS-Ganesha for VMWare which is the one im testing now.


--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Hello and thank you for your email.
Actually no, i didn't check the gfid of the vms.
If this will help, i can setup a new test cluster and get all the data you need.

Get Outlook for Android<https://aka.ms/ghei36>


From: Nithya Balachandran
Sent: Monday, March 20, 20:57
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
To: Krutika Dhananjay
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai, gluster-users@gluster.org 
List

Hi,

Do you know the GFIDs of the VM images which were corrupted?

Regards,

Nithya

On 20 March 2017 at 20:37, Krutika Dhananjay 
mailto:kdhan...@redhat.com>> wrote:

I looked at the logs.

>From the time the new graph (since the add-brick command you shared where 
>bricks 41 through 44 are added) is switched to (line 3011 onwards in 
>nfs-gfapi.log), I see the following kinds of errors:

1. Lookups to a bunch of files failed with ENOENT on both replicas which 
protocol/client converts to ESTALE. I am guessing these entries got migrated to

other subvolumes leading to 'No such file or directory' errors.

DHT and thereafter shard get the same error code and log the following:

 0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] 
[dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht: 
: failed to lookup the file on 
vmware2-dht [Stale file handle]
  1 [2017-03-17 14:04:26.353528] E [MSGID: 133014] 
[shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed: 
a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]

which is fine.

2. The other kind are from AFR logging of possible split-brain which I suppose 
are harmless too.
[2017-03-17 14:23:36.968883] W [MSGID: 108008] 
[afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable subvolume 
-1 found with event generation 2 for gfid 
74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)

Since you are saying the bug is hit only on VMs that are undergoing IO while 
rebalance is running (as opposed to those that remained powered off),

rebalance + IO could be causing some issues.

CC'ing DHT devs

Raghavendra/Nithya/Susant,

Could you take a look?

-Krutika



On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Thank you for your email mate.

Yes, im aware of this but, to save costs i chose replica 2, this cluster is all 
flash.

In version 3.7.x i had issues with ping timeout, if one hosts went down for few 
seconds the whole cluster hangs and become unavailable, to avoid this i 
adjusted the ping timeout to 5 seconds.

As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE or 
gfapi) im stuck with NFS for this volume.

The other volume is mounted using gfapi in oVirt cluster.




--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 2:01:49 PM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption



While I'm still going through the logs, just wanted to point out a couple of 
things:

1. It is recommended that you use 3-way replication (replica count 3) for VM 
store use case

2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.

Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?

Will get back with anything else I might find or more questions if I have any.

-Krutika

On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Thanks mate,

Kindly, check the attachment.


--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 10:00:22 AM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption



In that case could you share the ganesha-gfapi logs?

-Krutika

On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

I have two volumes, one is mounted using libgfapi for ovirt mount, the other 
one is exported via NFS-Ganesha for VMWare which is the one im testing now.


--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 8:02:19 AM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption




On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Kindly, check the attached new log file, i dont know if it's helpful or not 
but, i couldn't find the log with the name you just described.

No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?



-Krutika

--

Respectfully
Mahdi A. Mahdi

From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Saturday, March 18, 2017 6:10:40 PM

To: Mahdi A

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

Hi,

So it looks like Satheesaran managed to recreate this issue. We will be
seeking his help in debugging this. It will be easier that way.

-Krutika

On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan 
wrote:

> Hello and thank you for your email.
> Actually no, i didn't check the gfid of the vms.
> If this will help, i can setup a new test cluster and get all the data you
> need.
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> From: Nithya Balachandran
> Sent: Monday, March 20, 20:57
> Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
> To: Krutika Dhananjay
> Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
> gluster-users@gluster.org List
>
> Hi,
>
> Do you know the GFIDs of the VM images which were corrupted?
>
> Regards,
>
> Nithya
>
> On 20 March 2017 at 20:37, Krutika Dhananjay  wrote:
>
> I looked at the logs.
>
> From the time the new graph (since the add-brick command you shared where
> bricks 41 through 44 are added) is switched to (line 3011 onwards in
> nfs-gfapi.log), I see the following kinds of errors:
>
> 1. Lookups to a bunch of files failed with ENOENT on both replicas which
> protocol/client converts to ESTALE. I am guessing these entries got
> migrated to
>
> other subvolumes leading to 'No such file or directory' errors.
>
> DHT and thereafter shard get the same error code and log the following:
>
>  0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] 
> [dht-helper.c:1198:dht_migration_complete_check_task]
> 17-vmware2-dht: : failed
> to lookup the file on vmware2-dht [Stale file handle]
>
>
>   1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
> [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
> a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]
>
> which is fine.
>
> 2. The other kind are from AFR logging of possible split-brain which I
> suppose are harmless too.
> [2017-03-17 14:23:36.968883] W [MSGID: 108008]
> [afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable
> subvolume -1 found with event generation 2 for gfid
> 74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)
>
> Since you are saying the bug is hit only on VMs that are undergoing IO
> while rebalance is running (as opposed to those that remained powered off),
>
> rebalance + IO could be causing some issues.
>
> CC'ing DHT devs
>
> Raghavendra/Nithya/Susant,
>
> Could you take a look?
>
> -Krutika
>
>
> On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
> wrote:
>
> Thank you for your email mate.
>
> Yes, im aware of this but, to save costs i chose replica 2, this cluster
> is all flash.
>
> In version 3.7.x i had issues with ping timeout, if one hosts went down
> for few seconds the whole cluster hangs and become unavailable, to avoid
> this i adjusted the ping timeout to 5 seconds.
>
> As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE
> or gfapi) im stuck with NFS for this volume.
>
> The other volume is mounted using gfapi in oVirt cluster.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 2:01:49 PM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
>
>
> While I'm still going through the logs, just wanted to point out a couple
> of things:
>
> 1. It is recommended that you use 3-way replication (replica count 3) for
> VM store use case
>
> 2. network.ping-timeout at 5 seconds is way too low. Please change it to
> 30.
>
> Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?
>
> Will get back with anything else I might find or more questions if I have
> any.
>
> -Krutika
>
> On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
> wrote:
>
> Thanks mate,
>
> Kindly, check the attachment.
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 10:00:22 AM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
>
>
> In that case could you share the ganesha-gfapi logs?
>
> -Krutika
>
> On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
> wrote:
>
> I have two volumes, one is mounted using libgfapi for ovirt mount, the
> other one is exported via NFS-Ganesha for VMWare which is the one im
> testing now.
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 8:02:19 AM
>
> *To:* Mahdi Adnan
> *Cc:* glust

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Nithya Balachandran]

Hi,

Do you know the GFIDs of the VM images which were corrupted?

Regards,
Nithya

On 20 March 2017 at 20:37, Krutika Dhananjay  wrote:

> I looked at the logs.
>
> From the time the new graph (since the add-brick command you shared where
> bricks 41 through 44 are added) is switched to (line 3011 onwards in
> nfs-gfapi.log), I see the following kinds of errors:
>
> 1. Lookups to a bunch of files failed with ENOENT on both replicas which
> protocol/client converts to ESTALE. I am guessing these entries got
> migrated to
> other subvolumes leading to 'No such file or directory' errors.
> DHT and thereafter shard get the same error code and log the following:
>
>  0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] 
> [dht-helper.c:1198:dht_migration_complete_check_task]
> 17-vmware2-dht: : failed
> to lookup the file on vmware2-dht [Stale file handle]
>
>
>   1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
> [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
> a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]
>
> which is fine.
>
> 2. The other kind are from AFR logging of possible split-brain which I
> suppose are harmless too.
> [2017-03-17 14:23:36.968883] W [MSGID: 108008]
> [afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable
> subvolume -1 found with event generation 2 for gfid
> 74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)
>
> Since you are saying the bug is hit only on VMs that are undergoing IO
> while rebalance is running (as opposed to those that remained powered off),
> rebalance + IO could be causing some issues.
>
> CC'ing DHT devs
>
> Raghavendra/Nithya/Susant,
>
> Could you take a look?
>
> -Krutika
>
>
>
> On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
> wrote:
>
>> Thank you for your email mate.
>>
>>
>> Yes, im aware of this but, to save costs i chose replica 2, this cluster
>> is all flash.
>>
>> In version 3.7.x i had issues with ping timeout, if one hosts went down
>> for few seconds the whole cluster hangs and become unavailable, to avoid
>> this i adjusted the ping timeout to 5 seconds.
>>
>> As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE
>> or gfapi) im stuck with NFS for this volume.
>>
>> The other volume is mounted using gfapi in oVirt cluster.
>>
>>
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Sunday, March 19, 2017 2:01:49 PM
>>
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> While I'm still going through the logs, just wanted to point out a couple
>> of things:
>>
>> 1. It is recommended that you use 3-way replication (replica count 3) for
>> VM store use case
>> 2. network.ping-timeout at 5 seconds is way too low. Please change it to
>> 30.
>>
>> Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?
>>
>> Will get back with anything else I might find or more questions if I have
>> any.
>>
>> -Krutika
>>
>> On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
>> wrote:
>>
>>> Thanks mate,
>>>
>>> Kindly, check the attachment.
>>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Sunday, March 19, 2017 10:00:22 AM
>>>
>>> *To:* Mahdi Adnan
>>> *Cc:* gluster-users@gluster.org
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>> In that case could you share the ganesha-gfapi logs?
>>>
>>> -Krutika
>>>
>>> On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> I have two volumes, one is mounted using libgfapi for ovirt mount, the
>>>> other one is exported via NFS-Ganesha for VMWare which is the one im
>>>> testing now.
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Respectfully
>>>> *Mahdi A. Mahdi*
>>>>
>>>> --
>>>> *From:* Krutika Dhananjay 
>>>> *Sent:* Sunday, March 19, 2017 8:02:19 AM
>>>>
>>>> *To:* Mahdi Adnan
>>>> *Cc:* gluster-users@gluster.org
>>>> *Subject:* Re: [Gluste

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

I looked at the logs.

>From the time the new graph (since the add-brick command you shared where
bricks 41 through 44 are added) is switched to (line 3011 onwards in
nfs-gfapi.log), I see the following kinds of errors:

1. Lookups to a bunch of files failed with ENOENT on both replicas which
protocol/client converts to ESTALE. I am guessing these entries got
migrated to
other subvolumes leading to 'No such file or directory' errors.
DHT and thereafter shard get the same error code and log the following:

 0 [2017-03-17 14:04:26.353444] E [MSGID: 109040]
[dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht:
: failed to lookup the file
on vmware2-dht [Stale file
handle]

  1 [2017-03-17 14:04:26.353528] E [MSGID: 133014]
[shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed:
a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]

which is fine.

2. The other kind are from AFR logging of possible split-brain which I
suppose are harmless too.
[2017-03-17 14:23:36.968883] W [MSGID: 108008]
[afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable
subvolume -1 found with event generation 2 for gfid
74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)

Since you are saying the bug is hit only on VMs that are undergoing IO
while rebalance is running (as opposed to those that remained powered off),
rebalance + IO could be causing some issues.

CC'ing DHT devs

Raghavendra/Nithya/Susant,

Could you take a look?

-Krutika



On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan 
wrote:

> Thank you for your email mate.
>
>
> Yes, im aware of this but, to save costs i chose replica 2, this cluster
> is all flash.
>
> In version 3.7.x i had issues with ping timeout, if one hosts went down
> for few seconds the whole cluster hangs and become unavailable, to avoid
> this i adjusted the ping timeout to 5 seconds.
>
> As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE
> or gfapi) im stuck with NFS for this volume.
>
> The other volume is mounted using gfapi in oVirt cluster.
>
>
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 2:01:49 PM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> While I'm still going through the logs, just wanted to point out a couple
> of things:
>
> 1. It is recommended that you use 3-way replication (replica count 3) for
> VM store use case
> 2. network.ping-timeout at 5 seconds is way too low. Please change it to
> 30.
>
> Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?
>
> Will get back with anything else I might find or more questions if I have
> any.
>
> -Krutika
>
> On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
> wrote:
>
>> Thanks mate,
>>
>> Kindly, check the attachment.
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Sunday, March 19, 2017 10:00:22 AM
>>
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> In that case could you share the ganesha-gfapi logs?
>>
>> -Krutika
>>
>> On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
>> wrote:
>>
>>> I have two volumes, one is mounted using libgfapi for ovirt mount, the
>>> other one is exported via NFS-Ganesha for VMWare which is the one im
>>> testing now.
>>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Sunday, March 19, 2017 8:02:19 AM
>>>
>>> *To:* Mahdi Adnan
>>> *Cc:* gluster-users@gluster.org
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>>
>>>
>>> On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Kindly, check the attached new log file, i dont know if it's helpful or
>>>> not but, i couldn't find the log with the name you just described.
>>>>
>>> No. Are you using FUSE or libgfapi for accessing the volume? Or is it
>>> NFS?
>>>
>>> -Krutika
>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Respectfully
>>>> *Mahdi A. Mahdi*
>>>>
>>>> -

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Thank you for your email mate.


Yes, im aware of this but, to save costs i chose replica 2, this cluster is all 
flash.

In version 3.7.x i had issues with ping timeout, if one hosts went down for few 
seconds the whole cluster hangs and become unavailable, to avoid this i 
adjusted the ping timeout to 5 seconds.

As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE or 
gfapi) im stuck with NFS for this volume.

The other volume is mounted using gfapi in oVirt cluster.




--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay 
Sent: Sunday, March 19, 2017 2:01:49 PM
To: Mahdi Adnan
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

While I'm still going through the logs, just wanted to point out a couple of 
things:

1. It is recommended that you use 3-way replication (replica count 3) for VM 
store use case
2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.

Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?

Will get back with anything else I might find or more questions if I have any.

-Krutika

On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Thanks mate,

Kindly, check the attachment.


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 10:00:22 AM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

In that case could you share the ganesha-gfapi logs?

-Krutika

On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

I have two volumes, one is mounted using libgfapi for ovirt mount, the other 
one is exported via NFS-Ganesha for VMWare which is the one im testing now.


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Sunday, March 19, 2017 8:02:19 AM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption



On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Kindly, check the attached new log file, i dont know if it's helpful or not 
but, i couldn't find the log with the name you just described.

No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?

-Krutika


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Saturday, March 18, 2017 6:10:40 PM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse mount 
logs? It should be right under /var/log/glusterfs/ directory
named after the mount point name, only hyphenated.

-Krutika

On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hello Krutika,


Kindly, check the attached logs.


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Saturday, March 18, 2017 3:29:03 PM
To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

Hi Mahdi,

Could you attach mount, brick and rebalance logs?

-Krutika

On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hi,

I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a 
volume contains few VMs.
After the completion of rebalance, i have rebooted the VMs, some of ran just 
fine, and others just crashed.
Windows boot to recovery mode and Linux throw xfs errors and does not boot.
I ran the test again and it happened just as the first one, but i have noticed 
only VMs doing disk IOs are affected by this bug.
The VMs in power off mode started fine and even md5 of the disk file did not 
change after the rebalance.

anyone else can confirm this ?


Volume info:

Volume Name: vmware2
Type: Distributed-Replicate
Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
Status: Started
Snapshot Count: 0
Number of Bricks: 22 x 2 = 44
Transport-type: tcp
Bricks:
Brick1: gluster01:/mnt/disk1/vmware2
Brick2: gluster03:/mnt/disk1/vmware2
Brick3: gluster02:/mnt/disk1/vmware2
Brick4: gluster04:/mnt/disk1/vmware2
Brick5: gluster01:/mnt/disk2/vmware2
Brick6: gluster03:/mnt/disk2/vmware2
Brick7: gluster02:/mnt/disk2/vmware2
Brick8: gluster04:/mnt/disk2/vmware2
Brick9: gluster01:/mnt/disk3/vmware2
Brick10: gluster03:/mnt/disk3/vmware2
Brick11: gluster02:/mnt/disk3/vmware2
Brick12: gluster04:/mnt/disk3/vmware2
Brick13: gluster01:/

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

While I'm still going through the logs, just wanted to point out a couple
of things:

1. It is recommended that you use 3-way replication (replica count 3) for
VM store use case
2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.

Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?

Will get back with anything else I might find or more questions if I have
any.

-Krutika

On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan 
wrote:

> Thanks mate,
>
> Kindly, check the attachment.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 10:00:22 AM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> In that case could you share the ganesha-gfapi logs?
>
> -Krutika
>
> On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
> wrote:
>
>> I have two volumes, one is mounted using libgfapi for ovirt mount, the
>> other one is exported via NFS-Ganesha for VMWare which is the one im
>> testing now.
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Sunday, March 19, 2017 8:02:19 AM
>>
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>>
>>
>> On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
>> wrote:
>>
>>> Kindly, check the attached new log file, i dont know if it's helpful or
>>> not but, i couldn't find the log with the name you just described.
>>>
>> No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?
>>
>> -Krutika
>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Saturday, March 18, 2017 6:10:40 PM
>>>
>>> *To:* Mahdi Adnan
>>> *Cc:* gluster-users@gluster.org
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>> mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse
>>> mount logs? It should be right under /var/log/glusterfs/ directory
>>> named after the mount point name, only hyphenated.
>>>
>>> -Krutika
>>>
>>> On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Hello Krutika,
>>>>
>>>>
>>>> Kindly, check the attached logs.
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Respectfully
>>>> *Mahdi A. Mahdi*
>>>>
>>>> --
>>>> *From:* Krutika Dhananjay 
>>>> *Sent:* Saturday, March 18, 2017 3:29:03 PM
>>>> *To:* Mahdi Adnan
>>>> *Cc:* gluster-users@gluster.org
>>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>>
>>>> Hi Mahdi,
>>>>
>>>> Could you attach mount, brick and rebalance logs?
>>>>
>>>> -Krutika
>>>>
>>>> On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have upgraded to Gluster 3.8.10 today and ran the add-brick
>>>>> procedure in a volume contains few VMs.
>>>>> After the completion of rebalance, i have rebooted the VMs, some of
>>>>> ran just fine, and others just crashed.
>>>>> Windows boot to recovery mode and Linux throw xfs errors and does not
>>>>> boot.
>>>>> I ran the test again and it happened just as the first one, but i have
>>>>> noticed only VMs doing disk IOs are affected by this bug.
>>>>> The VMs in power off mode started fine and even md5 of the disk file
>>>>> did not change after the rebalance.
>>>>>
>>>>> anyone else can confirm this ?
>>>>>
>>>>>
>>>>> Volume info:
>>>>>
>>>>> Volume Name: vmware2
>>>>> Type: Distributed-Replicate
>>>>> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
>>>>> Status: Started
>>>>> Snapshot Count: 0
>>>>> Number of Bricks: 22 x 2 = 44
>>>>> Transport-t

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

In that case could you share the ganesha-gfapi logs?

-Krutika

On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan 
wrote:

> I have two volumes, one is mounted using libgfapi for ovirt mount, the
> other one is exported via NFS-Ganesha for VMWare which is the one im
> testing now.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Sunday, March 19, 2017 8:02:19 AM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
>
>
> On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
> wrote:
>
>> Kindly, check the attached new log file, i dont know if it's helpful or
>> not but, i couldn't find the log with the name you just described.
>>
> No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?
>
> -Krutika
>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> --
>> *From:* Krutika Dhananjay 
>> *Sent:* Saturday, March 18, 2017 6:10:40 PM
>>
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse
>> mount logs? It should be right under /var/log/glusterfs/ directory
>> named after the mount point name, only hyphenated.
>>
>> -Krutika
>>
>> On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
>> wrote:
>>
>>> Hello Krutika,
>>>
>>>
>>> Kindly, check the attached logs.
>>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Krutika Dhananjay 
>>> *Sent:* Saturday, March 18, 2017 3:29:03 PM
>>> *To:* Mahdi Adnan
>>> *Cc:* gluster-users@gluster.org
>>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>>
>>> Hi Mahdi,
>>>
>>> Could you attach mount, brick and rebalance logs?
>>>
>>> -Krutika
>>>
>>> On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure
>>>> in a volume contains few VMs.
>>>> After the completion of rebalance, i have rebooted the VMs, some of ran
>>>> just fine, and others just crashed.
>>>> Windows boot to recovery mode and Linux throw xfs errors and does not
>>>> boot.
>>>> I ran the test again and it happened just as the first one, but i have
>>>> noticed only VMs doing disk IOs are affected by this bug.
>>>> The VMs in power off mode started fine and even md5 of the disk file
>>>> did not change after the rebalance.
>>>>
>>>> anyone else can confirm this ?
>>>>
>>>>
>>>> Volume info:
>>>>
>>>> Volume Name: vmware2
>>>> Type: Distributed-Replicate
>>>> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 22 x 2 = 44
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: gluster01:/mnt/disk1/vmware2
>>>> Brick2: gluster03:/mnt/disk1/vmware2
>>>> Brick3: gluster02:/mnt/disk1/vmware2
>>>> Brick4: gluster04:/mnt/disk1/vmware2
>>>> Brick5: gluster01:/mnt/disk2/vmware2
>>>> Brick6: gluster03:/mnt/disk2/vmware2
>>>> Brick7: gluster02:/mnt/disk2/vmware2
>>>> Brick8: gluster04:/mnt/disk2/vmware2
>>>> Brick9: gluster01:/mnt/disk3/vmware2
>>>> Brick10: gluster03:/mnt/disk3/vmware2
>>>> Brick11: gluster02:/mnt/disk3/vmware2
>>>> Brick12: gluster04:/mnt/disk3/vmware2
>>>> Brick13: gluster01:/mnt/disk4/vmware2
>>>> Brick14: gluster03:/mnt/disk4/vmware2
>>>> Brick15: gluster02:/mnt/disk4/vmware2
>>>> Brick16: gluster04:/mnt/disk4/vmware2
>>>> Brick17: gluster01:/mnt/disk5/vmware2
>>>> Brick18: gluster03:/mnt/disk5/vmware2
>>>> Brick19: gluster02:/mnt/disk5/vmware2
>>>> Brick20: gluster04:/mnt/disk5/vmware2
>>>> Brick21: gluster01:/mnt/disk6/vmware2
>>>> Brick22: gluster03:/mnt/disk6/vmware2
>>>> Brick23: gluster02:/mnt/disk6/vmware2
>>>> Brick24: gluster04:/mn

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

I have two volumes, one is mounted using libgfapi for ovirt mount, the other 
one is exported via NFS-Ganesha for VMWare which is the one im testing now.


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay 
Sent: Sunday, March 19, 2017 8:02:19 AM
To: Mahdi Adnan
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption



On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Kindly, check the attached new log file, i dont know if it's helpful or not 
but, i couldn't find the log with the name you just described.

No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?

-Krutika


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Saturday, March 18, 2017 6:10:40 PM

To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse mount 
logs? It should be right under /var/log/glusterfs/ directory
named after the mount point name, only hyphenated.

-Krutika

On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hello Krutika,


Kindly, check the attached logs.


--

Respectfully
Mahdi A. Mahdi


From: Krutika Dhananjay mailto:kdhan...@redhat.com>>
Sent: Saturday, March 18, 2017 3:29:03 PM
To: Mahdi Adnan
Cc: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

Hi Mahdi,

Could you attach mount, brick and rebalance logs?

-Krutika

On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
mailto:mahdi.ad...@outlook.com>> wrote:

Hi,

I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a 
volume contains few VMs.
After the completion of rebalance, i have rebooted the VMs, some of ran just 
fine, and others just crashed.
Windows boot to recovery mode and Linux throw xfs errors and does not boot.
I ran the test again and it happened just as the first one, but i have noticed 
only VMs doing disk IOs are affected by this bug.
The VMs in power off mode started fine and even md5 of the disk file did not 
change after the rebalance.

anyone else can confirm this ?


Volume info:

Volume Name: vmware2
Type: Distributed-Replicate
Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
Status: Started
Snapshot Count: 0
Number of Bricks: 22 x 2 = 44
Transport-type: tcp
Bricks:
Brick1: gluster01:/mnt/disk1/vmware2
Brick2: gluster03:/mnt/disk1/vmware2
Brick3: gluster02:/mnt/disk1/vmware2
Brick4: gluster04:/mnt/disk1/vmware2
Brick5: gluster01:/mnt/disk2/vmware2
Brick6: gluster03:/mnt/disk2/vmware2
Brick7: gluster02:/mnt/disk2/vmware2
Brick8: gluster04:/mnt/disk2/vmware2
Brick9: gluster01:/mnt/disk3/vmware2
Brick10: gluster03:/mnt/disk3/vmware2
Brick11: gluster02:/mnt/disk3/vmware2
Brick12: gluster04:/mnt/disk3/vmware2
Brick13: gluster01:/mnt/disk4/vmware2
Brick14: gluster03:/mnt/disk4/vmware2
Brick15: gluster02:/mnt/disk4/vmware2
Brick16: gluster04:/mnt/disk4/vmware2
Brick17: gluster01:/mnt/disk5/vmware2
Brick18: gluster03:/mnt/disk5/vmware2
Brick19: gluster02:/mnt/disk5/vmware2
Brick20: gluster04:/mnt/disk5/vmware2
Brick21: gluster01:/mnt/disk6/vmware2
Brick22: gluster03:/mnt/disk6/vmware2
Brick23: gluster02:/mnt/disk6/vmware2
Brick24: gluster04:/mnt/disk6/vmware2
Brick25: gluster01:/mnt/disk7/vmware2
Brick26: gluster03:/mnt/disk7/vmware2
Brick27: gluster02:/mnt/disk7/vmware2
Brick28: gluster04:/mnt/disk7/vmware2
Brick29: gluster01:/mnt/disk8/vmware2
Brick30: gluster03:/mnt/disk8/vmware2
Brick31: gluster02:/mnt/disk8/vmware2
Brick32: gluster04:/mnt/disk8/vmware2
Brick33: gluster01:/mnt/disk9/vmware2
Brick34: gluster03:/mnt/disk9/vmware2
Brick35: gluster02:/mnt/disk9/vmware2
Brick36: gluster04:/mnt/disk9/vmware2
Brick37: gluster01:/mnt/disk10/vmware2
Brick38: gluster03:/mnt/disk10/vmware2
Brick39: gluster02:/mnt/disk10/vmware2
Brick40: gluster04:/mnt/disk10/vmware2
Brick41: gluster01:/mnt/disk11/vmware2
Brick42: gluster03:/mnt/disk11/vmware2
Brick43: gluster02:/mnt/disk11/vmware2
Brick44: gluster04:/mnt/disk11/vmware2
Options Reconfigured:
cluster.server-quorum-type: server
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
features.shard: on
cluster.data-self-heal-algorithm: full
features.cache-invalidation: on
ganesha.enable: on
features.shard-block-size: 256MB
client.event-threads: 2
server.event-threads: 2
cluster.favorite-child-policy: size
storage.build-pgfid: off
network.ping-timeout: 5
cluster.enable-shared-storage: enable
nfs-ganesha: enable
cluster.server-quorum-

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Dev Sidious]

Unfortunately, gandalf is precisely right with the point he made on data
consistency in GlusterFS.

> If gluster isn't able to ensure data consistency when doing it's
> primary role, scaling up a storage, i'm sorry but it can't be
> considered "enterprise" ready or production ready.

In my short experience with GlusterFS I have known it to fail PRECISELY
on data consistency (data representation consistency to be more
precise). Namely:

a) files partially or not at all replicated due to
b) errors such as: "Transport endpoint not connected"

with more or less random frequency.

I solved all these by disabling SSL. Since I disabled SSL, the system
APPEARS to be reliable.

To me, a system exhibiting such a behavior is not a solid system.

If it's "production ready" or not, now that's a more subjective topic
and I will leave it to the arm chair computer scientists and the
philosophers.




On 3/19/2017 12:53 AM, Krutika Dhananjay wrote:
> 
> 
> On Sat, Mar 18, 2017 at 11:15 PM, Gandalf Corvotempesta
>  > wrote:
> 
> Krutika, it wasn't an attack directly to you.
> It wasn't an attack at all.
> 
> 
> Gluster is a "SCALE-OUT" software defined storage, the folllowing is
> wrote in the middle of the homepage:
> "GlusterFS is a scalable network filesystem"
> 
> So, scaling a cluster is one of the primary goal of gluster.
> 
> A critical bug that prevent gluster from being scaled without loosing
> data was discovered 1 year ago, and took 1 year to be fixed. 
> 
> 
> If gluster isn't able to ensure data consistency when doing it's
> primary role, scaling up a storage, i'm sorry but it can't be
> considered "enterprise" ready or production ready.
> 
> 
> That's not entirely true. VM use-case is just one of the many workloads
> users
> use Gluster for. I think I've clarified this before. The bug was in
> dht-shard interaction.
> And shard is *only* supported in VM use-case as of today. This means that
> scaling out has been working fine on all but the VM use-case.
> That doesn't mean that Gluster is not production-ready. At least users
> who've deployed Gluster
> in non-VM use-cases haven't complained of add-brick not working in the
> recent past.
> 
> 
> -Krutika
>  
> 
> Maybe SOHO for small offices or home users, but in enterprises, data
> consistency and reliability is the most important thing and gluster
> isn't able to guarantee this even
> doing a very basic routine procedure that should be considered as the
> basis of the whole gluster project (as wrote on gluster's homepage)
> 
> 
> 2017-03-18 14:21 GMT+01:00 Krutika Dhananjay  >:
> >
> >
> > On Sat, Mar 18, 2017 at 3:18 PM, Gandalf Corvotempesta
> >  > wrote:
> >>
> >> 2017-03-18 2:09 GMT+01:00 Lindsay Mathieson
> mailto:lindsay.mathie...@gmail.com>>:
> >> > Concerning, this was supposed to be fixed in 3.8.10
> >>
> >> Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
> 
> >> Now let's see how much time they require to fix another CRITICAL bug.
> >>
> >> I'm really curious.
> >
> >
> > Hey Gandalf!
> >
> > Let's see. There have been plenty of occasions where I've sat and
> worked on
> > users' issues on weekends.
> > And then again, I've got a life too outside of work (or at least I'm
> > supposed to), you know.
> > (And hey you know what! Today is Saturday and I'm sitting here and
> > responding to your mail and collecting information
> > on Mahdi's issue. Nobody asked me to look into it. I checked the
> mail and I
> > had a choice to ignore it and not look into it until Monday.)
> >
> > Is there a genuine problem Mahdi is facing? Without a doubt!
> >
> > Got a constructive feedback to give? Please do.
> > Do you want to give back to the community and help improve
> GlusterFS? There
> > are plenty of ways to do that.
> > One of them is testing out the releases and providing feedback.
> Sharding
> > wouldn't have worked today, if not for Lindsay's timely
> > and regular feedback in several 3.7.x releases.
> >
> > But this kind of criticism doesn't help.
> >
> > Also, spending time on users' issues is only one of the many
> > responsibilities we have as developers.
> > So what you see on mailing lists is just the tip of the iceberg.
> >
> > I have personally tried several times to recreate the add-brick
> bug on 3
> > machines I borrowed from Kaleb. I haven't had success in
> recreating it.
> > Reproducing VM-related bugs, in my experience, wasn't easy. I
> don't use
> > Proxmox. Lindsay and Kevin did. There are a myriad qemu options
> used when
> > launching vms. Different VM management project

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan 
wrote:

> Kindly, check the attached new log file, i dont know if it's helpful or
> not but, i couldn't find the log with the name you just described.
>
No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?

-Krutika

>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Saturday, March 18, 2017 6:10:40 PM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse
> mount logs? It should be right under /var/log/glusterfs/ directory
> named after the mount point name, only hyphenated.
>
> -Krutika
>
> On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
> wrote:
>
>> Hello Krutika,
>>
>>
>> Kindly, check the attached logs.
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> ----------
>> *From:* Krutika Dhananjay 
>> *Sent:* Saturday, March 18, 2017 3:29:03 PM
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users@gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> Hi Mahdi,
>>
>> Could you attach mount, brick and rebalance logs?
>>
>> -Krutika
>>
>> On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
>> wrote:
>>
>>> Hi,
>>>
>>> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure
>>> in a volume contains few VMs.
>>> After the completion of rebalance, i have rebooted the VMs, some of ran
>>> just fine, and others just crashed.
>>> Windows boot to recovery mode and Linux throw xfs errors and does not
>>> boot.
>>> I ran the test again and it happened just as the first one, but i have
>>> noticed only VMs doing disk IOs are affected by this bug.
>>> The VMs in power off mode started fine and even md5 of the disk file did
>>> not change after the rebalance.
>>>
>>> anyone else can confirm this ?
>>>
>>>
>>> Volume info:
>>>
>>> Volume Name: vmware2
>>> Type: Distributed-Replicate
>>> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 22 x 2 = 44
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: gluster01:/mnt/disk1/vmware2
>>> Brick2: gluster03:/mnt/disk1/vmware2
>>> Brick3: gluster02:/mnt/disk1/vmware2
>>> Brick4: gluster04:/mnt/disk1/vmware2
>>> Brick5: gluster01:/mnt/disk2/vmware2
>>> Brick6: gluster03:/mnt/disk2/vmware2
>>> Brick7: gluster02:/mnt/disk2/vmware2
>>> Brick8: gluster04:/mnt/disk2/vmware2
>>> Brick9: gluster01:/mnt/disk3/vmware2
>>> Brick10: gluster03:/mnt/disk3/vmware2
>>> Brick11: gluster02:/mnt/disk3/vmware2
>>> Brick12: gluster04:/mnt/disk3/vmware2
>>> Brick13: gluster01:/mnt/disk4/vmware2
>>> Brick14: gluster03:/mnt/disk4/vmware2
>>> Brick15: gluster02:/mnt/disk4/vmware2
>>> Brick16: gluster04:/mnt/disk4/vmware2
>>> Brick17: gluster01:/mnt/disk5/vmware2
>>> Brick18: gluster03:/mnt/disk5/vmware2
>>> Brick19: gluster02:/mnt/disk5/vmware2
>>> Brick20: gluster04:/mnt/disk5/vmware2
>>> Brick21: gluster01:/mnt/disk6/vmware2
>>> Brick22: gluster03:/mnt/disk6/vmware2
>>> Brick23: gluster02:/mnt/disk6/vmware2
>>> Brick24: gluster04:/mnt/disk6/vmware2
>>> Brick25: gluster01:/mnt/disk7/vmware2
>>> Brick26: gluster03:/mnt/disk7/vmware2
>>> Brick27: gluster02:/mnt/disk7/vmware2
>>> Brick28: gluster04:/mnt/disk7/vmware2
>>> Brick29: gluster01:/mnt/disk8/vmware2
>>> Brick30: gluster03:/mnt/disk8/vmware2
>>> Brick31: gluster02:/mnt/disk8/vmware2
>>> Brick32: gluster04:/mnt/disk8/vmware2
>>> Brick33: gluster01:/mnt/disk9/vmware2
>>> Brick34: gluster03:/mnt/disk9/vmware2
>>> Brick35: gluster02:/mnt/disk9/vmware2
>>> Brick36: gluster04:/mnt/disk9/vmware2
>>> Brick37: gluster01:/mnt/disk10/vmware2
>>> Brick38: gluster03:/mnt/disk10/vmware2
>>> Brick39: gluster02:/mnt/disk10/vmware2
>>> Brick40: gluster04:/mnt/disk10/vmware2
>>> Brick41: gluster01:/mnt/disk11/vmware2
>>> Brick42: gluster03:/mnt/disk11/vmware2
>>> Brick43: gluster02:/mnt/disk11/vmware2
>>> Brick44: gluster04:/mnt/disk11/vmware2
>>> Options Reconfigured:
>&g

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

On Sat, Mar 18, 2017 at 11:15 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Krutika, it wasn't an attack directly to you.
> It wasn't an attack at all.
>

> Gluster is a "SCALE-OUT" software defined storage, the folllowing is
> wrote in the middle of the homepage:
> "GlusterFS is a scalable network filesystem"
>
> So, scaling a cluster is one of the primary goal of gluster.
>
> A critical bug that prevent gluster from being scaled without loosing
> data was discovered 1 year ago, and took 1 year to be fixed.
>

> If gluster isn't able to ensure data consistency when doing it's
> primary role, scaling up a storage, i'm sorry but it can't be
> considered "enterprise" ready or production ready.
>

That's not entirely true. VM use-case is just one of the many workloads
users
use Gluster for. I think I've clarified this before. The bug was in
dht-shard interaction.
And shard is *only* supported in VM use-case as of today. This means that
scaling out has been working fine on all but the VM use-case.
That doesn't mean that Gluster is not production-ready. At least users
who've deployed Gluster
in non-VM use-cases haven't complained of add-brick not working in the
recent past.


-Krutika


> Maybe SOHO for small offices or home users, but in enterprises, data
> consistency and reliability is the most important thing and gluster
> isn't able to guarantee this even
> doing a very basic routine procedure that should be considered as the
> basis of the whole gluster project (as wrote on gluster's homepage)
>
>
> 2017-03-18 14:21 GMT+01:00 Krutika Dhananjay :
> >
> >
> > On Sat, Mar 18, 2017 at 3:18 PM, Gandalf Corvotempesta
> >  wrote:
> >>
> >> 2017-03-18 2:09 GMT+01:00 Lindsay Mathieson <
> lindsay.mathie...@gmail.com>:
> >> > Concerning, this was supposed to be fixed in 3.8.10
> >>
> >> Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
> >> Now let's see how much time they require to fix another CRITICAL bug.
> >>
> >> I'm really curious.
> >
> >
> > Hey Gandalf!
> >
> > Let's see. There have been plenty of occasions where I've sat and worked
> on
> > users' issues on weekends.
> > And then again, I've got a life too outside of work (or at least I'm
> > supposed to), you know.
> > (And hey you know what! Today is Saturday and I'm sitting here and
> > responding to your mail and collecting information
> > on Mahdi's issue. Nobody asked me to look into it. I checked the mail
> and I
> > had a choice to ignore it and not look into it until Monday.)
> >
> > Is there a genuine problem Mahdi is facing? Without a doubt!
> >
> > Got a constructive feedback to give? Please do.
> > Do you want to give back to the community and help improve GlusterFS?
> There
> > are plenty of ways to do that.
> > One of them is testing out the releases and providing feedback. Sharding
> > wouldn't have worked today, if not for Lindsay's timely
> > and regular feedback in several 3.7.x releases.
> >
> > But this kind of criticism doesn't help.
> >
> > Also, spending time on users' issues is only one of the many
> > responsibilities we have as developers.
> > So what you see on mailing lists is just the tip of the iceberg.
> >
> > I have personally tried several times to recreate the add-brick bug on 3
> > machines I borrowed from Kaleb. I haven't had success in recreating it.
> > Reproducing VM-related bugs, in my experience, wasn't easy. I don't use
> > Proxmox. Lindsay and Kevin did. There are a myriad qemu options used when
> > launching vms. Different VM management projects (ovirt/Proxmox) use
> > different defaults for these options. There are too many variables to be
> > considered
> > when debugging or trying to simulate the users' test.
> >
> > It's why I asked for Mahdi's help before 3.8.10 was out for feedback on
> the
> > fix:
> > http://lists.gluster.org/pipermail/gluster-users/2017-
> February/030112.html
> >
> > Alright. That's all I had to say.
> >
> > Happy weekend to you!
> >
> > -Krutika
> >
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> >
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

Krutika, it wasn't an attack directly to you.
It wasn't an attack at all.

Gluster is a "SCALE-OUT" software defined storage, the folllowing is
wrote in the middle of the homepage:
"GlusterFS is a scalable network filesystem"

So, scaling a cluster is one of the primary goal of gluster.

A critical bug that prevent gluster from being scaled without loosing
data was discovered 1 year ago, and took 1 year to be fixed.

If gluster isn't able to ensure data consistency when doing it's
primary role, scaling up a storage, i'm sorry but it can't be
considered "enterprise" ready or production ready.
Maybe SOHO for small offices or home users, but in enterprises, data
consistency and reliability is the most important thing and gluster
isn't able to guarantee this even
doing a very basic routine procedure that should be considered as the
basis of the whole gluster project (as wrote on gluster's homepage)


2017-03-18 14:21 GMT+01:00 Krutika Dhananjay :
>
>
> On Sat, Mar 18, 2017 at 3:18 PM, Gandalf Corvotempesta
>  wrote:
>>
>> 2017-03-18 2:09 GMT+01:00 Lindsay Mathieson :
>> > Concerning, this was supposed to be fixed in 3.8.10
>>
>> Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
>> Now let's see how much time they require to fix another CRITICAL bug.
>>
>> I'm really curious.
>
>
> Hey Gandalf!
>
> Let's see. There have been plenty of occasions where I've sat and worked on
> users' issues on weekends.
> And then again, I've got a life too outside of work (or at least I'm
> supposed to), you know.
> (And hey you know what! Today is Saturday and I'm sitting here and
> responding to your mail and collecting information
> on Mahdi's issue. Nobody asked me to look into it. I checked the mail and I
> had a choice to ignore it and not look into it until Monday.)
>
> Is there a genuine problem Mahdi is facing? Without a doubt!
>
> Got a constructive feedback to give? Please do.
> Do you want to give back to the community and help improve GlusterFS? There
> are plenty of ways to do that.
> One of them is testing out the releases and providing feedback. Sharding
> wouldn't have worked today, if not for Lindsay's timely
> and regular feedback in several 3.7.x releases.
>
> But this kind of criticism doesn't help.
>
> Also, spending time on users' issues is only one of the many
> responsibilities we have as developers.
> So what you see on mailing lists is just the tip of the iceberg.
>
> I have personally tried several times to recreate the add-brick bug on 3
> machines I borrowed from Kaleb. I haven't had success in recreating it.
> Reproducing VM-related bugs, in my experience, wasn't easy. I don't use
> Proxmox. Lindsay and Kevin did. There are a myriad qemu options used when
> launching vms. Different VM management projects (ovirt/Proxmox) use
> different defaults for these options. There are too many variables to be
> considered
> when debugging or trying to simulate the users' test.
>
> It's why I asked for Mahdi's help before 3.8.10 was out for feedback on the
> fix:
> http://lists.gluster.org/pipermail/gluster-users/2017-February/030112.html
>
> Alright. That's all I had to say.
>
> Happy weekend to you!
>
> -Krutika
>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse
mount logs? It should be right under /var/log/glusterfs/ directory
named after the mount point name, only hyphenated.

-Krutika

On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan 
wrote:

> Hello Krutika,
>
>
> Kindly, check the attached logs.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Krutika Dhananjay 
> *Sent:* Saturday, March 18, 2017 3:29:03 PM
> *To:* Mahdi Adnan
> *Cc:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> Hi Mahdi,
>
> Could you attach mount, brick and rebalance logs?
>
> -Krutika
>
> On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
> wrote:
>
>> Hi,
>>
>> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure
>> in a volume contains few VMs.
>> After the completion of rebalance, i have rebooted the VMs, some of ran
>> just fine, and others just crashed.
>> Windows boot to recovery mode and Linux throw xfs errors and does not
>> boot.
>> I ran the test again and it happened just as the first one, but i have
>> noticed only VMs doing disk IOs are affected by this bug.
>> The VMs in power off mode started fine and even md5 of the disk file did
>> not change after the rebalance.
>>
>> anyone else can confirm this ?
>>
>>
>> Volume info:
>>
>> Volume Name: vmware2
>> Type: Distributed-Replicate
>> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 22 x 2 = 44
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster01:/mnt/disk1/vmware2
>> Brick2: gluster03:/mnt/disk1/vmware2
>> Brick3: gluster02:/mnt/disk1/vmware2
>> Brick4: gluster04:/mnt/disk1/vmware2
>> Brick5: gluster01:/mnt/disk2/vmware2
>> Brick6: gluster03:/mnt/disk2/vmware2
>> Brick7: gluster02:/mnt/disk2/vmware2
>> Brick8: gluster04:/mnt/disk2/vmware2
>> Brick9: gluster01:/mnt/disk3/vmware2
>> Brick10: gluster03:/mnt/disk3/vmware2
>> Brick11: gluster02:/mnt/disk3/vmware2
>> Brick12: gluster04:/mnt/disk3/vmware2
>> Brick13: gluster01:/mnt/disk4/vmware2
>> Brick14: gluster03:/mnt/disk4/vmware2
>> Brick15: gluster02:/mnt/disk4/vmware2
>> Brick16: gluster04:/mnt/disk4/vmware2
>> Brick17: gluster01:/mnt/disk5/vmware2
>> Brick18: gluster03:/mnt/disk5/vmware2
>> Brick19: gluster02:/mnt/disk5/vmware2
>> Brick20: gluster04:/mnt/disk5/vmware2
>> Brick21: gluster01:/mnt/disk6/vmware2
>> Brick22: gluster03:/mnt/disk6/vmware2
>> Brick23: gluster02:/mnt/disk6/vmware2
>> Brick24: gluster04:/mnt/disk6/vmware2
>> Brick25: gluster01:/mnt/disk7/vmware2
>> Brick26: gluster03:/mnt/disk7/vmware2
>> Brick27: gluster02:/mnt/disk7/vmware2
>> Brick28: gluster04:/mnt/disk7/vmware2
>> Brick29: gluster01:/mnt/disk8/vmware2
>> Brick30: gluster03:/mnt/disk8/vmware2
>> Brick31: gluster02:/mnt/disk8/vmware2
>> Brick32: gluster04:/mnt/disk8/vmware2
>> Brick33: gluster01:/mnt/disk9/vmware2
>> Brick34: gluster03:/mnt/disk9/vmware2
>> Brick35: gluster02:/mnt/disk9/vmware2
>> Brick36: gluster04:/mnt/disk9/vmware2
>> Brick37: gluster01:/mnt/disk10/vmware2
>> Brick38: gluster03:/mnt/disk10/vmware2
>> Brick39: gluster02:/mnt/disk10/vmware2
>> Brick40: gluster04:/mnt/disk10/vmware2
>> Brick41: gluster01:/mnt/disk11/vmware2
>> Brick42: gluster03:/mnt/disk11/vmware2
>> Brick43: gluster02:/mnt/disk11/vmware2
>> Brick44: gluster04:/mnt/disk11/vmware2
>> Options Reconfigured:
>> cluster.server-quorum-type: server
>> nfs.disable: on
>> performance.readdir-ahead: on
>> transport.address-family: inet
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> features.shard: on
>> cluster.data-self-heal-algorithm: full
>> features.cache-invalidation: on
>> ganesha.enable: on
>> features.shard-block-size: 256MB
>> client.event-threads: 2
>> server.event-threads: 2
>> cluster.favorite-child-policy: size
>> storage.build-pgfid: off
>> network.ping-timeout: 5
>> cluster.enable-shared-storage: enable
>> nfs-ganesha: enable
>> cluster.server-quorum-ratio: 51%
>>
>>
>> Adding bricks:
>> gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2
>> gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2
>> gluster04:/mnt/disk11/vmware2
>>
>>
>> starting fix layout:
>> gluster volume rebalance vmware2 fix-layout start
>>
>> Starting rebalance:
>> gluster volume rebalance vmware2  start
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

On Sat, Mar 18, 2017 at 3:18 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2017-03-18 2:09 GMT+01:00 Lindsay Mathieson :
> > Concerning, this was supposed to be fixed in 3.8.10
>
> Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
> Now let's see how much time they require to fix another CRITICAL bug.
>
> I'm really curious.
>

Hey Gandalf!

Let's see. There have been plenty of occasions where I've sat and worked on
users' issues on weekends.
And then again, I've got a life too outside of work (or at least I'm
supposed to), you know.
(And hey you know what! Today is Saturday and I'm sitting here and
responding to your mail and collecting information
on Mahdi's issue. Nobody asked me to look into it. I checked the mail and I
had a choice to ignore it and not look into it until Monday.)

Is there a genuine problem Mahdi is facing? Without a doubt!

Got a constructive feedback to give? Please do.
Do you want to give back to the community and help improve GlusterFS? There
are plenty of ways to do that.
One of them is testing out the releases and providing feedback. Sharding
wouldn't have worked today, if not for Lindsay's timely
and regular feedback in several 3.7.x releases.

But this kind of criticism doesn't help.

Also, spending time on users' issues is only one of the many
responsibilities we have as developers.
So what you see on mailing lists is just the tip of the iceberg.

I have personally tried several times to recreate the add-brick bug on 3
machines I borrowed from Kaleb. I haven't had success in recreating it.
Reproducing VM-related bugs, in my experience, wasn't easy. I don't use
Proxmox. Lindsay and Kevin did. There are a myriad qemu options used when
launching vms. Different VM management projects (ovirt/Proxmox) use
different defaults for these options. There are too many variables to be
considered
when debugging or trying to simulate the users' test.

It's why I asked for Mahdi's help before 3.8.10 was out for feedback on the
fix:
http://lists.gluster.org/pipermail/gluster-users/2017-February/030112.html

Alright. That's all I had to say.

Happy weekend to you!

-Krutika

___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

Hi Mahdi,

Could you attach mount, brick and rebalance logs?

-Krutika

On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan 
wrote:

> Hi,
>
> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in
> a volume contains few VMs.
> After the completion of rebalance, i have rebooted the VMs, some of ran
> just fine, and others just crashed.
> Windows boot to recovery mode and Linux throw xfs errors and does not boot.
> I ran the test again and it happened just as the first one, but i have
> noticed only VMs doing disk IOs are affected by this bug.
> The VMs in power off mode started fine and even md5 of the disk file did
> not change after the rebalance.
>
> anyone else can confirm this ?
>
>
> Volume info:
>
> Volume Name: vmware2
> Type: Distributed-Replicate
> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 22 x 2 = 44
> Transport-type: tcp
> Bricks:
> Brick1: gluster01:/mnt/disk1/vmware2
> Brick2: gluster03:/mnt/disk1/vmware2
> Brick3: gluster02:/mnt/disk1/vmware2
> Brick4: gluster04:/mnt/disk1/vmware2
> Brick5: gluster01:/mnt/disk2/vmware2
> Brick6: gluster03:/mnt/disk2/vmware2
> Brick7: gluster02:/mnt/disk2/vmware2
> Brick8: gluster04:/mnt/disk2/vmware2
> Brick9: gluster01:/mnt/disk3/vmware2
> Brick10: gluster03:/mnt/disk3/vmware2
> Brick11: gluster02:/mnt/disk3/vmware2
> Brick12: gluster04:/mnt/disk3/vmware2
> Brick13: gluster01:/mnt/disk4/vmware2
> Brick14: gluster03:/mnt/disk4/vmware2
> Brick15: gluster02:/mnt/disk4/vmware2
> Brick16: gluster04:/mnt/disk4/vmware2
> Brick17: gluster01:/mnt/disk5/vmware2
> Brick18: gluster03:/mnt/disk5/vmware2
> Brick19: gluster02:/mnt/disk5/vmware2
> Brick20: gluster04:/mnt/disk5/vmware2
> Brick21: gluster01:/mnt/disk6/vmware2
> Brick22: gluster03:/mnt/disk6/vmware2
> Brick23: gluster02:/mnt/disk6/vmware2
> Brick24: gluster04:/mnt/disk6/vmware2
> Brick25: gluster01:/mnt/disk7/vmware2
> Brick26: gluster03:/mnt/disk7/vmware2
> Brick27: gluster02:/mnt/disk7/vmware2
> Brick28: gluster04:/mnt/disk7/vmware2
> Brick29: gluster01:/mnt/disk8/vmware2
> Brick30: gluster03:/mnt/disk8/vmware2
> Brick31: gluster02:/mnt/disk8/vmware2
> Brick32: gluster04:/mnt/disk8/vmware2
> Brick33: gluster01:/mnt/disk9/vmware2
> Brick34: gluster03:/mnt/disk9/vmware2
> Brick35: gluster02:/mnt/disk9/vmware2
> Brick36: gluster04:/mnt/disk9/vmware2
> Brick37: gluster01:/mnt/disk10/vmware2
> Brick38: gluster03:/mnt/disk10/vmware2
> Brick39: gluster02:/mnt/disk10/vmware2
> Brick40: gluster04:/mnt/disk10/vmware2
> Brick41: gluster01:/mnt/disk11/vmware2
> Brick42: gluster03:/mnt/disk11/vmware2
> Brick43: gluster02:/mnt/disk11/vmware2
> Brick44: gluster04:/mnt/disk11/vmware2
> Options Reconfigured:
> cluster.server-quorum-type: server
> nfs.disable: on
> performance.readdir-ahead: on
> transport.address-family: inet
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> features.shard: on
> cluster.data-self-heal-algorithm: full
> features.cache-invalidation: on
> ganesha.enable: on
> features.shard-block-size: 256MB
> client.event-threads: 2
> server.event-threads: 2
> cluster.favorite-child-policy: size
> storage.build-pgfid: off
> network.ping-timeout: 5
> cluster.enable-shared-storage: enable
> nfs-ganesha: enable
> cluster.server-quorum-ratio: 51%
>
>
> Adding bricks:
> gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2
> gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2
> gluster04:/mnt/disk11/vmware2
>
>
> starting fix layout:
> gluster volume rebalance vmware2 fix-layout start
>
> Starting rebalance:
> gluster volume rebalance vmware2  start
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Although i have tested the patch before it got released, but apparently it 
was't a thorough test.

In Gluster 3.7.x i lost around 100 VMs, now in 3.8.x i just lost a few test VMs.

I hope there will be a fix soon.


--

Respectfully
Mahdi A. Mahdi


From: gluster-users-boun...@gluster.org  on 
behalf of Gandalf Corvotempesta 
Sent: Saturday, March 18, 2017 12:48:33 PM
To: Lindsay Mathieson
Cc: gluster-users
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

2017-03-18 2:09 GMT+01:00 Lindsay Mathieson :
> Concerning, this was supposed to be fixed in 3.8.10

Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
Now let's see how much time they require to fix another CRITICAL bug.

I'm really curious.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Gandalf Corvotempesta]

2017-03-18 2:09 GMT+01:00 Lindsay Mathieson :
> Concerning, this was supposed to be fixed in 3.8.10

Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
Now let's see how much time they require to fix another CRITICAL bug.

I'm really curious.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Lindsay Mathieson]

On 18/03/2017 12:20 AM, Mahdi Adnan wrote:


I have upgraded to Gluster 3.8.10 today and ran the add-brick 
procedure in a volume contains few VMs.


After the completion of rebalance, i have rebooted the VMs, some of 
ran just fine, and others just crashed.


Windows boot to recovery mode and Linux throw xfs errors and does not 
boot.


I ran the test again and it happened just as the first one, but i have 
noticed only VMs doing disk IOs are affected by this bug.


The VMs in power off mode started fine and even md5 of the disk file 
did not change after the rebalance.






Concerning, this was supposed to be fixed in 3.8.10

--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Hi,


I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a 
volume contains few VMs.

After the completion of rebalance, i have rebooted the VMs, some of ran just 
fine, and others just crashed.

Windows boot to recovery mode and Linux throw xfs errors and does not boot.

I ran the test again and it happened just as the first one, but i have noticed 
only VMs doing disk IOs are affected by this bug.

The VMs in power off mode started fine and even md5 of the disk file did not 
change after the rebalance.


anyone else can confirm this ?



Volume info:

Volume Name: vmware2
Type: Distributed-Replicate
Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
Status: Started
Snapshot Count: 0
Number of Bricks: 22 x 2 = 44
Transport-type: tcp
Bricks:
Brick1: gluster01:/mnt/disk1/vmware2
Brick2: gluster03:/mnt/disk1/vmware2
Brick3: gluster02:/mnt/disk1/vmware2
Brick4: gluster04:/mnt/disk1/vmware2
Brick5: gluster01:/mnt/disk2/vmware2
Brick6: gluster03:/mnt/disk2/vmware2
Brick7: gluster02:/mnt/disk2/vmware2
Brick8: gluster04:/mnt/disk2/vmware2
Brick9: gluster01:/mnt/disk3/vmware2
Brick10: gluster03:/mnt/disk3/vmware2
Brick11: gluster02:/mnt/disk3/vmware2
Brick12: gluster04:/mnt/disk3/vmware2
Brick13: gluster01:/mnt/disk4/vmware2
Brick14: gluster03:/mnt/disk4/vmware2
Brick15: gluster02:/mnt/disk4/vmware2
Brick16: gluster04:/mnt/disk4/vmware2
Brick17: gluster01:/mnt/disk5/vmware2
Brick18: gluster03:/mnt/disk5/vmware2
Brick19: gluster02:/mnt/disk5/vmware2
Brick20: gluster04:/mnt/disk5/vmware2
Brick21: gluster01:/mnt/disk6/vmware2
Brick22: gluster03:/mnt/disk6/vmware2
Brick23: gluster02:/mnt/disk6/vmware2
Brick24: gluster04:/mnt/disk6/vmware2
Brick25: gluster01:/mnt/disk7/vmware2
Brick26: gluster03:/mnt/disk7/vmware2
Brick27: gluster02:/mnt/disk7/vmware2
Brick28: gluster04:/mnt/disk7/vmware2
Brick29: gluster01:/mnt/disk8/vmware2
Brick30: gluster03:/mnt/disk8/vmware2
Brick31: gluster02:/mnt/disk8/vmware2
Brick32: gluster04:/mnt/disk8/vmware2
Brick33: gluster01:/mnt/disk9/vmware2
Brick34: gluster03:/mnt/disk9/vmware2
Brick35: gluster02:/mnt/disk9/vmware2
Brick36: gluster04:/mnt/disk9/vmware2
Brick37: gluster01:/mnt/disk10/vmware2
Brick38: gluster03:/mnt/disk10/vmware2
Brick39: gluster02:/mnt/disk10/vmware2
Brick40: gluster04:/mnt/disk10/vmware2
Brick41: gluster01:/mnt/disk11/vmware2
Brick42: gluster03:/mnt/disk11/vmware2
Brick43: gluster02:/mnt/disk11/vmware2
Brick44: gluster04:/mnt/disk11/vmware2
Options Reconfigured:
cluster.server-quorum-type: server
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
features.shard: on
cluster.data-self-heal-algorithm: full
features.cache-invalidation: on
ganesha.enable: on
features.shard-block-size: 256MB
client.event-threads: 2
server.event-threads: 2
cluster.favorite-child-policy: size
storage.build-pgfid: off
network.ping-timeout: 5
cluster.enable-shared-storage: enable
nfs-ganesha: enable
cluster.server-quorum-ratio: 51%


Adding bricks:
gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 
gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 
gluster04:/mnt/disk11/vmware2


starting fix layout:
gluster volume rebalance vmware2 fix-layout start

Starting rebalance:
gluster volume rebalance vmware2  start







--

Respectfully
Mahdi A. Mahdi

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Kevin Lemonnier]

Hi,

Yes, that's an old-ish bug. They fixed it recently, but I guess the fix
hasn't made it's way into a public release yet.
You'll have to get everything up from backups I think, been there :/


On Fri, Mar 17, 2017 at 06:44:14PM +, Mahdi Adnan wrote:
> Hi,
> 
> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a 
> volume contains few VMs.
> After the completion of rebalance, i have rebooted the VMs, some of ran just 
> fine, and others just crashed.
> Windows boot to recovery mode and Linux throw xfs errors and does not boot.
> I ran the test again and it happened just as the first one, but i have 
> noticed only VMs doing disk IOs are affected by this bug.
> The VMs in power off mode started fine and even md5 of the disk file did not 
> change after the rebalance.
> 
> anyone else can confirm this ?
> 
> 
> Volume info:
> 
> Volume Name: vmware2
> Type: Distributed-Replicate
> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 22 x 2 = 44
> Transport-type: tcp
> Bricks:
> Brick1: gluster01:/mnt/disk1/vmware2
> Brick2: gluster03:/mnt/disk1/vmware2
> Brick3: gluster02:/mnt/disk1/vmware2
> Brick4: gluster04:/mnt/disk1/vmware2
> Brick5: gluster01:/mnt/disk2/vmware2
> Brick6: gluster03:/mnt/disk2/vmware2
> Brick7: gluster02:/mnt/disk2/vmware2
> Brick8: gluster04:/mnt/disk2/vmware2
> Brick9: gluster01:/mnt/disk3/vmware2
> Brick10: gluster03:/mnt/disk3/vmware2
> Brick11: gluster02:/mnt/disk3/vmware2
> Brick12: gluster04:/mnt/disk3/vmware2
> Brick13: gluster01:/mnt/disk4/vmware2
> Brick14: gluster03:/mnt/disk4/vmware2
> Brick15: gluster02:/mnt/disk4/vmware2
> Brick16: gluster04:/mnt/disk4/vmware2
> Brick17: gluster01:/mnt/disk5/vmware2
> Brick18: gluster03:/mnt/disk5/vmware2
> Brick19: gluster02:/mnt/disk5/vmware2
> Brick20: gluster04:/mnt/disk5/vmware2
> Brick21: gluster01:/mnt/disk6/vmware2
> Brick22: gluster03:/mnt/disk6/vmware2
> Brick23: gluster02:/mnt/disk6/vmware2
> Brick24: gluster04:/mnt/disk6/vmware2
> Brick25: gluster01:/mnt/disk7/vmware2
> Brick26: gluster03:/mnt/disk7/vmware2
> Brick27: gluster02:/mnt/disk7/vmware2
> Brick28: gluster04:/mnt/disk7/vmware2
> Brick29: gluster01:/mnt/disk8/vmware2
> Brick30: gluster03:/mnt/disk8/vmware2
> Brick31: gluster02:/mnt/disk8/vmware2
> Brick32: gluster04:/mnt/disk8/vmware2
> Brick33: gluster01:/mnt/disk9/vmware2
> Brick34: gluster03:/mnt/disk9/vmware2
> Brick35: gluster02:/mnt/disk9/vmware2
> Brick36: gluster04:/mnt/disk9/vmware2
> Brick37: gluster01:/mnt/disk10/vmware2
> Brick38: gluster03:/mnt/disk10/vmware2
> Brick39: gluster02:/mnt/disk10/vmware2
> Brick40: gluster04:/mnt/disk10/vmware2
> Brick41: gluster01:/mnt/disk11/vmware2
> Brick42: gluster03:/mnt/disk11/vmware2
> Brick43: gluster02:/mnt/disk11/vmware2
> Brick44: gluster04:/mnt/disk11/vmware2
> Options Reconfigured:
> cluster.server-quorum-type: server
> nfs.disable: on
> performance.readdir-ahead: on
> transport.address-family: inet
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> features.shard: on
> cluster.data-self-heal-algorithm: full
> features.cache-invalidation: on
> ganesha.enable: on
> features.shard-block-size: 256MB
> client.event-threads: 2
> server.event-threads: 2
> cluster.favorite-child-policy: size
> storage.build-pgfid: off
> network.ping-timeout: 5
> cluster.enable-shared-storage: enable
> nfs-ganesha: enable
> cluster.server-quorum-ratio: 51%
> 
> 
> Adding bricks:
> gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 
> gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 
> gluster04:/mnt/disk11/vmware2
> 
> 
> starting fix layout:
> gluster volume rebalance vmware2 fix-layout start
> 
> Starting rebalance:
> gluster volume rebalance vmware2  start
> 
> 
> 
> --
> 
> Respectfully
> Mahdi A. Mahdi
> 

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users


-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Mahdi Adnan]

Hi,

I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a 
volume contains few VMs.
After the completion of rebalance, i have rebooted the VMs, some of ran just 
fine, and others just crashed.
Windows boot to recovery mode and Linux throw xfs errors and does not boot.
I ran the test again and it happened just as the first one, but i have noticed 
only VMs doing disk IOs are affected by this bug.
The VMs in power off mode started fine and even md5 of the disk file did not 
change after the rebalance.

anyone else can confirm this ?


Volume info:

Volume Name: vmware2
Type: Distributed-Replicate
Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
Status: Started
Snapshot Count: 0
Number of Bricks: 22 x 2 = 44
Transport-type: tcp
Bricks:
Brick1: gluster01:/mnt/disk1/vmware2
Brick2: gluster03:/mnt/disk1/vmware2
Brick3: gluster02:/mnt/disk1/vmware2
Brick4: gluster04:/mnt/disk1/vmware2
Brick5: gluster01:/mnt/disk2/vmware2
Brick6: gluster03:/mnt/disk2/vmware2
Brick7: gluster02:/mnt/disk2/vmware2
Brick8: gluster04:/mnt/disk2/vmware2
Brick9: gluster01:/mnt/disk3/vmware2
Brick10: gluster03:/mnt/disk3/vmware2
Brick11: gluster02:/mnt/disk3/vmware2
Brick12: gluster04:/mnt/disk3/vmware2
Brick13: gluster01:/mnt/disk4/vmware2
Brick14: gluster03:/mnt/disk4/vmware2
Brick15: gluster02:/mnt/disk4/vmware2
Brick16: gluster04:/mnt/disk4/vmware2
Brick17: gluster01:/mnt/disk5/vmware2
Brick18: gluster03:/mnt/disk5/vmware2
Brick19: gluster02:/mnt/disk5/vmware2
Brick20: gluster04:/mnt/disk5/vmware2
Brick21: gluster01:/mnt/disk6/vmware2
Brick22: gluster03:/mnt/disk6/vmware2
Brick23: gluster02:/mnt/disk6/vmware2
Brick24: gluster04:/mnt/disk6/vmware2
Brick25: gluster01:/mnt/disk7/vmware2
Brick26: gluster03:/mnt/disk7/vmware2
Brick27: gluster02:/mnt/disk7/vmware2
Brick28: gluster04:/mnt/disk7/vmware2
Brick29: gluster01:/mnt/disk8/vmware2
Brick30: gluster03:/mnt/disk8/vmware2
Brick31: gluster02:/mnt/disk8/vmware2
Brick32: gluster04:/mnt/disk8/vmware2
Brick33: gluster01:/mnt/disk9/vmware2
Brick34: gluster03:/mnt/disk9/vmware2
Brick35: gluster02:/mnt/disk9/vmware2
Brick36: gluster04:/mnt/disk9/vmware2
Brick37: gluster01:/mnt/disk10/vmware2
Brick38: gluster03:/mnt/disk10/vmware2
Brick39: gluster02:/mnt/disk10/vmware2
Brick40: gluster04:/mnt/disk10/vmware2
Brick41: gluster01:/mnt/disk11/vmware2
Brick42: gluster03:/mnt/disk11/vmware2
Brick43: gluster02:/mnt/disk11/vmware2
Brick44: gluster04:/mnt/disk11/vmware2
Options Reconfigured:
cluster.server-quorum-type: server
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
features.shard: on
cluster.data-self-heal-algorithm: full
features.cache-invalidation: on
ganesha.enable: on
features.shard-block-size: 256MB
client.event-threads: 2
server.event-threads: 2
cluster.favorite-child-policy: size
storage.build-pgfid: off
network.ping-timeout: 5
cluster.enable-shared-storage: enable
nfs-ganesha: enable
cluster.server-quorum-ratio: 51%


Adding bricks:
gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 
gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 
gluster04:/mnt/disk11/vmware2


starting fix layout:
gluster volume rebalance vmware2 fix-layout start

Starting rebalance:
gluster volume rebalance vmware2  start



--

Respectfully
Mahdi A. Mahdi

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users