Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Mahdi Adnan]

I first encountered this bug about a year ago, and lost more than 100 VM.

Sharding is essential to VM datastores and i think Gluster is't that useful 
without this feature for VMs.

I appreciate all the hard work that the developers putting on this bug, but i 
think a warning in CLI or something would be really helpful for Gluster users 
until the fix is ready.


--

Respectfully
Mahdi A. Mahdi


From: gluster-users-boun...@gluster.org  on 
behalf of Joe Julian 
Sent: Monday, May 1, 2017 6:37:00 PM
To: gluster-us...@gluster.org; Gluster Devel
Subject: [Gluster-users] Don't allow data loss via add-brick (was Re: Add 
single server)


On 04/30/2017 01:13 AM, lemonni...@ulrar.net wrote:
>> So I was a little but luck. If I has all the hardware part, probably i
>> would be firesd after causing data loss by using a software marked as stable
> Yes, we lost our data last year to this bug, and it wasn't a test cluster.
> We still hear from it from our clients to this day.
>
>> Is known that this feature is causing data loss and there is no evidence or
>> no warning in official docs.
>>
> I was (I believe) the first one to run into the bug, it happens and I knew it
> was a risk when installing gluster.
> But since then I didn't see any warnings anywhere except here, I agree
> with you that it should be mentionned in big bold letters on the site.
>
> Might even be worth adding a warning directly on the cli when trying to
> add bricks if sharding is enabled, to make sure no-one will destroy a
> whole cluster for a known bug.

I absolutely agree - or, just disable the ability to add-brick with
sharding enabled. Losing data should never be allowed.
___
Gluster-users mailing list
gluster-us...@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Gandalf Corvotempesta]

2017-05-03 14:22 GMT+02:00 Atin Mukherjee :
> Fix is up @ https://review.gluster.org/#/c/17160/ . The only thing which
> we'd need to decide (and are debating on) is that should we bypass this
> validation with rebalance start force or not. What do others think?

This is a good way to manage bugs. Release notes was updated accordingly and
gluster cli doesn't allow "stupid" operation like rebalance on a
sharded volume that
lead to data loss.

Well done.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Atin Mukherjee]

On Wed, May 3, 2017 at 3:41 PM, Raghavendra Talur  wrote:

> On Tue, May 2, 2017 at 8:46 PM, Nithya Balachandran 
> wrote:
> >
> >
> > On 2 May 2017 at 16:59, Shyam  wrote:
> >>
> >> Talur,
> >>
> >> Please wait for this fix before releasing 3.10.2.
> >>
> >> We will take in the change to either prevent add-brick in
> >> sharded+distrbuted volumes, or throw a warning and force the use of
> --force
> >> to execute this.
>
> Agreed, I have filed bug and marked as blocker for 3.10.2.
> https://bugzilla.redhat.com/show_bug.cgi?id=1447608
>
>
> >>
> > IIUC, the problem is less the add brick operation and more the
> > rebalance/fix-layout. It is those that need to be prevented (as someone
> > could trigger those without an add-brick).
>
> Yes, that problem seems to be with fix-layout/rebalance and not add-brick.
> However, depending on how users have arranged their dir structure, a
> add-brick without a fix-layout might be useless for them.
>
> I also had a look at the code to see if I can do the cli/glusterd
> change myself. However, sharding is enabled just as a xlator and not
> added to glusterd_volinfo_t.
> If someone from dht team could work with glusterd team here it would
> fix the issue faster.
>
> Action item on Nithya/Atin to assign bug 1447608 to someone. I will
> wait for the fix for 3.10.2.
>

Fix is up @ https://review.gluster.org/#/c/17160/ . The only thing which
we'd need to decide (and are debating on) is that should we bypass this
validation with rebalance start force or not. What do others think?


> Thanks,
> Raghavendra Talur
>
> >
> > Nithya
> >>
> >> Let's get a bug going, and not wait for someone to report it in
> bugzilla,
> >> and also mark it as blocking 3.10.2 release tracker bug.
> >>
> >> Thanks,
> >> Shyam
> >>
> >> On 05/02/2017 06:20 AM, Pranith Kumar Karampuri wrote:
> >>>
> >>>
> >>>
> >>> On Tue, May 2, 2017 at 9:16 AM, Pranith Kumar Karampuri
> >>> mailto:pkara...@redhat.com>> wrote:
> >>>
> >>> Yeah it is a good idea. I asked him to raise a bug and we can move
> >>> forward with it.
> >>>
> >>>
> >>> +Raghavendra/Nitya who can help with the fix.
> >>>
> >>>
> >>>
> >>> On Mon, May 1, 2017 at 9:07 PM, Joe Julian  >>> > wrote:
> >>>
> >>>
> >>> On 04/30/2017 01:13 AM, lemonni...@ulrar.net
> >>>  wrote:
> >>>
> >>> So I was a little but luck. If I has all the hardware
> >>> part, probably i
> >>> would be firesd after causing data loss by using a
> >>> software marked as stable
> >>>
> >>> Yes, we lost our data last year to this bug, and it wasn't
> a
> >>> test cluster.
> >>> We still hear from it from our clients to this day.
> >>>
> >>> Is known that this feature is causing data loss and
> >>> there is no evidence or
> >>> no warning in official docs.
> >>>
> >>> I was (I believe) the first one to run into the bug, it
> >>> happens and I knew it
> >>> was a risk when installing gluster.
> >>> But since then I didn't see any warnings anywhere except
> >>> here, I agree
> >>> with you that it should be mentionned in big bold letters
> on
> >>> the site.
> >>>
> >>> Might even be worth adding a warning directly on the cli
> >>> when trying to
> >>> add bricks if sharding is enabled, to make sure no-one will
> >>> destroy a
> >>> whole cluster for a known bug.
> >>>
> >>>
> >>> I absolutely agree - or, just disable the ability to add-brick
> >>> with sharding enabled. Losing data should never be allowed.
> >>> ___
> >>> Gluster-devel mailing list
> >>> Gluster-devel@gluster.org 
> >>> http://lists.gluster.org/mailman/listinfo/gluster-devel
> >>> 
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Pranith
> >>>
> >>> ___
> >>> Gluster-users mailing list
> >>> gluster-us...@gluster.org 
> >>> http://lists.gluster.org/mailman/listinfo/gluster-users
> >>> 
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Pranith
> >>>
> >>>
> >>> ___
> >>> Gluster-devel mailing list
> >>> Gluster-devel@gluster.org
> >>> http://lists.gluster.org/mailman/listinfo/gluster-devel
> >>>
> >> ___
> >> Gluster-devel mailing list
> >> Gluster-devel@gluster.org
> >> http://lists.gluster.org/mailman/listinfo/gluster-devel
> >
> >
>
___
Gluster-devel mailing list
Gluster-devel@glu

Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Raghavendra Talur]

On Tue, May 2, 2017 at 8:46 PM, Nithya Balachandran  wrote:
>
>
> On 2 May 2017 at 16:59, Shyam  wrote:
>>
>> Talur,
>>
>> Please wait for this fix before releasing 3.10.2.
>>
>> We will take in the change to either prevent add-brick in
>> sharded+distrbuted volumes, or throw a warning and force the use of --force
>> to execute this.

Agreed, I have filed bug and marked as blocker for 3.10.2.
https://bugzilla.redhat.com/show_bug.cgi?id=1447608


>>
> IIUC, the problem is less the add brick operation and more the
> rebalance/fix-layout. It is those that need to be prevented (as someone
> could trigger those without an add-brick).

Yes, that problem seems to be with fix-layout/rebalance and not add-brick.
However, depending on how users have arranged their dir structure, a
add-brick without a fix-layout might be useless for them.

I also had a look at the code to see if I can do the cli/glusterd
change myself. However, sharding is enabled just as a xlator and not
added to glusterd_volinfo_t.
If someone from dht team could work with glusterd team here it would
fix the issue faster.

Action item on Nithya/Atin to assign bug 1447608 to someone. I will
wait for the fix for 3.10.2.

Thanks,
Raghavendra Talur

>
> Nithya
>>
>> Let's get a bug going, and not wait for someone to report it in bugzilla,
>> and also mark it as blocking 3.10.2 release tracker bug.
>>
>> Thanks,
>> Shyam
>>
>> On 05/02/2017 06:20 AM, Pranith Kumar Karampuri wrote:
>>>
>>>
>>>
>>> On Tue, May 2, 2017 at 9:16 AM, Pranith Kumar Karampuri
>>> mailto:pkara...@redhat.com>> wrote:
>>>
>>> Yeah it is a good idea. I asked him to raise a bug and we can move
>>> forward with it.
>>>
>>>
>>> +Raghavendra/Nitya who can help with the fix.
>>>
>>>
>>>
>>> On Mon, May 1, 2017 at 9:07 PM, Joe Julian >> > wrote:
>>>
>>>
>>> On 04/30/2017 01:13 AM, lemonni...@ulrar.net
>>>  wrote:
>>>
>>> So I was a little but luck. If I has all the hardware
>>> part, probably i
>>> would be firesd after causing data loss by using a
>>> software marked as stable
>>>
>>> Yes, we lost our data last year to this bug, and it wasn't a
>>> test cluster.
>>> We still hear from it from our clients to this day.
>>>
>>> Is known that this feature is causing data loss and
>>> there is no evidence or
>>> no warning in official docs.
>>>
>>> I was (I believe) the first one to run into the bug, it
>>> happens and I knew it
>>> was a risk when installing gluster.
>>> But since then I didn't see any warnings anywhere except
>>> here, I agree
>>> with you that it should be mentionned in big bold letters on
>>> the site.
>>>
>>> Might even be worth adding a warning directly on the cli
>>> when trying to
>>> add bricks if sharding is enabled, to make sure no-one will
>>> destroy a
>>> whole cluster for a known bug.
>>>
>>>
>>> I absolutely agree - or, just disable the ability to add-brick
>>> with sharding enabled. Losing data should never be allowed.
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org 
>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>> 
>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>>>
>>> ___
>>> Gluster-users mailing list
>>> gluster-us...@gluster.org 
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>> 
>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>>>
>>>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Nithya Balachandran]

On 2 May 2017 at 16:59, Shyam  wrote:

> Talur,
>
> Please wait for this fix before releasing 3.10.2.
>
> We will take in the change to either prevent add-brick in
> sharded+distrbuted volumes, or throw a warning and force the use of --force
> to execute this.
>
> IIUC, the problem is less the add brick operation and more the
rebalance/fix-layout. It is those that need to be prevented (as someone
could trigger those without an add-brick).

Nithya

> Let's get a bug going, and not wait for someone to report it in bugzilla,
> and also mark it as blocking 3.10.2 release tracker bug.
>
> Thanks,
> Shyam
>
> On 05/02/2017 06:20 AM, Pranith Kumar Karampuri wrote:
>
>>
>>
>> On Tue, May 2, 2017 at 9:16 AM, Pranith Kumar Karampuri
>> mailto:pkara...@redhat.com>> wrote:
>>
>> Yeah it is a good idea. I asked him to raise a bug and we can move
>> forward with it.
>>
>>
>> +Raghavendra/Nitya who can help with the fix.
>>
>>
>>
>> On Mon, May 1, 2017 at 9:07 PM, Joe Julian > > wrote:
>>
>>
>> On 04/30/2017 01:13 AM, lemonni...@ulrar.net
>>  wrote:
>>
>> So I was a little but luck. If I has all the hardware
>> part, probably i
>> would be firesd after causing data loss by using a
>> software marked as stable
>>
>> Yes, we lost our data last year to this bug, and it wasn't a
>> test cluster.
>> We still hear from it from our clients to this day.
>>
>> Is known that this feature is causing data loss and
>> there is no evidence or
>> no warning in official docs.
>>
>> I was (I believe) the first one to run into the bug, it
>> happens and I knew it
>> was a risk when installing gluster.
>> But since then I didn't see any warnings anywhere except
>> here, I agree
>> with you that it should be mentionned in big bold letters on
>> the site.
>>
>> Might even be worth adding a warning directly on the cli
>> when trying to
>> add bricks if sharding is enabled, to make sure no-one will
>> destroy a
>> whole cluster for a known bug.
>>
>>
>> I absolutely agree - or, just disable the ability to add-brick
>> with sharding enabled. Losing data should never be allowed.
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>> 
>>
>>
>>
>>
>> --
>> Pranith
>>
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>> 
>>
>>
>>
>>
>> --
>> Pranith
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Shyam]

Talur,

Please wait for this fix before releasing 3.10.2.

We will take in the change to either prevent add-brick in 
sharded+distrbuted volumes, or throw a warning and force the use of 
--force to execute this.


Let's get a bug going, and not wait for someone to report it in 
bugzilla, and also mark it as blocking 3.10.2 release tracker bug.


Thanks,
Shyam

On 05/02/2017 06:20 AM, Pranith Kumar Karampuri wrote:



On Tue, May 2, 2017 at 9:16 AM, Pranith Kumar Karampuri
mailto:pkara...@redhat.com>> wrote:

Yeah it is a good idea. I asked him to raise a bug and we can move
forward with it.


+Raghavendra/Nitya who can help with the fix.



On Mon, May 1, 2017 at 9:07 PM, Joe Julian mailto:j...@julianfamily.org>> wrote:


On 04/30/2017 01:13 AM, lemonni...@ulrar.net
 wrote:

So I was a little but luck. If I has all the hardware
part, probably i
would be firesd after causing data loss by using a
software marked as stable

Yes, we lost our data last year to this bug, and it wasn't a
test cluster.
We still hear from it from our clients to this day.

Is known that this feature is causing data loss and
there is no evidence or
no warning in official docs.

I was (I believe) the first one to run into the bug, it
happens and I knew it
was a risk when installing gluster.
But since then I didn't see any warnings anywhere except
here, I agree
with you that it should be mentionned in big bold letters on
the site.

Might even be worth adding a warning directly on the cli
when trying to
add bricks if sharding is enabled, to make sure no-one will
destroy a
whole cluster for a known bug.


I absolutely agree - or, just disable the ability to add-brick
with sharding enabled. Losing data should never be allowed.
___
Gluster-devel mailing list
Gluster-devel@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-devel





--
Pranith

___
Gluster-users mailing list
gluster-us...@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users





--
Pranith


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Don't allow data loss via add-brick (was Re: Add single server)

[Pranith Kumar Karampuri]

On Tue, May 2, 2017 at 9:16 AM, Pranith Kumar Karampuri  wrote:

> Yeah it is a good idea. I asked him to raise a bug and we can move forward
> with it.
>

+Raghavendra/Nitya who can help with the fix.


>
> On Mon, May 1, 2017 at 9:07 PM, Joe Julian  wrote:
>
>>
>> On 04/30/2017 01:13 AM, lemonni...@ulrar.net wrote:
>>
>>> So I was a little but luck. If I has all the hardware part, probably i
 would be firesd after causing data loss by using a software marked as
 stable

>>> Yes, we lost our data last year to this bug, and it wasn't a test
>>> cluster.
>>> We still hear from it from our clients to this day.
>>>
>>> Is known that this feature is causing data loss and there is no evidence
 or
 no warning in official docs.

 I was (I believe) the first one to run into the bug, it happens and I
>>> knew it
>>> was a risk when installing gluster.
>>> But since then I didn't see any warnings anywhere except here, I agree
>>> with you that it should be mentionned in big bold letters on the site.
>>>
>>> Might even be worth adding a warning directly on the cli when trying to
>>> add bricks if sharding is enabled, to make sure no-one will destroy a
>>> whole cluster for a known bug.
>>>
>>
>> I absolutely agree - or, just disable the ability to add-brick with
>> sharding enabled. Losing data should never be allowed.
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Pranith
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel