Re: [Gluster-devel] About split-brain-resolution.t
- Original Message - > From: "Emmanuel Dreyfus" > To: "Anuradha Talur" , "Pranith Kumar Karampuri" > > Cc: gluster-devel@gluster.org > Sent: Tuesday, March 31, 2015 9:55:24 PM > Subject: Re: [Gluster-devel] About split-brain-resolution.t > > Anuradha Talur wrote: > > > 1) I send a patch today to revert the .t and send it again along with the > > fix. > > Or... > > 2) Can this be failure be ignored till the fix is merged in? > > We can ignore: NetBSD regresssion skips the test for now. Hi, I've sent a patch to fix this issue, it is currently being reviewed : http://review.gluster.org/#/c/10134/ . > > -- > Emmanuel Dreyfus > http://hcpnet.free.fr/pubz > m...@netbsd.org > -- Thanks, Anuradha. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
Anuradha Talur wrote: > 1) I send a patch today to revert the .t and send it again along with the fix. > Or... > 2) Can this be failure be ignored till the fix is merged in? We can ignore: NetBSD regresssion skips the test for now. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
- Original Message - > From: "Pranith Kumar Karampuri" > To: "Emmanuel Dreyfus" > Cc: gluster-devel@gluster.org, "Anuradha Talur" > Sent: Monday, 30 March, 2015 6:09:58 PM > Subject: Re: [Gluster-devel] About split-brain-resolution.t > > > On 03/30/2015 06:01 PM, Emmanuel Dreyfus wrote: > > On Mon, Mar 30, 2015 at 05:44:23PM +0530, Pranith Kumar Karampuri wrote: > >> Problem here is that ' inode_forget' is coming even before it gets to > >> inspect the file. We initially thought we should 'ref' the inode when the > >> user specifies the choice and 'unref' it at the time of 'finalize' or > >> 'abort' of the operation. But that may lead to un-necessary leaks when the > >> user forgets to do either finalize/abort the operation. One way to get > >> around it is to ref the inode for some 'pre-determined time' when 'choice' > >> is given. > > That suggests the design is not finalized ans the implementation ought to > > have unwanted behaviors. IMO the test should be retired until the design > > and implementation is completed. > I will work with Anuradha tomorrow about this one and either send a > patch to remove the .t file or send the fix which makes things right. > > Pranith Hi Emmanuel, I spoke with Pranith about the issue. I'll need 2 days to send a fix. One of the two things can be done : Either.. 1) I send a patch today to revert the .t and send it again along with the fix. Or... 2) Can this be failure be ignored till the fix is merged in? > > > > -- Thanks, Anuradha. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
On 03/30/2015 06:01 PM, Emmanuel Dreyfus wrote: On Mon, Mar 30, 2015 at 05:44:23PM +0530, Pranith Kumar Karampuri wrote: Problem here is that ' inode_forget' is coming even before it gets to inspect the file. We initially thought we should 'ref' the inode when the user specifies the choice and 'unref' it at the time of 'finalize' or 'abort' of the operation. But that may lead to un-necessary leaks when the user forgets to do either finalize/abort the operation. One way to get around it is to ref the inode for some 'pre-determined time' when 'choice' is given. That suggests the design is not finalized ans the implementation ought to have unwanted behaviors. IMO the test should be retired until the design and implementation is completed. I will work with Anuradha tomorrow about this one and either send a patch to remove the .t file or send the fix which makes things right. Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
On Mon, Mar 30, 2015 at 05:44:23PM +0530, Pranith Kumar Karampuri wrote: > Problem here is that ' inode_forget' is coming even before it gets to > inspect the file. We initially thought we should 'ref' the inode when the > user specifies the choice and 'unref' it at the time of 'finalize' or > 'abort' of the operation. But that may lead to un-necessary leaks when the > user forgets to do either finalize/abort the operation. One way to get > around it is to ref the inode for some 'pre-determined time' when 'choice' > is given. That suggests the design is not finalized ans the implementation ought to have unwanted behaviors. IMO the test should be retired until the design and implementation is completed. -- Emmanuel Dreyfus m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
On 03/30/2015 06:34 PM, Emmanuel Dreyfus wrote: Pranith Kumar Karampuri wrote: Since spb_choice is not saved as an attribute for the file on the bricks, it cannot be recovered when the context is reallocated. Either that "save" feature has been forgotten, or going to afr_destroy() here is a bug. Here is the backtrace leading there: This is a known issue :-(. I will need to talk to Anuradha once about this issue. She is not in today. Will let you know about the decision. It seems the thing arise because a threads quits and decide to cleanup stuff. Do we have an idea what this thread is? For the test to pass we need to keep the thread alive. Of course that works around a real problem. Why don't we immediatly clear pending xattr when replica.split-brain-choice is set? That would clear the split brain state. this is how the feature is supposed to work: https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md Basically the choice is given to inspect the 'data' of the file. Then one can finalize the choice which will clear the pending xattrs after resolving the split-brain. Problem here is that ' inode_forget' is coming even before it gets to inspect the file. We initially thought we should 'ref' the inode when the user specifies the choice and 'unref' it at the time of 'finalize' or 'abort' of the operation. But that may lead to un-necessary leaks when the user forgets to do either finalize/abort the operation. One way to get around it is to ref the inode for some 'pre-determined time' when 'choice' is given. Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
Pranith Kumar Karampuri wrote: > > Since spb_choice is not saved as an attribute for the file on the > > bricks, it cannot be recovered when the context is reallocated. Either > > that "save" feature has been forgotten, or going to afr_destroy() here > > is a bug. Here is the backtrace leading there: > This is a known issue :-(. I will need to talk to Anuradha once about > this issue. She is not in today. Will let you know about the decision. It seems the thing arise because a threads quits and decide to cleanup stuff. Do we have an idea what this thread is? For the test to pass we need to keep the thread alive. Of course that works around a real problem. Why don't we immediatly clear pending xattr when replica.split-brain-choice is set? That would clear the split brain state. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
On 03/29/2015 02:23 PM, Emmanuel Dreyfus wrote: Pranith Kumar Karampuri wrote: I see split-brain-resolution.t uses attribute replica.split-brain-choice to choose what replica should be used. This attribute is not in privilegied space (trusted. prefixed). Is it on purpose? Yes, these are used as internal commands to make a choice when file is in split-brain. Here is how the feature is broken on NetBSD: setting split-brain-resolution.t causes afr inode context's spb_choice to be set to the desired source. That works. But when I try to read from the file, spb_choice is -1. This is because in the meantime, the context has been destroyed by afr_destroy() and reallocated with sbp_choice set to default value -1. Since spb_choice is not saved as an attribute for the file on the bricks, it cannot be recovered when the context is reallocated. Either that "save" feature has been forgotten, or going to afr_destroy() here is a bug. Here is the backtrace leading there: This is a known issue :-(. I will need to talk to Anuradha once about this issue. She is not in today. Will let you know about the decision. Pranith 0xbb751313 <_gf_msg_backtrace_nomem+0xc0> at /autobuild/install/lib/libglusterfs.so.0 0xb9b441af at /autobuild/install/lib/glusterfs/3.7dev/xlator/cluster/replicate.so 0xbb771801 at /autobuild/install/lib/libglusterfs.so.0 0xbb7718af at /autobuild/install/lib/libglusterfs.so.0 0xbb773a38 at /autobuild/install/lib/libglusterfs.so.0 0xbb771dd4 at /autobuild/install/lib/libglusterfs.so.0 0xbb277518 at /autobuild/install/lib/glusterfs/3.7dev/xlator/mount/fuse.so 0xbb277639 at /autobuild/install/lib/glusterfs/3.7dev/xlator/mount/fuse.so 0xbb28d70f at /autobuild/install/lib/glusterfs/3.7dev/xlator/mount/fuse.so 0xbb6a2bca <__libc_thr_exit+0x1f8> at /usr/lib/libpthread.so.1 ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
Pranith Kumar Karampuri wrote: > > I see split-brain-resolution.t uses attribute replica.split-brain-choice > > to choose what replica should be used. This attribute is not in > > privilegied space (trusted. prefixed). Is it on purpose? > Yes, these are used as internal commands to make a choice when file is > in split-brain. Here is how the feature is broken on NetBSD: setting split-brain-resolution.t causes afr inode context's spb_choice to be set to the desired source. That works. But when I try to read from the file, spb_choice is -1. This is because in the meantime, the context has been destroyed by afr_destroy() and reallocated with sbp_choice set to default value -1. Since spb_choice is not saved as an attribute for the file on the bricks, it cannot be recovered when the context is reallocated. Either that "save" feature has been forgotten, or going to afr_destroy() here is a bug. Here is the backtrace leading there: 0xbb751313 <_gf_msg_backtrace_nomem+0xc0> at /autobuild/install/lib/libglusterfs.so.0 0xb9b441af at /autobuild/install/lib/glusterfs/3.7dev/xlator/cluster/replicate.so 0xbb771801 at /autobuild/install/lib/libglusterfs.so.0 0xbb7718af at /autobuild/install/lib/libglusterfs.so.0 0xbb773a38 at /autobuild/install/lib/libglusterfs.so.0 0xbb771dd4 at /autobuild/install/lib/libglusterfs.so.0 0xbb277518 at /autobuild/install/lib/glusterfs/3.7dev/xlator/mount/fuse.so 0xbb277639 at /autobuild/install/lib/glusterfs/3.7dev/xlator/mount/fuse.so 0xbb28d70f at /autobuild/install/lib/glusterfs/3.7dev/xlator/mount/fuse.so 0xbb6a2bca <__libc_thr_exit+0x1f8> at /usr/lib/libpthread.so.1 -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
Pranith Kumar Karampuri wrote: > > I see split-brain-resolution.t uses attribute replica.split-brain-choice > > to choose what replica should be used. This attribute is not in > > privilegied space (trusted. prefixed). Is it on purpose? > Yes, these are used as internal commands to make a choice when file is > in split-brain. Sure, but since it is not privilegied namespace, it means unprivilegied user can fiddle with it. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] About split-brain-resolution.t
On 03/28/2015 09:51 PM, Emmanuel Dreyfus wrote: Hi I see split-brain-resolution.t uses attribute replica.split-brain-choice to choose what replica should be used. This attribute is not in privilegied space (trusted. prefixed). Is it on purpose? Yes, these are used as internal commands to make a choice when file is in split-brain. While there, just in case someone has an idea on this: on NetBSD setting this attribute is ignored. The brick gets some request but does not set the attribute. Not yet investigated, but just in case someone has an idea... Yes it is treated as command and not set on the file. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
