Re: [Gluster-users] Problem adding replicated bricks on FreeBSD

[Mark Staudinger]

Mark Staudinger wrote:

Hi Folks,

I'm trying to debug an issue that I've found while attempting to 
qualify GlusterFS for potential distributed storage projects on the 
FreeBSD-11.1 server platform - using the existing package of GlusterFS 
v3.11.1_4


The main issue I've encountered is that I cannot add new bricks while 
setting/increasing the replica count.


If I create a replicated volume "poc" on two hosts, say 
s1:/gluster/1/poc and s2:/gluster/1/poc, the volume is created 
properly and shows replicated status, files are written to both volumes.


If I create a single volume: s1:/gluster/1/poc as a single / 
distributed brick, and then try to run


gluster volume add-brick poc replica 2 s2:/gluster/1/poc

it will always fail (sometimes after a pause, sometimes not.)  The 
only error I'm seeing on the server hosting the new brick, aside from 
the generic "Unable to add bricks" message, is like so:


I [MSGID: 106578] 
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 
0-management: replica-count is set 2
I [MSGID: 106578] 
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 
0-management: type is set 2, need to change it
E [MSGID: 106054] 
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 
0-management: Failed to set extended attribute trusted.add-brick : 
Operation not supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick] 
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 
0-management: Add-brick commit failed.


I was initially using ZFS and noted that ZFS on FreeBSD does not 
support xattr, so I reverted to using UFS as the storage type for the 
brick, and still encounter this behavior.


I also recompiled the port (again, GlusterFS v3.11.1) with the patch 
from https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals 
specifically with xattr handling in FreeBSD.


To recap - I'm able to create any type of volume (2 or 3-way 
replicated or distributed), but I'm unable to add replicated bricks to 
a volume.


I was, however, able to add a second distributed brick ( gluster 
volume add-brick poc s2:/gluster/1/poc ) - so the issue seems specific 
to adding and/or changing the replica count while adding a new brick.


Please let me know if there are any other issues in addition to bug 
#1452961 I should be aware of, or additional log or debug info I can 
provide.




Hi Folks,

As a follow-up to this, and it not making sense that xattrs would not be 
supported on UFS2, I am attaching the 'truss' output of the extattr 
syscalls from two operations.  In both situations, the command was 
initiated on server1, and traced on server2.


The first is a working "create-brick" issued from server1, traced from 
server2: ( gluster volume create poc transport tcp replica 2 
s1:/gluster/1/poc s2:/gluster/1/poc )


extattr_set_link(0x807a55440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x807a55440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'

extattr_set_link(0x807a55440,0x1,0x803b24c17,0x807dfe250,0x10) = 16 (0x10)


And the second is from the not-working situation where the brick from s2 
is being added to the volume ( gluster volume add-brick poc replica 2 
s2:/gluster/1/poc commit )


extattr_set_link(0x80545c440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x80545c440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'

extattr_set_link(0x80545c440,0x1,0x803b24c17,0x8051e9c20,0x10) = 16 (0x10)

Re: [Gluster-users] Problem adding replicated bricks on FreeBSD

[Mark Staudinger]

Kaushal M wrote:

On Thu, Apr 26, 2018 at 9:06 PM Mark Staudinger 
wrote:


Hi Folks,
I'm trying to debug an issue that I've found while attempting to qualify
GlusterFS for potential distributed storage projects on the FreeBSD-11.1
server platform - using the existing package of GlusterFS v3.11.1_4
The main issue I've encountered is that I cannot add new bricks while
setting/increasing the replica count.
If I create a replicated volume "poc" on two hosts, say s1:/gluster/1/poc
and s2:/gluster/1/poc, the volume is created properly and shows replicated
status, files are written to both volumes.
If I create a single volume: s1:/gluster/1/poc as a single / distributed
brick, and then try to run
gluster volume add-brick poc replica 2 s2:/gluster/1/poc
it will always fail (sometimes after a pause, sometimes not.)  The only
error I'm seeing on the server hosting the new brick, aside from the
generic "Unable to add bricks" message, is like so:
I [MSGID: 106578]
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:
replica-count is set 2
I [MSGID: 106578]
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:
type is set 2, need to change it
E [MSGID: 106054]
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:
Failed to set extended attribute trusted.add-brick : Operation not
supported [Operation not supported]

The log here seems to indicate the filesystem on the new brick being added
doesn't seem to support setting xattrs.
Maybe check the new brick again?


E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:
Add-brick commit failed.
I was initially using ZFS and noted that ZFS on FreeBSD does not support
xattr, so I reverted to using UFS as the storage type for the brick, and
still encounter this behavior.
I also recompiled the port (again, GlusterFS v3.11.1) with the patch from
https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
specifically with xattr handling in FreeBSD.
To recap - I'm able to create any type of volume (2 or 3-way replicated or
distributed), but I'm unable to add replicated bricks to a volume.
I was, however, able to add a second distributed brick ( gluster volume
add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding
and/or changing the replica count while adding a new brick.
Please let me know if there are any other issues in addition to bug
#1452961 I should be aware of, or additional log or debug info I can
provide.
Best Regards,
Mark
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Hi Kaushal,

That's what the error message would indicate, but UFS2 does support 
xattr so that's a bit unexpected. Since I'm able to add a distributed 
brick, I can rule out basic setup issues.


All the same, I can see via 'truss'

extattr_set_link(0x8081ff890,0x1,0x803b27886,0x805258c40,0x400) ERR#45 
'Operation not supported’


Is anyone else using a similar setup (FreeBSD >= 11.1-RELEASE with the 
official pkg) that has tested adding bricks in this way? Are there any 
changes to the newfs or mount settings required for this to work that 
aren't obvious?


Cheers,
-=Mark
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Problem adding replicated bricks on FreeBSD

[Kaushal M]

On Thu, Apr 26, 2018 at 9:06 PM Mark Staudinger 
wrote:

> Hi Folks,

> I'm trying to debug an issue that I've found while attempting to qualify
> GlusterFS for potential distributed storage projects on the FreeBSD-11.1
> server platform - using the existing package of GlusterFS v3.11.1_4

> The main issue I've encountered is that I cannot add new bricks while
> setting/increasing the replica count.

> If I create a replicated volume "poc" on two hosts, say s1:/gluster/1/poc
> and s2:/gluster/1/poc, the volume is created properly and shows replicated
> status, files are written to both volumes.

> If I create a single volume: s1:/gluster/1/poc as a single / distributed
> brick, and then try to run

> gluster volume add-brick poc replica 2 s2:/gluster/1/poc

> it will always fail (sometimes after a pause, sometimes not.)  The only
> error I'm seeing on the server hosting the new brick, aside from the
> generic "Unable to add bricks" message, is like so:

> I [MSGID: 106578]
> [glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:
> replica-count is set 2
> I [MSGID: 106578]
> [glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:
> type is set 2, need to change it
> E [MSGID: 106054]
> [glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:
> Failed to set extended attribute trusted.add-brick : Operation not
> supported [Operation not supported]

The log here seems to indicate the filesystem on the new brick being added
doesn't seem to support setting xattrs.
Maybe check the new brick again?

> E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
> 0-glusterd: Unable to add bricks
> E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:
> Add-brick commit failed.

> I was initially using ZFS and noted that ZFS on FreeBSD does not support
> xattr, so I reverted to using UFS as the storage type for the brick, and
> still encounter this behavior.

> I also recompiled the port (again, GlusterFS v3.11.1) with the patch from
> https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
> specifically with xattr handling in FreeBSD.

> To recap - I'm able to create any type of volume (2 or 3-way replicated or
> distributed), but I'm unable to add replicated bricks to a volume.

> I was, however, able to add a second distributed brick ( gluster volume
> add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding
> and/or changing the replica count while adding a new brick.

> Please let me know if there are any other issues in addition to bug
> #1452961 I should be aware of, or additional log or debug info I can
> provide.

> Best Regards,
> Mark
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Problem adding replicated bricks on FreeBSD

[Mark Staudinger]

Hi Folks,

I'm trying to debug an issue that I've found while attempting to qualify  
GlusterFS for potential distributed storage projects on the FreeBSD-11.1  
server platform - using the existing package of GlusterFS v3.11.1_4


The main issue I've encountered is that I cannot add new bricks while  
setting/increasing the replica count.


If I create a replicated volume "poc" on two hosts, say s1:/gluster/1/poc  
and s2:/gluster/1/poc, the volume is created properly and shows replicated  
status, files are written to both volumes.


If I create a single volume: s1:/gluster/1/poc as a single / distributed  
brick, and then try to run


gluster volume add-brick poc replica 2 s2:/gluster/1/poc

it will always fail (sometimes after a pause, sometimes not.)  The only  
error I'm seeing on the server hosting the new brick, aside from the  
generic "Unable to add bricks" message, is like so:


I [MSGID: 106578]  
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:  
replica-count is set 2
I [MSGID: 106578]  
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:  
type is set 2, need to change it
E [MSGID: 106054]  
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:  
Failed to set extended attribute trusted.add-brick : Operation not  
supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]  
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:  
Add-brick commit failed.


I was initially using ZFS and noted that ZFS on FreeBSD does not support  
xattr, so I reverted to using UFS as the storage type for the brick, and  
still encounter this behavior.


I also recompiled the port (again, GlusterFS v3.11.1) with the patch from  
https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals  
specifically with xattr handling in FreeBSD.


To recap - I'm able to create any type of volume (2 or 3-way replicated or  
distributed), but I'm unable to add replicated bricks to a volume.


I was, however, able to add a second distributed brick ( gluster volume  
add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding  
and/or changing the replica count while adding a new brick.


Please let me know if there are any other issues in addition to bug  
#1452961 I should be aware of, or additional log or debug info I can  
provide.


Best Regards,
Mark
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users