Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Ankireddypalle Reddy]

Thanks Pranith. We are waiting for a downtime on our production setup. Will 
update you once we are able to apply this on our production setup.

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Thursday, July 13, 2017 4:13 AM
To: Ankireddypalle Reddy
Cc: Sanoj Unnikrishnan; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost

Ram,
  I sent https://review.gluster.org/17765 to fix the possibility in bulk 
removexattr. But I am not sure if this is indeed the reason for this issue.

On Mon, Jul 10, 2017 at 6:30 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Thanks for the swift turn around. Will try this out and let you know.

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Monday, July 10, 2017 8:31 AM
To: Sanoj Unnikrishnan
Cc: Ankireddypalle Reddy; Gluster Devel 
(gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>); 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>

Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost

Ram,
  If you see it again, you can use this. I am going to send out a patch for 
the code path which can lead to removal of gfid/volume-id tomorrow.

On Mon, Jul 10, 2017 at 5:19 PM, Sanoj Unnikrishnan 
mailto:sunni...@redhat.com>> wrote:
Please use the systemtap 
script(https://paste.fedoraproject.org/paste/EGDa0ErwX0LV3y-gBYpfNA) to check 
which process is invoking remove xattr calls.
It prints the pid, tid and arguments of all removexattr calls.
I have checked for these fops at the protocol/client and posix translators.

To run the script ..
1) install systemtap and dependencies.
2) install glusterfs-debuginfo
3) change the path of the translator in the systemtap script to appropriate 
values for your system
(change "/usr/lib64/glusterfs/3.12dev/xlator/protocol/client.so" and 
"/usr/lib64/glusterfs/3.12dev/xlator/storage/posix.so")
4) run the script as follows
#stap -v fop_trace.stp

The o/p would look like these .. additionally arguments will also be dumped if 
glusterfs-debuginfo is also installed (i had not done it here.)
pid-958: 0 glusterfsd(3893):->posix_setxattr
pid-958:47 glusterfsd(3893):<-posix_setxattr
pid-966: 0 glusterfsd(5033):->posix_setxattr
pid-966:57 glusterfsd(5033):<-posix_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423:37 glusterfs(1431):<-client_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423:41 glusterfs(1431):<-client_setxattr
Regards,
Sanoj



On Mon, Jul 10, 2017 at 2:56 PM, Sanoj Unnikrishnan 
mailto:sunni...@redhat.com>> wrote:
@ pranith , yes . we can get the pid on all removexattr call and also print the 
backtrace of the glusterfsd process when trigerring removing xattr.
I will write the script and reply back.

On Sat, Jul 8, 2017 at 7:06 AM, Pranith Kumar Karampuri 
mailto:pkara...@redhat.com>> wrote:
Ram,
   As per the code, self-heal was the only candidate which *can* do it. 
Could you check logs of self-heal daemon and the mount to check if there are 
any metadata heals on root?
+Sanoj
Sanoj,
   Is there any systemtap script we can use to detect which process is 
removing these xattrs?

On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
We lost the attributes on all the bricks on servers glusterfs2 and glusterfs3 
again.

[root@glusterfs2 Log_Files]# gluster volume info

Volume Name: StoragePool
Type: Distributed-Disperse
Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
Status: Started
Number of Bricks: 20 x (2 + 1) = 60
Transport-type: tcp
Bricks:
Brick1: glusterfs1sds:/ws/disk1/ws_brick
Brick2: glusterfs2sds:/ws/disk1/ws_brick
Brick3: glusterfs3sds:/ws/disk1/ws_brick
Brick4: glusterfs1sds:/ws/disk2/ws_brick
Brick5: glusterfs2sds:/ws/disk2/ws_brick
Brick6: glusterfs3sds:/ws/disk2/ws_brick
Brick7: glusterfs1sds:/ws/disk3/ws_brick
Brick8: glusterfs2sds:/ws/disk3/ws_brick
Brick9: glusterfs3sds:/ws/disk3/ws_brick
Brick10: glusterfs1sds:/ws/disk4/ws_brick
Brick11: glusterfs2sds:/ws/disk4/ws_brick
Brick12: glusterfs3sds:/ws/disk4/ws_brick
Brick13: glusterfs1sds:/ws/disk5/ws_brick
Brick14: glusterfs2sds:/ws/disk5/ws_brick
Brick15: glusterfs3sds:/ws/disk5/ws_brick
Brick16: glusterfs1sds:/ws/disk6/ws_brick
Brick17: glusterfs2sds:/ws/disk6/ws_brick
Brick18: glusterfs3sds:/ws/disk6/ws_brick
Brick19: glusterfs1sds:/ws/disk7/ws_brick
Brick20: glusterfs2sds:/ws/disk7/ws_brick
Brick21: glusterfs3sds:/ws/disk7/ws_brick
Brick22: glusterfs1sds:/ws/disk8/ws_brick
Brick23: glusterfs2sds:/ws/disk8/ws_brick
Brick24: glusterfs3sds:/ws/disk8/ws_brick
Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
B

Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Ankireddypalle Reddy]

Thanks for the swift turn around. Will try this out and let you know.

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Monday, July 10, 2017 8:31 AM
To: Sanoj Unnikrishnan
Cc: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost

Ram,
  If you see it again, you can use this. I am going to send out a patch for 
the code path which can lead to removal of gfid/volume-id tomorrow.

On Mon, Jul 10, 2017 at 5:19 PM, Sanoj Unnikrishnan 
mailto:sunni...@redhat.com>> wrote:
Please use the systemtap 
script(https://paste.fedoraproject.org/paste/EGDa0ErwX0LV3y-gBYpfNA) to check 
which process is invoking remove xattr calls.
It prints the pid, tid and arguments of all removexattr calls.
I have checked for these fops at the protocol/client and posix translators.

To run the script ..
1) install systemtap and dependencies.
2) install glusterfs-debuginfo
3) change the path of the translator in the systemtap script to appropriate 
values for your system
(change "/usr/lib64/glusterfs/3.12dev/xlator/protocol/client.so" and 
"/usr/lib64/glusterfs/3.12dev/xlator/storage/posix.so")
4) run the script as follows
#stap -v fop_trace.stp

The o/p would look like these .. additionally arguments will also be dumped if 
glusterfs-debuginfo is also installed (i had not done it here.)
pid-958: 0 glusterfsd(3893):->posix_setxattr
pid-958:47 glusterfsd(3893):<-posix_setxattr
pid-966: 0 glusterfsd(5033):->posix_setxattr
pid-966:57 glusterfsd(5033):<-posix_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423:37 glusterfs(1431):<-client_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423:41 glusterfs(1431):<-client_setxattr
Regards,
Sanoj



On Mon, Jul 10, 2017 at 2:56 PM, Sanoj Unnikrishnan 
mailto:sunni...@redhat.com>> wrote:
@ pranith , yes . we can get the pid on all removexattr call and also print the 
backtrace of the glusterfsd process when trigerring removing xattr.
I will write the script and reply back.

On Sat, Jul 8, 2017 at 7:06 AM, Pranith Kumar Karampuri 
mailto:pkara...@redhat.com>> wrote:
Ram,
   As per the code, self-heal was the only candidate which *can* do it. 
Could you check logs of self-heal daemon and the mount to check if there are 
any metadata heals on root?

+Sanoj
Sanoj,
   Is there any systemtap script we can use to detect which process is 
removing these xattrs?

On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
We lost the attributes on all the bricks on servers glusterfs2 and glusterfs3 
again.

[root@glusterfs2 Log_Files]# gluster volume info

Volume Name: StoragePool
Type: Distributed-Disperse
Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
Status: Started
Number of Bricks: 20 x (2 + 1) = 60
Transport-type: tcp
Bricks:
Brick1: glusterfs1sds:/ws/disk1/ws_brick
Brick2: glusterfs2sds:/ws/disk1/ws_brick
Brick3: glusterfs3sds:/ws/disk1/ws_brick
Brick4: glusterfs1sds:/ws/disk2/ws_brick
Brick5: glusterfs2sds:/ws/disk2/ws_brick
Brick6: glusterfs3sds:/ws/disk2/ws_brick
Brick7: glusterfs1sds:/ws/disk3/ws_brick
Brick8: glusterfs2sds:/ws/disk3/ws_brick
Brick9: glusterfs3sds:/ws/disk3/ws_brick
Brick10: glusterfs1sds:/ws/disk4/ws_brick
Brick11: glusterfs2sds:/ws/disk4/ws_brick
Brick12: glusterfs3sds:/ws/disk4/ws_brick
Brick13: glusterfs1sds:/ws/disk5/ws_brick
Brick14: glusterfs2sds:/ws/disk5/ws_brick
Brick15: glusterfs3sds:/ws/disk5/ws_brick
Brick16: glusterfs1sds:/ws/disk6/ws_brick
Brick17: glusterfs2sds:/ws/disk6/ws_brick
Brick18: glusterfs3sds:/ws/disk6/ws_brick
Brick19: glusterfs1sds:/ws/disk7/ws_brick
Brick20: glusterfs2sds:/ws/disk7/ws_brick
Brick21: glusterfs3sds:/ws/disk7/ws_brick
Brick22: glusterfs1sds:/ws/disk8/ws_brick
Brick23: glusterfs2sds:/ws/disk8/ws_brick
Brick24: glusterfs3sds:/ws/disk8/ws_brick
Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
Brick28: glusterfs4sds.commvault.com:/ws/disk10/ws_brick
Brick29: glusterfs5sds.commvault.com:/ws/disk10/ws_brick
Brick30: glusterfs6sds.commvault.com:/ws/disk10/ws_brick
Brick31: glusterfs4sds.commvault.com:/ws/disk11/ws_brick
Brick32: glusterfs5sds.commvault.com:/ws/disk11/ws_brick
Brick33: glusterfs6sds.commvault.com:/ws/disk11/ws_brick
Brick34: glusterfs4sds.commvault.com:/ws/disk12/ws_brick
Brick35: glusterfs5sds.commvault.com:/ws/disk12/ws_brick
Brick36: glusterfs6sds.commvault.com:/ws/disk12/ws_brick
Brick37: glusterfs4sds.commvault.com:/ws/disk2/ws_brick
Brick38: glusterfs5sds.commvault.com:/ws/disk2/ws_brick
Brick39: glusterfs6sds.commvault.com:/ws/disk2/ws_brick
Brick40: glusterfs4sds.commvault.com:/ws/disk3/ws_brick
Brick41: glusterfs5sds.commvault.com:/ws/disk3/ws_brick
Brick42: glusterfs6sds.commvault.c

Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Ankireddypalle Reddy]

We lost the attributes on all the bricks on servers glusterfs2 and glusterfs3 
again.

[root@glusterfs2 Log_Files]# gluster volume info

Volume Name: StoragePool
Type: Distributed-Disperse
Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
Status: Started
Number of Bricks: 20 x (2 + 1) = 60
Transport-type: tcp
Bricks:
Brick1: glusterfs1sds:/ws/disk1/ws_brick
Brick2: glusterfs2sds:/ws/disk1/ws_brick
Brick3: glusterfs3sds:/ws/disk1/ws_brick
Brick4: glusterfs1sds:/ws/disk2/ws_brick
Brick5: glusterfs2sds:/ws/disk2/ws_brick
Brick6: glusterfs3sds:/ws/disk2/ws_brick
Brick7: glusterfs1sds:/ws/disk3/ws_brick
Brick8: glusterfs2sds:/ws/disk3/ws_brick
Brick9: glusterfs3sds:/ws/disk3/ws_brick
Brick10: glusterfs1sds:/ws/disk4/ws_brick
Brick11: glusterfs2sds:/ws/disk4/ws_brick
Brick12: glusterfs3sds:/ws/disk4/ws_brick
Brick13: glusterfs1sds:/ws/disk5/ws_brick
Brick14: glusterfs2sds:/ws/disk5/ws_brick
Brick15: glusterfs3sds:/ws/disk5/ws_brick
Brick16: glusterfs1sds:/ws/disk6/ws_brick
Brick17: glusterfs2sds:/ws/disk6/ws_brick
Brick18: glusterfs3sds:/ws/disk6/ws_brick
Brick19: glusterfs1sds:/ws/disk7/ws_brick
Brick20: glusterfs2sds:/ws/disk7/ws_brick
Brick21: glusterfs3sds:/ws/disk7/ws_brick
Brick22: glusterfs1sds:/ws/disk8/ws_brick
Brick23: glusterfs2sds:/ws/disk8/ws_brick
Brick24: glusterfs3sds:/ws/disk8/ws_brick
Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
Brick28: glusterfs4sds.commvault.com:/ws/disk10/ws_brick
Brick29: glusterfs5sds.commvault.com:/ws/disk10/ws_brick
Brick30: glusterfs6sds.commvault.com:/ws/disk10/ws_brick
Brick31: glusterfs4sds.commvault.com:/ws/disk11/ws_brick
Brick32: glusterfs5sds.commvault.com:/ws/disk11/ws_brick
Brick33: glusterfs6sds.commvault.com:/ws/disk11/ws_brick
Brick34: glusterfs4sds.commvault.com:/ws/disk12/ws_brick
Brick35: glusterfs5sds.commvault.com:/ws/disk12/ws_brick
Brick36: glusterfs6sds.commvault.com:/ws/disk12/ws_brick
Brick37: glusterfs4sds.commvault.com:/ws/disk2/ws_brick
Brick38: glusterfs5sds.commvault.com:/ws/disk2/ws_brick
Brick39: glusterfs6sds.commvault.com:/ws/disk2/ws_brick
Brick40: glusterfs4sds.commvault.com:/ws/disk3/ws_brick
Brick41: glusterfs5sds.commvault.com:/ws/disk3/ws_brick
Brick42: glusterfs6sds.commvault.com:/ws/disk3/ws_brick
Brick43: glusterfs4sds.commvault.com:/ws/disk4/ws_brick
Brick44: glusterfs5sds.commvault.com:/ws/disk4/ws_brick
Brick45: glusterfs6sds.commvault.com:/ws/disk4/ws_brick
Brick46: glusterfs4sds.commvault.com:/ws/disk5/ws_brick
Brick47: glusterfs5sds.commvault.com:/ws/disk5/ws_brick
Brick48: glusterfs6sds.commvault.com:/ws/disk5/ws_brick
Brick49: glusterfs4sds.commvault.com:/ws/disk6/ws_brick
Brick50: glusterfs5sds.commvault.com:/ws/disk6/ws_brick
Brick51: glusterfs6sds.commvault.com:/ws/disk6/ws_brick
Brick52: glusterfs4sds.commvault.com:/ws/disk7/ws_brick
Brick53: glusterfs5sds.commvault.com:/ws/disk7/ws_brick
Brick54: glusterfs6sds.commvault.com:/ws/disk7/ws_brick
Brick55: glusterfs4sds.commvault.com:/ws/disk8/ws_brick
Brick56: glusterfs5sds.commvault.com:/ws/disk8/ws_brick
Brick57: glusterfs6sds.commvault.com:/ws/disk8/ws_brick
Brick58: glusterfs4sds.commvault.com:/ws/disk9/ws_brick
Brick59: glusterfs5sds.commvault.com:/ws/disk9/ws_brick
Brick60: glusterfs6sds.commvault.com:/ws/disk9/ws_brick
Options Reconfigured:
performance.readdir-ahead: on
diagnostics.client-log-level: INFO
auth.allow: 
glusterfs1sds,glusterfs2sds,glusterfs3sds,glusterfs4sds.commvault.com,glusterfs5sds.commvault.com,glusterfs6sds.commvault.com

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Friday, July 07, 2017 12:15 PM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org
Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost



On Fri, Jul 7, 2017 at 9:25 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
3.7.19

These are the only callers for removexattr and only _posix_remove_xattr has the 
potential to do removexattr as posix_removexattr already makes sure that it is 
not gfid/volume-id. And surprise surprise _posix_remove_xattr happens only from 
healing code of afr/ec. And this can only happen if the source brick doesn't 
have gfid, which doesn't seem to match with the situation you explained.

   #   line  filename / context / line
   1   1234  xlators/mgmt/glusterd/src/glusterd-quota.c 
<>
 ret = sys_lremovexattr (abspath, QUOTA_LIMIT_KEY);
   2   1243  xlators/mgmt/glusterd/src/glusterd-quota.c 
<>
 ret = sys_lremovexattr (abspath, QUOTA_LIMIT_OBJECTS_KEY);
   3   6102  xlators/mgmt/glusterd/src/glusterd-utils.c 
<>
 sys_lremovexattr (path, "trusted.glusterfs.test");
   4 80  xlators/storage/posix/src/posix-handle.h <>
 op_ret = sys_lremovexattr (path, key); \
   5   5026  xlators/storage/posix/src/posix.c <<_pos

Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Ankireddypalle Reddy]

3.7.19

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Friday, July 07, 2017 11:54 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org
Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost



On Fri, Jul 7, 2017 at 9:20 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Pranith,
 Thanks for looking in to the issue. The bricks were mounted 
after the reboot. One more thing that I noticed was when the attributes were 
manually set when glusterd was up then on starting the volume the attributes 
were again lost. Had to stop glusterd set attributes and then start glusterd. 
After that the volume start succeeded.

Which version is this?


Thanks and Regards,
Ram

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Friday, July 07, 2017 11:46 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel 
(gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>); 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost

Did anything special happen on these two bricks? It can't happen in the I/O 
path:
posix_removexattr() has:
  0 if (!strcmp (GFID_XATTR_KEY, name)) {
  1 gf_msg (this->name, GF_LOG_WARNING, 0, 
P_MSG_XATTR_NOT_REMOVED,
  2 "Remove xattr called on gfid for file %s", 
real_path);
  3 op_ret = -1;
  4 goto out;
  5 }
  6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name)) {
  7 gf_msg (this->name, GF_LOG_WARNING, 0, 
P_MSG_XATTR_NOT_REMOVED,
  8 "Remove xattr called on volume-id for file %s",
  9 real_path);
 10 op_ret = -1;
 11 goto out;
 12 }
I just found that op_errno is not set correctly, but it can't happen in the I/O 
path, so self-heal/rebalance are off the hook.
I also grepped for any removexattr of trusted.gfid from glusterd and didn't 
find any.
So one thing that used to happen was that sometimes when machines reboot, the 
brick mounts wouldn't happen and this would lead to absence of both 
trusted.gfid and volume-id. So at the moment this is my wild guess.


On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
   We faced an issue in the production today. We had to stop the volume and 
reboot all the servers in the cluster.  Once the servers rebooted starting of 
the volume failed because the following extended attributes were not present on 
all the bricks on 2 servers.

1)  trusted.gfid

2)  trusted.glusterfs.volume-id

We had to manually set these extended attributes to start the volume.  Are 
there any such known issues.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-devel mailing list
gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-devel



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Ankireddypalle Reddy]

Pranith,
 Thanks for looking in to the issue. The bricks were mounted 
after the reboot. One more thing that I noticed was when the attributes were 
manually set when glusterd was up then on starting the volume the attributes 
were again lost. Had to stop glusterd set attributes and then start glusterd. 
After that the volume start succeeded.

Thanks and Regards,
Ram

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Friday, July 07, 2017 11:46 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org
Subject: Re: [Gluster-devel] gfid and volume-id extended attributes lost

Did anything special happen on these two bricks? It can't happen in the I/O 
path:
posix_removexattr() has:
  0 if (!strcmp (GFID_XATTR_KEY, name)) {
  1 gf_msg (this->name, GF_LOG_WARNING, 0, 
P_MSG_XATTR_NOT_REMOVED,
  2 "Remove xattr called on gfid for file %s", 
real_path);
  3 op_ret = -1;
  4 goto out;
  5 }
  6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name)) {
  7 gf_msg (this->name, GF_LOG_WARNING, 0, 
P_MSG_XATTR_NOT_REMOVED,
  8 "Remove xattr called on volume-id for file %s",
  9 real_path);
 10 op_ret = -1;
 11 goto out;
 12 }
I just found that op_errno is not set correctly, but it can't happen in the I/O 
path, so self-heal/rebalance are off the hook.
I also grepped for any removexattr of trusted.gfid from glusterd and didn't 
find any.
So one thing that used to happen was that sometimes when machines reboot, the 
brick mounts wouldn't happen and this would lead to absence of both 
trusted.gfid and volume-id. So at the moment this is my wild guess.


On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
   We faced an issue in the production today. We had to stop the volume and 
reboot all the servers in the cluster.  Once the servers rebooted starting of 
the volume failed because the following extended attributes were not present on 
all the bricks on 2 servers.

1)  trusted.gfid

2)  trusted.glusterfs.volume-id

We had to manually set these extended attributes to start the volume.  Are 
there any such known issues.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-devel mailing list
gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-devel



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gfid and volume-id extended attributes lost

[Ankireddypalle Reddy]

Hi,
   We faced an issue in the production today. We had to stop the volume and 
reboot all the servers in the cluster.  Once the servers rebooted starting of 
the volume failed because the following extended attributes were not present on 
all the bricks on 2 servers.

1)  trusted.gfid

2)  trusted.glusterfs.volume-id

We had to manually set these extended attributes to start the volume.  Are 
there any such known issues.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Enabling shard on EC

[Ankireddypalle Reddy]

Krutika,
  Thanks for letting me know that this is already in works. The 
use case is to provide a Scaleout NAS for large media files.

Thanks and Regards,
Ram
From: Krutika Dhananjay [mailto:kdhan...@redhat.com]
Sent: Friday, May 05, 2017 2:24 AM
To: Pranith Kumar Karampuri
Cc: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] Enabling shard on EC

Hi,
Work is in progress for this (and going a bit slow at the moment because of 
other priorities).

At the moment we support sharding only for VM image store use-case - most 
common large file + single writer use case we know of.
Just curious, what is the use case where you want to use shard+EC?
-Krutika

On Thu, May 4, 2017 at 4:05 PM, Pranith Kumar Karampuri 
mailto:pkara...@redhat.com>> wrote:
+Krutika
Krutika started work on this. But it is very long term. Not a simple thing to 
do.

On Thu, May 4, 2017 at 3:53 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Pranith,
 Thanks. Is there any work in progress to add this support.

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Thursday, May 04, 2017 6:17 AM

To: Ankireddypalle Reddy
Cc: Gluster Devel 
(gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>); 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-devel] Enabling shard on EC



On Thu, May 4, 2017 at 3:43 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Pranith,
 Thanks. Does it mean that a given file can be written by only 
one client at a time. If multiple clients try to access the file in write mode, 
does it lead to any kind of data inconsistencies.

We only tested it for single writer cases such as VM usecases. We need to bring 
in transaction framework for sharding to work with multiple writers.


Thanks and Regards,
Ram
From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Thursday, May 04, 2017 6:07 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel 
(gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>); 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-devel] Enabling shard on EC

It is never been tested. That said, I don't see any missing pieces that we know 
of for it to work. Please note that sharding works only for single writer cases 
at the moment. Do let us know if you find any problems and we will fix them.

On Wed, May 3, 2017 at 2:17 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
  Are there any known negatives of enabling shard on EC. Is this a 
recommended configuration?.

Thanks and Regards,
Ram


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-devel mailing list
gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-devel



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**


--
Pranith

***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] libgfapi access to snapshot volume

[Ankireddypalle Reddy]

Vijay,
  Thanks for pointing to the relevant source code.

Sent from my iPhone

On May 4, 2017, at 6:32 PM, Vijay Bellur 
mailto:vbel...@redhat.com>> wrote:


On Thu, May 4, 2017 at 9:12 AM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
   Can glusterfs snapshot volume be accessed through libgfapi.


Yes, activated snapshots can be accessed through libgfapi. User serviceable 
snapshots in Gluster makes use of gfapi to access activated snapshots. 
xlators/features/snapview-server contains code that accesses snapshots through 
libgfapi.

Regards,
Vijay


Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-devel mailing list
gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-devel

***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] libgfapi access to snapshot volume

[Ankireddypalle Reddy]

Rafi,
   Thanks. Will change the volume name to the notation that you 
provided and try it out.

Thanks and Regards,
Ram
From: Mohammed Rafi K C [mailto:rkavu...@redhat.com]
Sent: Thursday, May 04, 2017 11:11 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] libgfapi access to snapshot volume


Hi Ram,



You can access snapshot through libgfapi, it is just that the volname will 
become something like /snaps// . I can give you some example 
programs if you have any trouble in doing so.



Or you can use uss feature to use snapshot through main volume via libgfapi (it 
is also uses the above method internally).



Regards

Rafi KC


On 05/04/2017 06:42 PM, Ankireddypalle Reddy wrote:
Hi,
   Can glusterfs snapshot volume be accessed through libgfapi.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



___

Gluster-devel mailing list

gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>

http://lists.gluster.org/mailman/listinfo/gluster-devel

***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] libgfapi access to snapshot volume

[Ankireddypalle Reddy]

Hi,
   Can glusterfs snapshot volume be accessed through libgfapi.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Enabling shard on EC

[Ankireddypalle Reddy]

Pranith,
 Thanks. Is there any work in progress to add this support.

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Thursday, May 04, 2017 6:17 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org
Subject: Re: [Gluster-devel] Enabling shard on EC



On Thu, May 4, 2017 at 3:43 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Pranith,
 Thanks. Does it mean that a given file can be written by only 
one client at a time. If multiple clients try to access the file in write mode, 
does it lead to any kind of data inconsistencies.

We only tested it for single writer cases such as VM usecases. We need to bring 
in transaction framework for sharding to work with multiple writers.


Thanks and Regards,
Ram
From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com<mailto:pkara...@redhat.com>]
Sent: Thursday, May 04, 2017 6:07 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel 
(gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>); 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-devel] Enabling shard on EC

It is never been tested. That said, I don't see any missing pieces that we know 
of for it to work. Please note that sharding works only for single writer cases 
at the moment. Do let us know if you find any problems and we will fix them.

On Wed, May 3, 2017 at 2:17 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
  Are there any known negatives of enabling shard on EC. Is this a 
recommended configuration?.

Thanks and Regards,
Ram


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-devel mailing list
gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-devel



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Enabling shard on EC

[Ankireddypalle Reddy]

Pranith,
 Thanks. Does it mean that a given file can be written by only 
one client at a time. If multiple clients try to access the file in write mode, 
does it lead to any kind of data inconsistencies.

Thanks and Regards,
Ram
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Thursday, May 04, 2017 6:07 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org
Subject: Re: [Gluster-devel] Enabling shard on EC

It is never been tested. That said, I don't see any missing pieces that we know 
of for it to work. Please note that sharding works only for single writer cases 
at the moment. Do let us know if you find any problems and we will fix them.

On Wed, May 3, 2017 at 2:17 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
  Are there any known negatives of enabling shard on EC. Is this a 
recommended configuration?.

Thanks and Regards,
Ram


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-devel mailing list
gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-devel



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Enabling shard on EC

[Ankireddypalle Reddy]

Hi,
  Are there any known negatives of enabling shard on EC. Is this a 
recommended configuration?.

Thanks and Regards,
Ram


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Disperse mkdir fails

[Ankireddypalle Reddy]

Xavi,
   Thanks for checking this.  We have an external metadata server which 
keeps track of every file that gets written to the volume and has the 
capability to validate the file contents. Will use this capability to validate 
the file contents. Once the data is verified will the following sequence of 
steps be sufficient to restore the volume.

1) Rebalance the volume.
2) After rebalance is complete, stop ingesting more data to the volume.
3) Let the pending heals complete.
4) Stop the volume
5) For any heals that fail because of mismatching version/dirty extended 
attributes on the directories,  set this to a matching value on all the nodes.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Tuesday, March 14, 2017 5:28 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-users] Disperse mkdir fails

Hi Ram,

On 13/03/17 15:02, Ankireddypalle Reddy wrote:
> Xavi,
>CV_MAGNETIC directory on a single brick  has  155683 entries.  
> There are altogether 60 bricks in the volume. I could provide the output if 
> you still need that.

The problem is that not all bricks have the same number of entries:

glusterfs1:disk1 155674
glusterfs2:disk1 155675
glusterfs3:disk1 155718

glusterfs1:disk2 155688
glusterfs2:disk2 155687
glusterfs3:disk2 155730

glusterfs1:disk3 155675
glusterfs2:disk3 155674
glusterfs3:disk3 155717

glusterfs1:disk4 155684
glusterfs2:disk4 155683
glusterfs3:disk4 155726

glusterfs1:disk5 155698
glusterfs2:disk5 155695
glusterfs3:disk5 155738

glusterfs1:disk6 155668
glusterfs2:disk6 155667
glusterfs3:disk6 155710

glusterfs1:disk7 155687
glusterfs2:disk7 155689
glusterfs3:disk7 155732

glusterfs1:disk8 155673
glusterfs2:disk8 155675
glusterfs3:disk8 155718

glusterfs4:disk1 149097
glusterfs5:disk1 149097
glusterfs6:disk1 149098

glusterfs4:disk2 149097
glusterfs5:disk2 149097
glusterfs6:disk2 149098

glusterfs4:disk3 149097
glusterfs5:disk3 149097
glusterfs6:disk3 149098

glusterfs4:disk4 149097
glusterfs5:disk4 149097
glusterfs6:disk4 149098

glusterfs4:disk5 149097
glusterfs5:disk5 149097
glusterfs6:disk5 149098

glusterfs4:disk6 149097
glusterfs5:disk6 149097
glusterfs6:disk6 149098

glusterfs4:disk7 149097
glusterfs5:disk7 149097
glusterfs6:disk7 149098

glusterfs4:disk8 149097
glusterfs5:disk8 149097
glusterfs6:disk8 149098

An small difference could be explained by concurrent operations while 
retrieving this data, but some bricks are way out of sync.

trusted.ec.dirty and trusted.ec.version also show many discrepancies:

glusterfs1:disk1 trusted.ec.dirty=0x0ba4
glusterfs2:disk1 trusted.ec.dirty=0x0bb8
glusterfs3:disk1 trusted.ec.dirty=0x0016
glusterfs1:disk1 trusted.ec.version=0x00084db400084e11
glusterfs2:disk1 trusted.ec.version=0x00084e0700084e0c
glusterfs3:disk1 trusted.ec.version=0x0008426a00084e11

glusterfs1:disk2 trusted.ec.dirty=0x0ba5
glusterfs2:disk2 trusted.ec.dirty=0x0bb6
glusterfs3:disk2 trusted.ec.dirty=0x0017
glusterfs1:disk2 trusted.ec.version=0x0005ccb70005cd0a
glusterfs2:disk2 trusted.ec.version=0x0005cd05cd05
glusterfs3:disk2 trusted.ec.version=0x0005c1660005cd0a

glusterfs1:disk3 trusted.ec.dirty=0x0ba5
glusterfs2:disk3 trusted.ec.dirty=0x0bb5
glusterfs3:disk3 trusted.ec.dirty=0x0016
glusterfs1:disk3 trusted.ec.version=0x0005d0cb0005d123
glusterfs2:disk3 trusted.ec.version=0x0005d1190005d11e
glusterfs3:disk3 trusted.ec.version=0x0005c57f0005d123

glusterfs1:disk4 trusted.ec.dirty=0x0ba0
glusterfs2:disk4 trusted.ec.dirty=0x0bb1
glusterfs3:disk4 trusted.ec.dirty=0x0013
glusterfs1:disk4 trusted.ec.version=0x00084e2e00084e78
glusterfs2:disk4 trusted.ec.version=0x00084e6e00084e73
glusterfs3:disk4 trusted.ec.version=0x000842d500084e78

glusterfs1:disk5 trusted.ec.dirty=0x0b9a
glusterfs2:disk5 trusted.ec.dirty=0x2e27
glusterfs3:disk5 trusted.ec.dirty=0x2295
glusterfs1:disk5 trusted.ec.version=0x0005aa1f0005cd18
glusterfs2:disk5 trusted.ec.version=0x0005cd0d0005cd13
glusterfs3:disk5 trusted.ec.version=0x0005c185cd18

glusterfs1:disk6 trusted.ec.dirty=0x0ba2
glusterfs2:disk6 trusted.ec.dirty=0x0bad
glusterfs3:disk6 trusted.ec.dirty=0x000f
glusterfs1:disk6 truste

Re: [Gluster-users] Disperse mkdir fails

[Ankireddypalle Reddy]

Xavi,
   CV_MAGNETIC directory on a single brick  has  155683 entries.  
There are altogether 60 bricks in the volume. I could provide the output if you 
still need that.  

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, March 13, 2017 9:56 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-users] Disperse mkdir fails

Hi Ram,

On 13/03/17 14:13, Ankireddypalle Reddy wrote:
> Attachment (1):
>
> 1
>
>   
>
> data.txt
> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co
> mmvault.com/webconsole/api/drive/publicshare/346714/file/02bb2e2504a54
> 3e58cc89bce9f350f8c/action/preview&downloadUrl=https://imap.commvault.
> com/webconsole/api/contentstore/publicshare/346714/file/02bb2e2504a543
> e58cc89bce9f350f8c/action/download>
> [Download]
> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34
> 6714/file/02bb2e2504a543e58cc89bce9f350f8c/action/download>(17.63
> KB)
>
> Xavier,
>Please find attached the required info from all the six 
> nodes of the cluster.

I asked for the contents of the CV_MAGNETIC because this is the damaged 
directory, not the parent. But anyway we can see that the number of hard links 
of the directory differs for each brick, so this means that the number of 
subdirectories is different on each brick. A small difference could be 
explainable by the current activity of the volume while the data has been 
captured, but the differences are too big.

>  We need to find
>1) What is the solution through which this problem can 
> be avoided.
>2) How do we fix the current state of the cluster.
>
> Thanks and Regards,
> Ram
> -Original Message-
> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
> Sent: Friday, March 10, 2017 3:34 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
> gluster-users@gluster.org
> Subject: Re: [Gluster-users] Disperse mkdir fails
>
> Hi Ram,
>
> On 09/03/17 20:15, Ankireddypalle Reddy wrote:
>> Xavi,
>> Thanks for checking this.
>> 1) mkdir returns errnum 5. EIO.
>> 2)  The specified directory is the parent directory under
> which all the data in the gluster volume will be stored. Current 
> around 160TB of 262 TB is  consumed.
>
> I only need the first level entries of that directory, not the entire 
> tree of entries. This should be in the order of thousands, right ?
>
> We need to make sure that all bricks have the same entries in this 
> directory. Otherwise we would need to check other things.
>
>> 3)  It is extremely difficult to list the exact sequence
> of FOPS that would have been issued to the directory. The storage is 
> heavily used and lot of sub directories are present inside this directory.
>>
>>Are you looking for the extended attributes for this
> directory from all the bricks inside the volume.  There are about 60 bricks.
>
> If possible, yes.
>
> However, if there's a lot of modifications on that directory while you 
> are getting the xattr, it's possible that you get inconsistent values, 
> but they are not really inconsistent.
>
> If possible, you should get that information pausing all activity to 
> that directory.
>
> Xavi
>
>>
>> Thanks and Regards,
>> Ram
>>
>> -Original Message-
>> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
>> Sent: Thursday, March 09, 2017 11:15 AM
>> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
>> gluster-users@gluster.org
>> Subject: Re: [Gluster-users] Disperse mkdir fails
>>
>> Hi Ram,
>>
>> On 09/03/17 16:52, Ankireddypalle Reddy wrote:
>>> Attachment (1):
>>>
>>> 1
>>>
>>>
>>>
>>> info.txt
>>> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.
>>> c
>>> o
>>> mmvault.com/webconsole/api/drive/publicshare/346714/file/3037641a3f9
>>> b
>>> 4
>>> 133920b1b251ed32d5d/action/preview&downloadUrl=https://imap.commvault.
>>> com/webconsole/api/contentstore/publicshare/346714/file/3037641a3f9b
>>> 4
>>> 1
>>> 33920b1b251ed32d5d/action/download>
>>> [Download]
>>> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/
>>> 3
>>> 4
>>> 6714/file/3037641a3f9b4133920b1b251ed32d5d/action/download>(3.35
>>> KB)
>>>
>>> Hi,
>>>
>>>  

Re: [Gluster-users] Disperse mkdir fails

[Ankireddypalle Reddy]

Attachment (1):

1

data.txt<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/02bb2e2504a543e58cc89bce9f350f8c/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/02bb2e2504a543e58cc89bce9f350f8c/action/download>
 
[Download]<https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/02bb2e2504a543e58cc89bce9f350f8c/action/download>
 (17.63 KB)


Xavier,
   Please find attached the required info from all the six nodes of 
the cluster.  We need to find
   1) What is the solution through which this problem can be 
avoided.
   2) How do we fix the current state of the cluster.

Thanks and Regards,
Ram
-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es]
Sent: Friday, March 10, 2017 3:34 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-users] Disperse mkdir fails

Hi Ram,

On 09/03/17 20:15, Ankireddypalle Reddy wrote:
> Xavi,
> Thanks for checking this.
> 1) mkdir returns errnum 5. EIO.
> 2)  The specified directory is the parent directory under which 
> all the data in the gluster volume will be stored. Current around 160TB of 
> 262 TB is  consumed.

I only need the first level entries of that directory, not the entire tree of 
entries. This should be in the order of thousands, right ?

We need to make sure that all bricks have the same entries in this directory. 
Otherwise we would need to check other things.

> 3)  It is extremely difficult to list the exact sequence of FOPS 
> that would have been issued to the directory. The storage is heavily used and 
> lot of sub directories are present inside this directory.
>
>Are you looking for the extended attributes for this directory 
> from all the bricks inside the volume.  There are about 60 bricks.

If possible, yes.

However, if there's a lot of modifications on that directory while you are 
getting the xattr, it's possible that you get inconsistent values, but they are 
not really inconsistent.

If possible, you should get that information pausing all activity to that 
directory.

Xavi

>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
> Sent: Thursday, March 09, 2017 11:15 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org);
> gluster-users@gluster.org
> Subject: Re: [Gluster-users] Disperse mkdir fails
>
> Hi Ram,
>
> On 09/03/17 16:52, Ankireddypalle Reddy wrote:
>> Attachment (1):
>>
>> 1
>>
>>
>>
>> info.txt
>> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.c
>> o
>> mmvault.com/webconsole/api/drive/publicshare/346714/file/3037641a3f9b
>> 4
>> 133920b1b251ed32d5d/action/preview&downloadUrl=https://imap.commvault.
>> com/webconsole/api/contentstore/publicshare/346714/file/3037641a3f9b4
>> 1
>> 33920b1b251ed32d5d/action/download>
>> [Download]
>> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/3
>> 4
>> 6714/file/3037641a3f9b4133920b1b251ed32d5d/action/download>(3.35
>> KB)
>>
>> Hi,
>>
>> I have a disperse gluster volume  with 6 servers. 262TB of
>> usable capacity.  Gluster version is 3.7.19.
>>
>> glusterfs1, glusterf2 and glusterfs3 nodes were initially
>> used for creating the volume. Nodes glusterf4, glusterfs5 and
>> glusterfs6 were later added to the volume.
>>
>>
>>
>> Directory creation failed on a directory called
>> /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC.
>>
>> # file: ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC
>>
>> glusterfs.gfid.string="e8e51015-616f-4f04-b9d2-92f46eb5cfc7"
>>
>>
>>
>> gluster mount log contains lot of following errors:
>>
>> [2017-03-09 15:32:36.773937] W [MSGID: 122056]
>> [ec-combine.c:875:ec_combine_check] 0-StoragePool-disperse-7:
>> Mismatching xdata in answers of 'LOOKUP' for
>> e8e51015-616f-4f04-b9d2-92f46eb5cfc7
>>
>>
>>
>> The directory seems to be out of sync between nodes
>> glusterfs1,
>> glusterfs2 and glusterfs3. Each has different version.
>>
>>
>>
>>  trusted.ec.version=0x000839f83a4d
>>
>>  trusted.ec.version=0x00082ea400083a4b
>>
>>  trusted.ec.version=0x00083a7600083a7b
>>
>>

Re: [Gluster-users] Disperse mkdir fails

[Ankireddypalle Reddy]

Xavi,
Thanks for checking this.
1) mkdir returns errnum 5. EIO.  
2)  The specified directory is the parent directory under which all 
the data in the gluster volume will be stored. Current around 160TB of 262 TB 
is  consumed.
3)  It is extremely difficult to list the exact sequence of FOPS 
that would have been issued to the directory. The storage is heavily used and 
lot of sub directories are present inside this directory.

   Are you looking for the extended attributes for this directory from 
all the bricks inside the volume.  There are about 60 bricks.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Thursday, March 09, 2017 11:15 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-users] Disperse mkdir fails

Hi Ram,

On 09/03/17 16:52, Ankireddypalle Reddy wrote:
> Attachment (1):
>
> 1
>
>   
>
> info.txt
> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co
> mmvault.com/webconsole/api/drive/publicshare/346714/file/3037641a3f9b4
> 133920b1b251ed32d5d/action/preview&downloadUrl=https://imap.commvault.
> com/webconsole/api/contentstore/publicshare/346714/file/3037641a3f9b41
> 33920b1b251ed32d5d/action/download>
> [Download]
> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34
> 6714/file/3037641a3f9b4133920b1b251ed32d5d/action/download>(3.35
> KB)
>
> Hi,
>
> I have a disperse gluster volume  with 6 servers. 262TB of 
> usable capacity.  Gluster version is 3.7.19.
>
> glusterfs1, glusterf2 and glusterfs3 nodes were initially used 
> for creating the volume. Nodes glusterf4, glusterfs5 and glusterfs6 
> were later added to the volume.
>
>
>
> Directory creation failed on a directory called 
> /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC.
>
> # file: ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC
>
> glusterfs.gfid.string="e8e51015-616f-4f04-b9d2-92f46eb5cfc7"
>
>
>
> gluster mount log contains lot of following errors:
>
> [2017-03-09 15:32:36.773937] W [MSGID: 122056] 
> [ec-combine.c:875:ec_combine_check] 0-StoragePool-disperse-7:
> Mismatching xdata in answers of 'LOOKUP' for
> e8e51015-616f-4f04-b9d2-92f46eb5cfc7
>
>
>
> The directory seems to be out of sync between nodes 
> glusterfs1,
> glusterfs2 and glusterfs3. Each has different version.
>
>
>
>  trusted.ec.version=0x000839f83a4d
>
>  trusted.ec.version=0x00082ea400083a4b
>
>  trusted.ec.version=0x00083a7600083a7b
>
>
>
>  Self-heal does not seem to be healing this directory.
>

This is very similar to what happened the other time. Once more than 1 brick is 
damaged, self-heal cannot do anything to heal it on a 2+1 configuration.

What error does return the mkdir request ?

Does the directory you are trying to create already exist on some brick ?

Can you show all the remaining extended attributes of the directory ?

It would also be useful to have the directory contents on each brick (an 'ls 
-l'). In this case, include the name of the directory you are trying to create.

Can you explain a detailed sequence of operations done on that directory since 
the last time you successfully created a new subdirectory ? 
including any metadata change.

Xavi

>
>
> Thanks and Regards,
>
> Ram
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>

***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Disperse mkdir fails

[Ankireddypalle Reddy]

Attachment (1):

1

info.txt
 
[Download]
 (3.35 KB)

Hi,
I have a disperse gluster volume  with 6 servers. 262TB of usable 
capacity.  Gluster version is 3.7.19.
glusterfs1, glusterf2 and glusterfs3 nodes were initially used for 
creating the volume. Nodes glusterf4, glusterfs5 and glusterfs6 were later 
added to the volume.

Directory creation failed on a directory called 
/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC.
# file: ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC
glusterfs.gfid.string="e8e51015-616f-4f04-b9d2-92f46eb5cfc7"

gluster mount log contains lot of following errors:
[2017-03-09 15:32:36.773937] W [MSGID: 122056] 
[ec-combine.c:875:ec_combine_check] 0-StoragePool-disperse-7: Mismatching xdata 
in answers of 'LOOKUP' for e8e51015-616f-4f04-b9d2-92f46eb5cfc7

The directory seems to be out of sync between nodes glusterfs1, 
glusterfs2 and glusterfs3. Each has different version.

 trusted.ec.version=0x000839f83a4d
 trusted.ec.version=0x00082ea400083a4b
 trusted.ec.version=0x00083a7600083a7b

 Self-heal does not seem to be healing this directory.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
   Thanks.  Please let me know the functions that we need to track for 
any inconsistencies in the return codes from multiple bricks for issue 1. I 
will start doing that.

  1. Why the write fails in first place

Thanks and Regards,
Ram
 
-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Friday, January 20, 2017 2:41 AM
To: Ankireddypalle Reddy; Ashish Pandey
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse 
volume

Hi Ram,

On 20/01/17 08:02, Ankireddypalle Reddy wrote:
> Ashish,
>
>  Thanks for looking in to the issue. In the given
> example the size/version matches for file on glusterfs4 and glusterfs5
> nodes. The file is empty on glusterfs6. Now what happens if glusterfs5
> goes down. Though the SLA factor of 2 is met  still I will not be able
> to access the data.

True, but having a brick with inconsistent data is the same as having it 
down. You would have lost 2 bricks out of 3.

The problem is how to detect the inconsistent data, what is causing it 
and why self-heal (apparently) is not healing it.

> The problem is that writes did not fail for the file
> to indicate the issue to the application.

That's the expected behavior. Since we have redundancy 1, a loss or 
failure on a single fragment is hidden from the application. However 
this triggers internal procedures to repair the problem on the 
background. This also seems to not be working.

There are two issues to identify here:

1. Why the write fails in first place
2. Why self-heal is unable to repair it

Probably the root cause is the same for both problems, but I'm not sure.

For 1 there should be some warning or error in the mount log, since one 
of the bricks is reporting an error, even if ec is able to report 
success to the application later.

For 2 the analysis could be more complex, but most probably there should 
be some warning or error message in the mount log and/or self-heal log 
of one of the servers.

Xavi

>  It’s not that we are
> encountering issue for every file on the mount point. The issue happens
> randomly for different files.
>
>
>
> [root@glusterfs4 glusterfs]# gluster volume info
>
>
>
> Volume Name: glusterfsProd
>
> Type: Distributed-Disperse
>
> Volume ID: 622aa3ee-958f-485f-b3c1-fb0f6c8db34c
>
> Status: Started
>
> Number of Bricks: 12 x (2 + 1) = 36
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: glusterfs4sds:/ws/disk1/ws_brick
>
> Brick2: glusterfs5sds:/ws/disk1/ws_brick
>
> Brick3: glusterfs6sds:/ws/disk1/ws_brick
>
> Brick4: glusterfs4sds:/ws/disk10/ws_brick
>
> Brick5: glusterfs5sds:/ws/disk10/ws_brick
>
> Brick6: glusterfs6sds:/ws/disk10/ws_brick
>
> Brick7: glusterfs4sds:/ws/disk11/ws_brick
>
> Brick8: glusterfs5sds:/ws/disk11/ws_brick
>
> Brick9: glusterfs6sds:/ws/disk11/ws_brick
>
> Brick10: glusterfs4sds:/ws/disk2/ws_brick
>
> Brick11: glusterfs5sds:/ws/disk2/ws_brick
>
> Brick12: glusterfs6sds:/ws/disk2/ws_brick
>
> Brick13: glusterfs4sds:/ws/disk3/ws_brick
>
> Brick14: glusterfs5sds:/ws/disk3/ws_brick
>
> Brick15: glusterfs6sds:/ws/disk3/ws_brick
>
> Brick16: glusterfs4sds:/ws/disk4/ws_brick
>
> Brick17: glusterfs5sds:/ws/disk4/ws_brick
>
> Brick18: glusterfs6sds:/ws/disk4/ws_brick
>
> Brick19: glusterfs4sds:/ws/disk5/ws_brick
>
> Brick20: glusterfs5sds:/ws/disk5/ws_brick
>
> Brick21: glusterfs6sds:/ws/disk5/ws_brick
>
> Brick22: glusterfs4sds:/ws/disk6/ws_brick
>
> Brick23: glusterfs5sds:/ws/disk6/ws_brick
>
> Brick24: glusterfs6sds:/ws/disk6/ws_brick
>
> Brick25: glusterfs4sds:/ws/disk7/ws_brick
>
> Brick26: glusterfs5sds:/ws/disk7/ws_brick
>
> Brick27: glusterfs6sds:/ws/disk7/ws_brick
>
> Brick28: glusterfs4sds:/ws/disk8/ws_brick
>
> Brick29: glusterfs5sds:/ws/disk8/ws_brick
>
> Brick30: glusterfs6sds:/ws/disk8/ws_brick
>
> Brick31: glusterfs4sds:/ws/disk9/ws_brick
>
> Brick32: glusterfs5sds:/ws/disk9/ws_brick
>
> Brick33: glusterfs6sds:/ws/disk9/ws_brick
>
> Brick34: glusterfs4sds:/ws/disk12/ws_brick
>
> Brick35: glusterfs5sds:/ws/disk12/ws_brick
>
> Brick36: glusterfs6sds:/ws/disk12/ws_brick
>
> Options Reconfigured:
>
> storage.build-pgfid: on
>
> performance.readdir-ahead: on
>
> nfs.export-dirs: off
>
> nfs.export-volumes: off
>
> nfs.disable: on
>
> auth.allow: glusterfs4sds,glusterfs5sds,glusterfs6sds
>
> diagnostics.client-log-level: INFO
>
> [root@glusterfs4 glusterfs]#
>
>
>
> Thanks and Regards,
>
> Ram
>
> *From:*Ashish Pandey [mailto:aspan...@redhat.com]
> *Sent:* Thursday, January 19, 2017 10:36 PM
>

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
Finally I am able to locate the files for which we are seeing the 
mismatch.

[2017-01-19 21:20:10.002737] W [MSGID: 122056] 
[ec-combine.c:875:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP' for f5e61224-17f6-4803-9f75-dd74cd79e248
[2017-01-19 21:20:10.002776] W [MSGID: 122006] 
[ec-combine.c:207:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
combine iatt (inode: 11490333518038950472-11490333518038950472, links: 1-1, 
uid: 0-0, gid: 0-0, rdev: 0-0, size: 4800512-4734976, mode: 100775-100775) for 
f5e61224-17f6-4803-9f75-dd74cd79e248

GFID f5e61224-17f6-4803-9f75-dd74cd79e248 maps to file 
/ws/disk1/ws_brick/Folder_01.05.2017_21.15/CV_MAGNETIC/V_32410/CHUNK_424795/SFILE_CONTAINER_158

The size field differs between the bricks of this sub volume.

[root@glusterfs4 glusterfs]# getfattr -d -e hex -m . 
/ws/disk1/ws_brick/Folder_01.05.2017_21.15/CV_MAGNETIC/V_32410/CHUNK_424795/SFILE_CONTAINER_158
trusted.ec.size=0x000e
trusted.ec.version=0x00080008

[root@glusterfs5 ws]# getfattr -d -e hex -m . 
/ws/disk1/ws_brick/Folder_01.05.2017_21.15/CV_MAGNETIC/V_32410/CHUNK_424795/SFILE_CONTAINER_158
trusted.ec.size=0x000e
trusted.ec.version=0x00080008

[root@glusterfs6 glusterfs]# getfattr -d -e hex -m . 
/ws/disk1/ws_brick/Folder_01.05.2017_21.15/CV_MAGNETIC/V_32410/CHUNK_424795/SFILE_CONTAINER_158
getfattr: Removing leading '/' from absolute path names
trusted.ec.size=0x
trusted.ec.version=0x0008

On glusterfs6 writes seem to have failed and the file appears to be an empty 
file. 
[root@glusterfs6 glusterfs]# stat 
/ws/disk1/ws_brick/Folder_01.05.2017_21.15/CV_MAGNETIC/V_32410/CHUNK_424795/SFILE_CONTAINER_158
  File: 
'/ws/disk1/ws_brick/Folder_01.05.2017_21.15/CV_MAGNETIC/V_32410/CHUNK_424795/SFILE_CONTAINER_158'
  Size: 0   Blocks: 0  IO Block: 4096   regular empty file

This looks like a major issue. Since we are left with no redundancy for these 
files and if the brick on other node fails then we will not be able to access 
this data though the redundancy factor of 2 is met in a 3:1 disperse volume. 

Thanks and Regards,
Ram

-Original Message-
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Monday, January 16, 2017 9:41 AM
To: Xavier Hernandez
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse 
volume

Xavi,
   Thanks. I will start by tracking the delta if any in return codes 
from the bricks for writes in EC. I have a feeling that  if we could get to the 
bottom of this issue then the EIO errors could be ultimately avoided.  Please 
note that while we are testing all these on a standby setup we continue to 
encounter the EIO errors on our production setup which is running @ 3.7.18.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, January 16, 2017 6:54 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-devel] [Gluster-users] Lot of EIO errors in disperse 
volume

Hi Ram,

On 16/01/17 12:33, Ankireddypalle Reddy wrote:
> Xavi,
>   Thanks. Is there any other way to map from GFID to path.

The only way I know is to search all files from bricks and lookup for 
the trusted.gfid xattr.

> I will look for a way to share the TRACE logs. Easier way might be to add 
> some extra logging. I could do that if you could let me know functions in 
> which you are interested..

The problem is that I don't know where the problem is. One possibility 
could be to track all return values from all bricks for all writes and 
then identify which ones belong to an inconsistent file.

But if this doesn't reveal anything interesting we'll need to look at 
some other place. And this can be very tedious and slow.

Anyway, what we are looking now is not the source of an EIO, since there 
are two bricks with consistent state and the file should be perfectly 
readable and writable. It's true that there's some problem here and it 
could derive in EIO if one of the healthy bricks degrades, but at least 
this file shouldn't be giving EIO errors for now.

Xavi

>
> Sent on from my iPhone
>
>> On Jan 16, 2017, at 6:23 AM, Xavier Hernandez  wrote:
>>
>> Hi Ram,
>>
>>> On 13/01/17 18:41, Ankireddypalle Reddy wrote:
>>> Xavi,
>>> I enabled TRACE logging. The log grew up to 120GB and could not 
>>> make much out of it. Then I started logging GFID in the code where we were 
>>> seeing errors.
>>>
>>> [2017-01-13 17:02:01.761349] I [dict.c

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
   Thanks. I will start by tracking the delta if any in return codes 
from the bricks for writes in EC. I have a feeling that  if we could get to the 
bottom of this issue then the EIO errors could be ultimately avoided.  Please 
note that while we are testing all these on a standby setup we continue to 
encounter the EIO errors on our production setup which is running @ 3.7.18.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, January 16, 2017 6:54 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-devel] [Gluster-users] Lot of EIO errors in disperse 
volume

Hi Ram,

On 16/01/17 12:33, Ankireddypalle Reddy wrote:
> Xavi,
>   Thanks. Is there any other way to map from GFID to path.

The only way I know is to search all files from bricks and lookup for 
the trusted.gfid xattr.

> I will look for a way to share the TRACE logs. Easier way might be to add 
> some extra logging. I could do that if you could let me know functions in 
> which you are interested..

The problem is that I don't know where the problem is. One possibility 
could be to track all return values from all bricks for all writes and 
then identify which ones belong to an inconsistent file.

But if this doesn't reveal anything interesting we'll need to look at 
some other place. And this can be very tedious and slow.

Anyway, what we are looking now is not the source of an EIO, since there 
are two bricks with consistent state and the file should be perfectly 
readable and writable. It's true that there's some problem here and it 
could derive in EIO if one of the healthy bricks degrades, but at least 
this file shouldn't be giving EIO errors for now.

Xavi

>
> Sent on from my iPhone
>
>> On Jan 16, 2017, at 6:23 AM, Xavier Hernandez  wrote:
>>
>> Hi Ram,
>>
>>> On 13/01/17 18:41, Ankireddypalle Reddy wrote:
>>> Xavi,
>>> I enabled TRACE logging. The log grew up to 120GB and could not 
>>> make much out of it. Then I started logging GFID in the code where we were 
>>> seeing errors.
>>>
>>> [2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
>>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761365] W [MSGID: 122056] 
>>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>>> xdata in answers of 'LOOKUP'
>>> [2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
>>> 0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 
>>> 8)
>>> [2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
>>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
>>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>>> [2017-01-13 17:02:01.761433] W [MSGID: 122056] 
>>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>>> xdata in answers of 'LOOKUP'
>>> [2017-01-13 17:02:01.761442] W [MSGID: 122006] 
>>> [ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
>>> combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
>>> 60b990ed-d741-4176-9c7b-4d3a25fb8252  -  
>>> 60b990ed-d741-4176-9c7b-4d3a25fb8252,  links: 1-1, uid: 0-0, gid: 0-0, 
>>> rdev: 0-0,size: 406650880-406683648, mode: 100775-100775)
>>>
>>> The file for which we are seeing this error turns out to be having a GFID 
>>> of 60b990ed-d741-4176-9c7b-4d3a25fb8252
>>>
>>> Then I tried looking for find out the file with this GFID. It pointed me to 
>>> following path. I was expecting a real file system path from the following 
>>> turorial:
>>> https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/
>>
>> I think this method only works if bricks have the inode cached.
>>
>>>
>>> getfattr -n trusted.glusterfs.pathinfo -e text 
>>> /mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb825

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
  Thanks. Is there any other way to map from GFID to path. I will look 
for a way to share the TRACE logs. Easier way might be to add some extra 
logging. I could do that if you could let me know functions in which you are 
interested..

Sent on from my iPhone

> On Jan 16, 2017, at 6:23 AM, Xavier Hernandez  wrote:
> 
> Hi Ram,
> 
>> On 13/01/17 18:41, Ankireddypalle Reddy wrote:
>> Xavi,
>> I enabled TRACE logging. The log grew up to 120GB and could not 
>> make much out of it. Then I started logging GFID in the code where we were 
>> seeing errors.
>> 
>> [2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761365] W [MSGID: 122056] 
>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
>> 0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 
>> 8)
>> [2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
>> ((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
>> ((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
>> [2017-01-13 17:02:01.761433] W [MSGID: 122056] 
>> [ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-13 17:02:01.761442] W [MSGID: 122006] 
>> [ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
>> combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
>> 60b990ed-d741-4176-9c7b-4d3a25fb8252  -  
>> 60b990ed-d741-4176-9c7b-4d3a25fb8252,  links: 1-1, uid: 0-0, gid: 0-0, rdev: 
>> 0-0,size: 406650880-406683648, mode: 100775-100775)
>> 
>> The file for which we are seeing this error turns out to be having a GFID of 
>> 60b990ed-d741-4176-9c7b-4d3a25fb8252
>> 
>> Then I tried looking for find out the file with this GFID. It pointed me to 
>> following path. I was expecting a real file system path from the following 
>> turorial:
>> https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/
> 
> I think this method only works if bricks have the inode cached.
> 
>> 
>> getfattr -n trusted.glusterfs.pathinfo -e text 
>> /mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.glusterfs.pathinfo="( 
>> ( 
>> 
>>  
>> ))"
>> 
>> Then I looked for the xatttrs for these files from all the 3 bricks
>> 
>> [root@glusterfs4 glusterfs]# getfattr -d -m . -e hex 
>> /ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: 
>> ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.bit-rot.version=0x02005877a8dc00041138
>> trusted.ec.config=0x080301000200
>> trusted.ec.size=0x
>> trusted.ec.version=0x2a38
>> trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252
>> 
>> [root@glusterfs5 bricks]# getfattr -d -m . -e hex 
>> /ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute path names
>> # file: 
>> ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> trusted.bit-rot.version=0x02005877a8dc000c92d0
>> trusted.ec.config=0x080301000200
>> trusted.ec.dirty=0x0016
>> trusted.ec.size=0x306b
>> trusted.ec.version=0x2a382a38
>> trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252
>> 
>> [root@glusterfs6 ee]# getfattr -d -m . -e hex 
>> /ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
>> getfattr: Removing leading '/' from absolute pat

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
 I enabled TRACE logging. The log grew up to 120GB and could not 
make much out of it. Then I started logging GFID in the code where we were 
seeing errors.

[2017-01-13 17:02:01.761349] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bc690 
((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761360] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761365] W [MSGID: 122056] 
[ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-13 17:02:01.761405] I [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-13 17:02:01.761417] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bbb14 
((trusted.ec.size:0:0:0:0:30:6b:0:0:)(trusted.ec.version:0:0:0:0:0:0:2a:38:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761428] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7fa6706bed64 
((trusted.ec.size:0:0:0:0:0:0:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2a:38:))
[2017-01-13 17:02:01.761433] W [MSGID: 122056] 
[ec-combine.c:881:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-13 17:02:01.761442] W [MSGID: 122006] 
[ec-combine.c:214:ec_iatt_combine] 0-glusterfsProd-disperse-0: Failed to 
combine iatt (inode: 11275691004192850514-11275691004192850514, gfid: 
60b990ed-d741-4176-9c7b-4d3a25fb8252  -  60b990ed-d741-4176-9c7b-4d3a25fb8252,  
links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0,size: 406650880-406683648, mode: 
100775-100775)

The file for which we are seeing this error turns out to be having a GFID of 
60b990ed-d741-4176-9c7b-4d3a25fb8252  

Then I tried looking for find out the file with this GFID. It pointed me to 
following path. I was expecting a real file system path from the following 
turorial:
https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/

getfattr -n trusted.glusterfs.pathinfo -e text 
/mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: mnt/gfid/.gfid/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.glusterfs.pathinfo="( 
( 

 
))"

Then I looked for the xatttrs for these files from all the 3 bricks

[root@glusterfs4 glusterfs]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc00041138
trusted.ec.config=0x080301000200
trusted.ec.size=0x
trusted.ec.version=0x2a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

[root@glusterfs5 bricks]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc000c92d0
trusted.ec.config=0x080301000200
trusted.ec.dirty=0x0016
trusted.ec.size=0x306b
trusted.ec.version=0x2a382a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

[root@glusterfs6 ee]# getfattr -d -m . -e hex 
/ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
getfattr: Removing leading '/' from absolute path names
# file: ws/disk1/ws_brick/.glusterfs/60/b9/60b990ed-d741-4176-9c7b-4d3a25fb8252
trusted.bit-rot.version=0x02005877a8dc000c9436
trusted.ec.config=0x080301000200
trusted.ec.dirty=0x0016
trusted.ec.size=0x306b
trusted.ec.version=0x2a382a38
trusted.gfid=0x60b990edd74141769c7b4d3a25fb8252

It turns out that the size and version in fact does not match for one of the 
files. 

Thanks and Regards,
Ram

-Original Message-
From: gluster-devel-boun...@gluster.org 
[mailto:gluster-devel-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Friday, January 13, 2017 4:17 AM
To: Xavier Hernandez
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-devel] [Gluster-users] Lot of EIO errors in disperse 
volume

Xavi,
Thanks for explanation. Will collect TRACE logs today. 

Thanks and Regards,
Ram

Sent from my iPhone

> On Jan 13, 2017, at 3:03 AM, Xavier Hernandez  wrote:
> 
> Hi Ram,
> 
>> On 12/01/17 22:14, Ankireddypalle Reddy wrote:
>> Xavi,
>> I changed the logging to log the individual bytes. Consider the 
>

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
Thanks for explanation. Will collect TRACE logs today. 

Thanks and Regards,
Ram

Sent from my iPhone

> On Jan 13, 2017, at 3:03 AM, Xavier Hernandez  wrote:
> 
> Hi Ram,
> 
>> On 12/01/17 22:14, Ankireddypalle Reddy wrote:
>> Xavi,
>> I changed the logging to log the individual bytes. Consider the 
>> following from ws-glus.log file where /ws/glus is the mount point.
>> 
>> [2017-01-12 20:47:59.368102] I [MSGID: 109063] 
>> [dht-layout.c:718:dht_layout_normalize] 0-glusterfsProd-dht: Found anomalies 
>> in /Folder_01.05.2017_21.15/CV_MAGNETIC/V_30970/CHUNK_390607 (gfid = 
>> e694387f-dde7-410b-9562-914a994d5e85). Holes=1 overlaps=0
>> [2017-01-12 20:47:59.391218] I [MSGID: 109036] 
>> [dht-common.c:9082:dht_log_new_layout_for_dir_selfheal] 0-glusterfsProd-dht: 
>> Setting layout of /Folder_01.05.2017_21.15/CV_MAGNETIC/V_30970/CHUNK_390607 
>> with [Subvol_name: glusterfsProd-disperse-0, Err: -1 , Start: 2505397587 , 
>> Stop: 2863311527 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-1, Err: 
>> -1 , Start: 2863311528 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: 
>> glusterfsProd-disperse-10, Err: -1 , Start: 3221225469 , Stop: 3579139409 , 
>> Hash: 1 ], [Subvol_name: glusterfsProd-disperse-11, Err: -1 , Start: 
>> 3579139410 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: 
>> glusterfsProd-disperse-2, Err: -1 , Start: 3937053351 , Stop: 4294967295 , 
>> Hash: 1 ], [Subvol_name: glusterfsProd-disperse-3, Err: -1 , Start: 0 , 
>> Stop: 357913940 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-4, Err: -1 
>> , Start: 357913941 , Stop: 715827881 , Hash: 1 ], [Subvol_name: 
>> glusterfsProd-disperse-5, Err: -1 , Start: 715827882 , Stop: 1073741822 , 
>> Hash
 : 1 ], [Subvol_name: glusterfsProd-disperse-6, Err: -1 , Start: 1073741823 , 
Stop: 1431655763 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-7, Err: -1 , 
Start: 1431655764 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: 
glusterfsProd-disperse-8, Err: -1 , Start: 1789569705 , Stop: 2147483645 , 
Hash: 1 ], [Subvol_name: glusterfsProd-disperse-9, Err: -1 , Start: 2147483646 
, Stop: 2505397586 , Hash: 1 ],
>> 
>>Self-heal seems to be triggered for path  
>> /Folder_01.05.2017_21.15/CV_MAGNETIC/V_30970/CHUNK_390607 due to anomalies 
>> as per DHT.  It would be great if someone could explain what could be the 
>> anomaly here.  The setup where we encountered this is a fairly stable setup 
>> with no brick failures or no node failures.
> 
> This is not really a self-heal, at least from the point of view of ec. This 
> means that DHT has found a discrepancy in the layout of that directory, 
> however this doesn't mean any problem (notice the 'I' in the log, meaning 
> that it's informative, not a warning nor error).
> 
> Not sure how DHT works in this case or why it finds this "anomaly", but if 
> there aren't any previous errors before that message, it can be completely 
> ignored.
> 
> Not sure if it can be related to option cluster.weighted-rebalance that is 
> enabled by default.
> 
>> 
>>   Then Self-heal seems to have encountered the following error.
>> 
>> [2017-01-12 20:48:23.418432] I [dict.c:166:key_value_cmp] 
>> 0-glusterfsProd-disperse-2: 'trusted.ec.version' is different in two dicts 
>> (16, 16)
>> [2017-01-12 20:48:23.418496] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-2: dict=0x7f0b649520ac 
>> ((trusted.glusterfs.dht:0:0:0:1:0:0:0:0:0:0:0:0:15:55:55:54:)(trusted.ec.version:0:0:0:0:0:0:0:b:0:0:0:0:0:0:0:e:))
>> [2017-01-12 20:48:23.418519] I [dict.c:3065:dict_dump_to_log] 
>> 0-glusterfsProd-disperse-2: dict=0x7f0b6495b4e0 
>> ((trusted.glusterfs.dht:0:0:0:1:0:0:0:0:0:0:0:0:15:55:55:54:)(trusted.ec.version:0:0:0:0:0:0:0:d:0:0:0:0:0:0:0:e:))
>> [2017-01-12 20:48:23.418531] W [MSGID: 122056] 
>> [ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-2: Mismatching 
>> xdata in answers of 'LOOKUP'
> 
> That's a real problem. Here we have two bricks that differ in the 
> trusted.ec.version xattr. However this xattr not necessarily belongs to the 
> previous directory. They are unrelated messages.
> 
>> 
>> In this case glusterfsProd-disperse-2 sub volume actually 
>> consists of the following bricks.
>> glusterfs4sds:/ws/disk11/ws_brick, glusterfs5sds: 
>> /ws/disk11/ws_brick, glusterfs6sds: /ws/disk11/ws_brick
>> 
>> I went ahead and checked the value of trusted.ec.version on all 
>> the 3 bricks inside this sub vol:
>> 
>> [root@glusterfs6 ~]# getfattr -e hex -n trusted

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

-12 20:14:25.554608] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-2: dict=0x7f0b6495903c 
((glusterfs.open-fd-count:30:0:)(trusted.glusterfs.dht:0:0:0:1:0:0:0:0:0:0:0:0:15:55:55:54:)(trusted.ec.version:0:0:0:0:0:0:0:b:0:0:0:0:0:0:0:e:))
[2017-01-12 20:14:25.554624] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-2: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=3, bad=4)
[2017-01-12 20:14:25.554632] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-glusterfsProd-disperse-2: Heal failed [Invalid argument]
[2017-01-12 20:14:25.98] I [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-2: 'trusted.ec.version' is different in two dicts (16, 
16)
[2017-01-12 20:14:25.555622] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-2: dict=0x7f0b64956c24 
((glusterfs.open-fd-count:30:0:)(trusted.glusterfs.dht:0:0:0:1:0:0:0:0:0:0:0:0:15:55:55:54:)(trusted.ec.version:0:0:0:0:0:0:0:b:0:0:0:0:0:0:0:e:))
[2017-01-12 20:14:25.555638] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-2: dict=0x7f0b64964e8c 
((glusterfs.open-fd-count:30:0:)(trusted.glusterfs.dht:0:0:0:1:0:0:0:0:0:0:0:0:15:55:55:54:)(trusted.ec.version:0:0:0:0:0:0:0:d:0:0:0:0:0:0:0:e:))


In glustershd.log lot of similar errors are logged.

[2017-01-12 21:10:53.728770] I [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-0: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-12 21:10:53.728804] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7f21694b6f50 
((trusted.ec.size:0:0:0:0:42:3a:0:0:)(trusted.ec.version:0:0:0:0:0:0:37:5f:0:0:0:0:0:0:37:5f:))
[2017-01-12 21:10:53.728827] I [dict.c:3065:dict_dump_to_log] 
0-glusterfsProd-disperse-0: dict=0x7f21694b62bc 
((trusted.ec.size:0:0:0:0:0:ca:0:0:)(trusted.ec.version:0:0:0:0:0:0:0:a1:0:0:0:0:0:0:37:5f:))
[2017-01-12 21:10:53.728842] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-0: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-12 21:10:53.728854] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-0: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1)
[2017-01-12 21:10:53.728876] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-glusterfsProd-disperse-0: Heal failed [Invalid argument]

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Thursday, January 12, 2017 6:40 AM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org
Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse 
volume

Hi Ram,


On 12/01/17 11:49, Ankireddypalle Reddy wrote:
> Xavi,
>   As I mentioned before the error could happen for any FOP. Will try 
> to run with TRACE debug level. Is there a possibility that we are checking 
> for this attribute on a directory, because a directory does not seem to be 
> having this attribute set.

No, directories do not have this attribute and no one should be reading it from 
a directory.

> Also is the function to check size and version called after it is decided 
> that heal should be run or is this check is the one which decides whether a 
> heal should be run.

Almost all checks that trigger a heal are done in the lookup fop when some 
discrepancy is detected.

The function that checks size and version is called later once a lock on the 
inode is acquired (even if no heal is needed). However further failures in the 
processing of any fop can also trigger a self-heal.

Xavi

>
> Thanks and Regards,
> Ram
>
> Sent from my iPhone
>
>> On Jan 12, 2017, at 2:25 AM, Xavier Hernandez  wrote:
>>
>> Hi Ram,
>>
>>> On 12/01/17 02:36, Ankireddypalle Reddy wrote:
>>> Xavi,
>>>  I added some more logging information. The trusted.ec.size field 
>>> values are in fact different.
>>>   trusted.ec.sizel1 = 62719407423488l2 = 0
>>
>> That's very weird. Directories do not have this attribute. It's only present 
>> on regular files. But you said that the error happens while creating the 
>> file, so it doesn't make much sense because file creation always sets 
>> trusted.ec.size to 0.
>>
>> Could you reproduce the problem with diagnostics.client-log-level set to 
>> TRACE and send the log to me ? it will create a big log, but I'll have much 
>> more information about what's going on.
>>
>> Do you have a mixed setup with nodes of different types ? for example mixed 
>> 32/64 bits architectures or different operating systems ? I ask this because 
>> 62719407423488 in hex is 0x390B, which has the lower 32 bits set to 
>> 0, but has garbage above that.
>>
>>>
>>>   This is a fair

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
  As I mentioned before the error could happen for any FOP. Will try to 
run with TRACE debug level. Is there a possibility that we are checking for 
this attribute on a directory, because a directory does not seem to be having 
this attribute set. Also is the function to check size and version called after 
it is decided that heal should be run or is this check is the one which decides 
whether a heal should be run.

Thanks and Regards,
Ram

Sent from my iPhone

> On Jan 12, 2017, at 2:25 AM, Xavier Hernandez  wrote:
> 
> Hi Ram,
> 
>> On 12/01/17 02:36, Ankireddypalle Reddy wrote:
>> Xavi,
>>  I added some more logging information. The trusted.ec.size field 
>> values are in fact different.
>>   trusted.ec.sizel1 = 62719407423488l2 = 0
> 
> That's very weird. Directories do not have this attribute. It's only present 
> on regular files. But you said that the error happens while creating the 
> file, so it doesn't make much sense because file creation always sets 
> trusted.ec.size to 0.
> 
> Could you reproduce the problem with diagnostics.client-log-level set to 
> TRACE and send the log to me ? it will create a big log, but I'll have much 
> more information about what's going on.
> 
> Do you have a mixed setup with nodes of different types ? for example mixed 
> 32/64 bits architectures or different operating systems ? I ask this because 
> 62719407423488 in hex is 0x390B, which has the lower 32 bits set to 
> 0, but has garbage above that.
> 
>> 
>>   This is a fairly static setup with no brick/ node failure.  Please 
>> explain why  is that a heal is being triggered and what could have acutually 
>> caused these size xattrs to differ.  This is causing random I/O failures and 
>> is impacting the backup schedules.
> 
> The launch of self-heal is normal because it has detected an inconsistency. 
> The real problem is what originates that inconsistency.
> 
> Xavi
> 
>> 
>> [ 2017-01-12 01:19:18.256970] W [MSGID: 122056] 
>> [ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-8: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-12 01:19:18.257015] W [MSGID: 122053] 
>> [ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-8: Operation 
>> failed on some subvolumes (up=7, mask=7, remaining=0, good=3, bad=4)
>> [2017-01-12 01:19:18.257018] W [MSGID: 122002] 
>> [ec-common.c:71:ec_heal_report] 0-glusterfsProd-disperse-8: Heal failed 
>> [Invalid argument]
>> [2017-01-12 01:19:21.002028] E [dict.c:197:key_value_cmp] 
>> 0-glusterfsProd-disperse-4: 'trusted.ec.size' is different in two dicts (8, 
>> 8)
>> [2017-01-12 01:19:21.002056] E [dict.c:166:log_value] 
>> 0-glusterfsProd-disperse-4: trusted.ec.size [ l1 = 62719407423488 l2 = 0 i1 
>> = 0 i2 = 0 ]
>> [2017-01-12 01:19:21.002064] W [MSGID: 122056] 
>> [ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-4: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-12 01:19:21.209640] E [dict.c:197:key_value_cmp] 
>> 0-glusterfsProd-disperse-4: 'trusted.ec.size' is different in two dicts (8, 
>> 8)
>> [2017-01-12 01:19:21.209673] E [dict.c:166:log_value] 
>> 0-glusterfsProd-disperse-4: trusted.ec.size [ l1 = 62719407423488 l2 = 0 i1 
>> = 0 i2 = 0 ]
>> [2017-01-12 01:19:21.209686] W [MSGID: 122056] 
>> [ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-4: Mismatching 
>> xdata in answers of 'LOOKUP'
>> [2017-01-12 01:19:21.209719] W [MSGID: 122053] 
>> [ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-4: Operation 
>> failed on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1)
>> [2017-01-12 01:19:21.209753] W [MSGID: 122002] 
>> [ec-common.c:71:ec_heal_report] 0-glusterfsProd-disperse-4: Heal failed 
>> [Invalid argument]
>> 
>> Thanks and Regards,
>> Ram
>> 
>> -Original Message-
>> From: Ankireddypalle Reddy
>> Sent: Wednesday, January 11, 2017 9:29 AM
>> To: Ankireddypalle Reddy; Xavier Hernandez; Gluster Devel 
>> (gluster-de...@gluster.org); gluster-users@gluster.org
>> Subject: RE: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse 
>> volume
>> 
>> Xavi,
>>I built a debug binary to log more information. This is what is 
>> getting logged. Looks like it is the attribute trusted.ec.size which is 
>> different among the bricks in a sub volume.
>> 
>> In glustershd.log :
>> 
>> [2017-01-11 14:19:45.023845] N [MSGID: 122029] 
>> [ec-generic.c:683:ec_combine_lookup] 0-glusterfsProd-di

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
  I added some more logging information. The trusted.ec.size field 
values are in fact different.  
   trusted.ec.sizel1 = 62719407423488l2 = 0 
   
   This is a fairly static setup with no brick/ node failure.  Please 
explain why  is that a heal is being triggered and what could have acutually 
caused these size xattrs to differ.  This is causing random I/O failures and is 
impacting the backup schedules.

[ 2017-01-12 01:19:18.256970] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-8: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-12 01:19:18.257015] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-8: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=3, bad=4)
[2017-01-12 01:19:18.257018] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-glusterfsProd-disperse-8: Heal failed [Invalid argument]
[2017-01-12 01:19:21.002028] E [dict.c:197:key_value_cmp] 
0-glusterfsProd-disperse-4: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-12 01:19:21.002056] E [dict.c:166:log_value] 
0-glusterfsProd-disperse-4: trusted.ec.size [ l1 = 62719407423488 l2 = 0 i1 = 0 
i2 = 0 ]
[2017-01-12 01:19:21.002064] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-4: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-12 01:19:21.209640] E [dict.c:197:key_value_cmp] 
0-glusterfsProd-disperse-4: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-12 01:19:21.209673] E [dict.c:166:log_value] 
0-glusterfsProd-disperse-4: trusted.ec.size [ l1 = 62719407423488 l2 = 0 i1 = 0 
i2 = 0 ]
[2017-01-12 01:19:21.209686] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-4: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-12 01:19:21.209719] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-4: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1)
[2017-01-12 01:19:21.209753] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-glusterfsProd-disperse-4: Heal failed [Invalid argument]

Thanks and Regards,
Ram

-----Original Message-
From: Ankireddypalle Reddy 
Sent: Wednesday, January 11, 2017 9:29 AM
To: Ankireddypalle Reddy; Xavier Hernandez; Gluster Devel 
(gluster-de...@gluster.org); gluster-users@gluster.org
Subject: RE: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse 
volume

Xavi,
I built a debug binary to log more information. This is what is 
getting logged. Looks like it is the attribute trusted.ec.size which is 
different among the bricks in a sub volume. 

In glustershd.log :

[2017-01-11 14:19:45.023845] N [MSGID: 122029] 
[ec-generic.c:683:ec_combine_lookup] 0-glusterfsProd-disperse-8: Mismatching 
iatt in answers of 'GF_FOP_LOOKUP'
[2017-01-11 14:19:45.027718] E [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-6: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-11 14:19:45.027736] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-6: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-11 14:19:45.027763] E [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-6: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-11 14:19:45.027781] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-6: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-11 14:19:45.027793] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-6: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1)
[2017-01-11 14:19:45.027815] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-glusterfsProd-disperse-6: Heal failed [Invalid argument]
[2017-01-11 14:19:45.029035] E [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-8: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-11 14:19:45.029057] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-8: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-11 14:19:45.029089] E [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-8: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-11 14:19:45.029105] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-8: Mismatching 
xdata in answers of 'LOOKUP'
[2017-01-11 14:19:45.029121] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-8: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1)
[2017-01-11 14:19:45.032566] E [dict.c:166:key_value_cmp] 
0-glusterfsProd-disperse-6: 'trusted.ec.size' is different in two dicts (8, 8)
[2017-01-11 14:19:45.029138] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-glusterfsProd-disperse-8: Heal failed [Invalid argument]
[2017-01-11 14:19:45.032585] W [MS

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

ment]
[2017-01-11 14:20:17.807099] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-1: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.807286] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-10: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.807298] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-10: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.807409] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-11: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.807420] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-11: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.807448] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-4: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.807462] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-4: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.807539] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-2: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.807550] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-2: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.807723] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-3: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.807739] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-3: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.807785] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-5: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.807796] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-5: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.808020] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-9: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.808034] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-9: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.808054] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-6: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.808066] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-6: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.808282] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-8: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.808292] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-8: Invalid 
config xattr [Invalid argument]
[2017-01-11 14:20:17.809212] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 2-glusterfsProd-disperse-7: Invalid or 
corrupted config [Invalid argument]
[2017-01-11 14:20:17.809228] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 2-glusterfsProd-disperse-7: Invalid 
config xattr [Invalid argument]

 [2017-01-11 14:20:17.812660] I [MSGID: 109036] 
[dht-common.c:8043:dht_log_new_layout_for_dir_selfheal] 2-glusterfsProd-dht: 
Setting layout of /Folder_01.05.2017_21.15/CV_MAGNETIC/V_31500/CHUNK_402578 
with [Subvol_name: glusterfsProd-disperse-0, Err: -1 , Start: 1789569705 , 
Stop: 2147483645 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-1, Err: -1 , 
Start: 2147483646 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: 
glusterfsProd-disperse-10, Err: -1 , Start: 2505397587 , Stop: 2863311527 , 
Hash: 1 ], [Subvol_name: glusterfsProd-disperse-11, Err: -1 , Start: 2863311528 
, Stop: 3221225468 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-2, Err: -1 
, Start: 3221225469 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: 
glusterfsProd-disperse-3, Err: -1 , Start: 3579139410 , Stop: 3937053350 , 
Hash: 1 ], [Subvol_name: glusterfsProd-disperse-4, Err: -1 , Start: 3937053351 
, Stop: 4294967295 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-5, Err: -1 
, Start: 0 , Stop: 357913940 , Has
 h: 1 ], [Subvol_name: glusterfsProd-disperse-6, Err: -1 , Start: 357913941 , 
Stop: 715827881 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-7, Err: -1 , 
Start: 715827882 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: 
glusterfsProd-disperse-8, Err: -1 , Start: 1073741823 , Stop: 1431655763 , 
Hash: 1 ], [Subvol_name: glusterfsProd-disperse-9, Err: -1 , Start: 1431655764 
, Stop: 1789569704 , Hash: 1 ],


-Original Message-
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Tuesday, January 10, 2017 10:09 AM
To: Xavier Hernandez; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@g

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
   In this case it's the file creation which failed. So I provided the 
xattrs of the parent. 

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Tuesday, January 10, 2017 9:10 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram,

On 10/01/17 14:42, Ankireddypalle Reddy wrote:
> Attachments (2):
>
> 1
>
>   
>
> ec.txt
> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co
> mmvault.com/webconsole/api/drive/publicshare/346714/file/ee2d1536c2dc4
> dff94afb12132b4f8f6/action/preview&downloadUrl=https://imap.commvault.
> com/webconsole/api/contentstore/publicshare/346714/file/ee2d1536c2dc4d
> ff94afb12132b4f8f6/action/download>
> [Download]
> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34
> 6714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/download>(11.50
> KB)
>
> 2
>
>   
>
> ws-glus.log
> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co
> mmvault.com/webconsole/api/drive/publicshare/346714/file/cff3e0506e754
> b9a939db02da1cbbd58/action/preview&downloadUrl=https://imap.commvault.
> com/webconsole/api/contentstore/publicshare/346714/file/cff3e0506e754b
> 9a939db02da1cbbd58/action/download>
> [Download]
> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34
> 6714/file/cff3e0506e754b9a939db02da1cbbd58/action/download>(3.48
> MB)
>
> Xavi,
>   We are encountering errors for different kinds of FOPS.
>   The open failed for the following file:
>
>   cvd_2017_01_10_02_28_26.log:98182 1f9fe 01/10 00:57:10 8414465
> [MEDIAFS] 20117519-52075477 SingleInstancer_FS::StartDataFile2:
> Failed to create the data file
> [/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720
> /SFILE_CONTAINER_062], error=0xECCC0005:{CQiFile::Open(92)} + 
> {CQiUTFOSAPI::open(96)/ErrNo.5.(Input/output error)-Open failed, 
> File=/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_5134
> 2720/SFILE_CONTAINER_062, OperationFlag=0xC1, PermissionMode=0x1FF}
>
>   I've attached the extended attributes for the directories
>   /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/ and
>
> /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720
> from all the bricks.
>
>  The attributes look fine to me. I've also attached some log 
> cuts to illustrate the problem.

I need the extended attributes of the file itself, not the parent directories.

Xavi

>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
> Sent: Tuesday, January 10, 2017 7:53 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
> gluster-users@gluster.org
> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>
> Hi Ram,
>
> the error is caused by an extended attribute that does not match on 
> all
> 3 bricks of the disperse set. Most probable value is 
> trusted.ec.version, but could be others.
>
> At first sight, I don't see any change from 3.7.8 that could have 
> caused this. I'll check again.
>
> What kind of operations are you doing ? this can help me narrow the search.
>
> Xavi
>
> On 10/01/17 13:43, Ankireddypalle Reddy wrote:
>> Xavi,
>>   Thanks. If you could please explain what to look for in the
> extended attributes then I will check and let you know if I find 
> anything suspicious.  Also we noticed that some of these operations 
> would succeed if retried. Do you know of any communicated related 
> errors that are being reported/triaged.
>>
>> Thanks and Regards,
>> Ram
>>
>> -----Original Message-
>> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
>> Sent: Tuesday, January 10, 2017 7:23 AM
>> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
>> gluster-users@gluster.org
>> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>>
>> Hi Ram,
>>
>> On 10/01/17 13:14, Ankireddypalle Reddy wrote:
>>> Attachment (1):
>>>
>>> 1
>>>
>>>
>>>
>>> ecxattrs.txt
>>> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.
>>> c
>>> o
>>> mmvault.com/webconsole/api/drive/publicshare/346714/file/1272e682787
>>> 4
>>> 4
>>> f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault.
>>> com/webconsole/

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Attachments (2):

1

ec.txt<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/download>
 
[Download]<https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/ee2d1536c2dc4dff94afb12132b4f8f6/action/download>
 (11.50 KB)

2

ws-glus.log<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/cff3e0506e754b9a939db02da1cbbd58/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/cff3e0506e754b9a939db02da1cbbd58/action/download>
 
[Download]<https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/cff3e0506e754b9a939db02da1cbbd58/action/download>
 (3.48 MB)


Xavi,
  We are encountering errors for different kinds of FOPS.
  The open failed for the following file:

  cvd_2017_01_10_02_28_26.log:98182 1f9fe 01/10 00:57:10 8414465 
[MEDIAFS] 20117519-52075477 SingleInstancer_FS::StartDataFile2: Failed to 
create the data file 
[/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720/SFILE_CONTAINER_062],
 error=0xECCC0005:{CQiFile::Open(92)} + 
{CQiUTFOSAPI::open(96)/ErrNo.5.(Input/output error)-Open failed, 
File=/ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720/SFILE_CONTAINER_062,
 OperationFlag=0xC1, PermissionMode=0x1FF}

  I've attached the extended attributes for the directories
  /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/ and
  /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854974/CHUNK_51342720 
from all the bricks.

 The attributes look fine to me. I've also attached some log cuts to 
illustrate the problem.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es]
Sent: Tuesday, January 10, 2017 7:53 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram,

the error is caused by an extended attribute that does not match on all
3 bricks of the disperse set. Most probable value is trusted.ec.version, but 
could be others.

At first sight, I don't see any change from 3.7.8 that could have caused this. 
I'll check again.

What kind of operations are you doing ? this can help me narrow the search.

Xavi

On 10/01/17 13:43, Ankireddypalle Reddy wrote:
> Xavi,
>   Thanks. If you could please explain what to look for in the 
> extended attributes then I will check and let you know if I find anything 
> suspicious.  Also we noticed that some of these operations would succeed if 
> retried. Do you know of any communicated related errors that are being 
> reported/triaged.
>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
> Sent: Tuesday, January 10, 2017 7:23 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org);
> gluster-users@gluster.org
> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>
> Hi Ram,
>
> On 10/01/17 13:14, Ankireddypalle Reddy wrote:
>> Attachment (1):
>>
>> 1
>>
>>
>>
>> ecxattrs.txt
>> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.c
>> o
>> mmvault.com/webconsole/api/drive/publicshare/346714/file/1272e6827874
>> 4
>> f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault.
>> com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744
>> f
>> 15bf1a54f2b31b559d/action/download>
>> [Download]
>> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/3
>> 4
>> 6714/file/1272e68278744f15bf1a54f2b31b559d/action/download>(5.92
>> KB)
>>
>> Xavi,
>>  Please find attached the extended attributes for a
>> directory from all the bricks. Free space check failed for this with
>> error number EIO.
>
> What do you mean ? what operation have you made to check the free space on 
> that directory ?
>
> If it's a recursive check, I need the extended attributes from the exact file 
> that triggers the EIO. The attached attributes seem consistent and that 
> directory shouldn't cause any problem. Does an 'ls' on that directory fail or 
> does it show the contents ?
>
> Xavi
>
>>
>> Thanks and Regards,
>> Ram
>>
>> -Original Message-
>> From: Xavier Hernandez [mailto:xhe

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
  Thanks. If you could please explain what to look for in the extended 
attributes then I will check and let you know if I find anything suspicious.  
Also we noticed that some of these operations would succeed if retried. Do you 
know of any communicated related errors that are being reported/triaged.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Tuesday, January 10, 2017 7:23 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram,

On 10/01/17 13:14, Ankireddypalle Reddy wrote:
> Attachment (1):
>
> 1
>
>   
>
> ecxattrs.txt
> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co
> mmvault.com/webconsole/api/drive/publicshare/346714/file/1272e68278744
> f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault.
> com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744f
> 15bf1a54f2b31b559d/action/download>
> [Download]
> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34
> 6714/file/1272e68278744f15bf1a54f2b31b559d/action/download>(5.92
> KB)
>
> Xavi,
>  Please find attached the extended attributes for a 
> directory from all the bricks. Free space check failed for this with 
> error number EIO.

What do you mean ? what operation have you made to check the free space on that 
directory ?

If it's a recursive check, I need the extended attributes from the exact file 
that triggers the EIO. The attached attributes seem consistent and that 
directory shouldn't cause any problem. Does an 'ls' on that directory fail or 
does it show the contents ?

Xavi

>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
> Sent: Tuesday, January 10, 2017 6:45 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
> gluster-users@gluster.org
> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>
> Hi Ram,
>
> can you execute the following command on all bricks on a file that is 
> giving EIO ?
>
> getfattr -m. -e hex -d 
>
> Xavi
>
> On 10/01/17 12:41, Ankireddypalle Reddy wrote:
>> Xavi,
>> We have been running 3.7.8 on these servers. We upgraded
> to 3.7.18 yesterday. We upgraded all the servers at a time.  The 
> volume was brought down during upgrade.
>>
>> Thanks and Regards,
>> Ram
>>
>> -Original Message-
>> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
>> Sent: Tuesday, January 10, 2017 6:35 AM
>> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
>> gluster-users@gluster.org
>> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>>
>> Hi Ram,
>>
>> how did you upgrade gluster ? from which version ?
>>
>> Did you upgrade one server at a time and waited until self-heal
> finished before upgrading the next server ?
>>
>> Xavi
>>
>> On 10/01/17 11:39, Ankireddypalle Reddy wrote:
>>> Hi,
>>>
>>>   We upgraded to GlusterFS 3.7.18 yesterday.  We see lot of 
>>> failures in our applications. Most of the errors are EIO. The 
>>> following log lines are commonly seen in the logs:
>>>
>>>
>>>
>>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]
>>> 0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'"
>>> repeated 2 times between [2017-01-10 02:46:25.069809] and 
>>> [2017-01-10 02:46:25.069835]
>>>
>>> [2017-01-10 02:46:25.069852] W [MSGID: 122056] 
>>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5:
>>> Mismatching xdata in answers of 'LOOKUP'
>>>
>>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]
>>> 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'"
>>> repeated 2 times between [2017-01-10 02:46:25.069852] and 
>>> [2017-01-10 02:46:25.069873]
>>>
>>> [2017-01-10 02:46:25.069910] W [MSGID: 122056] 
>>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6:
>>> Mismatching xdata in answers of 'LOOKUP'
>>>
>>> ...
>>>
>>> [2017-01-10 02:46:26.520774] I [MSGID: 109036] 
>>> [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal]
>>> 0-StoragePool-dht: Setting layout of
>>> /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585 with
>>> [Subvol_nam

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Attachment (1):

1

ecxattrs.txt<https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/1272e68278744f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744f15bf1a54f2b31b559d/action/download>
 
[Download]<https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744f15bf1a54f2b31b559d/action/download>
 (5.92 KB)


Xavi,
 Please find attached the extended attributes for a directory from 
all the bricks. Free space check failed for this with error number EIO.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es]
Sent: Tuesday, January 10, 2017 6:45 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram,

can you execute the following command on all bricks on a file that is giving 
EIO ?

getfattr -m. -e hex -d 

Xavi

On 10/01/17 12:41, Ankireddypalle Reddy wrote:
> Xavi,
> We have been running 3.7.8 on these servers. We upgraded to 
> 3.7.18 yesterday. We upgraded all the servers at a time.  The volume was 
> brought down during upgrade.
>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Xavier Hernandez [mailto:xhernan...@datalab.es]
> Sent: Tuesday, January 10, 2017 6:35 AM
> To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org);
> gluster-users@gluster.org
> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume
>
> Hi Ram,
>
> how did you upgrade gluster ? from which version ?
>
> Did you upgrade one server at a time and waited until self-heal finished 
> before upgrading the next server ?
>
> Xavi
>
> On 10/01/17 11:39, Ankireddypalle Reddy wrote:
>> Hi,
>>
>>   We upgraded to GlusterFS 3.7.18 yesterday.  We see lot of
>> failures in our applications. Most of the errors are EIO. The
>> following log lines are commonly seen in the logs:
>>
>>
>>
>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]
>> 0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'"
>> repeated 2 times between [2017-01-10 02:46:25.069809] and [2017-01-10
>> 02:46:25.069835]
>>
>> [2017-01-10 02:46:25.069852] W [MSGID: 122056]
>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5:
>> Mismatching xdata in answers of 'LOOKUP'
>>
>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]
>> 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'"
>> repeated 2 times between [2017-01-10 02:46:25.069852] and [2017-01-10
>> 02:46:25.069873]
>>
>> [2017-01-10 02:46:25.069910] W [MSGID: 122056]
>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6:
>> Mismatching xdata in answers of 'LOOKUP'
>>
>> ...
>>
>> [2017-01-10 02:46:26.520774] I [MSGID: 109036]
>> [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal]
>> 0-StoragePool-dht: Setting layout of
>> /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585 with
>> [Subvol_name: StoragePool-disperse-0, Err: -1 , Start: 3221225466 ,
>> Stop: 3758096376 , Hash: 1 ], [Subvol_name: StoragePool-disperse-1, Err:
>> -1 , Start: 3758096377 , Stop: 4294967295 , Hash: 1 ], [Subvol_name:
>> StoragePool-disperse-2, Err: -1 , Start: 0 , Stop: 536870910 , Hash:
>> 1 ], [Subvol_name: StoragePool-disperse-3, Err: -1 , Start: 536870911
>> ,
>> Stop: 1073741821 , Hash: 1 ], [Subvol_name: StoragePool-disperse-4, Err:
>> -1 , Start: 1073741822 , Stop: 1610612732 , Hash: 1 ], [Subvol_name:
>> StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop:
>> 2147483643 ,
>> Hash: 1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 , Start:
>> 2147483644 , Stop: 2684354554 , Hash: 1 ], [Subvol_name:
>> StoragePool-disperse-7, Err: -1 , Start: 2684354555 , Stop:
>> 3221225465 ,
>> Hash: 1 ],
>>
>> [2017-01-10 02:46:26.522841] N [MSGID: 122031]
>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3:
>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>>
>> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]
>> 0-StoragePool-disperse-3: Mismatching dictionary in answers of
>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10
>> 02:46:26.522841] and [2017-01-10 02:46:26.522894]
>>
>> [2017-01-10 02:46:26.522898] W [MSGID: 122040]
>> [ec-common.c:91

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Xavi,
We have been running 3.7.8 on these servers. We upgraded to 3.7.18 
yesterday. We upgraded all the servers at a time.  The volume was brought down 
during upgrade. 

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Tuesday, January 10, 2017 6:35 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-de...@gluster.org); 
gluster-users@gluster.org
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram,

how did you upgrade gluster ? from which version ?

Did you upgrade one server at a time and waited until self-heal finished before 
upgrading the next server ?

Xavi

On 10/01/17 11:39, Ankireddypalle Reddy wrote:
> Hi,
>
>   We upgraded to GlusterFS 3.7.18 yesterday.  We see lot of 
> failures in our applications. Most of the errors are EIO. The 
> following log lines are commonly seen in the logs:
>
>
>
> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]
> 0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'"
> repeated 2 times between [2017-01-10 02:46:25.069809] and [2017-01-10 
> 02:46:25.069835]
>
> [2017-01-10 02:46:25.069852] W [MSGID: 122056] 
> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5:
> Mismatching xdata in answers of 'LOOKUP'
>
> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]
> 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'"
> repeated 2 times between [2017-01-10 02:46:25.069852] and [2017-01-10 
> 02:46:25.069873]
>
> [2017-01-10 02:46:25.069910] W [MSGID: 122056] 
> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6:
> Mismatching xdata in answers of 'LOOKUP'
>
> ...
>
> [2017-01-10 02:46:26.520774] I [MSGID: 109036] 
> [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal]
> 0-StoragePool-dht: Setting layout of
> /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585 with
> [Subvol_name: StoragePool-disperse-0, Err: -1 , Start: 3221225466 ,
> Stop: 3758096376 , Hash: 1 ], [Subvol_name: StoragePool-disperse-1, Err:
> -1 , Start: 3758096377 , Stop: 4294967295 , Hash: 1 ], [Subvol_name:
> StoragePool-disperse-2, Err: -1 , Start: 0 , Stop: 536870910 , Hash: 1 
> ], [Subvol_name: StoragePool-disperse-3, Err: -1 , Start: 536870911 ,
> Stop: 1073741821 , Hash: 1 ], [Subvol_name: StoragePool-disperse-4, Err:
> -1 , Start: 1073741822 , Stop: 1610612732 , Hash: 1 ], [Subvol_name:
> StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop: 2147483643 
> ,
> Hash: 1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 , Start:
> 2147483644 , Stop: 2684354554 , Hash: 1 ], [Subvol_name:
> StoragePool-disperse-7, Err: -1 , Start: 2684354555 , Stop: 3221225465 
> ,
> Hash: 1 ],
>
> [2017-01-10 02:46:26.522841] N [MSGID: 122031] 
> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3:
> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>
> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]
> 0-StoragePool-disperse-3: Mismatching dictionary in answers of 
> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 
> 02:46:26.522841] and [2017-01-10 02:46:26.522894]
>
> [2017-01-10 02:46:26.522898] W [MSGID: 122040] 
> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-3: 
> Failed to get size and version [Input/output error]
>
> [2017-01-10 02:46:26.523115] N [MSGID: 122031] 
> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-6:
> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>
> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]
> 0-StoragePool-disperse-6: Mismatching dictionary in answers of 
> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 
> 02:46:26.523115] and [2017-01-10 02:46:26.523143]
>
> [2017-01-10 02:46:26.523147] W [MSGID: 122040] 
> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-6: 
> Failed to get size and version [Input/output error]
>
> [2017-01-10 02:46:26.523302] N [MSGID: 122031] 
> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-2:
> Mismatching dictionary in answers of 'GF_FOP_XATTROP'
>
> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]
> 0-StoragePool-disperse-2: Mismatching dictionary in answers of 
> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 
> 02:46:26.523302] and [2017-01-10 02:46:26.523324]
>
> [2017-01-10 02:46:26.523328] W [MSGID: 122040] 
> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-2: 
> Failed to get size and version [Input/output error]
>
>
>
> [root@glusterfs3 Log_Files]# gluster -

[Gluster-users] Lot of EIO errors in disperse volume

[Ankireddypalle Reddy]

Hi,
  We upgraded to GlusterFS 3.7.18 yesterday.  We see lot of failures in our 
applications. Most of the errors are EIO. The following log lines are commonly 
seen in the logs:

The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 
0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'" repeated 2 
times between [2017-01-10 02:46:25.069809] and [2017-01-10 02:46:25.069835]
[2017-01-10 02:46:25.069852] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5: Mismatching xdata 
in answers of 'LOOKUP'
The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 
0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'" repeated 2 
times between [2017-01-10 02:46:25.069852] and [2017-01-10 02:46:25.069873]
[2017-01-10 02:46:25.069910] W [MSGID: 122056] 
[ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6: Mismatching xdata 
in answers of 'LOOKUP'
...
[2017-01-10 02:46:26.520774] I [MSGID: 109036] 
[dht-common.c:9076:dht_log_new_layout_for_dir_selfheal] 0-StoragePool-dht: 
Setting layout of /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585 
with [Subvol_name: StoragePool-disperse-0, Err: -1 , Start: 3221225466 , Stop: 
3758096376 , Hash: 1 ], [Subvol_name: StoragePool-disperse-1, Err: -1 , Start: 
3758096377 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: 
StoragePool-disperse-2, Err: -1 , Start: 0 , Stop: 536870910 , Hash: 1 ], 
[Subvol_name: StoragePool-disperse-3, Err: -1 , Start: 536870911 , Stop: 
1073741821 , Hash: 1 ], [Subvol_name: StoragePool-disperse-4, Err: -1 , Start: 
1073741822 , Stop: 1610612732 , Hash: 1 ], [Subvol_name: 
StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop: 2147483643 , Hash: 
1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 , Start: 2147483644 , Stop: 
2684354554 , Hash: 1 ], [Subvol_name: StoragePool-disperse-7, Err: -1 , Start: 
2684354555 , Stop: 3221225465 , Hash: 1 ],
[2017-01-10 02:46:26.522841] N [MSGID: 122031] 
[ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3: Mismatching 
dictionary in answers of 'GF_FOP_XATTROP'
The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 
0-StoragePool-disperse-3: Mismatching dictionary in answers of 
'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 02:46:26.522841] and 
[2017-01-10 02:46:26.522894]
[2017-01-10 02:46:26.522898] W [MSGID: 122040] 
[ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-3: Failed to get 
size and version [Input/output error]
[2017-01-10 02:46:26.523115] N [MSGID: 122031] 
[ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-6: Mismatching 
dictionary in answers of 'GF_FOP_XATTROP'
The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 
0-StoragePool-disperse-6: Mismatching dictionary in answers of 
'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 02:46:26.523115] and 
[2017-01-10 02:46:26.523143]
[2017-01-10 02:46:26.523147] W [MSGID: 122040] 
[ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-6: Failed to get 
size and version [Input/output error]
[2017-01-10 02:46:26.523302] N [MSGID: 122031] 
[ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-2: Mismatching 
dictionary in answers of 'GF_FOP_XATTROP'
The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 
0-StoragePool-disperse-2: Mismatching dictionary in answers of 
'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 02:46:26.523302] and 
[2017-01-10 02:46:26.523324]
[2017-01-10 02:46:26.523328] W [MSGID: 122040] 
[ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-2: Failed to get 
size and version [Input/output error]

[root@glusterfs3 Log_Files]# gluster --version
glusterfs 3.7.18 built on Dec  8 2016 06:34:26

[root@glusterfs3 Log_Files]# gluster volume info

Volume Name: StoragePool
Type: Distributed-Disperse
Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
Status: Started
Number of Bricks: 8 x (2 + 1) = 24
Transport-type: tcp
Bricks:
Brick1: glusterfs1sds:/ws/disk1/ws_brick
Brick2: glusterfs2sds:/ws/disk1/ws_brick
Brick3: glusterfs3sds:/ws/disk1/ws_brick
Brick4: glusterfs1sds:/ws/disk2/ws_brick
Brick5: glusterfs2sds:/ws/disk2/ws_brick
Brick6: glusterfs3sds:/ws/disk2/ws_brick
Brick7: glusterfs1sds:/ws/disk3/ws_brick
Brick8: glusterfs2sds:/ws/disk3/ws_brick
Brick9: glusterfs3sds:/ws/disk3/ws_brick
Brick10: glusterfs1sds:/ws/disk4/ws_brick
Brick11: glusterfs2sds:/ws/disk4/ws_brick
Brick12: glusterfs3sds:/ws/disk4/ws_brick
Brick13: glusterfs1sds:/ws/disk5/ws_brick
Brick14: glusterfs2sds:/ws/disk5/ws_brick
Brick15: glusterfs3sds:/ws/disk5/ws_brick
Brick16: glusterfs1sds:/ws/disk6/ws_brick
Brick17: glusterfs2sds:/ws/disk6/ws_brick
Brick18: glusterfs3sds:/ws/disk6/ws_brick
Brick19: glusterfs1sds:/ws/disk7/ws_brick
Brick20: glusterfs2sds:/ws/disk7/ws_brick
Brick21: glusterfs3sds:/ws/disk7/ws_brick
Brick22: glusterfs1sds:/ws/disk8/ws_brick
Brick23: glusterfs2sds:/ws/disk8/ws_brick
Brick24: glusterfs3sds:/ws/disk8/ws_brick
Options Reconf

[Gluster-users] Need to understand this logging in disperse volume

[Ankireddypalle Reddy]

Hi,
 In our application we replace a file named sample with a file name 
sample.compact. Here's the sequence of steps.

1)  Rename sample to sample.temp

2)  Rename sample.compact to sample

3)  Unlink sample.temp

Please note that this is a 3:1 disperse volume and each node is also a client. 
It's a FUSE mount.

Here's the corresponding logging:
[2016-12-24 09:23:25.407934] I [MSGID: 109066] [dht-rename.c:1413:dht_rename] 
0-StoragePool-dht: renaming 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002
 (hash=StoragePool-disperse-6/cache=StoragePool-disperse-6) => 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002.temp
 (hash=StoragePool-disperse-4/cache=)

[2016-12-24 09:23:25.829296] I [MSGID: 109066] [dht-rename.c:1413:dht_rename] 
0-StoragePool-dht: renaming 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002.compact
 (hash=StoragePool-disperse-0/cache=StoragePool-disperse-0) => 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002
 (hash=StoragePool-disperse-6/cache=)

At this point of time when I check for the bricks on which the file could be 
present I see the trusted.ec.config attribute set.


glusterfs.pathinfo="( ( 

 

 
))"


ws/disk1/ws_brick/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002

trusted.ec.config=0x080301000200

But I notice the following logging in the log files.

2016-12-24 09:23:26.759164] W [MSGID: 114031] 
[client-rpc-fops.c:1848:client3_3_xattrop_cbk] 0-StoragePool-client-14: remote 
operation failed. Path: 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002
 (cf977efe-e263-49b9-8099-06b36122e715) [No such file or directory]
[2016-12-24 09:23:26.759202] W [MSGID: 114031] 
[client-rpc-fops.c:1848:client3_3_xattrop_cbk] 0-StoragePool-client-12: remote 
operation failed. Path: 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002
 (cf977efe-e263-49b9-8099-06b36122e715) [No such file or directory]
[2016-12-24 09:23:26.759383] W [MSGID: 114031] 
[client-rpc-fops.c:1848:client3_3_xattrop_cbk] 0-StoragePool-client-13: remote 
operation failed. Path: 
/Folder_07.11.2016_23.02/CV_MAGNETIC/V_8772830/CHUNK_49113632/SFILE_CONTAINER_002
 (cf977efe-e263-49b9-8099-06b36122e715) [No such file or directory]

Client 14 is
volume StoragePool-client-14
...
option remote-subvolume /ws/disk5/ws_brick
...
end-volume

Why is it that the extended attribute is being checked on a wrong brick 
/ws/disk5 while the correct brick is /ws/disk1. I see lot of these errors being 
logged.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Error being logged in disperse volumes

[Ankireddypalle Reddy]

Ashish,
 Thanks for getting back on this. This does not happen always. 
Under heavy I/O we see such errors getting logged.

Thanks and Regards,
Ram

From: Ashish Pandey [mailto:aspan...@redhat.com]
Sent: Tuesday, December 20, 2016 10:25 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-users] Error being logged in disperse volumes


That means ec is not getting correct trusted.ec.config xattr from minimum 
number of bricks.

1 - Did you see any error on client side while accessing any file?
2 - If yes, check the file xattr's from all the bricks for such files.

It is too less information to find out the cause. If [1] is true then you have 
to give all client logs, getxattr from all the bricks for all the file giving 
error.
gluster v status, info and also gluster volume heal info.

Is there anything you changed recently on volume?

Ashish




From: "Ankireddypalle Reddy" mailto:are...@commvault.com>>
To: gluster-users@gluster.org<mailto:gluster-users@gluster.org>, "Gluster Devel 
(gluster-de...@gluster.org<mailto:gluster-de...@gluster.org>)" 
mailto:gluster-de...@gluster.org>>
Sent: Tuesday, December 20, 2016 7:42:29 PM
Subject: [Gluster-users] Error being logged in disperse volumes

Hi,
I am seeing many instances of the following error in the log files. 
What does this signify.

[2016-12-19 08:14:04.988004] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-1: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988027] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-1: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988038] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-0: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988055] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-0: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988179] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-3: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988193] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-3: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988228] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-2: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988248] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-2: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988338] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-4: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988350] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-4: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988374] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-5: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988388] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-5: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988460] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-7: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988478] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-7: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:05.508034] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-6: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:05.508072] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-6: Invalid 
config xattr [Invalid argument]

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-users mailing list
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users

***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
ple

[Gluster-users] Error being logged in disperse volumes

[Ankireddypalle Reddy]

Hi,
I am seeing many instances of the following error in the log files. 
What does this signify.

[2016-12-19 08:14:04.988004] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-1: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988027] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-1: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988038] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-0: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988055] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-0: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988179] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-3: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988193] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-3: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988228] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-2: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988248] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-2: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988338] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-4: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988350] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-4: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988374] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-5: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988388] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-5: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:04.988460] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-7: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:04.988478] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-7: Invalid 
config xattr [Invalid argument]
[2016-12-19 08:14:05.508034] E [MSGID: 122001] 
[ec-common.c:872:ec_config_check] 0-StoragePool-disperse-6: Invalid or 
corrupted config [Invalid argument]
[2016-12-19 08:14:05.508072] E [MSGID: 122066] 
[ec-common.c:969:ec_prepare_update_cbk] 0-StoragePool-disperse-6: Invalid 
config xattr [Invalid argument]

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Hole punch support

[Ankireddypalle Reddy]

Niels,
 Below are the two issues that I noticed and created bugs for 
tracking.
 1)  For disperse volumes hole punch is currently not supported. 
The support seems to be in works.
   https://bugzilla.redhat.com/show_bug.cgi?id=1394298

 2)  For a replica/distribute volume after punching hole in the 
file the reported size is not correct.
   You could see the description of problem in 
   https://bugzilla.redhat.com/show_bug.cgi?id=1398381

Thanks and Regards,
Ram   

-Original Message-
From: Niels de Vos [mailto:nde...@redhat.com] 
Sent: Monday, November 28, 2016 10:32 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org)
Subject: Re: [Gluster-devel] Hole punch support

On Thu, Nov 24, 2016 at 03:58:01PM +, Ankireddypalle Reddy wrote:
> Hi,
>   Gluster does not seem to be supporting hole punch support correctly. 
> This might be a limiting factor for people switching from hardware based 
> storage  to glusterfs for archival/backup use cases.  In these use cases hole 
> punch support is one of the key deciding factors. I request gluster 
> developers to expedite their efforts in this direction.

Hi Ram,

this is something we should be able to fix somehow. There is a similar bug 
reported for this already:

  fallocate + punch_hole used instead of fallocate + zero_range in 
glfs_discard()
  https://bugzilla.redhat.com/show_bug.cgi?id=1327857

>From a cursory look, libgfapi supports punching holes through the
glfs_discard() function. FUSE also has an implementation for fallocate().

How are you testing this, and what is the behaviour you see?

Thanks,
Niels
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Hole punch support

[Ankireddypalle Reddy]

Hi,
  Gluster does not seem to be supporting hole punch support correctly. This 
might be a limiting factor for people switching from hardware based storage  to 
glusterfs for archival/backup use cases.  In these use cases hole punch support 
is one of the key deciding factors. I request gluster developers to expedite 
their efforts in this direction.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gfid generation

[Ankireddypalle Reddy]

Kaushal/Pranith,
  Thanks for clarifying this. As I understand 
there are 2 id's. Please correct if there is a mistake in my assumptions:
  1) HASH generated by DHT and this will 
generate the same id for a given file all the time.
  2) GFID which is an version 4 UUID. As per 
the below links this is supposed to contain a time stamp field in it.  So this 
will not generate the same id for a given file all the time.
   
https://en.wikipedia.org/wiki/Universally_unique_identifier
   https://tools.ietf.org/html/rfc4122
 
Thanks and Regards,
ram
-Original Message-
From: Kaushal M [mailto:kshlms...@gmail.com] 
Sent: Tuesday, November 15, 2016 1:21 PM
To: Ankireddypalle Reddy
Cc: Pranith Kumar Karampuri; gluster-users@gluster.org; Gluster Devel
Subject: Re: [Gluster-users] gfid generation

On Tue, Nov 15, 2016 at 11:33 PM, Ankireddypalle Reddy  
wrote:
> Pranith,
>
>  Thanks for getting back on this. I am trying to see 
> how gfid can be generated programmatically. Given a file name how do 
> we generate gfid for it. I was reading some of the email threads about 
> it where it was mentioned that gfid is generated based upon parent 
> directory gfid and the file name. Given a same parent gfid and file 
> name do we always end up with the same gfid.

You're probably confusing the hash as generated for the elastic hash algorithm 
in DHT, with UUID. That is a combination of

I always thought that the GFID was a UUID, which was randomly generated. (The 
random UUID might be being modified a little to allow some leeway with 
directory listing, IIRC).

Adding gluster-devel to get more eyes on this.

>
>
>
> Thanks and Regards,
>
> ram
>
>
>
> From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> Sent: Tuesday, November 15, 2016 12:58 PM
> To: Ankireddypalle Reddy
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] gfid generation
>
>
>
> Sorry, didn't understand the question. Are you saying give a file on 
> gluster how to get gfid of the file?
>
> #getfattr -d -m. -e hex /path/to/file shows it
>
>
>
> On Fri, Nov 11, 2016 at 9:47 PM, Ankireddypalle Reddy 
> 
> wrote:
>
> Hi,
>
> Is the mapping from file name to gfid an idempotent operation.  
> If so please point me to the function that does this.
>
>
>
> Thanks and Regards,
>
> Ram
>
> ***Legal Disclaimer***
>
> "This communication may contain confidential and privileged material 
> for the
>
> sole use of the intended recipient. Any unauthorized review, use or 
> distribution
>
> by others is strictly prohibited. If you have received the message by 
> mistake,
>
> please advise the sender by reply email and delete the message. Thank you."
>
> **
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
>
> Pranith
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gfid generation

[Ankireddypalle Reddy]

Pranith,
 Thanks for getting back on this. I am trying to see how gfid 
can be generated programmatically. Given a file name how do we generate gfid 
for it. I was reading some of the email threads about it where it was mentioned 
that gfid is generated based upon parent directory gfid and the file name. 
Given a same parent gfid and file name do we always end up with the same gfid.

Thanks and Regards,
ram

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, November 15, 2016 12:58 PM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] gfid generation

Sorry, didn't understand the question. Are you saying give a file on gluster 
how to get gfid of the file?
#getfattr -d -m. -e hex /path/to/file shows it

On Fri, Nov 11, 2016 at 9:47 PM, Ankireddypalle Reddy 
mailto:are...@commvault.com>> wrote:
Hi,
Is the mapping from file name to gfid an idempotent operation.  If so 
please point me to the function that does this.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

___
Gluster-users mailing list
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users



--
Pranith
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gfid generation

[Ankireddypalle Reddy]

Hi,
Is the mapping from file name to gfid an idempotent operation.  If so 
please point me to the function that does this.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Hole punch support

[Ankireddypalle Reddy]

Ravi,
   Thanks for the explanation.  Filed a bug.

https://bugzilla.redhat.com/show_bug.cgi?id=1394298

Thanks and Regards,
Ram
From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Friday, November 11, 2016 10:01 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org; Gluster Devel
Subject: Re: [Gluster-users] Hole punch support

+ gluster-devel.

Can you raise an RFE bug for this and assign it to me?
The thing is,  FALLOC_FL_PUNCH_HOLE must be used in tandem with 
FALLOC_FL_KEEP_SIZE, and the latter is currently broken in gluster because 
there are some conversions done in iatt_from_stat() in gluster for quota to 
work. I'm not sure if these are needed anymore, or can be circumvented, but it 
is about time we looked into it.

Thanks,
Ravi

On 11/11/2016 07:55 PM, Ankireddypalle Reddy wrote:
Hi,
   Any idea about when will hole punch support be available with glusterfs.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



___

Gluster-users mailing list

Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>

http://www.gluster.org/mailman/listinfo/gluster-users


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Hole punch support

[Ankireddypalle Reddy]

Hi,
   Any idea about when will hole punch support be available with glusterfs.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] understanding dht value

[Ankireddypalle Reddy]

Thanks for pointing to the article. I have been following the article all the 
way.  What intrigues me is the dht values associated with sub directories.

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol
trusted.glusterfs.dht=0x00017de2

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/
trusted.glusterfs.dht=0x00017ffe

Does it mean that only files whose DHT value ranges from 0x00 to 0x7ffe can 
be saved inside the 'd' directory. But then it provides a very narrow range of 
0x7de2 to  0x7ffe to be created in that directory.

Thanks and Regards,
Ram

From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Julian
Sent: Tuesday, November 08, 2016 2:06 PM
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] understanding dht value


Here's an article explaining how dht works. The hash maps are per-directory.

https://joejulian.name/blog/dht-misses-are-expensive/

On 11/08/2016 11:04 AM, Ankireddypalle Reddy wrote:
Hi,
   I am trying to make sense of the hash values that get assigned/used by 
DHT.
   /brick1/vol and /brick2/vol are the directories that are being used as 
bricks in a distributed  replicated volume.

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick1/vol
getfattr: Removing leading '/' from absolute path names
# file: brick1/vol
trusted.glusterfs.dht=0x00017de1
This means that any file gets hashed to a value from 0x00 to 0x7de1 gets 
stored on this brick.


[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol
trusted.glusterfs.dht=0x00017de2
This means that any file that's hashed to a value from 0x7de2 to 0x 
gets stored on this brick.


What is confusing is the dht values that are shown for the directories inside 
these brick directories. What do the dht values associated with the sub 
directories signify.

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/
trusted.glusterfs.dht=0x00017ffe

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/e
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/e
trusted.glusterfs.dht=0x00017ffe


[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/f
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/f
trusted.glusterfs.dht=0x00017fff

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



___

Gluster-users mailing list

Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>

http://www.gluster.org/mailman/listinfo/gluster-users

***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] understanding dht value

[Ankireddypalle Reddy]

Hi,
   I am trying to make sense of the hash values that get assigned/used by 
DHT.
   /brick1/vol and /brick2/vol are the directories that are being used as 
bricks in a distributed  replicated volume.

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick1/vol
getfattr: Removing leading '/' from absolute path names
# file: brick1/vol
trusted.glusterfs.dht=0x00017de1
This means that any file gets hashed to a value from 0x00 to 0x7de1 gets 
stored on this brick.


[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol
trusted.glusterfs.dht=0x00017de2
This means that any file that's hashed to a value from 0x7de2 to 0x 
gets stored on this brick.


What is confusing is the dht values that are shown for the directories inside 
these brick directories. What do the dht values associated with the sub 
directories signify.

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/
trusted.glusterfs.dht=0x00017ffe

[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/e
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/e
trusted.glusterfs.dht=0x00017ffe


[root@glusterhackervm3 glus]# getfattr  -n trusted.glusterfs.dht -e hex 
/brick2/vol/d/f
getfattr: Removing leading '/' from absolute path names
# file: brick2/vol/d/f
trusted.glusterfs.dht=0x00017fff

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] rot-13 Translator query

[Ankireddypalle Reddy]

Xavi/Ravi,
 Thanks for the inputs. After disabling all the performance 
xlators on the client side and attaching to brick daemons before actually 
mounting the gluster volume I was able to get in to posix_readv function. 

Thanks and Regards,
Ram
-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, October 17, 2016 3:02 AM
To: Ankireddypalle Reddy; Ravishankar N; gluster-users@gluster.org
Subject: Re: [Gluster-users] rot-13 Translator query

Hi Ankireddypalle,

On 16/10/16 11:10, Ankireddypalle Reddy wrote:
> The encryption xlator is the last one before posix and it’s here that 
> the data is getting encrypted.  When the data is read back the 
> encrypted data is returned. Decryption is supposed to happen in read 
> callback which does not seem to be happening. The fact that encrypted 
> data is getting returned indicates that data in turn is getting 
> returned from the posix/underlying fs layer.  Is it possible that data 
> be returned by reading from the underlying fs by any translator other than 
> posix.

It could be because of quick-read translator. It caches some data from the 
beginning of files on lookups (even before an actual open and read is done on 
the file), so the first small read sent to the file could return cached data 
directly from what was obtained in the lookup fop without issuing a read fop.

You would need to handle that case in lookup also.

Anyway, to be sure you should do as Ravi has said and disable all performance 
xlators. In this case all reads should arrive as regular reads to your xlator.

Xavi

>
>
>
> Thanks and Regards,
>
> Ram
>
> *From:*Ravishankar N [mailto:ravishan...@redhat.com]
> *Sent:* Sunday, October 16, 2016 12:19 AM
> *To:* Ankireddypalle Reddy; gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] rot-13 Translator query
>
>
>
> On 10/15/2016 08:22 PM, Ankireddypalle Reddy wrote:
>
> Hi,
>
>   I am trying to follow the below document for developing a
> translator.
>
>
> 
> https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/t
> ranslator-development.md
>
>
>
>   I’ve created a replica volume and modified the vol file to
> include rot-13 translator. Below is the snippet from vol file.
>
>
>
> volume myvol-posix
>
> type storage/posix
>
> option volume-id b492191e-77a5-4fc3-9394-49218e36dae2
>
> option directory /brick1/repli
>
> end-volume
>
>
>
> volume *myvol-rot13*
>
> type encryption/rot-13
>
> subvolumes *myvol-posix*
>
> end-volume
>
>
>
> volume myvol-trash
>
> type features/trash
>
> option trash-internal-op off
>
> option brick-path /brick1/repli
>
> option trash-dir .trashcan
>
> subvolumes *myvol-rot13*
>
> end-volume
>
> …
>
>
>
> The writes are getting intercepted by the translator and the file is
> getting encrypted. But the reads don’t seem to be getting
> intercepted by the translator.  I tried setting break point in the
> posix_readv function and attach the brick daemons to gdb. But
> posix_readv does not seem to be getting called on the brick daemon
> and the read completes on the application side.
>
>
>
> Can someone please explain how the reads are getting serviced here
> without hitting the posix layer.
>
> It could be due to client side caching. I usually disable all 
> performance xlators (write-behind, read-head, io-cache, stat-prefetch, 
> quick-read, open-behind) when I want to remove caching effects while 
> debugging. drop-caches also helps.
>
> HTH,
> Ravi
>
>
>
>
> Thanks and Regards,
>
> Ram
>
> ***Legal Disclaimer***
>
> "This communication may contain confidential and privileged material 
> for the
>
> sole use of the intended recipient. Any unauthorized review, use or 
> distribution
>
> by others is strictly prohibited. If you have received the message by 
> mistake,
>
> please advise the sender by reply email and delete the message. Thank you."
>
> **
>
>
> ___
>
> Gluster-users mailing list
>
> Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sol

Re: [Gluster-users] rot-13 Translator query

[Ankireddypalle Reddy]

The encryption xlator is the last one before posix and it's here that the data 
is getting encrypted.  When the data is read back the encrypted data is 
returned. Decryption is supposed to happen in read callback which does not seem 
to be happening. The fact that encrypted data is getting returned indicates 
that data in turn is getting returned from the posix/underlying fs layer.  Is 
it possible that data be returned by reading from the underlying fs by any 
translator other than posix.

Thanks and Regards,
Ram
From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Sunday, October 16, 2016 12:19 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] rot-13 Translator query

On 10/15/2016 08:22 PM, Ankireddypalle Reddy wrote:
Hi,
  I am trying to follow the below document for developing a translator.
  
https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/translator-development.md

  I've created a replica volume and modified the vol file to include rot-13 
translator. Below is the snippet from vol file.

volume myvol-posix
type storage/posix
option volume-id b492191e-77a5-4fc3-9394-49218e36dae2
option directory /brick1/repli
end-volume

volume myvol-rot13
type encryption/rot-13
subvolumes myvol-posix
end-volume

volume myvol-trash
type features/trash
option trash-internal-op off
option brick-path /brick1/repli
option trash-dir .trashcan
subvolumes myvol-rot13
end-volume
...

The writes are getting intercepted by the translator and the file is getting 
encrypted. But the reads don't seem to be getting intercepted by the 
translator.  I tried setting break point in the posix_readv function and attach 
the brick daemons to gdb. But posix_readv does not seem to be getting called on 
the brick daemon and the read completes on the application side.

Can someone please explain how the reads are getting serviced here without 
hitting the posix layer.
It could be due to client side caching. I usually disable all performance 
xlators (write-behind, read-head, io-cache, stat-prefetch, quick-read, 
open-behind) when I want to remove caching effects while debugging. drop-caches 
also helps.

HTH,
Ravi



Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



___

Gluster-users mailing list

Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>

http://www.gluster.org/mailman/listinfo/gluster-users


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] rot-13 Translator query

[Ankireddypalle Reddy]

Hi,
  I am trying to follow the below document for developing a translator.
  
https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/translator-development.md

  I've created a replica volume and modified the vol file to include rot-13 
translator. Below is the snippet from vol file.

volume myvol-posix
type storage/posix
option volume-id b492191e-77a5-4fc3-9394-49218e36dae2
option directory /brick1/repli
end-volume

volume myvol-rot13
type encryption/rot-13
subvolumes myvol-posix
end-volume

volume myvol-trash
type features/trash
option trash-internal-op off
option brick-path /brick1/repli
option trash-dir .trashcan
subvolumes myvol-rot13
end-volume
...

The writes are getting intercepted by the translator and the file is getting 
encrypted. But the reads don't seem to be getting intercepted by the 
translator.  I tried setting break point in the posix_readv function and attach 
the brick daemons to gdb. But posix_readv does not seem to be getting called on 
the brick daemon and the read completes on the application side.

Can someone please explain how the reads are getting serviced here without 
hitting the posix layer.

Thanks and Regards,
Ram
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] CFP Gluster Developer Summit

[Ankireddypalle Reddy]

I propose to present the following topic.

Integration of GlusterFS in to Commvault data platform.

1) Brief introduction to Commvault data platform.
2) Why Commvault Choose to integrate GlusterFS in data platform.
3) Integration of GlusterFS in to Commvault data platform.
4) Orchestration by Commvault data platform to automatically create and extend 
a GlusterFS file system.
5) Health checks by Commvault data platform to monitor and report the file 
system health.
6) Features/Enhancements that Commvault data platform requires from GlusterFS.

Thanks and Regards,
Ram
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] libgfapi using unix domain sockets

[Ankireddypalle Reddy]

Poornima,
Thanks for checking this. We are using disperse volumes. Unix 
domain sockets will be used for communication between libgfapi and the brick 
daemons on the local server. The communication to brick daemons on the other 
nodes of the volume would  be through tcp/rdma. Is my assumption correct.

Thanks and Regards,
Ram

From: Poornima Gurusiddaiah [mailto:pguru...@redhat.com]
Sent: Wednesday, May 25, 2016 2:09 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi using unix domain sockets

Hi,

Whenever a new fd is created it is allocated from the mem-pool, if the mem pool 
is full it will be calloc'd. The current limit for fd-mem-pool is 1024, if 
there are more than 1024 fd's open, then the perf may be affected.
Also, the unix socket used while glfs_set_volfile_server() is only for Vol 
file, i.e. only for the connection btw Client and glusterd (management deamon). 
Hence, you may not see the IO performance increase, the patch 
http://review.gluster.org/#/c/12709/ introduces unix socket domain for IO path. 
This is what you may be interested in i guess.

Regards,
Poornima

____
From: "Ankireddypalle Reddy" mailto:are...@commvault.com>>
To: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Sent: Tuesday, May 24, 2016 9:16:31 PM
Subject: Re: [Gluster-users] libgfapi using unix domain sockets

Is there any suggested best practice for the number of glfs_fd_t  that can be 
associated with a glfs_t. Does having a single glfs_t in an application with 
large number of glfs_fd_t cause any resource contention issues.

Thanks and Regards,
Ram
From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Tuesday, May 24, 2016 11:34 AM
To: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] libgfapi using unix domain sockets

I figured it out.

Protocol: unix
Hostname: /var/run/glusterd.socket
Port: 0

Thanks and Regards,
Ram
From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Tuesday, May 24, 2016 10:20 AM
To: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: [Gluster-users] libgfapi using unix domain sockets

Hi,
I am trying to use libgfapi for connecting to a gluster  volume using 
unix domain sockets. I am not able to find the socket path that should be 
provided while making the “glfs_set_volfile_server” function call.

ps -eaf | grep gluster
root 15178 31450  0 09:52 pts/100:00:00 grep --color=auto gluster
root 26739 26291  0 May16 ?00:01:52 
/opt/commvault/Base/IndexingService -serviceName IndexingService_cleanup -cn 
glustervm6 cvshost:glustervm6*glustervm6 cvsport:58600:0 cvsmyplatform:2 
cvsremoteplatform:4 -vm Instance001
root 28335 1  0 May12 ?00:02:15 /usr/sbin/glusterd -p 
/var/run/glusterd.pid --log-level INFO
root 30047 1  0 May12 ?00:06:38 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk1-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk1-ws_brick.pid -S 
/var/run/gluster/9ed1d13b4265b95be4ed642578e7f28b.socket --brick-name 
/ws/disk1/ws_brick -l /var/log/glusterfs/bricks/ws-disk1-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49152 --xlator-option StoragePool-server.listen-port=49152
root 30066 1  0 May12 ?00:13:58 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk2-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk2-ws_brick.pid -S 
/var/run/gluster/be6fc96032a95d6bf00d41049ca0356a.socket --brick-name 
/ws/disk2/ws_brick -l /var/log/glusterfs/bricks/ws-disk2-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49153 --xlator-option StoragePool-server.listen-port=49153
root 30088 1  0 May12 ?00:00:21 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/glusterfs/nfs.log -S 
/var/run/gluster/93db4047a97542a6457b2178ce6512d7.socket
root 30093 1  0 May12 ?00:10:24 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/glustershd -p 
/var/lib/glusterd/glustershd/run/glustershd.pid -l 
/var/log/glusterfs/glustershd.log -S 
/var/run/gluster/3d435606821403370720761863000928.socket --xlator-option 
*replicate*.node-uuid=3ab81d79-9a99-4822-abb2-62e76a029240
root 30186 1  0 May12 ?00:00:31 /usr/sbin/glusterfs 
--volfile-server=glustervm6sds --volfile-id=/StoragePool /ws/glus

Thanks and Regards,
Ram





***Legal Disclaimer***

"This c

Re: [Gluster-users] libgfapi using unix domain sockets

[Ankireddypalle Reddy]

Is there any suggested best practice for the number of glfs_fd_t  that can be 
associated with a glfs_t. Does having a single glfs_t in an application with 
large number of glfs_fd_t cause any resource contention issues.

Thanks and Regards,
Ram
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Tuesday, May 24, 2016 11:34 AM
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi using unix domain sockets

I figured it out.

Protocol: unix
Hostname: /var/run/glusterd.socket
Port: 0

Thanks and Regards,
Ram
From: 
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Tuesday, May 24, 2016 10:20 AM
To: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: [Gluster-users] libgfapi using unix domain sockets

Hi,
I am trying to use libgfapi for connecting to a gluster  volume using 
unix domain sockets. I am not able to find the socket path that should be 
provided while making the "glfs_set_volfile_server" function call.

ps -eaf | grep gluster
root 15178 31450  0 09:52 pts/100:00:00 grep --color=auto gluster
root 26739 26291  0 May16 ?00:01:52 
/opt/commvault/Base/IndexingService -serviceName IndexingService_cleanup -cn 
glustervm6 cvshost:glustervm6*glustervm6 cvsport:58600:0 cvsmyplatform:2 
cvsremoteplatform:4 -vm Instance001
root 28335 1  0 May12 ?00:02:15 /usr/sbin/glusterd -p 
/var/run/glusterd.pid --log-level INFO
root 30047 1  0 May12 ?00:06:38 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk1-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk1-ws_brick.pid -S 
/var/run/gluster/9ed1d13b4265b95be4ed642578e7f28b.socket --brick-name 
/ws/disk1/ws_brick -l /var/log/glusterfs/bricks/ws-disk1-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49152 --xlator-option StoragePool-server.listen-port=49152
root 30066 1  0 May12 ?00:13:58 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk2-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk2-ws_brick.pid -S 
/var/run/gluster/be6fc96032a95d6bf00d41049ca0356a.socket --brick-name 
/ws/disk2/ws_brick -l /var/log/glusterfs/bricks/ws-disk2-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49153 --xlator-option StoragePool-server.listen-port=49153
root 30088 1  0 May12 ?00:00:21 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/glusterfs/nfs.log -S 
/var/run/gluster/93db4047a97542a6457b2178ce6512d7.socket
root 30093 1  0 May12 ?00:10:24 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/glustershd -p 
/var/lib/glusterd/glustershd/run/glustershd.pid -l 
/var/log/glusterfs/glustershd.log -S 
/var/run/gluster/3d435606821403370720761863000928.socket --xlator-option 
*replicate*.node-uuid=3ab81d79-9a99-4822-abb2-62e76a029240
root 30186 1  0 May12 ?00:00:31 /usr/sbin/glusterfs 
--volfile-server=glustervm6sds --volfile-id=/StoragePool /ws/glus

Thanks and Regards,
Ram





***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**





***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] libgfapi using unix domain sockets

[Ankireddypalle Reddy]

I figured it out.

Protocol: unix
Hostname: /var/run/glusterd.socket
Port: 0

Thanks and Regards,
Ram
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Tuesday, May 24, 2016 10:20 AM
To: gluster-users@gluster.org
Subject: [Gluster-users] libgfapi using unix domain sockets

Hi,
I am trying to use libgfapi for connecting to a gluster  volume using 
unix domain sockets. I am not able to find the socket path that should be 
provided while making the "glfs_set_volfile_server" function call.

ps -eaf | grep gluster
root 15178 31450  0 09:52 pts/100:00:00 grep --color=auto gluster
root 26739 26291  0 May16 ?00:01:52 
/opt/commvault/Base/IndexingService -serviceName IndexingService_cleanup -cn 
glustervm6 cvshost:glustervm6*glustervm6 cvsport:58600:0 cvsmyplatform:2 
cvsremoteplatform:4 -vm Instance001
root 28335 1  0 May12 ?00:02:15 /usr/sbin/glusterd -p 
/var/run/glusterd.pid --log-level INFO
root 30047 1  0 May12 ?00:06:38 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk1-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk1-ws_brick.pid -S 
/var/run/gluster/9ed1d13b4265b95be4ed642578e7f28b.socket --brick-name 
/ws/disk1/ws_brick -l /var/log/glusterfs/bricks/ws-disk1-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49152 --xlator-option StoragePool-server.listen-port=49152
root 30066 1  0 May12 ?00:13:58 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk2-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk2-ws_brick.pid -S 
/var/run/gluster/be6fc96032a95d6bf00d41049ca0356a.socket --brick-name 
/ws/disk2/ws_brick -l /var/log/glusterfs/bricks/ws-disk2-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49153 --xlator-option StoragePool-server.listen-port=49153
root 30088 1  0 May12 ?00:00:21 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/glusterfs/nfs.log -S 
/var/run/gluster/93db4047a97542a6457b2178ce6512d7.socket
root 30093 1  0 May12 ?00:10:24 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/glustershd -p 
/var/lib/glusterd/glustershd/run/glustershd.pid -l 
/var/log/glusterfs/glustershd.log -S 
/var/run/gluster/3d435606821403370720761863000928.socket --xlator-option 
*replicate*.node-uuid=3ab81d79-9a99-4822-abb2-62e76a029240
root 30186 1  0 May12 ?00:00:31 /usr/sbin/glusterfs 
--volfile-server=glustervm6sds --volfile-id=/StoragePool /ws/glus

Thanks and Regards,
Ram





***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] libgfapi using unix domain sockets

[Ankireddypalle Reddy]

Hi,
I am trying to use libgfapi for connecting to a gluster  volume using 
unix domain sockets. I am not able to find the socket path that should be 
provided while making the "glfs_set_volfile_server" function call.

ps -eaf | grep gluster
root 15178 31450  0 09:52 pts/100:00:00 grep --color=auto gluster
root 26739 26291  0 May16 ?00:01:52 
/opt/commvault/Base/IndexingService -serviceName IndexingService_cleanup -cn 
glustervm6 cvshost:glustervm6*glustervm6 cvsport:58600:0 cvsmyplatform:2 
cvsremoteplatform:4 -vm Instance001
root 28335 1  0 May12 ?00:02:15 /usr/sbin/glusterd -p 
/var/run/glusterd.pid --log-level INFO
root 30047 1  0 May12 ?00:06:38 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk1-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk1-ws_brick.pid -S 
/var/run/gluster/9ed1d13b4265b95be4ed642578e7f28b.socket --brick-name 
/ws/disk1/ws_brick -l /var/log/glusterfs/bricks/ws-disk1-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49152 --xlator-option StoragePool-server.listen-port=49152
root 30066 1  0 May12 ?00:13:58 /usr/sbin/glusterfsd -s 
glustervm6sds --volfile-id StoragePool.glustervm6sds.ws-disk2-ws_brick -p 
/var/lib/glusterd/vols/StoragePool/run/glustervm6sds-ws-disk2-ws_brick.pid -S 
/var/run/gluster/be6fc96032a95d6bf00d41049ca0356a.socket --brick-name 
/ws/disk2/ws_brick -l /var/log/glusterfs/bricks/ws-disk2-ws_brick.log 
--xlator-option *-posix.glusterd-uuid=3ab81d79-9a99-4822-abb2-62e76a029240 
--brick-port 49153 --xlator-option StoragePool-server.listen-port=49153
root 30088 1  0 May12 ?00:00:21 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/glusterfs/nfs.log -S 
/var/run/gluster/93db4047a97542a6457b2178ce6512d7.socket
root 30093 1  0 May12 ?00:10:24 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/glustershd -p 
/var/lib/glusterd/glustershd/run/glustershd.pid -l 
/var/log/glusterfs/glustershd.log -S 
/var/run/gluster/3d435606821403370720761863000928.socket --xlator-option 
*replicate*.node-uuid=3ab81d79-9a99-4822-abb2-62e76a029240
root 30186 1  0 May12 ?00:00:31 /usr/sbin/glusterfs 
--volfile-server=glustervm6sds --volfile-id=/StoragePool /ws/glus

Thanks and Regards,
Ram



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Possible error not being returned

[Ankireddypalle Reddy]

Xavier,
   Thanks for checking this. Will test this in 3.7.12.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Tuesday, May 24, 2016 2:24 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] Possible error not being returned

The error "XDR decoding failed" also happened to me recently. There's a bug 
report [1] and a patch for it [2] that should solve the problem.

Once communication failures are solved, random brick failures should disappear 
and ec should behave correctly. Anyway I'll try to see why ec is not able to 
detect these random failures.

The patch will be included in 3.7.12. Or you can apply it on top of
3.7.11 sources and compile if you want to try it.

Xavi

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1331502
[2] http://review.gluster.org/14292/

On 23/05/16 15:37, Ankireddypalle Reddy wrote:
> Xavier,
> Please find below logs from ws-glus.lg where /ws/glus is the 
> gluster mount point in question.
>
> 2016-05-18 21:13:00.477609] W [MSGID: 122002] 
> [ec-common.c:71:ec_heal_report] 0-SDSStoragePool-disperse-2: Heal 
> failed [Transport endpoint is not connected]
> [2016-05-18 21:13:00.556553] E [MSGID: 114030] 
> [client-rpc-fops.c:3022:client3_3_readv_cbk] 
> 0-SDSStoragePool-client-1: XDR decoding failed [Invalid argument]
> [2016-05-18 21:13:00.556588] W [MSGID: 114031] 
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 
> 0-SDSStoragePool-client-1: remote operation failed [Invalid argument]
> [2016-05-18 21:13:00.557754] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-0: 
> Operation failed on some subvolumes (up=7, mask=7, remaining=0, 
> good=5, bad=2)
> [2016-05-18 21:13:00.566626] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-0: 
> Operation failed on some subvolumes (up=7, mask=5, remaining=0, 
> good=5, bad=2)
> [2016-05-18 21:13:00.568694] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-0: 
> Operation failed on some subvolumes (up=7, mask=5, remaining=0, 
> good=5, bad=2)
> [2016-05-18 21:13:00.620563] W [MSGID: 122002] 
> [ec-common.c:71:ec_heal_report] 0-SDSStoragePool-disperse-0: Heal 
> failed [Transport endpoint is not connected]
> [2016-05-18 21:13:00.631860] W [MSGID: 122002] 
> [ec-common.c:71:ec_heal_report] 0-SDSStoragePool-disperse-0: Heal 
> failed [Transport endpoint is not connected]
> [2016-05-18 21:13:01.576095] E [MSGID: 114030] 
> [client-rpc-fops.c:3022:client3_3_readv_cbk] 
> 0-SDSStoragePool-client-13: XDR decoding failed [Invalid argument]
> [2016-05-18 21:13:01.576137] W [MSGID: 114031] 
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 
> 0-SDSStoragePool-client-13: remote operation failed [Invalid argument]
> [2016-05-18 21:13:01.576574] E [MSGID: 114030] 
> [client-rpc-fops.c:3022:client3_3_readv_cbk] 
> 0-SDSStoragePool-client-12: XDR decoding failed [Invalid argument]
> [2016-05-18 21:13:01.576598] W [MSGID: 114031] 
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 
> 0-SDSStoragePool-client-12: remote operation failed [Invalid argument]
> [2016-05-18 21:13:01.576810] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: 
> Operation failed on some subvolumes (up=7, mask=7, remaining=0, 
> good=6, bad=1)
> [2016-05-18 21:13:01.582926] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: 
> Operation failed on some subvolumes (up=7, mask=7, remaining=0, 
> good=5, bad=2)
> [2016-05-18 21:13:01.590063] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: 
> Operation failed on some subvolumes (up=7, mask=6, remaining=0, 
> good=6, bad=1)
> [2016-05-18 21:13:01.590798] E [MSGID: 122034] 
> [ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-4: 
> Insufficient available childs for this request (have 1, need 2)
> [2016-05-18 21:13:01.592769] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: 
> Operation failed on some subvolumes (up=7, mask=6, remaining=0, 
> good=6, bad=1) The message "W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: 
> Operation failed on some subvolumes (up=7, mask=6, remaining=0, 
> good=6, bad=1)" repeated 3 times between [2016-05-18 21:13:01.592769] 
> and [2016-05-18 21:13:01.598369]
> [2016-05-18 21:13:01.613843] I [MSGID: 122058] 
> [ec-heal.c:2338:ec_heal_do] 0-SDSStoragePool-disperse-4: 
> /Folder_05.11.2016_19.56/CV_MAGNETIC/V_22378/CHUNK_209062/SFILE_CONTAI
> NER_020: name heal successful on 7
> [2016-05-18 21:13:01.699580] W [MSGID: 122002] 
> [ec-common.c:71:ec_heal_report] 0-SDSStoragePool-d

Re: [Gluster-users] Possible error not being returned

[Ankireddypalle Reddy]

Xavier,
Please find below logs from ws-glus.lg where /ws/glus is the 
gluster mount point in question.

2016-05-18 21:13:00.477609] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-SDSStoragePool-disperse-2: Heal failed [Transport endpoint is not connected]
[2016-05-18 21:13:00.556553] E [MSGID: 114030] 
[client-rpc-fops.c:3022:client3_3_readv_cbk] 0-SDSStoragePool-client-1: XDR 
decoding failed [Invalid argument]
[2016-05-18 21:13:00.556588] W [MSGID: 114031] 
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-SDSStoragePool-client-1: remote 
operation failed [Invalid argument]
[2016-05-18 21:13:00.557754] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-0: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2016-05-18 21:13:00.566626] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-0: Operation failed 
on some subvolumes (up=7, mask=5, remaining=0, good=5, bad=2)
[2016-05-18 21:13:00.568694] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-0: Operation failed 
on some subvolumes (up=7, mask=5, remaining=0, good=5, bad=2)
[2016-05-18 21:13:00.620563] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-SDSStoragePool-disperse-0: Heal failed [Transport endpoint is not connected]
[2016-05-18 21:13:00.631860] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-SDSStoragePool-disperse-0: Heal failed [Transport endpoint is not connected]
[2016-05-18 21:13:01.576095] E [MSGID: 114030] 
[client-rpc-fops.c:3022:client3_3_readv_cbk] 0-SDSStoragePool-client-13: XDR 
decoding failed [Invalid argument]
[2016-05-18 21:13:01.576137] W [MSGID: 114031] 
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-SDSStoragePool-client-13: remote 
operation failed [Invalid argument]
[2016-05-18 21:13:01.576574] E [MSGID: 114030] 
[client-rpc-fops.c:3022:client3_3_readv_cbk] 0-SDSStoragePool-client-12: XDR 
decoding failed [Invalid argument]
[2016-05-18 21:13:01.576598] W [MSGID: 114031] 
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-SDSStoragePool-client-12: remote 
operation failed [Invalid argument]
[2016-05-18 21:13:01.576810] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1)
[2016-05-18 21:13:01.582926] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: Operation failed 
on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2016-05-18 21:13:01.590063] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: Operation failed 
on some subvolumes (up=7, mask=6, remaining=0, good=6, bad=1)
[2016-05-18 21:13:01.590798] E [MSGID: 122034] 
[ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-4: Insufficient 
available childs for this request (have 1, need 2)
[2016-05-18 21:13:01.592769] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-4: Operation failed 
on some subvolumes (up=7, mask=6, remaining=0, good=6, bad=1)
The message "W [MSGID: 122053] [ec-common.c:116:ec_check_status] 
0-SDSStoragePool-disperse-4: Operation failed on some subvolumes (up=7, mask=6, 
remaining=0, good=6, bad=1)" repeated 3 times between [2016-05-18 
21:13:01.592769] and [2016-05-18 21:13:01.598369]
[2016-05-18 21:13:01.613843] I [MSGID: 122058] [ec-heal.c:2338:ec_heal_do] 
0-SDSStoragePool-disperse-4: 
/Folder_05.11.2016_19.56/CV_MAGNETIC/V_22378/CHUNK_209062/SFILE_CONTAINER_020: 
name heal successful on 7
[2016-05-18 21:13:01.699580] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-SDSStoragePool-disperse-2: Heal failed [Transport endpoint is not connected]
[2016-05-18 21:13:01.833863] I [MSGID: 122058] [ec-heal.c:2338:ec_heal_do] 
0-SDSStoragePool-disperse-4: 
/Folder_05.11.2016_19.56/CV_MAGNETIC/V_22378/CHUNK_209062/SFILE_CONTAINER_020: 
name heal successful on 7
The message "I [MSGID: 122058] [ec-heal.c:2338:ec_heal_do] 
0-SDSStoragePool-disperse-4: 
/Folder_05.11.2016_19.56/CV_MAGNETIC/V_22378/CHUNK_209062/SFILE_CONTAINER_020: 
name heal successful on 7" repeated 5 times between [2016-05-18 
21:13:01.833863] and [2016-05-18 21:13:02.098833]  

The read from file 
/ws/glus/Folder_05.11.2016_19.56/CV_MAGNETIC/V_22378/CHUNK_209062/SFILE_CONTAINER_020
 succeeded but the CRC verification failed for the data read.

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, May 23, 2016 8:11 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] Possible error not being returned

In this case you should have more warnings/errors in your log files beside the 
ones related to EC. Can you post them ?

EC tries to keep track of good and bad bricks, however if multiple internal 
operations are failing (specially if related with communications), maybe some 
special case happens and it's unable to determi

Re: [Gluster-users] Possible error not being returned

[Ankireddypalle Reddy]

Xavier,
We are using disperse volume to save data being backed up by 
Commvault Simpana software. We are performing stress testing and have noticed 
that the issue happens when the 10G link is completely saturated consistently. 
The read would succeed but would return incorrect data. CRC checks fail on the 
returned data. The brick daemons are up and operational. To us it mostly 
appears to be communication issues. We have noticed lot of these issues when 1G 
NIC's were used. The error frequency went down drastically after moving the 
gluster traffic to 10G. 

Volume Name: SDSStoragePool
Type: Distributed-Disperse
Volume ID: c5ebb780-669f-4c31-9970-e12dae1f473c
Status: Started
Number of Bricks: 8 x (2 + 1) = 24
Transport-type: tcp
Bricks:
Brick1: cvltpbba1sds:/ws/disk1/ws_brick
Brick2: cvltpbba3sds:/ws/disk1/ws_brick
Brick3: cvltpbba4sds:/ws/disk1/ws_brick
Brick4: cvltpbba1sds:/ws/disk2/ws_brick
Brick5: cvltpbba3sds:/ws/disk2/ws_brick
Brick6: cvltpbba4sds:/ws/disk2/ws_brick
Brick7: cvltpbba1sds:/ws/disk3/ws_brick
Brick8: cvltpbba3sds:/ws/disk3/ws_brick
Brick9: cvltpbba4sds:/ws/disk3/ws_brick
Brick10: cvltpbba1sds:/ws/disk4/ws_brick
Brick11: cvltpbba3sds:/ws/disk4/ws_brick
Brick12: cvltpbba4sds:/ws/disk4/ws_brick
Brick13: cvltpbba1sds:/ws/disk5/ws_brick
Brick14: cvltpbba3sds:/ws/disk5/ws_brick
Brick15: cvltpbba4sds:/ws/disk5/ws_brick
Brick16: cvltpbba1sds:/ws/disk6/ws_brick
Brick17: cvltpbba3sds:/ws/disk6/ws_brick
Brick18: cvltpbba4sds:/ws/disk6/ws_brick
Brick19: cvltpbba1sds:/ws/disk7/ws_brick
Brick20: cvltpbba3sds:/ws/disk7/ws_brick
Brick21: cvltpbba4sds:/ws/disk7/ws_brick
Brick22: cvltpbba1sds:/ws/disk8/ws_brick
Brick23: cvltpbba3sds:/ws/disk8/ws_brick
Brick24: cvltpbba4sds:/ws/disk8/ws_brick
Options Reconfigured:
performance.readdir-ahead: on

Thanks and Regards,
Ram

-Original Message-
From: Xavier Hernandez [mailto:xhernan...@datalab.es] 
Sent: Monday, May 23, 2016 3:22 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] Possible error not being returned

It's possible that the operation that failed is an internal one made by 
disperse itself or any other translator, so this error is not reported to the 
application.

The read issued by the application will only fail if anything fails while 
processing the read itself. If everything goes well, the read will succeed and 
it should contain healthy data.

What configuration are you using ? (gluster volume info) What are you doing 
exactly ? (workload) Why is one brick down/damaged ? are you doing tests ? how 
are you doing them ?

Best regards,

Xavi

On 20/05/16 16:54, Ankireddypalle Reddy wrote:
> Hi,
>
> Did anyone get a chance to check this. We are intermittently 
> receiving corrupted data in read operations because of this.
>
>
>
> Thanks and Regards,
>
> Ram
>
>
>
> *From:*gluster-users-boun...@gluster.org
> [mailto:gluster-users-boun...@gluster.org] *On Behalf Of 
> *Ankireddypalle Reddy
> *Sent:* Thursday, May 19, 2016 3:59 PM
> *To:* gluster-users@gluster.org
> *Subject:* [Gluster-users] Possible error not being returned
>
>
>
> Hi,
>
>A disperse volume  was configured on  servers with limited 
> network bandwidth. Some of the read operations failed with error
>
>
>
> [2016-05-16 18:38:36.035559] E [MSGID: 122034] 
> [ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-2:
> Insufficient available childs for this request (have 1, need 2)
>
> [2016-05-16 18:38:36.035713] W [fuse-bridge.c:2213:fuse_readv_cbk]
> 0-glusterfs-fuse: 155121179: READ => -1 (Input/output error)
>
>
>
> For some read operations just the following error was logged but the 
> I/O did not fail.
>
> [2016-05-16 18:42:45.401570] E [MSGID: 122034] 
> [ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-3:
> Insufficient available childs for this request (have 1, need 2)
>
> [2016-05-16 18:42:45.402054] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-3: 
> Operation failed on some subvolumes (up=7, mask=6, remaining=0, 
> good=6, bad=1)
>
>
>
> We are receiving corrupted data in the read operation when the error 
> is logged but the read call did not return any error.
>
>
>
> Thanks and Regards,
>
> Ram
>
>
>
>
>
>
>
>
>
> ***Legal Disclaimer***
>
> "This communication may contain confidential and privileged material 
> for the
>
> sole use of the intended recipient. Any unauthorized review, use or 
> distribution
>
> by others is strictly prohibited. If you have received the message by 
> mistake,
>
> please advise the sender by reply email and delete the message. Thank you."
>
> ***

Re: [Gluster-users] Possible error not being returned

[Ankireddypalle Reddy]

Hi,
Did anyone get a chance to check this. We are intermittently receiving 
corrupted data in read operations because of this.

Thanks and Regards,
Ram

From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy
Sent: Thursday, May 19, 2016 3:59 PM
To: gluster-users@gluster.org
Subject: [Gluster-users] Possible error not being returned

Hi,
   A disperse volume  was configured on  servers with limited network 
bandwidth. Some of the read operations failed with error

[2016-05-16 18:38:36.035559] E [MSGID: 122034] 
[ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-2: Insufficient 
available childs for this request (have 1, need 2)
[2016-05-16 18:38:36.035713] W [fuse-bridge.c:2213:fuse_readv_cbk] 
0-glusterfs-fuse: 155121179: READ => -1 (Input/output error)

For some read operations just the following error was logged but the I/O did 
not fail.
[2016-05-16 18:42:45.401570] E [MSGID: 122034] 
[ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-3: Insufficient 
available childs for this request (have 1, need 2)
[2016-05-16 18:42:45.402054] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-3: Operation failed 
on some subvolumes (up=7, mask=6, remaining=0, good=6, bad=1)

We are receiving corrupted data in the read operation when the error is logged 
but the read call did not return any error.

Thanks and Regards,
Ram







***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Possible error not being returned

[Ankireddypalle Reddy]

Hi,
   A disperse volume  was configured on  servers with limited network 
bandwidth. Some of the read operations failed with error

[2016-05-16 18:38:36.035559] E [MSGID: 122034] 
[ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-2: Insufficient 
available childs for this request (have 1, need 2)
[2016-05-16 18:38:36.035713] W [fuse-bridge.c:2213:fuse_readv_cbk] 
0-glusterfs-fuse: 155121179: READ => -1 (Input/output error)

For some read operations just the following error was logged but the I/O did 
not fail.
[2016-05-16 18:42:45.401570] E [MSGID: 122034] 
[ec-common.c:461:ec_child_select] 0-SDSStoragePool-disperse-3: Insufficient 
available childs for this request (have 1, need 2)
[2016-05-16 18:42:45.402054] W [MSGID: 122053] 
[ec-common.c:116:ec_check_status] 0-SDSStoragePool-disperse-3: Operation failed 
on some subvolumes (up=7, mask=6, remaining=0, good=6, bad=1)

We are receiving corrupted data in the read operation when the error is logged 
but the read call did not return any error.

Thanks and Regards,
Ram




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] support for hole punching in glusterfs

[Ankireddypalle Reddy]

When is 3.8 expected to be GA?


Thanks and Regards,
Ram.

From: Pranith Kumar Karampuri 
Sent: Saturday, January 23, 2016 9:25 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Cc: Ashish Pandey; Xavier Hernandez
Subject: Re: [Gluster-users] support for hole punching in glusterfs



On 01/23/2016 09:23 AM, Ankireddypalle Reddy wrote:

Yes.

Fallocate is not implemented yet in EC xlator. We will complete this for 3.8. 
as well.

Pranith


Thanks and Regards,
Ram.

From: Pranith Kumar Karampuri <mailto:pkara...@redhat.com>
Sent: Saturday, January 23, 2016 9:20 AM
To: Ankireddypalle Reddy; 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] support for hole punching in glusterfs

Are you doing this for EC/disperse volume?

Pranith
On 01/22/2016 08:35 PM, Ankireddypalle Reddy wrote:
Hi,
Hole punching through fallocate succeeds for a ext4 mount path. But it 
fails for a glusterfs mount path. The volume is a disperse volume.

fallocate(fd,FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,0,sizeof(array)) 
fails with a return code of EOPNOTSUPP.

   The ext4 file system mount arguments are 
"rw,noexec,nosuid,nodev,user_xattr,discard".

Thanks and Regards,
Ram




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



___
Gluster-users mailing list
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users


***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] support for hole punching in glusterfs

[Ankireddypalle Reddy]

Yes.


Thanks and Regards,
Ram.

From: Pranith Kumar Karampuri 
Sent: Saturday, January 23, 2016 9:20 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] support for hole punching in glusterfs

Are you doing this for EC/disperse volume?

Pranith
On 01/22/2016 08:35 PM, Ankireddypalle Reddy wrote:
Hi,
Hole punching through fallocate succeeds for a ext4 mount path. But it 
fails for a glusterfs mount path. The volume is a disperse volume.

fallocate(fd,FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,0,sizeof(array)) 
fails with a return code of EOPNOTSUPP.

   The ext4 file system mount arguments are 
"rw,noexec,nosuid,nodev,user_xattr,discard".

Thanks and Regards,
Ram




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



___
Gluster-users mailing list
Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] support for hole punching in glusterfs

[Ankireddypalle Reddy]

I would be willing to assist with the development. 

Thanks and Regards,
Ram

-Original Message-
From: Niels de Vos [mailto:nde...@redhat.com] 
Sent: Friday, January 22, 2016 11:58 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] support for hole punching in glusterfs

On Fri, Jan 22, 2016 at 04:07:54PM +, Ankireddypalle Reddy wrote:
> Hi,
>   Thanks for checking this. If this is not supported through FUSE can 
> this be done through libgfapi.

Not yet. I was thinking of adding support for this after SEEK_HOLE/SEEK_DATA 
has been merged. Maybe we can include hole punching in gfapi for 3.8, where 
FUSE follows later. Would you be willing to assist with the development or 
testing of that?

Thanks,
Niels


> 
> Thanks and Regards,
> Ram
> 
> -Original Message-
> From: Niels de Vos [mailto:nde...@redhat.com]
> Sent: Friday, January 22, 2016 11:05 AM
> To: Ankireddypalle Reddy
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] support for hole punching in glusterfs
> 
> On Fri, Jan 22, 2016 at 03:05:16PM +, Ankireddypalle Reddy wrote:
> > Hi,
> > Hole punching through fallocate succeeds for a ext4 mount path. But 
> > it fails for a glusterfs mount path. The volume is a disperse volume.
> > 
> > 
> > fallocate(fd,FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,0,sizeof(array)) 
> > fails with a return code of EOPNOTSUPP.
> > 
> >The ext4 file system mount arguments are 
> > "rw,noexec,nosuid,nodev,user_xattr,discard".
> 
> This is currently not supported yet. I'm not sure if the FUSE kernel module 
> support punching holes yet, either. Please file a bug for this feature so 
> that we can figure out a plan to work on it.
> 
>   
> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=
> fuse
> 
> Note that support for SEEK_HOLE and SEEK_DATA is also not availeble yet.
> Some work has been done, but it is not ready for inclusion at this moment. 
> Hopefully it lands in the upcoming 3.8 release.
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1220173
> 
> Thanks,
> Niels
> 
> 
> 
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the message. Thank you."
> **



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] support for hole punching in glusterfs

[Ankireddypalle Reddy]

Hi,
  Thanks for checking this. If this is not supported through FUSE can 
this be done through libgfapi.

Thanks and Regards,
Ram

-Original Message-
From: Niels de Vos [mailto:nde...@redhat.com] 
Sent: Friday, January 22, 2016 11:05 AM
To: Ankireddypalle Reddy
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] support for hole punching in glusterfs

On Fri, Jan 22, 2016 at 03:05:16PM +, Ankireddypalle Reddy wrote:
> Hi,
> Hole punching through fallocate succeeds for a ext4 mount path. But 
> it fails for a glusterfs mount path. The volume is a disperse volume.
> 
> 
> fallocate(fd,FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,0,sizeof(array)) fails 
> with a return code of EOPNOTSUPP.
> 
>The ext4 file system mount arguments are 
> "rw,noexec,nosuid,nodev,user_xattr,discard".

This is currently not supported yet. I'm not sure if the FUSE kernel module 
support punching holes yet, either. Please file a bug for this feature so that 
we can figure out a plan to work on it.

  https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=fuse

Note that support for SEEK_HOLE and SEEK_DATA is also not availeble yet.
Some work has been done, but it is not ready for inclusion at this moment. 
Hopefully it lands in the upcoming 3.8 release.

  https://bugzilla.redhat.com/show_bug.cgi?id=1220173

Thanks,
Niels



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] support for hole punching in glusterfs

[Ankireddypalle Reddy]

Hi,
Hole punching through fallocate succeeds for a ext4 mount path. But it 
fails for a glusterfs mount path. The volume is a disperse volume.

fallocate(fd,FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,0,sizeof(array)) 
fails with a return code of EOPNOTSUPP.

   The ext4 file system mount arguments are 
"rw,noexec,nosuid,nodev,user_xattr,discard".

Thanks and Regards,
Ram






***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Notification for files that need healing

[Ankireddypalle Reddy]

Hi,
   Got it. Tried this on a 3.7.3 setup and now I am able to get the list of 
files that need healing.

Thanks and Regards,
Ram


From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, January 12, 2016 10:07 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] Notification for files that need healing


On 01/12/2016 08:29 PM, Ankireddypalle Reddy wrote:
Hi,
I tried the command. But looks like the command is not supported for 
disperse volumes.

[root@glusterlab1 ws_brick]# gluster volume heal StoragePool info
Volume StoragePool is not of type replicate
Volume heal failed
Oh are you not using 3.7.x release? I think my patch which implemented it, went 
in 3.7.1 or 3.7.2. Please use 3.7.x if you are running disperse.

Pranith


Thanks and Regards,
Ram

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, January 12, 2016 9:56 AM
To: Ankireddypalle Reddy; 
gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Subject: Re: [Gluster-users] Notification for files that need healing


On 01/12/2016 08:13 PM, Ankireddypalle Reddy wrote:
Hi,
   In a disperse volume how to determine programmatically the files that 
need healing. It's possible for a node to get   rebooted while I/O to the 
gluster volume is in progress. After the node comes up how to determine the 
list of files that need to be healed.
gluster volume heal  info gets us the files that need healing using 
the command. What exactly are you looking for? You want an api for this is it?

Pranith



Thanks and Regards,
Ram



***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**





___

Gluster-users mailing list

Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>

http://www.gluster.org/mailman/listinfo/gluster-users




***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Notification for files that need healing

[Ankireddypalle Reddy]

Hi,
I tried the command. But looks like the command is not supported for 
disperse volumes.

[root@glusterlab1 ws_brick]# gluster volume heal StoragePool info
Volume StoragePool is not of type replicate
Volume heal failed

Thanks and Regards,
Ram

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, January 12, 2016 9:56 AM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] Notification for files that need healing


On 01/12/2016 08:13 PM, Ankireddypalle Reddy wrote:
Hi,
   In a disperse volume how to determine programmatically the files that 
need healing. It's possible for a node to get   rebooted while I/O to the 
gluster volume is in progress. After the node comes up how to determine the 
list of files that need to be healed.
gluster volume heal  info gets us the files that need healing using 
the command. What exactly are you looking for? You want an api for this is it?

Pranith


Thanks and Regards,
Ram



***Legal Disclaimer***

"This communication may contain confidential and privileged material for the

sole use of the intended recipient. Any unauthorized review, use or distribution

by others is strictly prohibited. If you have received the message by mistake,

please advise the sender by reply email and delete the message. Thank you."

**




___

Gluster-users mailing list

Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>

http://www.gluster.org/mailman/listinfo/gluster-users




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Notification for files that need healing

[Ankireddypalle Reddy]

Hi,
   In a disperse volume how to determine programmatically the files that 
need healing. It's possible for a node to get   rebooted while I/O to the 
gluster volume is in progress. After the node comes up how to determine the 
list of files that need to be healed.

Thanks and Regards,
Ram



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

1) We are using  a gluster volume as a backend storage for storing backup data 
generated by Commvault Simpana software.  We tried creating one glfs_t instance 
for every glfd_t that was getting generated. But that did not work. After 
around 8 to 10 glfs_t objects were created glfs_init started failing. We are 
now creating 4 glfs_t objacts are making multiple glfd_t objects to use these 4 
objects. When I say multiple I mean hundreds/thousands  of glfd_t objects using 
the same glfs_t object. Would this create any performance/scale issues.
2) To circumvent the memcpy problem we tried using libgfapi Async I/O. But that 
was causing data loss. For 1000 write requests I got callback acknowledgment 
for all the 1000 write requests but only 44 requests made it to the file. Are 
there any known issues/limitations with libgfapi Async I/O.

Thanks and Regards,
Ram
-Original Message-
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com] 
Sent: Wednesday, December 16, 2015 7:15 AM
To: Poornima Gurusiddaiah; Ankireddypalle Reddy
Cc: Vijay Bellur; gluster-users@gluster.org; Shyam; Niels de Vos
Subject: Re: [Gluster-users] libgfapi access



On 12/16/2015 02:24 PM, Poornima Gurusiddaiah wrote:
> Answers inline
>
> - Original Message -
>> From: "Pranith Kumar Karampuri" 
>> To: "Ankireddypalle Reddy" , "Vijay Bellur" 
>> , gluster-users@gluster.org, "Shyam" 
>> , "Niels de Vos" 
>> Sent: Wednesday, December 16, 2015 1:14:35 PM
>> Subject: Re: [Gluster-users] libgfapi access
>>
>>
>>
>> On 12/16/2015 01:51 AM, Ankireddypalle Reddy wrote:
>>> Thanks for the explanation. Valgrind profiling shows multiple 
>>> memcpy's being invoked for each write through libgfapi. Is there a 
>>> way to avoid these memcpy's?. Also is there a limit on the number of 
>>> glfs_t* instances
> For every buffer passed by application to libgfapi, libgfapi does a 
> memcopy, for various reasons. As of yet there is no way to limit this, 
> there were some dicussions on adding capabilities for libgfapi to provide 
> buffer so that memcopies can be avoided.
>
>>> that can be allocated at a given point of time. I've encountered 
>>> cases where if more than 8 glfs_t* instances are being allocated 
>>> then glfs_init fails.
> Currently there is no way to limit the number of glfs_t instances for a 
> process.
> But it is quite easy to implement this in the application itself. What 
> application are using libgfapi for?
Do you think it is possible for you to share the c program which is leading to 
this problem? It would be easier to find the problem that way.

Pranith
>
>> Including maintainers of gfapi.
>>
>> Pranith
>>
>>>
>>>
>>> Thanks and Regards,
>>> Ram
>>>
>>> -----Original Message-
>>> From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
>>> Sent: Monday, December 14, 2015 11:13 PM
>>> To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
>>> Subject: Re: [Gluster-users] libgfapi access
>>>
>>>
>>>
>>> On 12/11/2015 08:58 PM, Ankireddypalle Reddy wrote:
>>>> Pranith,
>>>>Thanks for checking this. Though the time taken to run
>>>>was 18 seconds if you look at  the time consumed in
>>>>user land as well as kernel land for executing the
>>>>command then it is evident that fuse took almost half
>>>>the time as libgfapi. Also from the collected profiles
>>>>it is evident that the average latency for the write
>>>>command is less for fuse than for libgfapi. Are there
>>>>any recommendations for I/O through libgfapi for
>>>>disperse volumes. Is there any way to avoid the extra
>>>>memcpy's that are being made when performing I/O
>>>>through libgfapi.
>>> hi Ankireddy,
>>>Oh this is not a problem. If we use fuse, the system call 'write'
>>>from ./GlusterFuseTest will go through fuse-kernel, fuse kernel
>>>sends the write operation to glusterfs mount process which is a
>>>user process. Time taken to complete that call from then on is
>>>computed against the glusterfs mount process until it responds
>>>to the fuse-kernel, not against the ./GlusterFuseTest process.
>>>If we use gfapi, there is no system call ov

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Thanks for the explanation. Valgrind profiling shows multiple memcpy's being 
invoked for each write through libgfapi. Is there a way to avoid these 
memcpy's?. Also is there a limit on the number of glfs_t* instances that can be 
allocated at a given point of time. I've encountered cases where if more than 8 
glfs_t* instances are being allocated then glfs_init fails. 

Thanks and Regards,
Ram

-Original Message-
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com] 
Sent: Monday, December 14, 2015 11:13 PM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access



On 12/11/2015 08:58 PM, Ankireddypalle Reddy wrote:
> Pranith,
>  Thanks for checking this. Though the time taken to run was 
> 18 seconds if you look at  the time consumed in user land as well as kernel 
> land for executing the command then it is evident that fuse took almost half 
> the time as libgfapi. Also from the collected profiles it is evident that the 
> average latency for the write command is less for fuse than for libgfapi. Are 
> there any recommendations for I/O through libgfapi for disperse volumes. Is 
> there any way to avoid the extra memcpy's that are being made when performing 
> I/O through libgfapi.
hi Ankireddy,
 Oh this is not a problem. If we use fuse, the system call 'write' from 
./GlusterFuseTest will go through fuse-kernel, fuse kernel sends the write 
operation to glusterfs mount process which is a user process. Time taken to 
complete that call from then on is computed against the glusterfs mount process 
until it responds to the fuse-kernel, not against the ./GlusterFuseTest 
process. If we use gfapi, there is no system call over head, instead 
./GlusterFuseTest process directly makes calls with the bricks through gfapi 
library. So all the time that the process spends communicating with the bricks 
and getting the response is counted against ./GlusterFuseTest. That is the 
reason you see more 'user' time.

So again, There are quite a few workloads where gfapi has proven to give better 
response times than fuse mounts because we avoid the context switch costs of  
./GlusterFuseTest -> fuse-kernel -> glusterfs-mount -> fuse-kernel (for 
response)-> ./GlusterFuseTest (for response to 'write')

Hope that helps. Sorry for the delay in response, was in too many meetings 
yesterday.

Pranith
>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> Sent: Thursday, December 10, 2015 10:57 PM
> To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
> Subject: Re: [Gluster-users] libgfapi access
>
>
>
> On 12/10/2015 07:15 PM, Ankireddypalle Reddy wrote:
>> Hi,
>>Please let me know in case you need any more details. Even for only 
>> write operations fuse seems to outperform libgfapi. Is it because of 
>> disperse volumes?. Also I noticed a lot of data loss in case I use libgfapi 
>> asyn I/O for disperse volumes.
> Fuse and gfapi seem to take same amount of time to complete the run, i.e. 18 
> seconds. Could you let me know what you mean by fuse outperforming gfapi?
>
> Pranith
>> Thanks and Regards,
>> Ram
>>
>> -Original Message-
>> From: Ankireddypalle Reddy
>> Sent: Wednesday, December 09, 2015 5:01 PM
>> To: 'Pranith Kumar Karampuri'; Vijay Bellur; 
>> gluster-users@gluster.org
>> Subject: RE: [Gluster-users] libgfapi access
>>
>> Hi,
>>I upgraded my setup to gluster 3.7.3. I tested writes by 
>> performing writes through fuse and through libgfapi. Attached are the 
>> profiles generated from fuse and libgfapi. The test programs essentially 
>> writes 1 blocks each of 128K.
>>
>> [root@santest2 Base]# time ./GlusterFuseTest /ws/glus 131072 1 Mount 
>> path: /ws/glus Block size: 131072 Num of blocks: 1 Will perform write 
>> test on mount path : /ws/glus Succesfully created file 
>> /ws/glus/1449697583.glfs Successfully filled file /ws/glus/1449697583.glfs 
>> Write test succeeded Write test succeeded.
>>
>> real0m18.722s
>> user0m3.913s
>> sys 0m1.126s
>>
>> [root@santest2 Base]# time ./GlusterLibGFApiTest dispersevol santest2
>> 24007 131072 1 Host name: santest2
>> Volume: dispersevol
>> Port: 24007
>> Block size: 131072
>> Num of blocks: 1
>> Will perform write test on volume: dispersevol Successfully filled file 
>> 1449697651.glfs Write test succeeded Write test succeeded.
>>
>> real0m18.630s
>> user0m8.804s
>> sys 0m1.870s
>>
>> Thanks and Regards,
>> 

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Pranith,
Thanks for checking this. Though the time taken to run was 18 
seconds if you look at  the time consumed in user land as well as kernel land 
for executing the command then it is evident that fuse took almost half the 
time as libgfapi. Also from the collected profiles it is evident that the 
average latency for the write command is less for fuse than for libgfapi. Are 
there any recommendations for I/O through libgfapi for disperse volumes. Is 
there any way to avoid the extra memcpy's that are being made when performing 
I/O through libgfapi.

Thanks and Regards,
Ram

-Original Message-
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com] 
Sent: Thursday, December 10, 2015 10:57 PM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access



On 12/10/2015 07:15 PM, Ankireddypalle Reddy wrote:
> Hi,
>   Please let me know in case you need any more details. Even for only 
> write operations fuse seems to outperform libgfapi. Is it because of disperse 
> volumes?. Also I noticed a lot of data loss in case I use libgfapi asyn I/O 
> for disperse volumes.
Fuse and gfapi seem to take same amount of time to complete the run, i.e. 18 
seconds. Could you let me know what you mean by fuse outperforming gfapi?

Pranith
>
> Thanks and Regards,
> Ram
>
> -Original Message-
> From: Ankireddypalle Reddy
> Sent: Wednesday, December 09, 2015 5:01 PM
> To: 'Pranith Kumar Karampuri'; Vijay Bellur; gluster-users@gluster.org
> Subject: RE: [Gluster-users] libgfapi access
>
> Hi,
>   I upgraded my setup to gluster 3.7.3. I tested writes by performing 
> writes through fuse and through libgfapi. Attached are the profiles generated 
> from fuse and libgfapi. The test programs essentially writes 1 blocks 
> each of 128K.
>
> [root@santest2 Base]# time ./GlusterFuseTest /ws/glus 131072 1 Mount 
> path: /ws/glus Block size: 131072 Num of blocks: 1 Will perform write 
> test on mount path : /ws/glus Succesfully created file 
> /ws/glus/1449697583.glfs Successfully filled file /ws/glus/1449697583.glfs 
> Write test succeeded Write test succeeded.
>
> real0m18.722s
> user0m3.913s
> sys 0m1.126s
>
> [root@santest2 Base]# time ./GlusterLibGFApiTest dispersevol santest2 
> 24007 131072 1 Host name: santest2
> Volume: dispersevol
> Port: 24007
> Block size: 131072
> Num of blocks: 1
> Will perform write test on volume: dispersevol Successfully filled file 
> 1449697651.glfs Write test succeeded Write test succeeded.
>
> real0m18.630s
> user0m8.804s
> sys 0m1.870s
>
> Thanks and Regards,
> Ram
>
>    
>
> -Original Message-
> From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> Sent: Wednesday, December 09, 2015 1:39 AM
> To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
> Subject: Re: [Gluster-users] libgfapi access
>
>
>
> On 12/08/2015 08:28 PM, Ankireddypalle Reddy wrote:
>> Vijay,
>>We are trying to write data backed up by Commvault simpana to 
>> glusterfs volume.  The data being written is around 30 GB. Two kinds of 
>> write requests happen.
>>  1) 1MB requests
>>  2) Small write requests of size 128 bytes. In case of libgfapi access 
>> these are cached and a single 128KB write request is made where as in case 
>> of FUSE the 128 byte write request is handled to FUSE directly.
>>
>>  glusterfs 3.6.5 built on Aug 24 2015 10:02:43
>>
>>   Volume Name: dispersevol
>>  Type: Disperse
>>  Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0
>>  Status: Started
>>  Number of Bricks: 1 x (2 + 1) = 3
>>  Transport-type: tcp
>>  Bricks:
>>  Brick1: ssdtest:/mnt/ssdfs1/brick3
>>  Brick2: sanserver2:/data/brick3
>>  Brick3: santest2:/home/brick3
>>  Options Reconfigured:
>>  performance.cache-size: 512MB
>>  performance.write-behind-window-size: 8MB
>>  performance.io-thread-count: 32
>>  performance.flush-behind: on
> hi,
>Things look okay. May be we can find something using profile info.
>
> Could you post the results of the following operations:
> 1) gluster volume profile  start
> 2) Run the fuse workload
> 3) gluster volume profile  info > /path/to/file-1/to/send/us
> 4) Run the libgfapi workload
> 5)gluster volume profile  info > /path/to/file-2/to/send/us
>
> Send both these files to us to check what are the extra fops if any that are 
> sent over network which may be causing the delay.
>
> I see that you are using disperse volume. If you are going to use dis

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Hi,
 Please let me know in case you need any more details. Even for only write 
operations fuse seems to outperform libgfapi. Is it because of disperse 
volumes?. Also I noticed a lot of data loss in case I use libgfapi asyn I/O for 
disperse volumes. 

Thanks and Regards,
Ram

-Original Message-
From: Ankireddypalle Reddy 
Sent: Wednesday, December 09, 2015 5:01 PM
To: 'Pranith Kumar Karampuri'; Vijay Bellur; gluster-users@gluster.org
Subject: RE: [Gluster-users] libgfapi access

Hi,
 I upgraded my setup to gluster 3.7.3. I tested writes by performing 
writes through fuse and through libgfapi. Attached are the profiles generated 
from fuse and libgfapi. The test programs essentially writes 1 blocks each 
of 128K.

[root@santest2 Base]# time ./GlusterFuseTest /ws/glus 131072 1 Mount path: 
/ws/glus Block size: 131072 Num of blocks: 1 Will perform write test on 
mount path : /ws/glus Succesfully created file /ws/glus/1449697583.glfs 
Successfully filled file /ws/glus/1449697583.glfs Write test succeeded Write 
test succeeded.

real0m18.722s
user0m3.913s
sys 0m1.126s

[root@santest2 Base]# time ./GlusterLibGFApiTest dispersevol santest2 24007 
131072 1 Host name: santest2
Volume: dispersevol
Port: 24007
Block size: 131072
Num of blocks: 1
Will perform write test on volume: dispersevol Successfully filled file 
1449697651.glfs Write test succeeded Write test succeeded.

real0m18.630s
user0m8.804s
sys 0m1.870s

Thanks and Regards,
Ram

  

-Original Message-
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Wednesday, December 09, 2015 1:39 AM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access



On 12/08/2015 08:28 PM, Ankireddypalle Reddy wrote:
> Vijay,
>   We are trying to write data backed up by Commvault simpana to 
> glusterfs volume.  The data being written is around 30 GB. Two kinds of write 
> requests happen.
>   1) 1MB requests
>   2) Small write requests of size 128 bytes. In case of libgfapi access 
> these are cached and a single 128KB write request is made where as in case of 
> FUSE the 128 byte write request is handled to FUSE directly.
>
>   glusterfs 3.6.5 built on Aug 24 2015 10:02:43
>
>  Volume Name: dispersevol
>   Type: Disperse
>   Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0
>   Status: Started
>   Number of Bricks: 1 x (2 + 1) = 3
>   Transport-type: tcp
>   Bricks:
>   Brick1: ssdtest:/mnt/ssdfs1/brick3
>   Brick2: sanserver2:/data/brick3
>   Brick3: santest2:/home/brick3
>   Options Reconfigured:
>   performance.cache-size: 512MB
>   performance.write-behind-window-size: 8MB
>   performance.io-thread-count: 32
>   performance.flush-behind: on
hi,
  Things look okay. May be we can find something using profile info.

Could you post the results of the following operations:
1) gluster volume profile  start
2) Run the fuse workload
3) gluster volume profile  info > /path/to/file-1/to/send/us
4) Run the libgfapi workload
5)gluster volume profile  info > /path/to/file-2/to/send/us

Send both these files to us to check what are the extra fops if any that are 
sent over network which may be causing the delay.

I see that you are using disperse volume. If you are going to use disperse 
volume for production usecases, I suggest you use 3.7.x preferably 3.7.3. We 
fixed a bug in releases from 3.7.4 till 3.7.6 which will be released in 3.7.7.

Pranith
>
> Thanks and Regards,
> Ram
>
>
> -Original Message-
> From: Vijay Bellur [mailto:vbel...@redhat.com]
> Sent: Monday, December 07, 2015 6:13 PM
> To: Ankireddypalle Reddy; gluster-users@gluster.org
> Subject: Re: [Gluster-users] libgfapi access
>
> On 12/07/2015 10:29 AM, Ankireddypalle Reddy wrote:
>> Hi,
>>
>>  I am trying to use  libgfapi  interface to access gluster 
>> volume. What I noticed is that reads/writes to the gluster volume 
>> through libgfapi interface are slower than FUSE.  I was expecting the 
>> contrary. Are there any recommendations/settings suggested to be used 
>> while using libgfapi interface.
>>
> Can you please provide more details about your tests? Providing information 
> like I/O block size, file size, throughput would be helpful.
>
> Thanks,
> Vijay
>
>
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply ema

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Hi,
 I upgraded my setup to gluster 3.7.3. I tested writes by performing 
writes through fuse and through libgfapi. Attached are the profiles generated 
from fuse and libgfapi. The test programs essentially writes 1 blocks each 
of 128K.

[root@santest2 Base]# time ./GlusterFuseTest /ws/glus 131072 1
Mount path: /ws/glus
Block size: 131072
Num of blocks: 1
Will perform write test on mount path : /ws/glus
Succesfully created file /ws/glus/1449697583.glfs
Successfully filled file /ws/glus/1449697583.glfs
Write test succeeded
Write test succeeded.

real0m18.722s
user0m3.913s
sys 0m1.126s

[root@santest2 Base]# time ./GlusterLibGFApiTest dispersevol santest2 24007 
131072 1
Host name: santest2
Volume: dispersevol
Port: 24007
Block size: 131072
Num of blocks: 1
Will perform write test on volume: dispersevol
Successfully filled file 1449697651.glfs
Write test succeeded
Write test succeeded.

real0m18.630s
user0m8.804s
sys 0m1.870s

Thanks and Regards,
Ram

  

-Original Message-
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com] 
Sent: Wednesday, December 09, 2015 1:39 AM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access



On 12/08/2015 08:28 PM, Ankireddypalle Reddy wrote:
> Vijay,
>   We are trying to write data backed up by Commvault simpana to 
> glusterfs volume.  The data being written is around 30 GB. Two kinds of write 
> requests happen.
>   1) 1MB requests
>   2) Small write requests of size 128 bytes. In case of libgfapi access 
> these are cached and a single 128KB write request is made where as in case of 
> FUSE the 128 byte write request is handled to FUSE directly.
>
>   glusterfs 3.6.5 built on Aug 24 2015 10:02:43
>
>  Volume Name: dispersevol
>   Type: Disperse
>   Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0
>   Status: Started
>   Number of Bricks: 1 x (2 + 1) = 3
>   Transport-type: tcp
>   Bricks:
>   Brick1: ssdtest:/mnt/ssdfs1/brick3
>   Brick2: sanserver2:/data/brick3
>   Brick3: santest2:/home/brick3
>   Options Reconfigured:
>   performance.cache-size: 512MB
>   performance.write-behind-window-size: 8MB
>   performance.io-thread-count: 32
>   performance.flush-behind: on
hi,
  Things look okay. May be we can find something using profile info.

Could you post the results of the following operations:
1) gluster volume profile  start
2) Run the fuse workload
3) gluster volume profile  info > /path/to/file-1/to/send/us
4) Run the libgfapi workload
5)gluster volume profile  info > /path/to/file-2/to/send/us

Send both these files to us to check what are the extra fops if any that are 
sent over network which may be causing the delay.

I see that you are using disperse volume. If you are going to use disperse 
volume for production usecases, I suggest you use 3.7.x preferably 3.7.3. We 
fixed a bug in releases from 3.7.4 till 3.7.6 which will be released in 3.7.7.

Pranith
>
> Thanks and Regards,
> Ram
>
>
> -Original Message-
> From: Vijay Bellur [mailto:vbel...@redhat.com]
> Sent: Monday, December 07, 2015 6:13 PM
> To: Ankireddypalle Reddy; gluster-users@gluster.org
> Subject: Re: [Gluster-users] libgfapi access
>
> On 12/07/2015 10:29 AM, Ankireddypalle Reddy wrote:
>> Hi,
>>
>>  I am trying to use  libgfapi  interface to access gluster 
>> volume. What I noticed is that reads/writes to the gluster volume 
>> through libgfapi interface are slower than FUSE.  I was expecting the 
>> contrary. Are there any recommendations/settings suggested to be used 
>> while using libgfapi interface.
>>
> Can you please provide more details about your tests? Providing information 
> like I/O block size, file size, throughput would be helpful.
>
> Thanks,
> Vijay
>
>
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the message. Thank you."
> **
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users




***Legal Disclaimer***
"This communication may contain confidential and privileged mate

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Vijay,
 We are trying to write data backed up by Commvault simpana to 
glusterfs volume.  The data being written is around 30 GB. Two kinds of write 
requests happen.
1) 1MB requests
2) Small write requests of size 128 bytes. In case of libgfapi access 
these are cached and a single 128KB write request is made where as in case of 
FUSE the 128 byte write request is handled to FUSE directly.

glusterfs 3.6.5 built on Aug 24 2015 10:02:43

Volume Name: dispersevol
Type: Disperse
Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ssdtest:/mnt/ssdfs1/brick3
Brick2: sanserver2:/data/brick3
Brick3: santest2:/home/brick3
Options Reconfigured:
performance.cache-size: 512MB
performance.write-behind-window-size: 8MB
performance.io-thread-count: 32
performance.flush-behind: on

Thanks and Regards,
Ram


-Original Message-
From: Vijay Bellur [mailto:vbel...@redhat.com] 
Sent: Monday, December 07, 2015 6:13 PM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access

On 12/07/2015 10:29 AM, Ankireddypalle Reddy wrote:
> Hi,
>
> I am trying to use  libgfapi  interface to access gluster 
> volume. What I noticed is that reads/writes to the gluster volume 
> through libgfapi interface are slower than FUSE.  I was expecting the 
> contrary. Are there any recommendations/settings suggested to be used 
> while using libgfapi interface.
>

Can you please provide more details about your tests? Providing information 
like I/O block size, file size, throughput would be helpful.

Thanks,
Vijay





***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Hi,
   I started looking at the code and noticed that the application buffer is 
being copied to glusterfs internal buffers even for synchronous write 
operations. In my case the application tries to write 1MB of data in a single 
write operation. Looks to me that this 1MB is being copied in to Glusterfs 
internals buffers. Would be possible to use the application provided buffers 
directly in case of synchronous writes?
Thanks and Regards,
Ram

From: Ankireddypalle Reddy
Sent: Monday, December 07, 2015 10:29 AM
To: 'gluster-users@gluster.org'
Subject: libgfapi access

Hi,
   I am trying to use  libgfapi  interface to access gluster volume. What I 
noticed is that reads/writes to the gluster volume through libgfapi interface 
are slower than FUSE.  I was expecting the contrary. Are there any 
recommendations/settings suggested to be used while using libgfapi interface.

Thanks and Regards,
Ram




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] libgfapi access

[Ankireddypalle Reddy]

Hi,
   I am trying to use  libgfapi  interface to access gluster volume. What I 
noticed is that reads/writes to the gluster volume through libgfapi interface 
are slower than FUSE.  I was expecting the contrary. Are there any 
recommendations/settings suggested to be used while using libgfapi interface.

Thanks and Regards,
Ram




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users