Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

andey Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org) Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Hi Ram, On 20/01/17 08:02, Ankireddypalle Reddy wrote: Ashish, Thanks for looking in to the issue. In the given exampl

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

: Xavier Hernandez [mailto:xhernan...@datalab.es] Sent: Friday, January 20, 2017 2:41 AM To: Ankireddypalle Reddy; Ashish Pandey Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org) Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Hi Ram, On 20

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

g; Gluster Devel (gluster-de...@gluster.org) *Subject:* Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Ram, I don't understand what do you mean by saying "redundancy factor of 2 is met in a 3:1 disperse volume". You have given the xattr's of o

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

g] On Behalf Of Ankireddypalle Reddy Sent: Monday, January 16, 2017 9:41 AM To: Xavier Hernandez Cc: gluster-users@gluster.org; Gluster Devel (gluster-de...@gluster.org) Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Xavi, Thanks. I will start by track

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

failed [Invalid argument] >>>>> [2017-01-12 20:14:25.98] I [dict.c:166:key_value_cmp] >>>>> 0-glusterfsProd-disperse-2: 'trusted.ec.version' is different in two >>>>> dicts (16, 16) >>>>> [2017-01-12 20:14:25.555622] I [dict.

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Xavi Thanks and Regards, Ram -----Original Message----- From: Xavier Hernandez [mailto:xhernan...@datalab.es] Sent: Thursday, January 12, 2017 6:40 AM To: Ankireddypalle Reddy Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org Subject: Re: [Gluster-users] [Gluster-devel] Lot

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

rfsProd-disperse-6: 'trusted.ec.size' is different in two >>>>>>> dicts (8, 8) >>>>>>> [2017-01-11 14:19:45.037573] W [MSGID: 122056] >>>>>>> [ec-combine.c:873:ec_combine_check] 0-glusterfsProd-disperse-6: >>>>

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

be really useful. Xavi Thanks and Regards, Ram -Original Message----- From: Xavier Hernandez [mailto:xhernan...@datalab.es] Sent: Thursday, January 12, 2017 6:40 AM To: Ankireddypalle Reddy Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org Subject: Re: [Gluster-users] [G

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

0:53.728876] W [MSGID: 122002] >> [ec-common.c:71:ec_heal_report] 0-glusterfsProd-disperse-0: Heal >> failed [Invalid argument] > > This seems an attempt to heal a file, but I see a lot of differences between > both versions. The size on one brick is 13.238.272 bytes, but on t

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

12 01:19:18.257015] W [MSGID: 122053] >>>>> [ec-common.c:116:ec_check_status] 0-glusterfsProd-disperse-8: >>>>> Operation failed on some subvolumes (up=7, mask=7, remaining=0, >>>>> good=3, bad=4) >>>>> [2017-01-12 01:19:18.257018] W [MSG

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

fy the cause. Again, the TRACE log will be really useful. Xavi Thanks and Regards, Ram -Original Message- From: Xavier Hernandez [mailto:xhernan...@datalab.es] Sent: Thursday, January 12, 2017 6:40 AM To: Ankireddypalle Reddy Cc: Gluster Devel (gluster-de...@gluster.org); gluster-user

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

6:40 AM To: Ankireddypalle Reddy Cc: Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org Subject: Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Hi Ram, On 12/01/17 11:49, Ankireddypalle Reddy wrote: > Xavi, > As I mentioned before the e

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

ddypalle Reddy Sent: Wednesday, January 11, 2017 9:29 AM To: Ankireddypalle Reddy; Xavier Hernandez; Gluster Devel (gluster-de...@gluster.org); gluster-users@gluster.org Subject: RE: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Xavi, I built a debug binary to

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

rod-disperse-4: Operation >> failed on some subvolumes (up=7, mask=7, remaining=0, good=6, bad=1) >> [2017-01-12 01:19:21.209753] W [MSGID: 122002] >> [ec-common.c:71:ec_heal_report] 0-glusterfsProd-disperse-4: Heal failed >> [Invalid argument] >> >> Thanks an

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

gluster-de...@gluster.org); gluster-users@gluster.org Subject: RE: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Xavi, I built a debug binary to log more information. This is what is getting logged. Looks like it is the attribute trusted.ec.size which is diffe

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

luster.org); gluster-users@gluster.org Subject: RE: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume Xavi, I built a debug binary to log more information. This is what is getting logged. Looks like it is the attribute trusted.ec.size which is different among the brick

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

23 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: glusterfsProd-disperse-9, Err: -1 , Start: 1431655764 , Stop: 1789569704 , Hash: 1 ], -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Ankireddypalle Reddy Sent: Tuesday, Ja

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Xavi, In this case it's the file creation which failed. So I provided the xattrs of the parent. Thanks and Regards, Ram -Original Message- From: Xavier Hernandez [mailto:xhernan...@datalab.es] Sent: Tuesday, January 10, 2017 9:10 AM To: Ankireddypalle Reddy; Gluster Devel (g

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram, On 10/01/17 14:42, Ankireddypalle Reddy wrote: Attachments (2): 1 ec.txt

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Attachments (2): 1 ec.txt

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram, the error is caused by an extended attribute that does not match on all 3 bricks of the disperse set. Most probable value is trusted.ec.version, but could be others. At first sight, I don't see any change from 3.7.8 that could have caused this. I'll check again. What kind of operat

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Xavi, Thanks. If you could please explain what to look for in the extended attributes then I will check and let you know if I find anything suspicious. Also we noticed that some of these operations would succeed if retried. Do you know of any communicated related errors that are being

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram, On 10/01/17 13:14, Ankireddypalle Reddy wrote: Attachment (1): 1 ecxattrs.txt

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Attachment (1): 1 ecxattrs.txt

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram, can you execute the following command on all bricks on a file that is giving EIO ? getfattr -m. -e hex -d Xavi On 10/01/17 12:41, Ankireddypalle Reddy wrote: Xavi, We have been running 3.7.8 on these servers. We upgraded to 3.7.18 yesterday. We upgraded all the servers

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Hi Ram, how did you upgrade gluster ? from which version ? Did you upgrade one server at a time and waited until self-heal finished before upgrading the next server ? Xavi On 10/01/17 11:39, Ankireddypalle Reddy wrote: Hi, We upgraded to GlusterFS 3.7.18 yesterday. We see lot of fai

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

Xavi, We have been running 3.7.8 on these servers. We upgraded to 3.7.18 yesterday. We upgraded all the servers at a time. The volume was brought down during upgrade. Thanks and Regards, Ram -Original Message- From: Xavier Hernandez [mailto:xhernan...@datalab.es] Sent: Tu