I think this is the statesump of only one brick. We would required statedump from all the bricks and client process in case of fuse or nfs process if it is mounted through nfs.
---- Ashish ----- Original Message ----- From: "Serkan Çoban" <cobanser...@gmail.com> To: "Ashish Pandey" <aspan...@redhat.com> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel" <gluster-devel@gluster.org> Sent: Friday, April 15, 2016 2:11:57 PM Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size Sorry for typo, brick state dump file. On Fri, Apr 15, 2016 at 11:41 AM, Serkan Çoban <cobanser...@gmail.com> wrote: > Hi I reproduce the problem, brick log file is in below link: > https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0 > > > > On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspan...@redhat.com> wrote: >> Hi Serkan, >> >> Could you also provide us the statedump of all the brick processes and >> clients? >> >> Commands to generate statedumps for brick processes/nfs server/quotad >> >> For bricks: gluster volume statedump <volname> >> >> For nfs server: gluster volume statedump <volname> nfs >> >> >> We can find the directory where statedump files are created using 'gluster >> --print-statedumpdir' >> Also, the mount logs would help us to debug the issue. >> >> Ashish >> >> ________________________________ >> From: "Serkan Çoban" <cobanser...@gmail.com> >> To: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel" >> <gluster-devel@gluster.org> >> Sent: Thursday, April 14, 2016 6:27:10 PM >> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size >> >> >> Here is the related brick log: >> >> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E >> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks: Matching lock not >> found for unlock 0-9223372036854775807, by 94d29e885e7f0000 on >> 0x7f037413b990 >> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E >> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk] >> 0-v0-server: 712984: INODELK >> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605 >> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid >> argument] >> >> On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanser...@gmail.com> wrote: >>> Hi, >>> >>> During read/write tests to a 78x(16+4) distributed disperse volume >>> from 50 clients, One clients hangs on read/write with the following >>> logs: >>> >>> [2016-04-14 11:11:04.728580] W [MSGID: 122056] >>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata >>> in answers of 'LOOKUP' >>> [2016-04-14 11:11:04.728624] W [MSGID: 122053] >>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on >>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF, >>> bad=20000) >>> [2016-04-14 11:11:04.736689] I [MSGID: 122058] >>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2: >>> name heal successful on FFFFF >>> [2016-04-14 11:29:26.718036] W [MSGID: 122056] >>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata >>> in answers of 'LOOKUP' >>> [2016-04-14 11:29:26.718121] W [MSGID: 122053] >>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on >>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF, >>> bad=10000) >>> [2016-04-14 11:29:42.501760] I [MSGID: 122058] >>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2: >>> name heal successful on FFFFF >>> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat] >>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91) >>> [0x7f5ec9f942b1] >>> >>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57) >>> >>> [0x7f5ec9f94497] >>> >>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4) >>> >>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop, >>> fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size) >>> [2016-04-14 11:31:25.722372] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722411] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722450] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722477] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722503] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722577] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722605] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722742] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722794] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722818] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722840] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722883] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722906] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722958] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.722983] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.723037] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.723045] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.725044] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.741338] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.746602] E [MSGID: 114031] >>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57: remote >>> operation failed [Invalid argument] >>> [2016-04-14 11:31:25.746629] W [MSGID: 122015] >>> [ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode unlocking >>> failed (FSTAT) [Invalid argument] >>> [2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze] >>> >>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae) >>> >>> [0x7f5ec9fa009e] >>> >>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40) >>> [0x7f5ec9f97fc0] >>> >>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91) >>> >>> [0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs == lock->inserted >>> [2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze] >>> >>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae) >>> >>> [0x7f5ec9fa009e] >>> >>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40) >>> [0x7f5ec9f97fc0] >>> >>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2) >>> >>> [0x7f5ec9f97632] ) 0-: Assertion failed: list_empty(&lock->waiting) && >>> list_empty(&lock->owners) >>> [2016-04-14 11:40:54.139418] W [MSGID: 122056] >>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching xdata >>> in answers of 'LOOKUP' >>> [2016-04-14 11:40:54.139554] W [MSGID: 122053] >>> [ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed on >>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF, >>> bad=D0000) >>> [2016-04-14 11:40:54.143400] I [MSGID: 122058] >>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2: >>> name heal successful on FFFFF >>> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] >>> 0-v0-disperse-0: Mismatching xdata in answers of 'LOOKUP'" repeated 2 >>> times between [2016-04-14 11:40:54.139418] and [2016-04-14 >>> 11:40:54.139547] >>> [2016-04-14 11:42:31.471317] W [MSGID: 122056] >>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching >>> xdata in answers of 'LOOKUP' >>> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] >>> 0-v0-disperse-45: Mismatching xdata in answers of 'LOOKUP'" repeated 8 >>> times between [2016-04-14 11:42:31.471317] and [2016-04-14 >>> 11:42:31.471557] >>> >>> Gluster version is 3.7.10 >>> Any thoughts? >>> >>> Serkan >> _______________________________________________ >> Gluster-users mailing list >> gluster-us...@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ Gluster-users mailing list gluster-us...@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
