Re: [Gluster-users] help, all latency comes from FINODELK
Today， when I ran statedump, node22 still crashed and I got the pending frames [2014-02-07 13:32:52.356386] E [marker-quota-helper.c:230:mq_dict_set_contribution] (-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/debug/io-stats.so(io_stats_lookup+0x13e) [0x7f81ca048a3e] (-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/features/marker.so(marker_lookup+0x300) [0x7f81ca25e170] (-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/features/marker.so(mq_req_xattr+0x3c) [0x7f81ca2676ec]))) 0-marker: invalid argument: loc->parent pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-02-07 13:32:52 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.3.0.5rhs_iqiyi_7 /lib64/libc.so.6[0x35ef232920] /usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/protocol/server.so(ltable_dump+0x54)[0x7f81c9e130b4] /usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/protocol/server.so(server_inode+0xc7)[0x7f81c9e13597] /usr/lib64/libglusterfs.so.0(gf_proc_dump_xlator_info+0x112)[0x30630443f2] /usr/lib64/libglusterfs.so.0(gf_proc_dump_info+0x4b2)[0x3063044bc2] /usr/sbin/glusterfsd(glusterfs_sigwaiter+0xd2)[0x405cc2] /lib64/libpthread.so.0[0x35ef607851] /lib64/libc.so.6(clone+0x6d)[0x35ef2e767d] On Thu, Jan 30, 2014 at 3:14 PM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > Could you give us the back trace > > Pranith > - Original Message - > > From: "Mingfan Lu" > > To: "Pranith Kumar Karampuri" > > Cc: "haiwei.xie-soulinfo" , " > Gluster-users@gluster.org List" > > Sent: Thursday, January 30, 2014 12:43:44 PM > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > > > When I triedto execute the statedump, the brick server of the BAD node > > crashed. > > > > > > > > On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri < > > pkara...@redhat.com> wrote: > > > > > Could you take statedump of bricks and get that information please. > > > You can use > > > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html > > > for taking statedumps. > > > > > > Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276that > > > you raised for? If yes could you attach the files to that bug so that > I get > > > the notifications immediately. > > > > > > Pranith > > > - Original Message - > > > > From: "Mingfan Lu" > > > > To: "haiwei.xie-soulinfo" > > > > Cc: "Gluster-users@gluster.org List" > > > > Sent: Tuesday, January 28, 2014 7:44:14 AM > > > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > > > > > > > About 200+ clients > > > > > > > > How to print print lock info in bricks, print lock requst info in > afr & > > > dht? > > > > thanks. > > > > > > > > > > > > > > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo < > > > > haiwei@soulinfo.com > wrote: > > > > > > > > > > > > > > > > hi, > > > > looks FINODELK deak lock. how many clients, and nfs or fuse? > > > > Maybe the best way is to print lock info in bricks, print lock requst > > > info in > > > > afr & dht. > > > > > > > > > all my clients hang when they creating dir > > > > > > > > > > > > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu < > mingfan...@gmail.com > > > > > > wrote: > > > > > > > > > > > I ran glusterfs volume profile my_volume info, I got some thing > > > likes: > > > > > > > > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 185 > > > > > > RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 11 > > > > > > RELEASEDIR > > > > > > 0.00 66.50 us 54.00 us 79.00 us 2 > > > > > > SETATTR > > > > > > 0.00 44.83 us 25.00 us 94.00 us 6 > > > > > > READDIR > > > > > > 0.00 57.55 us 28.00 us 95.00 us 11 > > > > > > OPENDIR > > > > > > 0.00 325.00 us 279.00 us 371.00 us 2 > > > > > > RENAME > > > > &g
Re: [Gluster-users] help, all latency comes from FINODELK
t; > From: "Mingfan Lu" > > To: "Pranith Kumar Karampuri" > > Cc: "haiwei.xie-soulinfo" , " > Gluster-users@gluster.org List" > > Sent: Thursday, January 30, 2014 12:43:44 PM > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > > > When I triedto execute the statedump, the brick server of the BAD node > > crashed. > > > > > > > > On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri < > > pkara...@redhat.com> wrote: > > > > > Could you take statedump of bricks and get that information please. > > > You can use > > > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html > > > for taking statedumps. > > > > > > Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276that > > > you raised for? If yes could you attach the files to that bug so that > I get > > > the notifications immediately. > > > > > > Pranith > > > - Original Message - > > > > From: "Mingfan Lu" > > > > To: "haiwei.xie-soulinfo" > > > > Cc: "Gluster-users@gluster.org List" > > > > Sent: Tuesday, January 28, 2014 7:44:14 AM > > > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > > > > > > > About 200+ clients > > > > > > > > How to print print lock info in bricks, print lock requst info in > afr & > > > dht? > > > > thanks. > > > > > > > > > > > > > > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo < > > > > haiwei@soulinfo.com > wrote: > > > > > > > > > > > > > > > > hi, > > > > looks FINODELK deak lock. how many clients, and nfs or fuse? > > > > Maybe the best way is to print lock info in bricks, print lock requst > > > info in > > > > afr & dht. > > > > > > > > > all my clients hang when they creating dir > > > > > > > > > > > > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu < > mingfan...@gmail.com > > > > > > wrote: > > > > > > > > > > > I ran glusterfs volume profile my_volume info, I got some thing > > > likes: > > > > > > > > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > > > > > > 0.00 0.00 us 0.00 us 0.00 us 185 > > > > > > RELEASE > > > > > > 0.00 0.00 us 0.00 us 0.00 us 11 > > > > > > RELEASEDIR > > > > > > 0.00 66.50 us 54.00 us 79.00 us 2 > > > > > > SETATTR > > > > > > 0.00 44.83 us 25.00 us 94.00 us 6 > > > > > > READDIR > > > > > > 0.00 57.55 us 28.00 us 95.00 us 11 > > > > > > OPENDIR > > > > > > 0.00 325.00 us 279.00 us 371.00 us 2 > > > > > > RENAME > > > > > > 0.00 147.80 us 84.00 us 206.00 us 5 > > > > > > LINK > > > > > > 0.00 389.50 us 55.00 us 724.00 us 2 > > > > > > READDIRP > > > > > > 0.00 164.25 us 69.00 us 287.00 us 8 > > > > > > UNLINK > > > > > > 0.00 37.46 us 18.00 us 87.00 us 50 > > > > > > FSTAT > > > > > > 0.00 70.32 us 29.00 us 210.00 us 37 > > > > > > GETXATTR > > > > > > 0.00 77.75 us 42.00 us 216.00 us 55 > > > > > > SETXATTR > > > > > > 0.00 36.39 us 11.00 us 147.00 us 119 > > > > > > FLUSH > > > > > > 0.00 51.22 us 24.00 us 139.00 us 275 > > > > > > OPEN > > > > > > 0.00 180.14 us 84.00 us 457.00 us 96 > > > > > > XATTROP > > > > > > 0.00 3847.20 us 231.00 us 18218.00 us 5 > > > > > > MKNOD > > > > > > 0.00 70.08 us 15.00 us 6539.00 us 342 > > > > > > ENTRYLK > > > > > > 0.00 10338.86 us 184.00 us 34813.00 us 7 > > > > > > CREATE > > > > > > 0.00 896.65 us 12.00 us 83103.00 us 235 > > > > > > INODELK > > > > > > 0.00 187.86 us 50.00 us 668.00 us 1526 > > > > > > WRITE > > > > > > 0.00 40.66 us 13.00 us 400.00 us 10400 > > > > > > STATFS > > > > > > 0.00 313.13 us 66.00 us 2142.00 us 2049 > > > > > > FXATTROP > > > > > > 0.00 2794.97 us 26.00 us 78048.00 us 295 > > > > > > READ > > > > > > 0.00 24469.82 us 206.00 us 176157.00 us 34 > > > > > > MKDIR > > > > > > 0.00 40.49 us 13.00 us 507.00 us 21420 > > > > > > STAT > > > > > > 0.00 190.90 us 40.00 us 330032.00 us 45820 > > > > > > LOOKUP > > > > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > > > > > > FINODELK > > > > > > > > > > > > what happend? > > > > > > > > > > > > > > > > > > > > > > > > > > ___ > > > > Gluster-users mailing list > > > > Gluster-users@gluster.org > > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] help, all latency comes from FINODELK
Could you give us the back trace Pranith - Original Message - > From: "Mingfan Lu" > To: "Pranith Kumar Karampuri" > Cc: "haiwei.xie-soulinfo" , > "Gluster-users@gluster.org List" > Sent: Thursday, January 30, 2014 12:43:44 PM > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > When I triedto execute the statedump, the brick server of the BAD node > crashed. > > > > On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri < > pkara...@redhat.com> wrote: > > > Could you take statedump of bricks and get that information please. > > You can use > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html > > for taking statedumps. > > > > Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276 that > > you raised for? If yes could you attach the files to that bug so that I get > > the notifications immediately. > > > > Pranith > > - Original Message - > > > From: "Mingfan Lu" > > > To: "haiwei.xie-soulinfo" > > > Cc: "Gluster-users@gluster.org List" > > > Sent: Tuesday, January 28, 2014 7:44:14 AM > > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > > > > > About 200+ clients > > > > > > How to print print lock info in bricks, print lock requst info in afr & > > dht? > > > thanks. > > > > > > > > > > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo < > > > haiwei@soulinfo.com > wrote: > > > > > > > > > > > > hi, > > > looks FINODELK deak lock. how many clients, and nfs or fuse? > > > Maybe the best way is to print lock info in bricks, print lock requst > > info in > > > afr & dht. > > > > > > > all my clients hang when they creating dir > > > > > > > > > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu < mingfan...@gmail.com > > > > > wrote: > > > > > > > > > I ran glusterfs volume profile my_volume info, I got some thing > > likes: > > > > > > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > > > > > 0.00 0.00 us 0.00 us 0.00 us 185 > > > > > RELEASE > > > > > 0.00 0.00 us 0.00 us 0.00 us 11 > > > > > RELEASEDIR > > > > > 0.00 66.50 us 54.00 us 79.00 us 2 > > > > > SETATTR > > > > > 0.00 44.83 us 25.00 us 94.00 us 6 > > > > > READDIR > > > > > 0.00 57.55 us 28.00 us 95.00 us 11 > > > > > OPENDIR > > > > > 0.00 325.00 us 279.00 us 371.00 us 2 > > > > > RENAME > > > > > 0.00 147.80 us 84.00 us 206.00 us 5 > > > > > LINK > > > > > 0.00 389.50 us 55.00 us 724.00 us 2 > > > > > READDIRP > > > > > 0.00 164.25 us 69.00 us 287.00 us 8 > > > > > UNLINK > > > > > 0.00 37.46 us 18.00 us 87.00 us 50 > > > > > FSTAT > > > > > 0.00 70.32 us 29.00 us 210.00 us 37 > > > > > GETXATTR > > > > > 0.00 77.75 us 42.00 us 216.00 us 55 > > > > > SETXATTR > > > > > 0.00 36.39 us 11.00 us 147.00 us 119 > > > > > FLUSH > > > > > 0.00 51.22 us 24.00 us 139.00 us 275 > > > > > OPEN > > > > > 0.00 180.14 us 84.00 us 457.00 us 96 > > > > > XATTROP > > > > > 0.00 3847.20 us 231.00 us 18218.00 us 5 > > > > > MKNOD > > > > > 0.00 70.08 us 15.00 us 6539.00 us 342 > > > > > ENTRYLK > > > > > 0.00 10338.86 us 184.00 us 34813.00 us 7 > > > > > CREATE > > > > > 0.00 896.65 us 12.00 us 83103.00 us 235 > > > > > INODELK > > > > > 0.00 187.86 us 50.00 us 668.00 us 1526 > > > > > WRITE > > > > > 0.00 40.66 us 13.00 us 400.00 us 10400 > > > > > STATFS > > > > > 0.00 313.13 us 66.00 us 2142.00 us 2049 > > > > > FXATTROP > > > > > 0.00 2794.97 us 26.00 us 78048.00 us 295 > > > > > READ > > > > > 0.00 24469.82 us 206.00 us 176157.00 us 34 > > > > > MKDIR > > > > > 0.00 40.49 us 13.00 us 507.00 us 21420 > > > > > STAT > > > > > 0.00 190.90 us 40.00 us 330032.00 us 45820 > > > > > LOOKUP > > > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > > > > > FINODELK > > > > > > > > > > what happend? > > > > > > > > > > > > > > > > > > > > ___ > > > Gluster-users mailing list > > > Gluster-users@gluster.org > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] help, all latency comes from FINODELK
When I triedto execute the statedump, the brick server of the BAD node crashed. On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > Could you take statedump of bricks and get that information please. > You can use > https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html > for taking statedumps. > > Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276 that > you raised for? If yes could you attach the files to that bug so that I get > the notifications immediately. > > Pranith > - Original Message - > > From: "Mingfan Lu" > > To: "haiwei.xie-soulinfo" > > Cc: "Gluster-users@gluster.org List" > > Sent: Tuesday, January 28, 2014 7:44:14 AM > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > > > About 200+ clients > > > > How to print print lock info in bricks, print lock requst info in afr & > dht? > > thanks. > > > > > > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo < > > haiwei@soulinfo.com > wrote: > > > > > > > > hi, > > looks FINODELK deak lock. how many clients, and nfs or fuse? > > Maybe the best way is to print lock info in bricks, print lock requst > info in > > afr & dht. > > > > > all my clients hang when they creating dir > > > > > > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu < mingfan...@gmail.com > > > > wrote: > > > > > > > I ran glusterfs volume profile my_volume info, I got some thing > likes: > > > > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > > > > 0.00 0.00 us 0.00 us 0.00 us 185 > > > > RELEASE > > > > 0.00 0.00 us 0.00 us 0.00 us 11 > > > > RELEASEDIR > > > > 0.00 66.50 us 54.00 us 79.00 us 2 > > > > SETATTR > > > > 0.00 44.83 us 25.00 us 94.00 us 6 > > > > READDIR > > > > 0.00 57.55 us 28.00 us 95.00 us 11 > > > > OPENDIR > > > > 0.00 325.00 us 279.00 us 371.00 us 2 > > > > RENAME > > > > 0.00 147.80 us 84.00 us 206.00 us 5 > > > > LINK > > > > 0.00 389.50 us 55.00 us 724.00 us 2 > > > > READDIRP > > > > 0.00 164.25 us 69.00 us 287.00 us 8 > > > > UNLINK > > > > 0.00 37.46 us 18.00 us 87.00 us 50 > > > > FSTAT > > > > 0.00 70.32 us 29.00 us 210.00 us 37 > > > > GETXATTR > > > > 0.00 77.75 us 42.00 us 216.00 us 55 > > > > SETXATTR > > > > 0.00 36.39 us 11.00 us 147.00 us 119 > > > > FLUSH > > > > 0.00 51.22 us 24.00 us 139.00 us 275 > > > > OPEN > > > > 0.00 180.14 us 84.00 us 457.00 us 96 > > > > XATTROP > > > > 0.00 3847.20 us 231.00 us 18218.00 us 5 > > > > MKNOD > > > > 0.00 70.08 us 15.00 us 6539.00 us 342 > > > > ENTRYLK > > > > 0.00 10338.86 us 184.00 us 34813.00 us 7 > > > > CREATE > > > > 0.00 896.65 us 12.00 us 83103.00 us 235 > > > > INODELK > > > > 0.00 187.86 us 50.00 us 668.00 us 1526 > > > > WRITE > > > > 0.00 40.66 us 13.00 us 400.00 us 10400 > > > > STATFS > > > > 0.00 313.13 us 66.00 us 2142.00 us 2049 > > > > FXATTROP > > > > 0.00 2794.97 us 26.00 us 78048.00 us 295 > > > > READ > > > > 0.00 24469.82 us 206.00 us 176157.00 us 34 > > > > MKDIR > > > > 0.00 40.49 us 13.00 us 507.00 us 21420 > > > > STAT > > > > 0.00 190.90 us 40.00 us 330032.00 us 45820 > > > > LOOKUP > > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > > > > FINODELK > > > > > > > > what happend? > > > > > > > > > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] help, all latency comes from FINODELK
Could you take statedump of bricks and get that information please. You can use https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html for taking statedumps. Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276 that you raised for? If yes could you attach the files to that bug so that I get the notifications immediately. Pranith - Original Message - > From: "Mingfan Lu" > To: "haiwei.xie-soulinfo" > Cc: "Gluster-users@gluster.org List" > Sent: Tuesday, January 28, 2014 7:44:14 AM > Subject: Re: [Gluster-users] help, all latency comes from FINODELK > > About 200+ clients > > How to print print lock info in bricks, print lock requst info in afr & dht? > thanks. > > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo < > haiwei@soulinfo.com > wrote: > > > > hi, > looks FINODELK deak lock. how many clients, and nfs or fuse? > Maybe the best way is to print lock info in bricks, print lock requst info in > afr & dht. > > > all my clients hang when they creating dir > > > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu < mingfan...@gmail.com > > > wrote: > > > > > I ran glusterfs volume profile my_volume info, I got some thing likes: > > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > > > 0.00 0.00 us 0.00 us 0.00 us 185 > > > RELEASE > > > 0.00 0.00 us 0.00 us 0.00 us 11 > > > RELEASEDIR > > > 0.00 66.50 us 54.00 us 79.00 us 2 > > > SETATTR > > > 0.00 44.83 us 25.00 us 94.00 us 6 > > > READDIR > > > 0.00 57.55 us 28.00 us 95.00 us 11 > > > OPENDIR > > > 0.00 325.00 us 279.00 us 371.00 us 2 > > > RENAME > > > 0.00 147.80 us 84.00 us 206.00 us 5 > > > LINK > > > 0.00 389.50 us 55.00 us 724.00 us 2 > > > READDIRP > > > 0.00 164.25 us 69.00 us 287.00 us 8 > > > UNLINK > > > 0.00 37.46 us 18.00 us 87.00 us 50 > > > FSTAT > > > 0.00 70.32 us 29.00 us 210.00 us 37 > > > GETXATTR > > > 0.00 77.75 us 42.00 us 216.00 us 55 > > > SETXATTR > > > 0.00 36.39 us 11.00 us 147.00 us 119 > > > FLUSH > > > 0.00 51.22 us 24.00 us 139.00 us 275 > > > OPEN > > > 0.00 180.14 us 84.00 us 457.00 us 96 > > > XATTROP > > > 0.00 3847.20 us 231.00 us 18218.00 us 5 > > > MKNOD > > > 0.00 70.08 us 15.00 us 6539.00 us 342 > > > ENTRYLK > > > 0.00 10338.86 us 184.00 us 34813.00 us 7 > > > CREATE > > > 0.00 896.65 us 12.00 us 83103.00 us 235 > > > INODELK > > > 0.00 187.86 us 50.00 us 668.00 us 1526 > > > WRITE > > > 0.00 40.66 us 13.00 us 400.00 us 10400 > > > STATFS > > > 0.00 313.13 us 66.00 us 2142.00 us 2049 > > > FXATTROP > > > 0.00 2794.97 us 26.00 us 78048.00 us 295 > > > READ > > > 0.00 24469.82 us 206.00 us 176157.00 us 34 > > > MKDIR > > > 0.00 40.49 us 13.00 us 507.00 us 21420 > > > STAT > > > 0.00 190.90 us 40.00 us 330032.00 us 45820 > > > LOOKUP > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > > > FINODELK > > > > > > what happend? > > > > > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] help, all latency comes from FINODELK
About 200+ clients How to print print lock info in bricks, print lock requst info in afr & dht? thanks. On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo < haiwei@soulinfo.com> wrote: > > hi, >looks FINODELK deak lock. how many clients, and nfs or fuse? > Maybe the best way is to print lock info in bricks, print lock requst info > in afr & dht. > > > all my clients hang when they creating dir > > > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu > wrote: > > > > > I ran glusterfs volume profile my_volume info, I got some thing likes: > > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 > FORGET > > > 0.00 0.00 us 0.00 us 0.00 us185 > > > RELEASE > > > 0.00 0.00 us 0.00 us 0.00 us 11 > > > RELEASEDIR > > > 0.00 66.50 us 54.00 us 79.00 us 2 > > > SETATTR > > > 0.00 44.83 us 25.00 us 94.00 us 6 > > > READDIR > > > 0.00 57.55 us 28.00 us 95.00 us 11 > > > OPENDIR > > > 0.00 325.00 us 279.00 us 371.00 us 2 > > > RENAME > > > 0.00 147.80 us 84.00 us 206.00 us 5 > > > LINK > > > 0.00 389.50 us 55.00 us 724.00 us 2 > > > READDIRP > > > 0.00 164.25 us 69.00 us 287.00 us 8 > > > UNLINK > > > 0.00 37.46 us 18.00 us 87.00 us 50 > > > FSTAT > > > 0.00 70.32 us 29.00 us 210.00 us 37 > > > GETXATTR > > > 0.00 77.75 us 42.00 us 216.00 us 55 > > > SETXATTR > > > 0.00 36.39 us 11.00 us 147.00 us119 > > > FLUSH > > > 0.00 51.22 us 24.00 us 139.00 us275 > > > OPEN > > > 0.00 180.14 us 84.00 us 457.00 us 96 > > > XATTROP > > > 0.003847.20 us 231.00 us 18218.00 us 5 > > > MKNOD > > > 0.00 70.08 us 15.00 us6539.00 us342 > > > ENTRYLK > > > 0.00 10338.86 us 184.00 us 34813.00 us 7 > > > CREATE > > > 0.00 896.65 us 12.00 us 83103.00 us235 > > > INODELK > > > 0.00 187.86 us 50.00 us 668.00 us 1526 > > > WRITE > > > 0.00 40.66 us 13.00 us 400.00 us 10400 > > > STATFS > > > 0.00 313.13 us 66.00 us2142.00 us 2049 > > > FXATTROP > > > 0.002794.97 us 26.00 us 78048.00 us295 > > > READ > > > 0.00 24469.82 us 206.00 us 176157.00 us 34 > > > MKDIR > > > 0.00 40.49 us 13.00 us 507.00 us 21420 > > > STAT > > > 0.00 190.90 us 40.00 us 330032.00 us 45820 > > > LOOKUP > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > > > FINODELK > > > > > > what happend? > > > > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] help, all latency comes from FINODELK
hi, looks FINODELK deak lock. how many clients, and nfs or fuse? Maybe the best way is to print lock info in bricks, print lock requst info in afr & dht. > all my clients hang when they creating dir > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu wrote: > > > I ran glusterfs volume profile my_volume info, I got some thing likes: > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > > 0.00 0.00 us 0.00 us 0.00 us185 > > RELEASE > > 0.00 0.00 us 0.00 us 0.00 us 11 > > RELEASEDIR > > 0.00 66.50 us 54.00 us 79.00 us 2 > > SETATTR > > 0.00 44.83 us 25.00 us 94.00 us 6 > > READDIR > > 0.00 57.55 us 28.00 us 95.00 us 11 > > OPENDIR > > 0.00 325.00 us 279.00 us 371.00 us 2 > > RENAME > > 0.00 147.80 us 84.00 us 206.00 us 5 > > LINK > > 0.00 389.50 us 55.00 us 724.00 us 2 > > READDIRP > > 0.00 164.25 us 69.00 us 287.00 us 8 > > UNLINK > > 0.00 37.46 us 18.00 us 87.00 us 50 > > FSTAT > > 0.00 70.32 us 29.00 us 210.00 us 37 > > GETXATTR > > 0.00 77.75 us 42.00 us 216.00 us 55 > > SETXATTR > > 0.00 36.39 us 11.00 us 147.00 us119 > > FLUSH > > 0.00 51.22 us 24.00 us 139.00 us275 > > OPEN > > 0.00 180.14 us 84.00 us 457.00 us 96 > > XATTROP > > 0.003847.20 us 231.00 us 18218.00 us 5 > > MKNOD > > 0.00 70.08 us 15.00 us6539.00 us342 > > ENTRYLK > > 0.00 10338.86 us 184.00 us 34813.00 us 7 > > CREATE > > 0.00 896.65 us 12.00 us 83103.00 us235 > > INODELK > > 0.00 187.86 us 50.00 us 668.00 us 1526 > > WRITE > > 0.00 40.66 us 13.00 us 400.00 us 10400 > > STATFS > > 0.00 313.13 us 66.00 us2142.00 us 2049 > > FXATTROP > > 0.002794.97 us 26.00 us 78048.00 us295 > > READ > > 0.00 24469.82 us 206.00 us 176157.00 us 34 > > MKDIR > > 0.00 40.49 us 13.00 us 507.00 us 21420 > > STAT > > 0.00 190.90 us 40.00 us 330032.00 us 45820 > > LOOKUP > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > > FINODELK > > > > what happend? > > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] help, all latency comes from FINODELK
all my clients hang when they creating dir On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu wrote: > I ran glusterfs volume profile my_volume info, I got some thing likes: > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET > 0.00 0.00 us 0.00 us 0.00 us185 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 11 > RELEASEDIR > 0.00 66.50 us 54.00 us 79.00 us 2 > SETATTR > 0.00 44.83 us 25.00 us 94.00 us 6 > READDIR > 0.00 57.55 us 28.00 us 95.00 us 11 > OPENDIR > 0.00 325.00 us 279.00 us 371.00 us 2 > RENAME > 0.00 147.80 us 84.00 us 206.00 us 5 > LINK > 0.00 389.50 us 55.00 us 724.00 us 2 > READDIRP > 0.00 164.25 us 69.00 us 287.00 us 8 > UNLINK > 0.00 37.46 us 18.00 us 87.00 us 50 > FSTAT > 0.00 70.32 us 29.00 us 210.00 us 37 > GETXATTR > 0.00 77.75 us 42.00 us 216.00 us 55 > SETXATTR > 0.00 36.39 us 11.00 us 147.00 us119 > FLUSH > 0.00 51.22 us 24.00 us 139.00 us275 > OPEN > 0.00 180.14 us 84.00 us 457.00 us 96 > XATTROP > 0.003847.20 us 231.00 us 18218.00 us 5 > MKNOD > 0.00 70.08 us 15.00 us6539.00 us342 > ENTRYLK > 0.00 10338.86 us 184.00 us 34813.00 us 7 > CREATE > 0.00 896.65 us 12.00 us 83103.00 us235 > INODELK > 0.00 187.86 us 50.00 us 668.00 us 1526 > WRITE > 0.00 40.66 us 13.00 us 400.00 us 10400 > STATFS > 0.00 313.13 us 66.00 us2142.00 us 2049 > FXATTROP > 0.002794.97 us 26.00 us 78048.00 us295 > READ > 0.00 24469.82 us 206.00 us 176157.00 us 34 > MKDIR > 0.00 40.49 us 13.00 us 507.00 us 21420 > STAT > 0.00 190.90 us 40.00 us 330032.00 us 45820 > LOOKUP > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994 > FINODELK > > what happend? > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
