Re: [Gluster-users] Very slow ls - WARNING
On Tue, 2014-02-25 at 14:40 +, Justin Clift wrote: > On 23/02/2014, at 4:11 AM, Franco Broi wrote: > > All the client filesystems core-dumped. Lost a lot of production time. > > Ugh, that sounds remarkable bad. :( > > Out of curiosity, do you still have any of those core files around? Unfortunately no, I'm kicking myself now for not saving one but at the time I just wanted to get the filesystem back up. There's a little more information in the bug report https://bugzilla.redhat.com/show_bug.cgi?id=1067256 I think the layout information on the directories is somehow messed up but I'm a little wary of running a fix-layout again, I ran one after I doubled the capacity some weeks back and the mismatch messages are for directories created since then. Does fix-layout check and replace the xattrs for all directories? > > It _might_ be possible for someone to analyse them to figure out where > the bug is. > > Regards and best wishes, > > Justin Clift > > -- > Open Source and Standards @ Red Hat > > twitter.com/realjustinclift > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Very slow ls - WARNING
On 23/02/2014, at 4:11 AM, Franco Broi wrote: > All the client filesystems core-dumped. Lost a lot of production time. Ugh, that sounds remarkable bad. :( Out of curiosity, do you still have any of those core files around? It _might_ be possible for someone to analyse them to figure out where the bug is. Regards and best wishes, Justin Clift -- Open Source and Standards @ Red Hat twitter.com/realjustinclift ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Very slow ls - WARNING
FYI, another data point that echos Franco's experience. I turned this option (cluster.readdir-optimize) on after reading the thread post and in fact the 'ls' perf seem to increase quite a bit, but after a few days, this morning all 85 of our compute nodes reported no files on the mount point which was .. disconcerting to a number of users. The filesystem was still mounted and the data was intact, but 'ls' reported nothing, which makes it somewhat less than useful. After turning off that option and remounting, all the clients see their files again, albeit more slowly again. The config is gluster 3.4.2 on amd64/SL6.4 and is now $ gluster volume info gl Volume Name: gl Type: Distribute Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332 Status: Started Number of Bricks: 8 Transport-type: tcp,rdma Bricks: Brick1: bs2:/raid1 Brick2: bs2:/raid2 Brick3: bs3:/raid1 Brick4: bs3:/raid2 Brick5: bs4:/raid1 Brick6: bs4:/raid2 Brick7: bs1:/raid1 Brick8: bs1:/raid2 Options Reconfigured: cluster.readdir-optimize: off performance.write-behind-window-size: 1MB performance.flush-behind: on performance.cache-size: 268435456 nfs.disable: on performance.io-cache: on performance.quick-read: on performance.io-thread-count: 64 auth.allow: 10.2.*.*,10.1.*.* hjm On Sunday, February 23, 2014 04:11:28 AM Franco Broi wrote: > All the client filesystems core-dumped. Lost a lot of production time. > > I've disabled the cluster.readdir-optimize option and remounted all the > filesystems. > From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] > on behalf of Franco Broi [franco.b...@iongeo.com] Sent: Friday, February > 21, 2014 10:57 PM > To: Vijay Bellur > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Very slow ls > > Amazingly setting cluster.readdir-optimize has fixed the problem, ls is > still slow but there's no long pause on the last readdir call. > > What does this option do and why isn't it enabled by default? > ___ > From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] > on behalf of Franco Broi [franco.b...@iongeo.com] Sent: Friday, February > 21, 2014 7:25 PM > To: Vijay Bellur > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Very slow ls > > On 21 Feb 2014 22:03, Vijay Bellur wrote: > > On 02/18/2014 12:42 AM, Franco Broi wrote: > > > On 18 Feb 2014 00:13, Vijay Bellur wrote: > > > > On 02/17/2014 07:00 AM, Franco Broi wrote: > > > > > I mounted the filesystem with trace logging turned on and can see > > > > > that > > > > > after the last successful READDIRP there is a lot of other > > > > > connections > > > > > being made the clients repeatedly which takes minutes to complete. > > > > > > > > I did not observe anything specific which points to clients > > > > repeatedly > > > > reconnecting. Can you point to the appropriate line numbers for this? > > > > > > > > Can you also please describe the directory structure being referred > > > > here? > > > > > > I was tailing the log file while the readdir script was running and > > > could see respective READDIRP calls for each readdir, after the last > > > call all the rest of the stuff in the log file was returning nothing but > > > took minutes to complete. This particular example was a directory > > > containing a number of directories, one for each of the READDIRP calls > > > in the log file. > > > > One possible tuning that can possibly help: > > > > volume set cluster.readdir-optimize on > > > > Let us know if there is any improvement after enabling this option. > > I'll give it a go but I think this is a bug and not a performance issue. > I've filed a bug report on bugzilla. > > Thanks, > > Vijay > > > > > This email and any files transmitted with it are confidential and are > intended solely for the use of the individual or entity to whom they are > addressed. If you are not the original recipient or the person responsible > for delivering the email to the intended recipient, be advised that you > have received this email in error, and that any use, dissemination, > forwarding, printing, or copying of this email is strictly prohibited. If > you received this email in error, please immediately notify the sender and > delete the original. > > > > > This email and any files transmitted with it are confidential and are > intended solely for the use of the individual or entity to whom they are > addressed. If you are not the original recipient or the person responsible > for delivering the email to the intended recipient, be advised that you > have received this email in error, and that any use, dissemination, > forwarding, printing, or copying of this email is strictly prohibited. If > you received this email in error, please immediately notify the sender and > delete the original. > > ___
Re: [Gluster-users] Very slow ls - WARNING
All the client filesystems core-dumped. Lost a lot of production time. I've disabled the cluster.readdir-optimize option and remounted all the filesystems. From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of Franco Broi [franco.b...@iongeo.com] Sent: Friday, February 21, 2014 10:57 PM To: Vijay Bellur Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] Very slow ls Amazingly setting cluster.readdir-optimize has fixed the problem, ls is still slow but there's no long pause on the last readdir call. What does this option do and why isn't it enabled by default? ___ From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of Franco Broi [franco.b...@iongeo.com] Sent: Friday, February 21, 2014 7:25 PM To: Vijay Bellur Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] Very slow ls On 21 Feb 2014 22:03, Vijay Bellur wrote: > > On 02/18/2014 12:42 AM, Franco Broi wrote: > > > > On 18 Feb 2014 00:13, Vijay Bellur wrote: > > > > > > On 02/17/2014 07:00 AM, Franco Broi wrote: > > > > > > > > I mounted the filesystem with trace logging turned on and can see that > > > > after the last successful READDIRP there is a lot of other connections > > > > being made the clients repeatedly which takes minutes to complete. > > > > > > I did not observe anything specific which points to clients repeatedly > > > reconnecting. Can you point to the appropriate line numbers for this? > > > > > > Can you also please describe the directory structure being referred here? > > > > > > > I was tailing the log file while the readdir script was running and > > could see respective READDIRP calls for each readdir, after the last > > call all the rest of the stuff in the log file was returning nothing but > > took minutes to complete. This particular example was a directory > > containing a number of directories, one for each of the READDIRP calls > > in the log file. > > > > One possible tuning that can possibly help: > > volume set cluster.readdir-optimize on > > Let us know if there is any improvement after enabling this option. I'll give it a go but I think this is a bug and not a performance issue. I've filed a bug report on bugzilla. > > Thanks, > Vijay > > This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
