hi james! thanks for your help we are using 3.0.7 at the moment in our productive system
here are some server and then some client volume files: ----- server volume file 1 ----- volume posix type storage/posix option directory /gluster-storage/projekte/ksc/ # option o-direct enable option background-unlink yes # option export-statfs-size no end-volume volume locks type features/locks subvolumes posix end-volume volume ksc type performance/io-threads option thread-count 16 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option transport.socket.listen-port 7025 option auth.addr.ksc.allow 10.0.1.* subvolumes ksc end-volume ----- client volume file 1 ----- volume vgfs-01-001-ksc type protocol/client option transport-type tcp option remote-host vgfs-01-001 option remote-port 7025 option ping-timeout 10 option remote-subvolume ksc end-volume # distribute volume distribute1-ksc type cluster/distribute option lookup-unhashed auto option min-free-disk 5% subvolumes vgfs-01-001-ksc end-volume volume vgfs-01-002-ksc type protocol/client option transport-type tcp option remote-host vgfs-01-002 option remote-port 7025 option ping-timeout 10 option remote-subvolume ksc end-volume # distribute volume distribute2-ksc type cluster/distribute option lookup-unhashed auto option min-free-disk 5% subvolumes vgfs-01-002-ksc end-volume volume ksc-data-replicate type cluster/replicate subvolumes distribute1-ksc distribute2-ksc end-volume volume iocache type performance/io-cache option cache-size 64MB #1GB supported option cache-timeout 1 subvolumes ksc-data-replicate end-volume volume quick-read type performance/quick-read # option cache-timeout 10 (1 second) # option max-file-size 1048576 (64Kb) subvolumes iocache end-volume volume trace type debug/trace subvolumes quick-read # option include open,close,create,readdir,opendir,closedir # option exclude lookup,read,write end-volume ----- server volume file 2 ----- volume posix type storage/posix option directory /gluster-storage/projekte/macm02 end-volume volume locks type features/locks subvolumes posix end-volume volume macm02-data type performance/io-threads option thread-count 16 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option transport.socket.listen-port 7004 option auth.addr.macm02-data.allow 10.0.1.* subvolumes macm02-data end-volume ----- client volume file 2 ----- volume vgfs-01-001-macm02 type protocol/client option transport-type tcp option remote-host vgfs-01-001 option remote-port 7004 option ping-timeout 5 option remote-subvolume macm02-data end-volume # distribute volume distribute1-macm02 type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-001-macm02 end-volume volume vgfs-01-002-macm02 type protocol/client option transport-type tcp option remote-host vgfs-01-002 option remote-port 7004 option ping-timeout 5 option remote-subvolume macm02-data end-volume # distribute volume distribute2-macm02 type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-002-macm02 end-volume volume macm02-data-replicate type cluster/replicate subvolumes distribute1-macm02 distribute2-macm02 end-volume volume readahead type performance/read-ahead option page-count 16 # cache per file = (page-count x page-size) subvolumes macm02-data-replicate end-volume volume writebehind type performance/write-behind option cache-size 2048KB option flush-behind on subvolumes readahead end-volume #volume readahead # type performance/read-ahead # option page-count 16 # cache per file = (page-count x page-size) # subvolumes writebehind #end-volume volume iocache type performance/io-cache option cache-size 1GB #1GB supported option cache-timeout 1 subvolumes writebehind end-volume volume iothreads type performance/io-threads option thread-count 64 # default is 16 subvolumes iocache end-volume volume quickread type performance/quick-read option cache-timeout 1 option max-file-size 64kB subvolumes iothreads end-volume volume statprefetch type performance/stat-prefetch subvolumes quickread end-volume ----- server volume file 3 ----- volume posix type storage/posix option directory /gluster-storage/projekte/hosting2/ option o-direct enable option background-unlink yes end-volume volume locks type features/locks subvolumes posix end-volume volume hosting2 type performance/io-threads option thread-count 16 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option transport.socket.listen-port 7005 option auth.addr.hosting2.allow 10.0.1.* subvolumes hosting2 end-volume ----- client volume file 3 ----- volume vgfs-01-001-hosting2 type protocol/client option transport-type tcp option remote-host vgfs-01-001 option remote-port 7005 option ping-timeout 20 option remote-subvolume hosting2 end-volume volume vgfs-01-002-hosting2 type protocol/client option transport-type tcp option remote-host vgfs-01-002 option remote-port 7005 option ping-timeout 20 option remote-subvolume hosting2 end-volume # distribute volume distribute1-hosting2 type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-001-hosting2 end-volume # distribute volume distribute2-hosting2 type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-002-hosting2 end-volume volume backup-data-replicate type cluster/replicate subvolumes distribute1-hosting2 distribute2-hosting2 end-volume volume readahead type performance/read-ahead option page-count 16 # cache per file = (page-count x page-size) subvolumes backup-data-replicate end-volume volume iocache type performance/io-cache option cache-size 1024MB #1GB supported option cache-timeout 1 subvolumes readahead end-volume volume iothreads type performance/io-threads option thread-count 6 # default is 16 subvolumes iocache end-volume volume quickread type performance/quick-read option cache-timeout 30 option max-file-size 1024000 subvolumes iothreads end-volume ----- server volume file 4 ----- volume posix type storage/posix option directory /gluster-storage/projekte/xna-static end-volume volume locks type features/locks subvolumes posix end-volume volume iocache type performance/io-cache option cache-size 1GB #1GB supported option cache-timeout 1 subvolumes locks end-volume volume xna-static-data type performance/io-threads option thread-count 32 subvolumes iocache end-volume volume server type protocol/server option transport-type tcp option transport.socket.listen-port 7006 option auth.addr.xna-static-data.allow 10.0.1.* subvolumes xna-static-data end-volume ----- client volume file 4 ----- volume vgfs-01-001-xna-static type protocol/client option transport-type tcp option remote-host vgfs-01-001 option remote-port 7006 option ping-timeout 5 option remote-subvolume xna-static-data end-volume # distribute volume distribute1-xna-static type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-001-xna-static end-volume volume vgfs-01-002-xna-static type protocol/client option transport-type tcp option remote-host vgfs-01-002 option remote-port 7006 option ping-timeout 5 option remote-subvolume xna-static-data end-volume # distribute volume distribute2-xna-static type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-002-xna-static end-volume volume xna-static-data-replicate type cluster/replicate subvolumes distribute1-xna-static distribute2-xna-static end-volume volume iocache type performance/io-cache option cache-size 1GB #1GB supported option cache-timeout 1 subvolumes xna-static-data-replicate end-volume volume iothreads type performance/io-threads option thread-count 8 # default is 16 subvolumes iocache end-volume volume quickread type performance/quick-read option cache-timeout 1 option max-file-size 128kB subvolumes iothreads end-volume volume statprefetch type performance/stat-prefetch subvolumes quickread end-volume regards markus Am 27.01.2011 17:57, schrieb Burnash, James:
Hello. I have experienced this situation with the 3.0.4 release of Glusterfs - it was related to a bug that had to do with recursive file deletions (in my case). That bug has been fixed in 3.1.1 which is what I am currently running. Can you give us your Glusterfs version, and a copy of your volume files for server and client? That would help us to help you. Thanks, James -----Original Message----- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Markus Fröhlich Sent: Thursday, January 27, 2011 11:03 AM To: gluster-users@gluster.org Subject: [Gluster-users] sometimes connection errors - glusterfs limit? hi! problem description: 1-3 times a day glusterfs seems to hang - like freezed - network traffic is a few secounds again zero, because there is no glusterfs communication. in the log of the servers, that clients got disconnected and a little bit later connected again but the network traffic seems okay and is not at limit - there are no errors on the networkinterfaces. we also checked cables and switchports on the switches. we think that glusterfs in combination with the I/O is the bottleneck here. average we can say the storage setup is in use with 2/3 read and 1/3 write operations. is there a limitation of volumes, clients, mounts etc. in the glusterfs code? has some one made similar experience or troubles with such a setup? is it possible that to much clients are connected to few servers? has any one some tip for us? ------------------------------------- our setup: two glusterfs servers: * areca raid controller with raid5 setup * 3 LUNs each 11 TB with 70%-99% in use with ext3 formated * SLES11 x86_64 * glusterfs V 3.0.7 45 - 50 glusterfs client servers: * SLES10, SLES11, SLES11 SP1 * glusterfs V 3.0.7 * all VOLs client replicated to the both glusterfs servers ----------------------------------- volume files: server export vols look like this - some options differ: -- volume posix type storage/posix option directory /gluster-storage/projekte/ksc/ option background-unlink yes end-volume volume locks type features/locks subvolumes posix end-volume volume ksc type performance/io-threads option thread-count 16 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option transport.socket.listen-port 7025 option auth.addr.ksc.allow 10.0.1.* subvolumes ksc end-volume -- volume posix type storage/posix option directory /gluster-storage/projekte/hosting/ option o-direct enable option background-unlink yes end-volume volume locks type features/locks subvolumes posix end-volume volume hosting2 type performance/io-threads option thread-count 16 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option transport.socket.listen-port 7005 option auth.addr.hosting2.allow 10.0.1.* subvolumes hosting2 end-volume -- client repl. mount VOL files: volume vgfs-01-001-ksc type protocol/client option transport-type tcp option remote-host vgfs-01-001 option remote-port 7025 option ping-timeout 10 option remote-subvolume ksc end-volume # distribute volume distribute1-ksc type cluster/distribute option lookup-unhashed auto option min-free-disk 5% subvolumes vgfs-01-001-ksc end-volume volume vgfs-01-002-ksc type protocol/client option transport-type tcp option remote-host vgfs-01-002 option remote-port 7025 option ping-timeout 10 option remote-subvolume ksc end-volume # distribute volume distribute2-ksc type cluster/distribute option lookup-unhashed auto option min-free-disk 5% subvolumes vgfs-01-002-ksc end-volume volume ksc-data-replicate type cluster/replicate subvolumes distribute1-ksc distribute2-ksc end-volume volume iocache type performance/io-cache option cache-size 64MB #1GB supported option cache-timeout 1 subvolumes ksc-data-replicate end-volume volume quick-read type performance/quick-read # option cache-timeout 10 (1 second) # option max-file-size 1048576 (64Kb) subvolumes iocache end-volume volume trace type debug/trace subvolumes quick-read # option include open,close,create,readdir,opendir,closedir # option exclude lookup,read,write end-volume -- volume vgfs-01-001-hosting type protocol/client option transport-type tcp option remote-host vgfs-01-001 option remote-port 7005 option ping-timeout 20 option remote-subvolume hosting end-volume volume vgfs-01-002-hosting type protocol/client option transport-type tcp option remote-host vgfs-01-002 option remote-port 7005 option ping-timeout 20 option remote-subvolume hosting end-volume # distribute volume distribute1-hosting type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-001-hosting end-volume # distribute volume distribute2-hosting type cluster/distribute option lookup-unhashed yes option min-free-disk 5% subvolumes vgfs-01-002-hosting end-volume volume backup-data-replicate type cluster/replicate subvolumes distribute1-hosting distribute2-hosting subvolumes distribute2-hosting end-volume volume readahead type performance/read-ahead option page-count 16 # cache per file = (page-count x page-size) subvolumes backup-data-replicate end-volume volume iocache type performance/io-cache option cache-size 1024MB #1GB supported option cache-timeout 1 subvolumes readahead end-volume volume iothreads type performance/io-threads option thread-count 6 # default is 16 subvolumes iocache end-volume volume quickread type performance/quick-read option cache-timeout 30 option max-file-size 1024000 subvolumes iothreads end-volume -- regards markus
-- Mit freundlichen Grüssen Markus Fröhlich Techniker ________________________________________________________ Xidras GmbH Stockern 47 3744 Stockern Austria Tel: +43 (0) 2983 201 30503 Fax: +43 (0) 2983 201 305039 Email: markus.froehl...@xidras.com Web: http://www.xidras.com FN 317036 f | Landesgericht Krems | ATU64485024 ________________________________________________________________________________ VERTRAULICHE INFORMATIONEN! Diese eMail enthält vertrauliche Informationen und ist nur für den berechtigten Empfänger bestimmt. Wenn diese eMail nicht für Sie bestimmt ist, bitten wir Sie, diese eMail an uns zurückzusenden und anschließend auf Ihrem Computer und Mail-Server zu löschen. Solche eMails und Anlagen dürfen Sie weder nutzen, noch verarbeiten oder Dritten zugänglich machen, gleich in welcher Form. Wir danken für Ihre Kooperation! CONFIDENTIAL! This email contains confidential information and is intended for the authorised recipient only. If you are not an authorised recipient, please return the email to us and then delete it from your computer and mail-server. You may neither use nor edit any such emails including attachments, nor make them accessible to third parties in any manner whatsoever. Thank you for your cooperation ________________________________________________________________________________ _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users