Re: [Gluster-users] Gluster speed sooo slow
There is a big difference with working with small files (around 16kb) and big files (2mb). Performance is much better with big files. Witch is too bad for me ;( On 8/11/12 2:15 AM, Gandalf Corvotempesta wrote: What do you mean with small files? 16k ? 160k? 16mb? Do you know any workaround or any other software for this? Mee too i'm trying to create a clustered storage for many small file 2012/8/10 Philip Poten philip.po...@gmail.com mailto:philip.po...@gmail.com Hi Ivan, that's because Gluster has really bad many small files performance due to it's architecture. On all stat() calls (which rsync is doing plenty of), all replicas are being checked for integrity. regards, Philip 2012/8/10 Ivan Dimitrov dob...@amln.net mailto:dob...@amln.net: So I stopped a node to check the BIOS and after it went up, the rebalance kicked in. I was looking for those kind of speeds on a normal write. The rebalance is much faster than my rsync/cp. https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%202.04.09%20PM.png Best Regards Ivan Dimitrov On 8/10/12 1:23 PM, Ivan Dimitrov wrote: Hello What am I doing wrong?!? I have a test setup with 4 identical servers with 2 disks each in distribute-replicate 2. All servers are connected to a GB switch. I am experiencing really slow speeds at anything I do. Slow write, slow read, not to mention random write/reads. Here is an example: random-files is a directory with 32768 files with average size 16kb. [root@gltclient]:~# rsync -a /root/speedtest/random-files/ /home/gltvolume/ ^^ This will take more than 3 hours. On any of the servers if I do iostat the disks are not loaded at all: https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png This is similar result for all servers. Here is an example of simple ls command on the content. [root@gltclient]:~# unalias ls [root@gltclient]:~# /usr/bin/time -f %e seconds ls /home/gltvolume/ | wc -l 2.81 seconds 5393 almost 3 seconds to display 5000 files?!?! When they are 32,000, the ls will take around 35-45 seconds. This directory is on local disk: [root@gltclient]:~# /usr/bin/time -f %e seconds ls /root/speedtest/random-files/ | wc -l 1.45 seconds 32768 [root@gltclient]:~# /usr/bin/time -f %e seconds cat /home/gltvolume/* /dev/null 190.50 seconds [root@gltclient]:~# /usr/bin/time -f %e seconds du -sh /home/gltvolume/ 126M/home/gltvolume/ 75.23 seconds Here is the volume information. [root@glt1]:~# gluster volume info Volume Name: gltvolume Type: Distributed-Replicate Volume ID: 16edd852-8d23-41da-924d-710b753bb374 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 1.1.74.246:/home/sda3 Brick2: glt2.network.net:/home/sda3 Brick3: 1.1.74.246:/home/sdb1 Brick4: glt2.network.net:/home/sdb1 Brick5: glt3.network.net:/home/sda3 Brick6: gltclient.network.net:/home/sda3 Brick7: glt3.network.net:/home/sdb1 Brick8: gltclient.network.net:/home/sdb1 Options Reconfigured: performance.io-thread-count: 32 performance.cache-size: 256MB cluster.self-heal-daemon: on [root@glt1]:~# gluster volume status all detail Status of volume: gltvolume -- Brick: Brick 1.1.74.246:/home/sda3 Port : 24009 Online : Y Pid : 1479 File System : ext4 Device : /dev/sda3 Mount Options: rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11901550 -- Brick: Brick glt2.network.net:/home/sda3 Port : 24009 Online : Y Pid : 1589 File System : ext4 Device : /dev/sda3 Mount Options: rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11901550 -- Brick: Brick 1.1.74.246:/home/sdb1 Port : 24010 Online : Y Pid : 1485 File System : ext4 Device
Re: [Gluster-users] Gluster speed sooo slow
I think Gluster as it stands now and current level of development is more for Multimedia and Archival files, not for small files nor for running Virtual Machines. It requires still a fair amount of development which hopefully RedHat will put in place. Fernando From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Ivan Dimitrov Sent: 13 August 2012 08:33 To: gluster-users@gluster.org Subject: Re: [Gluster-users] Gluster speed sooo slow There is a big difference with working with small files (around 16kb) and big files (2mb). Performance is much better with big files. Witch is too bad for me ;( On 8/11/12 2:15 AM, Gandalf Corvotempesta wrote: What do you mean with small files? 16k ? 160k? 16mb? Do you know any workaround or any other software for this? Mee too i'm trying to create a clustered storage for many small file 2012/8/10 Philip Poten philip.po...@gmail.commailto:philip.po...@gmail.com Hi Ivan, that's because Gluster has really bad many small files performance due to it's architecture. On all stat() calls (which rsync is doing plenty of), all replicas are being checked for integrity. regards, Philip 2012/8/10 Ivan Dimitrov dob...@amln.netmailto:dob...@amln.net: So I stopped a node to check the BIOS and after it went up, the rebalance kicked in. I was looking for those kind of speeds on a normal write. The rebalance is much faster than my rsync/cp. https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%202.04.09%20PM.png Best Regards Ivan Dimitrov On 8/10/12 1:23 PM, Ivan Dimitrov wrote: Hello What am I doing wrong?!? I have a test setup with 4 identical servers with 2 disks each in distribute-replicate 2. All servers are connected to a GB switch. I am experiencing really slow speeds at anything I do. Slow write, slow read, not to mention random write/reads. Here is an example: random-files is a directory with 32768 files with average size 16kb. [root@gltclient]:~# rsync -a /root/speedtest/random-files/ /home/gltvolume/ ^^ This will take more than 3 hours. On any of the servers if I do iostat the disks are not loaded at all: https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png This is similar result for all servers. Here is an example of simple ls command on the content. [root@gltclient]:~# unalias ls [root@gltclient]:~# /usr/bin/time -f %e seconds ls /home/gltvolume/ | wc -l 2.81 seconds 5393 almost 3 seconds to display 5000 files?!?! When they are 32,000, the ls will take around 35-45 seconds. This directory is on local disk: [root@gltclient]:~# /usr/bin/time -f %e seconds ls /root/speedtest/random-files/ | wc -l 1.45 seconds 32768 [root@gltclient]:~# /usr/bin/time -f %e seconds cat /home/gltvolume/* /dev/null 190.50 seconds [root@gltclient]:~# /usr/bin/time -f %e seconds du -sh /home/gltvolume/ 126M/home/gltvolume/ 75.23 seconds Here is the volume information. [root@glt1]:~# gluster volume info Volume Name: gltvolume Type: Distributed-Replicate Volume ID: 16edd852-8d23-41da-924d-710b753bb374 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 1.1.74.246:/home/sda3 Brick2: glt2.network.net:/home/sda3 Brick3: 1.1.74.246:/home/sdb1 Brick4: glt2.network.net:/home/sdb1 Brick5: glt3.network.net:/home/sda3 Brick6: gltclient.network.net:/home/sda3 Brick7: glt3.network.net:/home/sdb1 Brick8: gltclient.network.net:/home/sdb1 Options Reconfigured: performance.io-thread-count: 32 performance.cache-size: 256MB cluster.self-heal-daemon: on [root@glt1]:~# gluster volume status all detail Status of volume: gltvolume -- Brick: Brick 1.1.74.246:/home/sda3 Port : 24009 Online : Y Pid : 1479 File System : ext4 Device : /dev/sda3 Mount Options: rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11901550 -- Brick: Brick glt2.network.net:/home/sda3 Port : 24009 Online : Y Pid : 1589 File System : ext4 Device : /dev/sda3 Mount Options: rw,noatime Inode Size : 256 Disk Space Free : 179.3GB Total Disk Space : 179.7GB Inode Count : 11968512 Free Inodes : 11901550 -- Brick: Brick 1.1.74.246:/home/sdb1 Port : 24010 Online : Y Pid : 1485 File System : ext4 Device : /dev/sdb1 Mount Options: rw,noatime Inode Size : 256 Disk Space
[Gluster-users] Problem with too many small files
I am not sure how it works on Gluster but to mitigate the problem with listing a lot of small files wouldn't it be suitable to keep on every node a copy of the directory tree. I think Isilon does that and there is probably a lot to be learned from them which seems quiet mature technology. Could also have another interesting thing added in the future, local SSD to keep the file system metadata for faster access. Regards, Fernando Frediani Lead Systems Engineer [Description: Description: Description: Description: Description: cid:image003.png@01CD637B.AB2C8040] 260-266 Goswell Road, London, EC1V 7EB, United Kingdom sales: +44 (0) 20 7150 3800 ddi: +44 (0) 20 7150 3803 fax:+44 (0) 20 7336 8420 web: http://www.qubenet.net/ Qube Managed Services Limited Company Number: 6215769 (Registered in England and Wales) VAT Registration Number: GB 933 8400 27 This e-mail and the information it contains are confidential. If you have received this e-mail in error please notify the sender immediately. You should not copy it for any purpose or disclose its contents to any other person. The contents of this e-mail do not necessarily reflect the views of the company. EOE P Please consider the environment before printing this email inline: image001.png___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster speed sooo slow
On Mon, Aug 13, 2012 at 09:40:49AM +, Fernando Frediani (Qube) wrote: I think Gluster as it stands now and current level of development is more for Multimedia and Archival files, not for small files nor for running Virtual Machines. It requires still a fair amount of development which hopefully RedHat will put in place. I know a large ISP is using gluster successfully for Maildir storage - or at least was a couple of years ago when I last spoke to them about it - which means very large numbers of small files. I think you need to be clear on the difference between throughput and latency. Any networked filesystem is going to have latency, and gluster maybe suffers more than most because of the FUSE layer at the client. This will show as poor throughput if a single client is sequentially reading or writing lots of small files, because it has to wait a round trip for each request. However, if you have multiple clients accessing at the same time, you can still have high total throughput. This is because the wasted time between requests from one client is used to service other clients. If gluster were to do aggressive client-side caching then it might be able to make responses appear faster to a single client, but this would be at the risk of data loss (e.g. responding that a file has been committed to disk, when in fact it hasn't). But this would make no difference to total throughput with multiple clients, which depends on the available bandwidth into the disk drives and across the network. So it all depends on your overall usage pattern. Only make your judgement based on a single-threaded benchmark if that's what your usage pattern is really going to be like: i.e. are you really going to have a single user accessing the filesystem, and their application reads or writes one file after the other rather than multiple files concurrently. Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster speed sooo slow
I heard from a Large ISP talking to someone that works there they were trying to use GlusteFS for Maildir and they had a hell because of the many small files and had customer complaining all the time. Latency is acceptable on a networked filesystem, but the results people are reporting are beyond any latency problems, they are due to the way Gluster is structured and that was already confirmed by some people on this list, so changed are indeed needed on the code. If you take even a Gigabit network the round trip isn't that much really, (not more than a quarter of a ms) so it shouldn't be a big thing. Yes FUSE might also contribute to decrease performance but still the performance problems are on the architecture of the filesystem. One thing that is new to Gluster and that in my opinion could contribute to increase performance is the Distributed-Stripped volumes, but that doesn't still work for all enviroemnts. So as it stands for Multimedia or Archive files fine, for other usages I wouldn't bet my chips and would rather test thoroughly first. -Original Message- From: Brian Candler [mailto:b.cand...@pobox.com] Sent: 13 August 2012 11:00 To: Fernando Frediani (Qube) Cc: 'Ivan Dimitrov'; 'gluster-users@gluster.org' Subject: Re: [Gluster-users] Gluster speed sooo slow On Mon, Aug 13, 2012 at 09:40:49AM +, Fernando Frediani (Qube) wrote: I think Gluster as it stands now and current level of development is more for Multimedia and Archival files, not for small files nor for running Virtual Machines. It requires still a fair amount of development which hopefully RedHat will put in place. I know a large ISP is using gluster successfully for Maildir storage - or at least was a couple of years ago when I last spoke to them about it - which means very large numbers of small files. I think you need to be clear on the difference between throughput and latency. Any networked filesystem is going to have latency, and gluster maybe suffers more than most because of the FUSE layer at the client. This will show as poor throughput if a single client is sequentially reading or writing lots of small files, because it has to wait a round trip for each request. However, if you have multiple clients accessing at the same time, you can still have high total throughput. This is because the wasted time between requests from one client is used to service other clients. If gluster were to do aggressive client-side caching then it might be able to make responses appear faster to a single client, but this would be at the risk of data loss (e.g. responding that a file has been committed to disk, when in fact it hasn't). But this would make no difference to total throughput with multiple clients, which depends on the available bandwidth into the disk drives and across the network. So it all depends on your overall usage pattern. Only make your judgement based on a single-threaded benchmark if that's what your usage pattern is really going to be like: i.e. are you really going to have a single user accessing the filesystem, and their application reads or writes one file after the other rather than multiple files concurrently. Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Unable to rebalance...status or stop after upgrade to 3.3
Harry- Thanks for the tip. My problem could well have been the same as yours. I have known for some time that gluster peer status doesn't give useful connection information but I didn't know about the gluster volume status commands; they must be new in version 3.3. I usually discover connection problems by seeing phrases like disconnected and anomalies in the logs. This has been happening more often since I upgraded to version 3.3, and I suspect it is being caused by the very high load experienced by some servers. I have seen this load problem discussed in other threads. The next time I attempt a rebalance operation I will run gluster volume status all detail first to check connectivity. -Dan On 08/08/2012 08:31 PM, Harry Mangalam wrote: This sounds similar, tho not identical to a problem that I had recently (descriibed here: http://gluster.org/pipermail/gluster-users/2012-August/011054.html My problems resulted were teh result of starting this kind of rebalance with a server node appearing to be connected (via the 'gluster peer status' output, but not actually being connected as shown by the 'gluster volume status all detail' output. Note especially the part that describes its online state. -- Brick: Brick pbs3ib:/bducgl Port : 24018 Online : N = Pid : 20953 File System : xfs You may have already verified this, but what I did was to start a rebalance / fix-layout with a disconnected brick and it went ahead and tried to do it, unsuccessfully as you might guess.. But when I finally was able to reconnect the downed brick, and restart the rebalance, it (astonishingly) was able to bring everything back. So props to the gluster team. hjm On Wed, Aug 8, 2012 at 11:58 AM, Dan Bretherton d.a.brether...@reading.ac.uk mailto:d.a.brether...@reading.ac.uk wrote: Hello All- I have noticed another problem after upgrading to version 3.3. I am unable to do gluster volume rebalance VOLUME fix-layout status or ...fix-layout ... stop after starting a rebalance operation with gluster volume rebalance VOLUME fix-layout start. The fix-layout operation seemed to be progressing normally on all the servers according to the log files, but all attempts to do status or stop result in the CLI usage message being returned. The only reference to the rebalance commands in the log files were these, which all the servers seem to have one or more of. [root@romulus glusterfs]# grep rebalance *.log etc-glusterfs-glusterd.vol.log:[2012-08-08 12:49:04.870709] W [socket.c:1512:__socket_proto_state_machine] 0-management: reading from socket failed. Error (Transport endpoint is not connected), peer (/var/lib/glusterd/vols/tracks/rebalance/cb21050d-05c2-42b3-8660-230954bab324.sock) tracks-rebalance.log:[2012-08-06 10:41:18.550241] I [graph.c:241:gf_add_cmdline_options] 0-tracks-dht: adding option 'rebalance-cmd' for volume 'tracks-dht' with value '4' The volume name is tracks by the way. I wanted to stop the rebalance operation because it seemed to be causing a very high load on some of the servers had been running for several days. I ended up having to manually kill the rebalance processes on all the servers followed by restarting glusterd. After that I found that one of the servers had rebalance_status=4 in file /var/lib/glusterd/vols/tracks/node_state.info http://node_state.info, whereas all the others had rebalance_status=0. I manually changed the '4' to '0' and restarted glusterd. I don't know if this was a consequence of the way I had killed the rebalance operation or the cause of the strange behaviour. I don't really want to start another rebalance going to test because the last one was so disruptive. Has anyone else experienced this problem since upgrading to 3.3? Regards, Dan. ___ Gluster-users mailing list Gluster-users@gluster.org mailto:Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster speed sooo slow
I have a low traffic free hosting and I converted some x,000 users on glusterfs a few months ago. I'm not impressed at all and would probably not convert any more users. It works ok for now, but with 88GB of 2TB volume. It's kind of pointless for now... :( I'm researching a way to convert my payed hosting users, but I can't find any system suitable for the job. Fernando, what gluster structure are you talking about? Best Regards Ivan Dimitrov Fernando, what On 8/13/12 2:16 PM, Fernando Frediani (Qube) wrote: I heard from a Large ISP talking to someone that works there they were trying to use GlusteFS for Maildir and they had a hell because of the many small files and had customer complaining all the time. Latency is acceptable on a networked filesystem, but the results people are reporting are beyond any latency problems, they are due to the way Gluster is structured and that was already confirmed by some people on this list, so changed are indeed needed on the code. If you take even a Gigabit network the round trip isn't that much really, (not more than a quarter of a ms) so it shouldn't be a big thing. Yes FUSE might also contribute to decrease performance but still the performance problems are on the architecture of the filesystem. One thing that is new to Gluster and that in my opinion could contribute to increase performance is the Distributed-Stripped volumes, but that doesn't still work for all enviroemnts. So as it stands for Multimedia or Archive files fine, for other usages I wouldn't bet my chips and would rather test thoroughly first. -Original Message- From: Brian Candler [mailto:b.cand...@pobox.com] Sent: 13 August 2012 11:00 To: Fernando Frediani (Qube) Cc: 'Ivan Dimitrov'; 'gluster-users@gluster.org' Subject: Re: [Gluster-users] Gluster speed sooo slow On Mon, Aug 13, 2012 at 09:40:49AM +, Fernando Frediani (Qube) wrote: I think Gluster as it stands now and current level of development is more for Multimedia and Archival files, not for small files nor for running Virtual Machines. It requires still a fair amount of development which hopefully RedHat will put in place. I know a large ISP is using gluster successfully for Maildir storage - or at least was a couple of years ago when I last spoke to them about it - which means very large numbers of small files. I think you need to be clear on the difference between throughput and latency. Any networked filesystem is going to have latency, and gluster maybe suffers more than most because of the FUSE layer at the client. This will show as poor throughput if a single client is sequentially reading or writing lots of small files, because it has to wait a round trip for each request. However, if you have multiple clients accessing at the same time, you can still have high total throughput. This is because the wasted time between requests from one client is used to service other clients. If gluster were to do aggressive client-side caching then it might be able to make responses appear faster to a single client, but this would be at the risk of data loss (e.g. responding that a file has been committed to disk, when in fact it hasn't). But this would make no difference to total throughput with multiple clients, which depends on the available bandwidth into the disk drives and across the network. So it all depends on your overall usage pattern. Only make your judgement based on a single-threaded benchmark if that's what your usage pattern is really going to be like: i.e. are you really going to have a single user accessing the filesystem, and their application reads or writes one file after the other rather than multiple files concurrently. Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster speed sooo slow
3.2 Ivan. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Ivan Dimitrov Sent: 13 August 2012 12:33 To: 'gluster-users@gluster.org' Subject: Re: [Gluster-users] Gluster speed sooo slow I have a low traffic free hosting and I converted some x,000 users on glusterfs a few months ago. I'm not impressed at all and would probably not convert any more users. It works ok for now, but with 88GB of 2TB volume. It's kind of pointless for now... :( I'm researching a way to convert my payed hosting users, but I can't find any system suitable for the job. Fernando, what gluster structure are you talking about? Best Regards Ivan Dimitrov Fernando, what On 8/13/12 2:16 PM, Fernando Frediani (Qube) wrote: I heard from a Large ISP talking to someone that works there they were trying to use GlusteFS for Maildir and they had a hell because of the many small files and had customer complaining all the time. Latency is acceptable on a networked filesystem, but the results people are reporting are beyond any latency problems, they are due to the way Gluster is structured and that was already confirmed by some people on this list, so changed are indeed needed on the code. If you take even a Gigabit network the round trip isn't that much really, (not more than a quarter of a ms) so it shouldn't be a big thing. Yes FUSE might also contribute to decrease performance but still the performance problems are on the architecture of the filesystem. One thing that is new to Gluster and that in my opinion could contribute to increase performance is the Distributed-Stripped volumes, but that doesn't still work for all enviroemnts. So as it stands for Multimedia or Archive files fine, for other usages I wouldn't bet my chips and would rather test thoroughly first. -Original Message- From: Brian Candler [mailto:b.cand...@pobox.com] Sent: 13 August 2012 11:00 To: Fernando Frediani (Qube) Cc: 'Ivan Dimitrov'; 'gluster-users@gluster.org' Subject: Re: [Gluster-users] Gluster speed sooo slow On Mon, Aug 13, 2012 at 09:40:49AM +, Fernando Frediani (Qube) wrote: I think Gluster as it stands now and current level of development is more for Multimedia and Archival files, not for small files nor for running Virtual Machines. It requires still a fair amount of development which hopefully RedHat will put in place. I know a large ISP is using gluster successfully for Maildir storage - or at least was a couple of years ago when I last spoke to them about it - which means very large numbers of small files. I think you need to be clear on the difference between throughput and latency. Any networked filesystem is going to have latency, and gluster maybe suffers more than most because of the FUSE layer at the client. This will show as poor throughput if a single client is sequentially reading or writing lots of small files, because it has to wait a round trip for each request. However, if you have multiple clients accessing at the same time, you can still have high total throughput. This is because the wasted time between requests from one client is used to service other clients. If gluster were to do aggressive client-side caching then it might be able to make responses appear faster to a single client, but this would be at the risk of data loss (e.g. responding that a file has been committed to disk, when in fact it hasn't). But this would make no difference to total throughput with multiple clients, which depends on the available bandwidth into the disk drives and across the network. So it all depends on your overall usage pattern. Only make your judgement based on a single-threaded benchmark if that's what your usage pattern is really going to be like: i.e. are you really going to have a single user accessing the filesystem, and their application reads or writes one file after the other rather than multiple files concurrently. Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster speed sooo slow
One thing that is new to Gluster and that in my opinion could contribute to increase performance is the Distributed-Stripped volumes, Only if you have a single huge file, and you are doing a large read or write to it - i.e. exactly the opposite case of lots of small files. for other usages I wouldn't bet my chips and would rather test thoroughly first. I agree 100% - for whatever solution you choose. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Problem with too many small files
On August 13, 2012 5:52:26 AM Fernando Frediani (Qube) fernando.fredi...@qubenet.net wrote: I am not sure how it works on Gluster but to mitigate the problem with listing a lot of small files wouldn't it be suitable to keep on every node a copy of the directory tree. I think Isilon does that and there is probably a lot to be learned from them which seems quiet mature technology. Could also have another interesting thing added in the future, local SSD to keep the file system metadata for faster access. We could do that, in fact I've been an advocate for it, but it must be understood that there's no such thing as a free lunch. Once you're caching directory structures on clients, you either have to give up a certain amount of consistency or make the entire protocol much more complex to perform cache invalidations etc. Who's volunteering to do that work? Who's even asking us to do that in the core team, once they understand that it means taking resources away from other priorities and permanently slowing down development because of that complexity? Nobody. At least, unlike Isilon, there's the possibility that somebody could take a stab at reducing consistency for the sake of performance themselves (as I myself have done e.g. with negative-lookup caching and replication bypass). There's not really all that much to be learned from a closed-source system that's not even described in papers. In fact, I *know* that they learn more from us than vice versa. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Replica with non-uniform bricks
Hello, as per http://gluster.org/pipermail/gluster-users/2011-May/007788.html I am advised to use bricks which are uniform in size, but for a number of reasons this is not possible at the moment in my current setup. So the question is: what is the expected behaviour when two bricks with different sizes are coupled to form a replica? Will the larger brick keep accepting writes even after the smallest brick has been filled up? In this scenario, what conditions become relevant in the process of rebalancing a volume? Is there a precedence of the first brick in a replica (as per volume definition)? Thank you very much in advance for your attention, Best regards ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] replica 3x corrupt files after restart
Hi everyone, I've build gluster 3.3 from source, setup 3x node Replicate. ( fro serving VM images to kvm ) All seems to be working fine till the power went off for weekend.. brick 1 was run till battery flat and safely shutdown. then brick 2 done the same. brick 3 was running till ups has no more power ( fault in UPS - which cause power off for the whole room..) So when I came in the morning - find the problem , replace UPS and start all of them in single mode to find which one has the more resent data. Found that is the brick3. OK - started - left 2 other stay off till the end of business day - so can start sync when less loading. OK work hours ending and I decide to think brick12. ( done backup first ) power on 1 and 2 start glusterd from brick 3 initiate self-heal. The result 2 biggest file from 6 become corrupted. Now restoring them back from backup Any glue why is this happening? from logs I can't see that those 2 corrupted files was healing ( and check md5sum on all 3 brick show same - corrupted ) again from logs I can see only 3 files was healed. I can see some other users experienced similar problem: http://www.bauer-power.net/2012/03/glusterfs-is-not-ready-for-san-storage.html -- -- Michael ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replica with non-uniform bricks
On Mon, Aug 13, 2012 at 06:03:36PM +0200, s19n wrote: So the question is: what is the expected behaviour when two bricks with different sizes are coupled to form a replica? Will the larger brick keep accepting writes even after the smallest brick has been filled up? I haven't tested, but I'd expect ENOSPC when the smaller one files up. I'd be inclined to split the larger server into two volumes: one the same size as the smaller server, to form a replica set, and put the remaining space in a single-volume brick for use as scratch space, backups, or whatever. Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replica with non-uniform bricks
On 08/13/2012 04:16 PM, Brian Candler wrote: On Mon, Aug 13, 2012 at 06:03:36PM +0200, s19n wrote: So the question is: what is the expected behaviour when two bricks with different sizes are coupled to form a replica? Will the larger brick keep accepting writes even after the smallest brick has been filled up? I haven't tested, but I'd expect ENOSPC when the smaller one files up. I'd be inclined to split the larger server into two volumes: one the same size as the smaller server, to form a replica set, and put the remaining space in a single-volume brick for use as scratch space, backups, or whatever. I second that. I haven't tested this myself, but I don't see any special handling of ENOSPC in the AFR code. I strongly suspect that what you'd see is an initial write error on the smaller replica, followed by repeated self-heal failures for the same reasons. Ick. Just be aware that you'd be sharing resources (e.g. disk heads) between the artificially small replica and the scratch space, so using both simultaneously might cause a significant performance degradation. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] question about list directory missing files or hang
Hi Gluster experts, I'm new to glusterfs and I have encountered a problem about list directory of glusters 3.3. I have a volume configuration of 3(distribute) * 2(replica). When write file on the glusterfs client mount directory some of the files can't be listed through ls command but the file exists. Some times the ls command hangs. Any one know what's the problem is? Thank you very much. -- 符永涛 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] question about list directory missing files or hang
Hi all, Any one helps? More information about this issue. for example if i create abc.zip by touch abc.zip then run ls it hangs but if I run rm -rf abc.zip then ls returns many file with same name seems there's bug here. ls hangs because it falls into a loop and the files stat are not valid. Thank you. 2012/8/14 符永涛 yongta...@gmail.com Hi Gluster experts, I'm new to glusterfs and I have encountered a problem about list directory of glusters 3.3. I have a volume configuration of 3(distribute) * 2(replica). When write file on the glusterfs client mount directory some of the files can't be listed through ls command but the file exists. Some times the ls command hangs. Any one know what's the problem is? Thank you very much. -- 符永涛 -- 符永涛 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
