Re: [Gluster-users] very bad performance on small files
On Mon, Jan 17, 2011 at 4:32 AM, Anand Avati wrote: > On Sun, Jan 16, 2011 at 5:02 PM, Max Ivanov wrote: > >> >> > time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec >> >> > (FUSE) 254 Kb/sec (NFS) >> >> >> > >> > This test shows why glusterfs native protocol is better than NFS when you >> > need to scale out storage. Even with a context switch overhead on the >> client >> > side, glusterfs scores better due to the "clustered nature" of its >> protocol. >> > NFS has to undergo a second hop when it has to fetch data not available >> in >> > the server it has mounted from whereas for glusterfs it is always a >> single >> > hop to any server it wants to get data from. >> >> My tests was done on 2 bircks setup mounted in replica mode, thereby >> all needed data was avaiable on NFS node and there was no need to do >> additional hop. >> >> > NFS still undergoes the second hop to complete lookups, verify sanity of > copies etc. The fops are - lookup(), open(), release(). > > Avati > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > It is also important to note that many on disk file systems such as FreeBSD's default file system do synchronous meta operations by default. Thus extracting http tar can take 30 seconds while a standard linux ext3 "seems" to almost extract this file instantly. So even some non-networked file systems are not fast at extracting large tars. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On Sun, Jan 16, 2011 at 5:02 PM, Max Ivanov wrote: > >> > time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec > >> > (FUSE) 254 Kb/sec (NFS) > >> > > > > This test shows why glusterfs native protocol is better than NFS when you > > need to scale out storage. Even with a context switch overhead on the > client > > side, glusterfs scores better due to the "clustered nature" of its > protocol. > > NFS has to undergo a second hop when it has to fetch data not available > in > > the server it has mounted from whereas for glusterfs it is always a > single > > hop to any server it wants to get data from. > > My tests was done on 2 bircks setup mounted in replica mode, thereby > all needed data was avaiable on NFS node and there was no need to do > additional hop. > > NFS still undergoes the second hop to complete lookups, verify sanity of copies etc. The fops are - lookup(), open(), release(). Avati ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
>> > time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec >> > (FUSE) 254 Kb/sec (NFS) >> > > This test shows why glusterfs native protocol is better than NFS when you > need to scale out storage. Even with a context switch overhead on the client > side, glusterfs scores better due to the "clustered nature" of its protocol. > NFS has to undergo a second hop when it has to fetch data not available in > the server it has mounted from whereas for glusterfs it is always a single > hop to any server it wants to get data from. My tests was done on 2 bircks setup mounted in replica mode, thereby all needed data was avaiable on NFS node and there was no need to do additional hop. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On Sun, 16 Jan 2011 02:45:50 +0530 Anand Avati wrote: > In any case comparing to local disk performance and network disk performance > is never right and is always misleading. > > Avati This statement is fundamentally broken. -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
> > > Sure, and all that applies equally to both NFS and gluster, yet in Max's > example NFS was ~50x faster than gluster for an identical small-file > workload. So what's gluster doing over and above what NFS is doing that's > taking so long, given that network and disk factors are equal? I'd buy a > factor of 2 for replication, but not 50. > > When using FUSE, the context switch the syscall undergoes even before glusterfs gets a hand on it is a _huge_ factor, especially when (wrongly) comparing with local filesystems. > In case you missed what I'm on about, it was these stats that Max posted: > > > Here is the results per command: > > dd if=/dev/zero of=M/tmp bs=1M count=16384 69.2 MB/se (Native) 69.2 > > MB/sec(FUSE) 52 MB/sec (NFS) > This test looks reasonable. Writes seem to be bottlenecked at the sustained write throughput of the disk itself. > > dd if=/dev/zero of=M/tmp bs=1K count=16384 88.1 MB/sec (Native) > > 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) > The huge drop of FUSE performance compared to NFS is due to the context switch overhead (which glusterfs cannot to much, as it is the latency coming in much before glusterfs even comes into the picture). Since both glusterfs and NFS does caching of writes, the comparison is really ending up being the latency of the context switch v/s no context switch (disregarding the network latency completely due to client side caching) - i.e just syscall delivery to the FS is more expensive in native glusterfs compared to NFS with no consideration of what each of them do after the syscall has been delivered. > > time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec > > (FUSE) 254 Kb/sec (NFS) > This test shows why glusterfs native protocol is better than NFS when you need to scale out storage. Even with a context switch overhead on the client side, glusterfs scores better due to the "clustered nature" of its protocol. NFS has to undergo a second hop when it has to fetch data not available in the server it has mounted from whereas for glusterfs it is always a single hop to any server it wants to get data from. In any case comparing to local disk performance and network disk performance is never right and is always misleading. Avati ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 01/15/2011 10:31 AM, Pan, Henry wrote: Hello Gluster Gurus, I'm trying to find out what performance data you could get while trying eDiscovery searching application in a namespace with over 3 billins small files on GlusterFS... Hmmm ... are you searching within the files or the file names for patterns? Gluster does have some pretty good caching facilities, with the downside being that we seem to catch lots of bugs in this layer. There are pre-fetch capabilities as well. The issue is that this is going to be a *very* seeky load. The underlying system IOP rate will be one of the bounding elements. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
Hello Gluster Gurus, I'm trying to find out what performance data you could get while trying eDiscovery searching application in a namespace with over 3 billins small files on GlusterFS... Thanks & Good w/e Henry PAN Sr. Data Storage Eng/Adm Iron Mountain 650-962-6184 (o) 650-930-6544 (c) henry@ironmountain.com -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of gluster-users-requ...@gluster.org Sent: Saturday, January 15, 2011 1:20 AM To: gluster-users@gluster.org Subject: Gluster-very bad performance on small files Send Gluster-users mailing list submissions to gluster-users@gluster.org To subscribe or unsubscribe via the World Wide Web, visit http://gluster.org/cgi-bin/mailman/listinfo/gluster-users or, via email, send a message with subject or body 'help' to gluster-users-requ...@gluster.org You can reach the person managing the list at gluster-users-ow...@gluster.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Gluster-users digest..." Today's Topics: 1. Re: very bad performance on small files (Marcus Bointon) 2. Re: very bad performance on small files (Joe Landman) 3. Re: very bad performance on small files (Max Ivanov) 4. Re: very bad performance on small files (Joe Landman) 5. Re: very bad performance on small files (Marcus Bointon) 6. Re: very bad performance on small files (Joe Landman) 7. Re: very bad performance on small files (Max Ivanov) 8. Re: very bad performance on small files (Rudi Ahlers) -- Message: 1 Date: Fri, 14 Jan 2011 22:50:37 +0100 From: Marcus Bointon Subject: Re: [Gluster-users] very bad performance on small files To: Gluster General Discussion List Message-ID: Content-Type: text/plain; charset=us-ascii On 14 Jan 2011, at 18:58, Jacob Shucart wrote: > This kind of thing is fine on local disks, but when you're talking about a > distributed filesystem the network latency starts to add up since 1 > request to the web server results in a bunch of file requests. I think the main objection is that it takes a huge amount of network latency to explain a > 1,500% overhead with only 2 machines. On 14 Jan 2011, at 15:20, Joe Landman wrote: > MB size or larger So does gluster become faster abruptly when file sizes cross some threshold? Or are average speeds are proportional to file size? Would be good to see a wider spread of values on benchmarks of throughput vs file size for the same overall volume (like Max's data but with more intermediate values) Marcus -- Message: 2 Date: Fri, 14 Jan 2011 17:12:01 -0500 From: Joe Landman Subject: Re: [Gluster-users] very bad performance on small files To: gluster-users@gluster.org Message-ID: <4d30ca31.9060...@scalableinformatics.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 01/14/2011 04:50 PM, Marcus Bointon wrote: > On 14 Jan 2011, at 18:58, Jacob Shucart wrote: > >> This kind of thing is fine on local disks, but when you're talking >> about a distributed filesystem the network latency starts to add up >> since 1 request to the web server results in a bunch of file >> requests. > > I think the main objection is that it takes a huge amount of network > latency to explain a> 1,500% overhead with only 2 machines. If most of your file access times are dominated by latency (e.g. small, seeky like loads), and you are going over a gigabit connection, yeah, your performance is going to crater on any cluster file system. Local latency to traverse the storage stack is on the order of 10's of microseconds. Physical latency of the disk medium is on the order of 10's of microseconds for RAMdisk, 100's of microseconds for flash/ssd, and 1000's of microseconds (e.g. milliseconds) for spinning rust. Now take 1 million small file writes. Say 1024 bytes. These million writes have to traverse the storage stack in the kernel to get to disk. Now add in a network latency event on the order of 1000's of microseconds for the remote storage stack and network stack to respond. I haven't measured it yet in a methodical manner, but I wouldn't be surprised to see IOP rates within a factor of 2 of the bare metal for a sufficiently fast network such as Infiniband, and within a factor of 4 or 5 for a slow network like Gigabit. Our own experience has been generally that you are IOP constrained because of the stack you have to traverse. If you add more latency into this stack, you have more to traverse, and therefore, you have more you need to wait. Which will have a magnification effect upon times for small IO ops which are seeky (stat, small writes, random ops). > > On 14 Jan 2011, at 15:2
Re: [Gluster-users] very bad performance on small files
On 01/15/2011 04:18 AM, Rudi Ahlers wrote: So what can one use for webhosting purposes? We use XEN / KVM virtual machines, hosted on NAS devices but the NAS devices doesn't have an easy upgrade path. We literally have to rsync all the data to the new device and then shutdown all the machines on the old one and restart them on the new one. They don't provide 100% uptime either. So I'm looking for something with easier upgrade (GlusterFS can do this) and better uptime (again, GlusterFS can do this). But it's clear that GlusterFS isn't made for small files, so what else could work well for us? XEN/KVM should be fine. Many small files should be locally cached if possible (not always possible though). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On Fri, Jan 14, 2011 at 7:58 PM, Jacob Shucart wrote: > For web hosting it is best to put user generated content(images, etc) on > Gluster but to leave application files like PHP files on the local disk. > This is because a single application file request could result in 20 other > file requests since applications like PHP use includes/inherits, etc. > This kind of thing is fine on local disks, but when you're talking about a > distributed filesystem the network latency starts to add up since 1 > request to the web server results in a bunch of file requests. > > -Original Message- > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Max Ivanov > Sent: Friday, January 14, 2011 6:09 AM > To: Burnash, James > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] very bad performance on small files > >> Gluster - and in fact most (all?) parallel filesystems are optimized for > very large files. That being the case, small files are not retrieved as > efficiently, and result in a larger number of file operations in total > because there are a fixed number for each file accessed. > > > Which makes glusterfs perfomance unacceptable for web hosting purposes =( > ___ So what can one use for webhosting purposes? We use XEN / KVM virtual machines, hosted on NAS devices but the NAS devices doesn't have an easy upgrade path. We literally have to rsync all the data to the new device and then shutdown all the machines on the old one and restart them on the new one. They don't provide 100% uptime either. So I'm looking for something with easier upgrade (GlusterFS can do this) and better uptime (again, GlusterFS can do this). But it's clear that GlusterFS isn't made for small files, so what else could work well for us? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
> Sure, and all that applies equally to both NFS and gluster, yet in Max's > example NFS was ~50x faster than gluster for an identical small-file > workload. So what's gluster doing over and above what NFS is doing that's > taking so long, given that network and disk factors are equal? I'd buy a > factor of 2 for replication, but not 50. > Sorry If I didnt make it clear but both NFS in my tests is not well known classic NFS but glusterfs in NFS mode. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 01/14/2011 06:26 PM, Marcus Bointon wrote: Our own experience has been generally that you are IOP constrained because of the stack you have to traverse. If you add more latency into this stack, you have more to traverse, and therefore, you have more you need to wait. Which will have a magnification effect upon times for small IO ops which are seeky (stat, small writes, random ops). Sure, and all that applies equally to both NFS and gluster, yet in Max's example NFS was ~50x faster than gluster for an identical small-file workload. So what's gluster doing over and above what NFS is doing that's taking so long, given that network and disk factors are equal? I'd buy a factor of 2 for replication, but not 50. If the NFS was doing attribute caching and the GlusterFS implementation had stat prefetch and other caching turned off, this could explain it. In case you missed what I'm on about, it was these stats that Max posted: Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS) dd if=/dev/zero of=M/tmp bs=1K count=16384 88.1 MB/sec (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv> /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS) Ok, I am not sure if I saw the numbers before. Thanks. In my case I'm running 30kiops SSDs over gigabit. At the moment my problem (running 3.0.6) isn't performance but reliability - files are occasionally reported as 'vanished' by front-end apps (like rsync) even though they are present on both backing stores; no errors in gluster logs, self-heal doesn't help. Check your stat-prefetch settings, and your time base. We've had some strange issues that seem to be correlated with time bases drifting. Including files disappearing. We have a few open tickets on this. The way we've worked around this problem is to abandon the NFS client and use the glusterfs client. Not our preferred option, but it provides a workaround for the moment. The NFS translator does appear to have a few issues. I am hoping we get more tuning knobs for it soon so we can see if we can work around this. Regards, Joe Marcus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 14 Jan 2011, at 23:12, Joe Landman wrote: > If most of your file access times are dominated by latency (e.g. small, seeky > like loads), and you are going over a gigabit connection, yeah, your > performance is going to crater on any cluster file system. > > Local latency to traverse the storage stack is on the order of 10's of > microseconds. Physical latency of the disk medium is on the order of 10's of > microseconds for RAMdisk, 100's of microseconds for flash/ssd, and 1000's of > microseconds (e.g. milliseconds) for spinning rust. > > Now take 1 million small file writes. Say 1024 bytes. These million writes > have to traverse the storage stack in the kernel to get to disk. > > Now add in a network latency event on the order of 1000's of microseconds for > the remote storage stack and network stack to respond. > > I haven't measured it yet in a methodical manner, but I wouldn't be surprised > to see IOP rates within a factor of 2 of the bare metal for a sufficiently > fast network such as Infiniband, and within a factor of 4 or 5 for a slow > network like Gigabit. > > Our own experience has been generally that you are IOP constrained because of > the stack you have to traverse. If you add more latency into this stack, you > have more to traverse, and therefore, you have more you need to wait. Which > will have a magnification effect upon times for small IO ops which are seeky > (stat, small writes, random ops). Sure, and all that applies equally to both NFS and gluster, yet in Max's example NFS was ~50x faster than gluster for an identical small-file workload. So what's gluster doing over and above what NFS is doing that's taking so long, given that network and disk factors are equal? I'd buy a factor of 2 for replication, but not 50. In case you missed what I'm on about, it was these stats that Max posted: > Here is the results per command: > dd if=/dev/zero of=M/tmp bs=1M count=16384 69.2 MB/se (Native) 69.2 > MB/sec(FUSE) 52 MB/sec (NFS) > dd if=/dev/zero of=M/tmp bs=1K count=16384 88.1 MB/sec (Native) > 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) > time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec > (FUSE) 254 Kb/sec (NFS) In my case I'm running 30kiops SSDs over gigabit. At the moment my problem (running 3.0.6) isn't performance but reliability - files are occasionally reported as 'vanished' by front-end apps (like rsync) even though they are present on both backing stores; no errors in gluster logs, self-heal doesn't help. Marcus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 01/14/2011 05:19 PM, Max Ivanov wrote: I haven't seen Max's data, so I can't comment on this. Understand that performance is going to be bound by many things. One of many things is the speed of the spinning disk if thats what you use. Another will be network. It is very similair to kernel source tree - tons of small (2-20kb) files. 1.1G in total. Ok, worth looking into ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
> I haven't seen Max's data, so I can't comment on this. Understand that > performance is going to be bound by many things. One of many things is the > speed of the spinning disk if thats what you use. Another will be network. > It is very similair to kernel source tree - tons of small (2-20kb) files. 1.1G in total. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 01/14/2011 04:50 PM, Marcus Bointon wrote: On 14 Jan 2011, at 18:58, Jacob Shucart wrote: This kind of thing is fine on local disks, but when you're talking about a distributed filesystem the network latency starts to add up since 1 request to the web server results in a bunch of file requests. I think the main objection is that it takes a huge amount of network latency to explain a> 1,500% overhead with only 2 machines. If most of your file access times are dominated by latency (e.g. small, seeky like loads), and you are going over a gigabit connection, yeah, your performance is going to crater on any cluster file system. Local latency to traverse the storage stack is on the order of 10's of microseconds. Physical latency of the disk medium is on the order of 10's of microseconds for RAMdisk, 100's of microseconds for flash/ssd, and 1000's of microseconds (e.g. milliseconds) for spinning rust. Now take 1 million small file writes. Say 1024 bytes. These million writes have to traverse the storage stack in the kernel to get to disk. Now add in a network latency event on the order of 1000's of microseconds for the remote storage stack and network stack to respond. I haven't measured it yet in a methodical manner, but I wouldn't be surprised to see IOP rates within a factor of 2 of the bare metal for a sufficiently fast network such as Infiniband, and within a factor of 4 or 5 for a slow network like Gigabit. Our own experience has been generally that you are IOP constrained because of the stack you have to traverse. If you add more latency into this stack, you have more to traverse, and therefore, you have more you need to wait. Which will have a magnification effect upon times for small IO ops which are seeky (stat, small writes, random ops). On 14 Jan 2011, at 15:20, Joe Landman wrote: MB size or larger So does gluster become faster abruptly when file sizes cross some threshold? Or are average speeds are proportional to file size? Would Its a continuous curve, and very much user load specific. The fewer seeky operations you can do the better (true of all cluster file systems). be good to see a wider spread of values on benchmarks of throughput vs file size for the same overall volume (like Max's data but with more intermediate values) I haven't seen Max's data, so I can't comment on this. Understand that performance is going to be bound by many things. One of many things is the speed of the spinning disk if thats what you use. Another will be network. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 14 Jan 2011, at 18:58, Jacob Shucart wrote: > This kind of thing is fine on local disks, but when you're talking about a > distributed filesystem the network latency starts to add up since 1 > request to the web server results in a bunch of file requests. I think the main objection is that it takes a huge amount of network latency to explain a > 1,500% overhead with only 2 machines. On 14 Jan 2011, at 15:20, Joe Landman wrote: > MB size or larger So does gluster become faster abruptly when file sizes cross some threshold? Or are average speeds are proportional to file size? Would be good to see a wider spread of values on benchmarks of throughput vs file size for the same overall volume (like Max's data but with more intermediate values) Marcus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
Hi, This can be offset by using a PHP bytecode caching system like Xcache or APC. Just make sure you have enough memory allocated. In this way you can have the benefit of a distributed filesystem for your application files while also having them accessed quickly (except for the 1st access). Having said that, I only went this way because we have 80GB worth of PHP files, if it were the standard few MBs then I'd keep them locally on the application server! Rafiq On Fri, Jan 14, 2011 at 5:58 PM, Jacob Shucart wrote: > For web hosting it is best to put user generated content(images, etc) on > Gluster but to leave application files like PHP files on the local disk. > This is because a single application file request could result in 20 other > file requests since applications like PHP use includes/inherits, etc. > This kind of thing is fine on local disks, but when you're talking about a > distributed filesystem the network latency starts to add up since 1 > request to the web server results in a bunch of file requests. > > -Original Message- > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Max Ivanov > Sent: Friday, January 14, 2011 6:09 AM > To: Burnash, James > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] very bad performance on small files > > > Gluster - and in fact most (all?) parallel filesystems are optimized for > very large files. That being the case, small files are not retrieved as > efficiently, and result in a larger number of file operations in total > because there are a fixed number for each file accessed. > > > Which makes glusterfs perfomance unacceptable for web hosting purposes =( > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 01/14/2011 12:58 PM, Jacob Shucart wrote: For web hosting it is best to put user generated content(images, etc) on Gluster but to leave application files like PHP files on the local disk. This is because a single application file request could result in 20 other file requests since applications like PHP use includes/inherits, etc. This kind of thing is fine on local disks, but when you're talking about a distributed filesystem the network latency starts to add up since 1 request to the web server results in a bunch of file requests. I'd probably go a little further than this ... place the originals on gluster, and on apache/other startup, copy the files to a local (RAM)disk, and work from there. You can even set up scripted push of originals with some simple scripting so that on (RAM)disk versions are kept in sync within some interval. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
For web hosting it is best to put user generated content(images, etc) on Gluster but to leave application files like PHP files on the local disk. This is because a single application file request could result in 20 other file requests since applications like PHP use includes/inherits, etc. This kind of thing is fine on local disks, but when you're talking about a distributed filesystem the network latency starts to add up since 1 request to the web server results in a bunch of file requests. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Max Ivanov Sent: Friday, January 14, 2011 6:09 AM To: Burnash, James Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] very bad performance on small files > Gluster - and in fact most (all?) parallel filesystems are optimized for very large files. That being the case, small files are not retrieved as efficiently, and result in a larger number of file operations in total because there are a fixed number for each file accessed. Which makes glusterfs perfomance unacceptable for web hosting purposes =( ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
On 01/14/2011 09:20 AM, Andrew Latham wrote: We are using it for image based storage with virtualization. What is the definition of large files for Gluster, that is a vague statement. MB size or larger -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
2011/1/14 Łukasz Jagiełło : > 2011/1/14 Max Ivanov : >>> Gluster - and in fact most (all?) parallel filesystems are optimized for >>> very large files. That being the case, small files are not retrieved as >>> efficiently, and result in a larger number of file operations in total >>> because there are a fixed number for each file accessed. >> >> >> Which makes glusterfs perfomance unacceptable for web hosting purposes =( > > Everything depends how you want to use it for web hosting. If it's > just storage for static content and all requests going over some > reverse proxy cache (ex. varnish) should be ok. > > -- > Łukasz Jagiełło > lukaszjagielloorg We are using it for image based storage with virtualization. What is the definition of large files for Gluster, that is a vague statement. ~~~ Andrew "lathama" Latham lath...@gmail.com ~~~ ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
2011/1/14 Max Ivanov : >> Gluster - and in fact most (all?) parallel filesystems are optimized for >> very large files. That being the case, small files are not retrieved as >> efficiently, and result in a larger number of file operations in total >> because there are a fixed number for each file accessed. > > > Which makes glusterfs perfomance unacceptable for web hosting purposes =( Everything depends how you want to use it for web hosting. If it's just storage for static content and all requests going over some reverse proxy cache (ex. varnish) should be ok. -- Łukasz Jagiełło lukaszjagielloorg ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
> Gluster - and in fact most (all?) parallel filesystems are optimized for very > large files. That being the case, small files are not retrieved as > efficiently, and result in a larger number of file operations in total > because there are a fixed number for each file accessed. Which makes glusterfs perfomance unacceptable for web hosting purposes =( ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
Hello. Gluster - and in fact most (all?) parallel filesystems are optimized for very large files. That being the case, small files are not retrieved as efficiently, and result in a larger number of file operations in total because there are a fixed number for each file accessed. While the Gluster Development team has put forth a major effort to remedy this as much as possible (using translators), small files simply are not what a filesystem like this was designed for. I have seen people reporting reasonable success in dealing with this issue on this mailing list - I encourage you to carefully search through it and see what they have done. "When all you have is a hammer, everything looks like a nail" :-) James Burnash, Unix Engineering -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of ???(yongjoon kong)/Cloud Computing Sent: Thursday, January 13, 2011 11:54 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] very bad performance on small files Maybe it's related with make the Metadata. And I have the same issue here. When I untar the kernel source( which has 34000 files) it took over 15 minute on replicated gluster but not more than 1 minute on local filesystem. Maybe this is because when gluster create metadata using DHT,it should contact brick to find the leftover space using 'du' Maybe it is to be a problem. But I don't know how to improve the performance Best Regards, Andrew Cloud Computing Business Team Andrew Kong Manager | andrew.k...@sk.com | T : +82-2-6400-4328 | M : +82-010-8776-5025 SK u-Tower, 25-1, Jeongja-dong, Bundang-gu, Seongnam-si, Gyeonggi-do, 463-844, Korea -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Max Ivanov Sent: Friday, January 14, 2011 7:32 AM To: gluster-users Subject: [Gluster-users] very bad performance on small files Hi! I've deployed glusterfs on 2 nodes using replication mode and done some tests. Peroformance drop was significant and I've no idea if its possible to improve it or not. Volume was mounted on one of nodes using both FUSE and NFS glusterfs clients. 1.1G of small files are stored on volume. To make command shorter M symbol is used as mountpoint label. Native performance is performance of command issued over native FS on one of the bricks. No other activities on both node happened. Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS) dd if=/dev/zero of=M/tmp bs=1K count=16384 88.1 MB/sec (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS) I use default configuration, no adjustments to /etc/gluster* where made. I use ext4 on bricks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] very bad performance on small files
Maybe it's related with make the Metadata. And I have the same issue here. When I untar the kernel source( which has 34000 files) it took over 15 minute on replicated gluster but not more than 1 minute on local filesystem. Maybe this is because when gluster create metadata using DHT,it should contact brick to find the leftover space using 'du' Maybe it is to be a problem. But I don't know how to improve the performance Best Regards, Andrew Cloud Computing Business Team Andrew Kong Manager | andrew.k...@sk.com | T : +82-2-6400-4328 | M : +82-010-8776-5025 SK u-Tower, 25-1, Jeongja-dong, Bundang-gu, Seongnam-si, Gyeonggi-do, 463-844, Korea -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Max Ivanov Sent: Friday, January 14, 2011 7:32 AM To: gluster-users Subject: [Gluster-users] very bad performance on small files Hi! I've deployed glusterfs on 2 nodes using replication mode and done some tests. Peroformance drop was significant and I've no idea if its possible to improve it or not. Volume was mounted on one of nodes using both FUSE and NFS glusterfs clients. 1.1G of small files are stored on volume. To make command shorter M symbol is used as mountpoint label. Native performance is performance of command issued over native FS on one of the bricks. No other activities on both node happened. Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS) dd if=/dev/zero of=M/tmp bs=1K count=16384 88.1 MB/sec (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS) I use default configuration, no adjustments to /etc/gluster* where made. I use ext4 on bricks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] very bad performance on small files
Hi! I've deployed glusterfs on 2 nodes using replication mode and done some tests. Peroformance drop was significant and I've no idea if its possible to improve it or not. Volume was mounted on one of nodes using both FUSE and NFS glusterfs clients. 1.1G of small files are stored on volume. To make command shorter M symbol is used as mountpoint label. Native performance is performance of command issued over native FS on one of the bricks. No other activities on both node happened. Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS) dd if=/dev/zero of=M/tmp bs=1K count=16384 88.1 MB/sec (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv > /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS) I use default configuration, no adjustments to /etc/gluster* where made. I use ext4 on bricks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
