Re: [OpenAFS] Tuning openafs write speed
memcache is much faster than the disk cache. memcache will not get any better if no one ever uses it so the openafs developers can get some bug reports. When we got out last system, the group at PDC decided that non-swappable/pageable memory was a too expensive resource to be used for file cache. If calc users buy a machine with 32GB of RAM they expect that _at_least_ 32GB of RAM can be used for caculations. Yeah, they are that way ;-). So for example taking 3GB (10%) is not acceptable if it can not be swapped out. If the memcache would be swappable (I'm talking about Linux, because calc users use Linux), that would be a completely other matter. Harald. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
[OpenAFS] Tuning openafs write speed
Hi folks! I would like to try tuning the speed of my openafs installation, but the only information I could google is this rather old thread (http://www.openafs.org/pipermail/openafs-info/2003-June/009753.html) and the hint to use a big cache-partition. For comparison I've created files with random data and different size (1MB, 2MB, 4MB, 8MB, 16MB, 32MB, 64MB and 128MB) on my local disk. I copied them into AFS and then I copied then to the same disk on the same host via scp (without compression). I've done that 10 times and computed the average. For the 1MB file AFS ist slightly faster then scp (factor 0,89). For the 2 and the 4MB file AFS needs about 1,4 of the time scp needs. For the 8, 16 and 32MB the factor is about 2,7 and for the 64 and the 128MB file it is about 3,3. I've already tried bigger cache-partitions, but it does not make a difference. Are there tuning parameters, which tell the system a threshold for the size of files, beyond which data won't be written to the cache? Greetings Kai Moritz -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
Kai Moritz wrote: Hi folks! I would like to try tuning the speed of my openafs installation, but the only information I could google is this rather old thread (http://www.openafs.org/pipermail/openafs-info/2003-June/009753.html) and the hint to use a big cache-partition. For comparison I've created files with random data and different size (1MB, 2MB, 4MB, 8MB, 16MB, 32MB, 64MB and 128MB) on my local disk. I copied them into AFS and then I copied then to the same disk on the same host via scp (without compression). I've done that 10 times and computed the average. For the 1MB file AFS ist slightly faster then scp (factor 0,89). For the 2 and the 4MB file AFS needs about 1,4 of the time scp needs. For the 8, 16 and 32MB the factor is about 2,7 and for the 64 and the 128MB file it is about 3,3. I've already tried bigger cache-partitions, but it does not make a difference. Are there tuning parameters, which tell the system a threshold for the size of files, beyond which data won't be written to the cache? Greetings Kai Moritz What are your data rates in MB/s? If you are on a fast network (Gbit Ethernet, Inifiband ...) a disk cache may be remarkably slower than the network. In this case memory cache can help. Another point is chunk size. The default (64 KB) is bad for reading where each chunk is fetched in a separate RPC. with disk cache bigger chunks (1 MB) can be recommanded, anyway. For memory cache of, say, 64 MB you would limit the number of chunks to only 64 which is certainly too low. Here ramdisks can help because many of the chunks are filled with short contents, such as directories and symbolic links. The additional overhead to go through the filesystem layer may be less than what you can earn from bigger chunks. With ramdisk 1 MB chunks aren't too bad. Hartmut -- - Hartmut Reuter e-mail [EMAIL PROTECTED] phone +49-89-3299-1328 RZG (Rechenzentrum Garching) fax +49-89-3299-1301 Computing Center of the Max-Planck-Gesellschaft (MPG) and the Institut fuer Plasmaphysik (IPP) - ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
What are your data rates in MB/s? scp says: 4.6MB/s If you are on a fast network (Gbit Ethernet, Inifiband ...) a disk cache may be remarkably slower than the network. In this case memory cache can help. I haven't tried that yet, becaus in the file /etc/openafs/afs.conf of my Debian Etch installation there is a comment that says: # Using the memory cache is not recommended. It's less stable than the disk # cache and doesn't improve performance as much as it might sound. Another point is chunk size. The default (64 KB) is bad for reading where each chunk is fetched in a separate RPC. with disk cache bigger chunks (1 MB) can be recommanded, anyway. For memory cache of, say, 64 MB you would limit the number of chunks to only 64 which is certainly too low. With automatically choosen values writing a 128 MB file in AFS takes about 44-45 seconds. On that machine I have a 3 GB cache. With the following options, which a have taken from an example in a Debian Configfile, writing the 128 MB file takes about 48 seconds :( -chunksize 20 -files 8 -dcache 1 -stat 15000 -daemons 6 -volume s 500 -rmtsys Greetings kai -- Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten Browser-Versionen downloaden: http://www.gmx.net/de/go/browser ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
In message [EMAIL PROTECTED],Kai Moritz writes: I haven't tried that yet, becaus in the file /etc/openafs/afs.conf of my Debian Etch installation there is a comment that says: # Using the memory cache is not recommended. It's less stable than the disk # cache and doesn't improve performance as much as it might sound. memcache is much faster than the disk cache. memcache will not get any better if no one ever uses it so the openafs developers can get some bug reports. i think memcache has improved quite a bit (but it could be better, i need to submit some patches) over the last couple years. i use '-memcache -chunksize 15 -dcache 1024'. if your system is memory starved this might be an issue. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
memcache is much faster than the disk cache. memcache will not get any better if no one ever uses it so the openafs developers can get some bug reports. i think memcache has improved quite a bit (but it could be better, i need to submit some patches) over the last couple years. i use '-memcache -chunksize 15 -dcache 1024'. if your system is memory starved this might be an issue. I did a whole bunch of testing regarding cache performances while we've been moving all of our users off of AFS-hosted mailspools, and here's what I've found -- this is on Sol 10 x86... * slowest: disk cache, of course. * medium: memory cache * fastest: ufs filesystem on a lofi-mounted block device hosted in / tmp (which is in-RAM) (I know this certainly wastes some cpu/memory resources and overhead, but... it works) ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
Kai Moritz wrote: What are your data rates in MB/s? scp says: 4.6MB/s Isn't great either. So may be you have some other problems in your network? When I do a scp of a 100 MB file to my laptop I get ~ 8 MB/s and there is in parallel running a remote rsync with about another .7 MB/s in both directions (rsyncd and AFS). So I normally get the full 100 Mbit/s bandwidth when I write into AFS or read from AFS: = 10 MB/s. If you are on a fast network (Gbit Ethernet, Inifiband ...) a disk cache may be remarkably slower than the network. In this case memory cache can help. I haven't tried that yet, becaus in the file /etc/openafs/afs.conf of my Debian Etch installation there is a comment that says: # Using the memory cache is not recommended. It's less stable than the disk # cache and doesn't improve performance as much as it might sound. We are using here in our Linux clusters and on the high performance AIX power 4/5 machines memcache without problems. It's my special OpenAFS-1.4.4 with OSD support which is expected to arrive soonly in the OpenAFS CVS. But I suppose also the normal OpenAFS-1.4.4 should work without problems with memcache Hartmut Another point is chunk size. The default (64 KB) is bad for reading where each chunk is fetched in a separate RPC. with disk cache bigger chunks (1 MB) can be recommanded, anyway. For memory cache of, say, 64 MB you would limit the number of chunks to only 64 which is certainly too low. With automatically choosen values writing a 128 MB file in AFS takes about 44-45 seconds. On that machine I have a 3 GB cache. With the following options, which a have taken from an example in a Debian Configfile, writing the 128 MB file takes about 48 seconds :( -chunksize 20 -files 8 -dcache 1 -stat 15000 -daemons 6 -volume s 500 -rmtsys Greetings kai -- - Hartmut Reuter e-mail [EMAIL PROTECTED] phone +49-89-3299-1328 RZG (Rechenzentrum Garching) fax +49-89-3299-1301 Computing Center of the Max-Planck-Gesellschaft (MPG) and the Institut fuer Plasmaphysik (IPP) - ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
memcache is much faster than the disk cache. memcache will not get any better if no one ever uses it so the openafs developers can get some bug reports. That's true, but I cannot annoy my users with starving machines... Hence, I can only run that on test-machines. Greetings kai -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
* slowest: disk cache, of course. * medium: memory cache * fastest: ufs filesystem on a lofi-mounted block device hosted in / tmp (which is in-RAM) (I know this certainly wastes some cpu/memory resources and overhead, but... it works) That sound intresting! I will give a ramdisk a try on some test-machines and report... Greetings Kai -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
chas williams - CONTRACTOR [EMAIL PROTECTED] writes: In message [EMAIL PROTECTED],Kai Moritz writes: I haven't tried that yet, becaus in the file /etc/openafs/afs.conf of my Debian Etch installation there is a comment that says: # Using the memory cache is not recommended. It's less stable than the disk # cache and doesn't improve performance as much as it might sound. memcache is much faster than the disk cache. memcache will not get any better if no one ever uses it so the openafs developers can get some bug reports. i think memcache has improved quite a bit (but it could be better, i need to submit some patches) over the last couple years. Sounds like that comment is obsolete. I'll drop it from the Debian packages. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Tuning openafs write speed
On Aug 23, 2007, at 10:49, Kai Moritz wrote: * slowest: disk cache, of course. * medium: memory cache * fastest: ufs filesystem on a lofi-mounted block device hosted in / tmp (which is in-RAM) (I know this certainly wastes some cpu/memory resources and overhead, but... it works) That sound intresting! I will give a ramdisk a try on some test-machines and report... Make sure you do it with a real filesystem. The AFS cache stuff won't work on top of most 'tmpfs' filesystems, hense the ufs- filesystem on the block device... ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
