[zfs-discuss] current status of SAM-QFS?
The subject says it all. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [developer] Setting default user/group quotas[usage accounting]?
On Apr 26, 2012, at 12:27 AM, Fred Liu wrote: zfs 'userused@' properties and 'zfs userspace' command are good enough to gather usage statistics. I think I mix that with NetApp. If my memory is correct, we have to set quotas to get usage statistics under DataOnTAP. Further, if we can add an ILM-like feature to poll the time-related info(atime,mtime,ctime,etc) with that statistic from ZFS, that will be really cool. In general, file-based ILM has limitations that cause all sorts of issues for things like operating systems, where files might only be needed infrequently, but when they are needed, they are needed right now Have you looked at zfs diff for changed files? Here ILM-like feature, I mean we know how the data is distributed by time per pool/filesystem like how many data are modified/accessed before mm/dd/. And we don't need to do the actual storage-tying operations immediately(moving the infrequently-used data to tie-2 storage). The time-related usage statistics are very useful reference for us. zfs diff will show the delta but not come with the time info. Since no one is focusing on enabling default user/group quota now, the temporarily remedy could be a script which traverse all the users/groups in the directory tree. Tough it is not so decent. The largest market for user/group quotas is .edu. But they represent only a small market when measured by $. There are also many corner cases in this problem space. One might pine for the days of VMS and its file resource management features, those features don't scale well to company- wide LDAP and thousands of file systems. My understanding is the quota management is needed as long as zfs storage is used in NAS way(shared by multi-users). So, for now, the fastest method to solve the problem might be to script some walkers. Yes. That is ture. Thanks. Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] IOzone benchmarking
I'm trying to run some IOzone benchmarking on a new system to get a feel for baseline performance. Unfortunately, the system has a lot of memory (144GB), but I have some time so am approaching my runs as follows: Throughput: iozone -m -t 8 -T -r 128k -o -s 36G -R -b bigfile.xls IOPS: iozone -O -i 0 -i 1 -i 2 -e -+n -r 128K -s 288G iops.txt Not sure what I gain/lose by using threads or not. Am I off on this? System is a 240x2TB (7200RPM) system in 20 Dell MD1200 JBODs. 16 vdevs of 15 disks each -- RAIDZ3. NexentaStor 3.1.2. Ray ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] current status of SAM-QFS?
On 04/30/12 04:00, Fred Liu wrote: The subject says it all. Still a fully supported product from Oracle: http://www.oracle.com/us/products/servers-storage/storage/storage-software/qfs-software/overview/index.html -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [developer] Setting default user/group quotas[usage accounting]?
On Apr 29, 2012, at 7:59 PM, Fred Liu wrote: On Apr 26, 2012, at 12:27 AM, Fred Liu wrote: “zfs 'userused@' properties” and “'zfs userspace' command” are good enough to gather usage statistics. I think I mix that with NetApp. If my memory is correct, we have to set quotas to get usage statistics under DataOnTAP. Further, if we can add an ILM-like feature to poll the time-related info(atime,mtime,ctime,etc) with that statistic from ZFS, that will be really cool. In general, file-based ILM has limitations that cause all sorts of issues for things like operating systems, where files might only be needed infrequently, but when they are needed, they are needed right now Have you looked at zfs diff for changed files? Here “ILM-like feature”, I mean we know how the data is distributed by time per pool/filesystem like how many data are modified/accessed before mm/dd/. And we don’t need to do the actual storage-tying operations immediately(moving the infrequently-used data to tie-2 storage). The time-related usage statistics are very useful reference for us. “zfs diff” will show the delta but not come with the time info. The time is the creation time of the snapshots. -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cluster vs nfs
Instead we've switched to Linux and DRBD. And if that doesn't get me sympathy I don't know what will. SvSAN does something similar and it does it rather well, I think. http://www.stormagic.com/SvSAN.php ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On May 1, 2012, at 1:41 AM, Ray Van Dolson wrote: Throughput: iozone -m -t 8 -T -r 128k -o -s 36G -R -b bigfile.xls IOPS: iozone -O -i 0 -i 1 -i 2 -e -+n -r 128K -s 288G iops.txt Do you expect to be reading or writing 36 or 288Gb files very often on this array? The largest file size I've used in my still lengthy benchmarks was 16Gb. If you use the sizes you've proposed, it could take several days or weeks to complete. Try a web search for iozone examples if you want more details on the command switches. -Gary ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [developer] Setting default user/group quotas[usage accounting]?
2012/4/25 Richard Elling richard.ell...@gmail.com: On Apr 25, 2012, at 8:14 AM, Eric Schrock wrote: ZFS will always track per-user usage information even in the absence of quotas. See the the zfs 'userused@' properties and 'zfs userspace' command. tip: zfs get -H -o value -p userused@username filesystem Yes, and this is the logical size, not physical size. Some ZFS features increase logical size (copies) while others decrease physical size (compression, dedup) The size accounted for by the userused@ and groupused@ properties is the referenced space, which is used as the basis for many other space accounting values in ZFS (e.g. du / ls -s / stat(2), and the zfs accounting properties referenced, refquota, refreservation, refcompressratio, written). It includes changes local to the dataset (compression, the copies property, file-specific metadata such as indirect blocks), but ignores pool-wide or cross-dataset changes (space shared between a clone and its origin, mirroring, raid-z, dedup[*]). --matt [*] Although dedup can be turned on and off per-dataset, the data is deduplicated against all dedup-enabled data in the pool. Ie, identical data in different datasets will be stored only once, if dedup is enabled for both datasets. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On Mon, Apr 30, 2012 at 4:15 PM, Ray Van Dolson rvandol...@esri.com wrote:
I'm trying to run some IOzone benchmarking on a new system to get a
feel for baseline performance.
If you have compression turned on (and I highly recommend turning
it on if you have the CPU power to handle it), the IOzone data will be
flawed. I did not look deeper into it, but the data that IOzone uses
compresses very, very well. Much more so than any real data out there.
I used a combination of Filebench and Oracle's Orion to test ZFS
performance. Recently I started writing my own utilities for testing,
as _none_ of the existing offerings tested what I needed (lots and
lots of small, less than 64KB, files). My tool is only OK for relative
measures.
Unfortunately, the system has a lot of memory (144GB), but I have some
time so am approaching my runs as follows:
When I was testing systems with more RAM than I wanted (when does
that ever happen :-), I called the ARC to something rational (2GB, 4GB
etc) and ran the tests with file sizes four times the ARC limit.
Unfortunately, the siwiki site appears to be down (gone ???).
On Solaris 10, the following in /etc/system (and a reboot) will cap
the zfa arc to the amount of RAM specified (in bytes). Not sure on
Nextena (and I have not had to cap the arc on my Nexenta Core system
at home).
set zfs:zfs_arc_max = 4294967296
Throughput:
iozone -m -t 8 -T -r 128k -o -s 36G -R -b bigfile.xls
IOPS:
iozone -O -i 0 -i 1 -i 2 -e -+n -r 128K -s 288G iops.txt
Not sure what I gain/lose by using threads or not.
IOzone without threads is single threaded and will demonstrate the
performance a single user or application will achieve. When you use
threads in IOzone you see performance for N simultaneous users (or
applications). In my experience, the knee in the performance vs. # of
threads curve happens somewhere between one and two times the number
of CPUs in the system. In other words, with a 16 CPU system,
performance scales linearly as the number of threads increases until
you get to somewhere between 16 and 32. At that point the performance
will start flattening out and eventually _decreases_ as you add more
threads.
Using multiple threads (or processes or clients or etc.) is a
good way to measure how many simultaneous users your system can handle
(at a certain performance level).
Am I off on this?
System is a 240x2TB (7200RPM) system in 20 Dell MD1200 JBODs. 16 vdevs of 15
disks each -- RAIDZ3. NexentaStor 3.1.2.
--
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Assistant Technical Director, LoneStarCon 3 (http://lonestarcon3.org/)
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, Troy Civic Theatre Company
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On Mon, 30 Apr 2012, Ray Van Dolson wrote: I'm trying to run some IOzone benchmarking on a new system to get a feel for baseline performance. Unfortunately, benchmarking with IOzone is a very poor indicator of what performance will be like during normal use. Forcing the system to behave like it is short on memory only tests how the system will behave when it is short on memory. Testing multi-threaded synchronous writes with IOzone might actually mean something if it is representative of your work-load. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [developer] current status of SAM-QFS?
5.3 is released, not much of an idea about the 5.4 schedule. On Monday 30 April 2012 08:30 AM, Fred Liu wrote: The subject says it all. --- illumos-developer Archives: https://www.listbox.com/member/archive/182179/=now RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175038-3914b2ca Modify Your Subscription: https://www.listbox.com/member/?member_id=21175038id_secret=21175038-6fd90acc Powered by Listbox: http://www.listbox.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Subscribe
Subscribe -- ORACLE James Cypcar | Solaris and Network Domain, Global Systems Support Oracle Global Customer Services Log, update, and monitor your Service Request online usinghttps://support.oracle.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On Tue, May 01, 2012 at 03:21:05AM -0700, Gary Driggs wrote: On May 1, 2012, at 1:41 AM, Ray Van Dolson wrote: Throughput: iozone -m -t 8 -T -r 128k -o -s 36G -R -b bigfile.xls IOPS: iozone -O -i 0 -i 1 -i 2 -e -+n -r 128K -s 288G iops.txt Do you expect to be reading or writing 36 or 288Gb files very often on this array? The largest file size I've used in my still lengthy benchmarks was 16Gb. If you use the sizes you've proposed, it could take several days or weeks to complete. Try a web search for iozone examples if you want more details on the command switches. -Gary The problem is this box has 144GB of memory. If I go with a 16GB file size (which I did), then memory and caching influences the results pretty severely (I get around 3GB/sec for writes!). Obviously, I could yank RAM for purposes of benchmarking. :) Thanks, Ray ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On Tue, May 01, 2012 at 07:18:18AM -0700, Bob Friesenhahn wrote: On Mon, 30 Apr 2012, Ray Van Dolson wrote: I'm trying to run some IOzone benchmarking on a new system to get a feel for baseline performance. Unfortunately, benchmarking with IOzone is a very poor indicator of what performance will be like during normal use. Forcing the system to behave like it is short on memory only tests how the system will behave when it is short on memory. Testing multi-threaded synchronous writes with IOzone might actually mean something if it is representative of your work-load. Bob Sounds like IOzone may not be my best option here (though it does produce pretty graphs). bonnie++ actually gave me more realistic sounding numbers, and I've been reading good thigns about fio. Ray ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On 5/1/12, Ray Van Dolson wrote: The problem is this box has 144GB of memory. If I go with a 16GB file size (which I did), then memory and caching influences the results pretty severely (I get around 3GB/sec for writes!). The idea of benchmarking -- IMHO -- is to vaguely attempt to reproduce real world loads. Obviously, this is an imperfect science but if you're going to be writing a lot of small files (e.g. NNTP or email servers used to be a good real world example) then you're going to want to benchmark for that. If you're going to want to write a bunch of huge files (are you writing a lot of 16GB files?) then you'll want to test for that. Caching anywhere in the pipeline is important for benchmarks because you aren't going to turn off a cache or remove RAM in production are you? -Gary ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On Tue, May 1, 2012 at 1:45 PM, Gary gdri...@gmail.com wrote:
The idea of benchmarking -- IMHO -- is to vaguely attempt to reproduce
real world loads. Obviously, this is an imperfect science but if
you're going to be writing a lot of small files (e.g. NNTP or email
servers used to be a good real world example) then you're going to
want to benchmark for that. If you're going to want to write a bunch
of huge files (are you writing a lot of 16GB files?) then you'll want
to test for that. Caching anywhere in the pipeline is important for
benchmarks because you aren't going to turn off a cache or remove RAM
in production are you?
It also depends on what you are going to be tuning. When I needed
to decided on a zpool configuration (# of vdev's, type of vdev, etc.)
I did not want the effect of the cache hiding the underlying
performance limitations of the physical drive configuration. In that
case I either needed to use a very large test data set or reduce the
size (effect) of the RAM. By limiting the ARC to 2 GB for my test, I
was able to relatively easily quantify the performance differences
between the various configurations. Once we picked a configuration, we
let the ARC take as much RAM as it wanted and re-ran the benchmark to
see what kind of real world performance we would get. Unfortunately,
we could not easily simulate 400 real world people sitting at desktops
accessing the data. So our ARC limited benchmark was effectively a
worst case number and the full ARC the best case. The real world,
as usual, fell somewhere in between.
Finding a benchmark tool that matches _my_ work load is why I have
started kludging together my own.
--
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
On Tue, 1 May 2012, Ray Van Dolson wrote: Testing multi-threaded synchronous writes with IOzone might actually mean something if it is representative of your work-load. Sounds like IOzone may not be my best option here (though it does produce pretty graphs). bonnie++ actually gave me more realistic sounding numbers, and I've been reading good thigns about fio. None of these benchmarks is really useful other than to stress-test your hardware. Assuming that the hardware is working properly, when you intentionally break the cache, IOzone should produce numbers similar to what you could have estimated from hardware specification sheets and an understanding of the algorithms. Sun engineers used 'filebench' to do most of their performance testing because it allowed configuring the behavior to emulate various usage models. You can get it from https://sourceforge.net/projects/filebench/;. Zfs is all about caching so the cache really does need to be included (and not intentionally broken) in any realistic measurement of how the system will behave. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] IOzone benchmarking
more comments... On May 1, 2012, at 10:41 AM, Ray Van Dolson wrote: On Tue, May 01, 2012 at 07:18:18AM -0700, Bob Friesenhahn wrote: On Mon, 30 Apr 2012, Ray Van Dolson wrote: I'm trying to run some IOzone benchmarking on a new system to get a feel for baseline performance. Unfortunately, benchmarking with IOzone is a very poor indicator of what performance will be like during normal use. Forcing the system to behave like it is short on memory only tests how the system will behave when it is short on memory. Testing multi-threaded synchronous writes with IOzone might actually mean something if it is representative of your work-load. Bob Sounds like IOzone may not be my best option here (though it does produce pretty graphs). For performance analysis of ZFS systems, you need to consider the advantages of the hybrid storage pool. I wrote a white paper last summer describing a model that you can use with your performance measurements or data from vendor datasheets. http://info.nexenta.com/rs/nexenta/images/tech_brief_nexenta_performance.pdf And in presentation form, http://www.slideshare.net/relling/nexentastor-performance-tuning-openstorage-summit-2011 Recently, this model has been expanded and enhanced. Contact me offline, if you are interested. I have used IOzone, filebench, and vdbench for a lot of performance characterization lately. Each has their own strength, but all can build a full characterization profile of a system. For IOzone, I like to run a full characterization run, which precludes multithreaded runs, for a spectrum of I/O sizes and WSS. Such info can be useful to explore the boundaries of your system's performance and compare to other systems. Also, for systems with 50GB of RAM, there are some tunables needed for good scaling under heavy write load workloads. Alas, there is no perfect answer and no single tunable setting works optimally for all cases. WIP. YMMV. A single, summary metric is not very useful... bonnie++ actually gave me more realistic sounding numbers, and I've been reading good thigns about fio. IMNSHO, bonnie++ is a totally useless benchmark. Roch disected it rather nicely at https://bigip-blogs-cms-adc.oracle.com/roch/entry/decoding_bonnie [gag me! Does Oracle have butugly URLs or what? ;-)] -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
