[zfs-discuss] Re: ZFS + rsync, backup on steroids.
Are both of you doing a umount/mount (or export/import, I guess) of the source filesystem before both first and second test? Otherwise, there might still be a fair bit of cached data left over from the first test, which would give the 2nd an unfair advantage. I'm fairly sure unmounting a filesystem invalidates all cached pages associated with files on that filesystem, as well as any cached [iv]node entries, all of which in needed to ensure both tests are starting from the most similar situation possible. Ideally, all this would even be done in single-user mode, so that nothing else could interfere. If there were a list of precautions to take that would put comparisons like this on firmer ground, it might provide a good starting point for such comparisons to be more than anecdotes, saving time for all concerned, both those attempting to replicate a prior casual observation for reporting, and those looking at the report. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: zfs and vmware
Yes I did and it works ok enough for me. Hardware: Sempron 1600Mhz 1.5GB Ram 2x80GB sata mirror for the Debian host OS. Currently I run vmware server 1.0 with 3 OpenBSD guests and one Solaris guest. The Solaris guest has physical access to 3x160 pata and 3x320 sata drives set up in two storage pools (didn't want to mix sata and pata in the same pool). I only use this for my home network so I don't know how well it works in production when you put alot of load on it. I'm also not sure if the added vmware layer takes away some of the advantages with zfs. Would be nice to have vmware server for Solaris instead so I can run Solaris as the host and use zfs directly on the controllers. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: Re[2]: [zfs-discuss] ZFS + rsync, backup on steroids.
> > Why not make a snapshots on a production and then send incremental > > backups over net? Especially with a lot of files it should be MUCH > > faster than rsync. > > > because its a ZFS limited solution, if the source is not ZFS it won't > work, and i'm not sure how much faster incrementals would be than > rsysnc since rsync only shares checksums untill it finds a block that > has changed. At small sizes, everything may be fine. But here are two things I would watch out for when doing this. #1 Rsync can run into problems on very large (# of files) filesystems. I've used rsync to copy some pretty big datasets. I had some where rsync would take a couple of hours before even starting to send data because it had to run the entire filesystem first. (This has nothing to do with ZFS). #2 Rsync tries to minimize the network transport, not local I/O. If you have small files, this isn't a problem. If you have large DB files, it might be. While rsync can detect that only a small number of blocks have changed, it will attempt to update that file atomically on the target machine. It does this by using a temporary file and renaming it after completing any updates. ZFS will see this as a completely new file, losing potential space savings from snapshots. -- Darren Dunham [EMAIL PROTECTED] Senior Technical Consultant TAOShttp://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + rsync, backup on steroids.
Dick Davies wrote: On 30/08/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: 'zfs send' is *incredibly* faster than rsync. That's interesting. We had considered it as a replacement for a certain task (publishing a master docroot to multiple webservers) but a quick test with ~500Mb of data showed the zfs send/recv to be about 5x slower than rsync for the initial copy. You're saying subsequent copies (zfs send -i?) should be faster? There's definitely something anomalous going on if you are seeing 'zfs send|recv' being 5x slower than rsync. I just did some quick tests on some mediocre machines and send|recv is much faster than rsync. So it would be great if you could describe the setup where you are seeing it be 5x slower, so that we can try to diagnose it. I used two different source filesystems: "workspace" has 98,384 files totaling 1.86GB "big files" has 2 files totaling 2.13GB I ran zfs send|recv as: ptime zfs send pool/[EMAIL PROTECTED] | \ ssh -c blowfish hostname zfs recv -dv pool/recvd I ran rsync as: ptime rsync -a -e "ssh -c blowfish" /pool/fs hostname:/pool/rsync/fs And the results: zfs send|recv rsync full workspace: 220s (8.6MB/s) 312s (6.1MB/s) incremental workspace: <1s 51s full big files 207s (10.6MB/s) 204s (10.7MB/s) incremental big files: <1s <1s So as you can see, we are somewhat constrained by the 100mbit/sec link between these machines, but on the workspace, the full backup is 40% faster with send|recv, and the incremental > 50x faster. To eliminate the network, I tried it between two pools on the same machine (without using ssh): zfs send|recv rsync full workspace: 133s (14.9MB/s) 339 (5.6MB/s) incremental workspace: <1s 56s full big files 74s (29.6MB/s) 70s (31.1MB/s) Here, the full workspace backup was 2.5x faster with send|recv, the incremental >50x faster, and the full big files 5% slower. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + rsync, backup on steroids.
On 30/08/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: Yes. The architectural benefits of 'zfs send' over rsync only apply to sending incremental changes. When sending a full backup, both schemes have to traverse all the metadata and send all the data, so the *should* be about the same speed. Cool! I'll retry it then. However, as I mentioned, there's still some low-hanging performance issues with 'zfs send', although I'm surprised that it was 5x slower than rsync! I'd like to look into that issue some more... What type of files were you sending? Eg. approximately what size files, how many files, how many files/directory? It was a copy of /usr/ports from freebsd, so around 500mb of small textfiles. Bear in mind I'm talking from memory, and it was just a quick test. I'll retry and let you know if I see a similar problem - if you don't hear anything, I couldn't replicate it. Thanks! -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Oracle on ZFS
[EMAIL PROTECTED] schrieb: Robert and Daniel, How did you put oracle on ZFS: - one zpool+ one filesystem - one zpool+ many filesystems - a few zpools + one filesystem on each - a few zpools + many filesystem on each My goal was not to maximize tuning for ZFS but just compare ZFS vs. UFS for average setup. I just setup ZFS with default parameters (beside the tested ones: compression on/off; checksum off/on/sha256). So my ZFS setup was simply one zpool + one filesystem; no specific tuning on the record size. I did some preliminary tests on different ZFS recordsize and Oracle db_block_size/db_file_multiblock_read_count but had mixed results so I kept the defaults. If I have some time (and my test machine back) I might try different ZFS/Oracle performance parameters and their impact. Daniel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + rsync, backup on steroids.
On Wed, Aug 30, 2006 at 07:51:45PM +0100, Dick Davies wrote: > On 30/08/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: > > >'zfs send' is *incredibly* faster than rsync. > > That's interesting. We had considered it as a replacement for a > certain task (publishing a master docroot to multiple webservers) > but a quick test with ~500Mb of data showed the zfs send/recv > to be about 5x slower than rsync for the initial copy. > > You're saying subsequent copies (zfs send -i?) should be faster? Yes and no. Depends on the nature of the changes that have happened. Re-writing the same file contents would result in the whole file's contents appearing in the incremental ZFS backup, but the rsync overhead for synchronizing the same file would be minimal (just the size of the rsync checksum, which is proportional to the file size, but much smaller). ZFS tracks changes transactionally and block-wise. Whereas rsync detects changes and produces remove+insert deltas. So ZFS snapshots/zfs send -i and rsync are very different things, complementary things perhaps, but different. Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + rsync, backup on steroids.
Dick Davies wrote: On 30/08/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: 'zfs send' is *incredibly* faster than rsync. That's interesting. We had considered it as a replacement for a certain task (publishing a master docroot to multiple webservers) but a quick test with ~500Mb of data showed the zfs send/recv to be about 5x slower than rsync for the initial copy. You're saying subsequent copies (zfs send -i?) should be faster? Yes. The architectural benefits of 'zfs send' over rsync only apply to sending incremental changes. When sending a full backup, both schemes have to traverse all the metadata and send all the data, so the *should* be about the same speed. However, as I mentioned, there's still some low-hanging performance issues with 'zfs send', although I'm surprised that it was 5x slower than rsync! I'd like to look into that issue some more... What type of files were you sending? Eg. approximately what size files, how many files, how many files/directory? --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + rsync, backup on steroids.
On 30/08/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: 'zfs send' is *incredibly* faster than rsync. That's interesting. We had considered it as a replacement for a certain task (publishing a master docroot to multiple webservers) but a quick test with ~500Mb of data showed the zfs send/recv to be about 5x slower than rsync for the initial copy. You're saying subsequent copies (zfs send -i?) should be faster? -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + rsync, backup on steroids.
James Dickens wrote: Why not make a snapshots on a production and then send incremental backups over net? Especially with a lot of files it should be MUCH faster than rsync. because its a ZFS limited solution, if the source is not ZFS it won't work, and i'm not sure how much faster incrementals would be than rsysnc since rsync only shares checksums untill it finds a block that has changed. 'zfs send' is *incredibly* faster than rsync. rsync needs to traverse all the metadata, so it is fundamentally O(all metadata). It needs to read every directory and stat every file, to figure out what's been changed. Then it needs to read all of every changed file to figure out what parts of it have been changed. In contrast, 'zfs send' essentially only needs to read the changed data, so it is O(changed data). We can do this by leveraging our knowledge of the zfs internal structure, eg. block birth times. That said, there is still a bunch of low-hanging performance fruit in 'zfs send', which I'll be working on over the next few months. And of course if you need a cross-filesystem tool then 'zfs send' is not for you. But give it a try if you can, and let us know how it works for you! --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Can a zfs storage pool be imported readonly?
oab wrote: Hi, I'm new to ZFS so I was wondering if it is possible to concurrently share a ZFS storage pool between two separate machines. No. You may be interested in this RFE: 4996041 want to directly access snapshots from multiple machines --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Can a zfs storage pool be imported readonly?
oab wrote: I'm new to ZFS so I was wondering if it is possible to concurrently share a ZFS storage pool between two separate machines. I am currently evaluating Sybase IQ running on ZFS rather than raw devices(initial performance tests look very promising) and need now to evaluate whether the IQ query server > functionality will work. Not today. We are very early on in the process of defining a project to make ZFS a shared file system for Sun Cluster. When this project is complete (no idea when that might be, yet, sorry), it will be possible to share a zpool among nodes of a cluster. We have support for fail-over ZFS (HA-ZFS) in the next release of Sun Cluster, due out later this year. That doesn't provide concurrent access, but it does provide high availability. This simply means that the Query server will have readonly access to the data (no lock manage required as far as I can tell), so that is why I would need to import the storage pool readonly or at least without exporting it. It's conceivable that it might work if *all* the nodes imported the pool read-only, but that's not tested and certainly not supported. -- --Ed begin:vcard fn:Ed Gould n:Gould;Ed org:Sun Microsystems, Inc.;Sun Cluster adr;dom:M/S UMPK17-201;;17 Network Circle;Menlo Park;CA;94025 email;internet:[EMAIL PROTECTED] title:File System Architect tel;work:650/786-4937 x-mozilla-html:FALSE version:2.1 end:vcard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: Re[2]: [zfs-discuss] ZFS + rsync, backup on steroids.
On 8/30/06, Robert Milkowski <[EMAIL PROTECTED]> wrote: Hello Jason, Tuesday, August 29, 2006, 9:35:13 PM, you wrote: JAH> On Aug 29, 2006, at 12:17 PM, James Dickens wrote: >> ZFS + rsync, backup on steroids. >> >> I was thinking today about backing up filesystems, and came up with an >> awesome idea. Use the power of rsync and ZFS together. >> >> Start with a one or two large SATA/PATA drives if you use two and >> don't need the space you can mirror other wise just use as in raid0, >> enable compression unless your files are mostly precompressed, use >> rsync as the backup tool, the first time you just copy the data over. >> After you are done, take a snapshot, export the pool. And uninstall >> the drives until next time. When next time rolls around have rsync >> update the changed files, as it does block copies of changed data, >> only a small part of the data has changed. After than is done, take a >> snapshot. >> >> Now thanks to ZFS you have complete access to incremental backups, >> just look at the desired snapshots. For now rsync doesn't support >> nfsv4 acls, but at least you have the data. JAH> Yes I concur. This is how we do our backups, rsync + rolling over JAH> snapshots. Why not make a snapshots on a production and then send incremental backups over net? Especially with a lot of files it should be MUCH faster than rsync. because its a ZFS limited solution, if the source is not ZFS it won't work, and i'm not sure how much faster incrementals would be than rsysnc since rsync only shares checksums untill it finds a block that has changed. James -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re[2]: [zfs-discuss] ZFS + rsync, backup on steroids.
Hello Jason, Tuesday, August 29, 2006, 9:35:13 PM, you wrote: JAH> On Aug 29, 2006, at 12:17 PM, James Dickens wrote: >> ZFS + rsync, backup on steroids. >> >> I was thinking today about backing up filesystems, and came up with an >> awesome idea. Use the power of rsync and ZFS together. >> >> Start with a one or two large SATA/PATA drives if you use two and >> don't need the space you can mirror other wise just use as in raid0, >> enable compression unless your files are mostly precompressed, use >> rsync as the backup tool, the first time you just copy the data over. >> After you are done, take a snapshot, export the pool. And uninstall >> the drives until next time. When next time rolls around have rsync >> update the changed files, as it does block copies of changed data, >> only a small part of the data has changed. After than is done, take a >> snapshot. >> >> Now thanks to ZFS you have complete access to incremental backups, >> just look at the desired snapshots. For now rsync doesn't support >> nfsv4 acls, but at least you have the data. JAH> Yes I concur. This is how we do our backups, rsync + rolling over JAH> snapshots. Why not make a snapshots on a production and then send incremental backups over net? Especially with a lot of files it should be MUCH faster than rsync. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] File level compression
Hello Sanjeev, Wednesday, August 30, 2006, 3:26:52 PM, you wrote: SB> Hi, SB> We were trying out the "compression=on" feature of ZFS and were SB> wondering if it would make SB> sense to have ZFS do compression only on a certain kind of files (or SB> rather the otherway around). SB> Our observation : SB> - If ZFS finds that it cannot achieve a certain amount of compression it SB> does not compress the file. SB> However, to figure this out ZFS would have to first compress the SB> block's data. This means ZFS ends SB> up consuming resources and that overhead is not worth it. SB> Is the above observation correct. If so, is there someway this can be SB> tuned/controlled ? right now it's 12% minimum compression gain required. Unfortunately last I checked it was hard-coded. Additionally it's not file based but block based so for large files which are already compressed zfs during writing will try to compress block by block. SB> If not how about having a feature where we could tell ZFS something like : SB> " Don't try to compress the .gz files" ? We could probably use the SB> magic number of the file to SB> identify the type. SB> Does this make sense ? Or is this a stupid idea ? I guess some functionality like this would be great. IIRC it was also proposed by someone from ZFS team long time ago :) -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Solaris 6/06 ZFS and OpenSolaris ZFS
Hello Louwtjie, Wednesday, August 30, 2006, 12:04:57 PM, you wrote: LB> What are the major differences between the "first" zfs shipped in LB> 06/06 Solaris 10, compared to the latest built's of OpenSolaris? Right now it's 1. ZFS hot spare support 2. raidz2 3. performance bug fixes LB> Will there be any major functionality released to 06/06 Solaris zfs via patches? I guess all above three will be in U3 (and patches). LB> Will major zfs updates only be integrated into Solaris with the regular release cycles? LB> (ala 03/05 - 06/06 - etc.) I guess as usual fixes will be delivered also in form of patches. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re[2]: [zfs-discuss] Oracle on ZFS
Hello przemolicc, Wednesday, August 30, 2006, 9:49:33 AM, you wrote: ppf> On Fri, Aug 25, 2006 at 06:52:17PM +0200, Daniel Rock wrote: >> [EMAIL PROTECTED] schrieb: >> >Hi all, >> > >> >Does anybody use Oracle on ZFS in production (but not as a >> >background/testing database but as a front line) ? ppf> [ ... ] ppf> Robert and Daniel, ppf> How did you put oracle on ZFS: ppf> - one zpool+ one filesystem ppf> - one zpool+ many filesystems ppf> - a few zpools + one filesystem on each ppf> - a few zpools + many filesystem on each I used one pool + many file systems, with recordsize set for some of them. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Oracle on ZFS
I have seen the best oracle performance on ZFS by 1. match the zfs record size to oracle db_block_size 2. use the default 128k record size for oracle logs. 3. If possible use a separate zpool for the oracle logs. This is especially true if your workload has a high write component to it. Two other knobs help (vdev cache/stb and vq_max_pending) but i do not recommend them right now. I believe work is already underway to have better values out-of-the-box. hth, -neel ps:If you want to know about vq_max_pending, please read Eric Kustarz's blog @ http://blogs.sun.com/erickustarz/entry/vq_max_pending [EMAIL PROTECTED] wrote: On Fri, Aug 25, 2006 at 06:52:17PM +0200, Daniel Rock wrote: [EMAIL PROTECTED] schrieb: Hi all, Does anybody use Oracle on ZFS in production (but not as a background/testing database but as a front line) ? [ ... ] Robert and Daniel, How did you put oracle on ZFS: - one zpool+ one filesystem - one zpool+ many filesystems - a few zpools + one filesystem on each - a few zpools + many filesystem on each przemol ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] File level compression
Hi, We were trying out the "compression=on" feature of ZFS and were wondering if it would make sense to have ZFS do compression only on a certain kind of files (or rather the otherway around). Our observation : - If ZFS finds that it cannot achieve a certain amount of compression it does not compress the file. However, to figure this out ZFS would have to first compress the block's data. This means ZFS ends up consuming resources and that overhead is not worth it. Is the above observation correct. If so, is there someway this can be tuned/controlled ? If not how about having a feature where we could tell ZFS something like : " Don't try to compress the .gz files" ? We could probably use the magic number of the file to identify the type. Does this make sense ? Or is this a stupid idea ? Thanks and regards, Sanjeev. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Can a zfs storage pool be imported readonly?
Hi, I'm new to ZFS so I was wondering if it is possible to concurrently share a ZFS storage pool between two separate machines. I am currently evaluating Sybase IQ running on ZFS rather than raw devices(initial performance tests look very promising) and need now to evaluate whether the IQ query server functionality will work. This simply means that the Query server will have readonly access to the data (no lock manage required as far as I can tell), so that is why I would need to import the storage pool readonly or at least without exporting it. Any help would be appreciated and I will report my findings. Regards oab This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Solaris 6/06 ZFS and OpenSolaris ZFS
What are the major differences between the "first" zfs shipped in 06/06 Solaris 10, compared to the latest built's of OpenSolaris? Will there be any major functionality released to 06/06 Solaris zfs via patches? Will major zfs updates only be integrated into Solaris with the regular release cycles? (ala 03/05 - 06/06 - etc.) This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Re: Re: zpool status panics server
On Fri, Aug 25, 2006 at 01:21:25PM -0600, Cindy Swearingen wrote: > Hi Neal, > > The ZFS administration class, available in the fall, I think, covers > basically the same content as the ZFS admin guide only with extensive > lab exercises. You could not give any ZFS class and advertise it as: "ZFS is so simple that we don't have any ZFS class !". Just kidding ;-) przemol ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Oracle on ZFS
On Fri, Aug 25, 2006 at 06:52:17PM +0200, Daniel Rock wrote: > [EMAIL PROTECTED] schrieb: > >Hi all, > > > >Does anybody use Oracle on ZFS in production (but not as a > >background/testing database but as a front line) ? [ ... ] Robert and Daniel, How did you put oracle on ZFS: - one zpool+ one filesystem - one zpool+ many filesystems - a few zpools + one filesystem on each - a few zpools + many filesystem on each przemol ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
