Re: [Discuss] More Fun in ZFSland
On 5/16/2012 10:11 PM, Richard Pieri wrote: I'm not really paying for the encryption. My N40L is I/O bound at the network interface so I can't saturate the disks or CPU over SMB and AFP. It turns out that I was very much in error on this. My level 0 backup last weekend took about as long as I expected since it's I/O bound by USB2. But then the weekly scrub took several times longer than the unencrypted scrub and that seemed very odd to me. Nothing seemed obviously wrong. System load was fine but the scrub throughput was very low. So I installed the ZFS on Linux kernel module set to take FUSE out of the loop and did some testing. And then some Google searching proved what I found. kcryptd is stupid. dm-crypt sets up a kcryptd thread for each encrypted device, but kcryptd itself isn't SMP-aware. All of those threads run together on one core. When it peaks at 100% usage then that's the I/O throughput cap. There's simply no way to make it go any faster. Time to revert and rethink. Live and learn and pass on that knowledge. I'm reverting using the offline/resilver method: zpool offline the dm-crypt device, remove it from /dev/mapper, make a symlink to trick ZFS into thinking the raw partition is the /dev/mapper device, zpool replace and let it resilver. I want to see how reducing the number of dm-crypt devices affects throughput. I also want to see resilver in action. -- Rich P. ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
[Discuss] More Fun in ZFSland
One of the things missing from zfs-fuse is the encryption subsystem. ZFS encryption was introduced by Oracle after closing the Solaris 10 source code so we don't yet have an open source reference for it. So, how to get encrypted ZFS? Every disk-based device is a block device and they all share the same APIs. This is what makes nesting LVM + DRBD + dm-crypt possible. Nested block devices! It's an all-or-nothing solution, not as elegant as a native dataset encryption subsystem, but it can work. What I did: Started out making backups of everything courtesy of snapshots and zfs send. This would be a good opportunity to test a full recovery. Destroyed the zpool. Used gdisk to create single partitions on each of the storage disks. gdisk (GPT fdisk) is an fdisk-like tool that works on GUID disks. It's also aware of 4k disks and automatically sets the partition boundaries appropriately. Used cryptsetup/LUKS to create dm-crypt devices on the partitions. Then created a new raidz pool on top of those. And it works. There is some CPU overhead in the encryption layer but it is unnoticeable in normal operation. Restored everything via zfs receive. And it all works. Which means my notebook backups remain encrypted on disk. It's overkill for my music and video libraries but that comes with encrypting the vdev block devices. Finally wrote a little script to handle opening the encrypted devices and importing the zpool since it can't work unattended. -- Rich P. ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] More Fun in ZFSland
On 05/16/2012 04:41 PM, Richard Pieri wrote: Richard, I read this and say to myself, this sounds more like you want to solve a problem with ZFS instead of wanting to solve a problem the best way possible. If you want to do it with ZFS because you think you can, then cool, have fun. If you want to solve a problem, what is the specific problem? and is there a solution that is less of the hoop jumping through kind? Usually when I start seeing the need to do the sorts of things you seem to be doing, I think to my self, Someone else must be doing something similar, it should not be this hard to do. Sometimes I find, yes, no one else is doing this. Other times I get a doh! moment. I'm not judging, I'm just saying. I get worried about my data when I start to do interesting things with it. One of the things missing from zfs-fuse is the encryption subsystem. ZFS encryption was introduced by Oracle after closing the Solaris 10 source code so we don't yet have an open source reference for it. So, how to get encrypted ZFS? Every disk-based device is a block device and they all share the same APIs. This is what makes nesting LVM + DRBD + dm-crypt possible. Nested block devices! It's an all-or-nothing solution, not as elegant as a native dataset encryption subsystem, but it can work. What I did: Started out making backups of everything courtesy of snapshots and zfs send. This would be a good opportunity to test a full recovery. Destroyed the zpool. Used gdisk to create single partitions on each of the storage disks. gdisk (GPT fdisk) is an fdisk-like tool that works on GUID disks. It's also aware of 4k disks and automatically sets the partition boundaries appropriately. Used cryptsetup/LUKS to create dm-crypt devices on the partitions. Then created a new raidz pool on top of those. And it works. There is some CPU overhead in the encryption layer but it is unnoticeable in normal operation. Restored everything via zfs receive. And it all works. Which means my notebook backups remain encrypted on disk. It's overkill for my music and video libraries but that comes with encrypting the vdev block devices. Finally wrote a little script to handle opening the encrypted devices and importing the zpool since it can't work unattended. ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] More Fun in ZFSland
On May 16, 2012, at 5:41 PM, Mark Woodward wrote: If you want to solve a problem, what is the specific problem? and is there a solution that is less of the hoop jumping through kind? Usually when I start What hoop-jumping? Encrypting block devices is the typical method of implementing OTFE on Linux. In the old days we used the losetup utility to created encrypted block devices. Today we use device mapper's cryptsetup utility. I'd have used the same general procedures if I were using dm-raid instead of ZFS. The problem to solve is that once I decided to encrypt my gaming rig's disks in case a thief breaks in and steals it, I was foolish not to have it's backups encrypted as well. These backups are on the HP N40L previously discussed. I'm happy with ZFS and I want to continue using it as the LVM and file system. ZFS encrypted datasets isn't available in ZFS on Linux or FreeBSD yet which left me with the choice of using encrypted containers on ZFS or encrypted block devices underneath ZFS. I chose encrypted block devices. Once I made the decision to encrypt at all it was a simple jump to encrypt everything. It was a little more work for the initial setup than creating encrypted containers but it will be less work down the line to maintain it. --Rich P. ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] More Fun in ZFSland
Richard Pieri richard.pi...@gmail.com writes: I chose encrypted block devices. Once I made the decision to encrypt at all it was a simple jump to encrypt everything. It was a little more work for the initial setup than creating encrypted containers but it will be less work down the line to maintain it. Are you familiar with zvol? Your maintence concern makes it sound like you may not be. I certainly can't think of anything I've done with mine past creation that resembles maintain. zfs create -V 10G pool/myzvol produces a 10 gigabyte block device in /dev/zvol/pool/myzvol (give or take how your OS handles the details), that gets it blocks from your zpool. The major price is that your encrypted volume will have travel through the zpool layer twice. But you won't pay for encryption on your music library. Which way you want to pay is your call, obviously. ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] More Fun in ZFSland
On May 16, 2012, at 9:02 PM, Daniel Hagerty wrote: Are you familiar with zvol? Your maintence concern makes it sound like you may not be. I certainly can't think of anything I've done with mine past creation that resembles maintain. Familiar with? No. Aware of? Yes. ZFS is incorporates volume manager and file system. zvol allows access to the volume manager portion so one can create arbitrary raw block devices. Useful for databases and non-ZFS file systems. Going through the zpool layer twice isn't a big lose. Statically-sized volumes is a massive lose. Using zvols requires allocating volumes at creation time. Resizing volumes and file systems needs to be done by hand. It's not hard but I find it tedious. Avoiding this is one of the reasons I chose ZFS over LVM + ext3. dm-crypt on zvols was never under consideration. I'm not really paying for the encryption. My N40L is I/O bound at the network interface so I can't saturate the disks or CPU over SMB and AFP. --Rich P. ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
