Re: [zfs-discuss] encryption
From: Darren J Moffat [mailto:darr...@opensolaris.org] Sent: Monday, February 20, 2012 12:46 PM GRUB2 has support for encrypted ZFS file systems already. I assume this requires a pre-boot password, right? Then I have two questions... I noticed in solaris 11, when you init 6 it doesn't reboot the way other OSes reboot. So maybe init 6 will not need you to type in a password again? Maybe you just need a passsword one time when you power on? If you have to enter password at every reboot, of course, it lends some bias toward using that encryption solution only for laptops, where there's guaranteed to be a user present. It's not quite mutually exclusive with servers, but a strong bias away from being used on servers. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] encryption
On 02/21/12 13:27, Edward Ned Harvey wrote: From: Darren J Moffat [mailto:darr...@opensolaris.org] Sent: Monday, February 20, 2012 12:46 PM GRUB2 has support for encrypted ZFS file systems already. I assume this requires a pre-boot password, right? Then I have two questions... The ZFS encryption support in GRUB2 was written by the main GRUB2 developer and doesn't use any Solaris ZFS encryption code. The GRUB2 code has support for interactive prompting for the passphrase or for reading the passphrase or raw wrapping key from a file in some other filesystem that GRUB2 can see. Solaris 11 doesn't have GRUB2 at this time it uses GRUB 0.97 which does not have encryption support. You can't put the two parts together because the Solaris 11 kernel doesn't know how to mount an encrypted root filesystem even though GRUB2 could have loaded the kernel and boot_archive from one if you managed to craft together a GRUB2 and Solaris 11 system on your own. I noticed in solaris 11, when you init 6 it doesn't reboot the way other OSes reboot. What you are seeing is Fast Reboot where on x86 we completely avoid the trip back through the BIOS and the boot loader it just loads and rexecute the kernel directly. The situation on SPARC is similar but not identical. So maybe init 6 will not need you to type in a password again? Maybe you just need a passsword one time when you power on? Solaris 11 doesn't have support for encrypted root at all at this time. Doesn't mater if Fast Reboot is in use or not. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] encryption
On 02/16/12 15:35, David Magda wrote: On Thu, February 16, 2012 09:55, Edward Ned Harvey wrote: I've never used ZFS encryption. How does it work? Do you need to type in a pre-boot password? And if so, how do you do that with a server? Or does it use TPM or something similar, to avoid the need for a pre-boot password? Darren Moffat put up some good posts when the code was initially introduced: https://blogs.oracle.com/darren/en_GB/tags/zfs https://blogs.oracle.com/darren/en_GB/tags/crypto I don't believe encrypting the root volume is currently supported, so pre-boot stuff doesn't apply. (Please correct if I'm wrong here.) That is correct you can't currently encrypt the root/boot file system. This is because neither OBP or GRUB 0.97 have any knowledge of ZFS encrypted file systems and how to get keys for them. GRUB2 has support for encrypted ZFS file systems already. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] encryption
I've never used ZFS encryption. How does it work? Do you need to type in a pre-boot password? And if so, how do you do that with a server? Or does it use TPM or something similar, to avoid the need for a pre-boot password? Thanks... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] encryption
On Thu, February 16, 2012 09:55, Edward Ned Harvey wrote: I've never used ZFS encryption. How does it work? Do you need to type in a pre-boot password? And if so, how do you do that with a server? Or does it use TPM or something similar, to avoid the need for a pre-boot password? Darren Moffat put up some good posts when the code was initially introduced: https://blogs.oracle.com/darren/en_GB/tags/zfs https://blogs.oracle.com/darren/en_GB/tags/crypto I don't believe encrypting the root volume is currently supported, so pre-boot stuff doesn't apply. (Please correct if I'm wrong here.) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] encryption
From: David Magda [mailto:dma...@ee.ryerson.ca] Sent: Thursday, February 16, 2012 10:35 AM On Thu, February 16, 2012 09:55, Edward Ned Harvey wrote: I've never used ZFS encryption. How does it work? Do you need to type in a pre-boot password? And if so, how do you do that with a server? Or does it use TPM or something similar, to avoid the need for a pre-boot password? Darren Moffat put up some good posts when the code was initially introduced: https://blogs.oracle.com/darren/en_GB/tags/zfs https://blogs.oracle.com/darren/en_GB/tags/crypto I don't believe encrypting the root volume is currently supported, so pre-boot stuff doesn't apply. (Please correct if I'm wrong here.) Very nice. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.[GPU acceleration of ZFS]
-Original Message- From: David Magda [mailto:dma...@ee.ryerson.ca] Sent: 星期二, 六月 28, 2011 10:41 To: Fred Liu Cc: Bill Sommerfeld; ZFS Discuss Subject: Re: [zfs-discuss] Encryption accelerator card recommendations.[GPU acceleration of ZFS] On Jun 27, 2011, at 22:03, Fred Liu wrote: FYI There is another thread named -- GPU acceleration of ZFS in this list to discuss the possibility to utilize the power of GPGPU. I posted here: In a similar vein I recently came across SSLShader: http://shader.kaist.edu/sslshader/ http://www.usenix.org/event/nsdi11/tech/full_papers/Jang.pdf http://www.google.com/search?q=sslshader This could be handy for desktops doing ZFS crypto (and even browser SSL and/or SSH), but few servers have decent graphics cards (and SPARC systems don't even have video ports by :). Agree. The most challenging part is coding as long as there is an empty PCIE slot in server. Thanks. Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
Thanks for this pointer ... I have been looking for a small (low
power) server for a bit now and did not realize that HP had anything
in the line below the ML-1xx.
One of the reviews at the HP site note that the 5.25 media bay is
IDE only (from a BIOS perspective), can you confirm or deny this ? I
really want 6 drives (2 x 250 GB OS, 4 x 1 TB data), and using the
5.25 bay plus the eSata I can get there. Although if I can use a
couple 16 GB USB flash drives for OS I *might* go that route.
I am not planning on using encryption, so the CPU is probably not a
limitation for me.
On Mon, Jun 27, 2011 at 12:55 PM, Roberto Waltman li...@rwaltman.com wrote:
I recently bought an HP Proliant Microserver for a home file server.
( pics and more here:
http://arstechnica.com/civis/viewtopic.php?p=20968192 )
I installed 5 1.5TB (5900 RPM) drives, upgraded the memory to 8GB, and
installed Solaris 11 Express without a hitch.
A few simple tests using dd with 1gb and 2gb files showed excellent
transfer rates: ~200 MB/sec on a 5 drive raidz2 pool, ~310 MB/sec on a five
drive pool with no redundancy.
That is, until I enabled encryption, which brought the transfer rates down
to around 20 MB/sec...
--
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On Jun 27, 2011, at 17:16, Erik Trimble wrote: Think about how things were done with the i386 and i387. That's what I'm= after. With modern CPU buses like AMD Intel support, plopping a co-pro= cessor into another CPU socket would really, really help. Given the amount of transistors that are available nowadays I think it'd be= simpler to just create a series of SIMD instructions right in/on general C= PUs, and skip the whole co-processor angle. One of the VIA processors was one of the first with specific random and AES instructions. AMD Intel have followed suite and your can some information here: http://en.wikipedia.org/wiki/AES_instruction_set (Similar instructions have been added for SHA, MD5 (older CPUs), RSA, though typically using building blocks not a single long running instruction) A number of the crypto accelerators were much slower than the implementation of a direct implementation in opcodes; one issue, though, what register sets will be used and where will it be saved when the thread is preempted (I'm assuming that the reason why AMD and Intel use different instructions from VIA is possibly because of such details. The current implementation the T3 uses a co-processor (one per core, I think) Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On 06/27/11 11:32 PM, Bill Sommerfeld wrote: On 06/27/11 15:24, David Magda wrote: Given the amount of transistors that are available nowadays I think it'd be simpler to just create a series of SIMD instructions right in/on general CPUs, and skip the whole co-processor angle. see: http://en.wikipedia.org/wiki/AES_instruction_set Present in many current Intel CPUs; also expected to be present in AMD's Bulldozer based CPUs. I recall seeing a blog comparing the existing Solaris hand-tuned AES assembler performance with the (then) new AES instruction version, where the Intel AES instructions only got you about a 30% performance increase. I've seen reports of better performance improvements, but usually by comparing with the performance on older processors which are going to be slower for additional reasons then just missing the AES instructions. Also, you could claim better performance improvement if you compared against a less efficient original implementation of AES. What this means is that a faster CPU may buy you more crypto performance than the AES instructions alone will do. My understanding from reading the Intel AES instruction set (which I warn might not be completely correct) is that the AES encryption/decryption instruction is executed between 10 and 14 times (depending on key length) for each 128 bits (16 bytes) of data being encrypted/decrypted, so it's very much part of the regular instruction pipeline. The code will have to loop though this process multiple times to process a data block bigger than 16 bytes, i.e. a double nested loop, although I expect it's normally loop-unrolled a fair degree for optimisation purposes. Conversely, the crypto units in the T-series processors are separate from the CPU, and do the encryption/decryption whilst the CPU is getting on with something else, and they do it much faster than it could be done on the CPU. Small blocks are normally a problem for crypto offload engines because the overhead of farming off the work to the engine and getting the result back often means that you can do the crypto on the CPU faster than the time it takes to get the crypto engine started and stopped. However, T-series crypto is particularly good at handling small blocks efficiently, such as around 1kbyte which you are likely to find in a network packet, as it is much closer coupled to the CPU than a PCI crypto card can be, and performance with small packets was key for the crypto networking support T-series was designed for. Of course, it handles crypto of large blocks just fine too. -- Andrew ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
All (Ultra)SPARC T2, T2+, and T3 CPUs should have these capabilities; if you have some other CPU the capabilities are probably not present. Run 'prtdiag | head -20' to see the CPUs on your system/s; run cryptoadm(1M) with the list option (Solaris 10+) to see the software and hardware providers available. For further assistance your best bet would be crypto-discuss (this has gotten OT for zfs-discuss): http://mail.opensolaris.org/pipermail/crypto-discuss/ Thanks, I'll ask over there. I understood there was a broadcomm add on card for servers but from your answer it seems the CPU supports crypto operations. I don't understand what the purpose of having both support it is if they want to sell crypto cards. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Encryption accelerator card recommendations.
I recently bought an HP Proliant Microserver for a home file server. ( pics and more here: http://arstechnica.com/civis/viewtopic.php?p=20968192 ) I installed 5 1.5TB (5900 RPM) drives, upgraded the memory to 8GB, and installed Solaris 11 Express without a hitch. A few simple tests using dd with 1gb and 2gb files showed excellent transfer rates: ~200 MB/sec on a 5 drive raidz2 pool, ~310 MB/sec on a five drive pool with no redundancy. That is, until I enabled encryption, which brought the transfer rates down to around 20 MB/sec... Obviously the CPU is the bottleneck here, and I?m wondering what to do next. I can split the storage into file systems with and without encryption and allocate data accordingly. No need, for example, to encrypt open source code, or music. But I would like to have everything encrypted by default. My concern is not industrial espionage from a hacker in Belarus, but having a disk fail and send it for repair with my credit card statements easily readable on it, etc. I am new to (open or closed)Solaris. I found there is something called the Encryption Framework, and that there is hardware support for encryption. This server has two unused PCI-e slots, so I thought a card could be the solution, but the few I found seem to be geared to protect SSH and VPN connections, etc., not the file system. Cost is a factor also. I could build a similar server with a much faster processor for a few hundred dollars more, so a $1000 dollar card for a $1000 file server is not a reasonable option. Is there anything out there I could use? Thanks, Roberto Waltman ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
IMO a faster processor with built-in AES and other crypto support is most likely to give you the most bang for your buck, particularly if you're using closed Solaris 11, as Solaris engineering is likely to add support for new crypto instructions faster than Illumos (but I don't really know enough about Illumos to say for sure). Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On 6/27/2011 9:55 AM, Roberto Waltman wrote: I recently bought an HP Proliant Microserver for a home file server. ( pics and more here: http://arstechnica.com/civis/viewtopic.php?p=20968192 ) I installed 5 1.5TB (5900 RPM) drives, upgraded the memory to 8GB, and installed Solaris 11 Express without a hitch. A few simple tests using dd with 1gb and 2gb files showed excellent transfer rates: ~200 MB/sec on a 5 drive raidz2 pool, ~310 MB/sec on a five drive pool with no redundancy. That is, until I enabled encryption, which brought the transfer rates down to around 20 MB/sec... Obviously the CPU is the bottleneck here, and I?m wondering what to do next. I can split the storage into file systems with and without encryption and allocate data accordingly. No need, for example, to encrypt open source code, or music. But I would like to have everything encrypted by default. My concern is not industrial espionage from a hacker in Belarus, but having a disk fail and send it for repair with my credit card statements easily readable on it, etc. I am new to (open or closed)Solaris. I found there is something called the Encryption Framework, and that there is hardware support for encryption. This server has two unused PCI-e slots, so I thought a card could be the solution, but the few I found seem to be geared to protect SSH and VPN connections, etc., not the file system. Cost is a factor also. I could build a similar server with a much faster processor for a few hundred dollars more, so a $1000 dollar card for a $1000 file server is not a reasonable option. Is there anything out there I could use? Thanks, Roberto Waltman You're out of luck. The hardware-encryption device is seen as a small market by the vendors, and they price accordingly. All the solutions are FIPS-compliant, which makes them non-trivially expensive to build/test/verify. I have yet to see the basic crypto accelerator - which should be as simple as an FPGA with downloadable (and updateable) firmware. The other major cost point is the crypto plugins - sadly, there is no way to simply have the CPU farm off crypto jobs to a co-processor. That is, there's no way for the CPU to go oh, that looks like I'm running a crypto algorithm - I should hand it over to the crypto co-processor. Instead, you have to write custom plugin/drivers/libraries for each OS, and pray that each OS has some standardized crypto framework. Otherwise, you have to link apps with custom libraries. I'm always kind of surprised that there hasn't been a movement to create standardized crypto commands, like the various FP-specific commands that are part of MMX/SSE/etc. That way, most of this could be done in hardware seamlessly. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On Mon, June 27, 2011 15:24, Erik Trimble wrote: [...] I'm always kind of surprised that there hasn't been a movement to create standardized crypto commands, like the various FP-specific commands that are part of MMX/SSE/etc. That way, most of this could be done in hardware seamlessly. The (Ultra)SPARC T-series processors do, but to a certain extent it goes against a CPU manufacturers best (financial) interest to provide this: crypto is very CPU intensive using 'regular' instructions, so if you need to do a lot of it, it would force you to purchase a manufacturer's top-of-the-line CPUs, and to have as many sockets as you can to handle a load (and presumably you need to do useful work besides just en/decrypting traffic). If you have special instructions that do the operations efficiently, it means that you're not chewing up cycles as much, so a less powerful (and cheaper) processor can be purchased. I'm sure all the Web 2.0 companies would love to have these (and OpenSSL link use the instructions), so they could simply enable HTTPS for everything. (Of course it'd also be helpful for data-at-rest, on-disk encryption as well.) The last benchmarks I saw indicated that the SPARC T-series could do 45 Gb/s AES or some such, with gobs of RSA operations as well. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On 6/27/2011 1:13 PM, David Magda wrote: On Mon, June 27, 2011 15:24, Erik Trimble wrote: [...] I'm always kind of surprised that there hasn't been a movement to create standardized crypto commands, like the various FP-specific commands that are part of MMX/SSE/etc. That way, most of this could be done in hardware seamlessly. The (Ultra)SPARC T-series processors do, but to a certain extent it goes against a CPU manufacturers best (financial) interest to provide this: crypto is very CPU intensive using 'regular' instructions, so if you need to do a lot of it, it would force you to purchase a manufacturer's top-of-the-line CPUs, and to have as many sockets as you can to handle a load (and presumably you need to do useful work besides just en/decrypting traffic). If you have special instructions that do the operations efficiently, it means that you're not chewing up cycles as much, so a less powerful (and cheaper) processor can be purchased. I'm sure all the Web 2.0 companies would love to have these (and OpenSSL link use the instructions), so they could simply enable HTTPS for everything. (Of course it'd also be helpful for data-at-rest, on-disk encryption as well.) The last benchmarks I saw indicated that the SPARC T-series could do 45 Gb/s AES or some such, with gobs of RSA operations as well The T-series crypto isn't what I'm thinking of. AFAIK, you still need to use the Crypto framework in Solaris to access the on-chip functionality. Which makes the T-series no different than CPUs without a crypto module but a crypto add-in board instead. What I'm thinking of is something on the lines of what AMD proposed awhile ago, in combination with how we used to handle hardware that had FP optional. That is, you continue to make CPUs without any crypto functionality, EXCEPT that they support certain extensions a la MMX. If no Crypto accelerator was available, the CPU would trap any Crypto calls, and force them to done in software. You could then stick a crypto accellerator in a second CPU socket, and the CPU would recognized this was there, and pipe crypto calls to the dedicated co-processor. Think about how things were done with the i386 and i387. That's what I'm after. With modern CPU buses like AMD Intel support, plopping a co-processor into another CPU socket would really, really help. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On Jun 27, 2011, at 17:16, Erik Trimble wrote: Think about how things were done with the i386 and i387. That's what I'm after. With modern CPU buses like AMD Intel support, plopping a co-processor into another CPU socket would really, really help. Given the amount of transistors that are available nowadays I think it'd be simpler to just create a series of SIMD instructions right in/on general CPUs, and skip the whole co-processor angle. There's more and more sensitive data out there, so on-disk crypto could be deployed in more places to help prevent data loss (on both servers and desktops/laptops), and those systems that don't do disk IO probably do network IO, and so would be helped from a TLS/SSL/SSH perspective. If I were AMD I'd seriously be thinking about this, as it'd help boost volume and mindshare for a little while with all the folks doing any kind of web activity would pick up kit for HTTPS—at least until Intel brought out a similar thing. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On 06/27/11 15:24, David Magda wrote: Given the amount of transistors that are available nowadays I think it'd be simpler to just create a series of SIMD instructions right in/on general CPUs, and skip the whole co-processor angle. see: http://en.wikipedia.org/wiki/AES_instruction_set Present in many current Intel CPUs; also expected to be present in AMD's Bulldozer based CPUs. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On Jun 27, 2011, at 18:32, Bill Sommerfeld wrote: On 06/27/11 15:24, David Magda wrote: Given the amount of transistors that are available nowadays I think it'd be simpler to just create a series of SIMD instructions right in/on general CPUs, and skip the whole co-processor angle. see: http://en.wikipedia.org/wiki/AES_instruction_set Present in many current Intel CPUs; also expected to be present in AMD's Bulldozer based CPUs. Now compare that with the T-series stuff that also handles 3DES, RC4, RSA2048, DSA, DH, ECC, MD5, SHA1, SHA2, as well as a hardware RNG: http://en.wikipedia.org/wiki/UltraSPARC_T2 http://blogs.oracle.com/BestPerf/entry/20100920_sparc_t3_pk11rsaperf The initial TLS/SSL set up is actually the expensive part (20-58% of the time spent of the 'transaction'), and that AES can be performed decently even on non-AESNI CPUs: simply adding an RSA accelerator can double performance without session caching, and even ~20% with it. SSL session caching alone can help improve throughput by a factor of more than two. Performance Analysis of TLS Web Servers http://www.cs.rice.edu/~dwallach/pub/tls-tocs.pdf http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.16.1403 AESNI is certain better than nothing, but RSA, SHA, and the RNG would be nice as well. It'd also be handy for ZFS crypto in addition to all the network IO stuff. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.[GPU acceleration of ZFS]
FYI There is another thread named -- GPU acceleration of ZFS in this list to discuss the possibility to utilize the power of GPGPU. I posted here: Good day, I think ZFS can take advantage of using GPU for sha256 calculation, encryption and maybe compression. Modern video card, like 5xxx or 6xxx ATI HD Series can do calculation of sha256 50-100 times faster than modern 4 cores CPU. kgpu project for linux shows nice results. 'zfs scrub' would work freely on high performance ZFS pools. The only problem that there is no AMD/Nvidia drivers for Solaris that support hardware-assisted OpenCL. Is anyone interested in it? Best regards, Anatoly Legkodymov. On Tue, May 10, 2011 at 11:29 AM, Anatoly legko...@fastmail.fm wrote: Good day, I think ZFS can take advantage of using GPU for sha256 calculation, encryption and maybe compression. Modern video card, like 5xxx or 6xxx ATI HD Series can do calculation of sha256 50-100 times faster than modern 4 cores CPU. Ignoring optimizations from SIMD extensions like SSE and friends, this is probably true. However, the GPU also has to deal with the overhead of data transfer to itself before it can even begin crunching data. Granted, a Gen. 2 x16 link is quite speedy, but is CPU performance really that poor where a GPU can still out-perform it? My undergrad thesis dealt with computational acceleration utilizing CUDA, and the datasets had to scale quite a ways before there was a noticeable advantage in using a Tesla or similar over a bog-standard i7-920. The only problem that there is no AMD/Nvidia drivers for Solaris that support hardware-assisted OpenCL. This, and keep in mind that most of the professional users here will likely be using professional hardware, where a simple 8MB Rage XL gets the job done thanks to the magic of out-of-band management cards and other such facilities. Even as a home user, I have not placed a high-end videocard into my machine, I use a $5 ATI PCI videocard that saw about a hour of use whilst I installed Solaris 11. -- --khd IMHO, zfs need to run in all kind of HW T-series CMT server that can help sha calculation since T1 day, did not see any work in ZFS to take advantage it On 5/10/2011 11:29 AM, Anatoly wrote: Good day, I think ZFS can take advantage of using GPU for sha256 calculation, encryption and maybe compression. Modern video card, like 5xxx or 6xxx ATI HD Series can do calculation of sha256 50-100 times faster than modern 4 cores CPU. kgpu project for linux shows nice results. 'zfs scrub' would work freely on high performance ZFS pools. The only problem that there is no AMD/Nvidia drivers for Solaris that support hardware-assisted OpenCL. Is anyone interested in it? Best regards, Anatoly Legkodymov. On Tue, May 10, 2011 at 10:29 PM, Anatoly legko...@fastmail.fm wrote: Good day, I think ZFS can take advantage of using GPU for sha256 calculation, encryption and maybe compression. Modern video card, like 5xxx or 6xxx ATI HD Series can do calculation of sha256 50-100 times faster than modern 4 cores CPU. kgpu project for linux shows nice results. 'zfs scrub' would work freely on high performance ZFS pools. The only problem that there is no AMD/Nvidia drivers for Solaris that support hardware-assisted OpenCL. Is anyone interested in it? This isn't technically true. The NVIDIA drivers support compute, but there's other parts of the toolchain missing. /* I don't know about ATI/AMD, but I'd guess they likely don't support compute across platforms */ /* Disclaimer - The company I work for has a working HMPP compiler for Solaris/FreeBSD and we may soon support CUDA */ On 10 May 2011, at 16:44, Hung-Sheng Tsao (LaoTsao) Ph. D. wrote: IMHO, zfs need to run in all kind of HW T-series CMT server that can help sha calculation since T1 day, did not see any work in ZFS to take advantage it That support would be in the crypto framework though, not ZFS per se. So I think the OP might consider how best to add GPU support to the crypto framework. Chris ___ Thanks. Fred -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of David Magda Sent: 星期二, 六月 28, 2011 9:23 To: Bill Sommerfeld Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Encryption accelerator card recommendations. On Jun 27, 2011, at 18:32, Bill Sommerfeld wrote: On 06/27/11 15:24, David Magda wrote: Given the amount of transistors that are available nowadays I think it'd be simpler to just create a series of SIMD instructions right in/on general CPUs, and skip the whole co-processor angle. see: http://en.wikipedia.org/wiki/AES_instruction_set Present in many current Intel CPUs; also expected to be present in AMD's Bulldozer based CPUs. Now compare that with the T-series stuff that also handles 3DES, RC4, RSA2048, DSA, DH, ECC, MD5
Re: [zfs-discuss] Encryption accelerator card recommendations.[GPU acceleration of ZFS]
On Jun 27, 2011, at 22:03, Fred Liu wrote: FYI There is another thread named -- GPU acceleration of ZFS in this list to discuss the possibility to utilize the power of GPGPU. I posted here: In a similar vein I recently came across SSLShader: http://shader.kaist.edu/sslshader/ http://www.usenix.org/event/nsdi11/tech/full_papers/Jang.pdf http://www.google.com/search?q=sslshader This could be handy for desktops doing ZFS crypto (and even browser SSL and/or SSH), but few servers have decent graphics cards (and SPARC systems don't even have video ports by :). ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On Jun 27, 2011 9:24 PM, David Magda dma...@ee.ryerson.ca wrote: AESNI is certain better than nothing, but RSA, SHA, and the RNG would be nice as well. It'd also be handy for ZFS crypto in addition to all the network IO stuff. The most important reason for AES-NI might be not performance but defeating side-channel attacks. Also, really fast AES HW makes AES-based hash functions quite tempting. No, AES-NI is nothing to sneeze at. Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption accelerator card recommendations.
On Jun 27, 2011 4:15 PM, David Magda dma...@ee.ryerson.ca wrote: The (Ultra)SPARC T-series processors do, but to a certain extent it goes against a CPU manufacturers best (financial) interest to provide this: crypto is very CPU intensive using 'regular' instructions, so if you need to do a lot of it, it would force you to purchase a manufacturer's top-of-the-line CPUs, and to have as many sockets as you can to handle a load (and presumably you need to do useful work besides just en/decrypting traffic). I hope no CPU vendor thinks about the economics of chip making that way. I actually doubt any do. Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
Btw, if you want a commercially supported and maintained product, have you looked at NexentaStor? Regardless of what happens with OpenSolaris, we aren't going anywhere. (Full disclosure: I'm a Nexenta Systems employee. :-) -- Garrett Hi Garrett, I would like to know why you think Nexenta would continue to stay if OpenSolaris goes away. I feel the fate of Nexenta is no different than the fate of my startup company. Both of us are heavily dependent on zfs. And we know OpenSolaris version of zfs is the most stable version. Any business that is dependent on zfs must plan for two things as a contingency: 1. Look for an alternative for zfs 2. Look for an alternative for OpenSolaris Preferably both need to be open source with no licenses attached. Ideally, zfs lawsuit will be put to rest and Oracle will commit for continuing to support OpenSolaris. Regards, Peter -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Wed, 2010-07-14 at 01:06 -0700, Peter Taps wrote: Btw, if you want a commercially supported and maintained product, have you looked at NexentaStor? Regardless of what happens with OpenSolaris, we aren't going anywhere. (Full disclosure: I'm a Nexenta Systems employee. :-) -- Garrett Hi Garrett, I would like to know why you think Nexenta would continue to stay if OpenSolaris goes away. I feel the fate of Nexenta is no different than the fate of my startup company. Both of us are heavily dependent on zfs. And we know OpenSolaris version of zfs is the most stable version. Any business that is dependent on zfs must plan for two things as a contingency: 1. Look for an alternative for zfs 2. Look for an alternative for OpenSolaris Preferably both need to be open source with no licenses attached. Ideally, zfs lawsuit will be put to rest and Oracle will commit for continuing to support OpenSolaris. Regards, Peter Nexenta is investing in the technology within OpenSolaris heavily -- and we are working on an effort which will decouple our product from a hard dependency on bits from Oracle's Solaris. I can't talk too much about this right now, but I will probably have a lot more to say about this early next month. Upshot of this is is that if OpenSolaris goes away, we will continue to be able to work with the source code we have (plus in house source code we are creating to replace certain bits we don't have source to today!), so that our future is not so dependent on Oracle's continued good will. We're also hiring in engineering and growing a world class kernel team, so we can continue to sustain and improve the code even if Oracle were to pull the plug on OpenSolaris sources. We already have innovations in our code base which Oracle's tree lacks -- even though we have made our sources for the OS available for Oracle. That said, I feel extremely confident that while Oracle *may* pull the plug on this OpenSolaris thing, the *source* code which makes up the critical platform bits (ON) for both Solaris and OpenSolaris will probably continue to be released as source code going forward. There are simply too many positive benefits to its customers, and frankly too many customers would leave Oracle (not just Solaris, but go from Oracle to DB2) if Oracle were to pull a stunt such as ceasing the delivery of source code. (And further, even though the request-sponsor process has been a dismal failure, I suspect that there will always be a way for contributors to send improvements back to Oracle, even if the formal OpenSolaris process for it were to go away.) Now, the question of ZFS is IMO a lot more concerning. If we were suddenly unable to continue to work with ZFS due to patent considerations, there is no question that this would have a devastating impact on our business. It would have a devastating impact on Solaris too. We have mitigation plans in place for that eventuality which I cannot discuss, but it would be extremely painful to us, as well as everyone else in this ecosystem. The good news is that so far it seems likely that NetApp's suit will fail. - Garrett ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Wed, 14 Jul 2010, Peter Taps wrote: Any business that is dependent on zfs must plan for two things as a contingency: 1. Look for an alternative for zfs 2. Look for an alternative for OpenSolaris The existing OpenSolaris and zfs code bases are quite viable products today. If Oracle falters, there are lots of people familiar with SunOS source code and kernel development and it seems quite likely that OpenSolaris distributions would continue just as many other OS distributions do today (but under a new name). At the moment we are in a wait and see scenario, and it is not quite necessary to fork and create a truely independent distribution just yet. OpenSolaris is currently not left any more in the lurch than Solaris 10 is since paying Solaris 10 users are still wondering what happened to the U9 release which Sun had already promised. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
While we're on the topic, has anyone used ZFS much with Vormetric's encryption product? Any feedback? Doug Linder -- Learn more about Merchant Link at www.merchantlink.com. THIS MESSAGE IS CONFIDENTIAL. This e-mail message and any attachments are proprietary and confidential information intended only for the use of the recipient(s) named above. If you are not the intended recipient, you may not print, distribute, or copy this message or any attachments. If you have received this communication in error, please notify the sender by return e-mail and delete this message and any attachments from your computer. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
Freddie Cash wrote: You definitely want to do the ZFS bits from within FreeBSD. Why not using ZFS in OpenSolaris? At least it has most stable/tested implementation and also the newest one if needed? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
on 11/07/2010 14:21 Roy Sigurd Karlsbakk said the following: I'm planning on running FreeBSD in VirtualBox (with a Linux host) and giving it raw disk access to four drives, which I plan to configure as a raidz2 volume. Wouldn't it be better or just as good to use fuse-zfs for such a configuration? I/O from VirtualBox isn't really very good, but then, I haven't tested the linux/fbsd configuration... Hmm, an unexpected question IMHO - wouldn't it better to just install FreeBSD on the hardware? :-) If an original poster is using Linux as a host OS, then probably he has some very good reason to do that. But performance and etc -wise, directly using FreeBSD, of course, should win over fuse-zfs. Right? [Installing and maintaining one OS instead of two is the first thing that comes to mind] -- Andriy Gapon ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Freddie Cash ZFS-FUSE is horribly unstable, That may be true. I couldn't say. although that's more an indication of the stability of the storage stack on Linux. But this, I take issue with. Ext3 isn't unstable. It may not be as awesome as ZFS, but it isn't unstable. And the same goes for all the other storage and filesystems in Linux. Years ago, when I first started using Gimp, it was stable in Linux and not stable in Windows. I went to gimp.org, and looked at the FAQ's, and FAQ #1 said Gimp is unstable on windows. And the reply was Everything is unstable on Windows. Which was simply false. It was a developer (or group of developers) expressing a biased point of view, and blaming a platform that they were not fond of. In later versions of gimp, even on the same version of Windows, gimp evolved into something that was stable. To simply blame gimp, or simply blame windows, both positions are not accurate. The truth is, gimp was stable on linux, and other apps were stable on windows. The truth is, there was something unstable in the interaction between the two. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Mon, Jul 12, 2010 at 10:00 AM, Garrett D'Amore garr...@nexenta.comwrote: Btw, if you want a commercially supported and maintained product, have you looked at NexentaStor? Regardless of what happens with OpenSolaris, we aren't going anywhere. (Full disclosure: I'm a Nexenta Systems employee. :-) I'm trying to decide for myself when I'll give up on Oracle releasing another dev or release build and moving to something like Nexenta Core. I actually *like* the Solaris user space, so GNU/Debian userspace isn't that compelling for me. I see it enough at work that using something different at home is novel and helps keep me honest. I also don't see a roadmap for the upcoming releases or what release of Debian or Ubuntu they'll be based on. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Mon, 2010-07-12 at 12:55 -0700, Brandon High wrote: On Mon, Jul 12, 2010 at 10:00 AM, Garrett D'Amore garr...@nexenta.com wrote: Btw, if you want a commercially supported and maintained product, have you looked at NexentaStor? Regardless of what happens with OpenSolaris, we aren't going anywhere. (Full disclosure: I'm a Nexenta Systems employee. :-) I'm trying to decide for myself when I'll give up on Oracle releasing another dev or release build and moving to something like Nexenta Core. I actually *like* the Solaris user space, so GNU/Debian userspace isn't that compelling for me. The distinction is quickly shrinking. I think the trend has been to value compatibility with Linux over compatibility with legacy Solaris, at least for OpenSolaris and probably also for whatever next release of Solaris might be forthcoming. (At least at the shell/command line level. The *library* level -i.e. C API - is a totally different story, of course.) I see it enough at work that using something different at home is novel and helps keep me honest. I also don't see a roadmap for the upcoming releases or what release of Debian or Ubuntu they'll be based on. We have plans centered around 3.0.x and 3.1, and our plans for 4.0 are still forming. For 3.0. and 3.1, we will remain based on the same release of Ubuntu. For 4.0, there will be a major change, but the ultimate base of this is still under debate. I don't know if marketing has released any timelines yet, so I won't do so here. But you should contact our sales group if you want to find out more -- they can probably say more than I can. - Garrett -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
Garrett wrote: I don't know about ramifications (though I suspect that a broadening error scope would decrease ZFS' ability to isolate and work around problematic regions on the media), but one thing I do know. If you use FreeBSD disk encryption below ZFS, then you won't be able able to import your pools to another implementation -- you will be stuck with FreeBSD. This is an excellent point. Geli isn't a good option for me, then, though using encryption outside of the VM would still work. Btw, if you want a commercially supported and maintained product, have you looked at NexentaStor? Regardless of what happens with OpenSolaris, we aren't going anywhere. (Full disclosure: I'm a Nexenta Systems employee. :-) I probably ought to consider other OpenSolaris alternatives, like NexentaStor. (Though I'd be looking at the free version, not the commercial one: this is just for personal use, despite how careful I'm being with it. :) ) However (and please correct me if I'm wrong), isn't your future still tied to the future of OpenSolaris? The code is open, of course, but my understanding is that there isn't the same kind of developer community supporting OpenSolaris itself that you see with Linux (or even the BSDs). In other words, if Oracle stops development of OpenSolaris, there wouldn't be enough developers still working on it to keep it from stagnating. Or are you saying that you employ enough kernel hackers to keep up even without Oracle? (I am admittedly ignorant about the OpenSolaris developer community; this is all based on others' statements and opinions that I've read.) Michael ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Sun, Jul 11, 2010 at 11:51 AM, Michael Johnson mjjohnson@yahoo.com wrote: I'm planning on running FreeBSD in VirtualBox (with a Linux host) and giving it raw disk access to four drives, which I plan to configure as a raidz2 volume. On top of that, I'm considering using encryption. I understand that ZFS doesn't yet natively support encryption, so my idea was to set each drive up with full-disk encryption in the Linux host (e.g., using TrueCrypt or dmcrypt), mount the encrypted drives, and then give the virtual machine access to the virtual unencrypted drives. So the encryption would be transparent to FreeBSD. However, I don't know enough about ZFS to know if this is a good idea. I know that I need to specifically configure VirtualBox to respect cache flushes, so that data really is on disk when ZFS expects it to be. Would putting ZFS on top of full-disk encryption like this cause any problems? E.g., if the (encrypted) physical disk has a problem and as a result a larger chunk of the unencrypted data is corrupted, would ZFS handle that well? Are there any other possible consequences of this idea that I should know about? (I'm not too worried about any hits in performance; I won't be reading or writing heavily, nor in time-sensitive applications.) I should add that since this is a desktop I'm not nearly as worried about encryption as if it were a laptop (theft or loss are less likely), but encryption would still be nice. However, data integrity is the most important thing (I'm storing backups of my personal files on this), so if there's a chance that ZFS wouldn't handle errors well when on top of encryption, I'll just go without it. Thanks, Michael you can also create zfs on top of GELI[1][2] devices. Create the encrypted disks first and then use that to create zpool. Exact steps (assuming single disk, da1): - create the key # dd if=/dev/random of=/root/da1.key bs=64 count=1 - initialize GELI disk, if you want to only use the key as authentication method or automatically attach on boot, check the reference links for initialization and configuration (-K and -b) # geli init -s 4096 -K da1.key /dev/da1 - attach GELI disk # geli attach -k da1.key /dev/da1 - create zpool, either directly on geli disk or by creating it on top of GPT direct: # zpool create securepool da1.eli on top of GPT: # gpart create -s gpt da1.eli # gpart add -t freebsd-zfs da1.eli # zpool create securepool da1.elip1 - adjust rc.conf and loader.conf accordingly Another tutorial: http://forums.freebsd.org/showthread.php?t=2775 [1] http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/disks-encrypting.html [2] http://www.freebsd.org/cgi/man.cgi?query=geliapropos=0sektion=0manpath=FreeBSD+8.0-RELEASEformat=html -- O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
I'm planning on running FreeBSD in VirtualBox (with a Linux host) and giving it raw disk access to four drives, which I plan to configure as a raidz2 volume. Wouldn't it be better or just as good to use fuse-zfs for such a configuration? I/O from VirtualBox isn't really very good, but then, I haven't tested the linux/fbsd configuration... Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Sun, Jul 11, 2010 at 4:21 AM, Roy Sigurd Karlsbakk r...@karlsbakk.net wrote: I'm planning on running FreeBSD in VirtualBox (with a Linux host) and giving it raw disk access to four drives, which I plan to configure as a raidz2 volume. Wouldn't it be better or just as good to use fuse-zfs for such a configuration? I/O from VirtualBox isn't really very good, but then, I haven't tested the linux/fbsd configuration... ZFS-FUSE is horribly unstable, although that's more an indication of the stability of the storage stack on Linux. We've been testing it at work to see how dedupe support will affect our FreeBSD+ZFS storage servers. We can't keep it (Linux+ZFS) running for more than a few days. Drives drop off at random, the pool locks up, resilvers never complete. When it does work, it works nicely. It's just hard to keep it running. You definitely want to do the ZFS bits from within FreeBSD. -- Freddie Cash fjwc...@gmail.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
On Jul 11, 2010, at 5:11 PM, Freddie Cash fjwc...@gmail.com wrote: ZFS-FUSE is horribly unstable, although that's more an indication of the stability of the storage stack on Linux. Not really, more an indication of the pseudo-VFS layer implemented in fuse. Remember fuse provides it's own VFS API separate from the Linux VFS API so file systems can be implemented in user space. Fuse needs a little more work to handle ZFS as a file system. -Ross ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption?
on 11/07/2010 15:54 Andriy Gapon said the following: on 11/07/2010 14:21 Roy Sigurd Karlsbakk said the following: I'm planning on running FreeBSD in VirtualBox (with a Linux host) and giving it raw disk access to four drives, which I plan to configure as a raidz2 volume. Wouldn't it be better or just as good to use fuse-zfs for such a configuration? I/O from VirtualBox isn't really very good, but then, I haven't tested the linux/fbsd configuration... Like Freddie already mentioned, I'd heard that fuse-zfs wasn't really all that good of an option, and I wanted something that was more stable/reliable. Hmm, an unexpected question IMHO - wouldn't it better to just install FreeBSD on the hardware? :-) If an original poster is using Linux as a host OS, then probably he has some very good reason to do that. But performance and etc -wise, directly using FreeBSD, of course, should win over fuse-zfs. Right? [Installing and maintaining one OS instead of two is the first thing that comes to mind] I'm going with a virtual machine because the box I ended up building for this was way more powerful than I needed for just my file server; thus, I figured I'd use it as a personal machine too. (I wanted ECC RAM, and there just aren't that many motherboards that support ECC RAM that are also really cheap and low-powered.) And since I'm much more comfortable with Linux, I wanted to use it for the personal side of things. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Encryption?
I'm planning on running FreeBSD in VirtualBox (with a Linux host) and giving it raw disk access to four drives, which I plan to configure as a raidz2 volume. On top of that, I'm considering using encryption. I understand that ZFS doesn't yet natively support encryption, so my idea was to set each drive up with full-disk encryption in the Linux host (e.g., using TrueCrypt or dmcrypt), mount the encrypted drives, and then give the virtual machine access to the virtual unencrypted drives. So the encryption would be transparent to FreeBSD. However, I don't know enough about ZFS to know if this is a good idea. I know that I need to specifically configure VirtualBox to respect cache flushes, so that data really is on disk when ZFS expects it to be. Would putting ZFS on top of full-disk encryption like this cause any problems? E.g., if the (encrypted) physical disk has a problem and as a result a larger chunk of the unencrypted data is corrupted, would ZFS handle that well? Are there any other possible consequences of this idea that I should know about? (I'm not too worried about any hits in performance; I won't be reading or writing heavily, nor in time-sensitive applications.) I should add that since this is a desktop I'm not nearly as worried about encryption as if it were a laptop (theft or loss are less likely), but encryption would still be nice. However, data integrity is the most important thing (I'm storing backups of my personal files on this), so if there's a chance that ZFS wouldn't handle errors well when on top of encryption, I'll just go without it. Thanks, Michael ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Encryption
So far I'm using file container encryption using TrueCrypt on the client, but I would seriously like native encryption support on Solaris itself, especially in ZFS. From http://hub.opensolaris.org/bin/view/Project+zfs-crypto/ I see it's hopefully coming in Q1 2010? Are there any alternatives people use currently? Matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption
Matthew Carras wrote: So far I'm using file container encryption using TrueCrypt on the client, but I would seriously like native encryption support on Solaris itself, especially in ZFS. From http://hub.opensolaris.org/bin/view/Project+zfs-crypto/ I see it's hopefully coming in Q1 2010? Are there any alternatives people use currently? http://blogs.sun.com/darren/entry/encrypting_zfs_pools_using_lofi This is the basis of the the disk encryption used in ISC - Immutable Service Containers until the real ZFS crypto is finished: http://kenai.com/projects/isc/pages/Home -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption
Roger wrote: Hello, I am new to Solaris. Several PDFs out there suggest any of the following: a) Solaris comes with 128bit encryption (full filesystem) b) Solaris supports full root encryption. Can you send a pointer to these please, because the information is not correct and I would like to try and get it corrected. Any truth to any of this? The company I work for tis mandating full root encryption. Why is it mandated, is there no exception process ? It isn't currently part of the ZFS Crypto project to provide for an encrypted boot (ie root) filesystem. Part of the reason for this is because of the changes needed for GRUB (x86) and OBP (SPARC) and I would rather wait until we move to GRUB2 as somethings will be much easier. For ZFS pools that do not have the boot file system on them you can have all filesystems in the pool encrypted ie: # zpool create -O encryption=on tank c0t0d0s0 Even if you need to boot from a filesystem in the pool you *can* still have the swap ZVOL encrypted. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Encryption
Hello, I am new to Solaris. Several PDFs out there suggest any of the following: a) Solaris comes with 128bit encryption (full filesystem) b) Solaris supports full root encryption. Any truth to any of this? The company I work for tis mandating full root encryption. Thanks. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption
On Jul 20, 2009, at 15:54, Roger wrote: Several PDFs out there suggest any of the following: a) Solaris comes with 128bit encryption (full filesystem) b) Solaris supports full root encryption. Any truth to any of this? The company I work for tis mandating full root encryption. Part (a) is in-progress: http://opensolaris.org/os/project/zfs-crypto/ http://opensolaris.org/os/project/zfs-crypto/phase1/ There is some support for encryption with loop-back mounts: http://blogs.sun.com/darren/entry/opensolaris_disk_encryption_in_snv Part (b) has been considered, but there is no ETA AFAIK. Note that this is part of the OpenSolaris project, and not yet integrated in Solaris proper at this time. Out of curiosity, what operating system(s) currently support full root encryption? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Encryption through compression?
Hello everyone, My understanding is that the ZFS crypto framework will not release until 2010. In light of that, I'm wondering if the following approach to encryption could make sense for some subset of users: The idea is to use the compression framework to do both compression and encryption in one pass. This would be done by defining a new compression type, which might be called compress-encrypt or something like that. There could be two levels, one that does both compress and encrypt and another that does encrypt only. I see the following issues with this approach: 1. ZFS compression framework presently takes compressed data only if there was at least 12.5% reduction. For data that didn't compress, you would wind up storing it unencrypted, even if encryption was on. 2. Meta-data would not be encrypted. I.e., even if you don't have the key, you will be able to do directory listings and see file names, etc. 3. There is no key management framework. I would deal with these as follows: Issue #1 can be solved by changing ZFS code such that it always accepts the compressed data. I guess this is an easy change. Issue #2 may be a limitation to some and feature to others. May be OK. Issue #3 can be solved using encryption hardware (which my company happens to make). The keys are stored in hardware and can be used directly from that. Of course, this means that the solution will be specific to our hardware, but that's fine by me. The idea is that we would do this project on our own and supply this modified ZFS with our compression/encryption hardware to our customers. We may submit the patch for inclusion in some future version of OS, if the developers are amenable to that. Does anyone see any problems with this? There are probably various gotchas here that I haven't thought of. If you can think of any, please let me know. Thanks, Monish Monish Shah CEO, Indra Networks, Inc. www.indranetworks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption through compression?
Monish Shah wrote: Hello everyone, My understanding is that the ZFS crypto framework will not release until 2010. That is incorrect information, where did you get that from ? In light of that, I'm wondering if the following approach to encryption could make sense for some subset of users: The idea is to use the compression framework to do both compression and encryption in one pass. This would be done by defining a new compression type, which might be called compress-encrypt or something like that. There could be two levels, one that does both compress and encrypt and another that does encrypt only. I see the following issues with this approach: 1. ZFS compression framework presently takes compressed data only if there was at least 12.5% reduction. For data that didn't compress, you would wind up storing it unencrypted, even if encryption was on. 2. Meta-data would not be encrypted. I.e., even if you don't have the key, you will be able to do directory listings and see file names, etc. 3. There is no key management framework. That is impossible there has to be key management somewhere. I would deal with these as follows: Issue #1 can be solved by changing ZFS code such that it always accepts the compressed data. I guess this is an easy change. Issue #2 may be a limitation to some and feature to others. May be OK. Issue #3 can be solved using encryption hardware (which my company happens to make). The keys are stored in hardware and can be used directly from that. Of course, this means that the solution will be specific to our hardware, but that's fine by me. The idea is that we would do this project on our own and supply this modified ZFS with our compression/encryption hardware to our customers. We may submit the patch for inclusion in some future version of OS, if the developers are amenable to that. If it is specific to your companies hardware I doubt it would ever get integrated into OpenSolaris particularly given the existing zfs-crypto project has no hardware dependencies at all. The better way to use your encryption hardware is to get it plugged into the OpenSolaris cryptographic framework (see the crypto project on OpenSolaris.org) Does anyone see any problems with this? There are probably various gotchas here that I haven't thought of. If you can think of any, please let me know. The various gotchas are the things that have been taking me and the rest of the ZFS team a large part of the zfs-crypto project to resolve. It really isn't as simple as you think it is - if it were then the zfs-crypto project would be done by now! If you really want to help get encryption for ZFS then please come and join the already existing project rather than starting another one from scratch. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption through compression?
Monish Shah wrote: Hello Darren, Monish Shah wrote: Hello everyone, My understanding is that the ZFS crypto framework will not release until 2010. That is incorrect information, where did you get that from ? It was in Mike Shapiro's presentation at the Open Solaris Storage Summit that took place a couple of weeks ago. Perhaps I mis-read the slide, but I'm pretty sure it listed encryption as a feature for 2010. That is for its availablity in the S7000 appliance. It will be in OpenSolaris before that (it has to be because the S7000 is based on an OpenSolaris build). If the schedule is much sooner than 2010, I would definitely do so. What is your present schedule estimate? I can't commit to this yet but I expect somewhere around August 2009. Note that the code in hg.opensolaris.org/hg/zfs-crypto/gate actually works today and encrypts more than what your proposal work. It is just that we are making some design changes to simplify the model and ensure that encryption integrates with other ZFS features coming along. There will be a design update posted to zfs-crypto-discuss@ later this month. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption through compression?
Hello Darren, Monish Shah wrote: Hello everyone, My understanding is that the ZFS crypto framework will not release until 2010. That is incorrect information, where did you get that from ? It was in Mike Shapiro's presentation at the Open Solaris Storage Summit that took place a couple of weeks ago. Perhaps I mis-read the slide, but I'm pretty sure it listed encryption as a feature for 2010. ... 3. There is no key management framework. That is impossible there has to be key management somewhere. What I meant was, the compression framework does not have key management framework. Using our hardware (which I mentioned later in my mail), the key management would come with the hardware, since we store keys in the hardware. We provide a utility to manage the keys stored in the hardware. ... If it is specific to your companies hardware I doubt it would ever get integrated into OpenSolaris particularly given the existing zfs-crypto project has no hardware dependencies at all. The better way to use your encryption hardware is to get it plugged into the OpenSolaris cryptographic framework (see the crypto project on OpenSolaris.org) That was precisely what I want thinking originally. However, if it is out in 2010, there is temptation to do our own project, which I thought could be done in a couple of months. (In light of your comment below, my estimate may have been wildly optimistic, but the foregoing is merely an explanation of what I was thinking.) Does anyone see any problems with this? There are probably various gotchas here that I haven't thought of. If you can think of any, please let me know. The various gotchas are the things that have been taking me and the rest of the ZFS team a large part of the zfs-crypto project to resolve. It really isn't as simple as you think it is - if it were then the zfs-crypto project would be done by now! If you really want to help get encryption for ZFS then please come and join the already existing project rather than starting another one from scratch. If the schedule is much sooner than 2010, I would definitely do so. What is your present schedule estimate? -- Darren J Moffat Monish ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Encryption on ZFS / Disk Usage
Hi 2006/8/22, Constantin Gonzalez [EMAIL PROTECTED]: Thomas Deutsch wrote: I'm thinking about to change from Linux/Softwareraid to OpenSolaris/ZFS. During this, I've got some (probably stupid) questions: don't worry, there are no stupid questions :). 1. Is ZFS able to encrypt all the data? If yes: How safe is this encryption? I'm currently using dm-crypt on linux for doing this. Encryption for ZFS is a planned feature, but not available now. See: http://www.opensolaris.org/os/project/zfs-crypto/ Another project is an encrypted loopback device called xlofi which can be used on top of ZFS: http://www.opensolaris.org/os/community/security/projects/xlofi/ I understand that both approaches are independent of the encryption mechanism, so one would be free to choose a suitably safe cypher that is supported by the Solaris Cryptographic Framework. Thanks for this informaton. I'm waiting for encryption until I would use ZFS. 2. How big is the usable diskspace? I know that a rai5 is using the space of one disk for parity informations. A raid5 with four disk of 300GB has 900GB Space. How is it with ZFS? How much space do I have in this case? ZFS' RAIDZ1 uses one parity disk per RAIDZ set, similarly to RAID-5. ZFS' RAIDZ2 uses two parity disks per RAIDZ set. This means that RAIDZ2 allows problems with two disks? thanks, Thomas ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
