Re: swap on encrypted softraid, performance penalty?

[Fredrik Alm]

> On 21 May 2015, at 08:48, Ján Kušniar  wrote:
> 
>> I think you will find that hibernate doesn’t work with this setup if you try 
>> it.
>> 
>> I found this write-up explaining a little better:
>> http://undeadly.org/cgi?action=article&sid=20131112031806
>> 
>> Seems double-encrypted swap or dual swap partitions is the way to go if you 
>> want hibernate
>> to work and don’t want to recompile the kernel. I’ll start by trying out the 
>> double-encrypted
>> swap, since I won’t be running heavy loads on this machine and only have a 
>> 128gb ssd in it.
>> 
> 
> Nope. I have fully working hibernate on T430 with swap on softraid volume as 
> "b" partition
> and swap encryption disabled. There's only "a" partition on physical drive:

That’s what we have already confirmed to work. It’s when you put the swap on 
another partition
than the encrypted disk containing the / partition that it won’t work without 
reconfiguration.

Re: swap on encrypted softraid, performance penalty?

[Ján Kušniar]

> I think you will find that hibernate doesn’t work with this setup if you try 
> it.
> 
> I found this write-up explaining a little better:
> http://undeadly.org/cgi?action=article&sid=20131112031806
> 
> Seems double-encrypted swap or dual swap partitions is the way to go if you 
> want hibernate
> to work and don’t want to recompile the kernel. I’ll start by trying out the 
> double-encrypted
> swap, since I won’t be running heavy loads on this machine and only have a 
> 128gb ssd in it.
> 

Nope. I have fully working hibernate on T430 with swap on softraid volume as 
"b" partition
and swap encryption disabled. There's only "a" partition on physical drive:

$ disklabel sd1 (physical drive)
#size   offset  fstype [fsize bsize  cpg]
  a:468856961   64RAID   
  c:4688621280  unused  

$ disklabel sd2 (softraid volume)
#size   offset  fstype [fsize bsize  cpg]
  a:  2097152   64  4.2BSD   2048 163841 # /
  b: 33427536  2097216swap   # none
  c:4688564330  unused   
  d: 41940640 35524768  4.2BSD   2048 163841 # /var
  e:  4192960 77465408  4.2BSD   2048 163841 # /usr
  f:  2088448 81658368  4.2BSD   2048 163841 # /usr/X11R6
  g: 20964832 83746816  4.2BSD   2048 163841 # /usr/local
  h:364129280104711680  4.2BSD   4096 327681 # /home


System is default 5.7 stable without kernel reconfiguration.


J.

Re: swap on encrypted softraid, performance penalty?

[Fredrik Alm]

I think you will find that hibernate doesn’t work with this setup if you try it.

I found this write-up explaining a little better:
http://undeadly.org/cgi?action=article&sid=20131112031806

Seems double-encrypted swap or dual swap partitions is the way to go if you 
want hibernate
to work and don’t want to recompile the kernel. I’ll start by trying out the 
double-encrypted
swap, since I won’t be running heavy loads on this machine and only have a 
128gb ssd in it.


> On 19 May 2015, at 21:48, Jonathan Thornburg  wrote:
> 
> In message ,
> Fredrik Alm  asked about how to handle the
> swap partition when using whole-disk softraid crypto:
>> I've seen a few 'whole disk encryption' tutorials which puts the
>> swap outside of the partition used for the softraid encryption,
>> since openbsd already encrypts the swap partition anyway. I assume
>> that by putting the swap inside the encrypte d partition, there
>> will be performance penalties because encryption is done twice?
>> could someone shed a little light on this issue?
> 
> In message 
> dan mclaughlin  replied
> | where did you see those tutorials? i attempted this some months ago
> | (6-7) and it was not possible to have swap outside of the softraid.
> | i forget what the exact problem was (i should have taken better
> | notes...). i believe the system wouldn't boot properly, and i think
> | it was because the swap partition was on a different device.
> and later in the thread
> | honestly though, i don't know how the guy who wrote that tutorial got it to
> | work (if in fact he did...), i remember it being completely unworkable. i
> | think the only option was to rebuild the kernel, as you said, which really
> | isn't an option.
> 
> In message 
> Stefan Sperling  replied
> # Keeping swap on the same disk as the root filesystem has some advantages.
> # For historical reasons the system expects this in various places.
> # More things (such as hibernate) will work out of the box this way.
> 
> I can report that as of 5.6-stable/amd64, it *is* possible to have
> swap outside the softraid.  I currently have this configuration running
> on a pair of Thinkpad T60 laptops, and I'm fully satisfied with it.
> Suspend-to-RAM works fine; I haven't tried hibernate.
> 
> 
> 
> For this configuration, I wanted separate softraid-crypto partitions
> for the OS and for /home.
> 
> After a few false starts, I settled on the following layout:
> 
>  sd0
>  ---   
>   |  a-+- (sd1) softraid crypt, size = 44.5G
>   || a = root   256M
>   || d = root2  256M
>   || e = var2G
>   || f = var2   2G
>   || g = usr20G
>   || h = usr2   20G
>   |   -+-
>   |  b   swap   6G
>   |  j-+- (sd2) softraid crypt, size = all remaining space
>   || j = home
>  ---  -+-
> 
> sd0 is the physical disk
> It has 3 openbsd-partitions: a, b, and j
> 
> sd1 is a softraid-crypto disk living inside sd0a.  sd1 stores all the
> OS partitions, currently 5.6-stable in my case.
>   [In my case there are actually two sets of OS partitions,
>   but at present I'm only using the a,e,g root,var,usr ones.
>   The others are for future use as backups, in the same manner
>   as I described (for an older OpenBSD system) in message
>   .]
> 
> sd0b is the swap partition
> 
> sd2 is a softraid-crypto disk living inside sd0j.  sd2 stores /home.
> 
> 
> 
> Setting this up took a little bit of tinkering, but with a bit of guru
> help on misc@, everything eventually came out fine.  Here's the procedure
> that eventually worked, starting from a new-from-the-factory disk just
> installed into the laptop:
> 
> boot from 5.6 CD
> Install, Upgrade, Autoinstall, or Shell --> Shell
> 
> maybe type some commands so the kernel can accumulate some of entropy
> in the random-number subsystem
> 
> fill the entire disk with random data:
> (--> later steps won't leak which blocks have been written)
> (for a big disk this may take a day or so)
> 
>   # dd if=/dev/arandom bs=1m of=/dev/sd0c
> 
> I want to use the entire physical disk for OpenBSD:
> 
>   # fdisk -i sd0
> 
>   # disklabel -E sd0
>   add partitions
>   a @  offset 128, size 93323264 sectors, type RAID
>   bsize 6G, type swap
>   jsize everything-left, type RAID
> 
> now create softraid-crypto sd1
> 
>   # cd /dev
>   # sh MAKEDEV sd1
>   # dd if=/dev/zero bs=1m count=1 of=/dev/rsd0a
>   # bioctl -c C -r 10 -l /dev/sd0a softraid0
>   (enter sd1 passphrase)
>   (enter sd1 passphrase again)
> 
> This passphrase will be the boot passphrase.
> 
> Now install OpenBSD from the CD into sd1,
> 
>   # install
> 
> creating whatever OS partitions you like (in my case a,d,e,f,g,h,
> as noted above).  Two notes about this:  

Re: swap on encrypted softraid, performance penalty?

[Jonathan Thornburg]

In message ,
Fredrik Alm  asked about how to handle the
swap partition when using whole-disk softraid crypto:
> I've seen a few 'whole disk encryption' tutorials which puts the
> swap outside of the partition used for the softraid encryption,
> since openbsd already encrypts the swap partition anyway. I assume
> that by putting the swap inside the encrypte d partition, there
> will be performance penalties because encryption is done twice?
> could someone shed a little light on this issue?

In message 
dan mclaughlin  replied
| where did you see those tutorials? i attempted this some months ago
| (6-7) and it was not possible to have swap outside of the softraid.
| i forget what the exact problem was (i should have taken better
| notes...). i believe the system wouldn't boot properly, and i think
| it was because the swap partition was on a different device.
and later in the thread
| honestly though, i don't know how the guy who wrote that tutorial got it to
| work (if in fact he did...), i remember it being completely unworkable. i
| think the only option was to rebuild the kernel, as you said, which really
| isn't an option.

In message 
Stefan Sperling  replied
# Keeping swap on the same disk as the root filesystem has some advantages.
# For historical reasons the system expects this in various places.
# More things (such as hibernate) will work out of the box this way.

I can report that as of 5.6-stable/amd64, it *is* possible to have
swap outside the softraid.  I currently have this configuration running
on a pair of Thinkpad T60 laptops, and I'm fully satisfied with it.
Suspend-to-RAM works fine; I haven't tried hibernate.



For this configuration, I wanted separate softraid-crypto partitions
for the OS and for /home.

After a few false starts, I settled on the following layout:

  sd0
  ---   
   |  a-+- (sd1) softraid crypt, size = 44.5G
   || a = root   256M
   || d = root2  256M
   || e = var2G
   || f = var2   2G
   || g = usr20G
   || h = usr2   20G
   |   -+-
   |  b   swap   6G
   |  j-+- (sd2) softraid crypt, size = all remaining space
   || j = home
  ---  -+-

sd0 is the physical disk
It has 3 openbsd-partitions: a, b, and j

sd1 is a softraid-crypto disk living inside sd0a.  sd1 stores all the
OS partitions, currently 5.6-stable in my case.
[In my case there are actually two sets of OS partitions,
but at present I'm only using the a,e,g root,var,usr ones.
The others are for future use as backups, in the same manner
as I described (for an older OpenBSD system) in message
.]

sd0b is the swap partition

sd2 is a softraid-crypto disk living inside sd0j.  sd2 stores /home.



Setting this up took a little bit of tinkering, but with a bit of guru
help on misc@, everything eventually came out fine.  Here's the procedure
that eventually worked, starting from a new-from-the-factory disk just
installed into the laptop:

boot from 5.6 CD
Install, Upgrade, Autoinstall, or Shell --> Shell

maybe type some commands so the kernel can accumulate some of entropy
in the random-number subsystem

fill the entire disk with random data:
(--> later steps won't leak which blocks have been written)
(for a big disk this may take a day or so)

   # dd if=/dev/arandom bs=1m of=/dev/sd0c

I want to use the entire physical disk for OpenBSD:

   # fdisk -i sd0

   # disklabel -E sd0
   add partitions
   a @  offset 128, size 93323264 sectors, type RAID
   bsize 6G, type swap
   jsize everything-left, type RAID

now create softraid-crypto sd1

   # cd /dev
   # sh MAKEDEV sd1
   # dd if=/dev/zero bs=1m count=1 of=/dev/rsd0a
   # bioctl -c C -r 10 -l /dev/sd0a softraid0
   (enter sd1 passphrase)
   (enter sd1 passphrase again)

This passphrase will be the boot passphrase.

Now install OpenBSD from the CD into sd1,

   # install

creating whatever OS partitions you like (in my case a,d,e,f,g,h,
as noted above).  Two notes about this:  First, put the root partition
("a") at offset 256 as per Christian Weisgerber 's
super-helpful comments in message
.
And second, don't create either a swap partition ("b")
or a /home partition at this point -- those will come later.

Now boot the newly-installed system (this will require entering the
boot passphrase, of course).  Once it's up and running, edit /etc/fstab
to add sd0b as a swap partition:

   /dev/sd0b   none  swap  sw  0 0

Now setup up softraid-crypto sd2 to hold /home

   # dd if=/dev/zero bs=1m count=1 of=/dev/rsd0j
   # bioctl -c C -r 10 -l /dev/sd0j softraid0
   (enter sd2 passphrase)
   (enter sd2 passph

Re: swap on encrypted softraid, performance penalty?

[Ted Unangst]

dan mclaughlin wrote:
> in the end i found it easier to just leave it all in the softraid for other
> reasons in addition to that issue. as to swap encryption, i disabled it. no
> need to encrypt twice.
> 

to the contrary, uvm swap encrypt does a better job of expiring keys and
making old data unrecoverable.

Re: swap on encrypted softraid, performance penalty?

[Stefan Sperling]

On Sun, May 17, 2015 at 12:20:52AM +0200, Fredrik Alm wrote:
> I’ve seen a few “whole disk encryption” tutorials which puts the swap outside 
> of the partition used for the softraid encryption, since openbsd already 
> encrypts the swap partition anyway. I assume that by putting the swap inside 
> the encrypted partition, there will be performance penalties because 
> encryption is done twice? could someone shed a little light on this issue?

Keeping swap on the same disk as the root filesystem has some advantages.
For historical reasons the system expects this in various places.
More things (such as hibernate) will work out of the box this way.

If you really need to avoid a performance hit on swap, I'd recommend
you add more memory to the system. If that's impossible you can add
an additional swap device from a non-softraid part of the disk and
set it to higher priority than the default swap. See swapctl(8).
The result could look something like this (sd2 being softraid crypto,
sd0 being a swap partiion on bare disk):

$ swapctl
Device  512-blocks UsedAvail Capacity  Priority
/dev/sd0b 167831360 16783136 0%0
/dev/sd2b 167718630 16771863 0%1
Total 335549990 33554999 0%

Also note that if your machine suports aesni (AES cpu feature flag in dmesg)
softraid encryption overhead is reduced by hardware crypto.

Re: swap on encrypted softraid, performance penalty?

[dan mclaughlin]

On Sun, 17 May 2015 04:32:38 +0200 Fredrik Alm  wrote:
> > On 17 May 2015, at 02:19, dan mclaughlin  wrote:
> > 
> > On Sun, 17 May 2015 00:20:52 +0200 Fredrik Alm  wrote:
> >> I’ve seen a few “whole disk encryption”
> >> tutorials which puts the swap outside of the partition used for the 
> >> softraid
> >> encryption, since openbsd already encrypts the swap partition anyway. I
> >> assume that by putting the swap inside the encrypted partition, there will
> >> be performance penalties because encryption is done twice? could someone
> >> shed a little light on this issue?
> >> 
> > 
> > where did you see those tutorials? i attempted this some months ago (6-7) 
> > and
> > it was not possible to have swap outside of the softraid. i forget what the
> > exact problem was (i should have taken better notes...). i believe the
> > system wouldn't boot properly, and i think it was because the swap partition
> > was on a different device.
> > 
> > in the end i found it easier to just leave it all in the softraid for other
> > reasons in addition to that issue. as to swap encryption, i disabled it. no
> > need to encrypt twice.
> 
> this is one of the tutorials: http://www.bsdnow.tv/tutorials/fde
> 
> I found that when the swap was on a different disk
> (sd0b instead of sd1b, with the rest of the encrypted stuff on the softraid 
> disk)
> the swap had to be added manually to the fstab and even then it was
> defaulted to /dev/sdb1 (which didn’t exist) for coredumps. I assume this is
> why ZZZ exited with a kernel error instead of hibernating when I tried this
> disklayout. When I just put everything including the swap on the softraid it
> worked like normal. I’ll just try turning the swap encryption off then, 
> seems
> easier than reconfiguring the kernel to use sd0b as a dump device.
> 

your experience sounds familiar (swap expected to be on the root device),
and is why i think i abandoned the attempt to put the swap outside the
partition. though i am pretty sure i had problems right at boot, not later.

honestly though, i don't know how the guy who wrote that tutorial got it to
work (if in fact he did...), i remember it being completely unworkable. i
think the only option was to rebuild the kernel, as you said, which really
isn't an option.

also, those instructions to use bioctl will only work if there has not been
a softraid crypto volume there previously. you need to clear the space via
dd as in bioctl(8).

Re: swap on encrypted softraid, performance penalty?

[dan mclaughlin]

On Sun, 17 May 2015 00:20:52 +0200 Fredrik Alm  wrote:
> I’ve seen a few “whole disk encryption”
> tutorials which puts the swap outside of the partition used for the softraid
> encryption, since openbsd already encrypts the swap partition anyway. I
> assume that by putting the swap inside the encrypted partition, there will
> be performance penalties because encryption is done twice? could someone
> shed a little light on this issue?
> 

where did you see those tutorials? i attempted this some months ago (6-7) and
it was not possible to have swap outside of the softraid. i forget what the
exact problem was (i should have taken better notes...). i believe the
system wouldn't boot properly, and i think it was because the swap partition
was on a different device.

in the end i found it easier to just leave it all in the softraid for other
reasons in addition to that issue. as to swap encryption, i disabled it. no
need to encrypt twice.

Re: swap on encrypted softraid, performance penalty?

[Fredrik Alm]

Yep, since my last mail I set it up on one big encrypted softraid, including 
the swap
and turned off swap encryption and created a key disk on usb instead of a 
password.
Works a lot better now and ZZZ works as it should (any ZZZ issues left are most 
likely
related to not yet supported hardware).


> On 17 May 2015, at 08:08, dan mclaughlin  wrote:
> 
> On Sun, 17 May 2015 04:32:38 +0200 Fredrik Alm  wrote:
>>> On 17 May 2015, at 02:19, dan mclaughlin  wrote:
>>> 
>>> On Sun, 17 May 2015 00:20:52 +0200 Fredrik Alm  wrote:
 I’ve seen a few “whole disk encryption”
 tutorials which puts the swap outside of the partition used for the 
 softraid
 encryption, since openbsd already encrypts the swap partition anyway. I
 assume that by putting the swap inside the encrypted partition, there will
 be performance penalties because encryption is done twice? could someone
 shed a little light on this issue?
 
>>> 
>>> where did you see those tutorials? i attempted this some months ago (6-7) 
>>> and
>>> it was not possible to have swap outside of the softraid. i forget what the
>>> exact problem was (i should have taken better notes...). i believe the
>>> system wouldn't boot properly, and i think it was because the swap partition
>>> was on a different device.
>>> 
>>> in the end i found it easier to just leave it all in the softraid for other
>>> reasons in addition to that issue. as to swap encryption, i disabled it. no
>>> need to encrypt twice.
>> 
>> this is one of the tutorials: http://www.bsdnow.tv/tutorials/fde
>> 
>> I found that when the swap was on a different disk
>> (sd0b instead of sd1b, with the rest of the encrypted stuff on the softraid 
>> disk)
>> the swap had to be added manually to the fstab and even then it was
>> defaulted to /dev/sdb1 (which didn’t exist) for coredumps. I assume this is
>> why ZZZ exited with a kernel error instead of hibernating when I tried this
>> disklayout. When I just put everything including the swap on the softraid it
>> worked like normal. I’ll just try turning the swap encryption off then, seems
>> easier than reconfiguring the kernel to use sd0b as a dump device.
>> 
> 
> your experience sounds familiar (swap expected to be on the root device),
> and is why i think i abandoned the attempt to put the swap outside the
> partition. though i am pretty sure i had problems right at boot, not later.
> 
> honestly though, i don't know how the guy who wrote that tutorial got it to
> work (if in fact he did...), i remember it being completely unworkable. i
> think the only option was to rebuild the kernel, as you said, which really
> isn't an option.
> 
> also, those instructions to use bioctl will only work if there has not been
> a softraid crypto volume there previously. you need to clear the space via
> dd as in bioctl(8).

Re: swap on encrypted softraid, performance penalty?

[Fredrik Alm]

> On 17 May 2015, at 02:19, dan mclaughlin  wrote:
> 
> On Sun, 17 May 2015 00:20:52 +0200 Fredrik Alm  wrote:
>> I’ve seen a few “whole disk encryption”
>> tutorials which puts the swap outside of the partition used for the softraid
>> encryption, since openbsd already encrypts the swap partition anyway. I
>> assume that by putting the swap inside the encrypted partition, there will
>> be performance penalties because encryption is done twice? could someone
>> shed a little light on this issue?
>> 
> 
> where did you see those tutorials? i attempted this some months ago (6-7) and
> it was not possible to have swap outside of the softraid. i forget what the
> exact problem was (i should have taken better notes...). i believe the
> system wouldn't boot properly, and i think it was because the swap partition
> was on a different device.
> 
> in the end i found it easier to just leave it all in the softraid for other
> reasons in addition to that issue. as to swap encryption, i disabled it. no
> need to encrypt twice.

this is one of the tutorials: http://www.bsdnow.tv/tutorials/fde

I found that when the swap was on a different disk
(sd0b instead of sd1b, with the rest of the encrypted stuff on the softraid 
disk)
the swap had to be added manually to the fstab and even then it was
defaulted to /dev/sdb1 (which didn’t exist) for coredumps. I assume this is
why ZZZ exited with a kernel error instead of hibernating when I tried this
disklayout. When I just put everything including the swap on the softraid it
worked like normal. I’ll just try turning the swap encryption off then, seems
easier than reconfiguring the kernel to use sd0b as a dump device.

swap on encrypted softraid, performance penalty?

[Fredrik Alm]

I’ve seen a few “whole disk encryption” tutorials which puts the swap outside 
of the partition used for the softraid encryption, since openbsd already 
encrypts the swap partition anyway. I assume that by putting the swap inside 
the encrypted partition, there will be performance penalties because encryption 
is done twice? could someone shed a little light on this issue?