Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Patrick Dohman]

Another feature to look for is spin down of the dedicated hot spare.

Go Vikings :)
Patrick

> On Feb 21, 2016, at 7:23 AM, Marcus MERIGHI  wrote:
>
> ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 21:05 (CET):
>> So glad to understand better what's in the box.
>>
>> Also please note that I'm not trying to suggest to implement lots of
>> crap, am perfectly clear that high security is correlated with low
>> complexity.
>>
>> On 2016-02-21 00:29, Marcus MERIGHI wrote:
>>> ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 16:43 (CET):
>> ..
>>> You appear to mean bioctl(8). Thats the only place I could find the word
>>> 'patrol'. bioctl(8) can control more than softraid(4) devices.
>>>
>>> bio(4):
>>>The following device drivers register with bio for volume
>>>   management:
>>>
>>>  ami(4) American Megatrends Inc. MegaRAID
>>> PATA/SATA/SCSI RAID controller
>>>  arc(4) Areca Technology Corporation SAS/SATA RAID
>>> controller
>>>  cac(4) Compaq Smart Array 2/3/4 SCSI RAID controller
>>>  ciss(4)Compaq Smart Array SAS/SATA/SCSI RAID
>>>   controller
>>>  ips(4) IBM SATA/SCSI ServeRAID controller
>>>  mfi(4) LSI Logic & Dell MegaRAID SAS RAID controller
>>>  mpi(4) LSI Logic Fusion-MPT Message Passing Interface
>>>  mpii(4)LSI Logic Fusion-MPT Message Passing Interface
>>>   II
>>>  softraid(4)Software RAID
>>>
>>> It is talking about controlling a HW raid controller, in that 'patrol'
>>> paragraph, isn't it?
>>
>> So by this you mean that patrolling is really implemented for
>> softraid??
>
> No, I said the opposite.
>
> I'm sure my english language capabilities are not perfect. But what you
> make of it is really surprising! (And even funny in the cabaret way.)
>
> I'll keep trying. But sooner or later we'll have to take this off list.
> Or to newbies. There you get help from the same people but without
> having your misinterpretations in the 'official' archives for other poor
> souls to find ;-)
>
> http://mailman.theapt.org/listinfo/openbsd-newbies
>
>> (Karel and Constantine don't agree??)
>>
>> So I just do.. "bioctl -t start sdX" wher sdX is the name of my softraid
>> device, and it'll do the "scrub" as in reading through all underlying
>
> bioctl(8) is clear, I think:
> -t patrol-function
>  Control the RAID card's patrol functionality, if
>  supported. patrol-function may be one of:
>
> Why do you think it will work for softraid(4) when it says it does for
> hardware-RAID?
>
> I have a theory: you have some experience with other Operating Systems
> and their built in help system that have led you to not fully read but
> just search/skim for keywords. Do yourself (and me) a favour and read
> them fully. Top to bottom. Take every word as put there thoughtfully,
> not in a hurry. You can find manpage content discussions all over the
> archives. manpages are taken seriously.
>
> Please repeat: bio(4)/bioctl(8) controls RAID devices. These can be in
> hardware or software. Some functions (-a, -b, -H, -t, -u) are only
> useable/usefull when controlling a hardware RAID. The manpage even gives
> direct clues on whether hardware- or software RAID is the topic. First
> synopsis, second synopsis. 'The options for RAID controllers are as
> follows:' (=hardware) 'In addition to the relevant options listed above,
> the options for softraid(4) devices are as follows:' (=software).
> Did you note the 'relevant' part? That word is there on purpose, I
> suppose. It is there to tell you that not all, but the relevant parts of
> the hardware RAID parameters also apply to software RAID (that comes
> below). I would consider '-v' relevant, '-a' ('Control the RAID card's
> alarm functionality, if supported') not.
>
> (Example: what '-a' does for hardware RAID can be done with sensorsd(8)
> for software RAID (=softraid(4)). Once a softraid volume is configured,
> you get 'hw.sensors.softraid0.drive0=online (sd1), OK'.
> Try 'sysctl hw.sensors.softraid0'.)
>
>> physical media to check its internal integrity so for RAID1C that will be
>> data readability and that checksums are correct, and "doas bioctl
> softraid0"
>> will show me the % status, and if I don't get any errors before it goes
> back
>> to normal it means the patrol was successful right?
>
> No idea, never had a hardware RAID controller.
>
>> (And as usual patrol is implemented to have the lowest priority, so it
>> should not interfere extreemely much with ordinary SSD softraid
operation.)
>
> I think the patrolling is done by the hardware RAID controller.
> bioctl(8) just commands it to do so.
>
 * Rebuild - I think I saw some console dump of the status of a rebuild
 process on the net, so MAYBE or NO..?
>>>
>>> That's what it looks like:
>>>
>>> $ 

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Marcus MERIGHI]

ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 21:05 (CET):
> So glad to understand better what's in the box.
>
> Also please note that I'm not trying to suggest to implement lots of
> crap, am perfectly clear that high security is correlated with low
> complexity.
>
> On 2016-02-21 00:29, Marcus MERIGHI wrote:
> >ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 16:43 (CET):
> ..
> >You appear to mean bioctl(8). Thats the only place I could find the word
> >'patrol'. bioctl(8) can control more than softraid(4) devices.
> >
> >bio(4):
> > The following device drivers register with bio for volume
> >management:
> >
> >   ami(4) American Megatrends Inc. MegaRAID
> >  PATA/SATA/SCSI RAID controller
> >   arc(4) Areca Technology Corporation SAS/SATA RAID
> >  controller
> >   cac(4) Compaq Smart Array 2/3/4 SCSI RAID controller
> >   ciss(4)Compaq Smart Array SAS/SATA/SCSI RAID
> >controller
> >   ips(4) IBM SATA/SCSI ServeRAID controller
> >   mfi(4) LSI Logic & Dell MegaRAID SAS RAID controller
> >   mpi(4) LSI Logic Fusion-MPT Message Passing Interface
> >   mpii(4)LSI Logic Fusion-MPT Message Passing Interface
> >II
> >   softraid(4)Software RAID
> >
> >It is talking about controlling a HW raid controller, in that 'patrol'
> >paragraph, isn't it?
>
> So by this you mean that patrolling is really implemented for
> softraid??

No, I said the opposite.

I'm sure my english language capabilities are not perfect. But what you
make of it is really surprising! (And even funny in the cabaret way.)

I'll keep trying. But sooner or later we'll have to take this off list.
Or to newbies. There you get help from the same people but without
having your misinterpretations in the 'official' archives for other poor
souls to find ;-)

http://mailman.theapt.org/listinfo/openbsd-newbies

> (Karel and Constantine don't agree??)
>
> So I just do.. "bioctl -t start sdX" wher sdX is the name of my softraid
> device, and it'll do the "scrub" as in reading through all underlying

bioctl(8) is clear, I think:
 -t patrol-function
  Control the RAID card's patrol functionality, if
  supported. patrol-function may be one of:

Why do you think it will work for softraid(4) when it says it does for
hardware-RAID?

I have a theory: you have some experience with other Operating Systems
and their built in help system that have led you to not fully read but
just search/skim for keywords. Do yourself (and me) a favour and read
them fully. Top to bottom. Take every word as put there thoughtfully,
not in a hurry. You can find manpage content discussions all over the
archives. manpages are taken seriously.

Please repeat: bio(4)/bioctl(8) controls RAID devices. These can be in
hardware or software. Some functions (-a, -b, -H, -t, -u) are only
useable/usefull when controlling a hardware RAID. The manpage even gives
direct clues on whether hardware- or software RAID is the topic. First
synopsis, second synopsis. 'The options for RAID controllers are as
follows:' (=hardware) 'In addition to the relevant options listed above,
the options for softraid(4) devices are as follows:' (=software).
Did you note the 'relevant' part? That word is there on purpose, I
suppose. It is there to tell you that not all, but the relevant parts of
the hardware RAID parameters also apply to software RAID (that comes
below). I would consider '-v' relevant, '-a' ('Control the RAID card's
alarm functionality, if supported') not.

(Example: what '-a' does for hardware RAID can be done with sensorsd(8)
for software RAID (=softraid(4)). Once a softraid volume is configured,
you get 'hw.sensors.softraid0.drive0=online (sd1), OK'.
Try 'sysctl hw.sensors.softraid0'.)

> physical media to check its internal integrity so for RAID1C that will be
> data readability and that checksums are correct, and "doas bioctl
softraid0"
> will show me the % status, and if I don't get any errors before it goes
back
> to normal it means the patrol was successful right?

No idea, never had a hardware RAID controller.

> (And as usual patrol is implemented to have the lowest priority, so it
> should not interfere extreemely much with ordinary SSD softraid operation.)

I think the patrolling is done by the hardware RAID controller.
bioctl(8) just commands it to do so.

> >> * Rebuild - I think I saw some console dump of the status of a rebuild
> >>process on the net, so MAYBE or NO..?
> >
> >That's what it looks like:
> >
> >$ doas bioctl softraid0
> >Volume  Status   Size Device
> >softraid0 0 Rebuild12002360033280 sd6 RAID5 35% done
> >  0 Rebuild 4000786726912 0:0.0   noencl 
> >  1 Online  4000786726912 0:1.0   noencl 
> >  2 Online  

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Constantine A. Murenin]

On 20 February 2016 at 14:29, Tinker  wrote:
[..]
> On 2016-02-21 04:39, Constantine A. Murenin wrote:
[..]
>> When you do http://mdoc.su/o/newfs.8, it does not write to every
>> sector of the underlying partition; thus you cannot expect all sectors
>> to be the same.
>
>
> Ah right, so at least to prepare for a RAID1C rebuild to work, at raid setup
> time before disklabel/newfs, one should initialize by doing "dd if=/dev/zero
> of=thelogicalraiddevice".

This would make the logical view from within softraid_raid1 appear the
same (e.g., an `sd3` regardless of which chunk it is being read from),
but the underlying `sd{0,1,2}a` chunks that would be backing it up
would still be different, because metadata
(http://bxr.su/o/sys/dev/softraidvar.h#sr_metadata).

C.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Tinker]

On 2016-02-21 05:05, Karel Gardas wrote:
The RAID 1 discipline does not initialize the mirror upon 
creation. This is by design because all sectors that are read are 
written first. There is no point in wasting a lot of time syncing 
random data.


I'm afraid the claim "all sectors that are read are written first" is
not generally right. E.g. disklabel 


On 2016-02-21 04:39, Constantine A. Murenin wrote:
..
Wait, I don't see the philosophical problem that you seem to be 
highlighting
here ("sectors are .. written first" and "on top of the filesystem" 
and

"forcing reads to be done from a certain chunk only"), what am I
missing/what is it that I don't understand?


When you do http://mdoc.su/o/newfs.8, it does not write to every
sector of the underlying partition; thus you cannot expect all sectors
to be the same.


Ah right, so at least to prepare for a RAID1C rebuild to work, at raid 
setup time before disklabel/newfs, one should initialize by doing "dd 
if=/dev/zero of=thelogicalraiddevice".



Also, so you mean that "patrol" even if it's in the manual is not 
supported
for RAID1 nor for RAID1C nor for RAID5 or any other raid discipline. 
Looking

forward to see what Marcus says & test myself & read code.


Heh, I didn't even know about this "patrol"; learn something new every 
day!


A BXR.SU for "patrol" has a few MFI(4) hits across the BSDs, and
ultimately reveals the `bioc_patrol` symbol (well, a struct, really,
http://bxr.su/o/sys/dev/biovar.h#bioc_patrol), a search of which
reveals that http://bxr.su/o/sys/dev/ic/mfi.c#mfi_ioctl_patrol is the
only driver reference for this symbol.

So, I don't think you'll be getting any patrol from softraid(4); it
was added very recently by uebayasi@ on 2015-05-29, based on mfiutil
from FreeBSD (http://mdoc.su/f/mfiutil.8), and only for mfi(4).



Aha, so to sum up the findings:

 * Scrub aka patrol is not supported (on the level of softraid) - 
perfectly fine.


 * Hot spares are supported (plug in with "bioctl -H" and plug out with 
"bioctl -O")


 * Rebuild is supported, and I should just figure out how (what do the 
two "bioctl -R" variants do respectively, and can it be done live, and 
does that mean it works like a hot-plug-in).


 * Hotswap is supported if hot-plug-in is supported via "bioctl -R", so 
should clarify that.



So just some minor piece of clarity left now. And also the separate SATA 
controller post.



All in all I think it looks neat.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Karel Gardas]

On Sat, Feb 20, 2016 at 8:44 PM, Constantine A. Murenin
 wrote:
>
> Scrub cannot possibly be supported due to the design of the softraid:
>
> http://mdoc.su/o/softraid.4
>
> The RAID 1 discipline does not initialize the mirror upon creation. This 
> is by design because all sectors that are read are written first. There 
> is no point in wasting a lot of time syncing random data.

I'm afraid the claim "all sectors that are read are written first" is
not generally right. E.g. disklabel 

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Constantine A. Murenin]

On 20 February 2016 at 12:23, Tinker  wrote:
>
> On 2016-02-21 01:29, Karel Gardas wrote:
>>
>> scrub is IIRC not supported by any softraid yet.
>
>
> But there's "patrol"!
>
> "bioctl -t start mysoftraid"

[...]

> On 2016-02-21 02:44, Constantine A. Murenin wrote:
>>
>> On 20 February 2016 at 10:29, Karel Gardas  wrote:
>>>
>>> scrub is IIRC not supported by any softraid yet. Rebuild by all which
>>> support redundancy. Marcus recommendation to read man pages can just
>>> be highlighted here. Otherwise just read the code for ultimate
>>> reference of what is or is not done.
>>
>>
>> Scrub cannot possibly be supported due to the design of the softraid:
>>
>> http://mdoc.su/o/softraid.4
>>
>> The RAID 1 discipline does not initialize the mirror upon creation.
>> This is by design because all sectors that are read are written first. 
>> There
>> is no point in wasting a lot of time syncing random data.
>>
>>
>> IIRC, other raid disciplines are not that much different, either.
>>
>> E.g., a scrub implementation would have to be implemented on top of
>> the filesystem, and would have to be able to temporarily force the
>> reads to be done from a certain chunk only.
>>
>> Long-term, it'll probably be easier to re-do the logic to actually
>> zero-out all the unused sectors, if scrub support is deemed important.
>> Which is why things like ZFS are superior due to having the awareness
>> of the underlying storage blocks.
>
>
> Wait, I don't see the philosophical problem that you seem to be highlighting
> here ("sectors are .. written first" and "on top of the filesystem" and
> "forcing reads to be done from a certain chunk only"), what am I
> missing/what is it that I don't understand?

When you do http://mdoc.su/o/newfs.8, it does not write to every
sector of the underlying partition; thus you cannot expect all sectors
to be the same.

>
>
> Also, so you mean that "patrol" even if it's in the manual is not supported
> for RAID1 nor for RAID1C nor for RAID5 or any other raid discipline. Looking
> forward to see what Marcus says & test myself & read code.

Heh, I didn't even know about this "patrol"; learn something new every day!

A BXR.SU for "patrol" has a few MFI(4) hits across the BSDs, and
ultimately reveals the `bioc_patrol` symbol (well, a struct, really,
http://bxr.su/o/sys/dev/biovar.h#bioc_patrol), a search of which
reveals that http://bxr.su/o/sys/dev/ic/mfi.c#mfi_ioctl_patrol is the
only driver reference for this symbol.

So, I don't think you'll be getting any patrol from softraid(4); it
was added very recently by uebayasi@ on 2015-05-29, based on mfiutil
from FreeBSD (http://mdoc.su/f/mfiutil.8), and only for mfi(4).

C.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Karel Gardas]

On Sat, Feb 20, 2016 at 9:23 PM, Tinker  wrote:
>
> On 2016-02-21 01:29, Karel Gardas wrote:
>>
>> scrub is IIRC not supported by any softraid yet.
>
>
> But there's "patrol"!
>
> "bioctl -t start mysoftraid"

bioctl also supports hardware raid cards besides softraid, so that's
what you are looking on now IMHO.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Tinker]

On 2016-02-21 01:29, Karel Gardas wrote:

scrub is IIRC not supported by any softraid yet.


But there's "patrol"!

"bioctl -t start mysoftraid"


Rebuild by all which support redundancy.


Yey! Clarified by Marcus & looking forward to his clarification


Marcus recommendation to read man pages can just
be highlighted here. Otherwise just read the code for ultimate
reference of what is or is not done.

Yey!


On 2016-02-21 02:44, Constantine A. Murenin wrote:

On 20 February 2016 at 10:29, Karel Gardas  wrote:

scrub is IIRC not supported by any softraid yet. Rebuild by all which
support redundancy. Marcus recommendation to read man pages can just
be highlighted here. Otherwise just read the code for ultimate
reference of what is or is not done.


Scrub cannot possibly be supported due to the design of the softraid:

http://mdoc.su/o/softraid.4

The RAID 1 discipline does not initialize the mirror upon creation. 
This is by design because all sectors that are read are written 
first. There is no point in wasting a lot of time syncing random 
data.


IIRC, other raid disciplines are not that much different, either.

E.g., a scrub implementation would have to be implemented on top of
the filesystem, and would have to be able to temporarily force the
reads to be done from a certain chunk only.

Long-term, it'll probably be easier to re-do the logic to actually
zero-out all the unused sectors, if scrub support is deemed important.
Which is why things like ZFS are superior due to having the awareness
of the underlying storage blocks.


Wait, I don't see the philosophical problem that you seem to be 
highlighting here ("sectors are .. written first" and "on top of the 
filesystem" and "forcing reads to be done from a certain chunk only"), 
what am I missing/what is it that I don't understand?



Also, so you mean that "patrol" even if it's in the manual is not 
supported for RAID1 nor for RAID1C nor for RAID5 or any other raid 
discipline. Looking forward to see what Marcus says & test myself & read 
code.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Tinker]

Marcus,

Holy moly, that is beautiful.

So glad to understand better what's in the box.

Also please note that I'm not trying to suggest to implement lots of 
crap, am perfectly clear that high security is correlated with low 
complexity.



On 2016-02-21 00:29, Marcus MERIGHI wrote:

ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 16:43 (CET):

..
You appear to mean bioctl(8). Thats the only place I could find the 
word

'patrol'. bioctl(8) can control more than softraid(4) devices.

bio(4):
 The following device drivers register with bio for volume
management:

   ami(4) American Megatrends Inc. MegaRAID
  PATA/SATA/SCSI RAID controller
   arc(4) Areca Technology Corporation SAS/SATA RAID
  controller
   cac(4) Compaq Smart Array 2/3/4 SCSI RAID controller
   ciss(4)Compaq Smart Array SAS/SATA/SCSI RAID
controller
   ips(4) IBM SATA/SCSI ServeRAID controller
   mfi(4) LSI Logic & Dell MegaRAID SAS RAID controller
   mpi(4) LSI Logic Fusion-MPT Message Passing 
Interface
   mpii(4)LSI Logic Fusion-MPT Message Passing 
Interface

II
   softraid(4)Software RAID

It is talking about controlling a HW raid controller, in that 'patrol'
paragraph, isn't it?


So by this you mean that patrolling is really implemented for softraid?? 
 (Karel and Constantine don't agree??)


So I just do.. "bioctl -t start sdX" wher sdX is the name of my softraid 
device, and it'll do the "scrub" as in reading through all underlying 
physical media to check its internal integrity so for RAID1C that will 
be data readability and that checksums are correct, and "doas bioctl 
softraid0" will show me the % status, and if I don't get any errors 
before it goes back to normal it means the patrol was successful right?


(And as usual patrol is implemented to have the lowest priority, so it 
should not interfere extreemely much with ordinary SSD softraid 
operation.)


 * Rebuild - I think I saw some console dump of the status of a 
rebuild

process on the net, so MAYBE or NO..?


That's what it looks like:

$ doas bioctl softraid0
Volume  Status   Size Device
softraid0 0 Rebuild12002360033280 sd6 RAID5 35% done
  0 Rebuild 4000786726912 0:0.0   noencl 
  1 Online  4000786726912 0:1.0   noencl 
  2 Online  4000786726912 0:2.0   noencl 
  3 Online  4000786726912 0:3.0   noencl 


Yey!!

Wait, can you explain to me what I would write instead of "device" and 
"channel:target[.lun]" in "bioctl -R device" and "bioctl -R 
channel:target[.lun]", AND what effect those would have?


Say that my sd0 and sd1 SSD:s run a RAID1C already, can I then make 
softraid extend my RAID1C with my sd2 SSD by "rebuilding" it, as a way 
to live-copy in all my data to sd2, so this would work as a kind of live 
attach even if expensive?


Does it work for a softraid that's live already?

 * Hotspare - MAYBE, "man softraid" says "Currently there is no 
automated

mechanism to recover from failed disks.", but that is not so specific
wording, and I think I read a hint somewhere that there is hotspare
functionality.


bioctl(8)
 -H channel:target[.lun]
 If the device at channel:target[.lun] is currently marked
 ``Unused'', promote it to being a ``Hot Spare''.

That's the only mention of 'hot spare'. And again talking about
controlling a hardware RAID controller, isn't it?

What is 'not so specific' about 'no' (as in "Currently there is *no*
automated mechanism to recover from failed disks")?


Awesome.

I guess "bioctl softraid0" will list which hotspares there are 
currently, and that "-d" will drop a hotspare.



The fact that there is hotspare functionality, means that there are 
cases when softraid will take a disk out of use.


That will be when that disk reports itself as COMPLETELY out of use ALL 
BY ITSELF, such as self-detaching itself on the level of the SATA 
controller or reporting failure via some SMART command?


A disk just half-breaking with broken sectors and 99% IO slowdown will 
not cause it to go offline though so I guess I should buy enterprise 
drives with IO access time guarantees then.


 * Hotswap - MAYBE, this would depend on if there's rebuild. Only 
disconnect

("bioctl -O" I think; "bioctl -d" is to.. unmount or self-destruct a
softraid?)


bioctl -O should fail the chunk specified, simulating hardware failure.
After this command you have an 'Offline' chunk in the 'bioctl' output.

bioctl -d 'detach', not 'destroy'; just as sdX appears when you 
assamble

  a softraid volume, this makes it go away. better unmount before...


So "-d" is to take down a whole softraid. "-O" could work to take out a 
single physical disk but it's unclean.



So then, there is a very unrefined 

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Constantine A. Murenin]

On 20 February 2016 at 10:29, Karel Gardas  wrote:
> scrub is IIRC not supported by any softraid yet. Rebuild by all which
> support redundancy. Marcus recommendation to read man pages can just
> be highlighted here. Otherwise just read the code for ultimate
> reference of what is or is not done.

Scrub cannot possibly be supported due to the design of the softraid:

http://mdoc.su/o/softraid.4

 The RAID 1 discipline does not initialize the mirror upon creation. This 
 is by design because all sectors that are read are written first. There is 
 no point in wasting a lot of time syncing random data.

IIRC, other raid disciplines are not that much different, either.

E.g., a scrub implementation would have to be implemented on top of
the filesystem, and would have to be able to temporarily force the
reads to be done from a certain chunk only.

Long-term, it'll probably be easier to re-do the logic to actually
zero-out all the unused sectors, if scrub support is deemed important.
Which is why things like ZFS are superior due to having the awareness
of the underlying storage blocks.

C.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Karel Gardas]

scrub is IIRC not supported by any softraid yet. Rebuild by all which
support redundancy. Marcus recommendation to read man pages can just
be highlighted here. Otherwise just read the code for ultimate
reference of what is or is not done.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Marcus MERIGHI]

ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 16:43 (CET):
> On 2016-02-20 22:23, Marcus MERIGHI wrote:
> >ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 15:29 (CET):
> >>This email is an attempt to get some knowledge on how softraid works.
> >
> >So many of your questions are answered if you start with bioctl(8)[1],
> >and continue with softraid(4)[2]. Maybe bio(4)[3] helps, too.
> >
> >What's there is usually documented. What's not documented is usually not
> >there. Or was it the other way around? ;-)
> >
> >[1]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/bioctl.8
> >[2]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/softraid.4
> >[3]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/bio.4
> >
> >Happy reading, Marcus
> 
> Marcus, I read the docs carefully, and based on them, my best
> understanding is still unclear on all 4 points -
> 
>  * Scrub - MAYBE??? There's some words about "patrol" in the man page that
> could mean it's there.

You appear to mean bioctl(8). Thats the only place I could find the word
'patrol'. bioctl(8) can control more than softraid(4) devices.

bio(4):
 The following device drivers register with bio for volume
management:

   ami(4) American Megatrends Inc. MegaRAID
  PATA/SATA/SCSI RAID controller
   arc(4) Areca Technology Corporation SAS/SATA RAID
  controller
   cac(4) Compaq Smart Array 2/3/4 SCSI RAID controller
   ciss(4)Compaq Smart Array SAS/SATA/SCSI RAID
controller
   ips(4) IBM SATA/SCSI ServeRAID controller
   mfi(4) LSI Logic & Dell MegaRAID SAS RAID controller
   mpi(4) LSI Logic Fusion-MPT Message Passing Interface
   mpii(4)LSI Logic Fusion-MPT Message Passing Interface
II
   softraid(4)Software RAID

It is talking about controlling a HW raid controller, in that 'patrol'
paragraph, isn't it?

>  * Rebuild - I think I saw some console dump of the status of a rebuild
> process on the net, so MAYBE or NO..?

That's what it looks like:

$ doas bioctl softraid0
Volume  Status   Size Device  
softraid0 0 Rebuild12002360033280 sd6 RAID5 35% done 
  0 Rebuild 4000786726912 0:0.0   noencl 
  1 Online  4000786726912 0:1.0   noencl 
  2 Online  4000786726912 0:2.0   noencl 
  3 Online  4000786726912 0:3.0   noencl 

>  * Hotspare - MAYBE, "man softraid" says "Currently there is no automated
> mechanism to recover from failed disks.", but that is not so specific
> wording, and I think I read a hint somewhere that there is hotspare
> functionality.

bioctl(8)
 -H channel:target[.lun]
 If the device at channel:target[.lun] is currently marked
 ``Unused'', promote it to being a ``Hot Spare''.

That's the only mention of 'hot spare'. And again talking about
controlling a hardware RAID controller, isn't it?

What is 'not so specific' about 'no' (as in "Currently there is *no*
automated mechanism to recover from failed disks")?
 
>  * Hotswap - MAYBE, this would depend on if there's rebuild. Only disconnect
> ("bioctl -O" I think; "bioctl -d" is to.. unmount or self-destruct a
> softraid?)

bioctl -O should fail the chunk specified, simulating hardware failure.
After this command you have an 'Offline' chunk in the 'bioctl' output. 

bioctl -d 'detach', not 'destroy'; just as sdX appears when you assamble
  a softraid volume, this makes it go away. better unmount before...

> The man pages are sometimes over-minimalistic with respect to an individual
> user who's trying to learn, this is why I'm asking for your clarification.

I am quite sure the man pages are kept as condensed as they are on
purpose.

You can always read mplayer(1) if you want something lengthy ;-)

> So your clarifications would still be much appreciated.

Nothing authoritative from me!
I am just trying to flatten your learning curve. 

Bye, Marcus

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Tinker]

On 2016-02-20 22:23, Marcus MERIGHI wrote:

ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 15:29 (CET):

This email is an attempt to get some knowledge on how softraid works.


So many of your questions are answered if you start with bioctl(8)[1],
and continue with softraid(4)[2]. Maybe bio(4)[3] helps, too.

What's there is usually documented. What's not documented is usually 
not

there. Or was it the other way around? ;-)

[1]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/bioctl.8
[2]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/softraid.4
[3]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/bio.4

Happy reading, Marcus


Marcus, I read the docs carefully, and based on them, my best 
understanding is still unclear on all 4 points -


 * Scrub - MAYBE??? There's some words about "patrol" in the man page 
that could mean it's there.


 * Rebuild - I think I saw some console dump of the status of a rebuild 
process on the net, so MAYBE or NO..?


 * Hotspare - MAYBE, "man softraid" says "Currently there is no 
automated mechanism to recover from failed disks.", but that is not so 
specific wording, and I think I read a hint somewhere that there is 
hotspare functionality.


 * Hotswap - MAYBE, this would depend on if there's rebuild. Only 
disconnect ("bioctl -O" I think; "bioctl -d" is to.. unmount or 
self-destruct a softraid?)


The man pages are sometimes over-minimalistic with respect to an 
individual user who's trying to learn, this is why I'm asking for your 
clarification.


So your clarifications would still be much appreciated.

Re: OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Marcus MERIGHI]

ti...@openmailbox.org (Tinker), 2016.02.20 (Sat) 15:29 (CET):
> This email is an attempt to get some knowledge on how softraid works.

So many of your questions are answered if you start with bioctl(8)[1],
and continue with softraid(4)[2]. Maybe bio(4)[3] helps, too. 

What's there is usually documented. What's not documented is usually not
there. Or was it the other way around? ;-)

[1]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/bioctl.8
[2]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/softraid.4
[3]http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man4/bio.4

Happy reading, Marcus

> There's basically zero docs on these topics out here (all docs are
> about how to set it up first & subsequent times in non-failure cases).
>
> If you would be able to respond in "HOWTO form" would be awesome, then
> at least this will be the Ultimate Softraid Reference :D
>
> Q1:
> Can softraid do
> 
>  a) Scrub (as in read all the underlying media and check they're in check,
> e.g. "raid fsck"),
>
>  b) Hotspare (as in have one or more pre-designated physical disks that the
> softraid would switch into use if one of the physical disks in use already
> breaks),
> 
>  c) Hotswap (as in I could unplug one of the physical disks in a raid live,
> and presuming my HBA allows it, the OpenBSD kernel + softraid will be happy
> to plug in and plug out disks live - and I guess perhaps that needs to play
> together with the rebuild then)?
> 
> 
> Q2:
> And, how do you use the rebuild + these three things? Specifically:
> 
>  a) How do I check the status of a rebuild from the console? (E.g. disk
> blablaX rebuilding 12.3% done, get report if the rebuild failed, etc.)
>  b) How do I plug in a physical disk live into my RAID1 softraid?
> 
>  c) How do I plug out a physical disk live from my RAID1 softraid?
> 
>  d) How do I initiate a rebuild on a RAID1 softraid, such as plugging in the
> brand new disk sdX into my RAID1 softraid or sdX was used previously or
> suffered a power outage or whatever - how do I rebuild it?
> 
>  e) How do I get a listing of all my softraids and of all the physical disks
> plugged into them, from the console?
>
>  f) How do I tell softraid to "scrub" i.e. intregrity-check all the physical
> disks running in the softraid, including every sector so actually all of
> their surfaces are read (presuming that the RAID takes up all the disks)?
> 
>  g) How do I tell my RAID1 softraid that if sdX dies, then sdY should be
> plugged in (i.e. hot spare)?
> 
> Also, how do I get a printout of that configuration from the console,
> and how do I remove a hotspare?
> 
>  h) After I plug out a physical disk from my softraid ( c) above), what
> should I generally need to do before unplugging it also physically - just
> wait 60 seconds or do some magic "SATA power off" command or the like?
> (Given an ultra nice SATA or HW RAID controller such as LSI HBA or LSI
> Megaraid.)
> 
>  i) When I plug in a physical disk on my SAS/SATA HBA (be it an LSI HBA or
> LSI HW RAID), will OpenBSD autodetect it and assign a device name for it
> just like it does for USB devices so it's complete hotswap/plug and play?
> 
>  j) When I plug in a softraid that worked before, what kind of trouble could
> happen - if it won't get going at all, should I just reset from backup, or
> is there some kind of "rescue rebuild" I could attempt?
> 
>  k) Do I need to know anything about DUID vs. device name use within the
> softraid?
> 
> 
> Looking forward to your explanation, thanks!! :DD
> Tinker
> 
> 
> !DSPAM:56c879bd171961725512869!

OpenBSD softraid can do scrub, hotspare, hotswap? How do rebuild + those 3 really done? (Absence of docs and howtos - ultimate Q!)

[Tinker]

Hi,

This email is an attempt to get some knowledge on how softraid works. 
There's basically zero docs on these topics out here (all docs are about 
how to set it up first & subsequent times in non-failure cases).


If you would be able to respond in "HOWTO form" would be awesome, then 
at least this will be the Ultimate Softraid Reference :D



Q1:
Can softraid do

 a) Scrub (as in read all the underlying media and check they're in 
check, e.g. "raid fsck"),


 b) Hotspare (as in have one or more pre-designated physical disks that 
the softraid would switch into use if one of the physical disks in use 
already breaks),


 c) Hotswap (as in I could unplug one of the physical disks in a raid 
live, and presuming my HBA allows it, the OpenBSD kernel + softraid will 
be happy to plug in and plug out disks live - and I guess perhaps that 
needs to play together with the rebuild then)?



Q2:
And, how do you use the rebuild + these three things? Specifically:

 a) How do I check the status of a rebuild from the console? (E.g. disk 
blablaX rebuilding 12.3% done, get report if the rebuild failed, etc.)


 b) How do I plug in a physical disk live into my RAID1 softraid?

 c) How do I plug out a physical disk live from my RAID1 softraid?

 d) How do I initiate a rebuild on a RAID1 softraid, such as plugging in 
the brand new disk sdX into my RAID1 softraid or sdX was used previously 
or suffered a power outage or whatever - how do I rebuild it?


 e) How do I get a listing of all my softraids and of all the physical 
disks plugged into them, from the console?


 f) How do I tell softraid to "scrub" i.e. intregrity-check all the 
physical disks running in the softraid, including every sector so 
actually all of their surfaces are read (presuming that the RAID takes 
up all the disks)?


 g) How do I tell my RAID1 softraid that if sdX dies, then sdY should be 
plugged in (i.e. hot spare)?


Also, how do I get a printout of that configuration from the 
console, and how do I remove a hotspare?


 h) After I plug out a physical disk from my softraid ( c) above), what 
should I generally need to do before unplugging it also physically - 
just wait 60 seconds or do some magic "SATA power off" command or the 
like? (Given an ultra nice SATA or HW RAID controller such as LSI HBA or 
LSI Megaraid.)


 i) When I plug in a physical disk on my SAS/SATA HBA (be it an LSI HBA 
or LSI HW RAID), will OpenBSD autodetect it and assign a device name for 
it just like it does for USB devices so it's complete hotswap/plug and 
play?


 j) When I plug in a softraid that worked before, what kind of trouble 
could happen - if it won't get going at all, should I just reset from 
backup, or is there some kind of "rescue rebuild" I could attempt?


 k) Do I need to know anything about DUID vs. device name use within the 
softraid?



Looking forward to your explanation, thanks!! :DD
Tinker