Re: Updated CCD Mirroring HOWTO

[Robbert Haarman]

Greg,

Again, you raise some interesting issues. I wonder how likely the 
catastrophic failures you describe are, versus how likely it is that 
things fail in a way where ccd actually helps you. I was hoping someone 
else would comment on that, but that doesn't seem to have happened so 
far.

> So one thing that's still missing is a big, bold line at the top
> that says:
>  
>   CCD Mirroring will eventually eat your data and you shouldn't use it!!

It's missing, because I am not at all convinced that claim is true.

The way I see it: when you use CCD mirroring, your data is written to 
multiple disks, rather than just one. In some situations, this won't 
help you (all your disks die in a fire; you delete your own files; ...) 
In some situations, this will help you (one of your disks fails, but you 
still have correct data on others). In some situations, it is not as 
good as other techniques (the cases you describe). It may or may not 
still be better than no mirroring in these cases (for example, in the 
case where one file gets corrupted, you may still have everything else 
intact).

I definitely think that stating that CCD mirroring _will_ eat your data
is FUD; short of bugs, CCD doesn't cause you to lose data; at worst, it 
may not preserve data which other methods would have preserved.

> To promote the use of CCD Mirroring without noting the above major 
> problems is a disservice to the novice who is likely not aware of 
> the above failure modes.

You are right that it would be deceptive to advertize CCD mirroring as a 
silver bullet. It would be a lie to say CCD mirroring is the best 
mirroring method. However, my HOWTO does neither of these. It clearly 
mentions that mirroring is no silver bullet (and that goes for _any_ 
kind of mirroring), and that RAID is superior to CCD. The HOWTO might 
actually not emphasize these points enough; I'll have a look at it 
sometime and make changes if I deem them necessary.

> To me, until the above have satisfactory 
> answers, the only thing the CCD Mirroring HOWTO (and the ccd(4)/
> ccdconfig(8) man-pages!) should recommend is:
> 
>   Don't use CCD Mirroring -- at best, it provides a false sense of 
>   security.  At worst, it will eat your data.  If you need mirroring 
>   functionality, use RAIDframe.

Again, you're making bold claims. I would like if someone else could 
comment on them. Does CCD mirroring really provide only a false sense of 
security? Will it really eat your data? Or is it just that it's not as 
good as RAIDframe, but still a valuable improvement over not using any 
mirroring at all?

> Really.  RAIDframe works, and it doesn't suffer from the serious 
> problems noted above.  

Agreed. However, RAIDframe requires compiling a custom kernel. Now. And 
when you next upgrade your system. And the next time. Until it gets 
included in the shipped kernel. CCD is easy to set up (once you figure 
out the steps) and I think it provides some protection against harddisk 
failures.

Again, thanks for your comments.

Bob

PS. If anybody on the list is annoyed by this discussion continuing
despite people already having pointed out that my HOWTO is considered
harmful, please tell me so. Until that happens, I'm assuming people are
Ok with me discussing things here and I'll keep responding to messages
people send me.

---
A man should practice what he preaches, but a man should also preach
what he practices.
-- Confucius

Re: Updated CCD Mirroring HOWTO

[Robbert Haarman]

Dear JCR,

> To the rest of list users; Please pardon another long email from me on
> this. Helping reasonable people like Robbert understand why many people
> consider "HOWTO's" to be harmful is hopefully worth the added noise and
> bandwidth.

If this is a concern, why don't we take the discussion off-list? Also, I 
don't want to waste anyone's time with this discussion, so if you are 
tired of this discussion, just tell me to stop it and I will.

> >> If end-users are lazy and want to take the easy way out, they should
> >> go back to using linux and MS-Windows. They are not welcome here.
> >
> >That's a pity. I personally think OpenBSD is the _only_ operating system 
> >that takes security as seriously as it should be taken, and it would be 
> >in everybody's (well, almost everybody's) best interest if they used it. 
> >There is nothing wrong with the project not wanting certain users, but 
> >it leaves these users with a choice among evils, which is a pity.
> >
> 
> The pity is not whether or not some users are welcome. The real pity is
> current technology has yet to produce a computer that the average user
> can, own, operate and maintain without either significant knowledge of
> their own or significant resources to pay professionals to do the dirty
> work.

I disagree. A Linux distro (forgive my blasphemy) like Ubuntu is easy 
enough for computer illiterates to use and even maintain, since security 
patches are automatically announced and installed with a click of the 
mouse. If only Ubuntu had the advanced security mechanisms of OpenBSD, 
it would be a very secure system, even if the users didn't know much 
about computers.

As it stands, OpenBSD is the only operating system I am aware of that
has had the full base system completely audited and has buffer overrun
and other protections enabled for all software on it. This, by itself,
makes it more secure than other systems, regardless of what users do
with it. Even in the worst case, where users actively degrade the
security of the system, I would imagine OpenBSD's security would at
least not be _worse_ than that of another system.

> >The reason I wrote the HOWTO is that, in my opinion of course, the 
> >manpages don't make it clear how to set things up. Searching the 
> >archives for more information came up with some contradictory messages, 
> >and some instances of people being misled by the way things worked and 
> >the way things were described in the manpages. My HOWTO is an effort to 
> >gather the relevant information in one place, and provide clear steps 
> >for getting things working. 
> 
> Therein lies a significant difference of opinion between you and I. The
> steps provided by HOWTO documents do not give clarity,

They do. They explain some of the things that people were having 
problems with, such as the fact that labeling doesn't work the same way 
as it does for real disks (which Mickey says I got wrong - but can you 
blame me, given that the manpages don't say anything about it?).

No matter how much you compare my HOWTO to blindfolding people and 
possibly sending them off in wrong directions, it's a fact that the 
documentation that existed before it has led people in wrong directions 
and left them confused. The HOWTO is my attempt to provide instructions 
that work and don't leave people confused. Perhaps, instead of arguing 
back and forth about whether I did well to write the HOWTO, we should be 
working together to fix the mistakes and turn it into a document that 
provides correct and sufficient information?

> You are legally able to copy the OpenBSD man pages, so there is really
> nothing stopping you from quoting them a chunk at a time and adding your
> own insight, explanations and experience. By privately contacting the
> authors and maintainers of both the code and man pages, you can easily
> double check your work to prevent spreading misinformation. Provide
> explanations of the steps you took as well as explanations of all the
> other possible steps a user might want or need to take. 

There's something to that, too. I didn't want to bug the developers,
afraid as I was that the questions I had would annoy them, and result in
a pointer back to previous misc@ threads at best. Instead, I decided to
figure out for myself how I could get things to work, and document the
steps, so that others might benefit. It seems that, in doing so, I've
annoyed people even more. Again, I must apologize. This was not my
intention.

> better than a HOWTO that claims to be a short-cut way to set up
> mirroring but actually provides the steps needed to possibly fry your
> disks through misconfiguration.

Honestly, I think that's a stretch. I'm sure you can destroy your data 
with ccd, but frying your disks with a pure software feature?

> There might be other good ways to go about making things more accessible
> to users but the methods you are currently using are really a disservice
> to others in spite of your good intentions.

That'

Re: Updated CCD Mirroring HOWTO

[Robbert Haarman]

Dear JCR,

Thank you for your informative message.

> Things like "HowTo" documents, sites like openbsdsupport.org and lists
> like openbsd-newbie@ are more often than not considered garbage. The
> reason is simply because you are robbing the reader of the fundamental
> and important details that the reader _NEEDS_ to learn. By providing
> short-cut documents to just get things working, you are sabotaging the
> learning process of the reader.

I see. That's a good point.

> The "Quick Start" section of your document is missing a lot of things,
> in particular, the generally accepted way to "Quick Start" anything on
> OpenBSD.
> 
> 1.) Read *_ALL_* relevant man pages, completely and repeatedly until you
> understand them.
> 2.) Search the mailing list archives for related information.
> 3.) Search the commit logs for any new developments and/or corrections.

I will add these steps to the HOWTO, making it clear that nobody on the 
mailing list should be bugged with questions before these steps have 
been taken.

> If end-users are lazy and want to take the easy way out, they should
> go back to using linux and MS-Windows. They are not welcome here.

That's a pity. I personally think OpenBSD is the _only_ operating system 
that takes security as seriously as it should be taken, and it would be 
in everybody's (well, almost everybody's) best interest if they used it. 
There is nothing wrong with the project not wanting certain users, but 
it leaves these users with a choice among evils, which is a pity.

> People around here have been really trying to be polite recently. The
> thing that got me personally was realizing that these days there are
> young kids reading these lists. None the less, don't let my politeness
> fool you; I still think you are an arrogant, egotistical asshole who has
> shown blatant disregard for the education and well being of others as
> well as shown complete disrespect to the authors who spend their time
> and effort writing correct code and documentation so people can actually
> learn and do something useful.

I really appreciate your politeness, and also appreciate you pointing 
out your real feelings in no uncertain words.

> If you really want me or anyone around here to change their mind about
> you and the crap you're producing, then replace your supposed "HowTo"
> with links to the relevant man pages.

The reason I wrote the HOWTO is that, in my opinion of course, the 
manpages don't make it clear how to set things up. Searching the 
archives for more information came up with some contradictory messages, 
and some instances of people being misled by the way things worked and 
the way things were described in the manpages. My HOWTO is an effort to 
gather the relevant information in one place, and provide clear steps 
for getting things working. Replacing the HOWTO with links to the 
manpages is not an option, because, first of all, the manpages don't 
provide the information that the HOWTO does, and secondly, the manpages 
are already assumed to have been read, so linking to them again wouldn't 
add anything.

I completely understand your position that documentation should provide 
the details people need to understand how things work. I agree with 
that. I would have provided this information if I had, myself, known how 
things work. However, I don't know this, and it seems the only way to 
find it out would be to read the source code of various parts of the 
kernel. Still, I figured I had learned enough to write a useful and 
helpful document that at least described how to get things running. You 
clearly think this is harmful; I think it might help some people.

> If perchance this assessment of you and your work has incited you enough
> to turn on your personal flame thrower,

Not at all. Your piece is very well reasoned and even you calling me an 
arrogant asshole serves a good purpose.

> It is my opinion that you have made a serious mistake by not
> knowing or not understanding the generally accepted consensus about
> correct documentation

In that case, you couldn't have done better than to point it out to me, 
and I'm very grateful that you did.

> you won't change my mind or the minds of most long time users and
> contributors.

I have no intention of doing so. I think your position is perfectly 
valid. I also think my quest of providing clearer information and making 
things more accessible is perfectly valid. I can only hope you agree.

Sincerely,

Bob

---
Computers have made our lives much more efficient.
Now we can do many more useless things in one day.

Updated CCD Mirroring HOWTO

[Robbert Haarman]

Dear list, especially Greg and Mickey,

I've updated the working copy of the CCD Mirroring HOWTO. In particular, 
I've split off the comparison to software RAID into a separate section 
and clarified that ccd does not do automatic recovery, and I've 
rewritten the section on labeling to state that the c partition must be 
set to unused, and normal partitions created instead.

Please take a look at
http://morgenes.shire.sytes.net/~inglorion/documents/tutorials/ccd/ and
let me know what you think about the new wording.

Bob

---
The only thing you know for sure is that you never know anything for sure.

Re: CCD Mirroring HOWTO

[Robbert Haarman]

On Fri, Nov 25, 2005 at 04:50:34PM +0100, mickey wrote:

> default 'c' type is unused.
> at least on default systems...

On REAL disks, yes. On ccd disks, it seems to be different. Or maybe
ccdconfig screws it up.

> > 2. The OpenBSD slices of my disks start at sector 24659775. This is also
> > where the a partition of these disks start. It isn't that way because I
> > made it that way, it was set up that way by the initial label editor
> > when I did the installation.
>
> some pplz use 'c' for their ccd components -- WRONG!

Pardon my rudeness, but are you sure? It works fine for me. Other people
have reported it works fine for them. I don't know any implementation
details, but maybe you do: what actually goes wrong when you use the c
partition on a ccd device?

> > 3. From 2, I conclude that wherever the BSD disklabel is stored, this
> > does not affect where my partitions can be. The disklabel could be
> > stored in my root partition, for all I know.
>
> disklabel is the sector #1 of the fdisk partition

The fdisk partition that is the OpenBSD slice? Because my a partition
starts at the exact same sector as that slice, and there seem not to be
any problems. Again, this is how the system set it up for me, I didn't
change the start position of my a partition. Maybe ffs can accomodate
disk labels and there isn't any problem?

> > 4. I have a ccd device starting at sector 41110398, with a size of
> > 32901057 sectors. Inside the ccd device is a c partition of 32901056
> > sectors, starting at sector 0, with type 4.2BSD. This isn't because I
> > set it up this way, this is how the device was set up when I first ran
> > disklabel. I never changed anything there.
>
> oh uhm must be a bug in disklabel spoofing (:

You mean that it's actually unused, but some spoofing code reports it to
disklabel as 4.2BSD? I don't think so. If you change the type from
4.2BSD to unused, you can partition as normal. Make your a partition,
and all that. And disklabel will report it as unused afterwards. By all
appearances, it looks like it _really_ does get set to 4.2BSD.

> it's not about how the 'c' is setup.
> you can screw it on normal drive as well.
> just run a newfs on it!

I see how you can screw it up on a real drive, especially one that has
fdisk partitions. Can't have an fdisk partition and an ffs filesystem in
the same sector. About disklabel, I don't know. What I'm seeing (my a
partition starting on the same sector as my OpenBSD slice) suggests
that, perhaps, they can coexist.

> the point is that the ccd part must be started w/ an offset as well!

Well, there is that one missing sector. Maybe the ccd driver
automatically reserves space for the label? I'm just making wild
guesses, trying to explain why it works for me, even though you say it
shouldn't.

To recap, I would very much like to see the following questions answered
beyond all doubt:

1. Is it or is it not wrong to use the 4.2BSD c partition that gets
created for you?

2. If it is wrong, why does it get created?

So far, there is you answering question 1 with a definite "it's wrong",
and me answering with an "it's not wrong", backed up with the evidence
that it works for me without any problems.

I hope I'm not being too annoying about this. It's just that I wrote the
HOWTO to clarify things and makes sense of the scattered and
inconsistent information on ccd and mirroring. I have little desire to
incorporate any information in it before I am very sure it is correct. I
want to clear up the confusion, not add to it.

Cheers,

Bob

---
The more you learn about Windows, the more you are amazed it works at all
-- Pfhreakaz0id on Slashdot

[demime 1.01d removed an attachment of type application/pgp-signature]

Re: CCD Mirroring HOWTO

[Robbert Haarman]

On Fri, Nov 25, 2005 at 02:43:31PM +0100, mickey wrote:
> On Fri, Nov 25, 2005 at 02:41:47PM +0100, Robbert Haarman wrote:
> >
> > This still leaves the issue of ccdconfig setting up the c partition as
> > type 4.2BSD. If this can thrash the disklabel, that sounds like a
> > serious bug. Is that going to be fixed?
>
> of course oif you make a it use 'c' it will reset the type.
> so do not use it (:

I'm not sure what you mean here. What I meant is that ccdconfig will
automatically create a disklabel with partition c set to type 4.2BSD
when you run it the first time. This has bugged many people who were
trying to create their own partitions, as disklabel will not let you
create any partitions when the type of partition c is set to anything
other than unused.

I assumed, as others did, that the fact that ccdconfig sets things up
this way means that you can use the c partition for storing files on.
You just told me this is not true, and it can in fact thrash the
disklabel on your real disk. Therefore, it seems to me that
ccdconfig setting the type of partition c to 4.2BSD is wrong. So I was
wondering aloud if that is indeed a bug and if it will be fixed.

> please make your writing into real openbsd faq entry.
> i bet _then_ people will actually pay attention to it in the future (:

You mean like an entry in the FAQ on the OpenBSD website? How do I go
about doing that?

Cheers,

Bob

---
"A good engineer will go to any amount of effort to avoid extra effort."

[demime 1.01d removed an attachment of type application/pgp-signature]

Re: CCD Mirroring HOWTO

[Robbert Haarman]

On Fri, Nov 25, 2005 at 02:26:08PM +0100, mickey wrote:
> On Fri, Nov 25, 2005 at 02:14:35PM +0100, Robbert Haarman wrote:
> by using 'c' partition one may endup trashing real disk's label.
> DO NOT USE 'c' PARTITION.

Ok, I'll change the HOWTO and add a FAQ entry ASAP. Apologies for not 
believing you earlier; there's posts on the web that claim one way and 
posts that claim the opposite, I just went by my own experience.

This still leaves the issue of ccdconfig setting up the c partition as
type 4.2BSD. If this can thrash the disklabel, that sounds like a 
serious bug. Is that going to be fixed?

-- Bob

---
The chief cause of problems is solutions.
-- Eric Sevareid

Re: CCD Mirroring HOWTO

[Robbert Haarman]

> >Of course, if somebody who
> >actually knew the implementation details about ccd could weigh in,
> >that would resolve the issue.
> 
> they just did.
> 

Ok, thanks for pointing that out. I apologize for my uninformed 
comments, especially to mickey.

-- Bob

---
Coal powered the first steam engines, whose killer app was pumping
stagnant water out of coal mines. It powered the railroads, whose killer
app was moving coal.
-- Bruce Sterling

Re: CCD Mirroring HOWTO

[Robbert Haarman]

> > http://inglorion.net/documents/tutorials/ccd/.
> 
> labeling section is wrong.
> one MUST never use 'c' partition.
> one MUST always make an 'a' (for example)
> to skip first cylinder (at least).

That's true for real disks, but it doesn't seem to be true for ccd
devices. If it were true, ccdconfig shouldn't be setting the c partition
to type 4.2BSD, either. I use the c partition created by ccdconfig on
one of my ccd devices, and, so far, it works without problems. From what
information I have been able to gather from the web, it works for
others, too. I will add a warning, but I don't think my labeling section
is _wrong_. Of course, if somebody who actually knew the implementation
details about ccd could weigh in, that would resolve the issue.

Anyway, thanks for your comment.

Bob

---
"In one of the Bard's best-thought-of tragedies, our insistent hero,
Hamlet, queries on two fronts about how life turns rotten."

Re: CCD Mirroring HOWTO

[Robbert Haarman]

>  http://inglorion.net/documents/tutorials/ccd/
> 
> I have a few questions/comments about the above and about ccd
> in OpenBSD in general.
> 
> 1) You talk about "RAID only works with whole disks"?  I'm curious 
> which RAID implementation you're referring to... it seems like you 
> might be talking about RAIDframe, but it works with partitions 
> instead of disks too.  (In fact, it only cares about parititions.)

I might be wrong. I've only ever seen RAID used on whole disks. I'll 
look into RAIDframe; if that supports using partitions as components, 
I'll change the HOWTO. Thanks for pointing this out.

> 2) How does ccd keep track of the mirror getting out-of-sync?  (i.e. 
> if the system happens to fall over at the exact instant where a write 
> hits one of the disks, but not the other?)  I know how this is done in 
> RAIDframe, but I don't see a similar mechanism in ccd, and lacking 
> that mechanism would be Very Dangerous.  (if that data is never 
> overwritten, and a disk fails, you might be left with the disk with 
> correct data, or the one without correct data.  You flip the coin.)

AFAIK, ccd doesn't have any mechanisms for that. If one disk fails, all 
you can do is reconfigure the ccd device to use only the remaining 
disks, dump the filesystem, replace the failed disk, set up your old ccd 
configuration, newfs, and restore. This is all manual. To me, that's 
acceptible, because I don't expect my disks to fail more than once a 
year, and I don't mind spending an hour per year rebuilding filesystems.

> 3) The code appears to only ever do reads from the first partition
> in 'old_io'-mode (the current default). That is fine if only the 
> mirror fails, I suppose.  I havn't dug deep enough into ccd.c to 
> determine what's really going on in !old_io mode, but it looks like
> it reads from both parts of the mirror? (whether it needs to or not) 
> There doesn't appear to be any mechanism here to say "this write failed"
> and then to note that the data on a particular block is now invalid.  

I don't know any implementation details; I haven't actually looked at 
the code. To me, it's enough that I have a second copy of my data.

> 4) Nothing is mentioned about how to "recover" from a disk failure.  
> This is perhaps the most important part of any mirroring setup!! 

See my answer to point 2.

> I think I saw mention of 'dd' in another post -- if that is indeed 
> the mechanism, then, at a minimum, care must be taking to make sure 
> the filesystem is not being written while the 'dd' is being done!!

You do the recovery offline.

> I believe there are some very good reasons to be using the mirroring 
> in RAIDframe, and to not be even thinking of using the 
> so-called-"mirroring" support in ccd -- to me using ccd for 
> "mirroring" is just asking for trouble.

It depends on what you need, I guess. If you want availability and 
automatic recovery, ccd is simply not an option. If all you want is to 
have your data mirrored, then ccd seems to do just fine. I'm in the 
latter category - for now anyway.

> But that's just my $0.02, and yes, I'll admit to being biased towards 
> RAIDframe.  At the same time, however, I don't want to see to see people
> get badly burned by something that is advertised as a "solution" when
> it really isn't (and when other real solutions do exist -- RAIDframe 
> only adds 150K to SomeOtherBSD's kernel these days).

I think I should add a few more warnings to my HOWTO, just to make sure 
people aren't expecting ccd to things it simply doesn't do.

> Thanks for your time.

Thanks for yours. You've made some very insightful comments.

Regards,

Bob

---
I can't remember the last time I forgot something.

CCD Mirroring HOWTO

[Robbert Haarman]

For those who are interested, I've uploaded a tutorial on setting up 
mirroring using ccd(4) to http://inglorion.net/documents/tutorials/ccd/.

-- Bob