Re: the fdisk man page and the fdisk behaviour

[J.C. Roberts]

On Fri, 03 Apr 2009 10:52:17 -0600 Tobias Weingartner
 wrote:

> > > > sense if it offered me the first available lba sector with
> > > > partition type 0?  i mean even if it doesnt want to offer any
> > > > "responsible" value, 0 is wrong in any case on i386, as the
> > > > first offset has to be at least 63...
> > > 
> > > "WRONG" :)
> > 
> > ok, the number should be S, as in sector.  so if not usable
> > value is offered, at least it shouldnt offer to overwrite the MBR.
> 
> You did point a problem here.  Having the default value outside of the
> allowed range makes little sense.  :)  I'm asking for review and
> testing of a diff I've cooked up to fix this one issue.


If I understand things correctly, even a simple diff would result in
requiring a *MASSIVE* testing effort across tons of hardware/archs.


> > > A very long time ago, I posted a somewhat similar rant to this
> > > same list.  I got a very polite private message from Toby,
> > > saying, "Ok, I'm tired of these complaints, what would you do
> > > differently?" and to be honest...I never came up with a good
> > > answer.
> >
> > i think i highlighted a couple of issues with very easy patches.
> > esp. for much more skillful person than i am, e.g. Toby.
> 
> Ugh.  Ok, yes, I could do it.  But honestly, I don't care to.  The
> last time a list bitched and moaned for an improvement of a particular
> system, I spent a considerable amount of time implementing such.
> When I asked for tests of the system a while later, I had 6 people
> test.  Six. Out of the 50+ requests for the essential feature, *SIX*
> tested.  So, now you would love it if fdisk was better.  So would I.
> But, I'm not going to expend any energy on this, and then have 1/2
> dozen people test on a dozen machines, a feature that has potential
> for serious data loss if there are any bugs in it.
> 
> The actual code is not hard.  It's on the level of what a 2nd to 3rd
> year CS student is capabale of.  I'm just tired of being the one
> that has to chase enough variety of machines/people/etc to do enough
> different types of tests, with enough variety of other partition
> munging OSen to have decent test coverage.  Someone else can step up
> and do this testing.  And since the coding is fun, I'll leave the
> coding for them as well as an incentive.  :)
> 

If you believe it is worth doing, and you don't mind applying the clue
stick to teach me the failure/success modes I need to look/test for, I
actually could do a fairly "*MASSIVE*" testing effort. Though they
typically sit in stacks, unplugged, waiting for me to have a need for
them, I have most all archs here, and many variations (systems) within
each of the archs... and parts, lots of parts.

How much testing (in archs, and machines, and test time length) would be
really required?

Is it a good use of time compared to what we would gain?

For lack of a polite way to say it, you've basically described
something destined towards a "OpenFDISK" project of it's own. The real
trouble is I could not afford the power bill to continue wide coverage
testing on a long term basis for an always on-going fdisk project. I
would consider doing it once, but not forever.

-- 
J.C. Roberts

Re: the fdisk man page and the fdisk behaviour

[Tobias Weingartner]

Note, I'm not on this list, Theo forwarded the message for me.

On Tuesday, March 31, Theo de Raadt forwarded:
> Date: Tue, 31 Mar 2009 20:13:22 +0200
> From: frantisek holop 
> To: misc 
> Subject: Re: the fdisk man page and the fdisk behaviour
> 
> hmm, on Tue, Mar 24, 2009 at 11:34:38PM -0400, Nick Holland said that
> > I really don't want to ignore what you call an "exotic geometry".
> > A lot of people seem to think there is something "standard" about
> > what you are calling "n/255/63 geometry", but life ain't so

Ok, first we have to know which geometry we are talking about.  If it
is the geometry that the hardware (disk) gives us, then the 255/63
geometry is special in that it is usually prefixed with a specific
number of cylinders (the number escapes me right now), which is the
ATA/IDE way of telling you that the geometry is fake, and for the
operating system to use LBA methods to access the disk.

The other geometry is the BIOS geometry.  This is the geometry that the
BIOS exposes to the operating system and booting environment.  This is
usually the same as the disk (if possible), but does not have to be.  For
the purposes of the MBR and partitioning, this is the only geometry that
matters.  Note, on some platforms this "BIOS" geometry will be the same
as the disk reported geometry always.  IE: they don't have a BIOS to get
in the way of relatively sane hardware.


> well, the commercial sector making partitioning software seems
> to disagree here.  the quite popular PartitionMagic comes to my
> mind.  and openbsd itself had a commit in the past to show every
> sd(4) disk using this very same geometry if i am not mistaken.

Ok, that was the hardware geometry.  Again, different from BIOS geometry
in most cases where it matters.

> here is another mail i sent in a looong time ago, another saga
> of mine, where because of openbsd's different geometry settings
> overlapping partitions and other filesystem errors manifested
> themselves, no doubt because of my inability to calulate all
> the numbers using a different geometry.
> http://marc.info/?l=openbsd-misc&m=116475065104125&w=2

Yes, in general, fdisk(8) tries really hard to find and use the BIOS
geometry.  If it does not, one thing you can try to do is to boot into
the /boot environment (the "boot>" prompt), and type 'machine disk',
and have a look at what the geometry is for the disk you like to deal
with.  Then use the command line flags to fdisk(8) to set the right
geometry when you're editing the partitions on the disk in question.

Note, when you move a disk from one system to another, or change the
disk access method (legacy, large, etc, etc), it may well cause the
BIOS to change the disk address translation mechanism it will use,
which *WILL* cause you issues.  If it does not, you're simply lucky.


> all your examples are without a doubt valid, but are they
> representative of the current hard drive situation?  that's
> what typical means for me in this context.  if you have to
> hunt for these disks on ebay, they are hardly typical or
> representative for me...

Irrelevant.  Seriously.  I'm not against making fdisk better of easier
to use, but it needs to be smarter than simply say "Well, 90% of disks
today are this way...".

If you really want to look forward, code up the relevant pieces of the
GPT partitioning scheme...  if you do so, and it works/etc, you'll get
the benefit of picking the OpenBSD GPT UUID identifiers... :)


> numeruous people stated on this mailing list that geometries
> had become abstract numbers in the world of modern hard drives
> and are basically useless anyway.  if that is the case, i do
> not see a good reason why not go with the other mainstream
> operating systems and just do make with an n/255/63 geometry
> everywhere (where possible).

Again, most of the time that is a hardware geometry.  Most modern
BIOS will also use a 255/63, simply because it maps most easily to
the current hardware.  However, plug in a RAID card, and see it give
you something different.  Plug in a USB stick, and notice it getting
something completely different yet again.


> i havent seen hard drive's described by their c/h/s in any bios
> for years (yes, i know, you probably have that old machines
> as well ;-)

It does not matter.  Do you have Win2k on your box?  Does XP touch it?
Any number of OSes are going to require that the C/H/S based fields in
the MBR are correct for them not to step on Open's toes.


> > > violates the CAVEATS section of the man page on two counts:
> > > 
> > > 1. the ending boundary of the 0th partition does not "end at
> > >   cylinder boundaries."

This one is a little harder to fix.  But for GPT support so

Re: the fdisk man page and the fdisk behaviour

[Nick Holland]

J.C. Roberts wrote:
...
> On a the lower and fundamental levels, solid state storage does not have
> the limitations or organization of rotating storage (disks), but none
> the less, in current products the new tech has been (intentionally)
> impaired and implemented with the old ideas to provide ass-backwards
> compatibility. At present, the Solid State (storage) Disks/Devices
> (SSD's) currently available have been designed for the sake of selling
> into existing markets where being ass-backwards compatibility is,
> unfortunately, a requirement.

eh, that's just this generation.
They've been available in some seriously kick-butt (for the day)
formats for literally decades on various platforms...but the price
was high, the issues were complex, and therefore, sales were low
and prices stayed high.

(back in the mid 1980s, I used a mainframe at school with a Solid
State Disk system for swap, as a way around limited physical RAM
capabilities in the system.  I hate to say this, but I think it was
a whopping 16M in size (though there may have been several of them).)

(side trip: anyone under 40 remember bubble memory?  don't answer
that on list, please...)

The new generation of stuff is mindless and simple...and sells like
hotcakes, and the price has therefore dropped to the point where it
sells like hotcakes...on sale.

YES, it loses much of the point to have a disk with no moving parts
that interfaces like a disk with moving parts.  But then, it makes
no sense to have a disk with variable geometry being treated as a
drive with fixed geometry.  But we've had it for 25 years...and
don't wait up for the change...If it greases the wheels and lets
things happen, greatit beats not happening at all.

> Though my employers would shoot me for violating an NDA, and Theo
> would shoot me for signing an NDA in the first place, for me it's tough
> to make a living with out them, so I have to be vague and leave out
> important details.
> 
> There are efforts afoot to abandon the limitations and organization of
> rotational storage, so both existing "disk layout" tools such as fdisk
> and disklabel, and even file systems, will eventually need to change to
> benefit from new technology.

actually, more file systems and boot loaders than disk layout tools.
The disk tools are pretty much a "bunch of sectors" thing, and have
been for quite some time now.

> The whole concept of sectors, 512 byte or otherwise, on solid state
> storage systems/devices is really just a sad kludge. The concept of a
> "disk controller" is already out dated and will soon be abandoned.

well..flash memory still has pages or similar that have to be cleared
and rewritten, at least according to my (non-ECC) memory...So, kinda
sector-ish.

Really, I'd think the idea of "sectors" has been more of an OS construct
than a hardware construct for a long time.  Back in the Amiga days
(again from non-ECC personal memory), they were loading entire tracks
from the floppy into RAM at a time, rather than sector-by-sector, in
large parts because once you got over 100k of RAM or so..you could.

I don't think you will see OSs allocating disk space on one-byte
boundaries any time soon.  I could be wrong...but I've seen some OSs
that managed to handle data on subdivided blocks...and eventually
they decided it wasn't worth the effort.  As disk size has grown,
trying to save a few K here and there through added complexity
is highly questionable...

> The following is publicly available information, is nearly two years
> old, is still using ass-backwards compatibility, and is obviously quite
> a few redesign revisions behind reality, but it should give you
> an idea of where the storage world is headed:
> http://www.tgdaily.com/index.php?option=com_content&task=view&id=34065

Former employer could have used some of those for a poorly written
app they seemed to be stuck with.  At one point, we were looking
at some $250,000 battery-backed RAM disks.  Cost was the killer, but
it wasn't a "quarter million? HAHAHHAA!  NO FREAKING WAY", much more
"hm.  don't think we can justify that now".  However, the guy who
fired me sunk many times that into some very bad solutions to the
problem.

Another problem all there will be with going in a "new direction"
is that it is a multi-prong problem -- storage HW, computer HW and
OS all have to support whatever is done.  The fact that you are
signing NDAs causes me to believe it is going to be a specialty
combination of HW and SW for some time...

'course, I spent about 20 years hoping that One Day the IBM XT/AT
abomination that we've been stuck with would be replaced.  I think
I've given up.

Nick.

Re: the fdisk man page and the fdisk behaviour

[J.C. Roberts]

On Wed, 1 Apr 2009 16:32:23 +0200 frantisek holop 
wrote:

> it wouldn't be the pc industry if it didn't go on with such a basic
> data structure going undocumented, without a standard since its
> conception.

So True! --Pathetic, but true none the less.

-- 
J.C. Roberts

Re: the fdisk man page and the fdisk behaviour

[frantisek holop]

hmm, on Wed, Apr 01, 2009 at 06:54:44AM -0700, J.C. Roberts said that
> Though they could logically do it now, is there a *need* to do this,
> let alone available hardware?  -You're really talking about a personal
> hacking project with a *very* serious cash requirement to buy the
> necessary equipment.

wherever i look on the net, all people writing about mbr's and
other "beginning of disk" structures, sectors, all of them starts
with "bear in mind, there is no accepted standard for this".

it wouldn't be the pc industry if it didn't go on with such a basic
data structure going undocumented, without a standard since its conception.


> Can you imagine trying to access RAM with c/h/s or LBA ?

well LBA is just an index basically (on the surface).
looks like an adress in RAM to me...

-f
-- 
i know nothing, but i know it fluently.

Re: the fdisk man page and the fdisk behaviour

[J.C. Roberts]

On Wed, 1 Apr 2009 05:14:54 +0200 frantisek holop 
wrote:

> hmm, on Wed, Apr 01, 2009 at 05:05:10AM +0200, frantisek holop said
> that
> > i wonder what will happen when 65536/255/63 is reached,
> > what kind of frankenstein translation table will come into life
> > just to save c/h/s again.  maybe there will come an LBA only age,
> > i mean fdisk-wise, not disk-wise, those are already LBA only...
> 
> openbsd could be the first one to nuke c/h/s from the MBR
> entirely, replace it with 8 or 16 byte LBA values (4 bytes
> currently) and lead the way for terribly huge disks :]
> 
> -f

fantisek,

Though they could logically do it now, is there a *need* to do this,
let alone available hardware?  -You're really talking about a personal
hacking project with a *very* serious cash requirement to buy the
necessary equipment.

More importantly, is c/h/s or even LBA even relevant to the new storage
system designs? The first is archaic, and the second is a just kludge
for the continuance of an archaic concept.

Everything we have and use today for storage was built on top of the
concepts of rotating media. It is *habit* for all of us to think about
things in terms of rotating media. Very soon our habit will become
obsolete.

It's all storage. How the storage is arranged, designed or implemented
is under constant evolution, including natural selection where a new,
stronger species can eradicate a weaker species.

Originally we had volatile storage (memory). You could call it solid
state "RAM" and be correct, but it just as easily could be vacuum tubes.

We also created even more volatile and faster storage (cache)
implemented closer to the processor. We even made successively faster
levels of cache.

We created "less volatile" but slower storage (drum, cards, tape, rom,
disk, disc, ...). --In my opinion *all* storage is volatile, especially
if Nick is around with a Nail Gun (ramset). (;

Since we never have enough level 1 cache, we pull from the next level.
Since we never have enough cache, we pull from RAM.
Since we never have enough RAM, we pull from paging (swap) on disk.
Since we never have enough disk, we pull from tape archives.

The tricky thing to remember is the next species of "less volatile"
storage, will not only replace rotating media, but it is extremely
similar to memory/RAM as we currently know it. All of those cool tricks
we've learned from the memory/RAM world are generally applicable, but
since it is "less volatile" all of the tricks we've learned from the
disk/raid/tape world are also applicable.

In the biological sense, think about your super fast RAID subsystem
mating with your massively interleaved memory subsystem and having
offspring with most of the best traits from both parents.

Can you imagine trying to access RAM with c/h/s or LBA ?

I didn't think so.

-- 
J.C. Roberts

Re: the fdisk man page and the fdisk behaviour

[J.C. Roberts]

On Tue, 31 Mar 2009 21:44:44 -0400 Nick Holland
 wrote:

> > what people dont realize sometimes that yapping about
> > and not giving patches is a sign of "wanting something"
> > but "not giving back".  i echo my ideas on the mailing
> > list because if any of it is usable, a developer could
> > make it happen probably 100x sooner, or of finally i
> > get my sh*t together it is just flatly refused.  life
> > is too short for that.
> 
> Hey, I beat you to whining about fdisk and disklabel by probably
> almost ten years. :)  And actually, a lot HAS improved since
> then, in small, incremental ways.  LBA made life a LOT easier. :)
> 
> I think most the developers wouldn't mind seeing a smoother fdisk
> program...as long as it fit all the other criteria that a good
> fdisk program needs to fit.
> 
> Talking about it from a x/255/63 and i386 perspective, however
> is not a productive starting point.  There is way, way too much
> else in this very complicated world of small computers...

I've enjoyed the rants about the old and odd stuff, and I hope the two
of you don't mind me butting in, but both of you forgot to rant about
something important, namely, new types of storage. 

On a the lower and fundamental levels, solid state storage does not have
the limitations or organization of rotating storage (disks), but none
the less, in current products the new tech has been (intentionally)
impaired and implemented with the old ideas to provide ass-backwards
compatibility. At present, the Solid State (storage) Disks/Devices
(SSD's) currently available have been designed for the sake of selling
into existing markets where being ass-backwards compatibility is,
unfortunately, a requirement.

Though my employers would shoot me for violating an NDA, and Theo
would shoot me for signing an NDA in the first place, for me it's tough
to make a living with out them, so I have to be vague and leave out
important details.

There are efforts afoot to abandon the limitations and organization of
rotational storage, so both existing "disk layout" tools such as fdisk
and disklabel, and even file systems, will eventually need to change to
benefit from new technology.

The whole concept of sectors, 512 byte or otherwise, on solid state
storage systems/devices is really just a sad kludge. The concept of a
"disk controller" is already out dated and will soon be abandoned.

The following is publicly available information, is nearly two years
old, is still using ass-backwards compatibility, and is obviously quite
a few redesign revisions behind reality, but it should give you
an idea of where the storage world is headed:
http://www.tgdaily.com/index.php?option=com_content&task=view&id=34065

Thoughts?

-- 
J.C. Roberts

Re: the fdisk man page and the fdisk behaviour

[frantisek holop]

hmm, on Wed, Apr 01, 2009 at 01:41:52PM +0200, Henning Brauer said that
> * Nick  [2009-04-01 03:52]:
> > I think most the developers wouldn't mind seeing a smoother fdisk
> > program...
> 
> what is fdisk?
> 
> oh I remember using it. fdisk -i sd1. why it can do anything else is
> almost beyond me :)

yes, sometimes life is that easy :]

but it is a bit like using pf only for pass any---not a challenge :p

-f
-- 
je blesse seulement l'une que j'aime

Re: the fdisk man page and the fdisk behaviour

[Henning Brauer]

* Nick  [2009-04-01 03:52]:
> I think most the developers wouldn't mind seeing a smoother fdisk
> program...

what is fdisk?

oh I remember using it. fdisk -i sd1. why it can do anything else is
almost beyond me :)

-- 
Henning Brauer, h...@bsws.de, henn...@openbsd.org
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam

Re: the fdisk man page and the fdisk behaviour

[Nick Holland]

frantisek holop wrote:
> hmm, on Wed, Apr 01, 2009 at 05:05:10AM +0200, frantisek holop said that
>> i wonder what will happen when 65536/255/63 is reached,
>> what kind of frankenstein translation table will come into life
>> just to save c/h/s again.  maybe there will come an LBA only age,
>> i mean fdisk-wise, not disk-wise, those are already LBA only...
> 
> openbsd could be the first one to nuke c/h/s from the MBR
> entirely, replace it with 8 or 16 byte LBA values (4 bytes
> currently) and lead the way for terribly huge disks :]
> 
> -f

old news.
65536*255*63*512
539,051,950,080 bytes
(the marketing people would probably call that a 640G or 750G disk :)

And here is a 1T-ish system I've got running (which I mistakenly called
a 1.5T system on a prev. post, I think).

fdisk: 1> p
Disk: wd0   geometry: 1938037/16/63 [1953542144 Sectors]
Offset: 0   Signature: 0xAA55
Starting Ending LBA Info:
 #: id  C   H   S -  C   H   S [   start:size ]
---
 0: 00  0   0   0 -  0   0   0 [   0:   0 ] unused
 1: 00  0   0   0 -  0   0   0 [   0:   0 ] unused
 2: 00  0   0   0 -  0   0   0 [   0:   0 ] unused
*3: A6  0   1   1 - 1938036  15  63 [  63:  1953541233 ] OpenBSD
fdisk: 1> p g
Disk: wd0   geometry: 1938037/16/63 [932 Gigabytes]
Offset: 0   Signature: 0xAA55
Starting Ending LBA Info:
 #: id  C   H   S -  C   H   S [   start:size ]
---
 0: 00  0   0   0 -  0   0   0 [   0:   0G] unused
 1: 00  0   0   0 -  0   0   0 [   0:   0G] unused
 2: 00  0   0   0 -  0   0   0 [   0:   0G] unused
*3: A6  0   1   1 - 1938036  15  63 [  63: 932G] OpenBSD


These "problems" just aren't problems.

Nick.

Re: the fdisk man page and the fdisk behaviour

[Kenneth R Westerback]

On Tue, Mar 31, 2009 at 08:13:22PM +0200, frantisek holop wrote:
> first of all, thanks Nick for your time going through
> all of that.  here is my answer.  it is all quite i386
> specific though.
> 
> hmm, on Tue, Mar 24, 2009 at 11:34:38PM -0400, Nick Holland said that
> > I really don't want to ignore what you call an "exotic geometry".
> > A lot of people seem to think there is something "standard" about
> > what you are calling "n/255/63 geometry", but life ain't so
> 
> well, the commercial sector making partitioning software seems
> to disagree here.  the quite popular PartitionMagic comes to my
> mind.  and openbsd itself had a commit in the past to show every
> sd(4) disk using this very same geometry if i am not mistaken.

Also sprach the commit messag for r1.125 of sd.c:

"For devices unwilling or unable to report geometry, change the 'fake'
geometry used to 255 heads and 63 sectors/track from 64 heads and 32
sectors. 255/63 makes the cylinders as large as PC/BIOS compatibility
allows, increasing the size of devices that can be supported."

The sd driver tries to determine the geometry of the disk by reading the
PAGE_REDUCED_GEOMETRY, PAGE_RIGID_GEOMETRY or PAGE_FLEX_GEOMETRY mode
pages. If no coherent information is obtained (or the device is USB in
which case we throw up our hands in despair and don't even try to
read the info) then and only then is a 'default' geometry used.

As the code says:

/*
 * XXX THINK ABOUT THIS!!  Using values such that sectors * heads *
 * cyls is <= disk_size can lead to wasted space. We need a more
 * careful calculation/validation to make everything work out
 * optimally.
 */
if (dp->disksize > 0x && (dp->heads * dp->sectors) < 0x) {
dp->heads = 511;
dp->sectors = 255;
cyls = 0;
} else {
/*
 * Use standard geometry values for anything we still don't
 * know.
 */
dp->heads = (heads == 0) ? 255 : heads;
dp->sectors = (sectors == 0) ? 63 : sectors;
dp->rot_rate = (rpm == 0) ? 3600 : rpm;
}

dp->cyls = (cyls == 0) ? dp->disksize / (dp->heads * dp->sectors) :
cyls;

if (dp->cyls == 0) {
dp->heads = dp->cyls = 1;
dp->sectors = dp->disksize;
}

The development of a useful/careful calculation/validation is left as an 
exercise for the reader.

 Ken

Re: the fdisk man page and the fdisk behaviour (OT)

[Raymond Lillard]

frantisek holop wrote:

> 
> -f
>
> --
> number of vulcans to replace a bulb?  precisely 1.00.

Are you sure it isn't 1.000?  or maybe 1.?

Way back in the Dark Ages when I went to school, 1 was
thought to be more precise than 1.00.  I think it
still might be so in some circles.

:-)
Ray

PS  Just consider this a small diversion from the current flame war.

PPS: Thanks to all who are improving the fdisk and disklabel
 programs.

Re: the fdisk man page and the fdisk behaviour

[frantisek holop]

hmm, on Wed, Apr 01, 2009 at 05:05:10AM +0200, frantisek holop said that
> i wonder what will happen when 65536/255/63 is reached,
> what kind of frankenstein translation table will come into life
> just to save c/h/s again.  maybe there will come an LBA only age,
> i mean fdisk-wise, not disk-wise, those are already LBA only...

openbsd could be the first one to nuke c/h/s from the MBR
entirely, replace it with 8 or 16 byte LBA values (4 bytes
currently) and lead the way for terribly huge disks :]

-f
-- 
number of vulcans to replace a bulb?  precisely 1.00.

Re: the fdisk man page and the fdisk behaviour

[frantisek holop]

hi there,

i am just reading up on things, and here is a very
nice explanation how and why some of the bioses chose
their geometry.

http://www.pcguide.com/ref/hdd/bios/modesECHS-c.html

quite some russian rulette, innit :]

it also kind of explains why when a certain disk capacity
is surpassed, only the n/255/63 logical c/h/s geometry makes
sense---the numbers are just getting bigger and bigger...

i wonder what will happen when 65536/255/63 is reached,
what kind of frankenstein translation table will come into life
just to save c/h/s again.  maybe there will come an LBA only age,
i mean fdisk-wise, not disk-wise, those are already LBA only...

pc technology is the ugliest no matter where one looks
(640K, ATA disks, pciide, ACPI, the list goes on and on)
complete lack of visionary thinking.

and in the other side of the ring, complete overengineering
(ipv6, EFI, ASN.1, etc)

-f
-- 
self-reference (n.): see self-reference

Re: the fdisk man page and the fdisk behaviour

[Nick Holland]

frantisek holop wrote:
> first of all, thanks Nick for your time going through
> all of that.  here is my answer.  it is all quite i386
> specific though.

You have to remember, the OpenBSD project doesn't have
that luxury.  These guys bust their butts to keep OpenBSD
clean and 100% functional on 17 platforms, PLUS whatever
is coming up in the future.

fdisk kinda started out as an i386 tool (a bad and
complicated way to do a simple task, but that was how the
IBM AT spec'd things out), but a lot of new
platforms use it now, too, and that is likely to continue
as a lot of new platforms seem to be designed by people
who think the IBM AT way is The Way to do things...

> hmm, on Tue, Mar 24, 2009 at 11:34:38PM -0400, Nick Holland said that
>> I really don't want to ignore what you call an "exotic geometry".
>> A lot of people seem to think there is something "standard" about
>> what you are calling "n/255/63 geometry", but life ain't so
> 
> well, the commercial sector making partitioning software seems
> to disagree here.  the quite popular PartitionMagic comes to my
> mind.  and openbsd itself had a commit in the past to show every
> sd(4) disk using this very same geometry if i am not mistaken.

I've seen a tremendous variation in sd(4) geometry myself...

Not just on i386, not just OpenBSD...but also SPARC systems
running Solaris (one computer, two very very similar drives,
totally different geometries.  I'm not even going to try to
pretend to explain that one, or the fact that if one dd's over
the first meg or so of disk from one to the other, the recipient
changes geometry to match the source...)

Anyway...keep in mind the goals of OpenBSD's fdisk and the
commercial i386-centric systems are quite different.  I think
a degree of disagreement is fine here, and in fact, since I've
used OpenBSD's fdisk to fix or customize commercial disk layouts
before, I'm glad for it.

> here is another mail i sent in a looong time ago, another saga
> of mine, where because of openbsd's different geometry settings
> overlapping partitions and other filesystem errors manifested
> themselves, no doubt because of my inability to calulate all
> the numbers using a different geometry.
> http://marc.info/?l=openbsd-misc&m=116475065104125&w=2

looking back at that, I see a few clues of issues that are
potentially completely unrelated to geometry, and a few things
that might be related to fixed bugs.  Good detail provided,
though. :)

>> simple.  Not at all.  First of all, there are a lot of non-i386
>> and non-amd64 systems that use fdisk, and a lot of very usable
>> machines for OpenBSD are, as you put it, "from the history books".
>> 
>> Without looking very hard, I found these "exotics":
> 
> yes, but are they running -current?  

hey, if I'm not running a relatively fresh -current on EVERYTHING
I've got, Theo's gonna hurt me, and hurt me badly. :)
(I keep a few machines back at prev. -release to test the upgrade
instructions I write, but in this cycle, even several of those ended
up being -current, too)

> and most importantly,
> would you call them "typical", as in where typical means:
> 
> typ-i-cal, adj.
>   2. Of or relating to a representative specimen;
>   characteristic or distinctive.
> 
> all your examples are without a doubt valid, but are they
> representative of the current hard drive situation?  that's
> what typical means for me in this context.  if you have to
> hunt for these disks on ebay, they are hardly typical or
> representative for me...

For IDE drives, it's the computer, or more accurately, the
computer's BIOS (or the BIOS on the card or ..), not the drive.

When it comes to i386 systems, I'm torn between saying there
isn't much exotic...and there isn't much typical, either.

Nothing I showed was really strange, at least for its vintage.
All of it was picked to show oddities.  Several things shocked
me, like the machine with SATA and IDE, where the IDE showed
as you expected...and the SATA showed up looking like a very
old pre-LBA system..in spite of being attached to a nearly new
SATA card.

> numeruous people stated on this mailing list that geometries
> had become abstract numbers in the world of modern hard drives
> and are basically useless anyway.  if that is the case, i do
> not see a good reason why not go with the other mainstream
> operating systems and just do make with an n/255/63 geometry
> everywhere (where possible).

On modern LBA systems, yes, the disk is just a sequence of
sectors.  However, where it will bite you is with systems
with buggy LBA BIOSs or non-LBA BIOSs...and those, you will
probably need to be precise.

And then, we would probably have to have two fdisks, one for
i386 and one for everything else...for a non-reason.

> i havent seen hard drive's described by their c/h/s in any bios
> for years (yes, i know, you probably have that old machines
> as well ;-)

a couple.
for really, really large values of "couple" :-/

HOWEVER, they do exist.  And, there are a lot o

Re: the fdisk man page and the fdisk behaviour

[frantisek holop]

first of all, thanks Nick for your time going through
all of that.  here is my answer.  it is all quite i386
specific though.

hmm, on Tue, Mar 24, 2009 at 11:34:38PM -0400, Nick Holland said that
> I really don't want to ignore what you call an "exotic geometry".
> A lot of people seem to think there is something "standard" about
> what you are calling "n/255/63 geometry", but life ain't so

well, the commercial sector making partitioning software seems
to disagree here.  the quite popular PartitionMagic comes to my
mind.  and openbsd itself had a commit in the past to show every
sd(4) disk using this very same geometry if i am not mistaken.

here is another mail i sent in a looong time ago, another saga
of mine, where because of openbsd's different geometry settings
overlapping partitions and other filesystem errors manifested
themselves, no doubt because of my inability to calulate all
the numbers using a different geometry.
http://marc.info/?l=openbsd-misc&m=116475065104125&w=2


> simple.  Not at all.  First of all, there are a lot of non-i386
> and non-amd64 systems that use fdisk, and a lot of very usable
> machines for OpenBSD are, as you put it, "from the history books".
> 
> Without looking very hard, I found these "exotics":

yes, but are they running -current?  and most importantly,
would you call them "typical", as in where typical means:

typ-i-cal, adj.
2. Of or relating to a representative specimen;
characteristic or distinctive.

all your examples are without a doubt valid, but are they
representative of the current hard drive situation?  that's
what typical means for me in this context.  if you have to
hunt for these disks on ebay, they are hardly typical or
representative for me...

numeruous people stated on this mailing list that geometries
had become abstract numbers in the world of modern hard drives
and are basically useless anyway.  if that is the case, i do
not see a good reason why not go with the other mainstream
operating systems and just do make with an n/255/63 geometry
everywhere (where possible).

i havent seen hard drive's described by their c/h/s in any bios
for years (yes, i know, you probably have that old machines
as well ;-)

> > violates the CAVEATS section of the man page on two counts:
> > 
> > 1. the ending boundary of the 0th partition does not "end at
> > cylinder boundaries."
> > 2. the openbsd partition does not "start on a cylinder boundary
> > (head 0, sector 1), except when starting on track 0,
> > (these should begin at head 1, sector 1)."
> > 
> > both these are cited as necessary for i386 machines...
> 
> no, key word is "SHOULD", it is very far from "necessary" on
> most modern OSs.

i think this is besides the point somewhat.  ok, it is not
a problem to have partitions end/begin anywhere on the disk
(except where the MBR is) but what is the point of it?  why
make your life harder?  if the powers that were decided that
disks will have c/h/s, and now that disks got insanely bigger
(>8G), only the n/254/63 (n/max/max) settings are making any
slight sense (otherwise the cylinder number wouldn't fit into
a long long) then why not round the numbers up?

it is like saying, ok, let's meet at 10:59:53 instead of 11:00.
all this just because "we can".  even if it is not a requirement
anymore, it is just good practice.


> I really, REALLY don't like people thinking that there's something
> magic about 255 tracks per cylinder...or 63 sectors per track.  Every
> time you see someone say "first partition should be offset 63 sectors"
> you should be thinking "WRONG!".

it is true on systems with the n/255/63 geometry...
but if i said that, i meant it should be offset by the first sector
(for the MBR).

> well, you DID take the default for that.  As mentioned in faq4.html,
> when working around existing partitions, CHS mode is probably easier
> than sector count mode.

i did.  to see what it does actually.

> > sense if it offered me the first available lba sector with partition
> > type 0?  i mean even if it doesnt want to offer any "responsible"
> > value, 0 is wrong in any case on i386, as the first offset has to be
> > at least 63...
> 
> "WRONG" :)

ok, the number should be S, as in sector.  so if not usable
value is offered, at least it shouldnt offer to overwrite the MBR.


> ok, now test your algorithm against this case:
> Four partitions on a disk, laid out front to back as you would probably
> assume:
> 0  10G
> 1  100G
> 2  20G
> 3  120G
> 
> Now, delete partitions 1 and 3.
> Add a partition.  What's its starting sector?  What if I want to
> add a 120G partition?  How do you override it?

this is a nice example.  first of all because i havent tested
yet if fdisk checks for overlaps, i need to look into that.

there is no need for a complicated algorithm.  all fdisk would
do, is simply offer the c/h/s from no. 1 as default (up to
the beginning of the next partition (no 2.) or the end of the
disk).  if the user wants a 120G partit

Re: the fdisk man page and the fdisk behaviour

[Jason McIntyre]

On Mon, Mar 23, 2009 at 02:56:54AM +0100, frantisek holop wrote:
> 
> this is for the dying breed of fdisk gurus...
> prepare some snacks, it's long.
> 

i think nick covered your points pretty well, but just to follow up, we
did find a need for some changes, so i've included those changes below
(now committed to -current).

jmc

Index: fdisk.8
===
RCS file: /cvs/src/sbin/fdisk/fdisk.8,v
retrieving revision 1.66
diff -u -r1.66 fdisk.8
--- fdisk.8 13 May 2008 13:28:15 -  1.66
+++ fdisk.8 30 Mar 2009 04:47:21 -
@@ -178,10 +178,11 @@
 .El
 .Pp
 .Em NOTE :
-The BIOS geometry sectors field is
+The BIOS geometry sectors field (C/H/S) is
 .Dq 1 based ,
-but the start field is
+but the LBA "start" field is
 .Dq 0 based .
+.Pp
 The CHS values will need to be in the BIOS's geometry
 for the system to be able to boot and use the drive correctly.
 These values must be kept correctly synchronized or a variety of
@@ -256,17 +257,16 @@
 The prompt contains information about the state of the edit
 process.
 .Pp
-.Dl fdisk:*0>
+.Dl fdisk:*1\*(Gt
 .Pp
 .Sq *
 means that the in-memory copy of the boot block has been modified, but
 not yet written to disk.
 .Pp
-0 is the disk offset of the currently selected boot block being edited.
-This number could be something other than zero when extended MBR partitions
-are being edited (using the
-.Em select
-subcommand).
+1 is the disk offset of the currently selected boot block being edited.
+This number will be 2 when editing an extended MBR partition,
+3 when editing an extended MBR partition within an extended MBR partition,
+and so on.
 .Pp
 The list of commands and their explanations are given below.
 .Bl -tag -width Ds
@@ -362,8 +362,8 @@
 .Xr disklabel 8
 .Sh CAVEATS
 Hand crafted disk layouts are highly error prone.
-MBR partitions should start on a cylinder boundary
-(head 0, sector 1),
-except when starting on track 0,
-(these should begin at head 1, sector 1).
-MBR partitions should also end at cylinder boundaries.
+It is common practice,
+though by no means required,
+that MBR partitions start on a cylinder boundary
+(generally head 0, sector 1, but head 1, sector 1 for track 0),
+and that MBR partitions also end at cylinder boundaries.

Re: the fdisk man page and the fdisk behaviour

[Jonathan Thornburg]

In message ,
Nick Holland wrote
[[many examples of disks which don't have an n/255/63 geometry]]
> I'm actually somewhat surprised how many exceptions I found
> with very little effort.  The only thing that surprised me
> was the lack of my finding a "240 head" system, that's a
> classic Compaqism, but apparently my firewall system has too
> small a hard disk. :)  I think the majority of the machines
> I looked at (granted, guessing they would be "odd") were,
> well...odd.

To beat this dead horse just a bit more, here's a "240-head"
geometry from a quite "ordinary" system:

Disk: wd0   geometry: 20673/240/63 [312581808 Sectors]

The system is an (i386) IBM/Lenovo Thinkpad T42 laptop.
The disk is a 160GB Hitachi HTS541616J9AT00 which I bought
new in October 2008.  Needless to say, OpenBSD installed
without a hitch on this disk, and runs fine:
% dmesg|grep wd0
wd0 at pciide0 channel 0 drive 0: 
wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
root on wd0a swap on wd0b dump on wd0b
%

ciao,

-- 
-- "Jonathan Thornburg [remove -animal to reply]" 

   Dept of Astronomy, Indiana University, Bloomington, Indiana, USA
   "Washing one's hands of the conflict between the powerful and the
powerless means to side with the powerful, not to be neutral."
  -- quote by Freire / poster by Oxfam

Re: the fdisk man page and the fdisk behaviour

[Nick Holland]

frantisek holop wrote:
> hi there,
> 
> this is for the dying breed of fdisk gurus...
> prepare some snacks, it's long.
> 
> i am about to install openbsd -current (feb 28) on a notebook
> with a 320G hard drive in IDE mode, although it is AHCI really.
> bsd.rd dmesg at the end.
> 
> my goal is to have the 2 ntfs partitions followed by a 32G
> openbsd one and then the last partition will be an ext2.
> 
> i spent some quality reading time in the man page, and first
> of all let me confess, that i think that the i386 fdisk man
> page section sporting the title "TYPICAL LAYOUT" is more like
> a page out from the history books...  i claim to be no fdisk
> jockey but i have seen a couple in my time, and i say, that
> even if we ignore the now exotic geometry (as most drives now
> identify always with an n/255/63 geometry), this particular MBR

I really don't want to ignore what you call an "exotic geometry".
A lot of people seem to think there is something "standard" about
what you are calling "n/255/63 geometry", but life ain't so
simple.  Not at all.  First of all, there are a lot of non-i386
and non-amd64 systems that use fdisk, and a lot of very usable
machines for OpenBSD are, as you put it, "from the history books".

Without looking very hard, I found these "exotics":
Disk: wd0   geometry: 787/128/63 [6346368 Sectors]
Compaq PII class system (i386)

Disk: wd0   geometry: 969021/16/63 [976773168 Sectors]
Thecus (armish)

Disk: wd0   geometry: 387621/16/63 [390721968 Sectors]
MacPPC

Disk: wd0   geometry: 1938037/16/63 [1953542144 Sectors]
Disk: wd1   geometry: 60801/255/63 [976773168 Sectors]
1.3GHz P4 (i386) (gotta love that one...two different
geometries on one machine!)

Disk: sd0   geometry: 43580/255/32 [355612800 Sectors]
Compaq GL380g2 (1.2GHz PIII) (i386)

I'm actually somewhat surprised how many exceptions I found
with very little effort.  The only thing that surprised me
was the lack of my finding a "240 head" system, that's a
classic Compaqism, but apparently my firewall system has too
small a hard disk. :)  I think the majority of the machines
I looked at (granted, guessing they would be "odd") were,
well...odd.


> violates the CAVEATS section of the man page on two counts:
> 
> 1. the ending boundary of the 0th partition does not "end at
>   cylinder boundaries."
> 2. the openbsd partition does not "start on a cylinder boundary
>   (head 0, sector 1), except when starting on track 0,
>   (these should begin at head 1, sector 1)."
> 
> both these are cited as necessary for i386 machines...

no, key word is "SHOULD", it is very far from "necessary" on
most modern OSs.

IN FACT, if you move a disk from a n/240/63 machine and put it
on a n/255/63, things are most likely not going to end on nice
and neat cyl boundaries.  Some OSs deal with it better than
others, OpenBSD rocks here, it doesn't care, most others will
complain more about the change in HW than the change in
geometry.

> as a quick refresher, here is the offending part:
> 
>   # fdisk wd0
>   Disk: wd0   geometry: 5168/240/63 [78140160 Sectors]
>   Offset: 0   Signature: 0xAA55
> Starting  EndingLBA Info:
>#: idC   H  S -C   H  S [   start:  size   ]
>   
> -
>0: 040   1  1 -  170   0 63 [  63: 2570462 ] DOS FAT-16
>1: 000   0  0 -0   0  0 [   0:   0 ] unused
>2: 000   0  0 -0   0  0 [   0:   0 ] unused
>   *3: A6  170   1  1 - 5167 239 63 [ 2570525:75569697 ] OpenBSD
> 
> i am _not_ stating, that this example is a made up fantasy,
> contrarywise, probably it has been copied from a live working
> operating system.  i am merely pointing out, that it is far
> from "typical" and more confusing than helping, and i make no
> secret of that i'd like to see it being replaced with something
> fresher and/or more relevant (or "typical")... (the faq has both
> in section 4 and 14 newer layouts)

FAQ used to have a 240/63 system, but..uh..it was a very annoying
Compaq, and at one point, I finally decided it wasn't worth the
trouble anymore, so I replaced it with a Dell which had the less
interesting translation.  Guess I need to look a little further. :)

I really, REALLY don't like people thinking that there's something
magic about 255 tracks per cylinder...or 63 sectors per track.  Every
time you see someone say "first partition should be offset 63 sectors"
you should be thinking "WRONG!".

# fdisk sd0
Disk: sd0   geometry: 43580/255/32 [355612800 Sectors]
Offset: 0   Signature: 0xAA55
Starting Ending LBA Info:
 #: id  C   H   S -  C   H   S [   start:size ]
---
 0: 00  0   0   0 -  0   0   0 [   0:   0 ] unused

the fdisk man page and the fdisk behaviour

[frantisek holop]

hi there,

this is for the dying breed of fdisk gurus...
prepare some snacks, it's long.

i am about to install openbsd -current (feb 28) on a notebook
with a 320G hard drive in IDE mode, although it is AHCI really.
bsd.rd dmesg at the end.

my goal is to have the 2 ntfs partitions followed by a 32G
openbsd one and then the last partition will be an ext2.

i spent some quality reading time in the man page, and first
of all let me confess, that i think that the i386 fdisk man
page section sporting the title "TYPICAL LAYOUT" is more like
a page out from the history books...  i claim to be no fdisk
jockey but i have seen a couple in my time, and i say, that
even if we ignore the now exotic geometry (as most drives now
identify always with an n/255/63 geometry), this particular MBR
violates the CAVEATS section of the man page on two counts:

1. the ending boundary of the 0th partition does not "end at
cylinder boundaries."
2. the openbsd partition does not "start on a cylinder boundary
(head 0, sector 1), except when starting on track 0,
(these should begin at head 1, sector 1)."

both these are cited as necessary for i386 machines...
as a quick refresher, here is the offending part:

  # fdisk wd0
  Disk: wd0   geometry: 5168/240/63 [78140160 Sectors]
  Offset: 0   Signature: 0xAA55
Starting  EndingLBA Info:
   #: idC   H  S -C   H  S [   start:  size   ]
  -
   0: 040   1  1 -  170   0 63 [  63: 2570462 ] DOS FAT-16
   1: 000   0  0 -0   0  0 [   0:   0 ] unused
   2: 000   0  0 -0   0  0 [   0:   0 ] unused
  *3: A6  170   1  1 - 5167 239 63 [ 2570525:75569697 ] OpenBSD

i am _not_ stating, that this example is a made up fantasy,
contrarywise, probably it has been copied from a live working
operating system.  i am merely pointing out, that it is far
from "typical" and more confusing than helping, and i make no
secret of that i'd like to see it being replaced with something
fresher and/or more relevant (or "typical")... (the faq has both
in section 4 and 14 newer layouts)


now on to my problems...
here is the fdisk session during the install with my comments.

Disk: wd0   geometry: 38913/255/63 [625142448 Sectors]
Offset: 0   Signature: 0xAA55
Starting Ending LBA Info:
 #: id  C   H   S -  C   H   S [   start:size ]
---
*0: 07  0   1   1 -   8354 254  63 [  63:   134223012 ] HPFS/QNX/AUX
 1: 07   8355   0   1 -  25063 254  63 [   134223075:   268430085 ] HPFS/QNX/AUX
 2: 00  0   0   0 -  0   0   0 [   0:   0 ] unused  
 3: 00  0   0   0 -  0   0   0 [   0:   0 ] unused  
Enter 'help' for information
fdisk: 1> e 2
Starting Ending LBA Info:
 #: id  C   H   S -  C   H   S [   start:size ]
---
 2: 00  0   0   0 -  0   0   0 [   0:   0 ] unused  
Partition id ('0' to disable)  [0 - FF]: [0] (? for help) a6
Do you wish to edit in CHS mode? [n]
offset: [0]
size: [0]
fdisk:*1> p
Starting Ending LBA Info:
 #: id  C   H   S -  C   H   S [   start:size ]
---
*0: 07  0   1   1 -   8354 254  63 [  63:   134223012 ] HPFS/QNX/AUX
 1: 07   8355   0   1 -  25063 254  63 [   134223075:   268430085 ] HPFS/QNX/AUX
 2: 00  0   0   1 - 267349  89   4 [   0:   0 ] OpenBSD 
 3: 00  0   0   0 -  0   0   0 [   0:   0 ] unused  
fdisk:*1>

let's go line by line.  i said i want to edit partition 2, and defined
its type as a6, openbsd.  then fdisk goes into lba editing mode and
offers me offset 0.  this is one of those moments where i dont know
if this is fdisk's expected behaviour, or a bug.  would it not make
sense if it offered me the first available lba sector with partition
type 0?  i mean even if it doesnt want to offer any "responsible"
value, 0 is wrong in any case on i386, as the first offset has to be
at least 63...

i got curious and wanted to see where this leads, so i hit enter.
fdisk then offered size 0, and again, i dont know if this is normal
fdisk behaviour, but it would maybe make sense if it offered the last
free lba sector (starting from offset) or up to the end of the disk,
if that being the case.  in any case, even if the offered size was
0, the result is obviously wrong, both the starting and ending C/H/S
values are bogus, although the lba size value is correct.

let's stop for a moment at the number in the prompt: 1.  the man page