The main reason that this went unnoticed on my side is that I am slow in
upgrading my distro to a newer version, so I was using older cpmtools package
version for a while. And probably not many people use custom diskdefs...
Anyhow, thank you very much for the analysis and workaround solution, I would
not have figured it out on my own that fast. I did not know about transition to
libdsk, although I am familiar with the library and used it in other contexts.
I am leaving for work soon, but I will check this on my side later in the day.
---
---------------------------
From: "Jacob Nevins"
To: "Zarko Zivanov"
Cc: "undefined" <1079...@bugs.debian.org>
Sent: Monday, August 26, 2024 3:56 AM
Subject: Re: Bug#1079619: cpmtools package produces garbage output with cpmls
> [Zarko Zivanov:]
>> I have previously used cpmtools_2.20 and had no problems. With the
>> recent version from repository, I am getting garbage printed on
>> screen when I try to list the contents of an IMG file (or, actually,
>> many IMG files, this one is just an example).
I wrote:
> I think the significant difference is probably that between 2.20-2 and
> 2.23-1, Debian's package started to be built against libdsk.
(Really, 2.23-3.)
OK, I think I know what's going on.
(Sorry, this is a bit of a braindump.)
I think the main fix is going to have to be in upstream libdsk -- I
think it's a misfiring heuristic there -- but I don't have a complete
strategy yet.
If I look at your image contents with "fsed.cpm -f tim011 demo.img"
with my LibDsk build of cpmtools (which should be similar to Debian's),
I can see that the tracks are misnumbered compared to my plain build --
what ought to be odd-numbered tracks (1, 3, etc) end up together as
track 80+, with the first tracks displayed being what ought to be track
0, 2, 4, ...
So the CP/M directory ends up in the wrong place (diskdefs tells
cpmtools it should be at the start of track 2), and cpmls shows garbage.
Vanilla cpmtools doesn't really have a notion of heads or sidedness --
to it, this disc image is a consecutive set of 160 tracks in ascending
order 0..159, without worrying about the relationship of track number to
(cylinder, head).
libdsk has a more sophisticated notion of heads/sidedness, and supports
image formats where tracks aren't ordered in the obvious way, and has
various heuristics to determine disc geometry from image file contents,
so there's more to go wrong.
Using libdsk's 'dskid' tool (which in Debian lives in the libdsk-utils
package) on your image to see what libdsk guesses about its geometry in
the absence of cpmtools' diskdefs, I can see libdsk has got an idea of
the geometry from somewhere which is utter nonsense ("Heads: 128" etc):
$ dskid demo.img
demo.img:
Driver: Raw file driver (alternate sides)
Sidedness: OutOut
Cylinders: 110
Heads: 128
Sectors: 49
First sector: 1
Sector size: 32768
Data rate: SD
Record mode: MFM
Complement: No
R/W gap: 0x2a
Format gap: 0x52
Drive status: 0x28
In particular, I think the "Sidedness: OutOut" in libdsk's garbage
geometry is causing our trouble -- that will cause the tracks to be
reordered.
Within cpmtools, the libdsk driver can cross-check this auto-detected
geometry with what diskdefs says, and correctly rejects this as garbage.
However, it isn't careful enough to discard all fields from the
garbage geometry; in particular, "Sidedness" leaks through.
(cpmtools device_libdsk.c:Device_setGeometry())
So where did the nonsense come from?
In the absence of clues about geometry from the disc image format (of
which there aren't any with a raw disc image like this, for which libdsk
correctly auto-selects its 'raw' driver), libdsk tries to infer clues
from the (presumed) boot sector at the start of the disc image.
Reading the libdsk source, what I think is happening is that libdsk is
misdetecting this image as having the boot sector of an "Opus Discovery"
(libdsk lib/dskgeom.c:dg_opusgeom()).
That has very weak magic -- it just checks whether the first byte is
0x18, and if it is, then it reads geometry out of nearby bytes, without
any sanity checking. The result matches the garbage we see from 'dskid'.
And indeed the first byte of your disc image is 0x18 ("Z80 relative
jump", so hardly unlikely to show up in an unrelated Z80 machine's boot
sector).
So what needs to happen is:
- libdsk's dg_opusgeom() needs to get more discriminating, at the
very least, so it doesn't misfire on disc images like yours (but I
don't know enough about the Opus Discovery to speculate how);
- maybe libdsk could usefully learn about TIM 011 boot sectors
specifically, especially if they have geometry information embedded
in them. (Again, I know nothing about these; do you know of docs/
people who might know more? Ideally English-language, but
understand if not.)
But this isn't strictly necessary; if none of the boot sector
heuristics misfire, libdsk's dsk_defgetgeom() should return an
inoffensive default geometry that doesn't cause trouble.
- probably cpmtools' device_libdsk.c:Device_setGeometry() needs to be
more thorough about sanitising DSK_GEOMETRY if it decides the
autodetected geometry was garbage.
Workarounds:
One workaround which gets your image working with my libdsk-built
cpmtools is to add "-T rawoo" to the command-line.
This is a fragile bodge, though; it compensates "Sidedness: OutOut" at
a different layer rather than reversing it, and it might not stay
working with different disc images or libdsk versions.
Here's a less fragile workaround, which prevents the misfiring heuristic
running at all, by providing the libdsk layer with information about your
disc format (yes this does duplicate what's already in diskdefs to
some extent):
1. Define a custom libdsk format, either in ~/.libdskrc or... some
system-wide location I haven't checked, probably /usr/share
somewhere. Here's a minimal one that works for me (it could have
more parameters, and some of it is based on guesses/web searches
about the TIM 011, caveat emptor); the format is described in the
doc/ subdirectory of the libdsk source distribution:
[tim011]
Sides = Alt
Cylinders = 80
Heads = 2
Sectors = 5
SecBase = 17
SecSize = 1024
2. Tell cpmtools to use that format -- you can either use a
command-line option "-T raw,tim011" (which forces use of the 'raw'
driver with your new 'tim011' format), or add
"libdsk:format tim011" to your diskdefs entry (which leaves libdsk
free to select a driver, but it'll probably correctly choose
'raw').
Putting this down for now, but if no-one else jumps in, I'll make libdsk
upstream aware of this bug report, and we'll see what they have to say.
(Full disclosure: I've been testing with my own builds of cpmtools --
2.23 -- and libdsk -- 1.5.19, rather than the Debian versions. Based
on what I've found, I think my conclusions transfer, however.)
