Re: Re: PWS 433au (Miata) recovery update
On Wed, Jan 30, 2019 at 04:42:23PM -0500, Alex Winbow wrote: > I've heard word that /usr destined to be going away, but frankly I'm > very > surprised that multiple local filesystems is a rarity these days. The debian > installer even creates these semi-automatically. It is seriously the case > that "everyone" has /var and /tmp on the root filesystem? Can't speak to all distros, but at least for RedHat/CentOS and derivatives, the default partitioning scheme is to put all persistent files on a single partition, possibly spanning multiple spindles (a logical volume). Files of a temporary nature get put on filesystems of a correspondingly temporary nature, which is to say "/run", "/dev", "/tmp", and anything I may have left out that does not need to survive a reboot gets put on separate filesystems (other than "/"). Distributions that have made their peace with "systemd" *have* to have "/usr" present at boot or shortly thereafter. CentOS 7.X has no separate "/usr": it's all symlinked to corresponding directories under "/". Example: "/usr/lib" --> "/lib". That's why there's no supported upgrade path from CentOS 6.X to CentOS 7.X (except for *very* carefully defined server configurations), just in case you were wondering. The "old school" way of partitioning disks tried to separate things into partitions based somewhat on where the contents came from and how likely they were to change. Three basic categories of software: (1) came with the OS; (2) other supported -- typically commercial -- software; and (3) home-grown (user written) software. There was always debate about "/usr" vs. "/opt" for the second category, and the UN*X OS vendor typically decided for you :-). Prior to the advent of true temporary filesystems (only became possible as memory became cheap and plentiful), you wanted to give filesystems with high activity their own separate partitions. This would obviously include things like "/tmp", and "/var" was usually a good candidate for a separate partition because that's where print spoolers, mail directories, USENET feeds, and so forth typically live(d). USENET was a particularly "nasty" case where the default filesystem creation parameters were typically not what you wanted -- news feeds required lots of inodes to support many small files rather than few large files. These days, the average Linux hobbyist doesn't know or care about the history or reasons why separate filesystems might be a good idea. Disk drives are large, cheap, and fast enough that one generally doesn't have to worry about where the swap partition is created relative to the rest of the disk -- the performance impact just isn't that great, particularly if the system has a metric arse-load of RAM to begin with and hardly ever touches swap. Consider yourself one of the "lucky" hobbyists if you've got an Alpha PWS to play with, becuase you *don't* have enough RAM or disk space that you can afford to be so cavalier in your attitude about disk partitioning :-). I try to do things like put "swap" and "/tmp" near the center of the spindle to minimize seek times/distances from other partitions, and for my PWS, "/tmp" is a local persistent filesystem rather than tmpfs -- there's simply not enough memory on a PWS to waste it. No choice in the matter as far as "/dev", thanks to "systemd" and "udev", but the amount of memory consumed there is minimal relative to the init system itself. Well, I hope this has at least been somewhat entertaining if not helpful :-). I've been playing around with different flavors of UN*X since 1977. First exposure was AT UNIX Sixth Edition on a PDP 11/70. First Linux system was SoftLanding Systems (SLS) Linux on a 386 back in 1992. I inherited my Alpha PWS-433au from a fellow who originally bought it to do some mail server software development in a Digital UNIX environment, then decided he needed a cantilever shelf for his equipment rack more than he needed the computer: I *think* I got the better end of the deal, even with having to make the 90-mile round trip out to his house to deliver the shelf and pick up the computer :-). --Bob
Re: Re: PWS 433au (Miata) recovery update
On Sun, 27 Jan 2019, Frank Scheiner wrote: Does anyone have an archive of "samba-common_4.7.3+dfsg-1_all.deb"? That seems like it may be missing piece for installing the older samba 4.7.3 for now on alpha (the binary packages still being present in the archive). snapshot.debian.org seems to still have it on: https://snapshot.debian.org/package/samba/2%3A4.7.3%2Bdfsg-1/#samba-common_2:3a:4.7.3:2b:dfsg-1 Direct download from: https://snapshot.debian.org/archive/debian/20171124T034111Z/pool/main/s/samba/samba-common_4.7.3%2Bdfsg-1_all.deb Thanks, Frank! I'd forgotten about snapshot.debian.org. That worked, the older samba 4.7.3 is installable on alpha (and my fileserver is up!) On Sun, 27 Jan 2019, Bob Tracy wrote: On Sun, Jan 27, 2019 at 12:25:52PM -0800, Alex Winbow wrote: My understanding is that it's not initramfs-tools that mounts all the (non-root) local filesystems, but systemd (which it looks like you've reported as a bug elsewhere). I was able to pseudo-fix this by changing the fs_passno field in /etc/fstab to '0'. This tells us (or me, anyway) that systemd's logic for automatically setting up and running "fsck.fstype" for local filesystems is broken. I don't think the dynamic generation of services and dependencies for handling local filesystems was part of the "special sauce" for systemd versions prior to version 235-X, which was when things broke on my system. I'm really surprised by that. Multiple local filesystems must be incredibly common across the installed base of systems. I see no sign that what's failing is arch/alpha-specific, but surely this can't be a universal bug?! Have I mentioned today how much I detest "systemd"? :-) I'm trying to keep an open mind, but I will say that I found init scripts a good deal easier to reverse-engineer the logical flow and debug. Another problem I'm seeing is that /var fails to unmount at shutdown. Only /var, not my other local filesystems. This will get solved eventually, but it would get solved more quickly if the case of multiple local filesystems was more common today. I've heard word that /usr destined to be going away, but frankly I'm very surprised that multiple local filesystems is a rarity these days. The debian installer even creates these semi-automatically. It is seriously the case that "everyone" has /var and /tmp on the root filesystem? Thanks, -Alex
Re: Re: PWS 433au (Miata) recovery update
On Sun, Jan 27, 2019 at 12:25:52PM -0800, Alex Winbow wrote: > I'm seeing this also, after installing using the Debian 8.0 installer > and dist-upgrade'ing to unstable (using the SMP kernel trick to get past the > GENERIC issue). My understanding is that it's not initramfs-tools that > mounts all the (non-root) local filesystems, but systemd (which it looks > like you've reported as a bug elsewhere). I was able to pseudo-fix this by > changing the fs_passno field in /etc/fstab to '0'. This tells us (or me, anyway) that systemd's logic for automatically setting up and running "fsck.fstype" for local filesystems is broken. I don't think the dynamic generation of services and dependencies for handling local filesystems was part of the "special sauce" for systemd versions prior to version 235-X, which was when things broke on my system. Setting the fs_passno field to '0' (electing to not run file system checks before mounting) will bite you eventually. It's annoying to have to manually run "e2fsck -p" on three local filesystems (other than '/' and '/usr') and mount them, but at least the boot process isn't so badly broken I can't do it. > Couple other things I found: in /etc/network/interfaces, > "allow-hotplug eth0" doesn't seem to work nicely with systemd, but "auto > eth0" does. Hadn't dug into this enough to determine what's going on, but did notice that running "ifup eth0" (static IP configuration) while in the emergency shell was effective. It's interesting you mention "auto eth0" working, because I've got an IPv6 tunnel interface designated "auto" that depends on "eth0" being up to function properly, and "systemd" happily configures the tunnel interface without "eth0" being present. Have I mentioned today how much I detest "systemd"? :-) This will get solved eventually, but it would get solved more quickly if the case of multiple local filesystems was more common today. --Bob
Re: PWS 433au (Miata) recovery update
On 1/27/19 22:53, Alex Winbow wrote: On Mon, 28 Jan 2019, Michael Cree wrote: samba build-depends on ceph [1] but ceph hasn't built on Alpha for some time [2]. Looks like dtp-relative relocation errors during linking in the build of ceph [3] is the reason. I have a theory that gcc is not taking the spec file listed in its invocation arguments in the correct order with other passed arguments thus we are not getting correct linking for some shared libraries in the repository. That leads to FTBFS in other packages with these dtp-relative relocation errors. I haven't explored my theory enough to make a bug report against gcc. Thanks, Michael. How can I help debug this? The alpha I'm bringing up will be a replacement server. Does anyone have an archive of "samba-common_4.7.3+dfsg-1_all.deb"? That seems like it may be missing piece for installing the older samba 4.7.3 for now on alpha (the binary packages still being present in the archive). snapshot.debian.org seems to still have it on: https://snapshot.debian.org/package/samba/2%3A4.7.3%2Bdfsg-1/#samba-common_2:3a:4.7.3:2b:dfsg-1 Direct download from: https://snapshot.debian.org/archive/debian/20171124T034111Z/pool/main/s/samba/samba-common_4.7.3%2Bdfsg-1_all.deb
Re: Re: PWS 433au (Miata) recovery update
On Mon, 28 Jan 2019, Michael Cree wrote: samba build-depends on ceph [1] but ceph hasn't built on Alpha for some time [2]. Looks like dtp-relative relocation errors during linking in the build of ceph [3] is the reason. I have a theory that gcc is not taking the spec file listed in its invocation arguments in the correct order with other passed arguments thus we are not getting correct linking for some shared libraries in the repository. That leads to FTBFS in other packages with these dtp-relative relocation errors. I haven't explored my theory enough to make a bug report against gcc. Thanks, Michael. How can I help debug this? The alpha I'm bringing up will be a replacement server. Does anyone have an archive of "samba-common_4.7.3+dfsg-1_all.deb"? That seems like it may be missing piece for installing the older samba 4.7.3 for now on alpha (the binary packages still being present in the archive). Thanks, -Alex
Re: Re: PWS 433au (Miata) recovery update
On Sun, Jan 27, 2019 at 12:25:52PM -0800, Alex Winbow wrote: > Samba isn't installable, and in fact I can't even build-dep to build > myself locally; looks like the dependency chain goes all the way back to a > specific version of libboost!?!? samba build-depends on ceph [1] but ceph hasn't built on Alpha for some time [2]. Looks like dtp-relative relocation errors during linking in the build of ceph [3] is the reason. I have a theory that gcc is not taking the spec file listed in its invocation arguments in the correct order with other passed arguments thus we are not getting correct linking for some shared libraries in the repository. That leads to FTBFS in other packages with these dtp-relative relocation errors. I haven't explored my theory enough to make a bug report against gcc. Cheers Michael. [1] https://buildd.debian.org/status/package.php?p=samba=sid [2] https://buildd.debian.org/status/package.php?p=ceph=sid [3] https://buildd.debian.org/status/fetch.php?pkg=ceph=alpha=12.2.10%2Bdfsg1-1=1546145667=0
Re: Re: PWS 433au (Miata) recovery update
Bob, I'm seeing this also, after installing using the Debian 8.0 installer and dist-upgrade'ing to unstable (using the SMP kernel trick to get past the GENERIC issue). My understanding is that it's not initramfs-tools that mounts all the (non-root) local filesystems, but systemd (which it looks like you've reported as a bug elsewhere). I was able to pseudo-fix this by changing the fs_passno field in /etc/fstab to '0'. Now, systemd happily mounts all the local filesystems at boot, and I no longer get dropped to the emergency shell. So, the trouble seems to be with running fsck on these fs, but I've no idea what the cause may be. Couple other things I found: in /etc/network/interfaces, "allow-hotplug eth0" doesn't seem to work nicely with systemd, but "auto eth0" does. Samba isn't installable, and in fact I can't even build-dep to build myself locally; looks like the dependency chain goes all the way back to a specific version of libboost!?!? Thanks, -Alex
Re: PWS 433au (Miata) recovery update
On Fri, Jan 18, 2019 at 12:19:52AM +, Maciej W. Rozycki wrote: > (much useful information set in the appropriate historical context) Thank you for your thoughts. The earlier reported problem with "/lib/systemd/systemd-udevd" evidently requiring AF_UNIX socket support to be built-in rather than modular has been confirmed. Setting "CONFIG_UNIX=y" in the kernel configuration was enough to get me past that particular problem I was seeing with the initial ramdisk. So, per advice I was given a long time ago, *do* examine the "systemd" README file under "/usr/share/doc": many kernel configuration requirements are mentioned there. As far as gleaning the additional udev-related info, one *might* infer it from the error messages produced by the executable, *or* one can examine the udev- related files under "/lib/systemd/system", one of which explicitly mentions AF_UNIX in the context of a restricted address family. I also note that the current "initramfs-tools" have evidently forgotten how to automatically check and mount local file systems *other* than "/" and "/usr". Every boot since restoring my PWS thus far has dropped me into emergency mode with everything mounted read-write and ready to go (including swap) *other* than the local non-tmpfs file systems. Manually running the appropriate flavor of "fsck" and mounting the file systems before exiting emergency mode results in the expected normal startup of multi-user system services. "journalctl -xb" has, for the case of one such file system that didn't get checked/mounted, the following messages: (...) -- The job identifier is 271 and the job result is done. Dec 21 13:02:10 smirkin systemd[1]: Starting of /dev/sda2 not supported. -- Subject: A start job for unit dev-sda2.device has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit dev-sda2.device has finished with a failure. -- -- The job identifier is 307 and the job result is unsupported. Dec 21 13:02:10 smirkin systemd[1]: Dependency failed for File System Check on /dev/sda2. -- Subject: A start job for unit systemd-fsck@dev-sda2.service has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit systemd-fsck@dev-sda2.service has finished with a failure. -- -- The job identifier is 306 and the job result is dependency. Dec 21 13:02:10 smirkin systemd[1]: Dependency failed for /boot. -- Subject: A start job for unit boot.mount has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit boot.mount has finished with a failure. (...) My guess is, device naming conventions have, once again, changed as far as what the systemd service descriptions/templates expect. Anyone have any idea how and/or where to fix this most efficiently? --Bob
Re: PWS 433au (Miata) recovery update
On Wed, 16 Jan 2019, Bob Tracy wrote: > Odd thing about the disk partitioning scheme. The disk label definitely > has to be "bsd" for SRM to be happy, but if Linux is the only OS on the > disk, all the rest of the BSD partitioning conventions don't have to be > observed as far as slice "c" spanning the entire disk, slice "a" being > the "boot" slice, slice "b" being "swap", and so forth. I doubt dual- > booting with Digital UNIX or one of the *BSD variants is a practical > possibility for most people, particularly those with PWS systems having > limited disk space. SRM doesn't actually care about or know the disk partitioning scheme, however it does require a signature and a pointer to the location of the boot loader (with `aboot' being the usual choice for Linux) in the first physical sector, and these structures happen to clash with the location of the IBM PC's DOS-style partition table. As you observe the partitions for Linux use can be arbitrary. I used to install `aboot' outside any partition by just leaving a number of sectors at the beginning of the disk unassigned; `aboot' takes some 80KiB only. This is analogous to disks using a DOS-style partition table where you'd often leave the whole (simulated) first track of first cylinder unassigned for boot loader use. This saves a partition table entry for a data area that can hardly be used through the corresponding block device anyway. NB be aware if you ever boot Windows NT on an Alpha machine that has an SRM signature on some disks, then the first sector of these disks will get corrupted under NT's assumption that these disks have no data and need to have a DOS-style partition table installed in the first sector. Any OSF/1 disk label will be retained, however the SRM signature will be lost making such disks unbootable and will have to be restored to make them bootable again. Also the presence of the DOS-style partition table marker may confuse the Linux kernel if support for such partition tables has been compiled in in the addition to support for OSF/1 disk labels. > A brief note about partitioning programs: "fdisk" is NOT your friend > on the Alpha, especially in "modern" times. Use "parted" and save > yourself much frustration. I had a look at the GIT history of `util-linux' and it looks like in the conversion of the tool to use `libfdisk', also included with `util-linux', support has been lost, which I find rather unfortunate. It should be pretty straightforward to resurrect with the aid of old code, however it was never particularly clean (as has not been what still remains in `fdisk') due to the use of `#ifdef __alpha__' to select the label type, meaning you could not work with a disk intended for an Alpha system using fdisk compiled for a different host architecture. It looks to me like the current framework would make it easier to avoid this #ifdef hack, but obviously any remains would have to be cleaned up. There used to be a tool called `minlabel' too that you could use to partition your Alpha disk, but it was rather crude. > (1) "systemd" (and "udevd" by extension) don't play well with "/usr" > being on a separate partition from "/". If I have *any* advice to offer > both the battle-scarred veteran and the newbie, it would be to consider > consolidating those two partitions into a single partition. Me? I'd > prefer the younger generation of system programmers consider the > perfectly valid reasons why those filesystems might have been separate > to begin with, and respect those reasons. (Hint: much smaller disks.) In the old days it used to be a common practice to have /usr NFS-mounted in some installations, often where NIS was also used; perhaps it is not anymore. FWIW, Maciej
Re: PWS 433au (Miata) recovery update
On Wed, Jan 16, 2019 at 01:10:14AM -0600, Bob Tracy wrote: > (initramfs / systemd / udevd issue) I think I may have this one painted into a corner... CONFIG_UNIX (at least) needs to be "y" instead of "m". This is a relatively new dependency. For userland applications used in the boot process to have such dependencies on kernel configuration options is "unfortunate" to put it mildly. --Bob
PWS 433au (Miata) recovery update
Gentlemen, Figured it was past time for an update, now that I actually have the Alpha back on-line and functioning in its pre-meltdown capacity as my IPv6 router and Linux kernel git repository. The following narrative is going to necessarily be somewhat long-winded, seeing as it's intended to be a modern synthesis of knowledge gleaned from out-of-date Debian and Gentoo installation documents, mailing list archives, bitter experience, and source code examination. Make whatever use of it you will. My intent is to have this written down *somewhere* for "the next time". The ability to recover the machine in a somewhat timely fashion was predicated on having reasonably current backups and a way to get them onto the Alpha. I never figured the latter consideration would be the most difficult part of the job. Not to put too fine a point on it, but functional boot media for Alpha is more scarce than it should be. Special thanks to the people at Gentoo (and Matt Turner in particular) for being responsive and fixing the "qla1280" firmware issue that was preventing the effective use of Gentoo's "install-alpha-minimal" image as a recovery tool. After a few off-line conversations with Michael, I'm cautiously optimistic we'll eventually see a useful Debian NETINST image at some point in the not-too-distant future. The Gentoo image had neither the requisite USB drivers nor "ntfs-3g" filesystem support, so I had to mount my external USB drive remotely and copy my backups across the network. Not too much pain, even over a relatively slow 10 Mbit/s link. Perhaps somewhat fortuitously, I used a 36 GB disk in the PWS with a layout something like this: (reserved for aboot) /boot (about 75 MB) / (about 4 GB) swap(about 2 GB) /tmp(about 3 GB) /usr(about 13.5 GB) /opt(about 13.5 GB) Out of the lot, the real contents of "/opt" didn't have to be present for the system to function when I initially booted off the hard drive, so that was my staging area for the backups. Once I was running off the hard drive, the plan was to hook up the USB drive and restore "/opt". Odd thing about the disk partitioning scheme. The disk label definitely has to be "bsd" for SRM to be happy, but if Linux is the only OS on the disk, all the rest of the BSD partitioning conventions don't have to be observed as far as slice "c" spanning the entire disk, slice "a" being the "boot" slice, slice "b" being "swap", and so forth. I doubt dual- booting with Digital UNIX or one of the *BSD variants is a practical possibility for most people, particularly those with PWS systems having limited disk space. A brief note about partitioning programs: "fdisk" is NOT your friend on the Alpha, especially in "modern" times. Use "parted" and save yourself much frustration. Using "parted", I set the default units to "cyl" and created a "sacrificial" first partition beginning on cylinder 0 and ending on cylinder 1. This is detected by Linux as, for example "sda1" and should not be used for anything on the off chance "aboot" installation overwrites it. So, the sequence of partition creation commands was: mkpart 0 1 mkpart 1 a mkpart a b mkpart b c mkpart c d mkpart d e mkpart e f where the letters "a" through "f" represent starting and ending cylinder numbers for each partition, and the starting cylinder for each partition is the ending cylinder of the preceding partition, and yes, "parted" makes sure things don't overlap. Bonus: when it comes time to do "swriteboot", you don't have to specify "-f3" because there's no slice 3 spanning the entire disk to prevent "swriteboot" from writing the boot sectors. Once I copied my backups into place (with the exception of "/opt" as mentioned earlier) and wrote the boot sector, I ran into an interesting show-stopper. I had evidently upgraded the "initramfs-tools" package prior to creating my backups, and, long story short, I was getting dropped into an interactive shell with an "(initramfs)" prompt due to the following braindamage: (1) "systemd" (and "udevd" by extension) don't play well with "/usr" being on a separate partition from "/". If I have *any* advice to offer both the battle-scarred veteran and the newbie, it would be to consider consolidating those two partitions into a single partition. Me? I'd prefer the younger generation of system programmers consider the perfectly valid reasons why those filesystems might have been separate to begin with, and respect those reasons. (Hint: much smaller disks.) (2) Perhaps as a consequence of (1), "/lib/systemd/systemd-udevd" refuses to start/run on the initramfs, in spite of the appropriate support being enabled in the kernel configuration per systemd's README file. The error messages appearing on the console are: error getting socket: Address family not supported by protocol error initializing udev control socket could not listen on fds: Invalid argument This isn't necessarily a fatal error, EXCEPT... (3) The brain-dead
