Re: Re (3): Backup.
On 6/30/24 08:37, Andy Smith wrote: Thank you for that informative discussion of rsnapshot(1) and related. :-) My initial reaction to this thread was to recommend Preston [1]. I still think that is decent advice; both for noobs and for experienced people who missed it. David [1] Preston, W., 2007, "Backup & Recovery", O'Reilly Media, Inc. ISBN: 9780596102463, https://www.oreilly.com/library/view/backup-recovery/0596102461/
Re: Strange Problem
On 6/29/24 10:01, Stephen P. Molnar wrote: On 06/29/2024 12:28 PM, Darac Marjal wrote: On 29/06/2024 15:13, Stephen P. Molnar wrote: I have just restated by Xfce4 user on my Bookworm system and find that I can no longer resize some of the apps on the desktop and the icons in the upper right corner of the tool bar are missing. Often, this is an indication that the window manager isn't running. For XFCE4, the window manager is called "xfwm4". Try pressing "Alt+F2" to bring up the launch dialog, enter "xfwm4" at the prompt and press enter. This is only om my user directory, the root directory is still normal. Don't run X sessions (read: don't run XFCE) as root. I have not the faintest idea as to what might be going on and would greatly appreciate assistance. Thanks in advance. Well, the problem is back. I logged out and back in, and there it was! I tried "Alt+F2" then "xfwm4" (without the quotes, of course) and got nothing. Please power off your computer. Then power on your computer. Then document every command or value you enter into the computer and every response by the computer. If and when the computer does something other than what you expect, document what the computer did and what you expected it to do. Research the proper terminology and use it. Then power off, power on, follow your document, and see if the issues are repeatable. Then post your document. David
Re: installation
On 6/28/24 10:20, dewey rahn wrote: When I used to use Debian when a new release came out (like from 10 to 11) you had to completely reinstall the operation system. Is that the case now? I have invested myself in backup, recovery, and version control/ configuration management. So, a major version upgrade for me consists of backing up data on Debian version X, verifying that all of the system configuration files are checked in to the version control system (CVS), pulling the old system drive, installing a wiped disk drive, doing a fresh install of Debian version X+1, checking out the system configuration files to a side directory, merging system configuration files by hand, and restoring data. I suggest that you start by implementing a version control system that you can use over the network. David
Re: Current best practices for system configuration management?
On 6/22/24 11:33, Dmitrii Odintcov wrote: Hi all, Sorry to resurrect an old-ish thread, but I am facing the exact same task, minus the know-how. Basically I am looking to pre-configure a number of Debian setups - let's say, "server", "laptop" and "PC" - that would contain sets of packages to install (or uninstall), configuration files (including but not limited to /etc/), and possibly arbitrary scripts to execute upon installation (e.g. to make more precise edits to configs). I would like to store these in some central, git-controlled location where I could pull them from to my target machines. This kind of setup could be rather low-level, such as configuring APT sources and preferences, installing and configuring systemd-networkd, etc. Ideally, I would also be able to incorporate these into a Debian installer so I could get a new machine "up and running" with my defaults in one go. It is worth adding that I am not looking to build an "infrastructure", and no setup *between* these machines (networking, file sharing, etc.) is necessary. Generally, I would rather avoid complicated tools like Ansible and those with large dependencies such as interpreted languages (except Perl). So far, equivs and config-package-dev appear most relevant, but the former is somewhat lacking in documentation (or I am in documentation-finding ability), and the latter seems to be focused on config editing/deployment rather than package installation. Would be grateful for some advice! I think the "best" answer depends upon the scale of your installation. I have a SOHO network with a dozen or so Debian, Windows, macOS, and iOS clients, a FreeBSD/ZFS CVS, SSH, and Samba server, and a FreeBSD/ZFS backup server. For system administration, including configuration management, I have gone down the do-it-yourself (DIY) path using lowest-common-denominator FOSS command line tools. After the network, ssh(1), and rsync(1), the most valuable tool for system administration (including configuration management) has been a version control system. I prefer CVS over Git because CVS provides monotonically-increasing MAJOR.MINOR version numbers via keywords (e.g. $Revision$) that can be included inside managed plaintext files. I create a CVS project for everything I want to manage. For OS's, I check in a sysadmin log file with my notes and console sessions, a list of packages installed, various reports that I have run, and any system configuration files that I have modified, added, or deleted. After version control, the next most valuable tool has been scripting. Over the years, I have written numerous scripts to automate repetitive chores. When the needs are simple, I write Bourne shell scripts. When I want more power, I upgrade to Perl. All of the scripts are checked in to CVS. I expect Ansible, Puppet, etc., would work at my scale, but are designed for large installations. David
Re: mounting external hard drive from rescue mode shell?
On 6/22/24 10:37, Richard Owlett wrote: I ask about i386 Debian Live because I have a fine operational Sony laptop that currently runs Debian 9.0 and has a $20 price tag on its bottom. This machine has option to boot Debian 11 with an AMD64 kernel. I routinely run Debian 9.13 because its configuration is comfortable (i.e. useful). I have 2 other laptops which will have something >= Debian 12 before I abandon this machine. On 6/22/24 10:49, Richard Owlett wrote: > On 06/22/2024 12:13 PM, Stefan Monnier wrote: >> On 6/22/24 09:57, David Christensen wrote: >>> Rather than creating a customized Debian Live image, I install >>> Debian onto a USB flash drive or onto a 2.5" SATA SSD connected via >>> a USB-SATA adapter cable: >> >> +1 >> >> It's pretty easy to make a simple Debian install on some old USB key >> you have lying around and it comes really handy. > > Relevant laptop is so old I don't know if it can boot from a physical > USB device. I was suspecting that simplest thing would be copying > suitable image to hard drive and let GRUB earn its keep ;} So, the Sony laptop has Debian 9.13 i386 installed on the HDD/SSD? Debian 9 LTS support ended on 01 Jul 2022: https://endoflife.date/debian If the Sony laptop can boot Debian 11 amd64 via CD, it should be able to boot an amd64 OS via USB. I suggest copying debian-12.5.0-amd64-netinst.iso to a USB flash drive and trying to boot it. If it works, you can decide if you want to use d-i, if you want to burn a live distribution to a USB device, and/or if you want to install Debian onto a USB device. David
Re: mounting external hard drive from rescue mode shell?
On 6/22/24 04:43, Richard Owlett wrote: Thank you for reminding me of live images just now. Perfect timing. I have an i386 machine with some atypical constraints. https://www.debian.org/CD/live/ states only amd64 images are currently available. Questions: 1. What is latest i386 live image available in some archive? 2. I have a working machine that will take a current full install of an i386 system. Can an average user create his own i386 live install image? Rather than creating a customized Debian Live image, I install Debian onto a USB flash drive or onto a 2.5" SATA SSD connected via a USB-SATA adapter cable: https://www.startech.com/en-us/hdd/usb3s2sat3cb Installing Debian onto a USB device allows me to choose the tasks I want at install time and then to add, remove, update, upgrade, etc., the packages I want later. I load the USB Debian instances with all of my favorite Unix/ Linux/ Debian trouble-shooting tools, plus my own scripts. My Debian 11 amd64 BIOS/MBR USB instance works on all of my amd64 BIOS/MBR computers and works on most of my EUFI/GPT computers when set to BIOS/MBR mode. My Debian 11 am64 UEFI/GPT USB instance was created on a Windows 10 era machine with UEFI/GPT and only works on similar machines with similar settings. David
Re: mounting external hard drive from rescue mode shell?
On 6/20/24 19:10, Max Nikulin wrote: On 20/06/2024 12:06, David Christensen wrote: You can use the fdisk(8) command to list the partitions on a drive. lsblk --fs perhaps with "-o +SIZE" may be more convenient to get overview of drives. The debian-11.9.0-amd64-netinst rescue shell does not include lsblk(8): ~ # lsblk /bin/sh: lsblk: not found David
Re: How to recover when monitor goes blank.
On 6/19/24 13:45, Ram Ramesh wrote: Hi, I have my monitor, keyboard and mouse shared through a KVM switch. One host is Linux Debian bookworm 12.5 and another is laptop running Windows 11. When I leave KVM on the laptop side for extended period I have issues switching back to Debian side. When I switch, the screen is blank and KB does not respond as if Debian is running headless. I had to remote login and reboot Debian side with KVM locked on this side to get back the monitor/KB. This happens regardless of whether I am in Xorg or VT. I do not know how to force Debian/Linux to check for monitor/KB again after extended period of disconnect when it has assumed it is running headless. Any solutions? Regards Ramesh My Debian machines have Xfce. I configure Applications Menu -> Settings -> Power Manager -> Display -> Display power management -> Off. David
Re: mounting external hard drive from rescue mode shell?
On 6/19/24 12:23, Heriberto Avelino wrote: On Wed, Jun 19, 2024 at 9:04 AM Heriberto Avelino wrote: Is it possible to mount an external hard drive while running Debian in rescue mode? Furthermore, the ultimate question is how could I copy folders from the computer's hard drive to the external one while in rescue mode? > Thanks Eben and David! > I am now on a shell (BusyBox v.35.0 Debian 1:1.35.0-4+b3) > I don't see the mounting points to execute cp. > There is nothing under media nor root; under usr I can see only: bin > lib local sbin share. > Where are the internal h-drive and the external? > I would very much appreciate your further help. > Who could have envisioned a community like this back in the 80's? This > is great! > Heriberto We are glad to help. :-) On Debian GNU/Linux, SATA drives (internal and external) should have special files (device nodes) under /dev. You can use the ls(1) command and a glob pattern to find them: ~ # ls /dev/sd? /dev/sda You can use the fdisk(8) command to list the partitions on a drive. For example, here is the drive containing Debian in the computer I am using now: ~ # fdisk -l /dev/sda Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors Disk model: INTEL SSDSC2CW06 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x544032f5 Device BootStart End Sectors Size Id Type /dev/sda1 *2048 1953791 1951744 953M 83 Linux /dev/sda21953792 3907583 1953792 954M 83 Linux /dev/sda33907584 29298687 25391104 12.1G 83 Linux /dev/sda4 29298688 117229567 87930880 41.9G 83 Linux Partitions can contain one of several things. One possibility is a file system. I keep detailed records on all of my OS's and drives, so I know what is what. Figuring out the contents of an unknown drive using the Debian rescue shell is possible, but requires more knowledge and effort. Using a live Linux distribution instead of the Debian rescue shell can make such tasks easier. I install Debian onto a SATA SSD via a USB-SATA adapter cable for this purpose. Once you have identified the device node of a partition that contains the file system that you want to read and/or write, you must mount the file system. The first step is to create a mount point with mkdir(1): ~ # mkdir /scratch The second step is to mount the file system with mount(8): ~ # mount /dev/sda4 /scratch Repeat the above process to mount any additional file systems that you want to read and/or write. One the file system(s) are mounted, you can read and/or write files and/or directories. For example, you can use cp(1) to copy files from an internal drive to an external drive. "Learning the Unix Operating System" is a good book for learning how to use Unix/ Linux from the command line: https://www.oreilly.com/library/view/learning-the-unix/0596002610/ "UNIX and Linux System Administration Handbook" is a good book for learning how to administer Unix/ Linux from the command line: https://www.oreilly.com/library/view/unix-and-linux/9780134278308/ "Design of the UNIX Operating System" is a good book for understanding how Unix worked around the time Linux Torvalds wrote Linux: https://www.pearson.com/en-us/subject-catalog/p/design-of-the-unix-operating-system/P20009243/9780132017992 https://en.wikipedia.org/wiki/Linus_Torvalds David
Re: mounting external hard drive from rescue mode shell?
On 6/19/24 08:04, Heriberto Avelino wrote: Dear all: Is it possible to mount an external hard drive while running Debian in rescue mode? Yes. Furthermore, the ultimate question is how could I copy folders from the computer's hard drive to the external one while in rescue mode? Many thanks!! Heriberto This is how I start a rescue shell using debian-11.9.0-amd64-netboot media: Debian GNU/Linux installer menu (BIOS mode) -> Advanced options Advanced options -> Rescue mode Language -> C Continent or region -> North America Country, territory or area -> United States Keymap -> American English Hostname -> debianrescue Domain name -> tracy.holgerdanske.com Select your time zone -> Pacific Passphrase for /dev/sda3 -> -> Continue Device to use as a root file system -> Do not use a root file system Rescue operations -> Execute a shell in the installer environment Executing a shell -> Continue Once busybox(1) is running, you can issue mount(8) and cp(1) commands similar to using a shell. David
Re: can't connect to server from outside LAN
On 6/12/24 15:54, Greg Marks wrote: I'm running a Debian server from my home with a static IP address, with ssh configured to use key-based authentication rather than password-based. As of a couple weeks ago, I have been unable to ssh to my server from external locations. When I ssh from a laptop connected to the wireless network on the same router as my home server, I do successfully connect to the server. But when I ssh from an external location, I get this error: OpenSSH_8.4p1 Debian-5+deb11u3, OpenSSL 1.1.1w 11 Sep 2023 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug2: resolve_canonicalize: hostname xxx.xxx.xxx.xxx is address debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts' debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2' debug2: ssh_connect_direct debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22. debug1: Connection established. debug1: identity file /home/user/.ssh/id_rsa type -1 debug1: identity file /home/user/.ssh/id_rsa-cert type -1 debug1: identity file /home/user/.ssh/id_dsa type -1 debug1: identity file /home/user/.ssh/id_dsa-cert type -1 debug1: identity file /home/user/.ssh/id_ecdsa type -1 debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/user/.ssh/id_ed25519 type -1 debug1: identity file /home/user/.ssh/id_ed25519-cert type -1 debug1: identity file /home/user/.ssh/id_ed25519_sk type -1 debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/user/.ssh/id_xmss type -1 debug1: identity file /home/user/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3 kex_exchange_identification: read: Connection timed out banner exchange: Connection to xxx.xxx.xxx.xxx port 22: Connection timed out When I ping the server from external locations, I get 100% packet loss; whereas when I ping the server from my local wireless network, there is 0% packet loss. (I do have nftables set to drop connections from numerous IP addresses that have attempted hacks in the past; however, the problem persists after flushing nftables, and at any rate, using check-host.net and www.site24x7.com to ping my server from various worldwide locations also results in 100% packet loss.) Port 22 is open. The package ufw is not installed on my server. The apache2 Web server running on my home server is correctly hosting my Web pages: from external locations, my Web page gmarks.org will open in a Web browser (even though "ping -c 10 gmarks.org" shows 100% packet loss). Running "traceroute xxx.xxx.xxx.xxx" from external locations reported four successful steps, not reaching my server IP, followed by a series of "* * *" lines. Running "sudo service sshd status" on my server shows ssh.service is active and running. Running "ip address show" on my server shows nothing unusual. I've restarted my router, and I've restarted my server; neither helped. The problem began a couple weeks ago; previously (and for many years) I had been able to ssh to my server without issue. The first time it failed, I was using free wireless at an airport; I was able to ssh to my server from the hotel that morning, and maybe, the first time I tried, from the airport, but then subsequent ssh attempts from the airport failed to connect. I mention this only because nothing had changed in my server's configuration when this problem began. This is a real problem for me, as a lot of my work involves sending files via scp between work and home. Any suggestions about how to troubleshoot and hopefully fix the problem will be greatly appreciated. Best regards, Greg Marks Ping: 2024-06-12 16:19:25 dpchrist@laalaa ~ $ ping -c 1 -v gmarks.org PING gmarks.org (76.235.90.201) 56(84) bytes of data. --- gmarks.org ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms HTTPS works: https://gmarks.org/ Greg Marks Office: 313 Ritter Hall Phone: (314)977-7206 E-mail address: local-part consists of surname, domain name identical to that of the present Web page PGP encryption public key ID: 0x53F269E8 Postal Address: Department of Mathematics and Statistics St. Louis University 221 N. Grand Blvd. St. Louis, MO 63103-2006 U.S.A. Resources for students Resources for mathematicians Research papers It appears that someone who controls one or more of the routers between the Internet and your router has blocked ICMP echo requests, ICMP echo replies, and TCP port 22 (SSH). I suggest that you
Re: about 10th new install of bullseye
On 6/8/24 12:13, gene heskett wrote: On 6/8/24 03:22, David Christensen wrote: If you installed VirtualBox on your Debian primary workstation, you could create one Debian VM for each of your engineering/ manufacturing apps. This would give each app a clean Debian VM for installation, prevent apps from fighting each other, and prevent apps from modifying your base Debian installation. It is quite rare that a snap, appimage, or venv needs anything from the system. Memory or storage is generally done at whatever venv is started as the user. That venv equ is generally what they all claim to do. I see your reticence to make use of them as a restriction. My suggestion is a variation of the "divide and conquer" troubleshooting strategy. I am not familiar with snap, appimage, or venv. Regardless of the software distribution mechanism, I expect that each app is developed and tested against a list of supported OS's and releases using VM's. If you provide each app with its own VM containing a supported OS and release, the app should install and work correctly. And, your base Debian installation should remain stable. David
Re: about 10th new install of bullseye
On 6/7/24 22:41, gene heskett wrote: I OTOH, have found AppImages a good way to get uptodate, and keep uptodate, packages like OpenSCAD, FreeCAD and the miriad 3d slicers, most of which do a new AppImage in the first week of the month. So the OpenSCAD I'm running is nearly 4 years newer than the repo version, and probably 20x faster. I have found that installing software on Debian by any means other than official Debian packages is a recipe for disaster. I sometimes write Perl code that runs as root. I use VirtualBox and do my development and testing on virtual machines. Oracle provides Debian packages and integrates with sources.list(5) and apt-get(8). See "Debian-based Linux distributions": https://www.virtualbox.org/wiki/Linux_Downloads If you installed VirtualBox on your Debian primary workstation, you could create one Debian VM for each of your engineering/ manufacturing apps. This would give each app a clean Debian VM for installation, prevent apps from fighting each other, and prevent apps from modifying your base Debian installation. David
Re: about 10th new install of bullseye
On 6/6/24 22:14, gene heskett wrote: In experimenting I've found a name clash, there are appprently two orca's. one is a speech synth, one is a slicer for 3d printers I don't use. Typing orca in a shell locks the shell wo any output, for several minutes but comes back to a prompt with a ctl-c, so I've NDC which was being executed. Whatevver, the installation is quite voluminous: gene@coyote:~/AppImages$ locate orca |wc -l 1560 So I took orca out, which took gnome out. But now gnomes dependencies will put orca back in. So now I can't run autoremove. So one more time this broken damned bookworm install has bit me in a rear. No Cheers, Gene Heskett, CET. Here are my installation notes from when I migrated my daily driver from Debian 9 to Debian 11. It has orca, and orca has never bothered me: January 9, 2022 1. Wipe Intel SSD 520 Series 60 GB drive in Intel DQ67SW. Insert debian-11.2.0-amd64-netinst USB flash drive into USB 3.0 port adjacent Gigabit port. Boot: Debian GNU/Linux installer menu (BIOS mode) install LanguageC Continent or region North America Country, territory or area United States Keymap to use American English Hostnamelaalaa Domain name tracy.holgerdanske.com Root password Re-enter password Full name for new user debian Username for your account debian Choose a password Re-enter password Select your time zone Pacific Partitioning method Manual Select a partition... SCSI1 (0,0,0) (sda) - 60.0 GB ATA INTEL SSDSC2CW06 Create partition table Yes Select a partition... pri/log 60.0 GB FREE SPACE Create a new partition New partition size1 GB Type Primary Location Beginning Partition settings Use as Ext4 journaling file system Mount point /boot Mount options defaults Label laalaa_boot Reserved blocks 5% Typical usage standard Bootable flag on Done setting up the partition Select a partition... pri/log 59.0 GB FREE SPACE Create a new partition New partition size1 GB Type Primary Location Beginning Partition settings Use as physical volume for encryption Encryption method Device-mapper (dm-crypt) Encryption aes Key size256 IV algorithmxts-plain64 Encryption key Random key Erase data no Bootable flag off Done setting up the partition Select a partition... pri/log 58.0 GB FREE SPACE Create a new partition New partition size13 GB Type Primary Location Beginning Partition settings Use as physical volume for encryption Encryption method Device-mapper (dm-crypt) Encryption aes Key size256 IV algorithmxts-plain64 Encryption key Passphrase Erase data no Bootable flag off Done setting up the partition Configure encrypted volumes Write the changes to disk Yes Encryption configurationCreate encrypted volumes Devices to encrypt [*] /dev/sda2 (1000MB; crypto) [*] /dev/sda3 (13000MB; crypt) Continue Encryption configurationFinish Encryption passphrase Re-enter passphrase Select a partition... #1 13.0 GB f ext4 Partition settings Use asExt4 journaling file system Mount point / Mount options defaults Label laalaa_root Reserved blocks 5% Typical usage standard Done setting up the partition Finish partitioning and
Re: about 10th new install of bullseye BUT its not Bullseye, its bookworm!
On 6/5/24 08:21, gene heskett wrote:> But in asking how to get rid of [orca], the subject is always changed and I always get re-install instructions. Because that is the most practical and correct answer for your situation; especially given the disk access issues. AIUI assistive technologies have been standard on FOSS graphical workstations for years. It should be possible to turn assistance off, but it might not be possible to eliminate the machine code throughout the entire software stack. I install Debian with the Xfce desktop, SSH server, and standard system utilities onto minimal hardware. It takes a known amount of time and usually works. I have successfully ignored assistive technologies for years (decades?). Yes, the assistive technologies are wasting storage, memory, and cycles, and they create a larger threat surface, but those risks and costs are cheaper than me trying to understand and control all of the details. Succeeding with software requires that you devise strategies to work within the limitations of the software. Alternatively with FOSS, you can change the software. David
Re: about 10th new install of bullseye
https://www.mail-archive.com/debian-user%40lists.debian.org/msg779582.html Gene Heskett Fri, 18 Feb 2022 09:14:03 -0800 > On 2/19/22 06:31, Andrew M.A. Cater wrote: On 6/4/24 03:26, gene heskett wrote: How much longer till trixie is officially out?? What you are proposing sounds like several days work, and i have other irons in the fire. This release has been such a disaster for me because the install insists on installing and configuring orca and brltty w/o asking. I've done 40 some installs now, trying to stop it from wasting about a second while its yelling every keystroke at me because it thinks I'm blind. I finally have orca disabled and the computer is useful. The delays are a pain in the a$$ but i can do work now. It is not useful when orca is using 90% of a 6 core I5 yelling at me loud enough to announce and pronounce every keystroke or mouse motion/click loud enough to wake the neighbors. The first 23 installs never asked me if I wanted that crap. And if you nuked the orca executable it would not reboot but hung forever waiting for orca to start. I have it usable, the installer AFAIAC is broken and I don't want to have to go through all that again. Until the installer ASKS me if I want it because it thinks I am blind, I have only one nerve left and and the suggestion that I do yet another install, is standing on it. Trying to remove it now, it insists on removing gnome and every dependency. I just checked again with synaptic, removing either orca or brltty still wants to destroy the system, Yet all I get when I fuss about the broken installer is "won't fix, not broken'. I suggest: 1. Back up the system configuration and data. 2. Disconnect everything internal to the chassis except for the motherboard, power supply, front panel, fans, processor, memory, and one disk drive for the OS connected to the first IDE, SATA, or NVMe port. 3. Disconnect everything external to the chassis except AC power, wired keyboard, wired mouse, wired monitor, and Ethernet. 4. Boot into Setup and reset settings to factory defaults. Choose between BIOS/Legacy and UEFI, if there is a choice. Set the disk controller mode to AHCI. Set the clock to UTC. 5. Boot the disk manufacturer toolkit and wipe the OS drive -- secure erase for SSD's and zero-fill for HDD's. I seem to recall that you have a 1 TB WD Black. WD does not appear to offer a bootable disk drive toolkit (?): https://support-en.wd.com/app/products/downloads/softwaredownloads If you can find a FOSS toolkit to do a secure erase, that would be best. Alternatively, find the non-zero blocks and zero them (a good job for a script). 6. Boot debian-12.5.0-amd64-netinst.iso and install Debian onto the OS disk. I partition manually with 1 GB EFI system partition, 1 GB boot partition, 1 GB random encrypted swap partition, and a small passphrase encrypted root partition (twice your current root partition usage?). Save the remaining free space for CAD, CNC, 3-D, etc., working/ scratch files and over-provisioning, to be configured after installation. If your desktop environment of choice is not offered by d-i, do not install a desktop environment. 7. At the end of installation, reboot. Remove d-i media during POST. Verify the system boots from the OS disk. Login and check vitals, but do not change anything. Power off. 8. Boot your FOSS toolkit of choice, or d-i rescue shell, and take a compressed image of the OS disk to a file on a USB HDD. I use a version control system (CVS over SSH) for software development, but also find it to be very useful for system administration. David
Re: advanced scripting problems - or wrong approach?
On 6/2/24 21:35, DdB wrote: Am 02.06.2024 um 02:41 schrieb DdB: Will share my findings, once i made more progress... Here is what i've got before utilizing it: datakanja@PBuster-NFox:/mnt/tmp$ cat test #!/bin/bash -e # testing usefulness of coprocess to control host and backup machine from a single script. # beware: do not use subprocesses or pipes, as that will confuse the pipes setup by coproc! # At this point, this interface may not be very flexible # but trying to follow best practices for using coproc in bash scripts # todo (deferred): how to handle stderr inside coproc? # todo (deferred): what, if coproc dies unexpectedly? # setting up the coprocess: stdout_to_ssh_stdin=5 # arbitrary choice outside the range of used file desciptors stdin_from_ssh_stdout=6 coproc SSH { bash; } # for testing purposes, i refrain from really involving ssh just yet and replace it with bash: # save filedescriptors by duplicating them: eval "exec ${stdin_from_ssh_stdout}<&${SSH[0]} ${stdout_to_ssh_stdin}>&${SSH[1]}" echo The PID of the coproc is: $SSH_PID # possibly useful for inspection unique_eof_delimirer="" line="" # collect the output available and print it locally (synchonous): function print-immediate-output () { while IFS= read -r -u "${stdin_from_ssh_stdout}" line do if [[ "${line:0-5:5}" == "$unique_eof_delimirer" ]] # currently, the length is fixed then line="${line%}" if [[ ! -z $line ]] then printf '%s\n' "$line" fi break fi printf '%s\n' "$line" done } # send a single command via ssh and print output locally function send-single-ssh-command () { printf '%s\n' "$@" >&"${stdout_to_ssh_stdin}" printf '%s\n' "echo '"$unique_eof_delimirer"'" >&"${stdout_to_ssh_stdin}" print-immediate-output } send-single-ssh-command "find . -maxdepth 1 -name [a-z]\*" # more or less a standard command, that succeeds send-single-ssh-command "ls nothin" # more or less a standard command, that fails # tearing down the coprocess: printf "%s\n" "exit" >&"${stdout_to_ssh_stdin}" # not interested in any more output (probably none) wait # Descriptors must be closed to prevent leaking. eval "exec ${stdin_from_ssh_stdout}<&- ${stdout_to_ssh_stdin}>-" echo "waited for the coproc to end gracefully, done" datakanja@PBuster-NFox:/mnt/tmp$ ./test The PID of the coproc is: 28154 ./test ./out ls: Zugriff auf 'nothin' nicht möglich: Datei oder Verzeichnis nicht gefunden waited for the coproc to end gracefully, done datakanja@PBuster-NFox:/mnt/tmp$ "test" is both a program and a shell builtin. I suggest that you pick another, non-keyword, name for your script. I suggest adding the Bash option "-u' (nounset). Your file descriptor duplication, redirection, etc., seems overly complex. Would not it be easier to use the coproc handles directly? 2024-06-03 08:49:41 dpchrist@laalaa ~/sandbox/bash $ nl coproc-demo 1 #!/usr/bin/env bash 2 # $Id: coproc-demo,v 1.3 2024/06/03 15:49:36 dpchrist Exp $ 3 set -e 4 set -u 5 coproc COPROC { bash ; } 6 echo 'echo "hello, world!"' >&"${COPROC[1]}" 7 read -r reply <&"${COPROC[0]}" 8 echo $reply 9 echo "exit" >&"${COPROC[1]}" 10 wait $COPROC_PID 2024-06-03 08:49:44 dpchrist@laalaa ~/sandbox/bash $ bash -x coproc-demo + set -e + set -u + echo 'echo "hello, world!"' + bash + read -r reply + echo hello, 'world!' hello, world! + echo exit + wait 4229 David
Re: advanced scripting problems - or wrong approach?
On 6/1/24 00:20, DdB wrote: Hello, for years have i been using a self-made backup script, that did mount a drive via USB, performed all kinds of plausibility checks, before actually backing up incrementally. Finally verifying success and logging the activities while kicking the ISB drive out. Since a few months, i do have a real backup server instead, connecting to it via ssh i was able to have 2 terminals open and back up manually. Last time, i introduced a mistake by accident and since, i am trying to automate the whole thing once again, but that is difficult, as the load on the net is huge, mbuffer is useful in that regard. So i was intending to have just one script for all the operations using coproc to coordinate the 2 servers. But weird things are going on, i cant reliably communicate between host and backup server, at least not automatically. Searching the web, i found: https://github.com/reconquest/coproc.bash/blob/master/REFERENCE.md But i was unable to get this to work, which seems to indicate, that i am misunderstanding something. The only success i had was to "talk" to a chess engine in a coprocess, which did go well. But neither bash nor ssh are cooperating, i may have timing issues with the pipes or some other side effects. How can i describe? For example if i start this: #!/bin/bash -e coproc { bash; } exec 5<&${COPROC[0]} 6>&${COPROC[1]} fd=5 echo "ls" >&6 while IFS= read -ru $fd line do printf '%s\n' "$line" done printf "%s\n" "sleep 3;exit" >&6 while IFS= read -ru $fd line do printf '%s\n' "$line" done exec 5<&- 6>&- wait echo waited, done i get the output from ls, but then the thing is hanging indefinitely, apparently not reaching the exit line. :( Anyone who can share his experience to advance my experimenting? DdB https://en.wikipedia.org/wiki/XY_problem Please define the root problem you are trying to solve. David
Re: "Repeaters", etc.
On 5/30/24 03:14, Roger Price wrote: On Wed, 29 May 2024, David Christensen wrote: On 5/29/24 03:36, Roger Price wrote: On Tue, 28 May 2024, David Christensen wrote: On 5/28/24 00:28, Roger Price wrote: I wired my place Cat5. A lot of work, and I regretted it. I live in the hills behind Nice, an area with a lot of lightning. The overhead line to my place took a hit and thanks to the Cat5 conductivity I lost equipment. How do you know that the damage your equipment suffered was due to the Cat 5e wiring and not due to the electrical power conductors? Electrical power to my computers comes through 30mA differential circuit breakers to Eaton Ellipse 1600 UPS units. I had no such protection for the telephone signal, and I saw flashes at the telephone junction box. So I summise that the Cat5 wiring did the damage. Those UPS's should be able to protect telephone and Ethernet, in addition to electrical power. Have you applied the UPS's to the former two? The UPS's stand next to the workstations and well away from the place where the telephone line arrives, so I didn't use the UPS's to protect the telephone line. My fault. Later I added a surge protector to the copper telephone line. I am now in the process of migrating from copper to fiber so I will need an extra UPS next the fiber terminator. Roger === https://www.eaton.com/sg/en-us/catalog/backup-power-ups-surge-it-power-distribution/eaton-ellipse-pro-ups.html PS: I once had a lightning strike direct to the house. Frightening. Although every differential circuit breaker in the house tripped, the circuit board in the UPS melted. But even when melting, it protected the Dell T7500. No damage to the T7500, no data lost. I took a photo of the melt, sent it to Eaton, and they replaced the UPS. Have you consider applying lightning protection to your house? 1. Lightning rods, down lines, ground rods, perimeter ground loop, etc.. 2. Lightning arresters at the electrical, telephone, CATV, etc. service entrance points. David
Re: "Repeaters", etc.
On 5/29/24 03:36, Roger Price wrote: On Tue, 28 May 2024, David Christensen wrote: On 5/28/24 00:28, Roger Price wrote: I wired my place Cat5. A lot of work, and I regretted it. I live in the hills behind Nice, an area with a lot of lightning. The overhead line to my place took a hit and thanks to the Cat5 conductivity I lost equipment. How do you know that the damage your equipment suffered was due to the Cat 5e wiring and not due to the electrical power conductors? Electrical power to my computers comes through 30mA differential circuit breakers to Eaton Ellipse 1600 UPS units. I had no such protection for the telephone signal, and I saw flashes at the telephone junction box. So I summise that the Cat5 wiring did the damage. Roger https://www.eaton.com/sg/en-us/catalog/backup-power-ups-surge-it-power-distribution/eaton-ellipse-pro-ups.html https://standards.globalspec.com/std/104626/iec-61643-1 Those UPS's should be able to protect telephone and Ethernet, in addition to electrical power. Have you applied the UPS's to the former two? David
Re: "Repeaters", etc.
On 5/28/24 17:10, John Hasler wrote: David writes: AIUI in the USA for residential 120/240V single-phase three-wire service drops, electrical utilities either run all three phases along the distribution line or they run two phases. Running one phase and a neutral instead of two phases would reduce the power by the square root of 3 Here in rural Wisconsin the 7200V distribution line leaves the substation as three phases and a grounded neutral. This eventually branches out into three single phase lines consisting of a phase and a grounded neutral. The pole pigs are connected phase to neutral. Interesting. STFW I found an article and a web site that clarifies the above arrangements and more: https://electrical-engineering-portal.com/primary-distribution-circuits David
Re: "Repeaters", etc.
On 5/28/24 12:47, gene heskett wrote: On 5/28/24 15:29, Roy J. Tellason, Sr. wrote: On Tuesday 28 May 2024 01:49:52 pm Paul M Foster wrote: I've never see a 3 phase in a house. Common in commercial/industrial, though. Residential installations (talking in the US here) typically involve *one* transformer tapping a single phase out of the three that are up there on the pole. The secondary is center-tapped, and it's that point which is grounded at the service entrance. Running 3-phase power requires *three* transformers up on the pole, much more in the way of expense if you want that for some reason, and I don't know of anybody that does that. Even those who are into having some nontrivial machinery around seem these days to use a VFD to give them multiple phases at the machine, rather than going through the expense of having it run in from the pole... And here you have it from another CET. Cheers, Gene Heskett, CET. AIUI in the USA for residential 120/240V single-phase three-wire service drops, electrical utilities either run all three phases along the distribution line or they run two phases. Running one phase and a neutral instead of two phases would reduce the power by the square root of 3. Running one phase and using the Earth as the return conductor is very dangerous and not modern practice. David
Re: "Repeaters", etc.
On 5/28/24 00:28, Roger Price wrote: I wired my place Cat5. A lot of work, and I regretted it. I live in the hills behind Nice, an area with a lot of lightning. The overhead line to my place took a hit and thanks to the Cat5 conductivity I lost equipment. If your electrical utility uses pole-mounted distribution lines, transformers, service drops, etc., and lightning strikes the high-voltage conductors, there will be a surge on the customer service conductors that places persons and property at risk. If lightning jumps to the customer service conductors, then the risk to persons and property can be extreme. How do you know that the damage your equipment suffered was due to the Cat 5e wiring and not due to the electrical power conductors? David
Re: "Repeaters", etc.
On 5/27/24 19:05, Paul M Foster wrote: I did some more research, and it looks like I must have misstated the problem. Let's assume I can't get in the attic and wire the place. Let's assume that I've got a wireless router/modem in, say, the garage. Let's say I have three rooms with devices I want to connect (one way or another) to my router/modem. It appears there are two solutions. One is wifi extenders, and one is a mesh network. In both cases, the device sits in the room and communicates via wifi to the modem/router. The devices in the room connect to the device via ethernet cable. How does that sound? Any dissenting opinions? Any brand recommendations? On 5/27/24 20:14, Paul M Foster wrote: Coincidentally, I used to be an electrician too, but we almost never ran low voltage except for doorbells. The house in question appears to have a generous attic, but they've blown in two feet of insulation I'd rather not disturb. And that much insulation makes the headers of walls very hard to find. Also, I'm not in my 20s anymore, and crawling around in attics is difficult. In the house I'm living in now, I did go into the attic years ago with cat 5e and wired up the living room. FWIW, in the house we're buying, I need internet (wired) in the living room, bedroom 2 and bedroom 4. Also, it's concrete block construction (outer walls). On 5/27/24 21:50, Paul M Foster wrote: Well, if I'm gonna run cat 5, I might as well just put a jack in each room.> No POE needed. The only reason for wifi at all in this case is so I don't *have* to run cat 5. From what I've read, TP-Link gets good reviews. FYI my previous coax cable suggestion will likely involve MoCA: https://en.wikipedia.org/wiki/Multimedia_over_Coax_Alliance Another idea is power-line communication. I seem to recall reading that an RF choke should be installed on the incoming electrical service to prevent interference to/from the neighbors, but STFW I do not see any mention of that today: https://en.wikipedia.org/wiki/Powerline_Ethernet David
Re: "Repeaters", etc.
On 5/27/24 14:09, Paul M Foster wrote: Folks: At some point this year, I'm moving into a new house, and it is not wired for internet (WHY aren't new houses wired with Cat5/6/7?). The local internet provider will likely provide a wireless router, as they all do. My idea is to put a device which receives wireless signal from the router/modem, and has an RJ45 jack in it in each room. So each room would have one of these, and the devices in it would be hooked to that device via cat 5e. I hope that's clear. I'd like to shop for such a device, but I don't know what it's called. Can anyone provide advice, and possibly preferred brand names? I'd appreciate it. Paul Is the house wired for cable television (RG-6 coaxial cable)? If so, and you choose the right Internet provider, you might be able to get a "main box" with 1+ type F connectors, 4 @ RJ-45 Gigabit ports, Wi-Fi access point, etc., and "satellite boxes" with 1+ type F connector, 1+ RJ-45 Gigabit ports, Wi-Fi access point, etc.. Ask you cable television/ Internet provider. An alternative to running Ethernet cables inside walls is to run the cables on the surface -- e.g. staple to wall along floor molding, drill and pull through walls as required, paint to match, etc.. Cat 5e is smaller diameter and easier to work with than Cat 6a. I surface wired my house with Cat 5e ~20 years ago and have 1000BASE-T (Gigabit Ethernet) switches and NIC's. All of the cable runs are under ~20 meters, so I should be able to upgrade to 2.5GBASE-T. David
Re: HDD long-term data storage with ensured integrity
On 5/3/24 04:26, Marc SCHAEFER wrote: On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote: For off-site long-term offline archiving, no, I am not using RAID. Now, as I had to think a bit about ONLINE integrity, I found this comparison: https://github.com/t13a/dm-integrity-benchmarks Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid I tend to have a biais favoring UNIX layered solutions against "all-into-one" solutions, and it seems that performance-wise, it's also quite good. I wrote this script to convince myself of auto-correction of the ext4+dm-integrity+dm-raid layered approach. Thank you for devising a benchmark and posting some data. :-) FreeBSD also offers a layered solution. From the top down: * UFS2 file system, which supports snapshots (requires partitions with soft updates enabled). * gpart(8) for partitions (volumes). * graid(8) for redundancy and self-healing. * geli(8) providers with continuous integrity checking. AFAICT the FreeBSD stack is mature and production quality, which I find very appealing. But the feature set is not as sophisticated as ZFS, which leaves me wanting. Notably, I have not found a way to replicate UFS snapshots directly -- the best I can dream up is synchronizing a snapshot to a backup UFS2 filesystem and then taking a snapshot with the same name. I am coming to the conclusion that the long-term survivability of data requires several components -- good live file system, good backups, good archives, continuous internal integrity checking with self-healing, periodic external integrity checking (e.g. mtree(1)) with some form of recovery (e.g. manual), etc.. If I get the other pieces right, I could go with OpenZFS for the live and backup systems, and worry less about data corruption bugs. David
Re: realpath quoting
On 5/3/24 04:34, jeremy ardley wrote: On 3/5/24 19:06, Greg Wooledge wrote: I would suggest that if you need to use a debugger to track down a bug in your program, you should use filenames that don't require quoting when you set up your tests. 1970's style static test cases are not relevant here. In the real world... I download files generated by another system that are constantly changing content and with names I don't control. My workflow is to download a new file from a remote source and then run my processor over it. As a necessary consequence I need the fully quoted or escaped file name of the new file to feed to the processor/debugger. I can obviously add an extra step to the process to convert the new file name to something acceptable before processing. However, my question was how to avoid that extra step by getting fully quoted filenames to process. So, you are copying and pasting file names via some clipboard? emacs(1) might have a way to put a filter into that process, but I am unaware of a similar feature using Xfce and Terminal (my platform). I have tried renaming files in similar situations, but you will want to rename them everywhere if you use rsync(1). What if you downloaded files to a directory with a well-formed name and added a feature to your script to process files that appear in that directory? David
Re: realpath quoting
On 5/3/24 04:09, Greg Wooledge wrote: On Thu, May 02, 2024 at 10:18:03PM -0700, David Christensen wrote: I am unable to find $'string' in the dash(1) man page (?). As I typically write "#!/bin/sh" shell scripts, writing such to deal with file names containing non-printing characters is going to baffle me. Currently, $' quoting is a bash extension. It's supposed to appear in some future edition of POSIX, at which point shells like dash will be required to adopt it (whenever they get around to it). For now, though, you should consider it bash only. Thank you for the clarification. :-) David
Re: realpath quoting
On 5/2/24 19:56, Max Nikulin wrote: On 03/05/2024 09:19, Greg Wooledge wrote: I still insist that this is a workaround that should *not* be used to try to cancel out quoting bugs in one's shell scripts. There are still specific cases when quoting is necessary, e.g. ssh remote command +1 (however you have to be sure concerning shell on the remote host). +1 In BASH printf has %q format. GNU coreutils supports it as well, but dash does not, so be careful. My practice is to start with '#!/bin/sh' and migrate to '#!/usr/bin/env perl' as complexity increases. Likely Jeremy's case does not really require this kind of quoting. We need to see the full range of file names the OP is trying to deal with. While "ls -l" output is for humans, realpath is often used in scripts. Certainly it should nor return quoted output by default. I am in doubts if a dedicated option should be added to realpath. Thank you for helping me realize that my solution fails to print the resolved absolute file name. Here is the updated solution: 2024-05-02 21:57:56 dpchrist@laalaa ~ $ touch 'name with spaces' 2024-05-02 22:23:01 dpchrist@laalaa ~ $ touch 'name with > newline' 2024-05-02 22:28:36 dpchrist@laalaa ~ $ perl -MString::ShellQuote '-MFile::Spec::Functions qw(rel2abs)' -e 'print shell_quote(map { rel2abs $_ } @ARGV), "\n"' name* '/home/dpchrist/name with newline' '/home/dpchrist/name with spaces' David
Re: realpath quoting
On 5/2/24 19:19, Greg Wooledge wrote: On Thu, May 02, 2024 at 07:11:46PM -0700, David Christensen wrote: Perhaps Perl and the module String::ShellQuote ? 2024-05-02 18:50:28 dpchrist@laalaa ~ $ touch "name with spaces" 2024-05-02 18:50:45 dpchrist@laalaa ~ $ touch "name with\nnewline" You didn't create a name with a newline in it here. You created a name with a backslash in it. If you wanted a newline, you would have to use the $'...' quoting form (in bash). touch $'name with\nnewline' Thank you for the clarification. RTFM bash(1): QUOTING ... Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. ... The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or . ... Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. I found another way to obtain a file name containing a newline -- by pressing when typing a double-quoted string literal: 2024-05-02 21:52:23 dpchrist@laalaa ~ $ touch "foo > bar" 2024-05-02 21:52:29 dpchrist@laalaa ~ $ ls -l foo* -rw-r--r-- 1 dpchrist dpchrist 0 May 2 21:52 'foo'$'\n''bar' 2024-05-02 21:52:36 dpchrist@laalaa ~ $ perl -MString::ShellQuote -e 'print shell_quote(@ARGV), "\n"' foo* 'foo bar' It also seems to work for single-quoted string literals: 2024-05-02 21:57:08 dpchrist@laalaa ~ $ touch 'foo > bar' 2024-05-02 21:57:14 dpchrist@laalaa ~ $ ls -l foo* -rw-r--r-- 1 dpchrist dpchrist 0 May 2 21:57 'foo'$'\n''bar' 2024-05-02 21:57:18 dpchrist@laalaa ~ $ perl -MString::ShellQuote -e 'print shell_quote(@ARGV), "\n"' foo* 'foo bar' I am unable to find $'string' in the dash(1) man page (?). As I typically write "#!/bin/sh" shell scripts, writing such to deal with file names containing non-printing characters is going to baffle me. I still insist that this is a workaround that should *not* be used to try to cancel out quoting bugs in one's shell scripts. Just write the shell scripts correctly in the first place. I would if I could. While I am also unable to write Perl scripts correctly in the first place, the quoting rules are easier. David
Re: realpath quoting
On 5/2/24 15:59, jeremy ardley wrote: I have a need to get the full path of a file that has spaces in its name to use as a program argument e.g. jeremy@client:~$ ls -l name\ with\ spaces -rw-r--r-- 1 jeremy jeremy 0 May 3 06:51 'name with spaces' jeremy@client:~$ realpath name\ with\ spaces /home/jeremy/name with spaces The spaces without quotes cause problems with subsequent processing. Can realpath or other utility return a quoted pathname? Perhaps Perl and the module String::ShellQuote ? 2024-05-02 18:50:28 dpchrist@laalaa ~ $ touch "name with spaces" 2024-05-02 18:50:45 dpchrist@laalaa ~ $ touch "name with\nnewline" 2024-05-02 19:06:01 dpchrist@laalaa ~ $ perl -MString::ShellQuote -e 'print shell_quote(@ARGV), "\n"' name* 'name with spaces' 'name with\nnewline' David
Re: Debian@IBMx3550
On 4/23/24 14:35, Greg wrote: Hi there, I got refurb IBM x3550 M3 7944 server and I'm a bit lost. Is there any Linux/Debian software (some gui would be nice) to monitor fan speed, temperatures, voltages, disks.. ? Thanks in advance for any help Greg If you installed the Xfce desktop, add the "Sensor" plug-in/ applet to the Panel. Panel can display various temperatures, fan speeds, etc.. Note that you may to set the SUID bit on /usr/sbin/hddtemp for disk drive readings to be available: # chmod u+s /usr/sbin/hddtemp David
Re: Subject: Glitchy sound in Steam games after hard drive upgrade
On 4/23/24 09:02, debian-u...@howorth.org.uk wrote: Charlie Gibbs wrote: On 2024-04-22 16:50, Jeffrey Walton wrote: What are the old and new hard drive model numbers and specs? The old drive is a Western Digital WD5000YS (500GB SATA). The new drive is a Western Digital Red, WF40EFPX (4TB SATA). According to my searches, there's no such disk as a WF40EFPX. Are you sure that's what it is? If by any chance it is a WD40EFRX then that is certainly slower than your old drive, so may cause some problems as suggested. I doubt the new drive is slower than the old drive: - https://www.harddrivebenchmark.net/hdd.php?hdd=WDC%20WD5000YS WDC WD5000YS 425 - https://www.harddrivebenchmark.net/hdd.php?hdd=WDC%20WD40EFRX WDC WD40EFRX1,943 David
Re: Subject: Glitchy sound in Steam games after hard drive upgrade
On 4/22/24 21:26, Charlie Gibbs wrote: On 2024-04-22 16:50, Jeffrey Walton wrote: What are the old and new hard drive model numbers and specs? The old drive is a Western Digital WD5000YS (500GB SATA). https://www.newegg.com/western-digital-re2-wd5000ys-500gb/p/N82E16822136032?Item=N82E16822136032 The new drive is a Western Digital Red, WF40EFPX (4TB SATA). https://www.westerndigital.com/products/internal-drives/wd-red-plus-sata-3-5-hdd?sku=WD40EFPX Both drives are spinning rust. I'm upgrading for the increased capacity, i.e. to store more MP3s and videos. Many thanks to all who have replied. When my schedule permits me to continue experimenting, I'm going to try copying /etc from the old drive to the new one. I've already learned how _not_ to do this: Boot from the new drive $ su root # cd / # mv etc etc.ori # rsync -av /mnt/backup/etc . The second line makes the system fall over and makes logins impossible. It took a boot from the rescue CD to undo the damage, which fortunately was easy since the deadly step at least succeeded in backing up /etc. Next time I'll do it while booted from the old drive. Copying an entire /etc directory from one machine to another requires a highly controlled environment and lot of engineering. I have always migrated /etc settings from one OS instance to another OS instance by hand, one service/ configuration file at a time. Can you leave the 500 GB HDD operational and use the 4 TB HDD for data? David
Re: Strange New Installation Behavior
On 4/22/24 06:00, Stephen P. Molnar wrote: I am running Bookworm and cleaned up a couple of files too many resulting in a messed up Xfce Desktop. I decided that this would be a good time to reinstall the Bullseye. I made a backup of my /home/comp directory using Deja-dup. I downloaded and ran the 512 check sum on a copy of Debian-12.5.0-amd64-DVD-1.iso and ran the Graphical Install mode on the 1.0 TD SSD on my Computer. The installation went smoothly without any warning or error messages. I logged in as root to set up the Desktop and, much to my surprise, found that my previous Desktop configuration was still there!!??? This was also the case when I logged in user!!!??? I have been using computers in my work since the 1960, the era of the Hollerith Card and tape drives and Linux since early days of Slackware and the Red Hat Mother's Day Edition. Now I am not a computer expert but a Research Chemist. I have installed Linux OS's many times and consider Linux my primary computational platform. I have never encountered the situation and have no ideas as to what is going on. I have been runnind Debian since Etch. I would appreciate some insight into what might be going on. Thanks in advance. Stephen P. Molnar, Ph.D. https://insilicochemistry.net (614)312-7528 (c) Skype: smolnar1 On 4/22/24 09:34, Stephen P. Molnar wrote: I did not want to revert to Bullseye, but to reinstall to Bookworm. I suggest that you buy a good 16 GB USB flash drive and install Debian 12 with Xfce onto it. Having a working live USB stick is very useful for low-level disk drive chores such as examining, backing up, testing, repairing, restoring, wiping, etc.. Use it to: 1. Ensure that you have a good backup of your 1 TB SSD. 2. Make additional backups or archives of all or part of your 1 TB SSD. Note the mantra: "Data does not exist unless it exists in three places". 3. Wipe the SSD so that the Debian installer will see a blank disk and respond accordingly when you later install Debian onto the SSD. Regarding copying a home directory from one OS installation to another OS installation, please see my comments on another thread: https://lists.debian.org/debian-user/2024/04/msg00336.html Once you have logged in to your new account on your fresh install, I suggest that you restore your /home/comp backup to a subdirectory and manually copy/ move/ edit/ merge files and directories from the restore subdirectory into your fresh home directory. Be very careful not to damage or delete anything needed by your fresh desktop or applications. David
Re: Subject: Glitchy sound in Steam games after hard drive upgrade
On 4/21/24 22:33, Charlie Gibbs wrote: I should probably be posting this to the Steam forums, but most of the denizens there are Windows people so I might be better off letting you Debian gurus have a go at it first. TL;DR: Copying an existing /home into a fresh Debian installation causes audio in Steam games to glitch - but all other sound is OK. Full description: I have a machine in the living room that stores MP3s and videos and serves them to other machines on our network as well as playing them locally on our TV's big screen. I also play a few Steam games (e.g. Portal) on it. It's a 2007-vintage machine, but it has 8GB of RAM and enough CPU power to do the job, and runs the latest version of Bookworm. Recently I decided to upgrade its storage capacity, and replaced its 500GB hard drive (which was pretty large at the time I bought it) with a 4TB drive. I did an install from scratch using a network install CD, then copied my /home partition (using rsync) from the old drive. Everything works great with one exception: when I fire up Portal the sound gets glitches about once a second. This only happens with Steam games; I can play MP3s and videos with mpv and the sound is perfect, as it is when watching YouTube videos. If I swap the old drive back in everything is fine. Obviously my Steam programs and configuration files are in my home directory, since the updated system comes up icons and all without re-installing Steam, and can find everything it needs to run the games. But perhaps there are a few files somewhere else (/usr?) containing information critical to audio for Steam. Any ideas? (Side question: is this an acceptable way to upgrade a hard drive?) Copying a home directory from one OS instance to another OS instance sounds risky, especially as I run various OS's. I have several instances of Debian 11, and would not consider them to be identical enough to try it. I only touch the content I create or have learned how to manage. I put my OS on a small SSD and the vast majority of my data on HDD RAID in a file server. As I am the only user on my Debian daily driver, I leave the /home directory on the root file system and keep as little as possible in it. I mount the file server shares under /mnt, and create symlinks in my home directory that point into the mounted file system. I use CVS for project working directories. To migrate to a new home directory, I check in the projects in the old home and check out the project in the new home. I use Firefox and its sync feature. To migrate to a new home, I start Firefox, log in, wait for my settings to sync, and then check all of the settings by hand. I use Thunderbird. To migrate to a new home, I create a tarball of my Thunderbird profile directory on the old machine, expand the tarball on the new machine, and configure Thunderbird to use that profile. I do not attempt to migrate any of the various home directory configuration directories; I let the installer and/or package manager create them, and let the desktop, apps, etc., manage them. David
Re: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/14/24 05:29, David Christensen wrote: debian-user: I have a Dell Latitude E6520: 2024-04-14 04:28:39 dpchrist@laalaa ~ $ cat /etc/debian_version ; uname -a 11.9 Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux 2024-04-14 04:34:40 dpchrist@laalaa ~ $ dpkg-query -l xfce4 network-manager network-manager-gnome Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=---= ii network-manager 1.30.6-1+deb11u1 amd64 network management framework (daemon and userspace tools) ii network-manager-gnome 1.20.0-3 amd64 network management framework (GNOME frontend) ii xfce4 4.16 all Meta-package for the Xfce Lightweight Desktop Environment I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). I compared the problem machine against another with a working Xfce panel Network Manager applet, and discovered that the Status Tray Plugin was missing. I may have deleted Status Tray Plugin while cleaning, but did not notice the change immediately (?). So, I added Status Tray Plugin and now Network Manager has returned. David
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/19/24 00:16, Florent Rougon wrote: Another thing: did you look into ~/.xsession-errors? (Sorry if this was already mentioned and I missed it.) Please see attached copy of ~/.xsession-errors, taken immediately after system restart and login. "nm-applet" does not appear in .xsession-errors, but there are plenty of other warnings and error messages. Perhaps the failure of nm-applet is a symptom of a more fundamental failure (?). More involved: if you can't find any trace of the applet doing something, maybe rebuilding the package after adding a few fprintf() calls would help. I think it is time for a bug report. David .xsession-errors-20240419-121605.gz Description: application/gzip
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/18/24 09:46, Gareth Evans wrote: On Thu 18/04/2024 at 11:05, David Christensen wrote: Move aside the ~/.config/xfce4 directory: ... Restart -- screen with wallpaper alone. ... Hi David, Starting from Mate DE only and some old (bookworm) XFCE config files, if I: $ sudo apt install task-xfce-desktop then log out and into XFCE, I get an XFCE desktop with wallpaper, desktop icons, panel, main menu, launchers, window buttons, notification area. Then I: $ mv ~/.config/xfce4 ~/.config/xfce4.old then logout and into XFCE again, I get wallpaper, desktop icons, panel, main menu, window buttons, notification area, but only non-functional launcher placeholder icons. $ diff ~/.config/xfce4 ~/.config/xfce4.old Common subdirectories: /home/user/.config/xfce4/desktop and /home/user/.config/xfce4.old/desktop Only in /home/user/.config/xfce4.old: panel } Only in /home/user/.config/xfce4.old: src } <-- these are subdirectories Only in /home/user/.config/xfce4.old: terminal} Common subdirectories: /home/user/.config/xfce4/xfconf and /home/user/.config/xfce4.old/xfconf Common subdirectories: /home/user/.config/xfce4/xfwm4 and /home/user/.config/xfce4.old/xfwm4 $ (though there probably wasn't much there to start with) I'm not sure what would cause this difference in behaviour (though wonder if this might suggest more amiss with your XFCE installation) and I will watch this thread with interest. Also I just "rediscovered" that reinstalling task-xfce-desktop doesn't reinstall those packages which it brings in in the first place, though I'm sure you knew that :) Sorry I couldn't be of more help. Thanks for the suggestions. We have more information now. :-) David
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/18/24 07:28, Max Nikulin wrote: On 18/04/2024 17:05, David Christensen wrote: $ mv .config/xfce4-20240418-180045/ .config/xfce4 Restart -- back to Xfce panel with no Network Manager. Try to create a new system user and log in. Is nm-applet present? Logging in using another previously working account produces the same result -- Xfce Panel displays Notification Plugin (bell icon), but no Network Manager icon. Creating a new account and logging in produces the same result. David
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/18/24 05:34, e...@gmx.us wrote: On 4/18/24 05:27, David Christensen wrote: On 4/17/24 12:37, Richmond wrote: What are the permissions on the nm-applet binary? maybe it doesn't have permission to execute, or the process which starts it doesn't have permission. 2024-04-18 02:24:20 root@laalaa ~ # ls -l `which nm-applet` -rwxr-xr-x 1 root root 250784 Feb 27 2021 /usr/bin/nm-applet I do not know what binary starts nm-applet, but here is a WAG: 2024-04-18 02:27:13 root@laalaa ~ # ll -l `which xfce4-panel` -rwxr-xr-x 1 root root 363328 2021-02-27 08:29:44 /usr/bin/xfce4-panel Can PPID tell you? Yes. xfce4-session is the parent of nm-applet: 2024-04-18 10:27:32 dpchrist@laalaa ~ $ ps -Ao "%p %P %a" | grep nm-applet 15301379 nm-applet 39373865 grep nm-applet 2024-04-18 10:27:44 dpchrist@laalaa ~ $ ps -Ao "%p %P %a" | grep 1379 13791344 xfce4-session 14211379 /usr/bin/ssh-agent x-session-manager 14581379 xfwm4 --display :0.0 --sm-client-id 2d1bdc0a5-06f2-4835-944e-7e402203bd9f 14711379 xfsettingsd --display :0.0 --sm-client-id 29896f657-477c-498a-8156-db9d51f74bcf 14971379 xfce4-panel --display :0.0 --sm-client-id 23fd394e3-a4ee-485b-b43c-38bf845d6699 15011379 xfdesktop --display :0.0 --sm-client-id 2c913de99-acac-4d70-b6e9-8eda44b6f6da 15061379 xfce4-power-manager --restart --sm-client-id 24cebdb39-bdc2-496f-aab0-7b11397b48d3 15131379 /usr/lib/x86_64-linux-gnu/xfce4/notifyd/xfce4-notifyd 15151379 light-locker 15161379 /usr/bin/python3 /usr/share/system-config-printer/applet.py 15171379 /usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1 15241379 xiccd 15301379 nm-applet 39403865 grep 1379 2024-04-18 10:29:20 dpchrist@laalaa ~ $ ls -l `which xfce4-session` -rwxr-xr-x 1 root root 260496 Dec 23 2020 /usr/bin/xfce4-session David
Re: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/17/24 12:07, Charles Curley wrote: On Wed, 17 Apr 2024 11:41:24 -0700 David Christensen wrote: My WAG is that nm-applet is failing to start, but I have been unable to find if and where any error message is reported. My instance of nm-applet does run, and I see this as part of the boot process: root@hawk:~# journalctl -b | grep nm-app Apr 15 11:27:42 hawk NetworkManager[1354]: [1713202062.7737] agent-manager: agent[108f011a1115d508,:1.131/org.freedesktop.nm-applet/1000]: agent registered root@hawk:~# I suspect that if nm-applet doesn't start, you won't see any output from that command. 2024-04-18 03:07:20 root@laalaa ~ # journalctl -b | grep nm-app Apr 18 03:01:39 laalaa NetworkManager[833]: [1713434499.3660] agent-manager: agent[2a08cda7fa35849b,:1.48/org.freedesktop.nm-applet/13250]: agent registered David
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/17/24 19:41, Gareth Evans wrote: On Wed 17/04/2024 at 19:41, David Christensen wrote: On 4/17/24 03:47, Gareth Evans wrote: On Wed 17/04/2024 at 09:18, David Christensen wrote: On 4/16/24 08:56, Gareth Evans wrote: On 16 Apr 2024, at 00:18, David Christensen wrote: On 4/15/24 09:21, Gareth Evans wrote: On Sun 14/04/2024 at 13:29, David Christensen wrote: ... I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). ... There is apparently a long history of nm-applet/XFCE panel-related issues (and not many great answers), as evidenced by such as https://bbs.archlinux.org/viewtopic.php?id=161998 https://forum.xfce.org/viewtopic.php?id=6105 https://superuser.com/questions/900490/networkmanager-icon-on-notification-area-is-not-present I tried: 1. Remove Notification applet, restart, add Notification applet, no Network Manager, restart -- no Network Manager. 2. Look at /etc/NetworkManager/NetworkManager.conf 2024-04-18 02:40:49 dpchrist@laalaa ~ $ cat /etc/NetworkManager/NetworkManager.conf [main] plugins=ifupdown,keyfile [ifupdown] managed=false Change it to: 2024-04-18 02:43:35 root@laalaa ~ # cat /etc/NetworkManager/NetworkManager.conf [main] plugins=ifupdown,keyfile [ifupdown] managed=true Restart -- no Network Manager. Verify /etc//NetworkManager/NetworkManager.conf: 2024-04-18 02:45:05 dpchrist@laalaa ~ $ cat /etc/NetworkManager/NetworkManager.conf [main] plugins=ifupdown,keyfile [ifupdown] managed=true If you get ps output like this directly after a reboot: $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet 2024-04-18 02:45:10 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1526 0.1 0.2 426484 35256 ?Sl 02:44 0:00 nm-applet dpchrist1869 0.0 0.0 3240 712 pts/0S+ 02:47 0:00 grep nm-applet then I don't think the issue is with the starting/running of nm-applet itself, but rather some issue with the notification plugin, which I'm not sure how to begin troubleshooting. I reluctantly abandoned XFCE partly due to panel instability some time ago. The advice in the final link to rm -rf ~/.config/xfce* might be a bit extreme, but I might try renaming it to force a rebuild on next login, and possibly reinstall task-xfce-desktop (or selected packages) Move aside the ~/.config/xfce4 directory: 2024-04-18 02:50:20 dpchrist@laalaa ~ $ ls -ld .config/xfce4/ drwxr-xr-x 8 dpchrist dpchrist 4096 Apr 18 00:45 .config/xfce4/ 2024-04-18 02:50:23 dpchrist@laalaa ~ $ mv .config/xfce4/ .config/xfce4-20240418-180045 Restart -- screen with wallpaper alone. Right-click on desktop -> Applications -> Settings -> Panel brings up the Panel Preferences application. The Items tab shows my previous settings. Fumbling around, I am unable to get that panel to display (?). If I create a new panel, I am unable to get it to display (?). Open a Terminal and look if Xfce created a new configuration directory: 2024-04-18 02:57:31 dpchrist@laalaa ~ $ ls -ld .config/xfce4* drwxr-xr-x 7 dpchrist dpchrist 4096 Apr 18 02:57 .config/xfce4 drwxr-xr-x 8 dpchrist dpchrist 4096 Apr 18 00:45 .config/xfce4-20240418-180045 Yes, it did. Compare the old directory to the new directory: 2024-04-18 02:58:11 dpchrist@laalaa ~ $ diff -rq .config/xfce4-20240418-180045/ .config/xfce4 diff: .config/xfce4-20240418-180045/desktop/icons.screen.latest.rc: No such file or directory Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1008x725.rc Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1424x857.rc Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1664x1007.rc Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1904x1037.rc Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1904x1064.rc Only in .config/xfce4-20240418-180045/desktop: icons.screen0-2928x1037.rc Files .config/xfce4-20240418-180045/desktop/icons.screen0-3824x1037.rc and .config/xfce4/desktop/icons.screen0-3824x1037.rc differ Only in .config/xfce4-20240418-180045/: help.rc Only in .config/xfce4-20240418-180045/panel: launcher-15 Only in .config/xfce4/panel: launcher-17 Only in .config/xfce4/panel: launcher-18 Only in .config/xfce4/panel: launcher-19 Only in .config/xfce4/panel: launcher-20 Only in .config/xfce4-20240418-180045/panel: launcher-24 Only in .config/xfce4-20240418-180045/panel: launcher-28 Only in .config/xfce4-20240418-180045/panel: launcher-7 Only in .config/xfce4-20240418-180045/: src Files .config/xfce4-20240418-180045/terminal/accels.scm and .config/xfce4/terminal/accels.scm differ Only in .config/xfce4-20240418-180045/terminal: terminalrc Only in .config/xfce4-20240418-180045/: xfce4-screenshooter Only in .config/xfce4-20240418-18004
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/17/24 13:56, e...@gmx.us wrote: On 4/17/24 15:37, Richmond wrote: David Christensen writes: My WAG is that nm-applet is failing to start, but I have been unable to find if and where any error message is reported. What are the permissions on the nm-applet binary? And is its filesystem mounted with noexec? 2024-04-18 02:27:18 root@laalaa ~ # df `which nm-applet` Filesystem 1M-blocks Used Available Use% Mounted on /dev/mapper/sdb3_crypt12084M 8927M 2522M 78% / 2024-04-18 02:28:57 root@laalaa ~ # mount | grep sdb3 /dev/mapper/sdb3_crypt on / type ext4 (rw,relatime,errors=remount-ro) David
Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/17/24 12:37, Richmond wrote: David Christensen writes: On Sun 14/04/2024 at 13:29, David Christensen wrote: ... I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). ... What are the permissions on the nm-applet binary? maybe it doesn't have permission to execute, or the process which starts it doesn't have permission. 2024-04-18 02:24:20 root@laalaa ~ # ls -l `which nm-applet` -rwxr-xr-x 1 root root 250784 Feb 27 2021 /usr/bin/nm-applet I do not know what binary starts nm-applet, but here is a WAG: 2024-04-18 02:27:13 root@laalaa ~ # ll -l `which xfce4-panel` -rwxr-xr-x 1 root root 363328 2021-02-27 08:29:44 /usr/bin/xfce4-panel David
Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
Forwarded Message Subject: Re: Debian 11 Xfce panel Network Manager applet has disappeared Date: Wed, 17 Apr 2024 01:18:34 -0700 From: David Christensen To: Gareth Evans On 4/16/24 08:56, Gareth Evans wrote: On 16 Apr 2024, at 00:18, David Christensen wrote: On 4/15/24 09:21, Gareth Evans wrote: On Sun 14/04/2024 at 13:29, David Christensen wrote: ... I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). ... 2024-04-15 16:08:23 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet dpchrist1940 0.0 0.0 3240 644 pts/0S+ 16:15 0:00 grep nm-applet 2024-04-15 16:15:12 dpchrist@laalaa ~ $ nm-applet 2024-04-15 16:15:31 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet dpchrist1952 0.0 0.0 3240 644 pts/0S+ 16:15 0:00 grep nm-applet That seems to show it's running from the outset, just not being displayed on the panel. Does rebooting (or logging out and in again) bring it back? No. David
Fwd: Debian 11 Xfce panel Network Manager applet has disappeared
Forwarded Message Subject: Re: Debian 11 Xfce panel Network Manager applet has disappeared Date: Wed, 17 Apr 2024 11:38:49 -0700 From: David Christensen To: Gareth Evans On 4/17/24 03:47, Gareth Evans wrote: On Wed 17/04/2024 at 09:18, David Christensen wrote: On 4/16/24 08:56, Gareth Evans wrote: On 16 Apr 2024, at 00:18, David Christensen wrote: On 4/15/24 09:21, Gareth Evans wrote: On Sun 14/04/2024 at 13:29, David Christensen wrote: ... I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). ... 2024-04-15 16:08:23 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet dpchrist1940 0.0 0.0 3240 644 pts/0S+ 16:15 0:00 grep nm-applet 2024-04-15 16:15:12 dpchrist@laalaa ~ $ nm-applet 2024-04-15 16:15:31 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet dpchrist1952 0.0 0.0 3240 644 pts/0S+ 16:15 0:00 grep nm-applet That seems to show it's running from the outset, just not being displayed on the panel. Does rebooting (or logging out and in again) bring it back? No. OK. You may have checked this already, but in case not, if I install XFCE and go to Settings > Session and Startup > Application Autostart there is an entry in the list called "Network (Manage your network connections)" It is checked. which shows a tooltip of "command: nm-applet" Command: nm-applet Might this somehow have become unset? I'm not sure if it's possible for GUI config helpers to become detached from actual settings - this seems to describe the relevant locations: https://unix.stackexchange.com/questions/669372/xfce4-session-and-startup-where-are-autostart-items-saved 2024-04-17 11:21:06 dpchrist@laalaa ~ $ grep nm-applet ~/.config/autostart grep: /home/dpchrist/.config/autostart: No such file or directory 2024-04-17 11:33:11 dpchrist@laalaa ~ $ find .config -name autostart 2024-04-17 11:33:22 dpchrist@laalaa ~ $ 2024-04-17 11:34:14 dpchrist@laalaa ~ $ grep -r nm-applet /etc/xdg/autostart /etc/xdg/autostart/nm-applet.desktop:Exec=nm-applet /etc/xdg/autostart/nm-applet.desktop:X-GNOME-Bugzilla-Component=nm-applet My WAG is that nm-applet is failing to start, but I have been unable to find if and where any error message is reported. David
Re: Debian 11 Xfce panel Network Manager applet has disappeared
On 4/15/24 09:21, Gareth Evans wrote: On Sun 14/04/2024 at 13:29, David Christensen wrote: ... I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). ... Hi David, I can't speak for XFCE, but certainly for Mate there was a time when multiple notification area panel widgets were available, not all of which would show everything to be expected. Is that a possibility? Does $ ps aux |grep nm-applet show anything? Before or after $ nm-applet ? 2024-04-15 16:08:23 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet dpchrist1940 0.0 0.0 3240 644 pts/0S+ 16:15 0:00 grep nm-applet 2024-04-15 16:15:12 dpchrist@laalaa ~ $ nm-applet 2024-04-15 16:15:31 dpchrist@laalaa ~ $ ps aux | grep nm-applet dpchrist1518 0.1 0.2 426500 35380 ?Sl 16:06 0:00 nm-applet dpchrist1952 0.0 0.0 3240 644 pts/0S+ 16:15 0:00 grep nm-applet David
Debian 11 Xfce panel Network Manager applet has disappeared
debian-user: I have a Dell Latitude E6520: 2024-04-14 04:28:39 dpchrist@laalaa ~ $ cat /etc/debian_version ; uname -a 11.9 Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux 2024-04-14 04:34:40 dpchrist@laalaa ~ $ dpkg-query -l xfce4 network-manager network-manager-gnome Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=---= ii network-manager 1.30.6-1+deb11u1 amd64network management framework (daemon and userspace tools) ii network-manager-gnome 1.20.0-3 amd64network management framework (GNOME frontend) ii xfce4 4.16 all Meta-package for the Xfce Lightweight Desktop Environment I have used the Xfce panel Network Manager applet for many years. Tonight, I noticed that it has disappeared (!). But, the machine is connected to my LAN: 2024-04-14 05:24:10 root@laalaa ~ # ifconfig wlp3s0 wlp3s0: flags=4163 mtu 1500 inet 192.168.REDACTED netmask 255.255.255.0 broadcast 192.168.REDACTED inet6 REDACTED prefixlen 64 scopeid 0x20 ether REDACTED txqueuelen 1000 (Ethernet) RX packets 5786 bytes 2830592 (2.6 MiB) RX errors 0 dropped 119 overruns 0 frame 0 TX packets 3897 bytes 518278 (506.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Looking in the the Xfce panel Application Menu, I am unable to find Network Manager. Looking at the Debian WIKI page "NetworkManager": https://wiki.debian.org/NetworkManager It looks like the Network Manager daemon is running: 2024-04-14 04:32:49 dpchrist@laalaa ~ $ ps -A | grep -i network 828 ?00:00:00 NetworkManager nm-applet(1) looks like the program I want (?). Attempting to start it via a terminal has no effect: 2024-04-14 04:40:25 dpchrist@laalaa ~ $ which nm-applet /usr/bin/nm-applet 2024-04-14 05:27:05 dpchrist@laalaa ~ $ nm-applet 2024-04-14 05:27:08 dpchrist@laalaa ~ $ RTFM nm-applet(1), it seems the desktop session manager is failing to start nm-applet(1) (?): 2024-04-14 04:58:49 dpchrist@laalaa ~ $ man nm-applet | cat ... DESCRIPTION nm-applet is a GTK-based GUI applet to monitor network status and devices and to start and stop network connections managed by NetworkManager. nm-applet is normally started at login by the desktop session manager and does not need to be run manu- ally. nm-applet conforms to the XDG System Tray specification and requires that the desktop environment provide a System Tray implementation in which the applet will be embedded. I am unable to find relevant error messages under /var/log. The network Connection Editor can be run via a terminal: 2024-04-14 04:55:29 dpchrist@laalaa ~ $ which nm-connection-editor /usr/bin/nm-connection-editor 2024-04-14 04:55:38 dpchrist@laalaa ~ $ nm-connection-editor Does anyone know why the Network Manager Xfce panel applet is missing, how to get it back, and/or how to start it some other way? David
Re: HDD long-term data storage with ensured integrity
On 4/12/24 08:14, piorunz wrote: On 10/04/2024 12:10, David Christensen wrote: Those sound like some compelling features. I believe the last time I tried Btrfs was Debian 9 (?). I ran into problems because I did not do the required manual maintenance (rebalancing). Does the Btrfs in Debian 11 or Debian 12 still require manual maintenance? If so, what and how often? I don't do balance at all, it's not required. Scrub is recommended, because it will detect any bit-rot due to hardware errors on HDD media. It scans the entire surface of allocated sectors on all drives. I do scrub usually monthly. Thank you for the information. David
Re: HDD long-term data storage with ensured integrity
On 4/10/24 08:49, Paul Leiber wrote: Am 10.04.2024 um 13:10 schrieb David Christensen: Does the Btrfs in Debian 11 or Debian 12 still require manual maintenance? If so, what and how often? Scrub and balance are actions which have been recommended. I am using btrfsmaintenance scripts [1][2] to automate this. I am doing a weekly balance and a monthly scrub. After some reading today, I am getting unsure if this is approach is correct, especially if balance is necessary anymore (it usually doesn't find anything to do anyway), so please take these periods with caution. My main message is that such operations can be automated using the linked scripts. Best regards, Paul [1] https://packages.debian.org/bookworm/btrfsmaintenance [2] https://github.com/kdave/btrfsmaintenance Thank you. Those scripts should be useful. David
Re: HDD long-term data storage with ensured integrity
On 4/9/24 17:08, piorunz wrote: On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? I use Btrfs, on all my systems, including some servers, with soft Raid1 and Raid10 modes (because these modes are considered stable and production ready). I decided on Btrfs not ZFS, because Btrfs allows to migrate drives on the fly while partition is live and heavily used, replace them with different sizes and types, mixed capacities, change Raid levels, change amount of drives too. I could go from single drive to Raid10 on 4 drives and back while my data is 100% available at all times. It saved my bacon many times, including hard checksum corruption on NVMe drive which otherwise I would never know about. Thanks to Btrfs I located the corrupted files, fixed them, got hardware replaced under warranty. Also helped with corrupted RAM: Btrfs just refused to save file because saved copy couldn't match read checksum from the source due to RAM bit flips. Diagnosed, then replaced memory, all good. I like a lot when one of the drives get ATA reset for whatever reason, and all other drives continue to read and write, I can continue using the system for hours, if I even notice. Not possible in normal circumstances without Raid. Once the problematic drive is back, or after reboot if it's more serious, then I do "scrub" command and everything is resynced again. If I don't do that, then Btrfs dynamically correct checksum errors on the fly anyway. And list goes on - I've been using Btrfs for last 5 years, not a single problem to date, it survived hard resets, power losses, drive failures, countless migrations. Those sound like some compelling features. I believe the last time I tried Btrfs was Debian 9 (?). I ran into problems because I did not do the required manual maintenance (rebalancing). Does the Btrfs in Debian 11 or Debian 12 still require manual maintenance? If so, what and how often? [1] https://github.com/openzfs/zfs/issues/15526 [2] https://github.com/openzfs/zfs/issues/15933 Problems reported here are from Linux kernel 6.5 and 6.7 on Gentoo system. Does this even affects Debian Stable with 6.1 LTS? I do not know. -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄ David
Re: Why LVM
On 4/8/24 16:54, Stefan Monnier wrote: If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS drive and rack, etc.), can I put LVM on it such that when the device is connected to a Debian system with a graphical desktop (I use Xfce) an icon is displayed on the desktop that I can interact with to display the file systems in my file manager (Thunar)? In the past: definitely not. Currently: no idea. I suspect not, because I think the behavior on disconnection is still poor (you want to be extra careful to deactivate all the volumes on the drive *before* removing it, otherwise they tend to linger "for ever"). I guess that's one area where partitions are still significantly better than LVM. Stefan "who doesn't use much hot-plugging of mass storage" Thank you for the clarification. :-) David
Re: Why LVM
On 4/8/24 14:08, Stefan Monnier wrote: David Christensen [2024-04-08 11:28:04] wrote: Why LVM? Personally, I've been using LVM everywhere I can (i.e. everywhere except on my OpenWRT router, tho I've also used LVM there back when my router had an HDD. I also use LVM on my 2GB USB rescue image). To me the question is rather the reverse: why not? I basically see it as a more flexible form of partitioning. Even in the worst cases where I have a single LV volume, I appreciate the fact that it forces me to name things, isolating me from issue linked to predicting the name of the device and the issues that plague UUIDs (the fact they're hard to remember, and that they're a bit too magical/hidden for my taste, so they sometimes change when I don't want them to and vice versa). Stefan If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS drive and rack, etc.), can I put LVM on it such that when the device is connected to a Debian system with a graphical desktop (I use Xfce) an icon is displayed on the desktop that I can interact with to display the file systems in my file manager (Thunar)? David
Re: HDD long-term data storage with ensured integrity
On 4/8/24 13:04, Marc SCHAEFER wrote: Hello, On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote: So, an ext4 file system on an LVM logical volume? Why LVM? Are you implementing redundancy (RAID)? Is your data larger than a single disk (concatenation/ JBOD)? Something else? For off-site long-term offline archiving, no, I am not using RAID. No, it's not LVM+md, just plain LVM for flexibility. Typically I use 16 TB hard drives, and I tend to use one LV per data source, the LV name being the data source and the date of the copy. Or sometimes I just copy a raw volume (ext4 or something else) to a LV. With smaller drives (4 TB) I tend to not use LVM, just plain ext4 on the raw disk. I almost never use partitionning. However, I tend to use luks encryption (per ext4 filesystem) when the drives are stored off-site. So it's either LVM -> LV -> LUKS -> ext4 or raw disk -> LUKS -> ext4. You can find some of the scripts I use to automate this off-site long-term archiving here: https://git.alphanet.ch/gitweb/?p=various;a=tree;f=offsite-archival/LVM-LUKS Thank you for the clarification. :-) David
Re: HDD long-term data storage with ensured integrity
On 4/8/24 02:38, Marc SCHAEFER wrote: For offline storage: On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? I use LVM on ext4, and I add a MD5SUMS file at the root. I then power up the drives at least once a year and check the MD5SUMS. A simple CRC could also work, obviously. So far, I have not detected MORE corruption with this method than the drive ECC itself (current drives & buses are much better than they used to be). When I have errors detected, I replace the file with another copy (I usually have multiple off-site copies, and sometimes even on-site online copies, but not always). When the errors add up, it is time to buy another drive, usually after 5+ years or even sometimes 10+ years. So, just re-reading the content might be enough, once a year or so. This is for HDD (for SDD I have no offline storage experience, it could be shorter). Thank you for the reply. So, an ext4 file system on an LVM logical volume? Why LVM? Are you implementing redundancy (RAID)? Is your data larger than a single disk (concatenation/ JBOD)? Something else? David
Re: readonly installer, (SOLVED)
On 4/3/24 19:05, Stefan Monnier wrote: I have a 128 MB USB flash drive from back in the day that includes a write protect switch. There are few products today that offer that feature. Side note: AFAIK this "write protect switch" doesn't prevent writing. It just tells your card reader that you'd like to avoid writing to it. Whether it ends up doing what you want depends on the hardware exposing that info to the driver and the driver paying attention to it. Stefan No, it is a USB flash drive with a write protect switch. Here is a modern example: https://www.amazon.com/dp/B00JJIE95G David
Re: Debian ISOs on USB stick
On 4/3/24 05:56, Thomas Schmitt wrote: Hi, i read from bytes 2085412 to 2085479: "Info rrmation Syste rm VolumeSYSTEM~" which is similar to the alterations of one of the USB sticks shown in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998#35 The web knows about a Microsoft folder named "System Volume Information". https://answers.microsoft.com/en-us/windows/forum/all/system-volume-information-what-is-it-and-what-is/3bc81844-0baa-46bd-9949-4efb4678b677 "whenever I put my flash-drive or my micro sd adapter and sd card into my windows 8.1 something called "System Volume Information" is always getting added on." So did you perhaps show this USB stick to a running MS-Windows system ? Have a nice day :) Thomas It is possible the drive was inserted into a Windows computer. If and when I need a newer d-i, perhaps I will put the ISO onto a USB flash drive, conduct more experiments, and post the results. I apologize for blaming d-i for what might be Dell, Intel, BIOS/UEFI, Microsoft, and/or other bugs. David
Re: readonly installer, (SOLVED)
On 4/3/24 08:16, David Wright wrote: On Tue 02 Apr 2024 at 05:54:06 (-0700), David Christensen wrote: On 4/1/24 11:35, DdB wrote: Am 01.04.2024 um 18:52 schrieb David Christensen: A bad USB flash drive would explain why you cannot boot the Debian installer. Please buy a good quality USB 3.0+ flash drive and try again. A friend of mine just let me use an external CD-Drive with the netboot image. I thought about suggesting that in my last post, but did not want to complicate things. A key advantage of using a CD-R disc is that you can verify the disc contents and/or checksum against the ISO and/or checksum now and in the future. This is not true for a USB flash drive, because the Debian installer modifies the contents of the USB flash drive when it runs. If this troubles you, you can also use an SD card with a write-lock, or a µSD card with a lock on the SD adaptor. Check that the write-lock works with the logs when you plug it in, or run fdisk/gdisk and immediately quit. Cheers, David. I have a 128 MB USB flash drive from back in the day that includes a write protect switch. There are few products today that offer that feature. David
Re: HDD long-term data storage with ensured integrity
On 4/2/24 14:57, David Christensen wrote: AIUI neither LVM nor ext4 have data and metadata checksum and correction features. But, it should be possible to achieve such by including dm-integrity (for checksumming) and some form of RAID (for correction) in the storage stack. I need to explore that possibility further. I have RTFM dm-integrity before and it is still experimental. I need something that is production ready: https://manpages.debian.org/bookworm/cryptsetup-bin/cryptsetup.8.en.html Authenticated disk encryption (EXPERIMENTAL) David
Re: Debian ISOs on USB stick, was: SOLVED
On 4/3/24 03:36, David Christensen wrote: On 4/3/24 00:30, Thomas Schmitt wrote: Hi, David Christensen wrote: It's a relatively simple experiment to confirm that a USB flash drive with d-i changes after the first boot. This could still be https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998 where Lenovo BIOS and/or MS-Windows altered the USB stick. Same for finding which bytes change. I fail to find this particular info in Date: Tue, 2 Apr 2024 14:46:42 -0700 From: David Christensen Message-ID: If we have the exact ISO name (i.e. URL from where it stems) and the byte address of the alteration, xorriso can find the affected file, if any. In case of bug #1056998 it was the EFI partition image /boot/grub/efi.img. Mounting the altered and unaltered image files showed changes in the FAT filesystem which point to the culprits Lenovo and Microsoft. The other plausible way of altering the ISO image on the stick would be adding a new partition. The MBR partition table is part of the Debian ISO and thus part of the checksummed area. Even if all other alterations happen after the end of the checksummed ISO image, the changed partition table will cause the Debian checksum to become invalid. (I am not aware that Debian installer changes the table. If it does indeed then this might be worth a new bug discussion.) Have a nice day :) Thomas 2024-04-03 03:29:18 root@laalaa /samba/dpchrist/iso/debian/11.3.0 # cmp --verbose debian-11.3.0-amd64-netinst.iso /dev/sdb 2083201 0 377 2083202 0 377 2083203 0 377 2085249 0 377 2085250 0 377 2085251 0 377 2085409 0 102 2085410 0 40 2085412 0 111 2085414 0 156 2085416 0 146 2085418 0 157 2085420 0 17 2085422 0 162 2085423 0 162 2085425 0 155 2085427 0 141 2085429 0 164 2085431 0 151 2085433 0 157 2085437 0 156 2085441 0 1 2085442 0 123 2085444 0 171 2085446 0 163 2085448 0 164 2085450 0 145 2085452 0 17 2085454 0 162 2085455 0 155 2085457 0 40 2085459 0 126 2085461 0 157 2085463 0 154 2085465 0 165 2085469 0 155 2085471 0 145 2085473 0 123 2085474 0 131 2085475 0 123 2085476 0 124 2085477 0 105 2085478 0 115 2085479 0 176 2085480 0 61 2085481 0 40 2085482 0 40 2085483 0 40 2085484 0 26 2085486 0 167 2085487 0 174 2085488 0 277 2085489 0 235 2085490 0 124 2085491 0 235 2085492 0 124 2085495 0 175 2085496 0 277 2085497 0 235 2085498 0 124 2085500 0 5 4719105 0 56 4719106 0 40 4719107 0 40 4719108 0 40 4719109 0 40 4719110 0 40 4719111 0 40 4719112 0 40 4719113 0 40 4719114 0 40 4719115 0 40 4719116 0 20 4719118 0 167 4719119 0 174 4719120 0 277 4719121 0 235 4719122 0 124 4719123 0 235 4719124 0 124 4719127 0 175 4719128 0 277 4719129 0 235 4719130 0 124 4719132 0 5 4719137 0 56 4719138 0 56 4719139 0 40 4719140 0 40 4719141 0 40 4719142 0 40 4719143 0 40 4719144 0 40 4719145 0 40 4719146 0 40 4719147 0 40 4719148 0 20 4719150 0 167 4719151 0 174 4719152 0 277 4719153 0 235 4719154 0 124 4719155 0 235 4719156 0 124 4719159 0 175 4719160 0 277 4719161 0 235 4719162 0 124 4719169 0 102 4719170 0 107 4719172 0 165 4719174 0 151 4719176 0 144 4719180 0 17 4719182 0 377 4719183 0 377 4719184 0 377 4719185 0 377 4719186 0 377 4719187 0 377 4719188 0 377 4719189 0 377 4719190 0 377 4719191 0 377 4719192 0 377 4719193 0 377 4719194 0 377 4719197 0 377 4719198 0 377 4719199 0 377 4719200 0 377 4719201 0 1 4719202 0 111 4719204 0 156 4719206 0 144 4719208 0 145 4719210 0 170 4719212 0 17 4719214 0 377 4719215 0 145 4719217 0 162 4719219 0 126 4719221 0 157 4719223 0 154 4719225 0 165 4719229 0 155 4719231 0 145 4719233 0 111 4719234 0 116 4719235 0 104 4719236 0 105 4719237 0 130 4719238 0 105 4719239 0 176 4719240 0 61 4719241 0 40 4719242 0 40 4719243 0 40 4719244 0 40 4719246 0 171 4719247 0 174 4719248 0 277 4719249 0 235 4719250 0 124 4719251 0 235 4719252 0 124 4719255 0 175 4719256 0 277 4719257 0 235 4719258 0 124 4719259 0 1 4719260 0 5 4719261 0 114 4721153 0 173 4721155 0 71 4721157 0 101 4721159 0 65 4721161 0 104 4721163 0 101 4721165 0 106 4721167 0 65 4721169 0 67 4721171 0 55
Re: Debian ISOs on USB stick, was: SOLVED
On 4/3/24 00:30, Thomas Schmitt wrote: Hi, David Christensen wrote: It's a relatively simple experiment to confirm that a USB flash drive with d-i changes after the first boot. This could still be https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998 where Lenovo BIOS and/or MS-Windows altered the USB stick. Same for finding which bytes change. I fail to find this particular info in Date: Tue, 2 Apr 2024 14:46:42 -0700 From: David Christensen Message-ID: If we have the exact ISO name (i.e. URL from where it stems) and the byte address of the alteration, xorriso can find the affected file, if any. In case of bug #1056998 it was the EFI partition image /boot/grub/efi.img. Mounting the altered and unaltered image files showed changes in the FAT filesystem which point to the culprits Lenovo and Microsoft. The other plausible way of altering the ISO image on the stick would be adding a new partition. The MBR partition table is part of the Debian ISO and thus part of the checksummed area. Even if all other alterations happen after the end of the checksummed ISO image, the changed partition table will cause the Debian checksum to become invalid. (I am not aware that Debian installer changes the table. If it does indeed then this might be worth a new bug discussion.) Have a nice day :) Thomas 2024-04-03 03:29:18 root@laalaa /samba/dpchrist/iso/debian/11.3.0 # cmp --verbose debian-11.3.0-amd64-netinst.iso /dev/sdb 2083201 0 377 2083202 0 377 2083203 0 377 2085249 0 377 2085250 0 377 2085251 0 377 2085409 0 102 2085410 0 40 2085412 0 111 2085414 0 156 2085416 0 146 2085418 0 157 2085420 0 17 2085422 0 162 2085423 0 162 2085425 0 155 2085427 0 141 2085429 0 164 2085431 0 151 2085433 0 157 2085437 0 156 2085441 0 1 2085442 0 123 2085444 0 171 2085446 0 163 2085448 0 164 2085450 0 145 2085452 0 17 2085454 0 162 2085455 0 155 2085457 0 40 2085459 0 126 2085461 0 157 2085463 0 154 2085465 0 165 2085469 0 155 2085471 0 145 2085473 0 123 2085474 0 131 2085475 0 123 2085476 0 124 2085477 0 105 2085478 0 115 2085479 0 176 2085480 0 61 2085481 0 40 2085482 0 40 2085483 0 40 2085484 0 26 2085486 0 167 2085487 0 174 2085488 0 277 2085489 0 235 2085490 0 124 2085491 0 235 2085492 0 124 2085495 0 175 2085496 0 277 2085497 0 235 2085498 0 124 2085500 0 5 4719105 0 56 4719106 0 40 4719107 0 40 4719108 0 40 4719109 0 40 4719110 0 40 4719111 0 40 4719112 0 40 4719113 0 40 4719114 0 40 4719115 0 40 4719116 0 20 4719118 0 167 4719119 0 174 4719120 0 277 4719121 0 235 4719122 0 124 4719123 0 235 4719124 0 124 4719127 0 175 4719128 0 277 4719129 0 235 4719130 0 124 4719132 0 5 4719137 0 56 4719138 0 56 4719139 0 40 4719140 0 40 4719141 0 40 4719142 0 40 4719143 0 40 4719144 0 40 4719145 0 40 4719146 0 40 4719147 0 40 4719148 0 20 4719150 0 167 4719151 0 174 4719152 0 277 4719153 0 235 4719154 0 124 4719155 0 235 4719156 0 124 4719159 0 175 4719160 0 277 4719161 0 235 4719162 0 124 4719169 0 102 4719170 0 107 4719172 0 165 4719174 0 151 4719176 0 144 4719180 0 17 4719182 0 377 4719183 0 377 4719184 0 377 4719185 0 377 4719186 0 377 4719187 0 377 4719188 0 377 4719189 0 377 4719190 0 377 4719191 0 377 4719192 0 377 4719193 0 377 4719194 0 377 4719197 0 377 4719198 0 377 4719199 0 377 4719200 0 377 4719201 0 1 4719202 0 111 4719204 0 156 4719206 0 144 4719208 0 145 4719210 0 170 4719212 0 17 4719214 0 377 4719215 0 145 4719217 0 162 4719219 0 126 4719221 0 157 4719223 0 154 4719225 0 165 4719229 0 155 4719231 0 145 4719233 0 111 4719234 0 116 4719235 0 104 4719236 0 105 4719237 0 130 4719238 0 105 4719239 0 176 4719240 0 61 4719241 0 40 4719242 0 40 4719243 0 40 4719244 0 40 4719246 0 171 4719247 0 174 4719248 0 277 4719249 0 235 4719250 0 124 4719251 0 235 4719252 0 124 4719255 0 175 4719256 0 277 4719257 0 235 4719258 0 124 4719259 0 1 4719260 0 5 4719261 0 114 4721153 0 173 4721155 0 71 4721157 0 101 4721159 0 65 4721161 0 104 4721163 0 101 4721165 0 106 4721167 0 65 4721169 0 67 4721171 0 55 4721173 0 71 4721175 0 102 4721177 0 63 4721179 0 61 4721181 0 55 4721183 0 64 4721185 0 71 4721187 0 62 4721189 0 105 4721191 0 55 4721193 0 102 4721195 0 66 4721197 0 105
Re: Debian ISOs on USB stick, was: SOLVED
On 4/2/24 08:56, Thomas Schmitt wrote: Hi, David Christensen wrote: the Debian installer modifies the contents of the USB flash drive when it runs. Do you mean inside the range of the ISO image or outside by creating a new partition ? songbird wrote: if it is an iso image copied to the USB stick it should not be modified if you haven't somehow told the installer to install the system to that USB stick (somehow). There are other parties which feel entitled to operate on the EFI System Partition of a USB stick. In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998 we found that Lenovo Thinkpad firmware created directories for storing an empty file named "/efi/Lenovo/BIOS/SelfHealing.fd" and that MS-Windows created a 12-byte file named "/System Volume Information/WPSettings.dat" when it had contact with the USB stick. i guess if you wanted to be really sure you could mount it read-only. I think it's the installer which mounts the ISO 9660 filesystem. Whatever, the Linux kernel has no regular means to alter an ISO 9660 filesystem. Neither kernel nor Debain installer will be so daring to operate with byte level commands on that filesystem. But the FAT filesystem in file /boot/grub/efi.img of the ISO 9660 filesystem in debian-12.*-amd64-netinst.iso is advertised by the partition table of the image and thus attracts vermin. Have a nice day :) Thomas Please see my reply to songbird. It's a relatively simple experiment to confirm that a USB flash drive with d-i changes after the first boot. Same for finding which bytes change. The challenge is figuring out what performed the change(s) and why. I assumed it was d-i, but no longer own 64-bit BIOS-only computers to confirm. David
Re: HDD long-term data storage with ensured integrity
On 4/2/24 06:55, Stefan Monnier wrote: The most obvious alternative to ZFS on Debian would be Btrfs. Does anyone have any comments or suggestions regarding Btrfs and data corruption bugs, concurrency, CMM level, PSP, etc.? If you're worried about such things, I'd think "the most obvious alternative" is LVM+ext4. Both Btrfs and ZFS share the same underlying problem: more features => more code => more bugs. Stefan AIUI neither LVM nor ext4 have data and metadata checksum and correction features. But, it should be possible to achieve such by including dm-integrity (for checksumming) and some form of RAID (for correction) in the storage stack. I need to explore that possibility further. David
Re: SOLVED
On 4/2/24 07:55, songbird wrote: David Christensen wrote: I thought about suggesting that in my last post, but did not want to complicate things. A key advantage of using a CD-R disc is that you can verify the disc contents and/or checksum against the ISO and/or checksum now and in the future. This is not true for a USB flash drive, because the Debian installer modifies the contents of the USB flash drive when it runs. if it is an iso image copied to the USB stick it should not be modified if you haven't somehow told the installer to install the system to that USB stick (somehow). i guess if you wanted to be really sure you could mount it read-only. songbird I used to think that the d-i ran in memory when booted from read-write media, but discovered otherwise several years ago. I previously downloaded debian-11.3.0-amd64-netinst.iso and SHA512SUMS: 2022-04-29 22:16:19 dpchrist@tinkywinky ~/samba/dpchrist/iso/debian/11.3.0 $ ls -l total 380414 -rwxr-xr-x 1 dpchrist dpchrist 494 Apr 28 21:04 SHA512SUMS -rwxr-xr-x 1 dpchrist dpchrist 833 Apr 28 21:04 SHA512SUMS.sign -rwxr-xr-x 1 dpchrist dpchrist 396361728 Apr 28 21:05 debian-11.3.0-amd64-netinst.iso I verified the checksum of the ISO file: 2022-04-29 22:17:17 dpchrist@tinkywinky ~/samba/dpchrist/iso/debian/11.3.0 $ grep debian-11.3.0-amd64-netinst.iso SHA512SUMS | sha512sum -c debian-11.3.0-amd64-netinst.iso: OK I burned the ISO to a zeroed USB flash drive: 2022-04-29 22:39:25 root@tinkywinky ~/hardware/adata/usb-flash-drive/REDACTED # time dd if=/home/dpchrist/samba/dpchrist/iso/debian/11.3.0/debian-11.3.0-amd64-netinst.iso of=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1392303332110024-0\:0 bs=1M iflag=fullblock oflag=sync,noatime status=progress 394264576 bytes (394 MB, 376 MiB) copied, 76.1204 s, 5.2 MB/s 378+0 records in 378+0 records out 396361728 bytes (396 MB, 378 MiB) copied, 76.5701 s, 5.2 MB/s real1m16.582s user0m0.012s sys 0m0.584s I computed the checksum of the relevant blocks of the USB flash drive: 2022-04-29 22:43:56 root@tinkywinky ~/hardware/adata/usb-flash-drive/REDACTED # time dd if=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_REDACTED-0\:0 bs=1M count=378 iflag=fullblock | sha512sum 378+0 records in 378+0 records out 396361728 bytes (396 MB, 378 MiB) copied, 25.0641 s, 15.8 MB/s 2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53 - real0m25.068s user0m3.468s sys 0m0.720s The USB flash drive checksum matched the value stored in SHA512SUMS: 2022-04-29 22:44:58 root@tinkywinky ~/hardware/adata/usb-flash-drive/REDACTED # grep 2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53 /home/dpchrist/samba/dpchrist/iso/debian/11.3.0/SHA512SUMS 2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53 debian-11.3.0-amd64-netinst.iso I have since used the USB flash drive to install Debian onto one or more computers. If I compute the checksum of the relevant blocks of the USB flash drive today: 2024-04-02 14:32:43 root@laalaa ~ # time dd if=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_REDACTED-0\:0 bs=1M count=378 iflag=fullblock | sha512sum 378+0 records in 378+0 records out 6cbb7e54fccdf550cbb5535d7dd9357513b36b767f0aaa550b1d58b37ef827c881036dcf47b3c377d196c4e77d14c786a6dd975aa558a11a81cf9ae107062abc - 396361728 bytes (396 MB, 378 MiB) copied, 27.7592 s, 14.3 MB/s real0m27.766s user0m4.134s sys 0m1.380s The checksum has changed because d-i modified the USB flash drive. I previously confirmed this behavior years ago after booting a d-i USB flash drive only once. You must use read-only media (e.g. CD-R) if you want to be able to verify d-i after use. David
Re: SOLVED
On 4/1/24 11:35, DdB wrote: Am 01.04.2024 um 18:52 schrieb David Christensen: A bad USB flash drive would explain why you cannot boot the Debian installer. Please buy a good quality USB 3.0+ flash drive and try again. A friend of mine just let me use an external CD-Drive with the netboot image. I thought about suggesting that in my last post, but did not want to complicate things. A key advantage of using a CD-R disc is that you can verify the disc contents and/or checksum against the ISO and/or checksum now and in the future. This is not true for a USB flash drive, because the Debian installer modifies the contents of the USB flash drive when it runs. This is already the third time, i am restarting the installation process, due to my false assumptions about the intelligence within the installer. The last time, i was quite happy until i came to notice, that partitions were not aligned with physical sector boundaries, which i assumed would be elementary best practice. I chose manual partitioning and the Debian installer aligned the partitions to 2**20 byte boundaries: 2024-04-02 04:07:16 root@laalaa ~ # cat /etc/debian_version ; uname -a 11.9 Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux 2024-04-02 04:08:18 root@laalaa ~ # fdisk -l /dev/sda Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors Disk model: INTEL SSDSC2CW06 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x544032f5 Device BootStart End Sectors Size Id Type /dev/sda1 *2048 1953791 1951744 953M 83 Linux /dev/sda21953792 3907583 1953792 954M 83 Linux /dev/sda33907584 29298687 25391104 12.1G 83 Linux /dev/sda4 29298688 117229567 87930880 41.9G 83 Linux But apart from losing some of my illusions the hard way, all is well. A big thank you to all the crowd offering suggestions and encouragement. so long, DdB I'm glad you were able to install Debian. :-) David
HDD long-term data storage with ensured integrity
On 3/31/24 02:18, DdB wrote: > i intend to create a huge backup server from some oldish hardware. > Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a > 6core Intel with 64 GB RAM. ... the [Debian] installer ... aborts. On 4/1/24 11:35, DdB wrote: > A friend of mine just let me use an external CD-Drive with the netboot > image. ... all is well. Now you get to solve the same problem I have been stuck on since last November -- how to use those HDD's. ZFS has been my bulk storage solution of choice for the past ~4 years, but the recent data corruption bugs [1, 2] have me worried. From a technical perspective, it's about incorrect concurrent execution of GNU cp(1), Linux, and/or OpenZFS. From a management perspective, it's about Capability Maturity Model (CMM) [3] and Programming Systems Product (PSP) [4]. The most obvious alternative to ZFS on Debian would be Btrfs. Does anyone have any comments or suggestions regarding Btrfs and data corruption bugs, concurrency, CMM level, PSP, etc.? Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? David [1] https://github.com/openzfs/zfs/issues/15526 [2] https://github.com/openzfs/zfs/issues/15933 [3] https://en.wikipedia.org/wiki/Capability_maturity_model [4] https://en.wikipedia.org/wiki/The_Mythical_Man-Month
Re: help needed to get a bookworm install to succeed
On 4/1/24 03:10, DdB wrote: Am 01.04.2024 um 07:44 schrieb David Christensen: Please post a console session that identifies the ISO you are using, verifies the checksum, burns the ISO to a USB flash drive, and compares the ISO against the flash drive. Ok, in the meantime, i came to similar conclusions and found that the USB-stick i was using, had consistent read errors at the first 2 gigabytes after having been used for years as memory extension in my router. Fixed that and will replace the stick. A bad USB flash drive would explain why you cannot boot the Debian installer. Please buy a good quality USB 3.0+ flash drive and try again. David
Re: help needed to get a bookworm install to succeed
On 3/31/24 02:18, DdB wrote: Hello list, i intend to create a huge backup server from some oldish hardware. Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a 6core Intel with 64 GB RAM. Already before assembling the hardware, grub was working from the SSD, which got lvm partitioning and is basically empty. As i have no working CD drive nor can this old machine boot from USB, i put an ISO for bookworm onto an lvm-LV. Using grub, i can manually boot from that ISO and see the first installer screens. But after asking some questions, the installer wants to mount the external media (ISO), and does not find it on sd[a-z], then aborts. By switching to Desktop 4, i can see the attempt to search for the "CD"-drive, which is bound to fail. I am not familiar with the very restricted shell, that is available from the installer (busybox) and have not yet found an approach to circumvent my problems. i would like to use the installer, as debootstrapping would necessitate alot more knowledge than mine. Suggestions are welcome :-) DdB A computer with a 6-core processor, 64 GB memory, and 9 drive bays/ ports that cannot boot USB? That does not make sense. Please post a console session that identifies the ISO you are using, verifies the checksum, burns the ISO to a USB flash drive, and compares the ISO against the flash drive. Then insert the USB flash drive into a USB port on the the target computer, power up and enter Setup, reset the settings to factory defaults, enable USB booting, set the USB flash drive as the first boot device, save, and exit. The Debian installer should then boot. David
Re: Debian 12.5 up-to-date Xfce, Firefox clings to USB stick
On 3/30/24 08:17, Antti-Pekka Känsälä wrote: What could be the deal, when Firefox tries to stop me from unmounting a stick, after I've accessed files on it through Firefox? I worry about my stick security. Thanks. Linux knows what files are open on each file system. If you try to unmount a file system with open files or eject a mounted USB drive with open files, Linux will refuse and your desktop environment will display a suitable error dialog. This is a feature, not a bug. The solution is to close all the files on the file system, and then unmount it. David
Re: Debian 12.5.0 amd64 and OpenZFS bug #15526
On 3/25/24 15:05, Gareth Evans wrote: On Fri 22/03/2024 at 21:01, Gareth Evans wrote: As anyone interested can see from the ref to #15933 in the below, there seems to have been considerable effort in getting to grips with this bug (actually multiple bugs), and it looks like a fix may be forthcoming, though not sure at the time of writing if there may be some further polishing first https://github.com/openzfs/zfs/pull/16019 https://github.com/openzfs/zfs/issues/15933 is now closed as completed with fix https://github.com/openzfs/zfs/commit/102b468b5e190973fbaee6fe682727eb33079811 which for the moment necessarily adds synchronous writes. FYI. Gareth Thank you for keeping an eye on this. Looking at the github commit, the C code makes me worry -- it does not appear to use traditional C/C++ thread-safe programming techniques such as I learned in CS and used when I did systems programming (e.g. guard functions, critical sections, locks, semaphores, etc.). Do I need to look at more enclosing code to see such, are those techniques missing, are there some newer techniques I do not understand, or something else? David
Re: trying to parse lines from an awkwardly formatted HAR file ...
On 3/22/24 22:53, Albretch Mueller wrote: out of a HAR file containing lots of obfuscating js cr@p and all kinds of nonsense I was able to extract line looking like: var00='{\"index\":\"prod-h-006\",\"fields\":{\"identifier\":\"bub_gb_O2EAMAAJ\",\"title\":\"Die Wissenschaft vom subjectiven Geist\",\"creator\":[\"Karl Rosenkranz\", \"Mr. ABC123\"],\"collection\":[\"europeanlibraries\", \"americana\"],\"year\":1843,\"language\":[\"German\"],\"item_size\":797368506},\"_score\":[50.629513]}' echo "// __ \$var00: |$var00|" The final result that I need would look like: o var02='bub_gb_O2EAMAAJ|Die Wissenschaft vom subjectiven Geist|["Karl Rosenkranz", "Mr. ABC123"]|["europeanlibraries", "americana"]|1843|["German"]|797368506|[50.629513]' echo "// __ \$var02: |$var02|" I have tried substring substitution, sed et tr to no avail. lbrtchx My daily driver: 2024-03-23 04:02:27 dpchrist@laalaa ~/sandbox/perl/debian-users/20240322-2253-albretch-mueller $ cat /etc/debian_version; uname -a; perl -v | head -n 2 | grep . 11.9 Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-gnu-thread-multi Put the JSON into a data file, one record per line (my mailer is line-wrapping data.json -- it contains two lines): 2024-03-23 04:22:20 dpchrist@laalaa ~/sandbox/perl/debian-users/20240322-2253-albretch-mueller $ cat data.json {"index":"prod-h-006","fields":{"identifier":"bub_gb_O2EAMAAJ","title":"Die Wissenschaft vom subjectiven Geist","creator":["Karl Rosenkranz", "Mr. ABC123"],"collection":["europeanlibraries", "americana"],"year":1843,"language":["German"],"item_size":797368506},"_score":[50.629513]} {"index":"prod-h-007","fields":{"identifier":"abc_de_12FGHIJKLMNO","title":"My Title","creator":["Some Body", "Somebody Else"],"collection":["europeanlibraries", "americana"],"year":2024,"language":["English"],"item_size":1234567890},"_score":[12.345678]} A Perl script to read newline-delimited JSON records and pretty print each: 2024-03-23 04:28:59 dpchrist@laalaa ~/sandbox/perl/debian-users/20240322-2253-albretch-mueller $ cat munge-json #!/usr/bin/perl # $Id: munge-json,v 1.3 2024/03/23 11:28:58 dpchrist Exp $ # Refer to debian-user 3/22/24 22:53 Albretch Mueller # "trying to parse lines from an awkwardly formatted HAR file" # by David Paul Christensen dpchr...@holgerdanske.com # Public Domain use strict; use warnings; use Data::Dumper; use JSON; use Getopt::Long; $Data::Dumper::Sortkeys = 1; my $debug; GetOptions('debug|d' => \$debug) or die; while (<>) { my $rh = decode_json $_; print Data::Dumper->Dump([$rh], [qw(rh)]) if $debug; print join('|', $rh->{fields}{identifier}, $rh->{fields}{title}, '["' . join('", "', @{$rh->{fields}{creator}}) . '"]', '["' . join('", "', @{$rh->{fields}{collection}}) . '"]', $rh->{fields}{year}, '["' . join('", "', @{$rh->{fields}{language}}) . '"]', $rh->{fields}{item_size}, '[' . join(', ', @{$rh->{_score}}) . ']', ), "\n"; } Run the script as a Unix filter: 2024-03-23 04:30:16 dpchrist@laalaa ~/sandbox/perl/debian-users/20240322-2253-albretch-mueller $ ./munge-json data.json bub_gb_O2EAMAAJ|Die Wissenschaft vom subjectiven Geist|["Karl Rosenkranz", "Mr. ABC123"]|["europeanlibraries", "americana"]|1843|["German"]|797368506|[50.629513] abc_de_12FGHIJKLMNO|My Title|["Some Body", "Somebody Else"]|["europeanlibraries", "americana"]|2024|["English"]|1234567890|[12.345678] 2024-03-23 04:30:18 dpchrist@laalaa ~/sandbox/perl/debian-users/20240322-2253-albretch-mueller $ cat data.json | ./munge-json bub_gb_O2EAMAAJ|Die Wissenschaft vom subjectiven Geist|["Karl Rosenkranz", "Mr. ABC123"]|["europeanlibraries", "americana"]|1843|["German"]|797368506|[50.629513] abc_de_12FGHIJKLMNO|My Title|["Some Body", "Somebody Else"]|["europeanlibraries", "americana"]|2024|["English"]|1234567890|[12.345678] David
Re: electrons/the Internet doesn't like … that I like to eat raw garlic, ...
On 3/4/24 16:06, David Wright wrote: On Mon 04 Mar 2024 at 12:36:54 (-0800), David Christensen wrote: On 3/4/24 08:37, Albretch Mueller wrote: _LINK="https://christuniversity.in/uploads/course/E_21-25_Lateral Entry(1)_20210618043317.pdf" I ignored the filename, and pasted https://christuniversity.in/uploads/course/ into FF. Here's the text copy/pasted off the page that was displayed. It was accompanied by the image that is displayed at https://christuniversity.in/images/cour-btch-bnnr.jpg ... Just a data point. Testing ping again: 2024-03-04 17:31:14 dpchrist@laalaa ~ $ ping -c 3 christuniversity.in PING christuniversity.in (111.93.136.229) 56(84) bytes of data. 64 bytes from 111.93.136.229 (111.93.136.229): icmp_seq=1 ttl=49 time=273 ms 64 bytes from 111.93.136.229 (111.93.136.229): icmp_seq=2 ttl=49 time=272 ms 64 bytes from 111.93.136.229 (111.93.136.229): icmp_seq=3 ttl=49 time=273 ms --- christuniversity.in ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 271.914/272.406/272.725/0.352 ms 2024-03-04 17:31:23 dpchrist@laalaa ~ $ ping -c 3 103.105.225.131 PING 103.105.225.131 (103.105.225.131) 56(84) bytes of data. --- 103.105.225.131 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2060ms 2024-03-04 17:31:41 dpchrist@laalaa ~ $ ping -c 3 111.93.136.229 PING 111.93.136.229 (111.93.136.229) 56(84) bytes of data. 64 bytes from 111.93.136.229: icmp_seq=1 ttl=49 time=277 ms 64 bytes from 111.93.136.229: icmp_seq=2 ttl=49 time=280 ms 64 bytes from 111.93.136.229: icmp_seq=3 ttl=49 time=276 ms --- 111.93.136.229 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 276.118/277.636/280.041/1.719 ms So, same results as before -- one IP works, the other does not, and I got lucky with the FQDN (a previous test got the wrong IPv4 address and timed out). Testing Firefox: https://christuniversity.in/uploads/course/ https://103.105.225.131/uploads/course/ https://111.93.136.229/uploads/course/ All three time out. Doing whois searches on the A record IP addresses: 1. https://www.whois.com/whois/103.105.225.131 The IPv4 address holder appears to be a small ISP with 4 @ IPv4 class C ranges (1,024 addresses). It appears nothing is connected to the christuniversity.in IPv4 address. 2. https://www.whois.com/whois/111.93.136.229 The IPv4 address holder appears to be a larger ISP with 1 @ IPv4 class B range (65,535 addresses). It appears there is a host connected to the christuniversity.in IPv4 address, but I cannot connect to its web server. STFW for information about DNS A (Address) records, I see: https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/ What is a DNS A record? ... The vast majority of websites only have one A record, but it is possible to have several. Some higher profile websites will have several different A records as part of a technique called round robin load balancing, which can distribute request traffic to one of several IP addresses, each hosting identical content. So, two DNS A records for the same FQDN is allowed and can be useful. Searching for all DNS records for christuniversity.in : https://www.whatsmydns.net/dns-lookup?query=christuniversity.in=opendns I see the two A (address) records that we have been discussing: id 21430, opcode QUERY, rcode NOERROR, flags QR RD RA ;QUESTION christuniversity.in. IN A ;ANSWER christuniversity.in. 60 IN A 111.93.136.229 christuniversity.in. 60 IN A 103.105.225.131 ;AUTHORITY ;ADDITIONAL I see five MX (mail exchanger) records: id 52477, opcode QUERY, rcode NOERROR, flags QR RD RA ;QUESTION christuniversity.in. IN MX ;ANSWER christuniversity.in. 60 IN MX 1 aspmx.l.google.com. christuniversity.in. 60 IN MX 5 alt1.aspmx.l.google.com. christuniversity.in. 60 IN MX 5 alt2.aspmx.l.google.com. christuniversity.in. 60 IN MX 10 alt3.aspmx.l.google.com. christuniversity.in. 60 IN MX 10 alt4.aspmx.l.google.com. ;AUTHORITY ;ADDITIONAL I see two NS (nameserver) records: id 57399, opcode QUERY, rcode NOERROR, flags QR RD RA ;QUESTION christuniversity.in. IN NS ;ANSWER christuniversity.in. 60 IN NS ns1.christuniversity.in. christuniversity.in. 60 IN NS ns2.christuniversity.in. ;AUTHORITY ;ADDITIONAL I find it strange that there are no A records for: ns1.christuniversity.in ns2.christuniversity.in And yet dig(1) can find them: 2024-03-04 17:56:59 dpchrist@laalaa ~ $ dig @9.9.9.9 ns1.christuniversity.in ; <<>> DiG 9.16.48-Debian <<>> @9.9.9.9 ns1.christuniversity.in ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40331 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;ns1.christuniversity.in. IN A ;; ANSWER SECTION: ns1.christuniver
Re: resolv.conf (was Re: electrons/the Internet [racism redacted])
On 3/4/24 13:11, Greg Wooledge wrote: On Mon, Mar 04, 2024 at 12:36:54PM -0800, David Christensen wrote: I believe Debian rewrites /etc/resolv.conf on every boot. This is not correct. It's *partly* correct if you ignore a lot of complicating factors. Short version: read <https://wiki.debian.org/resolv.conf>. Long version follows: If you have a static network interface configuration, defined in /etc/network/interfaces, with no DHCP client, no VPN stuff going on, etc. then your /etc/resolv.conf will not be changed. You can edit the file and put whatever you want in it, and it'll remain as you wish it to be. Unfortunately, almost *nobody* has a setup like this any more. In a more typical environment, you get your IP address via DHCP, which means you're running a DHCP client daemon. Most DHCP client daemons will rewrite the /etc/resolv.conf file every time they refresh their DHCP lease. This may indeed happen at boot time, but it'll also happen a couple times a day during normal operations. So, a simple instruction like "edit /etc/resolv.conf" is no longer possible. Even worse, there's no single *alternative* either. You can't even say "do ___ instead". To put the correct values into your /etc/resolv.conf file nowdays, you have to select a *strategy*. You need to find an indirect way to put the right content into some *other* place, in such a way that it will eventually find its way into /etc/resolv.conf every time the file is rewritten. And there are *lots* of strategies that will work, so you can't even say "obviously this one is best". Life is not that simple. Or, you could use chattr +i to make the /etc/resolv.conf file immutable, so DHCP clients and other programs cannot overwrite it. Either way, you take ownership of whatever strategy you decide to use, together with its pros and cons. You'll have to understand that on *this* system, you went with *this* strategy, and remember where to put your changes, and how to make them. Or at the very least, you'll need to be *aware* of all the strategies you've got in play on all of your systems, and know how to identify which one is in use on any given system. Thank you for the clarification. Thankfully, my gateway DHCP server and my Debian instances work together. David
Re: electrons/the Internet doesn't like question authority niggahs?, or is it that I like to eat raw garlic, ...
On 3/4/24 08:37, Albretch Mueller wrote: Yes, networking problems are infuriating. Something that shouldn't be happening at all is that after I use traceroute once, it doesn't work again and my Internet access speed describes like a sinus curve which amplitude remains for the most part under 16KiB per second and for more than one second as 0B per second. _LINK="https://christuniversity.in/uploads/course/E_21-25_Lateral Entry(1)_20210618043317.pdf" I have AT Internet service in Tracy, California. My daily driver is: 2024-03-04 10:47:12 dpchrist@laalaa ~ $ cat /etc/debian_version ; uname -a 11.9 Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux When I click the above link in my mail client (Thunderbird), my browser (Firefox) attempts to open the URL. But, the URL is mangled by mail client line wrap and/or indentation (?), and the connection times out: https://christuniversity.in/uploads/course/E_21-25_Lateral An error occurred during a connection to christuniversity.in. The site could be temporarily unavailable or too busy. Try again in a few moments. If you are unable to load any pages, check your computer’s network connection. If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the web. The following URL's also time out (see DNS comments, below): http://christuniversity.in/ http://103.105.225.131/ http://111.93.136.229/ https://christuniversity.in/ https://103.105.225.131/ https://111.93.136.229/ 1) is the file actually there?: wget -q --spider "${_LINK}"; _WGETQ=$? I refrain from spidering web sites -- being blackholed is not good. $ ls -l /etc/resolv.conf -rw-r--r-- 1 root root 204 Mar 3 18:59 /etc/resolv.conf 2024-03-04 10:50:49 dpchrist@laalaa ~ $ ls -l /etc/resolv.conf -rw-r--r-- 1 root root 83 Mar 4 09:50 /etc/resolv.conf $ cat /etc/resolv.conf # Generated by NetworkManager # nameserver 192.168.1.254 # nameserver 192.168.68.1 # https://serverfault.com/questions/76421/wget-cant-resolve-host # RED 2013-03-31 nameserver 8.8.8.8 nameserver 8.8.4.4 2024-03-04 10:51:32 dpchrist@laalaa ~ $ cat /etc/resolv.conf # Generated by NetworkManager search tracy.holgerdanske.com nameserver 192.168.5.1 I believe Debian rewrites /etc/resolv.conf on every boot. Hard coding Google Public DNS servers should work, but letting your gateway do it for your LAN is easier to manage, is faster, and conserves WAN bandwidth. I would revert your changes. And, Google is watching you. STFW for DNS privacy: https://avoidthehack.com/best-dns-privacy I think I will configure my gateway to use Quad9: https://www.quad9.net/service/locations/ The next level up would be DNS over TLS (DoT), DNS over HTTPS (DoH), DNSCrypt, etc.: https://en.wikipedia.org/wiki/DNS_over_TLS $ ls -l /etc/nsswitch.conf -rw-r--r-- 1 root root 613 Mar 3 12:57 /etc/nsswitch.conf 2024-03-04 10:52:04 dpchrist@laalaa ~ $ ls -l /etc/nsswitch.conf -rw-r--r-- 1 root root 542 Jan 9 2022 /etc/nsswitch.conf $ cat /etc/nsswitch.conf # /etc/nsswitch.conf # # Example configuration of GNU Name Service Switch functionality. # If you have the `glibc-doc-reference' and `info' packages installed, try: # `info libc "Name Service Switch"' for information about this file. passwd: files group: files shadow: files gshadow:files #hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname hosts: files dns mdns4_minimal [NOTFOUND=return] dns myhostname networks: files protocols: db files services: db files ethers: db files rpc:db files netgroup: nis 2024-03-04 10:56:37 dpchrist@laalaa ~ $ cat /etc/nsswitch.conf # /etc/nsswitch.conf # # Example configuration of GNU Name Service Switch functionality. # If you have the `glibc-doc-reference' and `info' packages installed, try: # `info libc "Name Service Switch"' for information about this file. passwd: files systemd group: files systemd shadow: files gshadow:files hosts: files mdns4_minimal [NOTFOUND=return] dns networks: files protocols: db files services: db files ethers: db files rpc:db files netgroup: nis It appears my /etc/nsswitch.conf has not been touched since installation. I would revert your changes. ; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> +time christuniversity.in ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49715 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;christuniversity.in. IN A ;; ANSWER SECTION: christuniversity.in.60 IN A 103.105.225.131 christuniversity.in.60 IN A 111.93.136.229 ;; Query time: 327 msec ;; SERVER:
Re: Debian 12.5.0 amd64 and OpenZFS bug #15526
On 2/26/24 20:52, Gareth Evans wrote: Replied to OP by mistake, reposting to list. On Sun 25/02/2024 at 05:34, David Christensen wrote: debian-user: Is Debian 12.5.0 amd64 affected by OpenZFS bug #15526? https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.5.0-amd64-netinst.iso https://packages.debian.org/bookworm/zfs-dkms https://github.com/openzfs/zfs/issues/15526 Hi David, Given the complexity of the issues, I'm not sure if this truly answers your question, but https://github.com/openzfs/zfs/issues/15933 seems to suggest that or a similar issue is still ongoing with Open ZFS 2.2.3, which is later than the version currently available from bookworm or bookworm-backports. It seems bookworm-backports might eventually provide the solution, if at all, per the Debian wiki on ZFS: "it is recommended by Debian ZFS on Linux Team to install ZFS related packages from Backports archive. Upstream stable patches will be tracked and compatibility is always maintained." https://wiki.debian.org/ZFS Currently: $ apt policy zfs-dkms zfs-dkms: Installed: 2.2.2-4~bpo12+1 Candidate: 2.2.2-4~bpo12+1 Version table: *** 2.2.2-4~bpo12+1 100 100 https://deb.debian.org/debian bookworm-backports/contrib amd64 Packages 100 https://deb.debian.org/debian bookworm-backports/contrib i386 Packages 100 /var/lib/dpkg/status 2.1.11-1 500 500 https://deb.debian.org/debian bookworm/contrib amd64 Packages 500 https://deb.debian.org/debian bookworm/contrib i386 Packages Hope that helps. Gareth That you for citing OpenZFS bug #15933. These appear to be the ZFS packages for the available Debian releases: https://packages.debian.org/buster/zfs-dkms buster zfs-dkms (0.7.12-2+deb10u2) buster-backportszfs-dkms (2.0.3-9~bpo10+1) bullseyezfs-dkms (2.0.3-9+deb11u1) bullseye-backports zfs-dkms (2.1.11-1~bpo11+1) bookwormzfs-dkms (2.1.11-1) bookworm-backports zfs-dkms (2.2.2-4~bpo12+1) trixie zfs-dkms (2.2.2-4) The question is, how far back to go? Is OpenZFS 2.1.x buggy? OpenZFS 2.0.x? What is 0.7.12 -- OpenZFS, ZFS-on-Linux, or something else -- and is it buggy? David
Debian 12.5.0 amd64 and OpenZFS bug #15526
debian-user: Is Debian 12.5.0 amd64 affected by OpenZFS bug #15526? https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.5.0-amd64-netinst.iso https://packages.debian.org/bookworm/zfs-dkms https://github.com/openzfs/zfs/issues/15526 David
Re: Orphaned Inode Problem
On 2/21/24 03:00, Jörg-Volker Peetz wrote: Hi, did you take a look at the smartctl output? Somewhere I read, for maintainance of an SSD all it's cells should be read from time to time like this sudo dd if=/dev/DEVICE of=/dev/null bs=8M status=progress where device is something like sda or nvme0n1, especially if it was switched off for a longer period. At least, it shows the current read performance of the device. An SSD should regularly be trimmed, if in use. This is to assist it's wear leveling process. What's your opinion? Regards, Jörg. I prefer to run a SMART long test periodically. This should read every cell, including those that are reserved and not visible to the OS. AIUI So long as the SSD can maintain a supply of erased cells via manufacturer over-provisioning, trim is not required to maintain performance. If you have workload that does a lot of writes in a short period of time and exhausts the manufacturer over-provisioning, leaving free space on the SSD and trimming can be a work-around. If you are using strong encryption, not trimming will leave crypttext on disk that creates more work for an attacker. If you are using weak encryption, not trimming will leave crypttext on disk that an attacker can recover. For imaging/ cloning, trimming will zero blocks freed by the OS and facilitate compression of the image file. I have a SOHO network with about two dozen disks. Running smartctl by hand is a PITA. Running fstrim(8) by hand is easy enough. I try to do both once a month. I need to figure out smartd(8). David
Re: HDD error: Current_Pending_Sector
On 2/20/24 09:51, Default User wrote: Hi guys! I am running Debian 12 Stable, up to date, on a low-spec Dell Inspiron 15 3000 Model 3511. Firmware is also up to date. I have a 4 Gb Western Digital external usb SATA HDD, Model WDC WD40NDZW-11A8JS1. It has only one partition, formatted as ext4. The filesystem is labeled MSD00012. Every night, I use rsync to copy all contents of a (theoretically) identical drive, which has filesystem label MSD00014, to the drive with MSD00012. Two nights ago, I could not do the copy correctly. Apparently, as a safety measure, MSD00012 was automatically re-mounted as read only, due to a filesystem error. I used the gnome-disks utility to unmount and then remount it. It was remounted as read-write. Now it "works", BUT . . . I ran: sudo smartctl --test=long /dev/sdb on it, and it reports a Current_Pending_Sector error, at LBA 325904690. From sudo smartctl --all /dev/sdb > backup_drive_b_test.txt: smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-18-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Elements / My Passport (USB, AF) Device Model: WDC WD40NDZW-11A8JS1 Serial Number:WD- LU WWN Device Id: 5 0014ee 269112168 Firmware Version: 01.01A01 User Capacity:4,000,753,475,584 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate:5400 rpm Form Factor: 2.5 inches TRIM Command: Available, deterministic Device is:In smartctl database 7.3/5319 ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is:Tue Feb 20 11:32:04 2024 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection:(12240) seconds. Offline data collection capabilities:(0x1b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. No Selective Self-test supported. SMART capabilities:(0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability:(0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time:( 2) minutes. Extended self-test routine recommended polling time:( 24) minutes. SCT capabilities: (0x30b5) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 198 051Pre-fail Always - 41 3 Spin_Up_Time0x0027 253 253 021Pre-fail Always - 4741 4 Start_Stop_Count0x0032 099 099 000Old_age Always - 1175 5 Reallocated_Sector_Ct 0x0033 200 200 140Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000Old_age Always - 1311 10 Spin_Retry_Count0x0032 100 100 000Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000Old_age Always - 693 192 Power-Off_Retract_Count 0x0032 200 200 000Old_age Always - 23 193 Load_Cycle_Count0x0032 199 199 000Old_age Always - 3045 194 Temperature_Celsius 0x0022 114 102 000Old_age
Re: red SATA cables "notoriously bad"?
On 2/19/24 18:07, Felix Miata wrote: My experience with that particular color cables matches Gene's. Cut one open, and out comes a powdery substance instead of clean copper strands. I think most for gen 1.0 SATA 2 decades ago, so there shouldn't be many still around bogging down 3.0 drives. About 10 (?) years ago, I seem to recall trouble-shooting a SATA connection problem and coming to the conclusion that the (red) SATA cable was the problem. I cannot recall if I had heard Gene's story at the time. I believe I decided to cut off one end, taking a 50% chance of getting something I could use as a break-out/ pig tail. To my surprise, there was no copper within the cable, just brownish dust! Unfortunately, I did not photograph the cable and it is long gone. 4 or more years ago, I was plagued with SATA III connection issues; likely due to old SATA I and SATA II cables and mobile racks. I bought a bunch of black SATA cables marked "6 Gbps" with locking connectors and got rid of all of my existing cables (most of which were red). I later retired all of my SATA I and SATA II mobile racks, moved most of my drives internal, and bought a few SATA III mobile racks for off-site backup drives. My SATA connection problems are finally resolved. David
Re: partition reporting full, but not
On 2/18/24 19:20, Keith Bainbridge wrote: I am convinced that the missing space is used by btrfs snapshot process. Perhaps. But, are you re-balancing your btrfs file systems regularly? https://manpages.debian.org/bookworm/btrfs-progs/btrfs-balance.8.en.html Doing it by hand was not practical for me. I wrote a Perl script to automate the process. On SSD's, the results were decent. On USB flash drives, not so much. Searching for a power tool today, I see: 2024-02-18 23:27:43 dpchrist@laalaa ~/stretch-amd64 $ apt-cache search btrfs | grep mainten btrfsmaintenance - automate btrfs maintenance tasks on mountpoints or directories https://packages.debian.org/bookworm/btrfsmaintenance I suggest installing and trying the btrfsmaintenance package. David
Re: partition reporting full, but not
Keith Bainbridge composed on 2024-02-17 15:44 (UTC+1100): Yes the / partitions are btrfs Several years ago, I installed Debian (9?) using btrfs for root (and boot?). I failed to understand that btrfs required regular maintenance and/or I was too lazy to figure it out and do it. After a few months, the systems started running slowly and I seem to recall conflicting reports of storage usage. I STFW, RTFM, etc., and tried doing the maintenance by hand. I quickly came to the conclusion that I needed to run `btrfs balance start ...` many, many times. So, I wrote Perl script and let it hammer on the file systems for hours at a time (!). I was able to rescue most of the disks to decent performance, but one was especially bad and I was only able to rescue it to marginal performance. I continued running btrfs for a while and running the Perl script periodically. Ultimatedly, I backed up, put in fresh OS disks, and installed using ext4. This was one of my reasons to use FreeBSD and ZFS for my SOHO servers. David
Re: f3tools vs Silicon Power 4T drive
On 2/16/24 12:46, Stefan Monnier wrote: One of the 1T samsungs in the md raid10 isn't entirely happy but mdadm has not fussed about it, and smartctl seems to say its ok after testing. Other than that the gui access delay (30+ seconds) problems I have did NOT go away when I moved /home off the raid to another SSD, so I may move it back. One of the reasons I ma rsync'ing this /home back to it every other day or so, takes < 5 minutes. Please get a small SSD, do a fresh install, and test for the access delay. If the delay is not present, incrementally add and test applications. If you encounter the delay, please stop and post the details; console sessions are best. If not, then connect the disks with /home and test. If you encounter the delay, then please stop and post the details. If you do not encounter the delay, then your system is fixed. Take a Clonezilla image. FWIW, my crystal ball says "30s => software timeout rather than hardware problem" +1 David
Re: f3tools vs Silicon Power 4T drive
On 2/15/24 22:16, gene heskett wrote: I want to know with absolute certainty, with of the 4 drives in that raid10, actually has a belly ache. When it has a belly ache. I can't see any reason on this ball of rock and water, why I should be expected to replace a drive at a time until the belly ache goes away. I seem to recall the Samsung 1 TB SSD's in your /home RAID10 were worn out. I suggest installing the 2 TB M.2 WD Black, partitioning it with GPT, creating one large partition, mounting it at /data, and copying all of the data from /home to /data before the SSD's and RAID fail completely. I recently had an Intel SSD 520 Series 180 GB go from operational to toast, with nothing in between. If that happens to one of those Samsung 1 TB SSD's, there will be no way for the RAID10 to correct the bad blocks on the other other SSD. You will corrupt and lose data. I leave /home on my root partition. My working directories are in CVS. The only ephemeral data is in $HOME/.thunderbird. I have a mail filter that copies incoming mail to a second folder on the IMAP server. I Bcc outgoing mail to another mail account. If my OS disk dies, I restore the image from last month, update Debian, check out my work, reconnect Thunderbird to the various e-mail servers, and clean up the Thunderbird folders as required. No data is lost. David
Re: f3tools vs Silicon Power 4T drive
On 2/15/24 17:44, gene heskett wrote: One of the 1T samsungs in the md raid10 isn't entirely happy but mdadm has not fussed about it, and smartctl seems to say its ok after testing. Other than that the gui access delay (30+ seconds) problems I have did NOT go away when I moved /home off the raid to another SSD, so I may move it back. One of the reasons I ma rsync'ing this /home back to it every other day or so, takes < 5 minutes. Please get a small SSD, do a fresh install, and test for the access delay. If the delay is not present, incrementally add and test applications. If you encounter the delay, please stop and post the details; console sessions are best. If not, then connect the disks with /home and test. If you encounter the delay, then please stop and post the details. If you do not encounter the delay, then your system is fixed. Take a Clonezilla image. David
Re: f3tools vs Silicon Power 4T drive
On 2/15/24 12:59, gene heskett wrote: ... gigastones, I 5 of them but when all are plugged in there are only 3 becauae there are 2 pairs of matching serial numbers ... I recall 2 pairs of SSD's with matching serial numbers. Please remove one SSD of each pair so that the remaining SSD's all have unique serial numbers. Return them for a refund while you still can. If you cannot, put them in another computer or put them on the shelf as spares. David
Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?
On 2/16/24 10:56, Roy J. Tellason, Sr. wrote: On Friday 16 February 2024 04:52:22 am David Christensen wrote: I think the Raspberry Pi, etc., users on this list live with USB storage and have found it to be reliable enough for personal and SOHO network use. I have one, haven't done much with it. Are there any alternative ways to interface storage? Maybe add SATA ports or something? In general, there are many combinations of storage interfaces offered in the marketplace. For your specific single-board computer (SBC), I suggest checking the manual and checking the manufacturer sales and/or support web pages. David
Re: f3tools vs Silicon Power 4T drive
On 2/15/24 12:19, gene heskett wrote: On 2/15/24 11:21, Andy Smith wrote: ... redundancy plans ... Like which version of a raid is the best at tolerating a failed drive, which give he best balance between redundancy and capacity. Given a small number of disks, N (say, 4 to 8), the obvious choices are RAID5, RAID6, and RAID10. Regarding redundancy: * RAID5 can tolerate the loss of any one disk. * RAID6 can tolerate the loss of any two disks. * RAID10 can tolerate the loss of any one disk. If you get lucky, RAID10 can tolerate the loss of multiple disks if each lost disk is in a different mirror. Regarding capacity, if each disk stores B bytes: * RAID5 gives you (N-1) * B capacity. * RAID6 gives you (N-2) * B capacity. * RAID10 gives you (N/2) * B capacity. If each disk has performance P: * RAID5 has performance ranging from P to (N-1) * P. * RAID6 has performance ranging from P to (N-2) * P. * RAID10 with M mirrors of D disks each has write performance M * P and read performance M * D * P. Other factors to consider: * All of the above needs to be reconsidered when one or more disks fail -- e.g. the array is operating in degraded mode. * All of the above needs to be reconsidered when a failed disk has been replaced -- e.g. the array is resilvering. * All of the above needs to be reconsidered when disk(s) fail during resilvering (!). * RAID5 and RAID6 typically do not allow changes to topology -- e.g. the number of disks in the array and the number of bytes used in each disk. * RAID0, RAID1, and JBOD may allow some changes to topology. What is allowed depends upon implementation. * With more disks, you may be able to create hierarchies -- e.g. stripe of mirrors (RAID10). Redundancy, capacity, and/or performance under operational, degraded, resilvering, etc., modes all need to be reconsidered. * Hot spares can be added. Again, reconsider everything. * And more. So, it's a multi-dimensional problem and there are many combinations and permutations. The more disks you have, the more possibilities you have. I suggest picking two or three, and exploring them using a dedicated computer, a snapshot of your data, and your workload. I am currently using ZFS and a stripe of 2 mirrors with 2 @ 3 TB HDD's each and SSD read cache. I expect the same could be implemented with mdadm(8), lvm(8), bcache, dm-cache, btrfs, and others. David
Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?
On 2/15/24 07:41, The Wanderer wrote: On 2024-02-15 at 03:09, David Christensen wrote: On 2/14/24 18:54, The Wanderer wrote: On 2024-01-09 at 14:22, The Wanderer wrote: On 2024-01-09 at 14:01, Michael Kjörling wrote: On 9 Jan 2024 13:25 -0500, from The Wanderer I've ordered a 22TB external drive Make? Model? How it is interfaced to your computer? It's a WD Elements 20TB drive (I'm not sure where I got the 22 from); the back of the case has the part number WDBWLG0200HBK-X8 (or possibly -XB, the font is kind of ambiguous). The connection, per the packaging label, is USB-3. Okay. STFW it seems that drive uses CMR, which is good: https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/ The big change of plans in the middle of my month-plus process was the decision to replace the entire 8-drive array with a 6-drive array, and the reason for that was because the 8-drive array left me with no open SATA ports to be able to connect spare drives in order to do drive replacements without needing to rebuild the whole shaboozle. Having spare drive bays for RAID drive replacement is smart. If you have a processor, memory, PCIe slot, and HBA to match those SSD's, the performance of those SSD's should be very nice. The CPU is a Ryxen 5 5600X. The RAM is G-Skill DDR4 2666MHz, in two 32GB DIMMs. I don't know how to assess PCIe slots and HBA, but the motherboard is an Asus ROG Crosshair VIII Dark Hero, which I think was the top-of-the-line enthusiast motherboard (with the port set my criteria called for) the year I built this machine. I'm pretty sure my performance bottleneck for most things is the CPU (or the GPU, where that comes into play, which here it doesn't); storage-wise this seems so far to be at least as fast as what I had before, but it's hard to tell if it's faster. It would not surprise me if the Intel D3-S4510 server drives are somewhat slower than the Samsung EVO 870 desktop drives. But the Intel disks are designed to pull a heavy load all day for years on end. Do you have a tool to monitor disk throughput and utilization? I use Xfce panel Disk Performance Monitor applets and nmon(1) in a Terminal. Those plus CPU and memory monitoring tools should allow you to determine if your workload is CPU bound, memory bound, or I/O bound. The key concept is "data lifetime". (Or alternatively, "destruction policy".) I can see that for when you have a tiered backup structure, and are looking at the lifetimes of each backup copy. For my live system, my intended data lifetime (outside of caches and data kept in /tmp) is basically "forever". I try to group my data in anticipation of backup, etc., requirements. When I get it right, disaster preparedness and disaster recovery are easier. I believe ZFS can do more hard links. (Much more? Limited by available storage space?) I'm not sure, but I'll have to look into that, when I get to the point of trying to set up that tiered backup. ... ... without [rsnapshot hard link] deduplication there wouldn't have been enough space on the drive for more than the single copy, ... ZFS provides similarly useful results with built-in compression and de-duplication. I have the impression that there are risk and/or complexity aspects to it ... Of course. ZFS is sophisticated storage technology. It looks deceptively simple when you are window shopping, but becomes non-trivial once you put real data on it, have to live with it 24x7, have to prepare for disasters, and have to recover from disasters. There is a lot to learn and "more than enough rope to shoot yourself in the foot". That sounds like an N-way merge problem ... It does sound like that, yes. I'm already aware of jdupes, and of a few other tools (part of the work I already did in getting this far was rdfind, which is what I used to set up much of the hardlink deduplication that wound up biting me in the butt), but have not investigated LVM snapshot - and the idea of trying to script something like this, without an existing known-safe copy of the data to fall back on, leaves me *very* nervous. Figuring out how to be prepared to roll back is the other uncertain and nervous-making part. In some cases it's straightforward enough, but doing it at the scale of the size of those copies is at best daunting. https://html.duckduckgo.com/html?q=lvm%20snapshot%20restore Use another computer or a VM to learn and practice LVM snapshots and restores, then use those skills when doing the N-way merge. out of the way, shutting down all parts of the system that might be writing to the affected filesystems, and manually copying out the final state of the *other* parts of those filesystems via rsync, bypassing rsnapshot. That was on Saturday the 10th. Then I grabbed copies of various metadata about the filesystems, the LVM, and the mdraid config; modified /etc/fstab to not mount them; deactivated the md
Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?
On 2/14/24 18:54, The Wanderer wrote: TL;DR: It worked! I'm back up and running, with what appears to be all my data safely recovered from the failing storage stack! That is good to hear. :-) On 2024-01-09 at 14:22, The Wanderer wrote: On 2024-01-09 at 14:01, Michael Kjörling wrote: On 9 Jan 2024 13:25 -0500, from wande...@fastmail.fm (The Wanderer): I've ordered a 22TB external drive Make? Model? How it is interfaced to your computer? for the purpose of creating such a backup. Fingers crossed that things last long enough for it to get here and get the backup created. I suggest selecting, installing and configuring (as much as possible) whatever software you will use to actually perform the backup while you wait for the drive to arrive. It might save you a little time later. Opinions differ but I like rsnapshot myself; it's really just a front-end for rsync, so the copy is simply files, making partial or full restoration easy without any special tools. My intention was to shut down everything that normally runs, log out as the user who normally runs it, log in as root (whose home directory, like the main installed system, is on a different RAID array with different backing drives), and use rsync from that point. My understanding is that in that arrangement, the only thing accessing the RAID-6 array should be the rsync process itself. For additional clarity: the RAID-6 array is backing a pair of logical volumes, which are backing the /home and /opt partitions. The entire rest of the system is on a series of other logical volumes which are backed by a RAID-1 array, which is based on entirely different drives (different model, different form factor, different capacity, I think even different connection technology) and which has not seen any warnings arise. dmesg does have what appears to be an error entry for each of the events reported in the alert mails, correlated with the devices in question. I can provide a sample of one of those, if desired. As long as the drive is being honest about failures and is reporting failures rapidly, the RAID array can do its work. What you absolutely don't want to see is I/O errors relating to the RAID array device (for example, with mdraid, /dev/md*), because that would presumably mean that the redundancy was insufficient to correct for the failure. If that happens, you are falling off a proverbial cliff. Yeah, *that* would be indicative of current catastrophic failure. I have not seen any messages related to the RAID array itself. In the time since this, I continued mostly-normal but somewhat-curtailed use of the system, and saw few messages about these matters that did not arise from attempts to back up the data for later recovery purposes. Migrating large amounts of data from one storage configuration to another storage configuration is non-trivial. Anticipating problems and preparing for them ahead of time (e.g. backups) makes it even less trivial. The last time I lost data was during a migration when I had barely enough hardware. I made a conscious decision to always have a surplus of hardware. (For awareness: this is all a source of considerable psychological stress to me, to an extent that is leaving me on the edge of physically ill, and I am managing to remain on the good side of that line only by minimizing my mental engagement with the issue as much as possible. I am currently able to read and respond to these mails without pressing that line, but that may change at any moment, and if so I will stop replying without notice until things change again.) This need to stop reading wound up happening almost immediately after I sent the message to which I am replying. I remember reading your comment and then noticing you went silent. I apologize if I pushed your button. I now, however, have good news to report back: after more than a month, at least one change of plans, nearly $2200 in replacement hard drives, Ouch. If you have a processor, memory, PCIe slot, and HBA to match those SSD's, the performance of those SSD's should be very nice. much nervous stress, several days of running data copies to and from a 20+-terabyte mechanical hard drive over USB, and a complete manual removal of my old 8-drive RAID-6 array and build of a new 6-drive RAID-6 array (and of the LVM structure on top of it), I now appear to have complete success. I am now running on a restored copy of the data on the affected partitions, taken from a nearly-fully-shut-down system state, which is sitting on a new RAID-6 array built on what I understand to be data-center-class SSDs (which should, therefore, be more suitable to the 24/7-uptime read-mostly workload I expect of my storage). The current filesystems involved are roughly the same size as the ones previously in use, but the underlying drives are nearly 2x the size; I decided to leave the extra capacity for later allocation via LVM, if and when I may need it. When I was
Re: f3tools vs Silicon Power 4T drive
On 2/14/24 18:06, gene heskett wrote: Will the by-id string fit in the space reserved for a label?That IF there was a connection between the /dev/sdc that udev assigns and anything in this list: root@coyote:~# ls /dev/disk/by-id ata-ATAPI_iHAS424_B_3524253_327133504865 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W-part1 wwn-0x5002538f413394a5 ata-Gigastone_SSD_GST02TBG221146 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W-part2 wwn-0x5002538f413394a5-part1 ata-Gigastone_SSD_GST02TBG221146-part1 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W-part3 wwn-0x5002538f413394a5-part2 ata-Gigastone_SSD_GSTD02TB230102 ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V wwn-0x5002538f413394a5-part3 ata-Gigastone_SSD_GSTD02TB230102-part1 ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V-part1 wwn-0x5002538f413394a9 ata-Gigastone_SSD_GSTG02TB230206 ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V-part2 wwn-0x5002538f413394a9-part1 ata-Gigastone_SSD_GSTG02TB230206-part1 ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V-part3 wwn-0x5002538f413394a9-part2 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T ata-SPCC_Solid_State_Disk_AA231107S304KG00080 wwn-0x5002538f413394a9-part3 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T-part1 ata-SPCC_Solid_State_Disk_AA231107S304KG00080-part1 wwn-0x5002538f413394ae ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T-part2 md-name-coyote:0 wwn-0x5002538f413394ae-part1 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T-part3 md-name-coyote:0-part1 wwn-0x5002538f413394ae-part2 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E md-name-coyote:2 wwn-0x5002538f413394ae-part3 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E-part1 md-name-_none_:1 wwn-0x5002538f413394b0 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E-part2 md-uuid-3d5a3621:c0e32c8a:e3f7ebb3:318edbfb wwn-0x5002538f413394b0-part1 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E-part3 md-uuid-3d5a3621:c0e32c8a:e3f7ebb3:318edbfb-part1 wwn-0x5002538f413394b0-part2 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V md-uuid-57a88605:27f5a773:5be347c1:7c5e7342 wwn-0x5002538f413394b0-part3 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V-part1 md-uuid-bb6e03ce:19d290c8:5171004f:0127a392 wwn-0x5002538f42205e8e ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V-part2 usb-SPCC_Sol_id_State_Disk_1234567897E6-0:0 wwn-0x5002538f42205e8e-part1 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V-part3 usb-SPCC_Sol_id_State_Disk_1234567897E6-0:0-part1 wwn-0x5002538f42205e8e-part2 ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W usb-USB_Mass_Storage_Device_816820130806-0:0 wwn-0x5002538f42205e8e-part3 root@coyote:~# I dare you to find the disk that udev calls sdc in the above wall of text. Why can't you understand that I want a unique label for all of this stuff that is NOT a wall of HEX numbers no one can remember. Its not mounted, so blkid does NOT see it. For labeled disk partitions, use /dev/disk/by-label/* paths: 2024-02-14 18:22:34 root@taz ~ # ls -1 /dev/disk/by-label/ sda3_crypt taz_boot taz_root David
Re: f3tools vs Silicon Power 4T drive
On 2/14/24 17:48, gene heskett wrote: On 2/14/24 19:48, Andy Smith wrote: On Wed, Feb 14, 2024 at 05:09:02PM -0500, gene heskett wrote: I have made 1 full partiton om each one, a labeled those partitions as SiPwr_0 and SiPwr_1 Please show us the command you used¹ to do that, so we know what exactly you are talking about, because as previously discussed there's a lot of different things that you like to call "partition labels". This is what gparted calls a "partition label" and certainly does not need a 4.5 megabyte camera image to see. or even a 50k screen snap. Taking this screenshot was a pita, because the gparted window disappears behind the terminal screen when you click on take another shot, so you have to quit, then find the gparted on the tool bar to bring it back to the front, then move it and the terminal so its not totally hidden. Then rerun spectacle again waste a click bringing it fwd, then 30 seconds later the spectacal instructions finally show up and after 5 minutes of screwing around, finally get the screen shot attached to prove I'm not lieing. The easy and accurate answer is to use a root console, fdisk(8) with --list-details, select the console session, and paste into a mail reply: 2024-02-14 18:09:26 root@taz ~ # fdisk --list-details /dev/sda Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors Disk model: INTEL SSDSC2CW06 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 816CF78F-AFAD-4F70-AAA0-B08C6CE95AE7 First LBA: 34 Last LBA: 117231374 Alternative LBA: 117231407 Partition entries LBA: 2 Allocated partition entries: 128 DeviceStart End Sectors Type-UUID UUID Name Attrs /dev/sda1 2048 1953791 1951744 C12A7328-F81F-11D2-BA4B-00A0C93EC93B 5A1358F4-23A2-4CF6-A4E2-0A30A0FFC904 ESP /dev/sda2 1953792 3907583 1953792 0FC63DAF-8483-4772-8E79-3D69D8477DE4 B429D984-E32D-4BAE-A7AE-137168B0F0F3 taz_boot /dev/sda3 3907584 5861375 1953792 0FC63DAF-8483-4772-8E79-3D69D8477DE4 83862E6A-7B89-4AB9-A21D-BAEF3AD0F7A3 taz_swap_crypt /dev/sda4 5861376 29298687 23437312 0FC63DAF-8483-4772-8E79-3D69D8477DE4 2A708FD7-F6EE-49D7-8E23-65905BCD6512 taz_root_crypt /dev/sda5 29298688 117229567 87930880 0FC63DAF-8483-4772-8E79-3D69D8477DE4 B8468EA2-B66D-4D13-9FD1-E46AEDA58067 taz_scratch_crypt David
Re: shred bug? [was: Unidentified subject!]
On 2/13/24 09:40, debian-u...@howorth.org.uk wrote: Greg Wooledge wrote: Shred will determine the size of the file, then write data to the file, rewind, write data again, etc. On a traditional hard drive, that will overwrite the original private information. On modern devices, it may not. Thanks for the excellent explanation :) One nitpick. You say "On a traditional hard drive, that will overwrite the original private information" but that's not quite true. It also needs to be a "traditional" file system! That is, not journalled or COW. So nowadays I would expect shred not to work unless you got very lucky, or planned carefully. Perhaps zerofree(8)? David
Re: shred bug? [was: Unidentified subject!]
On 2/13/24 11:31, gene heskett wrote: Next experiment is a pair of 4T Silicon Power SSD's When they & the startech usb3 adapters arrive. I'll get that NAS built for amanda yet. 2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = $198.96 each set? https://www.amazon.com/dp/B0BVLRFFWQ https://www.amazon.com/dp/B00HJZJI84 Why not external USB drives for $192.99? https://www.amazon.com/dp/B0C6XVZS4K For $7 more, you can get the "Pro edition" in black with two cables: https://www.amazon.com/dp/B0C69QD5NK You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus PRIME Z370-A II motherboard. David
Re: Fast Random Data Generation (Was: Re: Unidentified subject!)
On 2/12/24 08:30, Linux-Fan wrote: David Christensen writes: On 2/11/24 02:26, Linux-Fan wrote: I wrote a program to automatically generate random bytes in multiple threads: https://masysma.net/32/big4.xhtml What algorithm did you implement? I copied the algorithm from here: https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml That Java code uses locks, which implies it uses global state and cannot be run multi-threaded (?). (E.g. one process with one JVM.) Is it possible to obtain parallel operation on an SMP machine with multiple virtual processors? (Other than multiple OS processes with one PRNG on one JVM each?) I found it during the development of another application where I needed a lot of random data for simulation purposes :) My implementation code is here: https://github.com/m7a/bo-big/blob/master/latest/Big4.java If I were to do it again today, I'd probably switch to any of these PRNGS: * https://burtleburtle.net/bob/rand/smallprng.html * https://www.pcg-random.org/ Hard core. I'll let the experts figure it out; and then I will use their libraries and programs. David
Re: shred bug? [was: Unidentified subject!]
On 2/12/24 08:50, Curt wrote: On 2024-02-11, wrote: On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote: [...] If FILE is -, shred standard output. =20 In every sentence, the word FILE appears. There's nothing in there which says "you can operate on a non-file". Point taken, yes. I thought everything was a file. "Everything is a file" is a design feature of the Unix operating system: https://en.wikipedia.org/wiki/Everything_is_a_file But, there is more than one kind of file. And, not every program supports every kind of file. The manual page for find(1) provides a shopping list of file types it supports: 2024-02-12 12:32:13 dpchrist@laalaa ~ $ man find | egrep -A 20 '^ .type c' -type c File is of type c: b block (buffered) special c character (unbuffered) special d directory p named pipe (FIFO) f regular file l symbolic link; this is never true if the -L option or the -follow option is in ef- fect, unless the symbolic link is broken. If you want to search for symbolic links when -L is in effect, use -xtype. s socket As for shred(1), the argument FILE is conventionally a regular file. We are discussing the special case described in the manual page: If FILE is -, shred standard output. David
Re: Fast Random Data Generation (Was: Re: Unidentified subject!)
On 2/11/24 02:26, Linux-Fan wrote: I wrote a program to automatically generate random bytes in multiple threads: https://masysma.net/32/big4.xhtml Before knowing about `fio` this way my way to benchmark SSDs :) Example: | $ big4 -b /dev/null 100 GiB | Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma. | For further info send an e-mail to ma_sys...@web.de. || 0.00% +0 MiB 0 MiB/s 0/102400 MiB | 3.48% +3562 MiB 3255 MiB/s 3562/102400 MiB | 11.06% +7764 MiB 5407 MiB/s 11329/102400 MiB | 19.31% +8436 MiB 6387 MiB/s 19768/102400 MiB | 27.71% +8605 MiB 6928 MiB/s 28378/102400 MiB | 35.16% +7616 MiB 7062 MiB/s 35999/102400 MiB | 42.58% +7595 MiB 7150 MiB/s 43598/102400 MiB | 50.12% +7720 MiB 7230 MiB/s 51321/102400 MiB | 58.57% +8648 MiB 7405 MiB/s 59975/102400 MiB | 66.96% +8588 MiB 7535 MiB/s 68569/102400 MiB | 75.11% +8343 MiB 7615 MiB/s 76916/102400 MiB | 83.38% +8463 MiB 7691 MiB/s 85383/102400 MiB | 91.74% +8551 MiB 7762 MiB/s 93937/102400 MiB | 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB || Wrote 102400 MiB in 13 s @ 7812.023 MiB/s What algorithm did you implement? Secure Random can be obtained from OpenSSL: | $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 * 1024)); done | | real 0m49.288s | user 0m44.710s | sys 0m4.579s Effectively 2078 MiB/s (quite OK for single-threaded operation). It is not designed to generate large amounts of random data as the size is limited by integer range... Thank you for posting the openssl(1) incantation. Benchmarking my daily driver laptop without Intel Secure Key: 2024-02-11 21:54:04 dpchrist@laalaa ~ $ lscpu | grep 'Model name' Model name: Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz 2024-02-11 21:54:09 dpchrist@laalaa ~ $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 * 1024)); done real1m40.149s user1m25.174s sys 0m14.952s So, ~1.072E+9 bytes per second. Benchmarking a workstation with Intel Secure Key: 2024-02-11 21:54:40 dpchrist@taz ~ $ lscpu | grep 'Model name' Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz 2024-02-11 21:54:46 dpchrist@taz ~ $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 * 1024)); done real1m14.696s user1m0.338s sys 0m14.353s So, ~1.437E+09 bytes per second. David
Re: shred bug? [was: Unidentified subject!]
On 2/11/24 06:54, Greg Wooledge wrote: On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote: On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote: On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote: [...] What Thomas was trying to do is to get a cheap, fast random number generator. Shred seems to have such. Well... I certainly wouldn't call it a bug. Maybe a feature request. Still there's the discrepancy between doc and behaviour. There isn't. The documentation says: SYNOPSIS shred [OPTION]... FILE... I interpret the above line to be a prototype for invoking the shred(1) program: * "shred" is the program name * "[OPTION]..." is one or more option specifiers that may be omitted. Each should be described below. * "FILE..." is one or more argument specifies that should be file system paths (strings). DESCRIPTION Overwrite the specified FILE(s) repeatedly, in order to make it harder for even very expensive hardware probing to recover the data. If FILE is -, shred standard output. I interpret the above line at face value -- if the caller provides a dash as the argument, shred(1) will operate on standard output. In every sentence, the word FILE appears. There's nothing in there which says "you can operate on a non-file". Dash is not a file, yet the above sentence says shred(1) can operate on it. Once you grasp what the command is *intended* to do (rewind and overwrite a file repeatedly), it makes absolutely perfect sense that it should only operate on rewindable file system objects. An expert may infer what you have stated, but I prefer manual pages that are explicit. The GNU project must have found a need for the FILE='-' feature, otherwise it would not exist. The manual page should describe that need (e.g. why) and how to properly use shred(1) to solve the need. If you want it to write a stream of data instead of performing its normal operation (rewinding and rewriting), that's a new feature. Humans are (in)famous for doing unexpected things. If you'd prefer the documentation to say explicitly "only regular files and block devices are allowed", that would be an upstream documentation *clarification* request. Apparently, shred(1) has both an info(1) page (?) and a man(1) page. The obvious solution is to write one document that is complete and correct, and use it everywhere -- e.g. DRY. David
Re: Unidentified subject!
On 2/11/24 03:13, Thomas Schmitt wrote: Hi, David Christensen wrote: Concurrency: threads throughput 8 205+198+180+195+205+184+184+189=1,540 MB/s There remains the question how to join these streams without losing speed in order to produce a single checksum. (Or one would have to divide the target into 8 areas which get checked separately.) I had similar thoughts. A FIFO should be able to join the streams. But, dividing the device by the number of virtual cores and putting a thread on each makes more sense. Either done right should fill the drive I/O capacity. Does this 8 thread generator cause any problems with the usability of the rest of the system ? Sluggish program behavior or so ? CPU Graph shows all eight virtual cores at 100%, so everything else on the system would be sluggish (unless you use nice(1)). Here is a processor with Intel Secure Key and otherwise unloaded: 2024-02-11 11:48:21 dpchrist@taz ~ $ lscpu | grep 'Model name' Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz 2024-02-11 11:59:55 dpchrist@taz ~ $ cat /etc/debian_version ; uname -a 11.8 Linux taz 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) x86_64 GNU/Linux 2024-02-11 12:02:52 dpchrist@taz ~ $ dd if=/dev/urandom of=/dev/null bs=1M count=10K 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 20.0469 s, 536 MB/s threads throughput 1 536 MB/s 2 512+512 = 1,024 MB/s 3 502+503+503 = 1,508 MB/s 4 492+491+492+492 = 1,967 MB/s 5 492+384+491+385+491 = 2,243 MB/s 6 379+491+492+379+379+379 = 2,499 MB/s 7 352+491+356+388+352+357+388 = 2,684 MB/s 8 355+354+344+348+344+354+353+349 = 2,801 MB/s I have to correct my previous measurement on the 4 GHz Xeon, which was made with a debuggable version of the program that produced the stream. The production binary which is compiled with -O2 can write 2500 MB/s into a pipe with a pacifier program which counts the data: $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 2s62gss463ar46492bni | $(scdbackup -where bin)/raedchen -step 100m -no_output -print_count 100.0g bytes real0m39.884s user0m30.629s sys 0m41.013s (One would have to install scdbackup to reproduce this and to see raedchen count the bytes while spinning the classic SunOS boot wheel: |/-\|/-\|/-\ http://scdbackup.webframe.org/main_eng.html http://scdbackup.webframe.org/examples.html Oh nostalgy ... ) Totally useless but yielding nearly 4000 MB/s: $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 2s62gss463ar46492bni >/dev/null real0m27.064s user0m23.433s sys 0m3.646s The main bottleneck in my proposal would be the checksummer: $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 2s62gss463ar46492bni | md5sum 5a6ba41c2c18423fa33355005445c183 - real2m8.160s user2m25.599s sys 0m22.663s That's quite exactly 800 MiB/s ~= 6.7 Gbps. Still good enough for vanilla USB-3 with a fast SSD, i'd say. Yes -- more than enough throughput. Before I knew of fdupes(1) and jdupes(1), I wrote a Perl script to find duplicate files. It uses the Digest module, and supports any algorithm supported by that module. Here are some runs against a local ext4 on LUKS (with AES-NI) on Intel SSD 520 Series 60 GB and check summing whole files: 2024-02-11 13:32:47 dpchrist@taz ~ $ time finddups --filter w --digest MD4 .thunderbird/ >/dev/null real0m0.878s user0m0.741s sys 0m0.137s 2024-02-11 13:33:14 dpchrist@taz ~ $ time finddups --filter w --digest MD5 .thunderbird/ >/dev/null real0m1.110s user0m0.977s sys 0m0.132s 2024-02-11 13:33:19 dpchrist@taz ~ $ time finddups --filter w --digest SHA-1 .thunderbird/ >/dev/null real0m1.306s user0m1.151s sys 0m0.156s 2024-02-11 13:36:40 dpchrist@taz ~ $ time finddups --filter w --digest SHA-256 .thunderbird/ >/dev/null real0m2.545s user0m2.424s sys 0m0.121s 2024-02-11 13:36:51 dpchrist@taz ~ $ time finddups --filter w --digest SHA-384 .thunderbird/ >/dev/null real0m1.808s user0m1.652s sys 0m0.157s 2024-02-11 13:37:00 dpchrist@taz ~ $ time finddups --filter w --digest SHA-512 .thunderbird/ >/dev/null real0m1.814s user0m1.673s sys 0m0.141s It is curious that SHA-384 and SHA-512 are faster than SHA-256. I can confirm similar results on: 2024-02-11 13:39:58 dpchrist@laalaa ~ $ lscpu | grep 'Model name' Model name: Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz David
Re: Unidentified subject!
On 2/11/24 00:07, Thomas Schmitt wrote: In the other thread about the /dev/sdm test: Gene Heskett wrote: Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32 [...] $ sudo f3probe --destructive --time-ops /dev/sdm Bad news: The device `/dev/sdm' is a counterfeit of type limbo Device geometry: *Usable* size: 59.15 GB (124050944 blocks) Announced size: 1.91 TB (409600 blocks) David Christensen wrote: Which other thread? Please provide a URL to archived post. https://lists.debian.org/msgid-search/e7a0c1a1-f973-4007-a86d-8d91d8d91...@shentel.net https://lists.debian.org/msgid-search/b566504b-5677-4d84-b5b3-b0de63230...@shentel.net Thank you. :-) David