Re: latest upgrade to systemd 252.12-1 error about invalid attributes /var/log/journal and slow sshd connections
On Sat, Jul 15, 2023 at 1:09 PM David Mehler wrote: > > [...] > > "2. "I noticed that when I change UsePAM yes to UsePAM no then this > issue is resolved." > > BINGO! I flipped that UsePAM setting to no and the problem has gone away. If you need a datapoint about UsePAM... I've been setting it to 'no' for years on the BSDs, Debian, Fedora, Hurd, Red Hat and Ubuntu. But I also disable all password authentication, and require public key authentication. $ cat /etc/ssh/sshd_config.d/10-pubkey_auth.conf # Disable passwords PasswordAuthentication no ChallengeResponseAuthentication no KerberosAuthentication no KerberosOrLocalPasswd no GSSAPIAuthentication no UsePAM no # Enable public key PubkeyAuthentication yes Jeff
Re: latest upgrade to systemd 252.12-1 error about invalid attributes /var/log/journal and slow sshd connections
On Sat 15 Jul 2023, at 17:52, David Mehler wrote: [...] > Regarding the original issue of the systemd upgrade and the invalid > attributes [...] here is the output that I've got: > [...] > Cannot set file attributes for '/var/log/journal', maybe due to > incompatibility in specified attributes, previous=0x0008, > current=0x0008, expected=0x0088, ignoring. > Cannot set file attributes for > '/var/log/journal/390b00d843d3401094a8fd44f1b7de82', maybe due to > incompatibility in specified attributes, previous=0x0008, > current=0x0008, expected=0x0088, ignoring. > Obsolete conffile /etc/systemd/resolved.conf has been modified by you. > Saving as /etc/systemd/resolved.conf.dpkg-bak ... User "seth" at https://bbs.archlinux.org/viewtopic.php?id=272893 suggests "The error itself is harmless; systemd tries to set an attribute on a filesystem that doesn't support it" which seems to go along with it being ignored. and later: "0x0080 is FS_NOCOW_FL - what is not a thing on directories. Edit except for apparently btrfs - what also seems the only supported FS here. Otherwise you get an error [...]" User j1simon suggests in https://bbs.archlinux.org/viewtopic.php?pid=2013787#p2013787 that the errors are present at boot. (I presume journalctl -b is how that output was obtained) I use ZFS and can't find any similar errors in boot log $ sudo journalctl -b|grep incompat $ so I wonder if ZFS supports it on directories too. man ioctl_iflags: "FS_NOCOW_FL 'C' (since Linux 2.6.39) The *file* will not be subject to copy-on-write updates. This flag has an effect only on filesystems that support copy-on-write semantics, such as Btrfs. See chattr(1) and btrfs(5)." https://man7.org/linux/man-pages/man2/ioctl_iflags.2.html The reporter in the first link above is asked if the bug has been reported to systemd developers. In another bug report re the same error (if in a slightly different context) on F2FS, systemd developer Lennart Poettering says "[...] this is a bug in the filesystem - They should not just eat up requests to set flags, but return an error. Please ping the f2fs maintainers." https://github.com/systemd/systemd/issues/26318 It looks like the same bug/issue on ext4 to me, and I imagine safe to ignore. Best wishes, Gareth
Re: latest upgrade to systemd 252.12-1 error about invalid attributes /var/log/journal and slow sshd connections
Hello, Thanks. The ssh issue has been solved. "The same symptoms appear in an answer to https://superuser.com/questions/166359/why-is-my-ssh-login-slow which includes various solutions, some more permanent/apparently likely to help you than others. Just out of interest, is the su command (on the ssh server machine) also affected by authentication delays? This apparently suggests a PAM issue." In answer yes su on the ssh machine also has these delays. It is looking like a pam issue. "1. "I found that PAM was reading the file /var/log/btmp, which had become huge as a result of people trying to brute-force my server. This was leading to login times of a minute. Clearing this file solved the problem." I did check for /var/log/btmp and it is a nice lovely 25MB in size. I did clear it, restarted sshd and this did not clear up the problem, still had the delays. "2. "I noticed that when I change UsePAM yes to UsePAM no then this issue is resolved." BINGO! I flipped that UsePAM setting to no and the problem has gone away. Regarding the original issue of the systemd upgrade and the invalid attributes (this sshd was a nice side venture but wasn't sure if it was connected or not) here is the output that I've got: Setting up systemd (252.11-1~deb12u1) ... Installing new version of config file /etc/systemd/journald.conf ... Installing new version of config file /etc/systemd/logind.conf ... Installing new version of config file /etc/systemd/networkd.conf ... Installing new version of config file /etc/systemd/pstore.conf ... Installing new version of config file /etc/systemd/sleep.conf ... Installing new version of config file /etc/systemd/system.conf ... Installing new version of config file /etc/systemd/user.conf ... Cannot set file attributes for '/var/log/journal', maybe due to incompatibility in specified attributes, previous=0x0008, current=0x0008, expected=0x0088, ignoring. Cannot set file attributes for '/var/log/journal/390b00d843d3401094a8fd44f1b7de82', maybe due to incompatibility in specified attributes, previous=0x0008, current=0x0008, expected=0x0088, ignoring. Obsolete conffile /etc/systemd/resolved.conf has been modified by you. Saving as /etc/systemd/resolved.conf.dpkg-bak ... Thanks. Dave. On 7/15/23, Gareth Evans wrote: > On Sat 15 Jul 2023, at 13:09, Gareth Evans wrote: >> >> 2. "I noticed that when I change UsePAM yes to UsePAM no then this >> issue is resolved." >> >> There may be security (or other) issues with (2). > > See, for example: > > https://unix.stackexchange.com/questions/673153/sshd-what-are-the-practical-effects-of-setting-usepam-no > >
Re: latest upgrade to systemd 252.12-1 error about invalid attributes /var/log/journal and slow sshd connections
On Sat 15 Jul 2023, at 13:09, Gareth Evans wrote: > > 2. "I noticed that when I change UsePAM yes to UsePAM no then this > issue is resolved." > > There may be security (or other) issues with (2). See, for example: https://unix.stackexchange.com/questions/673153/sshd-what-are-the-practical-effects-of-setting-usepam-no
Re: latest upgrade to systemd 252.12-1 error about invalid attributes /var/log/journal and slow sshd connections
On Wed 12 Jul 2023, at 18:29, Gareth Evans wrote: >> On 12 Jul 2023, at 15:12, David Mehler wrote: >> [sshd login takes a long time] > [...] > Does > > ssh -vvv ... > > (at client) shed any light? Replying to an off-list message from David in which he stated ssh -vvv waits after > debug1: Entering interactive session. > debug1: pledge: network The same symptoms appear in an answer to https://superuser.com/questions/166359/why-is-my-ssh-login-slow which includes various solutions, some more permanent/apparently likely to help you than others. Just out of interest, is the su command (on the ssh server machine) also affected by authentication delays? This apparently suggests a PAM issue. If you start a new ssh server on a different port and enable debugging: $ sudo /usr/sbin/sshd -ddd -p1234 then at what point does it hang when you ssh from the other machine? Don't forget to specify target port (with -p1234) If PAM-related, then answers at the above link suggest: 1. "I found that PAM was reading the file /var/log/btmp, which had become huge as a result of people trying to brute-force my server. This was leading to login times of a minute. Clearing this file solved the problem." 2. "I noticed that when I change UsePAM yes to UsePAM no then this issue is resolved." There may be security (or other) issues with (2). To avoid the risk of locking yourself out of VPS I would Copy /etc/ssh/sshd_config elsewhere Amend the copy to include UsePAM no $ sudo /var/sbin/sshd -f /path/to/sshd_config_copy -ddd -p1235 (NB use new port number if previous command still running) then see if you can ssh to it. If the issue is not solved by either of the above, please give any sshd debug output that seems relevant for a few lines before/after the wait. To view the systemd journal, see man journalctl You may however like to install rsyslog to get /var/log/syslog back. Not sure if it's retro-active though. HTH Gareth
Re: latest upgrade to systemd 252.12-1 error about invalid attributes /var/log/journal and slow sshd connections
> On 12 Jul 2023, at 15:12, David Mehler wrote: > > Hello, > > I'm running Debian 12 on a vps. I just upgraded it and am now > apparently running the latest systemd version 252.12-1. I saw an error > about invalid attributes on /var/log/journal then it said ignoring. > I've seen others with this error but only in reference as far as I can > tell to the btrfs filesystem which I'm not using. I've got a single > drive running ext4. I'm also seeing very slow like over a minute > connection times between when I authenticate via sshd and I get a > terminal prompt which is also since this upgrade. The initial server > connection goes as normal, it gets my public key then a good long > delay and then I finally get my terminal prompt. > > Any comments on either of these appreciated. Hi Dave, Can you specify the journal error messages? This suggests ssh login delay may be a DNS issue https://superuser.com/questions/166359/why-is-my-ssh-login-slow Does ssh -vvv ... (at client) shed any light? Thanks, Gareth > Thanks. > Dave. >
