Re: smtpd warn: not enough disk space

[Christian Schulte]

On 09.07.24 11:16, Stuart Henderson wrote:

On 2024-07-09, Christian Schulte  wrote:

For example: Just
remove the patches in this directory - well a lot of them - and see how
those GNU folks have turned into complete idiots. I don't get it.

https://github.com/openbsd/ports/tree/master/devel/gettext/patches


A lot of those patches are to avoid triggering warnings from ld when
linking other programs which use the gettext library due to the
api warnings openbsd has for some libc functions.




Of course they do. And there is a reason they do. This goes back to me 
running i386 instead of amd64 due to RAM constraints and using the gnome 
desktop environment, which makes heavy use of gettext. gettext authors 
have just sped up those functions, because they noticed themselves, that 
they are called way to often. Then those patches removed those speed ups 
uncovering those design flaws. Of course they noticed this themselves. 
They never rethought design decisions. Install i386 with a gnome desktop 
environment. So slow. Rebuild gettext wihtout some of those patches 
removing those speed ups. There you go. It's so noticeable and it's not 
those patches to blame. That's what I meant with "avoid hotspots" rather 
than trying to make them run faster. That's theire philosophy. And that 
is just wrong.

Re: smtpd warn: not enough disk space

[Christian Schulte]

On 09.07.24 11:18, Stuart Henderson wrote:

On 2024-07-09, Christian Schulte  wrote:



On 07.07.24 03:51, Jeremy Evans wrote:

On Fri, Jul 5, 2024 at 9:16 PM Christian Schulte mailto:schulte...@gmail.com>> wrote:

 Just wondering how the postgresql
 port is configured. Really should setup quotas automatically when
 pkg_adding in a way, just to ensure, that no one ever runs into a
 situation, that there is no way out of a disk full situation.


The port can't sanely do that, because it doesn't know how the admin
has configured their system.

Also, openbsd doesn't enable filesystem quotas by default.


I did not criticize the postgresql port in any way. I am just
suggesting, that when you want to setup a postgresql server in a
fire-and-forget way of things, it would be cool to restrict it from
eating up all available storage.


That is simple, use a separate filesystem for /var/postgresql.



Indeed. This is what my provider provided me with. Watch out for the 
mount point of /. This would not have been a problem, if postgresql 
would be able to reclaim diskspace without requiring disk space to do 
so. This is what you get, for not taking care of yourself and blindly 
relying on others to provide you with sane defaults. Was it really my 
fault? Maybe. A simple warning message during install of postgresql 
about what issue might I run into would have been - hmm - nice.


sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs 
(rw,nosuid,relatime,size=8168948k,nr_inodes=2042237,mode=755,inode64)
devpts on /dev/pts type devpts 
(rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs 
(rw,nosuid,nodev,noexec,relatime,size=1637608k,mode=755,inode64)
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs 
(rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs 
(rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 
(rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs 
(rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2495)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl 
(rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs 
(ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs 
(ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs 
(ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda1 on /boot type ext4 (rw,noatime)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs 
(ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc 
(rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs 
(rw,nosuid,nodev,relatime,size=1637604k,nr_inodes=409401,mode=700,uid=1000,gid=1000,inode64)

Re: smtpd warn: not enough disk space

[Stuart Henderson]

On 2024-07-09, Christian Schulte  wrote:
>
>
> On 07.07.24 03:51, Jeremy Evans wrote:
>> On Fri, Jul 5, 2024 at 9:16 PM Christian Schulte > > wrote:
>> 
>> Just wondering how the postgresql
>> port is configured. Really should setup quotas automatically when
>> pkg_adding in a way, just to ensure, that no one ever runs into a
>> situation, that there is no way out of a disk full situation.

The port can't sanely do that, because it doesn't know how the admin
has configured their system.

Also, openbsd doesn't enable filesystem quotas by default.

> I did not criticize the postgresql port in any way. I am just 
> suggesting, that when you want to setup a postgresql server in a 
> fire-and-forget way of things, it would be cool to restrict it from 
> eating up all available storage.

That is simple, use a separate filesystem for /var/postgresql.

-- 
Please keep replies on the mailing list.

Re: smtpd warn: not enough disk space

[Stuart Henderson]

On 2024-07-09, Christian Schulte  wrote:
>For example: Just 
> remove the patches in this directory - well a lot of them - and see how 
> those GNU folks have turned into complete idiots. I don't get it.
>
> https://github.com/openbsd/ports/tree/master/devel/gettext/patches

A lot of those patches are to avoid triggering warnings from ld when
linking other programs which use the gettext library due to the
api warnings openbsd has for some libc functions.


-- 
Please keep replies on the mailing list.

Re: smtpd warn: not enough disk space

[Christian Schulte]

On 07.07.24 03:51, Jeremy Evans wrote:
On Fri, Jul 5, 2024 at 9:16 PM Christian Schulte > wrote:


Just wondering how the postgresql
port is configured. Really should setup quotas automatically when
pkg_adding in a way, just to ensure, that no one ever runs into a
situation, that there is no way out of a disk full situation.


I'm not aware of any port that sets up quotas automatically, so I don't 
understand why you think this is an issue with the PostgreSQL port 
specifically.  Since you are wondering how the PostgreSQL port is 
configured: 
https://cvsweb.openbsd.org/ports/databases/postgresql/Makefile?rev=1.304=text/x-cvsweb-markup 


Jeremy



I am not wondering about anything. Just suggesting improvements. 
Currently I grabbed my old Atari 800XL and Atari Falcon 030 doing some 
cool things whit those machines using a PC for calculating data quicker, 
than had been possible in the 80s and 90s. Those 8 bit Ataris were 
really great machines. The ancestor Amiga as well. For example: Just 
remove the patches in this directory - well a lot of them - and see how 
those GNU folks have turned into complete idiots. I don't get it.


https://github.com/openbsd/ports/tree/master/devel/gettext/patches

They are introducing hotspots like hell and then try to speed those up 
by using insecure non sense. Those GNU folks have no soul. Just remove a 
bunch of those patches in the ports tree and see how stupid they were 
having not eliminated those hotspots in the first place but trying to 
make them as fast and insecure as possible.


Maybe time for me to write some 6502 on my Atari 800XL. There you do not 
need to cope with GNU idiots at all.


Regards,
--
Christian

Re: smtpd warn: not enough disk space

[Christian Schulte]

On 07.07.24 03:51, Jeremy Evans wrote:
On Fri, Jul 5, 2024 at 9:16 PM Christian Schulte > wrote:


Just wondering how the postgresql
port is configured. Really should setup quotas automatically when
pkg_adding in a way, just to ensure, that no one ever runs into a
situation, that there is no way out of a disk full situation.


I'm not aware of any port that sets up quotas automatically, so I don't 
understand why you think this is an issue with the PostgreSQL port 
specifically.  Since you are wondering how the PostgreSQL port is 
configured: 
https://cvsweb.openbsd.org/ports/databases/postgresql/Makefile?rev=1.304=text/x-cvsweb-markup 


Jeremy



I did not criticize the postgresql port in any way. I am just 
suggesting, that when you want to setup a postgresql server in a 
fire-and-forget way of things, it would be cool to restrict it from 
eating up all available storage. Just because only then will you notice 
how difficult it can get to get out of such a situation. That's all. 
Discussion started with an MTA blindly preserving 5% of space for 
temporary queue files, which are, well, temporary. Makes no sense. Queue 
will get emptied whatsoever automatically. Completely different scenario 
with a database system. The MTA can and will resolve such a situation 
automatically over a period of a few days. This does not hold true for a 
database system, which is not dealing with temporary data and just needs 
a way to ensure someone never runs into a non recoverable situation. No 
need to apply any changes to the postgresql port. If you know how nasty 
things can get, you can also just use GNU/Linux to shoot you into your 
own feet. So to say. There is a reason why we are here.


Regards,
--
Christian

Re: smtpd warn: not enough disk space

[Jeremy Evans]

On Fri, Jul 5, 2024 at 9:16 PM Christian Schulte 
wrote:

> Just wondering how the postgresql
> port is configured. Really should setup quotas automatically when
> pkg_adding in a way, just to ensure, that no one ever runs into a
> situation, that there is no way out of a disk full situation.
>

I'm not aware of any port that sets up quotas automatically, so I don't
understand why you think this is an issue with the PostgreSQL port
specifically.  Since you are wondering how the PostgreSQL port is
configured:
https://cvsweb.openbsd.org/ports/databases/postgresql/Makefile?rev=1.304=text/x-cvsweb-markup

Jeremy

Re: smtpd warn: not enough disk space

[Christian Schulte]

On 06.07.24 04:08, Eric Pruitt wrote:

On Sat, Jul 06, 2024 at 01:49:05AM +0200, Christian Schulte wrote:

A database admin would have monitored the system and just enhanced
storage when required. Bad thing for me was, that I could not vaccuum
the database, because postgresql copies tables to new files to reclaim
disk space afterwards. So the database was unusable for about 12
hours, until I could manage to xz a backup and transfer it to another
machine. Making postgresql aware of keeping some free disk space, it
would need to preserve around 50%, so that vaccuum can always be
performed.


This is tangential to the original issue, but I've heard of some
administrators leaving around large files so they can be removed when
the disk unexpectedly fills up buying them more time to address the
underlying issue, and CockroachDB implements this natively:
.

Eric


Those large files resided in /root/backup for me. Luckily I setup a 
cronjob doing some daily backups just to be save. If it would not have 
been possible to delete those, freeing some GB, I would have been really 
screwed. No way to get out of this situation without increasing storage. 
That would have been impossible with my provider as well. For the 
record: Never ever let postgresql fill up your storage 100%, when there 
is no way for you to increase storage. Just wondering how the postgresql 
port is configured. Really should setup quotas automatically when 
pkg_adding in a way, just to ensure, that no one ever runs into a 
situation, that there is no way out of a disk full situation.


Regards,
--
Christian

Re: smtpd warn: not enough disk space

[Eric Pruitt]

On Sat, Jul 06, 2024 at 01:49:05AM +0200, Christian Schulte wrote:
> A database admin would have monitored the system and just enhanced
> storage when required. Bad thing for me was, that I could not vaccuum
> the database, because postgresql copies tables to new files to reclaim
> disk space afterwards. So the database was unusable for about 12
> hours, until I could manage to xz a backup and transfer it to another
> machine. Making postgresql aware of keeping some free disk space, it
> would need to preserve around 50%, so that vaccuum can always be
> performed.

This is tangential to the original issue, but I've heard of some
administrators leaving around large files so they can be removed when
the disk unexpectedly fills up buying them more time to address the
underlying issue, and CockroachDB implements this natively:
.

Eric

Re: smtpd warn: not enough disk space

[Christian Schulte]

On 05.07.24 13:46, Jeremy Mates wrote:

On 2024-07-05 05:19:01 +0200, Christian Schulte wrote:

I have never seen an application performing such kind of checks.


Sendmail had a knob to refuse mail at a certain CPU load, on the
assumption that if a system was "too busy" it's in a bad state and
accepting mail would only make things worse. Also rogue(6) had CPU
checks, apparently they wanted to stop any rogue processes when the
system load got too high--maybe real work was being done? Speaking of
not checking the CPU, I have an amusing story of what a dual-processor
Linux box did at a CPU load of 5,000, something something sending the
wrong bytes to the wrong file descriptors, thus corrupting payment
transactions for a small internet retailer.

For application-specific filesystem checks I don't recall any off the
top of my head, but I tend to have monitoring send dire alerts when the
disk usage hits 90% (or less, if you want more breathing room to figure
out what is wrong, who to notify, them to act, etc) so would never have
seen anything that triggers at 5% free or whatever.


If there is not enough space, write would fail anyway.


Or the filesystem gets somewhat looped after being run at 100% full for
too long. Granted, that was Windows, and we did warn the user not to do
that, but they did, and then when a nice Windows admin tried to backup
the 500G disk, it filled up a 3T disk and wanted more. Who knows what
breaks when a stray dd(1) or package update or whatever eats too much of
the remaining space. Not breaking randomly is probably a design goal of
a SMTP server that tries to guarantee delivery? You could fiddle with
the percentage, or remove the code, but I'm going to guess that most
everyone else keeps their mail partitions not so full.


I just commented out the function call locally for my laptop. The issue 
had been discussed on the OpenSMTPD mailing list as well. As it seems, 
everyone seems to agree, that checking a fixed percentage value makes no 
sense. I do not disagree that checking/preserving some amount of free 
disk space makes sense. For example, I ran into an out of disk space 
situation on a server running postgresql filling up the disk 100% and 
then crashing. It's a personal VPS server I am not maintaining 
professionally. A database admin would have monitored the system and 
just enhanced storage when required. Bad thing for me was, that I could 
not vaccuum the database, because postgresql copies tables to new files 
to reclaim disk space afterwards. So the database was unusable for about 
12 hours, until I could manage to xz a backup and transfer it to another 
machine. Making postgresql aware of keeping some free disk space, it 
would need to preserve around 50%, so that vaccuum can always be 
performed. That would be just impractical. My objection is that 
something like this should not be coded into the application itself, but 
rather is a task of general system maintenance (e.g. setting up 
filesystem quotas and such). For what it's worth, it's just my private 
laptop...


Regards,
--
Christian

Re: smtpd warn: not enough disk space

[Christian Schulte]

On 05.07.24 13:50, Souji Thenria wrote:

On Fri Jul 5, 2024 at 4:19 AM BST, Christian Schulte wrote:

Hello,

Hi Christian,

What is the reasoning to check for disk space based on percentages? I 
have never seen an application performing such kind of checks. If 
there is not enough space, write would fail anyway. Checking for 
available bytes before write would make sense. Checking a percentage 
value? I don't get it. Not being able to send mail, although 3.2G 
space is available makes no sense, IMHO.


You are not the first one with this issue. However, it looks like it
went nowhere.
https://www.mail-archive.com/misc@opensmtpd.org/msg01007.html

It might be worth to restate the question on the OpenSMTPD mailing list.


Thank you. I should have used that mailing list, of course.

Regards,
--
Christian

Re: smtpd warn: not enough disk space

[Souji Thenria]

On Fri Jul 5, 2024 at 4:19 AM BST, Christian Schulte wrote:

Hello,

Hi Christian,

What is the reasoning to check for disk space based on percentages? I 
have never seen an application performing such kind of checks. If there 
is not enough space, write would fail anyway. Checking for available 
bytes before write would make sense. Checking a percentage value? I 
don't get it. Not being able to send mail, although 3.2G space is 
available makes no sense, IMHO.


You are not the first one with this issue. However, it looks like it
went nowhere.
https://www.mail-archive.com/misc@opensmtpd.org/msg01007.html

It might be worth to restate the question on the OpenSMTPD mailing list.

Regards,
Souji

Re: smtpd warn: not enough disk space

[Jeremy Mates]

On 2024-07-05 05:19:01 +0200, Christian Schulte wrote:
> I have never seen an application performing such kind of checks.

Sendmail had a knob to refuse mail at a certain CPU load, on the
assumption that if a system was "too busy" it's in a bad state and
accepting mail would only make things worse. Also rogue(6) had CPU
checks, apparently they wanted to stop any rogue processes when the
system load got too high--maybe real work was being done? Speaking of
not checking the CPU, I have an amusing story of what a dual-processor
Linux box did at a CPU load of 5,000, something something sending the
wrong bytes to the wrong file descriptors, thus corrupting payment
transactions for a small internet retailer.

For application-specific filesystem checks I don't recall any off the
top of my head, but I tend to have monitoring send dire alerts when the
disk usage hits 90% (or less, if you want more breathing room to figure
out what is wrong, who to notify, them to act, etc) so would never have
seen anything that triggers at 5% free or whatever.

> If there is not enough space, write would fail anyway.

Or the filesystem gets somewhat looped after being run at 100% full for
too long. Granted, that was Windows, and we did warn the user not to do
that, but they did, and then when a nice Windows admin tried to backup
the 500G disk, it filled up a 3T disk and wanted more. Who knows what
breaks when a stray dd(1) or package update or whatever eats too much of
the remaining space. Not breaking randomly is probably a design goal of
a SMTP server that tries to guarantee delivery? You could fiddle with
the percentage, or remove the code, but I'm going to guess that most
everyone else keeps their mail partitions not so full.