Re: flist -- "Killed" -- oom (*not* 1.8 related)

2023-03-03 Thread Greg Minshall
Ken,

> The bottom line is nmh (and MH before it) is just not going to perform
> well with billion-sized gaps in message numbers and fixing that is going
> to be very very hard.

understood.  thanks for the explanation.

cheers, Greg



Re: flist -- "Killed" -- oom (*not* 1.8 related)

2023-03-03 Thread Ken Hornstein
>while actual bytes of memory on my laptop are semi-precious, addresses
>in the address space are much less so.  here's somebody who uses mmap(2)
>to allocate a huge chunk of address space, and then madvise(2) (a call i
>think i've never used) to have that chunk backed by (lots and lots of)
>zeroes.
>
>https://robert.ocallahan.org/2016/06/managing-vast-sparse-memory-on-linux.html
>

While that is interesting, I see some issues:

- MAP_NORESERVE is a Linux-specific feature of mmap(), as far as I can tell.
  I'm not opposed to OS-specific features but we'd need to think about it
  carefully.
- As for if it would help ... well, it depends on what you are doing.
  In the specific case of flist(1), it would probably help because one
  of things folder_read() does is count up the total number of messages
  in a folder (mp->nummsg) and that's what flist uses.  But if you tried
  to use scan(1) on that folder, well ... what scan(1) does is start
  at "lowmsg" and calls does_exist() on every number between "lowmsg"
  and "highmsg" to determine if that message exists.  And does_exist()
  is using the msgstat array to see if a message exists, so you'd be
  reading every single msgstat array member.

The bottom line is nmh (and MH before it) is just not going to perform
well with billion-sized gaps in message numbers and fixing that is going
to be very very hard.

--Ken