Ken,
> It did occur to me that if you did a folder -pack in the MH-E index
> folder and you had a numeric sub-folder then your sub-folder would
> change its name and I am not sure what that would do to the MH-E index.
i *suspect* it would mean that i wouldn't be able to track down a
previous sear
>In the meantime, an occasional folder(1) -pack might solve the problem
>manually.
It did occur to me that if you did a folder -pack in the MH-E index
folder and you had a numeric sub-folder then your sub-folder would
change its name and I am not sure what that would do to the MH-E index.
--Ken
Ken wrote:
> This does suggest to me we should probably change the internal API
> so sparse message ranges are handled better;
Yes, that would be a nice enhancement.
In the meantime, an occasional folder(1) -pack might solve the problem manually.
David
Ken,
> The bottom line is nmh (and MH before it) is just not going to perform
> well with billion-sized gaps in message numbers and fixing that is going
> to be very very hard.
understood. thanks for the explanation.
cheers, Greg
>while actual bytes of memory on my laptop are semi-precious, addresses
>in the address space are much less so. here's somebody who uses mmap(2)
>to allocate a huge chunk of address space, and then madvise(2) (a call i
>think i've never used) to have that chunk backed by (lots and lots of)
>zeroes
Robert, et al., thanks very much.
possibly mh-e could add something like a comma before integers. i'll
ask and look.
on the more general issue, you all know a lot more about all of this
than me. but ... :)
while actual bytes of memory on my laptop are semi-precious, addresses
in the address sp
I've always done sortm -verbose after big delete fests. verbose because I
love watching the towers of hanoi shuffle along.
lots of GUI mail systems have 'compact mailbox' command options. I assumed
that everyone did periodic tidyup anyway.
I'm not saying this isn't a problem. But, I seriously won
>a folder with the highest message number of "N" will cause the array to be
>configured to support N messages, even if there are many fewer (perhaps
>even one) messages
No, that's not correct. If you have a single message in a folder with a
count of 100, you only get one entry allocated. The
On Thu, Mar 2, 2023 at 11:27 AM Ken Hornstein wrote:
>
>
> This does suggest to me we should probably change the internal API
> so sparse message ranges are handled better; right now all of the
> programs access the folder structure members directly and assume that
> there will be a msgstat struc
>From Ken's description above, these 111 messages would allocate almost
>800,000 msgstat structures. I don't know how huge the message numbers
>get in the results folder, but six digits is common. I don't recall if
>I've seen seven digit or larger message numbers.
I see Conrad pointed out that i
Hi Conrad,
Conrad Hughes wrote:
> Simon> Possibly somewhat related, Greg mentioned he uses mairix for
> Simon> search. mairix produces very "sparse" results folders.
>
> I use mairix and have never witnessed this. A quick experiment shows
> that it's because I use
>
> sort=date+
>
> in my .ma
Date:Thu, 02 Mar 2023 13:33:01 +
From:Ralph Corderoy
Message-ID: <20230302133301.8900121...@orac.inputplus.co.uk>
| The real issue is nmh doesn't forbid folders named with just decimal
| digits and even creates them when requested. MH-E is set a bad example.
Simon> Possibly somewhat related, Greg mentioned he uses mairix for
Simon> search. mairix produces very "sparse" results folders.
I use mairix and have never witnessed this. A quick experiment shows
that it's because I use
sort=date+
in my .mairixrc. At a guess, the default unsorted numberi
Ken Hornstein wrote:
> Exactly HOW many messages are in mhe-index?
>
> Ah, I think I see what's happening. That line is this:
>
> mp->msgstats = mh_xmalloc (MSGSTATSIZE(mp));
>
> MSGSTATSIZE is defined as:
>
> #define MSGSTATSIZE(mp) ((mp)->num_msgstats * sizeof *(mp)->msgstats)
>
> num_msg
Hi Ken,
> > > I think we have to push this back on the MH-E people
...
> >$ refile +31415 >$ folder +31415
> >31415+ has 1 message (1-1).
>
> I'm aware of that, but what happens if you have a subfolder that is
> all numeric? I believe all of the nmh tools will treat that subfolder
>> I think we have to push this back on the MH-E people; Robert's
>> suggestion to add a non-numeric prefix to directories it creats sounds
>> like the best answer to me.
>
>$ refile +31415 $ folder +31415
>31415+ has 1 message (1-1).
I'm aware of that, but what happens if you have a
Hi,
Ken wrote:
> I think we have to push this back on the MH-E people; Robert's
> suggestion to add a non-numeric prefix to directories it creats sounds
> like the best answer to me.
$ refile +31415
>it seems that at some point i had done a search for 74600607886815 (your
>basic "magic number" :). mh-e, i guess, had created a directory with
>that number as its name (it uses the search term to name subfolders
>under the normal mhe-index folder). and, i guess, flist decided that
>(under the ~/
Date:Thu, 02 Mar 2023 09:49:23 +0300
From:Greg Minshall
Message-ID: <814300.1677739763@archlinux>
| bash archlinux (master): {49603} ls -a 74600607886815/
That would do it.
| and, i guess, flist decided that
| (under the ~/Mail/MHE-INDEX folder) was a message
Ken,
> Exactly HOW many messages are in mhe-index?
exactly
bash archlinux (master): {49536} find ~/MHE-INDEX/ -type f | wc
173477 173477 7838026
but, is that that many? in my overall Mail directory
bash archlinux (master): {49516} find ~/Mail/ -type f | wc
1499737 1499739 63526
>ah, great! yes, that works. and, yes, to my ignorant eye, it appears
>that the call from `folder_read()` to `mh_xmalloc()` is where we are
>going south.
>[...]
>#2 0xf898 in mh_xmalloc (size=42269452928) at sbr/utils.c:38
>#3 0xacf6 in folder_read (name=0x555d5400
>"
Ralph,
> To watch progress, try
>
> strace -e openat flist -recurse -sequence xyz -all >/dev/null
thanks. Ken's hint about ulimit, and questioning the number of folders,
seems to have cleared things up.
> Also, is there anything odd about your tree of directories?
> Does it have cycles? S
hi, Ken,
many thanks for the reply, and the sparkling idea at the end!
> Is there a long delay when you run flist?
yes, there is a long delay, as the system does some N^m processing. the
system gets sluggish as always when there is a huge memory crunch.
> Do you have a lot of folders? Like a
Hi Greg,
> $ valgrind flist -sequence xyz -all
...
> ==729147== Warning: set address range perms: large range [0x59c8e040,
> 0xa313d8ac0) (undefined)
To watch progress, try
strace -e openat flist -recurse -sequence xyz -all >/dev/null
Also, is there anything odd about your tree of director
>> If you run under the debugger, you should stop when you receive the
>> signal from the OOM process.
>
>thanks. OOM is a pretty strange way to die...
Sigh, I guess I was thinking that ptrace() would be able to catch a
process killed by SIGKILL, but I guess not.
Is there a long delay when you r
Ken,
> If you run under the debugger, you should stop when you receive the signal
> from the OOM process.
thanks. OOM is a pretty strange way to die...
run -sequence xyz -all
Starting program: /usr/bin/flist -sequence xyz -all
[Thread debugging using libthread_db enabled]
Using host libthre
>and, if not, any thoughts on how to debug? if i build "cc -g", any
>thoughts on where to set breakpoints, or where to insert printf's, to
>try to track this down?
If you run under the debugger, you should stop when you receive the signal
from the OOM process.
That MIGHT be useful _if_ you hit t
hi.
for some years now, i occasionally get in a situation where flist
gobbles up lots of memory, and is then killed by the kernel's "oom"
procedure. eventually this goes away, though i'm not exactly sure how
that happens either.
bash archlinux (master): {50071} flist -sequence unseen -all -
28 matches
Mail list logo
