RE: Failing to archive many messages
> Today I realized that it was indeed the sa-learn that is slowing down my
> Archiving.
>
> I added an «&» on the end of the line that calls the sa-learn (called by
> the report-ham.sieve script):
>
> exec /usr/bin/sa-learn -u ${1} --ham &
>
> Does anyone thing this is a bad idea? I noticed that archiving got way
> faster this way…
You fork/spawn a new process. So depending on how many messages are moved, that
many processes are created. I am not using sa-learn, but I think this could be
a memory consuming task. So manybe check for OOM crashes or your swap space
slowing everything down.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
Thanks a lot Marc.
I actually found something that works for me:
I replaced `sa-learn -u xx --ham`
with
`spamc -u xx -L ham`
Archiving wasn't as fast as simply moving messages, but was considerably
faster - fast enough that no errors on Roundcube are reported.
Best,
Francis
---
Francis Augusto Medeiros-Logeay
Oslo, Norway
On 2024-05-18 12:06, Marc wrote:
Today I realized that it was indeed the sa-learn that is slowing down
my
Archiving.
I added an «&» on the end of the line that calls the sa-learn (called
by
the report-ham.sieve script):
exec /usr/bin/sa-learn -u ${1} --ham &
Does anyone thing this is a bad idea? I noticed that archiving got way
faster this way…
You fork/spawn a new process. So depending on how many messages are
moved, that many processes are created. I am not using sa-learn, but I
think this could be a memory consuming task. So manybe check for OOM
crashes or your swap space slowing everything down.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
Today I realized that it was indeed the sa-learn that is slowing down my
Archiving.
I added an «&» on the end of the line that calls the sa-learn (called by the
report-ham.sieve script):
exec /usr/bin/sa-learn -u ${1} --ham &
Does anyone thing this is a bad idea? I noticed that archiving got way faster
this way…
Best,
Francis
> On 9 May 2024, at 23:09, dovecot-requ...@dovecot.org wrote:
>
>> On 9 May 2024, at 19:45, dovecot-requ...@dovecot.org
>> wrote:
>>
>> Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 12:29:
>>
If you want to analyze emails, why not do it with scheduled crons
after they are archived?
>>
>> maybe ask for skip older then one day in here
>> https://github.com/isbg/isbg/tree/master
>>
>> while roundcube just set the needed flag for retest ?
>>
>>> wouldn't it reanalyze all archived messages anyway?
>>
>> sometimes email is ham at recpt stage, while its spam later on test,
>> this is why isbg is made imho :)
>
>
> I see.
>
> But I wonder if there is something wrong with what I already have. It worked
> fine for years, and it works fine for most of my users. I tested with a few
> accounts, and Archving (with its sa-learn script) works blazingly fast. But
> with one user - happens to be mine - it is super slow when archiving, most
> likely due the sa-learn.
>
> Best,
>
> Francis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
> On 9 May 2024, at 19:45, dovecot-requ...@dovecot.org wrote: > > Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 12:29: > >>> If you want to analyze emails, why not do it with scheduled crons >>> after they are archived? > > maybe ask for skip older then one day in here > https://github.com/isbg/isbg/tree/master > > while roundcube just set the needed flag for retest ? > >> wouldn't it reanalyze all archived messages anyway? > > sometimes email is ham at recpt stage, while its spam later on test, > this is why isbg is made imho :) I see. But I wonder if there is something wrong with what I already have. It worked fine for years, and it works fine for most of my users. I tested with a few accounts, and Archving (with its sa-learn script) works blazingly fast. But with one user - happens to be mine - it is super slow when archiving, most likely due the sa-learn. Best, Francis ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
> On 9 May 2024, at 12:55, dovecot-requ...@dovecot.org wrote: > > Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 11:28: > >> Actually, I don’t know here if that’s the sieve that’s slowing down the >> process, or if it is something else. >> I see that yeah, sa-learn takes a little while to execute. But maybe >> the issue is that Dovecot doesn’t seem to >> release the IMAP connection until each message went through sieve. This >> gives me some timeout on >> Roundcube (it says just «server error»). All the messages are indeed >> archived. >> >> Is that something here I could do better? > > use tmpfs to store tmp content, or try solve it without pipe with imho > creates tmp files > > https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/ > > try spamc Thanks Benny. Using spamc and sa-learn isn’t a bit the same? But I’ll check the docs. Bestm Francis ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 12:29: If you want to analyze emails, why not do it with scheduled crons after they are archived? maybe ask for skip older then one day in here https://github.com/isbg/isbg/tree/master while roundcube just set the needed flag for retest ? wouldn't it reanalyze all archived messages anyway? sometimes email is ham at recpt stage, while its spam later on test, this is why isbg is made imho :) ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: Failing to archive many messages
> On 2024-05-09 12:03, Marc wrote: > > I see report-ham, instead of archiving? I am not entirely sure if this > > is an efficient approach to archiving. My users can only set a flag for > > 'auto archiving' via sieve and then 3 months after every year messages > > are being moved by cron jobs, and modification logs are put into their > > accounts. This all is only happening in 'dead hours'. > > > > If you want to analyze emails, why not do it with scheduled crons after > > they are archived? > > > > wouldn't it reanalyze all archived messages anyway? No you should add search filter for messages like from today, this week or so. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
= On 2024-05-09 12:03, Marc wrote: I see report-ham, instead of archiving? I am not entirely sure if this is an efficient approach to archiving. My users can only set a flag for 'auto archiving' via sieve and then 3 months after every year messages are being moved by cron jobs, and modification logs are put into their accounts. This all is only happening in 'dead hours'. If you want to analyze emails, why not do it with scheduled crons after they are archived? wouldn't it reanalyze all archived messages anyway? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 11:28: Actually, I don’t know here if that’s the sieve that’s slowing down the process, or if it is something else. I see that yeah, sa-learn takes a little while to execute. But maybe the issue is that Dovecot doesn’t seem to release the IMAP connection until each message went through sieve. This gives me some timeout on Roundcube (it says just «server error»). All the messages are indeed archived. Is that something here I could do better? use tmpfs to store tmp content, or try solve it without pipe with imho creates tmp files https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/ try spamc ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: Failing to archive many messages
I see report-ham, instead of archiving? I am not entirely sure if this is an
efficient approach to archiving. My users can only set a flag for 'auto
archiving' via sieve and then 3 months after every year messages are being
moved by cron jobs, and modification logs are put into their accounts. This all
is only happening in 'dead hours'.
If you want to analyze emails, why not do it with scheduled crons after they
are archived?
> I noticed that my sieve rules are taking a very long time when archiving
> messages. I can see on my logs that every message takes about two-three
> seconds to be evaluated.
>
> My sieve for the Archive is this:
>
> cat /var/mail/vmail/mail/sieve/global/report-ham.sieve
> require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment",
> "variables"];
>
> if environment :matches "imap.mailbox" "*" {
> set "mailbox" "${1}";
> }
>
> if string "${mailbox}" "Trash" {
> stop;
> }
>
> if environment :matches "imap.user" "*" {
> set "username" "${1}";
> }
>
> pipe :copy "sa-learn-ham.sh" [ "${username}" ];
>
> And my sa-learn-ham.sh is this:
>
> #!/bin/sh
> export PATH="/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin:$PATH"
> exec /usr/bin/sa-learn -u ${1} —ham
>
>
>
>
> May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug:
> sieve: multi-script: Start execute sequence
> May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug:
>
> I also noticed this happens for a few accounts. Do you know what could be
> causing this?
>
> Best,
> Francis
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
> I noticed that my sieve rules are taking a very long time when archiving > messages. I can see on my logs that every message takes about two-three > seconds to be evaluated. > > I also noticed this happens for a few accounts. Do you know what could be > causing this? > > Best, > Francis > Actually, I don’t know here if that’s the sieve that’s slowing down the process, or if it is something else. I see that yeah, sa-learn takes a little while to execute. But maybe the issue is that Dovecot doesn’t seem to release the IMAP connection until each message went through sieve. This gives me some timeout on Roundcube (it says just «server error»). All the messages are indeed archived. Is that something here I could do better? Best, Francis ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Failing to archive many messages
I noticed that my sieve rules are taking a very long time when archiving
messages. I can see on my logs that every message takes about two-three seconds
to be evaluated.
My sieve for the Archive is this:
cat /var/mail/vmail/mail/sieve/global/report-ham.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];
if environment :matches "imap.mailbox" "*" {
set "mailbox" "${1}";
}
if string "${mailbox}" "Trash" {
stop;
}
if environment :matches "imap.user" "*" {
set "username" "${1}";
}
pipe :copy "sa-learn-ham.sh" [ "${username}" ];
And my sa-learn-ham.sh is this:
#!/bin/sh
export PATH="/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin:$PATH"
exec /usr/bin/sa-learn -u ${1} —ham
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
multi-script: Start execute sequence
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
Executing script from `/var/mail/vmail/mail/sieve/global/report-ham.svbin'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
multi-script: Run script `/var/mail/vmail/mail/sieve/global/report-ham.svbin'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
Started running script `/var/mail/vmail/mail/sieve/global/report-ham.svbin'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
Finished running script `ok'
(status=/var/mail/vmail/mail/sieve/global/report-ham.svbin, resource usage: no
usage recorded)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
multi-script: Execute result
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Executing result (status=ok, commit=no)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Starting execution of actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Executing actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Executing pipe action
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
action pipe: running program: sa-learn-ham.sh
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Created
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass
environment: USER=fran...@med-lo.eu
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass
environment: HOME=/var/mail/vmail/med-lo.eu/francis
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass
environment: HOST=fb58607074fd
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: Mailbox
Archive: UID 191458: Opened mail because: mail stream
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Finished executing pipe action (status=ok, keep=implicit)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Finished executing actions (status=ok, keep=implicit)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Finished executing result (no commit, status=ok, keep=yes)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
multi-script: Sequence active
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
multi-script: Finishing sequence (status=ok)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Executing result (status=ok, commit=yes)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Starting execution of actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Executing actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Finished executing actions (status=ok, keep=implicit)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Execute implicit keep (failure=no)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Start storing into mailbox Archive
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Executing implicit keep action
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Execute storing into mailbox 'Archive'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve:
uid=191458: Updated existing mail in mailbox 'Archive'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>:
Re: Failing to archive many messages
It is weird. At first, it seems that the logs are going fine. It goes like this for many minutes: May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.IN-REPLY-TO from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.MESSAGE-ID from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.SUBJECT from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.TO from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.X-PRIORITY from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.BCC from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.REPLY-TO from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.DISPOSITION-NOTIFICATION-TO from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Looked up field hdr.LIST-POST from mail cache May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: multi-script: Start execute sequence May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: Executing script from `/var/mail/vmail/mail/sieve/global/report-ham.svbin' May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: multi-script: Run script `/var/mail/vmail/mail/sieve/global/report-ham.svbin' May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: Started running script `/var/mail/vmail/mail/sieve/global/report-ham.svbin' May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: Finished running script `ok' (status=/var/mail/vmail/mail/sieve/global/report-ham.svbin, resource usage: no usage recorded) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: multi-script: Execute result May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Executing result (status=ok, commit=no) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Starting execution of actions May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Executing actions May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Executing pipe action May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: action pipe: running program: sa-learn-ham.sh May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: program exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Created May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: program exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass environment: USER=fran...@mydomain.com May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: program exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass environment: HOME=/var/mail/vmail/med-lo.eu/francis May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: program exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass environment: HOST=fb58607074fd May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: Mailbox Archive: UID 25719: Opened mail because: mail stream May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Finished executing pipe action (status=ok, keep=implicit) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Finished executing actions (status=ok, keep=implicit) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Finished executing result (no commit, status=ok, keep=yes) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: multi-script: Sequence active May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: multi-script: Finishing sequence (status=ok) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Executing result (status=ok, commit=yes) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Starting execution of actions May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Executing actions May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Finished executing actions (status=ok, keep=implicit) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Execute implicit keep (failure=no) May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Start storing into mailbox Archive May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Executing implicit keep action May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Execute storing into mailbox 'Archive' May 01 14:20:04 imap(fran...@mydomain.com)<226>: Debug: sieve: uid=25719: Updated existing mail in
Failing to archive many messages
Hi, When I archive one or two messages, the messages are correctly moved to the Archive. However, when I archive more messages - like 50 or more, dovecot seems to fail. The error I get is this one: Warning: Failed to do incremental sync for mailbox Archive, retry with a full sync (Modseq 17277 no longer in transaction log (highest=17323, last_common_uid=12216, nextuid=15767)) No messages end up being moved to the Archive folder. I tried to do this both from a mail client and from roundcube. What could be the problem here? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
