dovecot sieve duplicates detection
Hello, I have tested the sieve duplicate script with success so far, but I have a question. I would like to know if the "duplicate" sieve flag in Dovecot is global to all folders, or specific to one folder only. For instance, if I copy an email from one folder to another, and I have a discard action on duplicate email, is this action will be applied (in this case, discard) or not. If the duplicate is global to all folders, is there a way to restrict the search in one folder only. Thanks for your help. André
Re: lda fails in parse_angle_addr if sieve is enabled
Op 11-4-2018 om 20:42 schreef Bernard Spil: Hi Stephan, Shared the message to you in person only via separate mail. With to Return-Path, I've not seen any difference in failures, all messages were consistently failing with the same error. Thank you! Bernard. The fix for the panic itself is here: https://github.com/dovecot/core/commit/fbed9168dc3b104b09bd748409aec902328cd8e5 Given that you provided an example message without a return-path header, I am confused on how this panic would occur at your end. Are you really sure this is the message that Dovecot gets to see? If you extracted it from the STMP queue, that may not be the same. You could also obtain a GDB backtrace from dovecot-lda, which would tell me from which part of the code this panic is reached (I currently see only one obvious path). You can run dovecot-lda from command line within GDB to achieve this: `gdb --args dovecot-lda -p -d ` and then execute `r` .. wait until it crashes an then issue 'bt full` (https://dovecot.org/bugreport.html). Regards, Stephan. 2018-04-10 2:54 GMT+02:00 Stephan Bosch : Op 4/8/2018 om 8:10 PM schreef Bernard Spil: Hi, Since updating to 2.3.1 on my FreeBSD mailserver mail delivery using lda is broken if I have sieve enabled. (Before updating this was 2.2 and pigeonhole 0.4) FreeBSD 11.1-p8 amd64 Dovecot 2.3.1 Pigeonhole 0.5.1 Mailflow is OpenSMTPd as MTA, using mda delivery to rspamc which utlimately delivers using dovecot-lda. smtpd.conf deliver to mda "rspamc -h scan --mime -e \"/usr/local/libexec/dovecot/deliver -d %{user.username}\"" %{user.username} is the local user after virtusers, aliases etc. verified using a shell wrapper and capturing the username. conf.d/15-lda.conf protocol lda { mail_plugins = $mail_plugins sieve } Mail gets delivered. Don't understand why it is looking for a <>-address if sieve is enabled. I found the cause of the panic. However, what actually triggers it could be a little more nasty. Do you have an example of a message passed to dovecot-lda that causes the problem? I am particularly interested in the Return-Path header. Regards, Stephan.
Re: lda fails in parse_angle_addr if sieve is enabled
Hi Stephan, Shared the message to you in person only via separate mail. With to Return-Path, I've not seen any difference in failures, all messages were consistently failing with the same error. Thank you! Bernard. 2018-04-10 2:54 GMT+02:00 Stephan Bosch : > Op 4/8/2018 om 8:10 PM schreef Bernard Spil: >> Hi, >> >> Since updating to 2.3.1 on my FreeBSD mailserver mail delivery using >> lda is broken if I have sieve enabled. >> (Before updating this was 2.2 and pigeonhole 0.4) >> >> FreeBSD 11.1-p8 amd64 >> Dovecot 2.3.1 >> Pigeonhole 0.5.1 >> >> Mailflow is OpenSMTPd as MTA, using mda delivery to rspamc which >> utlimately delivers using dovecot-lda. >> >> smtpd.conf >> deliver to mda "rspamc -h scan --mime -e >> \"/usr/local/libexec/dovecot/deliver -d %{user.username}\"" >> %{user.username} is the local user after virtusers, aliases etc. >> verified using a shell wrapper and capturing the username. >> >> conf.d/15-lda.conf >> protocol lda { >> mail_plugins = $mail_plugins sieve >> } >> >> >> Mail gets delivered. >> >> Don't understand why it is looking for a <>-address if sieve is enabled. > > I found the cause of the panic. However, what actually triggers it could > be a little more nasty. Do you have an example of a message passed to > dovecot-lda that causes the problem? I am particularly interested in the > Return-Path header. > > Regards, > > Stephan. >
Re: two unrelated issues, lastlogin, and an out of memory fatal error
On 11.04.18 18:23, David Mehler wrote: Hello Aki, Thanks for your suggestion. I followed it and the wiki at: https://wiki.dovecot.org/Plugins/LastLogin Good news, I get a record in the users table, with the username that just logged in. This is good. Bad news is I got unexpected output from the last_login time stamp. Here's the table and the user: mysql> describe users; ++--+--+-+-+---+ | Field | Type | Null | Key | Default | Extra | ++--+--+-+-+---+ | userid | varchar(255) | NO | PRI | NULL| | | last_login | int(11) | YES | | NULL| | ++--+--+-+-+---+ 2 rows in set (0.00 sec) mysql> select * from users; +++ | userid | last_login | +++ | u...@domain.com | 1523459718 | +++ 1 row in set (0.01 sec) The issue is the 1523459718 I was expecting something like a time stamp. Is this fixable? Also, can I use last_login to see on which IP the user last logged in from? SELECT FROM_UNIXTIME(1523459718) Maybe that will help? Good luck, Reio
Re: two unrelated issues, lastlogin, and an out of memory fatal error
* David Mehler, 2018-04-11 17:23 > [...] > The issue is the 1523459718 I was expecting something like a time > stamp. Is this fixable? Also, can I use last_login to see on which IP > the user last logged in from? > [...] This is in fact a timestamp: [zlatko@disclosure:~]$ date -d @1523459718 Mit Apr 11 17:15:18 CEST 2018 Your output might look different depending on your locale. HTH, Thomas
Re: two unrelated issues, lastlogin, and an out of memory fatal error
Hello Aki, Thanks for your suggestion. I followed it and the wiki at: https://wiki.dovecot.org/Plugins/LastLogin Good news, I get a record in the users table, with the username that just logged in. This is good. Bad news is I got unexpected output from the last_login time stamp. Here's the table and the user: mysql> describe users; ++--+--+-+-+---+ | Field | Type | Null | Key | Default | Extra | ++--+--+-+-+---+ | userid | varchar(255) | NO | PRI | NULL| | | last_login | int(11) | YES | | NULL| | ++--+--+-+-+---+ 2 rows in set (0.00 sec) mysql> select * from users; +++ | userid | last_login | +++ | u...@domain.com | 1523459718 | +++ 1 row in set (0.01 sec) The issue is the 1523459718 I was expecting something like a time stamp. Is this fixable? Also, can I use last_login to see on which IP the user last logged in from? Any ideas on my other issue, the one on the running out of memory when attempting to access the imported gmail messages? Thanks for your help so far. Thanks. Dave. On 4/11/18, Aki Tuomi wrote: > Hi! > > > On 10.04.2018 19:31, David Mehler wrote: >> Hello, >> >> I've got a few misc issues that I'm trying to get a handle on. >> >> I'm having what is probably a simple error with last logins. It's not >> working, unable to write values to the database. Here's the log lines: >> >> 2018-04-10 10:43:34 imap-login: Info: Login: user=, >> method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=75934, TLS, >> session= >> 2018-04-10 10:43:34 imap(u...@domain.com)<75934>: >> Error: last_login_dict: Failed to write value for user >> u...@domain.com: dict-server returned failure: sql dict: commit >> failed: Field 'name' doesn't have a default value (reply took 0.013 >> secs (0.000 in dict wait, 0.012 in other ioloops, 0.000 in locks, >> async-id reply 0.002 secs ago, started on dict-server 0.005 secs ago, >> took 0.005 secs)) > It's attempting to insert a new row there for some reason. I'd recommend > using separate table for last-login values. > >> Here's an excerpt from my dovecot-dict-sql.conf.ext file: >> >> map { >> pattern = shared/last-login/$user >> table = accounts >> value_field = last_login >> value_type = uint >> >> fields { >> username = $user >> } >> } >> >> and the relevant database table: >> describe accounts; >> ++--+--+-+-++ >> >> | Field | Type | Null | Key | Default | Extra | >> >> ++--+--+-+-++ >> >> | id | int(10) unsigned | NO | PRI | NULL| auto_increment | >> >> | name | varchar(255) | NO | | NULL|| >> >> | username | varchar(64) | NO | MUL | NULL|| >> >> | domain | varchar(255) | NO | MUL | NULL|| >> >> | password | varchar(255) | NO | | NULL|| >> >> | quota | int(10) unsigned | YES | | 0 || >> >> | enabled| tinyint(1) | YES | | 0 || >> >> | sendonly | tinyint(1) | YES | | 0 || >> >> | last_login | int(11) | YES | | NULL|| >> >> >> Second issue, I've used imapsync to download my gmail messages to >> dovecot. I am atempting to access these messages and getting: >> 2018-04-08 11:11:58 >> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:StoZDD4xylqhrAAA0J78UA>: >> Info: Indexed 0 messages in INBOX >> 2018-04-08 11:11:58 >> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:0AaXFD4xylqhrAAA0J78UA>: >> Info: Indexed 0 messages in INBOX >> 2018-04-08 11:11:58 >> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:CP3TIT4xylqhrAAA0J78UA>: >> Info: Indexed 0 messages in INBOX >> 2018-04-08 11:11:58 >> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:eNEWMT4xylqhrAAA0J78UA>: >> Info: Indexed 0 messages in INBOX >> 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>: >> Fatal: block_alloc(8388608): Out of memory >> 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>: >> Fatal: master: service(imap): child 21374 returned error 83 (Out of >> memory (service imap { vsz_limit=256 MB }, you may need to increase >> it) - set CORE_OUTOFMEM=1 environment to get core dump) >> >> when I atempt to do so. For completeness the imapsync command I used was: >> >> imapsync --host1 imap.gmail.com --user1 Username --password1 Password >> --host2 localhost --user2 username --password2 Password --ssl1 >> --useheader 'Message-Id' --regextrans2 's/\[Gmail\]/user\@domain/' >> > > Increase memory, your cache takes more memo
Re: two unrelated issues, lastlogin, and an out of memory fatal error
Hi! On 10.04.2018 19:31, David Mehler wrote: > Hello, > > I've got a few misc issues that I'm trying to get a handle on. > > I'm having what is probably a simple error with last logins. It's not > working, unable to write values to the database. Here's the log lines: > > 2018-04-10 10:43:34 imap-login: Info: Login: user=, > method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=75934, TLS, > session= > 2018-04-10 10:43:34 imap(u...@domain.com)<75934>: > Error: last_login_dict: Failed to write value for user > u...@domain.com: dict-server returned failure: sql dict: commit > failed: Field 'name' doesn't have a default value (reply took 0.013 > secs (0.000 in dict wait, 0.012 in other ioloops, 0.000 in locks, > async-id reply 0.002 secs ago, started on dict-server 0.005 secs ago, > took 0.005 secs)) It's attempting to insert a new row there for some reason. I'd recommend using separate table for last-login values. > Here's an excerpt from my dovecot-dict-sql.conf.ext file: > > map { > pattern = shared/last-login/$user > table = accounts > value_field = last_login > value_type = uint > > fields { > username = $user > } > } > > and the relevant database table: > describe accounts; > ++--+--+-+-++ > > | Field | Type | Null | Key | Default | Extra | > > ++--+--+-+-++ > > | id | int(10) unsigned | NO | PRI | NULL| auto_increment | > > | name | varchar(255) | NO | | NULL|| > > | username | varchar(64) | NO | MUL | NULL|| > > | domain | varchar(255) | NO | MUL | NULL|| > > | password | varchar(255) | NO | | NULL|| > > | quota | int(10) unsigned | YES | | 0 || > > | enabled| tinyint(1) | YES | | 0 || > > | sendonly | tinyint(1) | YES | | 0 || > > | last_login | int(11) | YES | | NULL|| > > > Second issue, I've used imapsync to download my gmail messages to > dovecot. I am atempting to access these messages and getting: > 2018-04-08 11:11:58 > indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:StoZDD4xylqhrAAA0J78UA>: > Info: Indexed 0 messages in INBOX > 2018-04-08 11:11:58 > indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:0AaXFD4xylqhrAAA0J78UA>: > Info: Indexed 0 messages in INBOX > 2018-04-08 11:11:58 > indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:CP3TIT4xylqhrAAA0J78UA>: > Info: Indexed 0 messages in INBOX > 2018-04-08 11:11:58 > indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:eNEWMT4xylqhrAAA0J78UA>: > Info: Indexed 0 messages in INBOX > 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>: > Fatal: block_alloc(8388608): Out of memory > 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>: > Fatal: master: service(imap): child 21374 returned error 83 (Out of > memory (service imap { vsz_limit=256 MB }, you may need to increase > it) - set CORE_OUTOFMEM=1 environment to get core dump) > > when I atempt to do so. For completeness the imapsync command I used was: > > imapsync --host1 imap.gmail.com --user1 Username --password1 Password > --host2 localhost --user2 username --password2 Password --ssl1 > --useheader 'Message-Id' --regextrans2 's/\[Gmail\]/user\@domain/' > Increase memory, your cache takes more memory than you have anticipated. > Any help appreciated on either issue. > Thanks. > Dave. > > doveconf -n > # 2.3.1 (8e2f634): /usr/local/etc/dovecot/dovecot.conf > # Pigeonhole version 0.5.1 (d9bc6dfe) > # OS: FreeBSD 11.1-RELEASE-p4 amd64 > # Hostname: localhost > auth_cache_size = 24 M > auth_cache_ttl = 18 hours > auth_default_realm = domain.com > auth_mechanisms = plain login > dict { > acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext > lastlogin = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext > quota = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext > } > first_valid_gid = 999 > first_valid_uid = 999 > hostname = .. > imap_idle_notify_interval = 10 mins > last_valid_gid = 999 > last_valid_uid = 999 > lda_mailbox_autocreate = yes > lda_mailbox_autosubscribe = yes > lda_original_recipient_header = X-Original-To > listen = 127.0.0.1 xxx.xxx.xxx.xxx > log_path = /var/log/dovecot/dovecot.log > log_timestamp = "%Y-%m-%d %H:%M:%S " > mail_access_groups = vmail > mail_gid = vmail > mail_home = /home/vmail/mailboxes/%d/%n > mail_location = maildir:~/mail:LAYOUT=fs > mail_plugins = acl mail_log notify quota trash virtual welcome zlib > mail_privileged_group = vmail > mail_server_admin = mailto:postmas...@davemehler.com > mail_uid = vmail > mailbox_idle_check_interval = 59 secs > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > e