Segfaults after upgrade to Debian Jessie

[Andrew Beverley]

Hi,

I've just upgrade from Debian Wheezy to Debian Jessie and am getting:

Fatal: master: service(lmtp): child 6761 killed with signal 11 (core dumped)

It seems to be something to do with sieve. When I disable that from lmtp then
everything works fine.

OS: Debian Jessie
Dovecot version: 2.2.13
CPU: x86

This is the gdb output:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/lmtp'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f8e4c94f626 in sieve_validator_register_command () from 
/usr/lib/dovecot/libdovecot-sieve.so.0

And dovecot -n

# 2.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 8.1 ext3
auth_mechanisms = plain login
dict {
  sieve = mysql:/etc/dovecot/pigeonhole-sieve.dict
}
mail_location = maildir:/var/mail/vhosts/%d/%n
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date ihave 
vnd.dovecot.extdata
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix = 
}
passdb {
  driver = pam
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  sieve = dict:proxy::sieve;name=active
  sieve_dir = ~/sieve
  sieve_extdata_dict_uri = proxy::sieve
  sieve_plugins = sieve_extdata
}
protocols = " imap lmtp sieve"
service auth-worker {
  user = vmail
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
  }
  unix_listener auth-userdb {
mode = 0600
user = vmail
  }
  user = dovecot
}
service dict {
  unix_listener dict {
mode = 0600
user = vmail
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
  }
}
ssl = required
ssl_cert = 

Re: [Dovecot] Dovecot FTS using ElasticSearch

[Tim Groeneveld]

Hey James,

I did make progress, however, I have dropped the code that I was working on
in favor of Joshua Atkins' work on fts-elasticsearch.

Check it out here: https://github.com/ascendantcom/fts-elasticsearch/

Regards,
Tim

 On Tue, 26 Aug 2014 03:09:12 +1000 James Devine wrote 
 
 > Tim, 
 >  
 > Had you made any progress on this?  We've been using elasticsearch to index 
 > our mail backup for a while now and I am interested in possibly using it to 
 > index our live mail as well. 
 >  
 >  
 > On Wed, Jun 26, 2013 at 11:14 PM, Tim Groeneveld  wrote: 
 >  
 > > 
 > > - Original Message - 
 > > > I would like to know if is possible to use ElasticSearch instead of 
 > > > Solr for FTS. 
 > > 
 > > I have started work on an ElasticSearch implementation based on fts-solr. 
 > > 
 > > There is still around 20-30 hours more work for me to do until it is 
 > > complete 
 > > (and I need to hunt down the reason for a random crash that happens every 
 > > now and then) but I would be more then happy to share the code with you 
 > > when I am done if you are interested? 
 > > 
 > > Regards, 
 > > Tim 
 > > 
 > 

Re: POP3 getting previously delivered messages.

[Tim Dickson]

On 24/07/2015 10:37, Jas Reehal wrote:

Hi All,

This is my first post to this list. I have recently upgraded to 
dovecot version 2.2.10.


1. I have one POP3 user who does not want to delete his email on the
   server.
2. He has accumulated over 19000 messages, and the 'dovecot.uidlist'
   file has more than 19000 lines!
3. Occasionally now when he has received one or more new emails, and 
   his email client does a 'get new mail' operation (POP3)

   he begins to receive email he has already received previously.

last time this happened all 19000+ messages were delivered again, 
leaning to duplicate entries in all his mailboxes.


I have reproduced this issue with a dummy email account with a POP3 
client, leaving all messages on the server. I have delivered 2 
messages to this email account, each about 1.3k ins size and each  
with 


  Subject: Message number 
  Body containing"Message number " 
where  is 1 to 2


now occasionally if I send another message to this email account, and 
then check for new messages using a POP3 client I get more messages 
downloaded than I expect!


The 'dovecot.uidlist' file is not corrupt, but I beleive due ti it's 
size (number of lines) the is an issue on compiling a
' POP3 drop-list'. I have attached a file 'dovecot-a.txt' which shows 
the  'dovecot -a'  output from my server.


Based on what I have investigated I believe that this issue may not be 
correlated with a migration to version 2.2.10, but more to do with the 
number of messages retained on the server.


Is there anyone who has any insight into what is happening and if 
there is any remedy, perhaps a "pop3_client_workarounds =" or other 
POP3 or UIDL directive in the conf files?


 The user is stubbornly resisting migration to IMAP, because of the 
time and effort he says will be required to replicate his organised 
filing/storage of messages on the server.


not a fix for the technical issue, but it is very easy to delete the 
messages on the server, create an imap account on the same mua (email 
client) and then drag the emails/email folders from the pop account to 
the imap account. This will take a little while as the client instructs 
the imap server to recreate the structure server side. if you are using 
maildir format it is easy enough to check that the structure has been 
created properly. Then the pop account can be removed from the client, 
and his/her email directory structure has been preserved.
I used this method when migrating users to dovecot using imap from their 
isp using pop, when using thunderbird email client, and one of those 
users has over 24000 emails.
if using a different email client it is worth experimenting on a test 
account first.

It is also a good idea to backup the emails first "just in case".
regards, Tim

Kind Regards  Jas Reehal

POP3 getting previously delivered messages.

[Jas Reehal]

Hi All,

This is my first post to this list. I have recently upgraded to dovecot 
version 2.2.10.


1. I have one POP3 user who does not want to delete his email on the
   server.
2. He has accumulated over 19000 messages, and the 'dovecot.uidlist'
   file has more than 19000 lines!
3. Occasionally now when he has received one or more new emails, and 
   his email client does a 'get new mail' operation (POP3)

   he begins to receive email he has already received previously.

last time this happened all 19000+ messages were delivered again, 
leaning to duplicate entries in all his mailboxes.


I have reproduced this issue with a dummy email account with a POP3 
client, leaving all messages on the server. I have delivered 2 
messages to this email account, each about 1.3k ins size and each  with 



  Subject: Message number 
  Body containing"Message number " 
where  is 1 to 2


now occasionally if I send another message to this email account, and 
then check for new messages using a POP3 client I get more messages 
downloaded than I expect!


The 'dovecot.uidlist' file is not corrupt, but I beleive due ti it's 
size (number of lines) the is an issue on compiling a
' POP3 drop-list'. I have attached a file 'dovecot-a.txt' which shows 
the  'dovecot -a'  output from my server.


Based on what I have investigated I believe that this issue may not be 
correlated with a migration to version 2.2.10, but more to do with the 
number of messages retained on the server.


Is there anyone who has any insight into what is happening and if there 
is any remedy, perhaps a "pop3_client_workarounds =" or other POP3 or 
UIDL directive in the conf files?


 The user is stubbornly resisting migration to IMAP, because of the 
time and effort he says will be required to replicate his organised 
filing/storage of messages on the server.


Kind Regards  Jas Reehal


--
Jas Reehal | IT & Networks Consultant

*Zuken Limited - /The Partner for Success/*
1500 Aztec West, Almondsbury, Bristol, BS32 4RF, United Kingdom
T: +44 1454 207824 | F: +44 1454 207803
E: jas.ree...@zuken.co.uk  | W: 
www.zuken.com 


Connect with Zuken: blog  | youtube 
 | linkedin 
 | twitter  
| facebook 


Zuken Ltd // Registered office: Bristol // Registered No. 956340 // 
Directors: Jinya Katsube, Kazuhiro Kariya, S Yukawa


This e-mail message from Zuken including any attachments is intended 
only for the individual or entity to which it is addressed. This e-mail 
may contain privileged, proprietary, or otherwise private information. 
If you are not the intended recipient of this e-mail, you are hereby 
notified that any dissemination, distribution or copying of this 
communication is strictly prohibited. If you received this e-mail by 
accident, please notify the sender immediately and destroy this e-mail 
and all copies of it.


# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-229.1.2.el7.x86_64 x86_64 Red Hat Enterprise Linux Server 
release 7.1 (Maipo) 
# NOTE: Send doveconf -n output instead when asking for help.
auth_anonymous_username = anonymous
auth_cache_negative_ttl = 1 hours
auth_cache_size = 0
auth_cache_ttl = 1 hours
auth_debug = no
auth_debug_passwords = no
auth_default_realm = 
auth_failure_delay = 2 secs
auth_gssapi_hostname = 
auth_krb5_keytab = 
auth_master_user_separator = 
auth_mechanisms = plain
auth_proxy_self = 
auth_realms = 
auth_socket_path = auth-userdb
auth_ssl_require_client_cert = no
auth_ssl_username_from_cert = no
auth_use_winbind = no
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
auth_username_format = %Lu
auth_username_translation = 
auth_verbose = no
auth_verbose_passwords = no
auth_winbind_helper_path = /usr/bin/ntlm_auth
auth_worker_max_count = 30
base_dir = /var/run/dovecot
config_cache_size = 1 M
debug_log_path = 
default_client_limit = 1000
default_idle_kill = 1 mins
default_internal_user = dovecot
default_login_user = dovenull
default_process_limit = 200
default_vsz_limit = 256 M
deliver_log_format = msgid=%m: %$
dict_db_config = 
director_doveadm_port = 0
director_mail_servers = 
director_servers = 
director_user_expire = 15 mins
director_username_hash = %Lu
disable_plaintext_auth = yes
dotlock_use_excl = yes
doveadm_allowed_commands = 
doveadm_password = 
doveadm_port = 0
doveadm_socket_path = doveadm-server
doveadm_worker_count = 0
dsync_alt_char = _
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -U
first_valid_gid = 1
first_valid_uid = 100
hostname = 
imap_capability = 
imap_client_workarounds = 
imap_id_log = 
imap_id_send = name *
imap_idle_notify_interval = 2 mins
imap_logout_format = in=%i out=%o
imap_max_line_length = 64 k
imap_metadata = no
imap_urlauth_host = 
imap_urlauth_logout_f

possible patch for ACL SQL ERROR: duplicate key value violates unique constraint "user_shares_pkey"

[Steffen Kaiser]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I'm using Postgres as dict backend now and then I get this error, when 
someone changed ACLs, actually for some reason some users, who get 
permission from other users, get this error quite often:


dict: Error: sql dict: commit failed:
ERROR:  duplicate key value violates unique constraint "user_shares_pkey"
(query: INSERT INTO user_shares (dummy,to_user,from_user) VALUES 
('1','user1','user2'))

It looks like that this error is triggered, when a MUA is connecting to 
Dovecot after ACLs had been changed and opens several mailboxes at once. 
So two or parallel Dovecot instances try to insert the record 
simultaneously.


With a similiar technique to the quota problem, I added a trigger. The 
idea is to use the condition dummy ISNULL to detect, that the function was 
triggered from the INSERT of the function itself, hence, breaking the 
recursive triggers. This assumes that dummy is never NULL from Dovecot.


The same applies to anyone_shares, just remove the to_user field from the 
function.


Does somebody see something wrong with my hack?

=

CREATE FUNCTION insert_sharing() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
  IF NEW.dummy IS NULL THEN
-- ugly kludge: we came here from this function, really do try to insert
return NEW;
  END IF;

  LOOP
UPDATE user_shares SET dummy = NEW.dummy
  WHERE from_user = NEW.from_user AND to_user = NEW.to_user;
IF found THEN
  -- updated and inserted
  RETURN NULL;
END IF;

BEGIN
  INSERT INTO user_shares (from_user,to_user,dummy) VALUES 
(NEW.from_user,NEW.to_user,NULL);

  --- return NULL;
EXCEPTION WHEN unique_violation THEN
  -- someone just inserted the record, update it
END;
-- Update dummy after INSERT
  END LOOP;
END;
$$;

CREATE TRIGGER insertsharing
BEFORE INSERT ON user_shares
FOR EACH ROW
EXECUTE PROCEDURE insert_sharing();

- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBVbHxJXz1H7kL/d9rAQIn5gf/dNMQKmmigGrbccGQQrsGU66B6OHvTY7L
1vyh8hgr985KNk2/Ws/usLnAqpl5jhpo8kVPLI8TqcGYLAS/67TOIFrwLKAEBU96
e+QpbqqiSm66mZwWOQZgCI63+Fh7fdBUMREL6UqGleFF1GrzNnjmX4MGecbpu/L9
gQMAGUG++/S+ka9S6B275RXIj9DvWzq/oYqJ1M1grGckUxXfDk5szLyvuIJ981At
20qaCHEIlIOlkQ0ttE4W+RWkkYS2rMJvCzQelF4pcm6vaxhXxvqUryzmy9pv+UUT
osqS6igWt7BPXF7ku8+rKYVp/qPZrh9IWXMsajr/2nI+V5gRb6u2ow==
=gb1P
-END PGP SIGNATURE-