Re: index worker 2.3.7 undefined symbol errors

[Dirk Koopman via dovecot]

On 22/07/2019 16:24, Timo Sirainen wrote:

On 19 Jul 2019, at 13.20, Dirk Koopman via dovecot  wrote:

But I am left with this:

Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User  lookup 
failed: Couldn't load required plugin /usr/lib/dovecot/modules/lib90_sieve_plugin.so: 
dlopen() faile: /usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: 
mail_deliver_ctx_get_log_var_expand_table

Oh.. it's logged by indexer-worker.


mail_plugins = mail_log notify replication fts fts_lucene sieve

You can't load sieve globally. It needs to be inside protocol lmtp {}



That has fixed it, thank you.

It's not obvious from the documentation which are global plugins and 
which are protocol based ones. Would it be possible to put some words on 
plugin pages to indicate which is what?


Thanks for the help

Dirk

Re: index worker 2.3.7 undefined symbol errors (more info)

[Dirk Koopman via dovecot]

On 22/07/2019 13:32, Timo Sirainen via dovecot wrote:
On 21 Jul 2019, at 23.14, Dirk Koopman via dovecot 
mailto:dovecot@dovecot.org>> wrote:


Some supplemental information:

This is happening on every email delivered into Dovecot via LMTP. The 
curious things are that the message is a) successfully delivered and 
b) sieved into the correct directory.


Another observation is that:

mail_deliver_ctx_get_log_var_expand_table

is defined globally in core/src/lib-lda/mail-deliver.c (and used 
there) but the ONLY external call in the entire dovecot tree is from 
pigeonhole/src/plugins/lda-sieve/lda-sieve-log.c.


I am not using lda but it seems to be part of core. So, as I am only 
using lmtp, why is pigeonhole using lda-sieve at all?


It's part of lib-lda, which is also used by lmtp.

Can I work around the error message by some config magic (as I did by 
calling the correct plugin for imap_sieve) or is this an actual bug? 
Could this be fixed simply by including mail-deliver.h in 
lda-sieve-log.c?


I think you're not linking lmtp binary correctly somehow. That symbol 
should be part of it:


% nm /usr/libexec/dovecot/lmtp | grep 
mail_deliver_ctx_get_log_var_expand_table

00061960 T mail_deliver_ctx_get_log_var_expand_table



I agree, but I am using the ubuntu packages as provided by yourselves 
(not Ubuntu's). I'm guessing this must be some kind of linking/packaging 
issue as I have been using your packages for some years now and have not 
seen this particular error before. It only started appearing after I 
upgraded to 2.3.7 on July 19th.


All my executables in /usr/lib/dovecot/ are stripped so I cannot check 
my versions.


Dirk

Re: index worker 2.3.7 undefined symbol errors (more info)

[Dirk Koopman via dovecot]

Some supplemental information:

This is happening on every email delivered into Dovecot via LMTP. The 
curious things are that the message is a) successfully delivered and b) 
sieved into the correct directory.


Another observation is that:

mail_deliver_ctx_get_log_var_expand_table

is defined globally in core/src/lib-lda/mail-deliver.c (and used there) 
but the ONLY external call in the entire dovecot tree is from 
pigeonhole/src/plugins/lda-sieve/lda-sieve-log.c.


I am not using lda but it seems to be part of core. So, as I am only 
using lmtp, why is pigeonhole using lda-sieve at all?


Can I work around the error message by some config magic (as I did by 
calling the correct plugin for imap_sieve) or is this an actual bug? 
Could this be fixed simply by including mail-deliver.h in lda-sieve-log.c?


Dirk

On 19/07/2019 13:20, Dirk Koopman via dovecot wrote:

Suddenly I am getting undefined symbol errors having updated to 2.3.7.

Jul 18 01:02:49 localhost dovecot: indexer-worker: Error: User 
 lookup failed: Couldn't load required plugin 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: dlopen() failed: 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: undefined symbol: 
command_hook_register


(I think I fixed this one by a config change)

But I am left with this:

Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User 
 lookup failed: Couldn't load required plugin 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile: 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: 
mail_deliver_ctx_get_log_var_expand_table


My (redacted) dovecot -n:

# 2.3.7 (494d20bdc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7 (5a4e63b5)
# OS: Linux 4.15.0-54-generic x86_64 Ubuntu 18.04.2 LTS
# Hostname: localhost
auth_verbose = yes
first_valid_uid = 2000
imapc_features = rfc822.size fetch-headers
imapc_host = mailman.tobit.co.uk
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no
imapc_user = %n
lda_mailbox_autocreate = yes
mail_attachment_dir = %h/attachments
mail_attachment_min_size = 64 k
mail_location = mdbox:%h:INBOX=%h/mail
mail_plugins = mail_log notify replication fts fts_lucene sieve
mail_prefetch_count = 20
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext imapsieve vnd.dovecot.imapsieve

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = # hidden
  driver = sql
}
plugin {
  fts = lucene
  fts_autoindex = yes
  fts_autoindex_exclude = \Junk
  fts_autoindex_exclude2 = \Trash
  fts_autoindex_exclude3 = \Deleted
  fts_lucene = whitespace_chars=@.
  imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  mail_log_events = delete undelete expunge copy save mailbox_delete 
mailbox_rename mailbox_create

  mail_log_fields = uid box msgid size
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  sieve_pipe_bin_dir = /etc/dovecot/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_trace_dir = /tmp
  sieve_trace_level = matching
}
postmaster_address = postmas...@tobit.co.uk
protocols = imap lmtp sieve
service auth {
  unix_listener exim-auth {
    group = Debian-exim
    mode = 0660
    user = Debian-exim
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}
service lmtp {
  inet_listener lmtp {
    address = 127.0.0.1
    port = # hidden
  }
  unix_listener lmtp {
    mode = 0666
  }
  user = vmail
}
service managesieve-login {
  inet_listener sieve {
    port = # hidden
  }
  process_min_avail = 0
  service_count = 1
  vsz_limit = 64 M
}
service managesieve {
  process_limit = 100
}
ssl_cert = # hidden
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  args = # hidden
  driver = sql
}
protocol imap {
  mail_plugins = mail_log notify replication fts fts_lucene imap_sieve
}
protocol lmtp {
  mail_plugins = fts fts_lucene sieve
}

index worker 2.3.7 undefined symbol errors

[Dirk Koopman via dovecot]

Suddenly I am getting undefined symbol errors having updated to 2.3.7.

Jul 18 01:02:49 localhost dovecot: indexer-worker: Error: User  
lookup failed: Couldn't load required plugin 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: dlopen() failed: 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: undefined symbol: 
command_hook_register


(I think I fixed this one by a config change)

But I am left with this:

Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User  
lookup failed: Couldn't load required plugin 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile: 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: 
mail_deliver_ctx_get_log_var_expand_table


My (redacted) dovecot -n:

# 2.3.7 (494d20bdc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7 (5a4e63b5)
# OS: Linux 4.15.0-54-generic x86_64 Ubuntu 18.04.2 LTS
# Hostname: localhost
auth_verbose = yes
first_valid_uid = 2000
imapc_features = rfc822.size fetch-headers
imapc_host = mailman.tobit.co.uk
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no
imapc_user = %n
lda_mailbox_autocreate = yes
mail_attachment_dir = %h/attachments
mail_attachment_min_size = 64 k
mail_location = mdbox:%h:INBOX=%h/mail
mail_plugins = mail_log notify replication fts fts_lucene sieve
mail_prefetch_count = 20
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext imapsieve vnd.dovecot.imapsieve

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = # hidden
  driver = sql
}
plugin {
  fts = lucene
  fts_autoindex = yes
  fts_autoindex_exclude = \Junk
  fts_autoindex_exclude2 = \Trash
  fts_autoindex_exclude3 = \Deleted
  fts_lucene = whitespace_chars=@.
  imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  mail_log_events = delete undelete expunge copy save mailbox_delete 
mailbox_rename mailbox_create

  mail_log_fields = uid box msgid size
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  sieve_pipe_bin_dir = /etc/dovecot/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_trace_dir = /tmp
  sieve_trace_level = matching
}
postmaster_address = postmas...@tobit.co.uk
protocols = imap lmtp sieve
service auth {
  unix_listener exim-auth {
    group = Debian-exim
    mode = 0660
    user = Debian-exim
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}
service lmtp {
  inet_listener lmtp {
    address = 127.0.0.1
    port = # hidden
  }
  unix_listener lmtp {
    mode = 0666
  }
  user = vmail
}
service managesieve-login {
  inet_listener sieve {
    port = # hidden
  }
  process_min_avail = 0
  service_count = 1
  vsz_limit = 64 M
}
service managesieve {
  process_limit = 100
}
ssl_cert = # hidden
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  args = # hidden
  driver = sql
}
protocol imap {
  mail_plugins = mail_log notify replication fts fts_lucene imap_sieve
}
protocol lmtp {
  mail_plugins = fts fts_lucene sieve
}

Searching small(ish) sites on ubuntu

[Dirk Koopman]

I run a small dovecot site with ~5 users, some of whom have many large 
mailboxes. I used to use lucene until the packagers decided that it 
broke some licencing rules and stopped including it. I have dismally 
failed to configure Solr, plus it is huge and potentially exposes too 
many services as well. Squash is deprecated (but still used?) in 2.2+. I 
am currently on 2.2.33.


So what can I use? Or do I have to bite the bullet and install from 
source, thus losing the advantages of packaging and automatic updates?


Dirk

Re: Dovecot user lookup from sendmail

[Dirk Koopman]

Use exim instead of sendmail with an LDA or LMTP transport.


On 04/07/17 15:41, Garry Glendown wrote:

Hi *

after trying everything I could think of and find on the web, I'm
totally stumped ... maybe someone give me some hints on what I'm doing
wrong, or where to look ...

I have an ancient customer box with sendmail & cyrus imap, with imap
using an LDAP database of users (no locally configured users), all of
which is working fine.

This machine needs to be replaced by a new installation, which -
regrettably - is supposed to use SLES 12. Which does not have any Cyrus
IMAP packages available anymore, so I had to use the Dovecot imap
instead. I've already got the LDAP auth working, as well as the imap
storage of mails (used imapsync to copy all of the old mailboxes to the
new machine), everything fine so far.

Anyway, I'm running into problems getting sendmail to check whether a
recipient actually exists. Any local user is correctly identified, but
all virtual users are constantly rejected. I tried finding anything on
how sendmail actually does the user check ... on the cyrus install, all
I had to do is

define(`confLOCAL_MAILER',`cyrusv2')dnl
define(`CYRUSV2_MAILER_ARGS',`FILE /var/lib/imap/socket/lmtp')dnl
MAILER(cyrusv2)dnl

and both user checks and local delivery worked. What's the appropriate
config entry in the .mc for dovecot?

I tried settings like:

FEATURE(`local_lmtp',`[IPC]',`FILE /var/run/dovecot/lmtp')dnl

and used the dovecot.m4 file together with

MAILER(`dovecot')dnl

but couldn't see anything happening - or improve.

Hints and help greatly appreciated!


-garry

fsck looping duplicate mdbox GUIDs

[Dirk Koopman]

I have recently updated with 2.2.29.1-avh1~xenial1 ubuntu package on 16.04.

It appears that I have a couple of duplicate GUIDs which fsck can't 
clear so the machine is looping through doing continuous fsck runs.


Any way to determine what the offending messages are and how to delete 
or otherwise stop this?


Jul  1 17:09:24 post dovecot: doveadm(d...@tobit.co.uk): Warning: fscking 
index file /home/imap/djk/storage/dovecot.map.index
Jul  1 17:09:24 post dovecot: doveadm(d...@tobit.co.uk): Error: Fixed 
index file /home/imap/djk/storage/dovecot.map.index: 
log_file_tail_offset 0 -> 24
Jul  1 17:09:24 post dovecot: dsync-local(djk): Warning: fscking index 
file /home/imap/djk/storage/dovecot.map.index
Jul  1 17:09:24 post dovecot: doveadm(d...@tobit.co.uk): Warning: fscking 
index file /home/imap/djk/storage/dovecot.map.index
Jul  1 17:09:25 post dovecot: doveadm(d...@tobit.co.uk): Warning: mdbox 
/home/imap/djk/storage: rebuilding indexes
Jul  1 17:10:15 post dovecot: dsync-local(djk): expunge: box=mup-users, 
uid=1528, msgid=<48707d70.9070...@optusnet.com.au>, size=3159
Jul  1 17:11:00 post dovecot: doveadm(d...@tobit.co.uk): Error: mdbox 
/home/imap/djk/storage: Duplicate GUID a8e15732b4f91b55414ff80b9cab 
in m.385:10102880 (size=860) and m.100:3388046 (size=861)
Jul  1 17:11:11 post dovecot: doveadm(d...@tobit.co.uk): Error: mdbox 
/home/imap/djk/storage: Duplicate GUID 67d2801030161c557a71f80b9cab 
in m.389:363061 (size=895) and m.169:5349813 (size=896)
Jul  1 17:11:43 post dovecot: doveadm(d...@tobit.co.uk): Warning: 
Transaction log file /home/imap/djk/storage/dovecot.map.index.log was 
locked for 139 seconds (mdbox storage rebuild)
Jul  1 17:11:43 post dovecot: dsync-local(djk): Warning: Locking 
transaction log file /home/imap/djk/storage/dovecot.map.index.log took 
88 seconds (syncing)
Jul  1 17:11:43 post dovecot: doveadm(d...@tobit.co.uk): Warning: fscking 
index file /home/imap/djk/storage/dovecot.map.index


Dirk

Re: overview zlib efficiency?

[Dirk Koopman]

On 15/03/16 22:07, micah wrote:

Andrew McGlashan  writes:


On 16/03/2016 2:01 AM, Götz Reinicke - IT Koordinator wrote:

Hi,

may be someone has already done that: Do you have a script(?) tool which
shows the efficiency of the mail compression if zlib is used?

Something that shows the uncompressed size vrs. the compressed.

Remember one thing; emails are stored in plain text, the same text that
they are normally transmitted b/w servers.

Emails are not stored in plaintext if you are using dbox/mdbox.


They are on my machine. The fact that (m)dbox puts some "binary" records 
around those emails doesn't stop the files being (human) readable and 
the text extractable.