Replication: "cross-updates" of mail meta-data
Hello
I have a two-server dovecot setup using replication. Each server runs
two dovecot instances, one for director and another for the backend.
Initially I set up a single server, got it all working, then rsync'd the
data and index partitions to the new one and started the clusters (I
used rsync as a way to speed-up dovecot's initial replication). Both
servers listen on a virtual IP addresses load-balanced by IPVS as well
as their own IP addresses.
When I started dovecot in the second server, users started reporting
that some of their messages were missing. I noticed that while the mail
data seemed to be up to date in server1, it wasn't in server2; however,
the indexes were being updated in server2 but not in server1. For
example, "doveadm quota get -u some-user" would report the wrong value
in server1 and the right value in server2, while a "du" would show that
only server1 actually had the most up-to-date data.
At the moment I had to take down server2, because as soon as dovecot is
running in it, users start reporting missing messages again. To solve
that problem, a "doveadm force-resync" in server1 isn't enough. I either
rsync the index files from server2 or remove the indexes in server1
before the force-resync, which then works.
Has anyone ever saw anything similar to this? Any help would be much
appreciated.
I'm running Ubuntu 14.04 with dovecot 2.2.15 with the configurations below:
=== Backend instance ===
# dovecot -n
# 2.2.15: /etc/dovecot/dovecot.conf
# OS: Linux 3.16.0-37-generic x86_64 Ubuntu 14.04.2 LTS
auth_master_user_separator = *
auth_mechanisms = plain login
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@=
auth_worker_max_count = 10
disable_plaintext_auth = no
doveadm_password = secret
doveadm_port = 12345
listen = server1.mydomain.com
mail_fsync = never
mail_gid = dovemail
mail_plugins = " notify replication quota"
mail_uid = dovemail
managesieve_sieve_capability = fileinto reject envelope vacation
subaddress comparator-i;ascii-numeric relational regex copy include
vacation-seconds imapflags notify
namespace {
inbox = yes
location =
mailbox SPAM {
auto = subscribe
}
prefix =
separator = /
type = private
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-sql-master.conf.ext
driver = sql
master = yes
pass = yes
}
plugin {
mail_replica = tcp:192.168.0.2:12345 # server2's replication IP
quota = dict:User
quota::file:/var/lib/imap/user/%2.256RHn/%n/dovecot-quota
quota_rule = *:storage=1GB
sieve = /var/lib/imap/sieve/%2.256RHn/%n/dovecot.sieve
sieve_dir = /var/lib/imap/sieve/%2.256RHn/%n
sieve_extensions = comparator-i;ascii-numeric copy envelope fileinto
imapflags include notify regex reject relational subaddress vacation-seconds
sieve_max_redirects = 20
sieve_max_script_size = 512KB
sieve_quota_max_scripts = 2
sieve_vacation_min_period = 1
sieve_vacation_send_from_recipient = yes
}
protocols = imap pop3 lmtp sieve
replication_dsync_parameters = -f -d -N -l 30 -U
service aggregator {
fifo_listener replication-notify-fifo {
group = dovemail
mode = 0666
user = dovemail
}
unix_listener replication-notify {
group = dovemail
mode = 0666
user = dovemail
}
}
service auth-worker {
user = $default_internal_user
}
service auth {
client_limit = 10240
process_limit = 1
process_min_avail = 1
}
service doveadm {
inet_listener {
address = 192.168.0.1
port = 12345
}
vsz_limit = 1 G
}
service imap {
process_limit = 8192
process_min_avail = 10
}
service lmtp {
inet_listener {
address = lmtpcluster.internal.mydomain.com,
server1.internal.mydomain.com
port = 1124
}
process_limit = 512
process_min_avail = 10
}
service managesieve-login {
inet_listener sieve {
address = lmtpcluster.internal.mydomain.com,
server1.internal.mydomain.com
port = 4190
}
}
service managesieve {
process_limit = 20
process_min_avail = 10
}
service pop3 {
process_limit = 512
process_min_avail = 10
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
group = dovemail
mode = 0666
user = dovemail
}
}
ssl_cert =
signature.asc
Description: OpenPGP digital signature
[Dovecot] Error: write(auth socket) failed: Broken pipe
Hello >From time to time I'm seeing errors like this in my logs: Mar 16 10:45:55 box5 dovecot: lmtp(5438): Error: write(auth socket) failed: Broken pipe This causes delivery from postfix to fail until I restart dovecot. There doesn't seem to be any other messages in the logs related to this error. Has anyone ever seen this error? Thanks Andre
Re: [Dovecot] Best Cluster Storage
On Fri, 2011-01-14 at 03:48 +, Jonathan Tripathy wrote: > ocfs2 seems to work quite well with lots of small files (typical of > maildir). I'm guessing that since ocfs2 reboot a system automatically, > it doesn't require any additional fencing? We have a two-node active-active DRBD+OCFS2 Dovecot cluster. We're currently unable to fully use it due to (what I believe is) an OCFS2 bug: http://oss.oracle.com/bugzilla/show_bug.cgi?id=1297 so while DRBD is in a dual-primary setup and the dovecot volumes are mounted read/write on both cluster nodes, I had to remove one of them from the load balancer, and thus only one of them handles connections while the other is sitting there as a failover node. HTH, Andre
Re: [Dovecot] Indexes.
On Thu, 2010-12-02 at 13:40 +0100, Robert Schetterer wrote: > hm , i have no problems with ocfs2 (1.4.3-1: amd64 ) on drbd ubuntu lucid > using dovecot vers 2 recommended settings for cluster file systems > i have my index files in the maildir dir Robert, are you running an active-active setup with drbd+ocfs2? Have you ever seen errors like these in the logs? http://dovecot.markmail.org/message/u5qhqaguhqlsk2ui Thanks, Andre
Re: [Dovecot] Phantom email message
On Wed, 2010-11-10 at 11:59 -0800, Marc Perkel wrote: > Anyone have any ideas? This happens from time to time for me in Evolution. Moving a message from some folder to the folder displaying the phantom message seems to force it to recalculate and always fixes it here. HTH, Andre
Re: [Dovecot] Corrupted transaction log file
On Tue, 2010-10-26 at 23:52 -0200, Andre Nathan wrote: > On Tue, 2010-10-26 at 22:39 +0200, Timo Sirainen wrote: > > Have you set mmap_disable=yes? > > Yep. Just out of curiosity, is this setting really needed, or is it for performance reasons? OCFS2 claims to support mmap: http://www.oracle.com/us/technologies/linux/025995.htm Regards, Andre
Re: [Dovecot] Corrupted transaction log file
On Tue, 2010-10-26 at 23:52 -0200, Andre Nathan wrote: > They're not happening all the time. It showed up two or three times in > the logs during the night. A few errors of this kind appeared in the logs last night. The "Log synchronization error" appears more frequently than "Corrupted transaction log file", but I assume the former is a consequence of the latter. This really seems to be related to the shared accounts. Our load balancer makes no attempt to send the same users to the same server; it just sends connections based on server load. This allows two simultaneous connections to access an account, one on each server. Given a distributed FS, this should be OK, right? At least in theory it's no different than two CPU cores accessing the same account in a single server. Best regards, Andre
Re: [Dovecot] Corrupted transaction log file
On Tue, 2010-10-26 at 22:39 +0200, Timo Sirainen wrote: > Have you set mmap_disable=yes? Yep. > Those errors should fix themselves automatically. Still, it's not very good > if they keep happening. Sooner or later they will cause user visible problems. They're not happening all the time. It showed up two or three times in the logs during the night. Thanks, Andre
[Dovecot] Corrupted transaction log file
Hello I run dovecot 2.0.6 on a two-machine cluster using OCFS2 as the file system. I have some error messages like these in my log: Oct 25 01:07:50 box5 dovecot: lmtp(8886, suporte=100br@lmtp1.prv.f1.k8.com.br): Error: Corrupted transaction log file /var/lib/imap/user/a3/suporte=100br.com/stor age/dovecot.map.index.log seq 29: Transaction log corrupted unexpectedly at 21536: Invalid size 0 (type=0) (sync_offset=21908) Oct 25 01:07:50 box5 dovecot: lmtp(8886, suporte=100br@lmtp1.prv.f1.k8.com.br): Error: Index /var/lib/imap/user/a3/suporte=100br.com/storage/dovecot.map.index: Lost log for seq=29 offset=21796 Oct 25 01:07:55 box5 dovecot: lmtp(8885, suporte=100br@lmtp1.prv.f1.k8.com.br): Error: Log synchronization error at seq=30,offset=312 for /var/lib/imap/user/a3/su porte=100br.com/storage/dovecot.map.index: Append with UID 5404, but next_uid = 5405 Oct 25 02:37:53 box5 dovecot: lmtp(8885, suporte=hostnet.com...@lmtp1.prv.f1.k8.com.br): Error: Log synchronization error at seq=47,offset=22944 for /var/lib/imap/user/03/suporte=hostnet.com.br/storage/dovecot.map.index: Extension record update for invalid uid=21815 These two accounts happen to be accessed simultaneously by a number of people, but there are similar errors for "normal" accounts too, just not the same amount of logs. Is it not considered safe to do this kind of simultaneous access in a distributed filesystem like OCFS2? Is there a way to fix this error? Would a "doveadm force-resync" do it? Thanks, Andre
[Dovecot] Recommendation for OCFS2 filesystem type
Hello When creating an OCFS2 filesystem, one can specify a "filesystem type" option: -T filesystem-type Specify how the filesystem is going to be used, so that mkfs.ocfs2 can chose optimal filesystem parameters for that use. The supported filesystem types are: mail Appropriate for file systems which will have many meta data updates. Creates a larger journal. datafiles Appropriate for file systems which will host a rela‐ tively small number of very large files. A small journal is selected. Cluster size will be at least 128K. Considering I'm using a separate partition for the dovecot indices, and that I'm using mdbox, would it be more appropriate to use 'datafiles' for the mail location partition and 'mail' just for the indices? Or would 'mail' be appropriate for both even with the larger files created by mdbox? Thanks, Andre
Re: [Dovecot] Segfault in libdovecot-sieve
Hi I've upgraded to 2.0.6 and set the home directory. The segfaults are gone :) Thanks, Andre On Thu, 2010-10-21 at 22:13 +0100, Timo Sirainen wrote: > On 21.10.2010, at 23.14, Andre wrote: > > > This shows up in the mail error when the sieve error happens. It seems > > the sieve error is a side-effect of something else: > > > > Oct 21 17:58:11 box6 dovecot: lmtp(10424, > > catia.comercial=industriacarvalho.com...@lmtp1.prv.f1.k8.com.br): Error: > > lstat((null)) failed: Bad address > > This reminds me of a problem with missing home directory. That crash is fixed > in v2.0.6, but you should have a home directory set so that reject/vacation > messages won't be duplicated. http://wiki2.dovecot.org/VirtualUsers/Home >
Re: [Dovecot] Backup recovery with mdbox
On Mon, 2010-10-18 at 18:38 +0100, Timo Sirainen wrote: > Added to hg: > >This example imports all mails from a backup under a backup-20101026 >mailbox: > >doveadm import -u jane@example.org \ >mdbox:/backup/20101026/jane.doe/mdbox backup-20101026 all > >Another example that imports only messages from f...@example.org in > the >backup mdbox's INBOX to jane's INBOX: > >doveadm import -u jane@example.org \ >mdbox:~/mdbox-backup mailbox INBOX from f...@example.org Awesome :) Thanks a lot! Andre
Re: [Dovecot] Backup recovery with mdbox
On Wed, 2010-10-06 at 12:08 -0300, Andre Nathan wrote: > That worked, as long as I run a > > doveadm force-resync -u andre=sneakymustard.com INBOX > > after dsync, which is totally fine :) Hmm actually it doesn't work after a force-resync. The "backup-$date" mailbox is shown, with all it's sub-mailboxes as intended, but they're all empty. I found out, though, that if I run the dsync command twice the messages appear on the imap client. I get log messages like these on the second run: dsync(andre=sneakymustard.com): Info: SPAM: uidnext changed: 31 != 1 dsync(andre=sneakymustard.com): Info: INBOX: uidnext changed: 314 != 1 Is this what makes it work (it fixes the uids or something like that)? Does the second dsync do something that could cause the message listing to be fixed (apparently something that a force-resync doesn't do...)? I have no problem running the command twice, it's just a bit weird... Thanks, Andre
Re: [Dovecot] User authentication fields size limit
On Wed, 2010-10-06 at 16:09 -0300, Andre Nathan wrote: > Truncation appears in the "master out" string. Nevermind. Apparently there's an inconsistency in the MySQL cluster and some servers have it truncated. Sorry for the noise. Andre
Re: [Dovecot] User authentication fields size limit
On Wed, 2010-10-06 at 19:59 +0100, Timo Sirainen wrote:
> There's no 128 byte limit in Dovecot as far as I know. You sure it's not
> truncated to 128 bytes already on the SQL database? :) If that's not the
> case, set auth_debug=yes and show the logs.
Is the log from "doveadm user" enough? It's like below.
Oct 6 16:11:03 box5 dovecot: auth: Debug: master in:
USER#0111#011wellington.epaminondas=sbu.org.br#011service=doveadm
Oct 6 16:11:03 box5 dovecot: auth: Debug:
prefetch(wellington.epaminondas=sbu.org.br): passdb didn't return userdb
entries, trying the next userdb
Oct 6 16:11:03 box5 dovecot: auth: Debug:
sql(wellington.epaminondas=sbu.org.br): SELECT pass AS password,
mail_location AS mail, concat('*:bytes=', quota, 'M') AS quota_rule FROM
mail_users WHERE user = 'wellington.epaminondas=sbu.org.br' AND
server_id = 1
Oct 6 16:11:03 box5 dovecot: auth: Debug: master out:
USER#0111#011wellington.epaminondas=sbu.org.br#011password=$1$gT9MJF7o
$mactR0nGXtu8X6G.3mJuF0#011mail=mdbox:/var/spool/imap/partition7/b7/wellington.epaminondas=sbu.org.br:INDEX=/var/lib/imap/user/b7/wellington.epaminondas=sbu.org#011quota_rule=*:bytes=1024M
Truncation appears in the "master out" string.
Thanks,
Andre
[Dovecot] User authentication fields size limit
Hello I'm using authentication via SQL, and I'm storing the full mail_location in the database. This is a long string because we're using multiple partitions and hashing to create this path. It looks like this: # doveadm user andre=sneakymustard.com userdb: andre=sneakymustard.com password : $1$ixW87UeD$fLhORaJS0c5wNFmWqfJdI1 mail : mdbox:/var/spool/imap/partition7/11/andre=sneakymustard.com:INDEX=/var/lib/imap/user/11/andre=sneakymustard.com quota_rule: *:bytes=1024M Depending on the user's email, this string can be even longer. What I've noticed is that it is truncated when it reaches 128 bytes. Could this size limit be increased? Thanks, Andre
Re: [Dovecot] Backup recovery with mdbox
On Wed, 2010-10-06 at 15:26 +0100, Timo Sirainen wrote: > dsync -o > mail_location=mdbox:mdbox:/var/spool/imap/partition7/11/andre=sneakymustard.com:INDEX=/var/lib/imap/user/11/andre=sneakymustard.com:MAILBOXDIR=mailboxes/backup-1234 > -u ..etc.. > > So basically override the default mail_location but only change the > mailboxes/ directory to mailboxes/backup-1234. I think that should work. > Take a backup first. :) That worked, as long as I run a doveadm force-resync -u andre=sneakymustard.com INBOX after dsync, which is totally fine :) Thanks! Andre
Re: [Dovecot] Backup recovery with mdbox
On Tue, 2010-10-05 at 14:13 +0100, Timo Sirainen wrote:
> dsync could work, although with some small kludging:
>
> 1. Put the old backup to e.g. ~/backup
>
> 2. Rename the mailbox to whatever you want in destination, e.g. mv
> ~/backup/mailboxes/foo ~/backup/mailboxes/foo-$date
>
> 3. Copy the mailbox: dsync -m foo-$date mirror mdbox:~/backup
Hello
I'm doing the following:
* Backups are extracted in /home/andre/backup/{spool,lib}
* The user mail_location is
mdbox:/var/spool/imap/partition7/11/andre=sneakymustard.com:INDEX=/var/lib/imap/user/11/andre=sneakymustard.com
* I tried the following commands:
# dsync -R -u andre=sneakymustard.com -m backup_2010-10-01-23 backup
"mdbox:/home/andre/backup/spool:INDEX=/home/andre/backup/lib"
# dsync -u andre=sneakymustard.com -m backup_2010-10-01-23 -o
mail_location="mdbox:/home/andre/backup/spool:INDEX=/home/andre/backup/lib"
backup
"mdbox:/var/spool/imap/partition7/11/andre=sneakymustard.com:INDEX=/var/lib/imap/user/11/andre=sneakymustard.com
The intent is that a folder called "backup_2010-10-01-21" is created in
that mailbox, with the contents from /home/andre/backup stored in it,
but nothing is created. I believe this is because there's no
"backup_2010-10-01-21" folder in the recovered backup. Is there a way
around this?
Thanks,
Andre
[Dovecot] Backup recovery with mdbox
Hello I'm using mdbox for mail storage. I'd like to give users an option to recover backups into a new folder, say, Backup-$date. Is there a simple way to do this? I was looking at tools like dsync and doveadm-altmove but apparently they're not intended for this kind of stuff. Thanks in advance, Andre
