date:20130201

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

2013-02-01 Thread Daniel Parthey

Hi Frank,

Frank Lienhard wrote:
> Reindl Harald wrote:
> >Am 01.02.2013 17:22, schrieb Frank Lienhard:
> >>I'm about to replace my old server. Both servers are on my internal network.
> >>The old one ist running debian lenny(32bit) with dovecot 1.0.15 and the new 
> >>one debian Wheezy with dovecot 2.1.7.
> >
> > imapsync is version agnostic and if you have a database
> > with the user-logins it should be easy to generate a shellscript
> >
> sadly, this is not availble from the debian repos.

imapsync is written in perl and available on github:
https://github.com/imapsync/imapsync

This is IMHO the best tool where dsync cannot be applied (as in your case).
The only drawback of imapsync is that it cannot preserve UIDs since it uploads
the messages via IMAP and the server will assign new message UIDs.

Regards
Daniel
-- 
https://plus.google.com/103021802792276734820

Re: [Dovecot] lmtp-proxying in 2.1 slower than in 2.0.14 ?

2013-02-01 Thread Timo Sirainen

On 1.2.2013, at 19.00, Jan-Frode Myklebust  wrote:

> We upgraded our two dovecot directors from v2.0.14 to dovecot-ee
> 2.1.10.3 this week, and after that mail seems to be flowing a lot
> slower than before. The backend mailstores are untouched, on v2.0.14
> still. After the upgrade we've been hitting process_limit for lmtp a
> lot, and we're struggeling with large queues in the incoming
> mailservers that are using LMTP virtual transport towards our two
> directors.
> 
> I seem to remember 2.1 should have a new lmtp-proxying code. Is there
> anything in this that maybe needs to be tuned that's different from
> v2.0 ? I'm a bit scheptical to just increasing the process_limit for
> LMTP proxying, as  I doubt running many hundreds of simultaneous
> deliveries should work that much better against the backend storage..

Hmm. The main difference is that v2.1 writes temporary files to mail_temp_dir. 
If that's in tmpfs (and probably even if it isn't), it should still be pretty 
fast..

Have you checked if there's an increase in disk I/O usage, or system cpu usage?

Or actually .. It could simply be that in v2.0.15 service lmtp { client_limit } 
default was changed to 1 (from default_client_limit=1000). This is important 
with the backend, because writing to message store can be slow, but proxying 
should be able to handle more than 1 client per process, even with the new 
temporary file writing. So you could see if it helps to set lmtp { client_limit 
= 100 } or something.

Re: [Dovecot] dsync replication errors

2013-02-01 Thread Michael Grimm

[Sorry Oli for my previous mail to your address, only. Resent here]

Oli Schacher  wrote:

> There still seems to be a problem when changes to both mailboxes at
> the same time are involved

I can confirm your observation, although triggered by a different test
scenario, similar to the one I did use with 2.1 replicator before
(http://www.dovecot.org/list/dovecot/2012-March/064354.html).

This is v2.2.beta1 (78bdcb6642c7) with freshly created mailboxes "test"
at both servers "mx1" and "mx2", and replicator uses ssh for remote
access. Both servers run a recent postfix, use lmtp for local delivery,
and "test" is a virtual user.

Test script to produce local testmails of equal size at mx1:
| #!/bin/csh
| set INDEX= 101
| set endINDEX = 200
| while ( $INDEX <= $endINDEX )
|echo $INDEX
|echo "test" | mail -s $INDEX test@mx1
|if ( $INDEX % 1000 == 0 ) then
|   sleep 1
|endif
|@ INDEX = $INDEX + 1
|end
|exit 0

Test script to produce testmails of equal size at mx2:
| #!/bin/csh
| set INDEX= 1101
| set endINDEX = 1200
| while ( $INDEX <= $endINDEX )
|echo $INDEX
|echo "test" | mail -s $INDEX test@mx2
|if ( $INDEX % 1000 == 0 ) then
|   sleep 1
|endif
|@ INDEX = $INDEX + 1
|end
|exit 0

All tests are run with vanilla mailboxes, after restarting dovecot, and
without imap connections by MUA:

1) Simultaneous mailbomb approach: run both scripts simultaneously, and
  you'll end up with numerous duplicates in mailboxes "test". Very often
  you'll find multiples.

2) Mailbomb approach: run one script at one server only, and all mails
  will become perfectly well synchronised.

3) Mofify both scripts to "( $INDEX % 1 == 0 )" to add a second waiting
  between every mail injection, and run them simultaneously at both
  servers, and you'll end up with significantly less duplicates and no
  more multiples.

> Feb  1 07:12:52 doco1 dovecot: dsync-local(user1): Error: Mailbox INBOX: 
> Remote didn't send mail GUID=7a30ff22af5b0b510f0c960042f4 (UID=211)

> Feb  1 07:12:54 doco2 dovecot: dsync-local(user1): Error: Importing mailbox 
> INBOX failed

> Feb  1 07:13:24 doco2 dovecot: dsync-local(user1): Error: Remote command 
> process isn't dying, killing it

I do see those error messages as well, and in addition numerous of those:

| dovecot: dsync-local(test): Error: Mailbox INBOX: Unexpected GUID mismatch 
for UID=7153: 82c5df0a4ffa0b5141e36a0d5a02 != 
29cc9f284ffa0b5141c236abecbd

| doveadm: Error: dsync-remote(test): Error: Mailbox INBOX: Unexpected GUID 
mismatch for UID=7153: 82c5df0a4ffa0b5141e36a0d5a02 != 
29cc9f284ffa0b5141c236abecbd

| dovecot: lmtp(49752, test): Error: Corrupted index cache file 
/.../test/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: File too small

| Feb  1 18:35:16   mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: Error: 
Mailbox INBOX: Corrupted index, uidvalidity=0
| Feb  1 18:35:16  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:16   mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: Error: 
Mailbox INBOX: Corrupted index, uidvalidity=0
| Feb  1 18:35:16   mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: Error: 
mdbox /.../test/mailboxes/INBOX/dbox-Mails: Storage keeps breaking
| Feb  1 18:35:16   mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: Error: 
Mailbox INBOX: Corrupted index, uidvalidity=0
| Feb  1 18:35:16  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:16  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: mdbox /.../test/storage: rebuilding indexes
| Feb  1 18:35:17  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:17  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:17  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: mdbox /.../test/storage: rebuilding indexes
| Feb  1 18:35:17  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:17  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:17  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: mdbox /.../test/storage: rebuilding indexes
| Feb  1 18:35:18  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:18  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:18  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: mdbox /.../test/storage: rebuilding indexes
| Feb  1 18:35:18  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:27  mx1 dovecot: imap(test) BXeiKq3UBgBd3DLy: 
Warning: fscking index file /.../test/storage/dovecot.map.index
| Feb  1 18:35:

[Dovecot] cmsg cancel

2013-02-01 Thread Michael Grimm

ignore
Article cancelled by slrn 1.0.1

Re: [Dovecot] Userdb passwd and 'nologin' users

2013-02-01 Thread Ben Morrow

At  4AM +0100 on  1/02/13 you (Daniel Parthey) wrote:
> Hi Ben,
> 
> Ben Morrow wrote:
> > +if (set->check_nologin) {
> > +/* skip entries that don't have a valid shell.
> > +   they're again probably not real users. */
> > +if (strcmp(pw->pw_shell, "/bin/false") == 0 ||
> > +strcmp(pw->pw_shell, "/sbin/nologin") == 0 ||
> > +strcmp(pw->pw_shell, "/usr/sbin/nologin") == 0)
> > +return FALSE;
> > +}
> 
> Valid shells are defined in /etc/shells and "locked" users, I would
> strongly discourage from hardcoding a list of no-login shells here.

That list isn't mine, my patch just moves that code from one part of the
file to another and makes it conditional. Personally I don't think
checking the shell is sensible at all, which is why I'm trying to make
it optional.

> Users locked with "passwd -l" can also be detected by a ! at
> the beginning of the password hash.

That is system-specific, and in any case you have to be root (and on
non-BSD systems you have to make a shadow password call) to see the
password field. The userdb shouldn't be doing that: locked users will
already be prevented from logging in with a password because the
password won't match. (Of course, this doesn't cover e.g. Kerberos
logins, though I would usually lock a Kerberos user by telling the KDC
not to issue tickets rather than by locking the passwd account.)

Ben

Re: [Dovecot] Userdb passwd and 'nologin' users

2013-02-01 Thread Ben Morrow

At  1AM +0200 on  1/02/13 you (Timo Sirainen) wrote:
> On 1.2.2013, at 0.35, Ben Morrow  wrote:
> 
> > I am running Dovecot with system users (userdb passwd), but some of
> > those users don't have shell accounts on the IMAP server so their shell
> > on that machine is set to /usr/sbin/nologin. Currently I am using
> > maildirs and this is not a problem, but I am in the process of switching
> > to dbox which means I will need a cronjob running 'doveadm purge -A'.
> > 
> > During testing I found that those users with a 'nologin' shell are not
> > included in the list returned by the userdb iterator, and that the
> > iterator doesn't honour the first/last_valid_uid settings. This
> > inconsistency seems undesirable, so the attached patch
> > 
> >- makes lookup perform the same checks as iteration,
> 
> Hmmh. You could also just have them aliased to other users, so this
> wouldn't be necessary..

I don't understand what you mean. Alias them where?

> >- makes the 'nologin' check configurable,
> >- adds a new optional check that the user owns their home directory.
> 
> These settings are passwd-specific, so they would have to something like:
> 
> userdb {
>   driver = passwd
>   args = check-nologin=n check-home=y
> }

OK. New patch attached.

> > The last check was the one performed by qmail, and seems to me to be a
> > more reliable 'is this a real user' check than a nologin shell.
> 
> It also performs disk I/O, slowing down the lookup.

Hmm. OK, I've left that part out: my real users are segregated by UID
anyway, so all I really care about is getting rid of the nologin check.
(I would be perfectly happy if the check were just removed altogether.)

> > If this patch is applied, the release notes for the next release should
> > probably mention that system users with a 'nologin' shell will no longer
> > be allowed to log in to IMAP until the 'auth_check_nologin' setting is
> > changed from true to false.
> 
> The default will in any case be the same as it is now.

Well, yes; but authentication will now check for a nologin shell by
default, which it didn't before, so the visible behaviour will have
changed.

> > Also, there seem to be two first/last_valid_uid settings:
> > first_valid_uid itself, which is honoured by the storage subsystem, and
> > auth_first_valid_uid, which is honoured by the 'passwd' userdb. Is this
> > intentional?
> 
> Nope, that's a bug. Fixed that in v2.2:
> http://hg.dovecot.org/dovecot-2.2/rev/18661d1d6ed0

Cool. Will that be backported to 2.1 at some point?

Ben

diff -r 8f7dc71ad982 src/auth/userdb-passwd.c
--- a/src/auth/userdb-passwd.c	Fri Feb 01 17:46:08 2013 +
+++ b/src/auth/userdb-passwd.c	Fri Feb 01 18:13:31 2013 +
@@ -20,6 +20,8 @@
 	struct userdb_module module;
 	struct userdb_template *tmpl;
 
+unsigned int nologin:1;
+
 	unsigned int fast_count, slow_count;
 	unsigned int slow_warned:1;
 };
@@ -27,6 +29,8 @@
 struct passwd_userdb_iterate_context {
 	struct userdb_iterate_context ctx;
 	struct passwd_userdb_iterate_context *next_waiting;
+
+unsigned int nologin:1;
 };
 
 static struct passwd_userdb_iterate_context *cur_userdb_iter = NULL;
@@ -76,6 +80,29 @@
 	}
 }
 
+static bool
+passwd_want_pw(struct passwd *pw, const struct auth_settings *set,
+unsigned int nologin)
+{
+	/* skip entries not in valid UID range.
+	   they're users for daemons and such. */
+	if (pw->pw_uid < (uid_t)set->first_valid_uid)
+		return FALSE;
+	if (pw->pw_uid > (uid_t)set->last_valid_uid && set->last_valid_uid != 0)
+		return FALSE;
+
+if (nologin) {
+/* skip entries that don't have a valid shell.
+   they're again probably not real users. */
+if (strcmp(pw->pw_shell, "/bin/false") == 0 ||
+strcmp(pw->pw_shell, "/sbin/nologin") == 0 ||
+strcmp(pw->pw_shell, "/usr/sbin/nologin") == 0)
+return FALSE;
+}
+
+	return TRUE;
+}
+
 static void passwd_lookup(struct auth_request *auth_request,
 			  userdb_callback_t *callback)
 {
@@ -106,6 +133,13 @@
 		return;
 	}
 
+if (!passwd_want_pw(&pw, auth_request->set, module->nologin)) {
+auth_request_log_info(auth_request, "passwd",
+  "user has bad uid or shell");
+callback(USERDB_RESULT_USER_UNKNOWN, auth_request);
+return;
+}
+
 	auth_request_set_field(auth_request, "user", pw.pw_name, NULL);
 
 	auth_request_init_userdb_reply(auth_request);
@@ -125,11 +159,15 @@
 		userdb_iter_callback_t *callback, void *context)
 {
 	struct passwd_userdb_iterate_context *ctx;
+	struct userdb_module *_module = auth_request->userdb->userdb;
+	struct passwd_userdb_module *module =
+		(struct passwd_userdb_module *)_module;
 
 	ctx = i_new(struct passwd_userdb_iterate_context, 1);
 	ctx->ctx.auth_request = auth_request;
 	ctx->ctx.callback = callback;
 	ctx->ctx.context = context;
+ctx->nologin = module->nologin;

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

2013-02-01 Thread Reindl Harald



Am 01.02.2013 18:19, schrieb Frank Lienhard:
> Reindl Harald wrote:
>> Am 01.02.2013 17:22, schrieb Frank Lienhard:
>>  
>>> I'm about to replace my old server. Both servers are on my internal network.
>>> The old one ist running debian lenny(32bit) with dovecot 1.0.15 and the new 
>>> one debian Wheezy with dovecot 2.1.7.
>>> I set up the users on both systems identically (same gid uid).
>>> My first attempt was tho simply rsync the Maildirs along with the homes. 
>>> Both deovecot versions are set up with the
>>> maildir format, but this results in ~30% doublicated mails on the new 
>>> server.
>>> I then learned about doveadm/dsync, only to figure out that this isn't 
>>> available in v 1.0.15 and therefor won't
>>> work.
>>> So which could be the most painless way to solve that?
>>> 
>>
>> imapsync is version agnostic and if you have a database
>> with the user-logins it should be easy to geenrate a shellscript
>>
>> i did migration of around 20 messages and some hundret users
>> from Apple OSX EIMS to dbmail/dovecot in 2009 this way
>>
>>   
> sadly, this is not availble from the debian repos.
> I searched a bit around:
> syncmaildir, imapsync: only available in in Lenny OR Wheezy, but needed on 
> both servers

needed on both servers is simply not true
i have even a template in my admin-backend to generate imapsync scripts out of 
a db

imapsync --host1 [source] --user1 [user1] --password1 [pwd1] --authmech1 LOGIN 
--host2 [target] --user2 [user2]
--password2 [pwd2] --authmech2 LOGIN --skipsize



signature.asc
Description: OpenPGP digital signature

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

2013-02-01 Thread Frank Lienhard


Reindl Harald wrote:

Am 01.02.2013 17:22, schrieb Frank Lienhard:
  

I'm about to replace my old server. Both servers are on my internal network.
The old one ist running debian lenny(32bit) with dovecot 1.0.15 and the new one 
debian Wheezy with dovecot 2.1.7.
I set up the users on both systems identically (same gid uid).
My first attempt was tho simply rsync the Maildirs along with the homes. Both 
deovecot versions are set up with the
maildir format, but this results in ~30% doublicated mails on the new server.
I then learned about doveadm/dsync, only to figure out that this isn't 
available in v 1.0.15 and therefor won't work.
So which could be the most painless way to solve that?



imapsync is version agnostic and if you have a database
with the user-logins it should be easy to geenrate a shellscript

i did migration of around 20 messages and some hundret users
from Apple OSX EIMS to dbmail/dovecot in 2009 this way

  

sadly, this is not availble from the debian repos.
I searched a bit around:
syncmaildir, imapsync: only available in in Lenny OR Wheezy, but needed 
on both servers
isync, mailsync: mentioned to hold two maildirs in sync with a more or 
less difficult to configure setup

which  leaves me one option (from programms I found);
offlineimap: mentioned to backup mails from a remote/ISP mailserver in a 
local directory of a user.

Wonder if that works , when the local dir is one of an imap server...

[Dovecot] lmtp-proxying in 2.1 slower than in 2.0.14 ?

2013-02-01 Thread Jan-Frode Myklebust

We upgraded our two dovecot directors from v2.0.14 to dovecot-ee
2.1.10.3 this week, and after that mail seems to be flowing a lot
slower than before. The backend mailstores are untouched, on v2.0.14
still. After the upgrade we've been hitting process_limit for lmtp a
lot, and we're struggeling with large queues in the incoming
mailservers that are using LMTP virtual transport towards our two
directors.

I seem to remember 2.1 should have a new lmtp-proxying code. Is there
anything in this that maybe needs to be tuned that's different from
v2.0 ? I'm a bit scheptical to just increasing the process_limit for
LMTP proxying, as  I doubt running many hundreds of simultaneous
deliveries should work that much better against the backend storage..


## doveconf -n ##
# 2.1.10.3: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-194.32.1.el5 x86_64 Red Hat Enterprise Linux Server
release 5.5 (Tikanga)
default_client_limit = 4000
director_mail_servers = 192.168.42.7 192.168.42.8 192.168.42.9
192.168.42.10 192.168.42.28 192.168.42.29
director_servers = 192.168.42.15 192.168.42.17
disable_plaintext_auth = no
listen = *
lmtp_proxy = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
passdb {
  args = proxy=y nopassword=y
  driver = static
}
protocols = imap pop3 lmtp sieve
service anvil {
  client_limit = 6247
}
service auth {
  client_limit = 8292
  unix_listener auth-userdb {
user = dovecot
  }
}
service director {
  fifo_listener login/proxy-notify {
mode = 0666
  }
  inet_listener {
port = 5515
  }
  unix_listener director-userdb {
mode = 0600
  }
  unix_listener login/director {
mode = 0666
  }
}
service imap-login {
  executable = imap-login director
  process_limit = 4096
  process_min_avail = 4
  service_count = 0
  vsz_limit = 256 M
}
service lmtp {
  inet_listener lmtp {
address = *
port = 24
  }
  process_limit = 100
}
service managesieve-login {
  executable = managesieve-login director
  inet_listener sieve {
address = *
port = 4190
  }
  process_limit = 50
}
service pop3-login {
  executable = pop3-login director
  process_limit = 2048
  process_min_avail = 4
  service_count = 0
  vsz_limit = 256 M
}
ssl_cert =

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

2013-02-01 Thread Reindl Harald



Am 01.02.2013 17:22, schrieb Frank Lienhard:
> I'm about to replace my old server. Both servers are on my internal network.
> The old one ist running debian lenny(32bit) with dovecot 1.0.15 and the new 
> one debian Wheezy with dovecot 2.1.7.
> I set up the users on both systems identically (same gid uid).
> My first attempt was tho simply rsync the Maildirs along with the homes. Both 
> deovecot versions are set up with the
> maildir format, but this results in ~30% doublicated mails on the new server.
> I then learned about doveadm/dsync, only to figure out that this isn't 
> available in v 1.0.15 and therefor won't work.
> So which could be the most painless way to solve that?

imapsync is version agnostic and if you have a database
with the user-logins it should be easy to geenrate a shellscript

i did migration of around 20 messages and some hundret users
from Apple OSX EIMS to dbmail/dovecot in 2009 this way



signature.asc
Description: OpenPGP digital signature

[Dovecot] move mail from server with v1.0 to server w. v2.1?

2013-02-01 Thread Frank Lienhard


I'm about to replace my old server. Both servers are on my internal network.
The old one ist running debian lenny(32bit) with dovecot 1.0.15 and the 
new one debian Wheezy with dovecot 2.1.7.

I set up the users on both systems identically (same gid uid).
My first attempt was tho simply rsync the Maildirs along with the homes. 
Both deovecot versions are set up with the maildir format, but this 
results in ~30% doublicated mails on the new server.
I then learned about doveadm/dsync, only to figure out that this isn't 
available in v 1.0.15 and therefor won't work.

So which could be the most painless way to solve that?

Re: [Dovecot] Reviewing end-user ham/spam submissions before feeding them to sa-learn via Dovecot Antispam plug-in

2013-02-01 Thread Ben Johnson



On 2/1/2013 8:45 AM, Steffen Kaiser wrote:
> On Thu, 31 Jan 2013, Ben Johnson wrote:
> 
>> On 1/17/2013 4:31 AM, Steffen Kaiser wrote:
>>> On Wed, 16 Jan 2013, Ben Johnson wrote:
>>> 
>> I am using the Maildir format indeed.
> 
 any of your mail users need write permission those
 directories, the admin needs read permissions for the spooled
 files,
> 
>> By "mail users", do you mean, e.g., the "vmail" user account (I'm
>> on Debian/Ubuntu)? My understanding is that the "vmail" user
>> account
> 
> Yes, I mean that Unix account, Dovecot accesses the fils with. In
> a setup with virtual users "vmail" makes sense.
> 
>> handles all IMAP transactions; if this is true, then are you
>> saying that the only requisite to your suggestions is that the
>> "vmail" user has read/write access to the following two
>> directories?
> 
>> /path/to/admin/Maildir/.TrainingReview.spam/new/
> 
>> and
> 
>> /path/to/admin/Maildir/.TrainingReview.not_spam/new/
> 
> Yes.
> 
 you need some method to pass the reviewed messages to
 sa-learn.
> 
>> In the past, I have simply sorted the messages into "Ham" and
>> "Spam" sub-folders of the admin's training Inbox, and called
>> sa-learn, with the appropriate --ham/--spam switch on each, using
>> a cron job. It sounds as though this is what you are suggesting,
>> and I can continue
> 
> That's what I mean.
> 
>> with this approach.
> 
>> I went ahead and tried to reconfigure Dovecot's Antispam plug-in
>> to use the spool2dir backend, but I'm receiving a
>> less-than-helpful message from the plug-in when I try to move a
>> message from Inbox to Junk or vice versa: "CANNOT: antispam
>> plugin not configured".
> 
>> Please note that I am using Dovecot 1.2.9 in Ubuntu 10.04 LTS.
>> By
> 
> Oh, I have no experience with Dovecot v1.2; in v1.0 you have to
> compile one particular backend into antispam-plugin. Maybe,
> distributors have another, non-Dovecot way to select between the
> backends.
> 
>> extension, I am using the Antispam plug-in for Dovecot 1 (not 2),
>> the manpage for which is at 
>> http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html
>>
>> 
. So, the configuration option names and expected values differ
>> slightly from those in your example.
> 
> The man-page contains:
> 
> "INSTALLATION
> 
> First copy the ‘defconfig’ file to ‘.config’ and edit it as 
> necessary. You  need  to  have  the  dovecot  headers installed and
> possibly other things depending on the backend you choose.  Then,
> assuming  you have configured the INSTALLDIR correctly, simply run
> ‘make install’. "
> 
> -and-
> 
> "BACKENDS
> 
> The  plugin supports multiple backends, there are currently two 
> working backends included in the distribution: "
> 
> This suggests that my guess is true - although I don't understand
> the "there are currently two backends included" part.
> 
> ==
> 
> So you could re-compile another antispam-plugin with spool2dir
> backend, or - - you are using these settings, right?
> 
> #= # mail sending plugin # # Because of the way
> this plugin works, you can also use it # to train via an arbitrary
> program that receives the message # on standard input, in that case
> you can use the config # options antispam_mail_spam and
> antispam_mail_notspam for # the argument that distinguishes between
> ham and spam. # For example: #   antispam_mail_sendmail =
> /path/to/mailtrain #   antispam_mail_sendmail_args = --for;%u #
> antispam_mail_spam = --spam #   antispam_mail_notspam = --ham
> 
> change antispam_mail_sendmail into a script, that drops the
> message into the correct mail folder, e.g.:
> 
> #!/bin/bash
> 
> mode= for opt; do if test "x$opt" == x--ham; then mode=HAM break 
> elif test "x$opt" == x--ham; then mode=SPAM break fi done
> 
> if test -n "$mode"; then # options from
> http://wiki1.dovecot.org/LDA /path/to/dovecot-deliver -d spamadmin
> -m Training.$mode fi
> 
> This sends the message bypassing a MTA to the spamadmin user.
> 
> Kind regards,
> 
> -- Steffen Kaiser

Steffen,

It seems you're correct in that the spool2dir back-end isn't included
with version 1 of the plug-in. I like the idea of using a pipe script
better than changing the back-end.

Thank you for providing a solid example; it works beautifully with a
couple of small changes. Here's the final script:

---
#!/bin/bash

mode=
for opt; do
if test "x$*" == "x--ham"; then
mode=HAM
break
elif test "x$*" == "x--spam"; then
mode=SPAM
break
fi
done

if test -n "$mode"; then
# options from http://wiki1.dovecot.org/LDA
/usr/lib/dovecot/deliver -d u...@example.com -m Training.$mode
fi

exit 0
---

For anyone who is curious, here are the Antispam plug-in options:

---
  # For Dovecot < 2.0.
  antispam_spam_pattern_ignorecase = SPAM;JUNK
  antispam_mail_

Re: [Dovecot] dsync timeout?

2013-02-01 Thread micah anderson

Sean Kamath  writes:

> On Jan 30, 2013, at 3:46 PM, micah anderson  wrote:
>> Seems that only the above process was still around and no other dsync
>> processes. I have three machines that all have this happening it seems.
>> 
>> I wonder if there is a ssh configuration option I could set to make
>> these die off.
>
> If the ssh process isn't sending anything, and just waiting for read()s, and 
> keepalives are turned off, the SSH session might never know the remote side 
> is long gone. . .
>
> If any data were transmitted, it would discover the remote side is turned off.
>
> See man ssh_config and the option TCPKeepAlive.
>
> BTW: Since it's not on the command line, it's likely in /etc/ssh_config or 
> /etc/ssh/ssh_config.  Or ~/.ssh/config.

In /etc/ssh/sshd_config on the server I'm sending to, TCPKeepAlive yes
is set.

The default on this system, according to the man page, seems to be to
have TCPKeepAlive set. 

Perhaps I should set ServerAliveInterval?

micah

Re: [Dovecot] Reviewing end-user ham/spam submissions before feeding them to sa-learn via Dovecot Antispam plug-in

2013-02-01 Thread Steffen Kaiser


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, 31 Jan 2013, Ben Johnson wrote:


On 1/17/2013 4:31 AM, Steffen Kaiser wrote:

On Wed, 16 Jan 2013, Ben Johnson wrote:


I am using the Maildir format indeed.


any of your mail users need write permission those directories,
the admin needs read permissions for the spooled files,


By "mail users", do you mean, e.g., the "vmail" user account (I'm on
Debian/Ubuntu)? My understanding is that the "vmail" user account


Yes, I mean that Unix account, Dovecot accesses the fils with. In a setup 
with virtual users "vmail" makes sense.



handles all IMAP transactions; if this is true, then are you saying
that the only requisite to your suggestions is that the "vmail" user
has read/write access to the following two directories?

/path/to/admin/Maildir/.TrainingReview.spam/new/

and

/path/to/admin/Maildir/.TrainingReview.not_spam/new/


Yes.


you need some method to pass the reviewed messages to sa-learn.


In the past, I have simply sorted the messages into "Ham" and "Spam"
sub-folders of the admin's training Inbox, and called sa-learn, with
the appropriate --ham/--spam switch on each, using a cron job. It
sounds as though this is what you are suggesting, and I can continue


That's what I mean.


with this approach.

I went ahead and tried to reconfigure Dovecot's Antispam plug-in to
use the spool2dir backend, but I'm receiving a less-than-helpful
message from the plug-in when I try to move a message from Inbox to
Junk or vice versa: "CANNOT: antispam plugin not configured".

Please note that I am using Dovecot 1.2.9 in Ubuntu 10.04 LTS. By


Oh, I have no experience with Dovecot v1.2; in v1.0 you have to compile 
one particular backend into antispam-plugin. Maybe, distributors have 
another, non-Dovecot way to select between the backends.



extension, I am using the Antispam plug-in for Dovecot 1 (not 2), the
manpage for which is at
http://manpages.ubuntu.com/manpages/lucid/man7/dovecot-antispam.7.html
. So, the configuration option names and expected values differ
slightly from those in your example.


The man-page contains:

"INSTALLATION

   First copy the ‘defconfig’ file to ‘.config’ and edit it as 
necessary.
   You  need  to  have  the  dovecot  headers installed and possibly 
other
   things depending on the backend you choose.  Then,  assuming  you 
have

   configured the INSTALLDIR correctly, simply run ‘make install’.
"

- -and-

"BACKENDS

   The  plugin supports multiple backends, there are currently two 
working

   backends included in the distribution:
"

This suggests that my guess is true - although I don't understand the 
"there are currently two backends included" part.


==

So you could re-compile another antispam-plugin with spool2dir backend, or 
- - you are using these settings, right?


#=
   # mail sending plugin
   #
   # Because of the way this plugin works, you can also use it
   # to train via an arbitrary program that receives the message
   # on standard input, in that case you can use the config
   # options antispam_mail_spam and antispam_mail_notspam for
   # the argument that distinguishes between ham and spam.
   # For example:
   #   antispam_mail_sendmail = /path/to/mailtrain
   #   antispam_mail_sendmail_args = --for;%u
   #   antispam_mail_spam = --spam
   #   antispam_mail_notspam = --ham

change antispam_mail_sendmail into a script, that drops the message
into the correct mail folder, e.g.:

#!/bin/bash

mode=
for opt; do
if test "x$opt" == x--ham; then
mode=HAM
break
elif test "x$opt" == x--ham; then
mode=SPAM
break
fi
done

if test -n "$mode"; then
# options from http://wiki1.dovecot.org/LDA
/path/to/dovecot-deliver -d spamadmin -m Training.$mode
fi

This sends the message bypassing a MTA to the spamadmin user.

Kind regards,

- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUQvHDl3r2wJMiz2NAQJEeAf/XmxSzh+cPqviAax/ucThVaYfygrARz6G
qXRLbea/8fnhlRfO2seL75tElDmRsirVXGPu5awpf0WUEzFD96HWrmcrKMMRPfyE
uFylqzVB2dnmc+KOLolGb08hKRooMOaTPQt1Y9eVDGAplQM8PoNu+K3QE+rqkCbf
OJiL2pxJrEbiTOxzVhFSgUY/VdJVYLUBY4BpC5iZp7nNvXvub4scNlcd7OX0T1Kj
nlPnjpw2eNWX+UBCmjbfuVQKVFLBIFQFL9gnxZMphCMzjjYYgPaGHpSBlO00C+aM
ddiR46SrcjJIP4pXZsJyf5xw5aOCIUk2PXGr4aQFj409rcVJaK3CsQ==
=NgZB
-END PGP SIGNATURE-

Re: [Dovecot] dsync replication errors

2013-02-01 Thread Oli Schacher

On Thu, 31 Jan 2013 22:17:28 +0200
Timo Sirainen  wrote:

> On Thu, 2013-01-31 at 21:51 +0200, Timo Sirainen wrote:
> > On 31.1.2013, at 19.41, Oli Schacher  wrote:
> > 
> > >>> Jan 31 17:13:11 doco1 dovecot: dsync-local(user1): Error:
> > >>> Mailbox INBOX: Remote didn't send mail
> > >>> GUID=33dabe0f11980a51200c960042f4 (UID=104)
> > 
> > I guess there's some bug that causes this to happen in some
> > situations.. But the reason for mail duplication should be fixed
> > by: http://hg.dovecot.org/dovecot-2.2/rev/138f1c76c0ec
> > 
> > Except that shouldn't have been necessary. doveadm-server returns
> > success before it has finished running dsync. Not sure why, need to
> > debug it further.
> 
> Fixed with a bit of a kludge:
> http://hg.dovecot.org/dovecot-2.2/rev/e9e6a95cea21
> 
> 

I can confirm that it has become significantly harder to produce errors
with the latest patches. There still seems to be a problem when changes
to both mailboxes at the same time are involved, however, today I didn't
have time to test "scientifically", i just updated to latest hg and
clicked around, so this report probably won't be of much use to
you,sorry. I'll try to make reproducible tests again next week.

I'll post the errors from my clicking session anyway, maybe it helps you
figuring out what went wrong even without knowing how to reproduce. At
least the "Operation not permitted" error below when killing the
dsync process sounds unintended?


Logoutput is from changeset 78bdcb6642c7 running on both servers.


Server 1:
Feb  1 07:12:52 doco1 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Remote didn't send mail GUID=7a30ff22af5b0b510f0c960042f4
(UID=211)
Feb  1 07:12:52 doco1 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Remote didn't send mail GUID=7a30ff22af5b0b510f0c960042f4
(UID=205)
Feb  1 07:12:52 doco1 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Remote didn't send mail GUID=7a30ff22af5b0b510f0c960042f4
(UID=208)
Feb  1 07:12:54 doco1 dovecot: doveadm: Error: dsync-remote(user1):
Error: Mailbox INBOX: Unexpected GUID mismatch for UID=205:
7a30ff22af5b0b510f0c960042f4 != 8230ff22af5b0b510f0c960042f4
Feb  1 07:12:54 doco1 dovecot: doveadm: Error: dsync-remote(user1):
Error: Mailbox INBOX: Remote didn't send mail
GUID=7b30ff22af5b0b510f0c960042f4 (UID=228)
[...]
Feb  1 07:12:55 doco1 dovecot: doveadm: Error: dsync-remote(user1):
Error: Importing mailbox INBOX failed
Feb  1 07:12:56 doco1 dovecot: dsync-local(user1): Error:
read(vmail@192.168.23.62) failed: EOF
Feb  1 07:12:56 doco1 dovecot: dsync-local(user1): Error:
read(vmail@192.168.23.62) failed: Broken pipe
Feb  1 07:12:56 doco1 dovecot: dsync-local(user1): Error: Remote
command returned error 75
[...]
Feb  1 07:12:57 doco1 dovecot: doveadm: Error: dsync-remote(user1):
Error: Mailbox INBOX: Unexpected GUID mismatch for UID=291:
7b30ff22af5b0b510f0c960042f4 != 8d30ff22af5b0b510f0c960042f4
Feb  1 07:12:57 doco1 dovecot: doveadm: Error: dsync-remote(user1):
Panic: file dsync-mailbox-import.c: line 1112
(dsync_mailbox_import_change): assertion failed: (change->type == 
DSYNC_MAIL_CHANGE_TYPE_SAVE)
Feb  1 07:12:57 doco1 dovecot: doveadm: Error: dsync-remote(user1):
Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x5d4ea) 
[0x7f19cf5954ea]
-> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x32) 
[0x7f19cf5955d2] -> /usr/lib64/dovecot/libdovecot.so.0(+0x1f6ca) 
[0x7f19cf5576ca] -> /usr/bin/doveadm(dsync_mailbox_import_change+0x501) 
[0x42c881] -> /usr/bin/doveadm(dsync_brain_sync_mails+0x3a2) 
[0x4290c2] -> /usr/bin/doveadm(dsync_brain_run+0x169) 
[0x425e29] -> /usr/bin/doveadm() [0x426380] -> /usr/bin/doveadm() 
[0x434aa0] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f19cf5a4076]
-> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0xa7) 
[0x7f19cf5a5107]
-> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x28)
[0x7f19cf5a4018] -> /usr/bin/doveadm() 
[0x424134] -> /usr/bin/doveadm() 
[0x40fe4f] -> /usr/bin/doveadm() 
[0x41067d] -> /usr/bin/doveadm(doveadm_mail_try_run+0x141) 
[0x410ba1] -> /usr/bin/doveadm(main+0x3f1) 
[0x417bc1] -> /lib64/libc.so.6(__libc_start_main+0xfd) 
[0x7f19cf1c3cdd] -> /usr/bin/doveadm() [0x40f839]
Feb  1 07:12:57 doco1 dovecot: dsync-local(user1): Error:
read(vmail@192.168.23.62) failed: EOF


Server 2:
Feb  1 07:12:54 doco2 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Unexpected GUID mismatch for UID=205:
7a30ff22af5b0b510f0c960042f4 != 8230ff22af5b0b510f0c960042f4
Feb  1 07:12:54 doco2 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Remote didn't send mail GUID=7b30ff22af5b0b510f0c960042f4
(UID=228)
Feb  1 07:12:54 doco2 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Remote didn't send mail GUID=7b30ff22af5b0b510f0c960042f4
(UID=234)
Feb  1 07:12:54 doco2 dovecot: dsync-local(user1): Error: Mailbox
INBOX: Remote didn't send mail GUID=7b30ff22af5b0b510f0c960042f4
(UID=238)
Feb  1 07:12:54 doco2 dovecot: dsync-local

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

Re: [Dovecot] lmtp-proxying in 2.1 slower than in 2.0.14 ?

Re: [Dovecot] dsync replication errors

[Dovecot] cmsg cancel

Re: [Dovecot] Userdb passwd and 'nologin' users

Re: [Dovecot] Userdb passwd and 'nologin' users

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

[Dovecot] lmtp-proxying in 2.1 slower than in 2.0.14 ?

Re: [Dovecot] move mail from server with v1.0 to server w. v2.1?

[Dovecot] move mail from server with v1.0 to server w. v2.1?

Re: [Dovecot] Reviewing end-user ham/spam submissions before feeding them to sa-learn via Dovecot Antispam plug-in

Re: [Dovecot] dsync timeout?

Re: [Dovecot] Reviewing end-user ham/spam submissions before feeding them to sa-learn via Dovecot Antispam plug-in

Re: [Dovecot] dsync replication errors

15 matches

Site Navigation

Mail list logo

Footer information