Re: Please do not remove replication

2024-01-25 Thread Gerben Wierda
I have two postfix/(replicating)dovecot/rspamd/etc servers, 6-8 human users, on
a total of 15-20 devices (iOS and macOS)

Yours, 

Gerben Wierda (LinkedIn, Mastodon)
R&A_IT_Strategy (main site)
Book: Chess_and_the_Art_of_Enterprise Architecture
Book: Mastering_ArchiMate
YouTube_Channel

 On 25 Jan 2024, at 10:17, Marc  wrote:

 How many servers do you have? How many active clients do you have?


  Respectfully, I would like to ask: please do not remove
  replication, please
  rethink this.

  Currently, replication is my life saver. I run two postfix/
  dovecot combos
  (on different operating systems), with dovecot
  synchronising via
  replication. Both are behind a HAProxy running on the
  router (OPNsense), one
  as active, one as backup.

  If one of the two fails, the other takes over, and when it
  comes up again
  everything works fine and is up to date. I have had these
  kinds of system
  failures (very hard to find and turned to be hardware
  related) and it was
  the replication that made me survive the issues (even when
  I was far away
  from my systems). Mail for my small group of users (about
  8) never went
  down, no mail message was ever lost, no manual
  interventions to sync were
  ever needed.

  If I want to create the same level of availability without
  replication, I
  need those two dovecots to use shared (NFS cluster)
  storage. But then, I
  have another single point of failure (NFS storage) again.
  So, I need two
  separate NFS machines that synchronise, Apart from the
  nightmare of making
  NFS secure, it means that I need to double my hardware
  (from two systems to
  four) to be protected against hardware failure (which is my
  goal).

  The replication service is the perfect small scale
  solution. Together with
  HAProxy, it enables HA in the most simple and effective
  way. Going the 'NFS
  cluster' route is not feasible for me, so if replication is
  removed and I am
  forced to upgrade, I will lose HA.

  So please, take small scale users like me into account.

      Gerben Wierda (LinkedIn <https://www.linkedin.com/in/
  gerbenwierda>, Mastodon
  <https://newsie.social/@gctwnl>)
  R&A IT Strategy <https://ea.rna.nl/> (main site)
  Book: Chess and the Art of Enterprise Architecture 
  Book: Mastering ArchiMate <https://ea.rna.nl/the-book-
  edition-iii/>
  YouTube Channel <http://www.youtube.com/@GerbenWierda>

   On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot
   
  wrote:

   Hi!

   Yes, director and replicator are removed, and
   won't be available for pro
  users either.

   For NFS setups (or similar shared setups), we
   have documented a way to use
  Lua to run a director-like setup, see

   https://doc.dovecot.org/3.0/configuration_manual/
   howto/director_with_lua/

   Regards to replication, doveadm sync is not being
   removed. So you can
  still run doveadm sync on your system to have a primary /
  backup setup.

   Aki

On 16/07/2023 18:34 EEST William
Edwards via dovecot
   wrote:


Top posting because nothing specific to
reply to, sorry. Not exactly
  sure, but there’s another thread about the removal of
  Director in favour of
  Dovecot Pro on 3.x. Perhaps this change is related.

William Edwards

 Op 16 jul. 2023 om 16:33
 heeft Daniele  het
  volgende geschreven:

 Hello,

 Just like Vladimir, I'm a bit
 concerned about this change,
 and I'd
  really appreciate if someone could let us know if the
  replication feature
  (that works so well!) will be replaced or removed; and, in
  case of removal,
  what would be recommended replacement?
 Thanks in advance and best
 regards,
 Daniele

  On 09-Jul-23 9:36
  PM, Vladimir
  Mishonov via
  dovecot wrote:
  Hello everyone.

  Just saw this
  commit in the
 

Please do not remove replication

2024-01-24 Thread Gerben Wierda
Respectfully, I would like to ask: please do not remove replication, please
rethink this.

Currently, replication is my life saver. I run two postfix/dovecot combos (on
different operating systems), with dovecot synchronising via replication. Both
are behind a HAProxy running on the router (OPNsense), one as active, one as
backup.

If one of the two fails, the other takes over, and when it comes up again
everything works fine and is up to date. I have had these kinds of system
failures (very hard to find and turned to be hardware related) and it was the
replication that made me survive the issues (even when I was far away from my
systems). Mail for my small group of users (about 8) never went down, no mail
message was ever lost, no manual interventions to sync were ever needed.

If I want to create the same level of availability without replication, I need
those two dovecots to use shared (NFS cluster) storage. But then, I have
another single point of failure (NFS storage) again. So, I need two separate
NFS machines that synchronise, Apart from the nightmare of making NFS secure,
it means that I need to double my hardware (from two systems to four) to be
protected against hardware failure (which is my goal).

The replication service is the perfect small scale solution. Together with
HAProxy, it enables HA in the most simple and effective way. Going the 'NFS
cluster' route is not feasible for me, so if replication is removed and I am
forced to upgrade, I will lose HA.

So please, take small scale users like me into account.

Gerben Wierda (LinkedIn, Mastodon)
R&A_IT_Strategy (main site)
Book: Chess_and_the_Art_of_Enterprise Architecture
Book: Mastering_ArchiMate
YouTube_Channel

 On 16 Jul 2023, at 18:54, Aki Tuomi via dovecot 
 wrote:

 Hi!

 Yes, director and replicator are removed, and won't be available for
 pro users either.

 For NFS setups (or similar shared setups), we have documented a way
 to use Lua to run a director-like setup, see

 https://doc.dovecot.org/3.0/configuration_manual/howto/
 director_with_lua/

 Regards to replication, doveadm sync is not being removed. So you can
 still run doveadm sync on your system to have a primary / backup
 setup.

 Aki

  On 16/07/2023 18:34 EEST William Edwards via dovecot
   wrote:


  Top posting because nothing specific to reply to, sorry.
  Not exactly sure, but there’s another thread about the
  removal of Director in favour of Dovecot Pro on 3.x.
  Perhaps this change is related.

  William Edwards

   Op 16 jul. 2023 om 16:33 heeft Daniele
het volgende geschreven:

   Hello,

   Just like Vladimir, I'm a bit concerned about
   this change, and I'd really appreciate if someone
   could let us know if the replication feature
   (that works so well!) will be replaced or
   removed; and, in case of removal, what would be
   recommended replacement?
   Thanks in advance and best regards,
   Daniele

On 09-Jul-23 9:36 PM, Vladimir Mishonov
via dovecot wrote:
Hello everyone.

Just saw this commit in the official
Github repo:

https://github.com/dovecot/core/commit/
4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff-

5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8


Looking at the commit details, it
appears that it completely removes the
replication feature. I'm a bit
perplexed by this change and am not
sure what might be the justification
for it. Personally, I find replication
to be very useful, as it allows me to
maintain a synchronized mirror of all
of my mailboxes on my home server, for
use as backup in case the primary
server goes down for some reason.

Perhaps there's some sort of
replacement being planned for this
feature? Or maybe the relevant code is
simply going to be refactored to a
plugin or external program, and there's
nothing to worry about at all?

In any case, I'd greatly appreciate if
one of the developers could comment on
this change.


   ___
   dovecot mailing list -- dovecot@dovecot.org
   To unsubscribe send an email to dovecot-
   le...@dovecot.org


  _

Re: Replication going away?

2023-11-28 Thread gerben . wierda
>From what version on is replication gone? I am running 2.3.20 and it still 
>there.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


I've sent a question to dovecot@dovecot.org via mail. Why is it not ending up here?

2023-11-28 Thread gerben . wierda
This is what my postfix server says:

Nov 28 14:23:12 albus postfix/qmgr[654]: 823D71CDE2714: 
from=, size=9144, nrcpt=1 (que
ue active)
Nov 28 14:23:12 albus postfix/smtp[2294]: Untrusted TLS connection established 
to talvi.dovecot.org[94.237.1
05.223]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) 
key-exchange ECDHE (prime256v1) server
-signature RSA-PSS (2048 bits) server-digest SHA256
Nov 28 14:23:13 albus postfix/smtp[2294]: 823D71CDE2714: 
to=, relay=talvi.dovecot.org[9
4.237.105.223]:25, delay=2.1, delays=0.51/0.02/0.75/0.83, dsn=2.0.0, 
status=sent (250 2.0.0 Ok: queued as E5
C425DEEF)
Nov 28 14:23:13 albus postfix/qmgr[654]: 823D71CDE2714: removed

But I can't see it here nor do I get it sent back to me.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: dovecot replication - new and cur folders on mx1 and mx2

2023-01-17 Thread Gerben Wierda
It might have a noticeable effect on clients.

I encountered (probably triggered by this in some way?) that I was unable to 
het the 'read' bit set in macOS Mail.app. Maybe (as I am doing HA with round 
robin) the Mail.app client got to one dovecot repository on one tcp connection 
and then on the other.

Is there a reason why syncing tis move from new to cur is a bad idea?

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>

> On 17 Jan 2023, at 22:41, Gerben Wierda  wrote:
> 
> I can confirm this in a slightly different setting, but still using two-way 
> sync between two dovecots. On e is 2.3.19.1 running on macOS Monterey, the 
> other is 2.3.20 running in an alpine container on Ubuntu.
> 
> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
> R&A IT Strategy <https://ea.rna.nl/> (main site)
> Book: Chess and the Art of Enterprise Architecture 
> <https://ea.rna.nl/the-book/>
> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
> 
>> On 15 Jan 2023, at 23:12, Tomaz Kavcic > <mailto:tomaz.kav...@futurion.si>> wrote:
>> 
>> Hello,
>> 
>> I have a question in regards to specific dovecot replication behaviour and 
>> I'm just wondering if this is actually an expected/normal behaviour, or just 
>> a version issue.
>> 
>> I'm using dovecot 2.3.16 which is packed by default with latest Ubuntu 
>> 22.04.1 LTS server release. I setup dovecot replication pair (mx1 - mx2) 
>> which is working ok. MX1 has priority 10, MX2 has priority 20. I use maildir 
>> (postfix + dovecot lmtp).
>> 
>> The "strange" behaviour is this. When new mail arrives, it's by default 
>> delivered into "new" folder inside user directory. This email is then 
>> replicated to both servers (mx1 and mx2). When I login to mx1 via IMAP 
>> client (roundcube, outlook, etc.) that specific email is moved from "new" to 
>> "cur" folder on server mx1 and it's flagged also with "S", which probably 
>> means read flag. On server mx2, that email filename is also flagged with 
>> "S", but the email stays inside the "new" folder and it's not moved to 
>> "cur". If I want this email to be moved to "cur" on mx2 server, I have to 
>> login to that IMAP server as well, click on that email (which is already 
>> flagged as read), and after click, the email is also moved to "cur" on 
>> server mx2.
>> 
>> Simply said, all new mails on mx1 server are moved to "cur" when accessed, 
>> but the stay in "new" folder on server mx2 until they're physically accessed 
>> there as well. Is this normal behaviour?
>> 
>> I tried setup with TCP and SSH replication, and the situation is the same in 
>> all cases. Lastly, I tried TCPS (with SSL) as well, but that option has 
>> issues in 2.3.16, which is probably known already as I found multiple posts 
>> about these issues in this version.
>> 
>> Thank you in advance for your answer and kind regards, Tomaz Kavcic.
>> 
>> ---
>> 
>> As suggested by mr. Timo Sarainen, this should be synced, so I'm posting 
>> doveconf -n as attachment for both servers as well.
>> 
>> 
> 



Re: replicator: Panic: data stack: Out of memory when allocating 268435496 bytes

2023-01-17 Thread Gerben Wierda

> On 6 Jan 2023, at 08:53, Aki Tuomi  wrote:
> 
> 
> 
> On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda  
> wrote:
>> One step further in my quest to create a replacement mail server.
>> 
>> I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new 
>> (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it 
>> works, but, after a while I see:
>> 
>> Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 
>> 268435496 bytes
>> Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 
>> killed with signal 6 (core dumped)
>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync 
>> failure: 
>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): 
>> Remote sent invalid input: -
>> 
>> I've removed synchronous operation for now (found a message on the net 
>> suggesting that) but is this known and what does it mean?
>> 
>> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
>> R&A IT Strategy <https://ea.rna.nl/> (main site)
>> Book: Chess and the Art of Enterprise Architecture 
>> <https://ea.rna.nl/the-book/>
>> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
>> 
> 
> Dovecot default memory limit is 256M. You should probably set
> 
> service replicator {
>  vsz_limit = 2G
> }
> 
> because replicator might have to use more memory, especially for larger 
> indexes. 
> 
> Aki

Coming back to this one. doveadm config says everything is vsz_limit = 
18446744073709551615 B

Isn't that more than 2G already (nd certainly more than 256M?)?

I also tried putting

default_vsz_limit = 2G

in dovecot.conf but that doesn't change anything either.

What am I missing?

Re: dovecot replication - new and cur folders on mx1 and mx2

2023-01-17 Thread Gerben Wierda
I can confirm this in a slightly different setting, but still using two-way 
sync between two dovecots. On e is 2.3.19.1 running on macOS Monterey, the 
other is 2.3.20 running in an alpine container on Ubuntu.

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>

> On 15 Jan 2023, at 23:12, Tomaz Kavcic  wrote:
> 
> Hello,
> 
> I have a question in regards to specific dovecot replication behaviour and 
> I'm just wondering if this is actually an expected/normal behaviour, or just 
> a version issue.
> 
> I'm using dovecot 2.3.16 which is packed by default with latest Ubuntu 
> 22.04.1 LTS server release. I setup dovecot replication pair (mx1 - mx2) 
> which is working ok. MX1 has priority 10, MX2 has priority 20. I use maildir 
> (postfix + dovecot lmtp).
> 
> The "strange" behaviour is this. When new mail arrives, it's by default 
> delivered into "new" folder inside user directory. This email is then 
> replicated to both servers (mx1 and mx2). When I login to mx1 via IMAP client 
> (roundcube, outlook, etc.) that specific email is moved from "new" to "cur" 
> folder on server mx1 and it's flagged also with "S", which probably means 
> read flag. On server mx2, that email filename is also flagged with "S", but 
> the email stays inside the "new" folder and it's not moved to "cur". If I 
> want this email to be moved to "cur" on mx2 server, I have to login to that 
> IMAP server as well, click on that email (which is already flagged as read), 
> and after click, the email is also moved to "cur" on server mx2.
> 
> Simply said, all new mails on mx1 server are moved to "cur" when accessed, 
> but the stay in "new" folder on server mx2 until they're physically accessed 
> there as well. Is this normal behaviour?
> 
> I tried setup with TCP and SSH replication, and the situation is the same in 
> all cases. Lastly, I tried TCPS (with SSL) as well, but that option has 
> issues in 2.3.16, which is probably known already as I found multiple posts 
> about these issues in this version.
> 
> Thank you in advance for your answer and kind regards, Tomaz Kavcic.
> 
> ---
> 
> As suggested by mr. Timo Sarainen, this should be synced, so I'm posting 
> doveconf -n as attachment for both servers as well.
> 
> 



Re: replicator: Panic: data stack: Out of memory when allocating 268435496 bytes

2023-01-06 Thread Gerben Wierda
dovecot-pgsql.conf
>  driver = sql
> 
> }
> 
> protocol imap {
>  mail_max_userip_connections = 50
>  mail_plugins = $mail_plugins notify replication
> }
> 
> protocol pop3 {
>  mail_max_userip_connections = 50
>  mail_plugins = $mail_plugins notify replication
> }
> 
> protocol imaps {
>  mail_max_userip_connections = 25
>  mail_plugins = $mail_plugins notify replication
> }
> 
> protocol pop3s {
>  mail_max_userip_connections = 25
>  mail_plugins = $mail_plugins notify replication
> }
> 
> 
> service managesieve-login {
>  process_limit = 1000
>  vsz_limit = 1g
>  inet_listener sieve {
>port = 4190
>  }
> }
> 
> verbose_proctitle = yes
> 
> replication_max_conns = 100
> 
> replication_full_sync_interval = 1d
> 
> service replicator {
>  client_limit = 0
>  drop_priv_before_exec = no
>  idle_kill = 4294967295s
>  process_limit = 1
>  process_min_avail = 0
>  service_count = 0
>  vsz_limit = 8g
>unix_listener replicator-doveadm {
>mode = 0600
>user = vmail
>  }
>  vsz_limit = 8192M
> }
> 
> 
> service aggregator {
>  process_limit = 1000
>  #vsz_limit = 1g
>  fifo_listener replication-notify-fifo {
>user = vmail
>group = vmail
>mode = 0666
>  }
> 
> }
> 
> service pop3-login {
>  process_limit = 1000
>  client_limit = 100
>  vsz_limit = 512m
> }
> 
> 
> service imap-urlauth-login {
>  process_limit = 1000
>  client_limit = 1000
>  vsz_limit = 1g
> }
> 
> 
> service imap-login {
>  process_limit=1000
>  client_limit = 1000
>  vsz_limit = 1g
> }
> 
> 
> protocol sieve {
>  managesieve_implementation_string = Dovecot Pigeonhole
>  managesieve_max_line_length = 65536
> }
> 
> 
> 
> 
> #Addition ssl config
> !include sni.conf
> 
> with sni cert support (examples)
> 
> # cat sni.conf
> #sni.conf
> ssl = yes
> verbose_ssl = yes
> ssl_dh = ssl_prefer_server_ciphers = yes
> #ssl_min_protocol = TLSv1.2
> 
> #Default *.scom.ca
> ssl_key = ssl_cert = ssl_ca = 
> local_name .scom.ca {
> ssl_key = ssl_cert = ssl_ca = 
> }
> 
> local_name mail.clancyca.com {
>  ssl_key =  ssl_cert =  ssl_ca = }
> 
> local_name mail.paulkudla.net {
>  ssl_key =  ssl_cert =  ssl_ca = }
> 
> local_name mail.ekst.ca {
>  ssl_key =  ssl_cert =  ssl_ca = }
> 
> local_name mail.hamletdevelopments.ca {
>  ssl_key =  ssl_cert =  ssl_ca = }
> 
> pg sql support supporting replication
> 
> # cat dovecot-pgsql.conf
> driver = pgsql
> connect = host=localhost port=5433 dbname=scom_billing user=pgsql password=
> default_pass_scheme = PLAIN
> 
> password_query = SELECT username as user, password FROM email_users WHERE 
> username = '%u' and password <> 'alias' and status = True and destination = 
> '%u'
> 
> user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and 
> password <> 'alias' and status = True and destination = '%u'
> 
> #iterate_query = SELECT user, password FROM email_users WHERE username = '%u' 
> and password <> 'alias' and status = True and destination = '%u'
> 
> iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
> status = True and alias_flag = False
> 
> 
> 
> 
> 
> Happy Friday !!!
> Thanks - paul
> 
> Paul Kudla
> 
> 
> Scom.ca Internet Services <http://www.scom.ca>
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
> 
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> Email p...@scom.ca
> 
> On 1/6/2023 5:32 AM, Gerben Wierda wrote:
>>> On 6 Jan 2023, at 08:53, Aki Tuomi >> <mailto:aki.tu...@open-xchange.com>> wrote:
>>> 
>>> 
>>> 
>>> On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda 
>>> mailto:gerben.wie...@rna.nl>> wrote:
>>>> One step further in my quest to create a replacement mail server.
>>>> 
>>>> I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new 
>>>> (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it 
>>>> works, but, after a while I see:
>>>> 
>>>> Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when 
>>>> allocating 268435496 bytes
>>>> Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 
>>>> killed with signal 6 (core dumped)
>>>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): 
>>>> Sync failure:
>>>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): 
>>>> Remote sent invalid input: -
>>>> 
>>>> I've removed synchronous operation for now (found a message on the net 
>>>> suggesting that) but is this known and what does it mean?
>>>> 
>>>> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda 
>>>> <https://www.linkedin.com/in/gerbenwierda>>)
>>>> R&A IT Strategy <https://ea.rna.nl/ <https://ea.rna.nl/>> (main site)
>>>> Book: Chess and the Art of Enterprise Architecture 
>>>> <https://ea.rna.nl/the-book/ <https://ea.rna.nl/the-book/>>
>>>> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/ 
>>>> <https://ea.rna.nl/the-book-edition-iii/>>
>>>> 
>>> 
>>> Dovecot default memory limit is 256M. You should probably set
>>> 
>>> service replicator {
>>>  vsz_limit = 2G
>>> }
>>> 
>>> because replicator might have to use more memory, especially for larger 
>>> indexes.
>>> 
>>> Aki
>> That is a good tip as well.
>> I had followed this bit of experience from someone else: 
>> https://marc.info/?l=dovecot&m=164438199727640 
>> <https://marc.info/?l=dovecot&m=164438199727640>, haven't seen any err 
>> message since. But that might be because they are in sync now and both sides 
>> are aware. Can I trigger full replication again so I can test?
>> Gerben
>> -- 
>> This message has been scanned for viruses and
>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
>> believed to be clean.



Re: replicator: Panic: data stack: Out of memory when allocating 268435496 bytes

2023-01-06 Thread Gerben Wierda

> On 6 Jan 2023, at 08:53, Aki Tuomi  wrote:
> 
> 
> 
> On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda  
> wrote:
>> One step further in my quest to create a replacement mail server.
>> 
>> I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new 
>> (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it 
>> works, but, after a while I see:
>> 
>> Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 
>> 268435496 bytes
>> Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 
>> killed with signal 6 (core dumped)
>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync 
>> failure: 
>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): 
>> Remote sent invalid input: -
>> 
>> I've removed synchronous operation for now (found a message on the net 
>> suggesting that) but is this known and what does it mean?
>> 
>> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
>> R&A IT Strategy <https://ea.rna.nl/> (main site)
>> Book: Chess and the Art of Enterprise Architecture 
>> <https://ea.rna.nl/the-book/>
>> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
>> 
> 
> Dovecot default memory limit is 256M. You should probably set
> 
> service replicator {
>  vsz_limit = 2G
> }
> 
> because replicator might have to use more memory, especially for larger 
> indexes. 
> 
> Aki

That is a good tip as well. 

I had followed this bit of experience from someone else: 
https://marc.info/?l=dovecot&m=164438199727640 
<https://marc.info/?l=dovecot&m=164438199727640>, haven't seen any err message 
since. But that might be because they are in sync now and both sides are aware. 
Can I trigger full replication again so I can test?

Gerben

replicator: Panic: data stack: Out of memory when allocating 268435496 bytes

2023-01-05 Thread Gerben Wierda
One step further in my quest to create a replacement mail server.

I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, 
Alpine Linux, Docker, apk package). When I turn on replication it works, but, 
after a while I see:

Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 
268435496 bytes
Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 
killed with signal 6 (core dumped)
Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync 
failure: 
Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote 
sent invalid input: -

I've removed synchronous operation for now (found a message on the net 
suggesting that) but is this known and what does it mean?

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>



Re: postfix connects to dovecot lmtp socket, but nothing is delivered

2023-01-05 Thread Gerben Wierda
Hmm: according to the documentation, the default is "imap pop3 lmtp"

But my installation (2.3.20, Alpine apk) for some reason only ran "imap" by 
default

G

> On 5 Jan 2023, at 21:27, Gerben Wierda  wrote:
> 
> Boy do I feel stupid.
> 
> protocols was not set in the new version's dovecot.conf. And default it only 
> contains imap. My old server had an override in dovecot.conf and I copied 
> conf.d over but started with afresh dovecot.conf, so lmtp was missing.
> 
> G
> 
>> On 5 Jan 2023, at 18:39, Gerben Wierda > <mailto:gerben.wie...@rna.nl>> wrote:
>> 
>>> On 5 Jan 2023, at 18:17, dove...@ptld.com <mailto:dove...@ptld.com> wrote:
>>> 
>>>> service lmtp {
>>>> inet_listener lmtp {
>>>> address = 127.0.0.1
>>>> port = 24
>>>> }
>>> 
>>>> unix_listener /var/spool/postfix/private/lmtp {
>>>> mode = 0660
>>>> user = postfix
>>>> group = postfix
>>>> }
>>> 
>>> 
>>> Looks like you are activating both a socket and a port. Im not sure what 
>>> dovecot does, does it bind to both or does it pick one and ignore the 
>>> other. You only need to use one or the other.
>>> 
>>> And what does your postfix side look like? Where is postfix trying to 
>>> connect? To the socket or the port? For using a socket postfix would look 
>>> something like:
>>> 
>>>virtual_transport = lmtp:unix:private/lmtp
>>> 
>> 
>> I removed the tcp port for now.
>> 
>> / # doveconf service/lmtp
>> service lmtp {
>>   chroot = 
>>   client_limit = 1
>>   drop_priv_before_exec = no
>>   executable = lmtp
>>   extra_groups = $default_internal_group
>>   group = 
>>   idle_kill = 0
>>   privileged_group = 
>>   process_limit = 0
>>   process_min_avail = 0
>>   protocol = lmtp
>>   service_count = 0
>>   type = 
>>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
>> group = mail
>> mode = 0660
>> user = postfix
>>   }
>>   unix_listener lmtp {
>> group = 
>> mode = 0666
>> user = 
>>   }
>>   user = 
>>   vsz_limit = 18446744073709551615 B
>> }
>> 
>> postfix config:
>> 
>> mailbox_transport = lmtp:unix:private/dovecot-lmtp
>> virtual_transport = lmtp:unix:private/dovecot-lmtp
>> 
>> I had confused postfix's own lmtp socket with dovecot's, which was 
>> originally named dovecot-lmtp but when it wasn't created I though my config 
>> was wrong.
>> 
>> What seems to happen is that /var/spool/postfix/private/dovecot-lmtp isn't 
>> created by dovecot at launch, but /var/spool/postfix/private/auth is
>> 
>> G
> 



Re: postfix connects to dovecot lmtp socket, but nothing is delivered

2023-01-05 Thread Gerben Wierda
Boy do I feel stupid.

protocols was not set in the new version's dovecot.conf. And default it only 
contains imap. My old server had an override in dovecot.conf and I copied 
conf.d over but started with afresh dovecot.conf, so lmtp was missing.

G

> On 5 Jan 2023, at 18:39, Gerben Wierda  wrote:
> 
>> On 5 Jan 2023, at 18:17, dove...@ptld.com <mailto:dove...@ptld.com> wrote:
>> 
>>> service lmtp {
>>> inet_listener lmtp {
>>> address = 127.0.0.1
>>> port = 24
>>> }
>> 
>>> unix_listener /var/spool/postfix/private/lmtp {
>>> mode = 0660
>>> user = postfix
>>> group = postfix
>>> }
>> 
>> 
>> Looks like you are activating both a socket and a port. Im not sure what 
>> dovecot does, does it bind to both or does it pick one and ignore the other. 
>> You only need to use one or the other.
>> 
>> And what does your postfix side look like? Where is postfix trying to 
>> connect? To the socket or the port? For using a socket postfix would look 
>> something like:
>> 
>>virtual_transport = lmtp:unix:private/lmtp
>> 
> 
> I removed the tcp port for now.
> 
> / # doveconf service/lmtp
> service lmtp {
>   chroot = 
>   client_limit = 1
>   drop_priv_before_exec = no
>   executable = lmtp
>   extra_groups = $default_internal_group
>   group = 
>   idle_kill = 0
>   privileged_group = 
>   process_limit = 0
>   process_min_avail = 0
>   protocol = lmtp
>   service_count = 0
>   type = 
>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
> group = mail
> mode = 0660
> user = postfix
>   }
>   unix_listener lmtp {
> group = 
> mode = 0666
> user = 
>   }
>   user = 
>   vsz_limit = 18446744073709551615 B
> }
> 
> postfix config:
> 
> mailbox_transport = lmtp:unix:private/dovecot-lmtp
> virtual_transport = lmtp:unix:private/dovecot-lmtp
> 
> I had confused postfix's own lmtp socket with dovecot's, which was originally 
> named dovecot-lmtp but when it wasn't created I though my config was wrong.
> 
> What seems to happen is that /var/spool/postfix/private/dovecot-lmtp isn't 
> created by dovecot at launch, but /var/spool/postfix/private/auth is
> 
> G



Re: postfix connects to dovecot lmtp socket, but nothing is delivered

2023-01-05 Thread Gerben Wierda
> On 5 Jan 2023, at 18:17, dove...@ptld.com wrote:
> 
>> service lmtp {
>> inet_listener lmtp {
>> address = 127.0.0.1
>> port = 24
>> }
> 
>> unix_listener /var/spool/postfix/private/lmtp {
>> mode = 0660
>> user = postfix
>> group = postfix
>> }
> 
> 
> Looks like you are activating both a socket and a port. Im not sure what 
> dovecot does, does it bind to both or does it pick one and ignore the other. 
> You only need to use one or the other.
> 
> And what does your postfix side look like? Where is postfix trying to 
> connect? To the socket or the port? For using a socket postfix would look 
> something like:
> 
>virtual_transport = lmtp:unix:private/lmtp
> 

I removed the tcp port for now.

/ # doveconf service/lmtp
service lmtp {
  chroot = 
  client_limit = 1
  drop_priv_before_exec = no
  executable = lmtp
  extra_groups = $default_internal_group
  group = 
  idle_kill = 0
  privileged_group = 
  process_limit = 0
  process_min_avail = 0
  protocol = lmtp
  service_count = 0
  type = 
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = mail
mode = 0660
user = postfix
  }
  unix_listener lmtp {
group = 
mode = 0666
user = 
  }
  user = 
  vsz_limit = 18446744073709551615 B
}

postfix config:

mailbox_transport = lmtp:unix:private/dovecot-lmtp
virtual_transport = lmtp:unix:private/dovecot-lmtp

I had confused postfix's own lmtp socket with dovecot's, which was originally 
named dovecot-lmtp but when it wasn't created I though my config was wrong.

What seems to happen is that /var/spool/postfix/private/dovecot-lmtp isn't 
created by dovecot at launch, but /var/spool/postfix/private/auth is

G

Re: postfix connects to dovecot lmtp socket, but nothing is delivered

2023-01-05 Thread Gerben Wierda


> On 5 Jan 2023, at 18:17, dove...@ptld.com wrote:
> 
>> service lmtp {
>> inet_listener lmtp {
>> address = 127.0.0.1
>> port = 24
>> }
> 
>> unix_listener /var/spool/postfix/private/lmtp {
>> mode = 0660
>> user = postfix
>> group = postfix
>> }
> 
> 
> Looks like you are activating both a socket and a port. Im not sure what 
> dovecot does, does it bind to both or does it pick one and ignore the other. 
> You only need to use one or the other.
> 
> And what does your postfix side look like? Where is postfix trying to 
> connect? To the socket or the port? For using a socket postfix would look 
> something like:
> 
>virtual_transport = lmtp:unix:private/lmtp

Ha, my mistake. As dovecot had not created the socket I assumed th elmtp socket 
was the one the new config had created, so I changed the config to use that one.

Now I only have to find out why dovecot doesn't create the 
/var/spool/postfix/private/dovecot-lmtp socket

The inet_listener was added to see if I could work around this.

G

postfix connects to dovecot lmtp socket, but nothing is delivered

2023-01-05 Thread Gerben Wierda
I am setting up a new server combination (postfix + dovecot). dovecot's auth 
service works for postfix. Just lmtp does not and I need to find out why. I 
don't see anything in the dovecot logging, but I see this on the postfix side 
(debug)

Jan 05 16:11:59 snape postfix/lmtp[127]: connection established
Jan 05 16:11:59 snape postfix/lmtp[127]: master_notify: status 0
Jan 05 16:11:59 snape postfix/lmtp[127]: deliver_request_initial: send initial 
response
Jan 05 16:11:59 snape postfix/lmtp[127]: send attr protocol = 
delivery_request_protocol
Jan 05 16:16:59 snape postfix/lmtp[126]: smtp_get: timeout
Jan 05 16:16:59 snape postfix/lmtp[126]: connect to subsystem private/defer
Jan 05 16:16:59 snape postfix/lmtp[126]: private/defer socket: wanted 
attribute: protocol
Jan 05 16:16:59 snape postfix/lmtp[126]: input attribute name: protocol
Jan 05 16:16:59 snape postfix/lmtp[126]: input attribute value: 
delivery_status_protocol
Jan 05 16:16:59 snape postfix/lmtp[126]: private/defer socket: wanted 
attribute: (list terminator)
Jan 05 16:16:59 snape postfix/lmtp[126]: input attribute name: (end)
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr nrequest = 0
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr flags = 0
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr queue_id = C71B3D1262
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr original_recipient = 
sy...@rna.nl
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr recipient = sy...@rna.nl
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr offset = 18446744073709551615
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr dsn_orig_rcpt = 
rfc822;sy...@rna.nl
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr notify_flags = 0
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr status = 4.4.2
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr diag_type = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr diag_text = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr mta_type = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr mta_mname = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr action = delayed
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr reason = conversation with 
snape.rna.nl[private/lmtp] timed out while receiving the initial server greeting
Jan 05 16:16:59 snape postfix/lmtp[126]: private/defer socket: wanted 
attribute: status
Jan 05 16:16:59 snape postfix/lmtp[126]: input attribute name: status
Jan 05 16:16:59 snape postfix/lmtp[126]: input attribute value: 0
Jan 05 16:16:59 snape postfix/lmtp[126]: private/defer socket: wanted 
attribute: (list terminator)
Jan 05 16:16:59 snape postfix/lmtp[126]: input attribute name: (end)
Jan 05 16:16:59 snape postfix/lmtp[126]: C71B3D1262: to=, 
relay=snape.rna.nl[private/lmtp], delay=300, delays=0.02/0/300/0, dsn=4.4.2, 
status=deferred (conversation with snape.rna.nl[private/lmtp] timed out while 
receiving the initial server greeting)
Jan 05 16:16:59 snape postfix/lmtp[126]: flush_add: site rna.nl id C71B3D1262
Jan 05 16:16:59 snape postfix/lmtp[126]: match_list_match: rna.nl: no match
Jan 05 16:16:59 snape postfix/lmtp[126]: flush_add: site rna.nl id C71B3D1262 
status 4
Jan 05 16:16:59 snape postfix/lmtp[126]: deliver_request_final: send: 
"conversation with snape.rna.nl[private/lmtp] timed out while receiving the 
initial server greeting" -1
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr status = 4.4.2
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr diag_type = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr diag_text = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr mta_type = 
Jan 05 16:16:59 snape postfix/lmtp[127]: master_notify: status 1
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr mta_mname = 
Jan 05 16:16:59 snape postfix/lmtp[127]: connection closed
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr action = 
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr reason = conversation with 
snape.rna.nl[private/lmtp] timed out while receiving the initial server greeting
Jan 05 16:16:59 snape postfix/lmtp[126]: send attr status = 4294967295
Jan 05 16:16:59 snape postfix/lmtp[126]: master_notify: status 1
Jan 05 16:16:59 snape postfix/lmtp[126]: connection closed

It seems dovecot doesn't react to whatever postfix is doing on the socket. 
Another possibility is that the socket doesn't really work and dovecot never 
gets a connection request. How can I debug this?

Might I test if the socket is the problem by running lmtp on the dovecot side 
on TCP on 127.0.0.1 and have postfix use that? Just to see if it works? If so, 
how do I tell postfix to do that? In dovecot I have added the tcp:

service lmtp {
  inet_listener lmtp {
address = 127.0.0.1
port = 24
  }
  unix_listener /var/spool/postfix/private/lmtp {
mode = 0660
user = postfix
group = postfix
  }
}
service lmtp {
   executable = lmtp -L
}
protocol lmtp {
   info_log_path = /var/log/mail/dovecot-lmtp.log
}


Any other tips?

Gerben Wierda (LinkedIn <https:/

Up to date dovecot packages on Ubuntu 22.4LTS

2023-01-04 Thread Gerben Wierda
I just found out that Ubuntu Linux 22.04 LTS has a dovecot apt package that is 
2.3.16. Debian 11 (stable) is at dovecot 2.3.13 with apt. if I looked 
correctly. Dovecot itself is at 2.3.20.

That surprised me (I am new to the Linux distro and package management world) 
as Ubuntu 22.04 is a Long-Time Support version, and I was expecting packages to 
be updated as well (as many users will want updated packages.  How does that 
work in the dovecot world? How come macOS MacPorts (2.3.19) has more recent 
packages than any of the large Debian-based Linuxes?

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>



Re: Migrating, syncing, maybe load-balancing/failover two dovecot servers?

2023-01-04 Thread Gerben Wierda
ylib   
   0x0001050f52e6 io_loop_run + 81 -> 18  libdovecot.0.dylib
  0x00010506d5e0 master_service_run + 24 -> 19  doveadm-server  
0x0001048aac3f main + 292 -> 20  dyld   
 0x00011487652e start + 462
Jan 04 20:17:05 doveadm(74522): Fatal: master: service(doveadm): child 74522 
killed with signal 6 (core dumps disabled - 
https://dovecot.org/bugreport.html#coredumps)

Turns out, this is a known (and pretty old) problem 
(https://www.mail-archive.com/dovecot%40dovecot.org/msg85388.html) and my 
dovecot on the old server (macOS + MacPorts) is newer than the dovecot on the 
new one. I should go back to a 2.3.16 on the old server.

It seems the syncing works (or has worked) nonetheless, but it doesn't feel 
good.

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>

> On 4 Jan 2023, at 13:54, Paul Kudla  wrote:
> 
> 
> maybe look a replicator / replication
> 
> its designed to do exactly that
> 
> 
> 
> 
> Happy Wednesday !!!
> Thanks - paul
> 
> Paul Kudla
> 
> 
> Scom.ca Internet Services <http://www.scom.ca>
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
> 
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> Email p...@scom.ca
> 
> On 1/4/2023 7:46 AM, Gerben Wierda wrote:
>> I am in the process of migrating from dovecot on one OS (macOS/darwin) to a 
>> new server running dovecot with another OS (Ubuntu Linux 22.4).
>> I have mostly copied/adapted the setup of the old server to the new. I am in 
>> the process of finishing that and adding some stuff that still needs to be 
>> added/migrated, like rspamd. And the data of course before the new one takes 
>> over from the old.
>> I have done a migration before (MacOS X Server dovecot to MacPorts dovecot 
>> on macOS), many years ago, I recall that I used dovecot syncing but also 
>> rsync and I don't really recall (and anyway, the software has changed since)
>> I have been thinking about keeping them both alive, with one as a failover 
>> for the other. They will not share their storage (e.g. NFS), So, I was 
>> wondering if I can do something with syncing between instances and dovecot 
>> director. I have been looking at the documentation, but a quick scan reveals 
>> I cannot locate some sort of tutorial and I am uncertain what will work and 
>> what not.
>> If keeping both alive in parallel is too problematic, it is OK to have 
>> regular syncing in one direction (old to new) at first and then switch over 
>> and have syncing in the other direction (new to old)
>> Can someone enlighten me?
>> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
>> R&A IT Strategy <https://ea.rna.nl/> (main site)
>> Book: Chess and the Art of Enterprise Architecture 
>> <https://ea.rna.nl/the-book/>
>> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
>> -- 
>> This message has been scanned for viruses and
>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
>> believed to be clean.



CRAM MD5 passwd db permission issue

2023-01-04 Thread Gerben Wierda
I am busy migrating. I am moving from macOS+MacPorts to Ubuntu+Docker

On the old system, I have this in the dovecot config:
mail_uid = _dovecot
mail_gid = mail
mail_privileged_group = mail
mail_access_groups = mail

This seems weird to me, I think the dovecot user should be in group dovecot 
only if I understand the docs. On the old system dovecot, postfix, dovenull and 
rspamd are all members of the mail group.

On that system, the cram md5 passwd database (file) has these permissions:

drwxr-xr-x   3 root  wheel96 Feb  2  2021 .
drwxr-xr-x  22 root  admin   704 Jan  4 15:17 ..
-rw-r-   1 root  mail   1234 Feb  2  2021 cram-md5.pwd

and that has worked like that for many years, basically starting with Mac OS X 
Server, surviving all kinds of macOS migrations.

On my new Ubuntu system I've copied this setup over:
drwxr-xr-x 2 root root 4096 Jan  4 09:49 .
drwxr-xr-x 7 root root 4096 Jan  4 15:21 ..
-rw-r- 1 root mail 1234 Feb  2  2021 cram-md5.pwd

mail_uid = dovecot
mail_gid = mail
mail_privileged_group = mail
mail_access_groups = mail

But:
Jan 04 15:40:08 auth: Error: passwd-file 
/etc/dovecot/etc/cram-md5.pwd:open(/etc/dovecot/etc/cram-md5.pwd) failed: 
Permission denied (euid=91(dovecot) egid=91(dovecot) missing +r perm: 
/etc/dovecot/etc/cram-md5.pwd, we're not in group 8(mail), dir owned by 0:0 
mode=0755)

And really, dovecot is in group mail. From /etc/group:
mail:x:8:postfix,dovecot
dovenull:x:90:
dovecot:x:91:
And from /etc/passwd:
dovenull:x:90:90::/home/dovenull:/usr/sbin/nologin
dovecot:x:91:91::/home/dovecot:/usr/sbin/nologin

So, that I get this error baffles me.

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>



Migrating, syncing, maybe load-balancing/failover two dovecot servers?

2023-01-04 Thread Gerben Wierda
I am in the process of migrating from dovecot on one OS (macOS/darwin) to a new 
server running dovecot with another OS (Ubuntu Linux 22.4).

I have mostly copied/adapted the setup of the old server to the new. I am in 
the process of finishing that and adding some stuff that still needs to be 
added/migrated, like rspamd. And the data of course before the new one takes 
over from the old.

I have done a migration before (MacOS X Server dovecot to MacPorts dovecot on 
macOS), many years ago, I recall that I used dovecot syncing but also rsync and 
I don't really recall (and anyway, the software has changed since)

I have been thinking about keeping them both alive, with one as a failover for 
the other. They will not share their storage (e.g. NFS), So, I was wondering if 
I can do something with syncing between instances and dovecot director. I have 
been looking at the documentation, but a quick scan reveals I cannot locate 
some sort of tutorial and I am uncertain what will work and what not.

If keeping both alive in parallel is too problematic, it is OK to have regular 
syncing in one direction (old to new) at first and then switch over and have 
syncing in the other direction (new to old)

Can someone enlighten me?

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>



macOS ManageSieve client?

2022-02-19 Thread Gerben Wierda
I have a dovecot & dovecot-sieve running under MacPorts on my macOS ’server’. I 
do a very low tech maintenance on sieve, by using the cli on the server and 
edit sieve by hand for myself. I’d like to open this up to other users.

How do people use this from their macOS clients? For this, the ManageSieve 
protocol exists and this is implemented by dovecot-sieve, but other than 
installing roundcube and offering a web-based mail client that also supports 
ManageSieve, is there another way? A ManageSieve client that directly runs on 
the macOS client and interfaces with dovecot-sieve on the server?

After all, routing mail to different mailboxes better happens server-side so 
that all clients profit.

Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda>)
R&A IT Strategy <https://ea.rna.nl/> (main site)
Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>



Re: Is it possible to enable push notifications for iOS/macOS mail clients on a stock dovecot?

2020-01-08 Thread Gerben Wierda
This requires the ongoing use of macOS Server (as does the patch that is part 
of the MacPorts port of dovecot2). The issue then is: how do you keep getting 
certificates for push mail if mail services have been removed from macOS 
Server? I can configure this today but on Oct 20, my apns certificate expires. 
Then what?

Gerben Wierda
Chess and the Art of Enterprise Architecture
Mastering ArchiMate
Architecture for Real Enterprises at InfoWorld
On Slippery Ice at EAPJ

> On 9 Jan 2020, at 00:19, Steffan Cline  wrote:
> 
> https://github.com/st3fan/dovecot-xaps-daemon
> 
> Works PERFECTLY. I use it on CentOS 6 and 7. 
> 
> 
> Thanks,
> Steffan Cline
> stef...@hldns.com
> 602-793-0014
> 
>>> On Jan 8, 2020, at 2:39 PM, infoomatic  wrote:
>>> 
>> I have no idea about that Mac proprietary stuff ... however,
>> https://doc.dovecot.org/configuration_manual/push_notification/ might be
>> a solution/workaround for you.
>> 
>> regards,
>> 
>> infoomatic
>> 
>> 
>>> Am 08.01.20 um 22:33 schrieb Gerben Wierda:
>>> Now that Apple has abandoned mail server capabilities in macOS Server.app 
>>> we have to move to an open source solution. I have done this using Macports 
>>> and it works well (no thanks to Apple as it never published the in 2018 
>>> promised migration info for mail services).
>>> 
>>> The MacPorts Apple Push Notification variant based on a patch written a few 
>>> years ago for dovecot supports APNS but it requires you to export a 
>>> certificate form your old High Sierra Server.app. With the patch and that 
>>> certificate, dovecot tells an email client it supports XAPPLEPUSHSERVICE 
>>> and that makes Apple mail clients think they are talking to macOS Server 
>>> and they switch to push notifications for updates. Now, that certificate 
>>> needs to be renewed yearly by Server.app and imported by hand etc. At some 
>>> point this will stop working (apart from that you need to keep an old macOS 
>>> Server around to renew)
>>> 
>>> Is there another way to get this working with an open source dovecot, 
>>> outside of using Apple’s push notification service? Some other standard? 
>>> Some other service? Probably not, right?
>>> 
>>> G
>> 


Is it possible to enable push notifications for iOS/macOS mail clients on a stock dovecot?

2020-01-08 Thread Gerben Wierda
Now that Apple has abandoned mail server capabilities in macOS Server.app we 
have to move to an open source solution. I have done this using Macports and it 
works well (no thanks to Apple as it never published the in 2018 promised 
migration info for mail services).

The MacPorts Apple Push Notification variant based on a patch written a few 
years ago for dovecot supports APNS but it requires you to export a certificate 
form your old High Sierra Server.app. With the patch and that certificate, 
dovecot tells an email client it supports XAPPLEPUSHSERVICE and that makes 
Apple mail clients think they are talking to macOS Server and they switch to 
push notifications for updates. Now, that certificate needs to be renewed 
yearly by Server.app and imported by hand etc. At some point this will stop 
working (apart from that you need to keep an old macOS Server around to renew)

Is there another way to get this working with an open source dovecot, outside 
of using Apple’s push notification service? Some other standard? Some other 
service? Probably not, right?

G

Re: iOS Client Not Showing Email Previews, Dovecot Issue?

2020-01-02 Thread Gerben Wierda
> On 2 Jan 2020, at 01:18, a...@globalchangemusic.org wrote:
> 
> Greetings,
> 
> Just wondering if anybody would know why one of our users' iOS Mail app 
> client doesn't show the email previews (the first few sentences of the email)?
> 
> We had migrated from an earlier (2.2.4) version of Dovecot to the latest 
> version and once that was done, the user began to report this issue.  I don't 
> know if it's a Dovecot issue or not, but can anyone point me in the right 
> direction to begin to debug?

I am running the latest MacPorts dovecot and latest iOS and see the first line.

I have run into some issues with Mail.app on macOS that were solved by removing 
the account and recreating again. macOS Mail.app kept settings around on how to 
connect to IMAP (from years of upgrading an existing dovecot). These were 
solved only when I recreated the mail account on the client side so th eclient 
queried the server anew and created its settings anew.

It’s not really debugging but trial and error, but then, debugging anything on 
iOS (anything Apple, really) is hard to do.

G



spamsieve question

2019-12-31 Thread Gerben Wierda
My new dovecot covers spam/ham training through the standard Spam_train and 
Notspam_train mailboxes. When a message is copied (I’m using macOS/iOS standard 
Apple clients) it triggers the spam/ham learning by rspamd.

So far so good.

When I move a message to Spam_train I’d like it to be removed immediately after 
learning.
When I move a message to Notspam_train, I’d like it to move back where it came 
from after having been piped

The first is I think doable, but how? Is the second doable?

Gerben Wierda
Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ



Works, but why? Was: Trouble using reverse dsync backup (log in successful but almost nothing happens)

2019-12-23 Thread Gerben Wierda
Answering my own question: removing the existing mailbox and running again 
worked. I don’t know why, I think it should have made the existing mailbox 
equivalent to the backupped one, shouldn’t it?

Gerben Wierda
Chess and the Art of Enterprise Architecture <http://enterprisechess.com/>
Mastering ArchiMate <http://masteringarchimate.com/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ

> On 23 Dec 2019, at 01:47, Gerben Wierda  wrote:
> 
> I am migrating an old dovecot 2.2.30 (macOS High SIerra, macO S Server) to a 
> new server with dovecot 2.3.7 (macOS Mojave, macports).
> 
> To get the data over, I’m trying to do a reverse backup with doveadm between 
> an old and a new mail server, but failing to do so. I’m trying (from the new 
> server, the old server is 192.168.2.67):
> 
> doveadm -Dv \
>   -o imapc_host=192.168.2.67 \
>   -o imapc_user=gerben \
>   -o imapc_password=’[snip]' \
>   -o imapc_features='rfc822.size fetch-headers’ \
>   -o mail_prefetch_count=20 \
>   -o mail_fsync=never \
>   -o imapc_ssl=starttls \
>   -o imapc_ssl_verify=no \
>backup -R -u gerben imapc:
> 
> It seems to start to work:
> 
> dsync(gerben): Debug: imapc(192.168.2.67:143): Created new connection
> dsync(gerben): Debug: imapc(192.168.2.67:143): Looking up IP address 
> (reconnect_ok=true, last_connect=1577059834)
> dsync(gerben): Debug: imapc(192.168.2.67:143): Connecting to 192.168.2.67:143
> dsync(gerben): Info: imapc(192.168.2.67:143): Connected to 192.168.2.67:143 
> (local 192.168.2.66:50804)
> dsync(gerben): Debug: imapc(192.168.2.67:143): Server capabilities: IMAP4rev1 
> LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED 
> XAPPLEPUSHSERVICE
> dsync(gerben): Debug: imapc(192.168.2.67:143): Starting SSL handshake
> dsync(gerben): Debug: imapc(192.168.2.67:143): SSL handshake successful, 
> ignoring invalid certificate: SSL certificate doesn't match expected host 
> name 192.168.2.67: No match to 2 SubjectAltNames
> dsync(gerben): Debug: imapc(192.168.2.67:143): Server capabilities: IMAP4rev1 
> LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE XAPPLEPUSHSERVICE 
> AUTH=CRAM-MD5 AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=GSSAPI
> dsync(gerben): Debug: imapc(192.168.2.67:143): Authenticating as gerben
> dsync(gerben): Debug: imapc(192.168.2.67:143): Server capabilities: IMAP4rev1 
> LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY 
> THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL 
> CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 
> CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS 
> BINARY MOVE XAPPLEPUSHSERVICE COMPRESS=DEFLATE QUOTA ACL RIGHTS=texk
> dsync(gerben): Debug: imapc(192.168.2.67:143): Authenticated successfully
> dsync(gerben): Debug: imapc: root=, index=, indexpvt=, control=, inbox=, alt=
> dsync(gerben): Debug: fts: Indexes disabled for namespace ''
> dsync(gerben): Debug: brain S: Namespace  has location imapc:
> dsync(gerben): Debug: brain M: Local mailbox tree: INBOX 
> guid=184a4c066b219a5dd7379e3b11ac uid_validity=1570382187 uid_next=18 
> subs=no last_change=0 last_subs=0
> dsync(gerben): Debug: brain S: Local mailbox tree: INBOX 
> guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1232523592 uid_next=482711 
> subs=no last_change=0 last_subs=0
> dsync(gerben): Debug: brain M: Local mailbox tree: Notspam_train 
> guid=e0cc8c076b219a5dd7379e3b11ac uid_validity=1570382188 uid_next=10 
> subs=no last_change=0 last_subs=0
> dsync(gerben): Debug: brain S: Local mailbox tree: zzz-Old 
> guid=c71fb29dcfd52e98ffb476033482311a uid_validity=1246525241 uid_next=1 
> subs=yes last_change=0 last_subs=0
> dsync(gerben): Debug: brain M: Local mailbox tree: Sent 
> guid=e8c3b0076b219a5dd7379e3b11ac uid_validity=1570382189 uid_next=17 
> subs=yes last_change=0 last_subs=1570382187
> dsync(gerben): Debug: brain S: Local mailbox tree: zzz-Old/Werk 
> guid=493c1b0ff8829799ae854dcdd35bdc49 uid_validity=1246525654 uid_next=1 
> subs=yes last_change=0 last_subs=0
> dsync(gerben): Debug: brain M: Local mailbox tree: Spam_train 
> guid=7076d5076b219a5dd7379e3b11ac uid_validity=1570382190 uid_next=3719 
> subs=no last_change=0 last_subs=0
> dsync(gerben): Debug: brain S: Local mailbox tree: 
> zzz-Old/Werk/UitNFIMailboxen guid=1da8c05ea358b80ee1c2f3a01cc7f2cc 
> uid_validity=1232523595 uid_next=118 subs=yes last_change=0 last_subs=0
> dsync(gerben): Debug: brain M: Local mailbox tree: Trash 
> guid=4810f5076b219a5dd7379e3b11ac uid_validity=1570382191 uid_next=3 
> subs=no l

Trouble using reverse dsync backup (log in successful but almost nothing happens)

2019-12-22 Thread Gerben Wierda
ug: brain S: Remote mailbox tree: Drafts 
guid=60cd72148903005e6b5501009e3b11ac uid_validity=1570382198 uid_next=1 
subs=yes last_change=0 last_subs=1570382187
dsync(gerben): Debug: imapc(192.168.2.67:143): Disconnected

But nothing has been copied over and while it states it has deleted local 
mailboxes (such as Subfolder/SUbSubFolder), they are still there. In other 
words: nothing happens.

Well, not quite. The first time I ran this, local directory 
/var/mail/nl.rna.mail/rules/gerben/gerben/tmp was created.

Can anyone see here what is going wrong?

Gerben Wierda
Chess and the Art of Enterprise Architecture <http://enterprisechess.com/>
Mastering ArchiMate <http://masteringarchimate.com/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ



Re: Is it safe to mix dovecot import/backup/sync between dovecot versions 2.2.30.2 and 2.3.7.1?

2019-12-22 Thread Gerben Wierda
Thank you.

> On 22 Dec 2019, at 12:36, Sami Ketola  wrote:
> 
> 
>> On 22 Dec 2019, at 12.22, Gerben Wierda  wrote:
>> 
>> I am migrating a server. The old one uses dovecot 2.2.30.2 and the new one 
>> 2.3.7.1. I don’t want to use any tool that might influence flags on 
>> messages. I am using Maildir format on both ends and I have a way to let one 
>> server access the data on the other (either using an rsync or a direct 
>> access).
>> 
>> Could I set up a sync between both to create a smooth migration?
> 
> rsync probably works out fine, but I would use native dovecot dsync myself. 

So just for certainty: I canjust perfectly copy over the 2.2.30 Maildir to the 
new server for each user and dovecot 2.3.7 would just work? No need to do 
anything else, such as recreating indexes etc? That would probably be perfect 
for me.

> 
> Something like this run on the new server:
> 
> /usr/bin/doveadm backup -u  -R ssh adminuser@old-server "sudo 
> /usr/bin/doveadm dsync-server -u ”

For this I would have to create the ssh-setup for adminuser which I’re rather 
prevent to have to do, because the _dovecot user doesn’t have a login shell 
etc. and the servers are rather closed down with respect to ssh etc for 
security reasons.

I have to admit I haven't really had to administer dovecot before (macOS Server 
was doing that for me) so I’m quite overwhelmed with all the settings, options, 
configurations and workings.

It would be nice to have something like this in a crontab of the *old* server:

doveadm sync remote:server-replica.example.com

but I have no idea what I have to do in terms of getting this to work safely. 
Probably set up a master user on the new server’s end? Since all instructions I 
can find are based on some sort of Linux (and my servers are macOS) I have no 
idea yet what to do *exactly*. There are also no imapc entries yet on both 
servers in the config. User database on macOS is different as far as I know 
(LDAP: Open DIrectory). I am running into not understanding this, for instance 
from the Wiki:
# Authenticate as masteruser / masteruser-secret, but use a separate login user.
# If you don't have a master user, remove the imapc_master_user setting.
imapc_user = %u
imapc_master_user = masteruser
imapc_password = masteruser-secret
And that leaves me wondering: what does "imapc_user = %u" do exactly? Can I use 
any existing local user to run the doveadm commands and will the %u become the 
user name of that user for instance?

I other words, I’m drowning in details and a simple copy of the mail 
directories would be a perfect solution.

G

Is it safe to mix dovecot import/backup/sync between dovecot versions 2.2.30.2 and 2.3.7.1?

2019-12-22 Thread Gerben Wierda
I am migrating a server. The old one uses dovecot 2.2.30.2 and the new one 
2.3.7.1. I don’t want to use any tool that might influence flags on messages. I 
am using Maildir format on both ends and I have a way to let one server access 
the data on the other (either using an rsync or a direct access).

Could I set up a sync between both to create a smooth migration?

Gerben Wierda
Chess and the Art of Enterprise Architecture <http://enterprisechess.com/>
Mastering ArchiMate <http://masteringarchimate.com/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ



Is it possible to create and use some lookup table to replace (for instance) %u in mail_location?

2019-10-06 Thread Gerben Wierda via dovecot
Suppose my mail_location is:

mail_location = maildir:/private/var/mail/nl.rna.mail/%u

But I would like to use a simple lookup table to replace %u with another value, 
say '8DB4E345-9144-4F92-8C9F-9D47BB61CBF8’ instead of ‘username’. Is there a 
(simple) way to do this?

Gerben Wierda
Chess and the Art of Enterprise Architecture <http://enterprisechess.com/>
Mastering ArchiMate <http://masteringarchimate.com/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ



Re: New to dovecot admin, question about using LDAP for user-specific values

2019-09-13 Thread Gerben Wierda via dovecot

> On 13 Sep 2019, at 11:51, Jean-Daniel Dupas  wrote:
> 
> 
> 
>> Le 13 sept. 2019 à 09:29, Gerben Wierda via dovecot > <mailto:dovecot@dovecot.org>> a écrit :
>> 
>> Nobody?
>> 
>>> On 10 Sep 2019, at 11:58, Gerben Wierda via dovecot >> <mailto:dovecot@dovecot.org>> wrote:
>>> 
>>> I am new to dovecot administration. I’ve read the Wiki but that hasn’t 
>>> given me the understanding I need.
>>> 
>>> When I query my LDAP (on macOS) on a value for user ‘gerben’, I can get 
>>> that:
>>> 
>>> dumbledore:~ gerben$ dscl /LDAPv3/127.0.0.1 -read /users/gerben GeneratedUID
>>> GeneratedUID: 780D870E-6B00-478E-AB70-3D3307215A82
>>> 
>>> I would like to use that value in dovecot settings, e.g. something like
>>> 
>>> user_attrs = \
>>>   =mail=maildir://Library/Server/Mail/Data/mail/%{ldap:GeneratedUID} 
>>> 
>>> 
>>> Is this possible and if so what do I exactly need to do to get this working?
> 
> As the answer is in the question, it is hard to give you any hint about what 
> should be done.
> 
> What is wrong with 
> 
> user_attrs = \
>   =mail=maildir://Library/Server/Mail/Data/mail/%{ldap:GeneratedUID} 
> 
> 
> Did you try it ? Have you got any issue with it ?

I haven’t tried anything yet as I am trying to learn before I do anything (and 
trial and error is a very slow method), but it seems to me that just that line 
cannot be enough. Because how does “ldap:” know to go looking in the LDAP 
structure at "/Users//“ in the LDAP “/LDAPv3/127.0.0.1”? Somehow I 
shall have to tell dovecot that.

But if trial and error is the only way, I’ll have to try.

G

Re: New to dovecot admin, question about using LDAP for user-specific values

2019-09-13 Thread Gerben Wierda via dovecot
Nobody?

> On 10 Sep 2019, at 11:58, Gerben Wierda via dovecot  
> wrote:
> 
> I am new to dovecot administration. I’ve read the Wiki but that hasn’t given 
> me the understanding I need.
> 
> When I query my LDAP (on macOS) on a value for user ‘gerben’, I can get that:
> 
> dumbledore:~ gerben$ dscl /LDAPv3/127.0.0.1 -read /users/gerben GeneratedUID
> GeneratedUID: 780D870E-6B00-478E-AB70-3D3307215A82
> 
> I would like to use that value in dovecot settings, e.g. something like
> 
> user_attrs = \
>   =mail=maildir://Library/Server/Mail/Data/mail/%{ldap:GeneratedUID} 
> 
> 
> Is this possible and if so what do I exactly need to do to get this working?
> 
> Gerben Wierda
> Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
> Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
> Architecture for Real Enterprises 
> <https://www.infoworld.com/blog/architecture-for-real-enterprises/> at 
> InfoWorld
> On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ
> 



New to dovecot admin, question about testing a dovecot configuration

2019-09-10 Thread Gerben Wierda via dovecot
I am new to dovecot administration, not new to postfix administration.

When setting up a new postfix/dovecot combo from scratch, can I start with 
dovecot alone and still be able to test it? E.g. can I address the local 
delivery agent with some sort of test email? If so, how? I haven’t been able to 
find it in the wiki.

Gerben Wierda
Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ



New to dovecot admin, question about using LDAP for user-specific values

2019-09-10 Thread Gerben Wierda via dovecot
I am new to dovecot administration. I’ve read the Wiki but that hasn’t given me 
the understanding I need.

When I query my LDAP (on macOS) on a value for user ‘gerben’, I can get that:

dumbledore:~ gerben$ dscl /LDAPv3/127.0.0.1 -read /users/gerben GeneratedUID
GeneratedUID: 780D870E-6B00-478E-AB70-3D3307215A82

I would like to use that value in dovecot settings, e.g. something like

user_attrs = \
  =mail=maildir://Library/Server/Mail/Data/mail/%{ldap:GeneratedUID}

Is this possible and if so what do I exactly need to do to get this working?

Gerben Wierda
Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/>
Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/>
Architecture for Real Enterprises 
<https://www.infoworld.com/blog/architecture-for-real-enterprises/> at InfoWorld
On Slippery Ice <https://eapj.org/on-slippery-ice/> at EAPJ



Checking sieve

2016-12-19 Thread Gerben Wierda
A simple question: I’ve installed roundcube on macOS Sierra with Server 5.2 and 
created a set of sieve rules in my dovecot setup. How can I check their 
functioning (other than send test mails) e.g. is there some logging I can set / 
look at?

G

Managing Sieve on macOS 10.12 Sierra with Server 5.2?

2016-12-17 Thread Gerben Wierda
I don’t know if this is considered OT, but dovecot has support for sieve. I had 
a rule set running under Mac OS X 10.8.5 Server 2.2.5 which I did maintain by 
hand on the server because a while before Mac OS X Server had dropped support 
for Squirrelmail.

Now I wonder if there is a decent way to get sieve running again on macOS 10.12 
Sierra with Server 5.2

G

Maildir: do message files need to have a unique name across all directories?

2016-12-15 Thread Gerben Wierda
While testing a new mail server I created a temporary account and filled its 
mailbox (Milder, dovecot) with the contents of another account. That means the 
actual message files in each account have identical names. Is that a problem? 
(I’m asking because I see some weird behaviour of visible/invisible messages 
and I wonder if this could have been the reason).

G

Re: Messed up dovecot mail store, need some repair advice

2016-10-22 Thread Gerben Wierda

> On 22 Oct 2016, at 22:09, Gerben Wierda  wrote:
> 
>> 
>> On 22 Oct 2016, at 19:04, Aki Tuomi  wrote:
>> 
>> 
>>> On October 22, 2016 at 7:51 PM Gerben Wierda  wrote:
>>> 
>>> 
>>> Hello folks,
>>> 
>>> I have an older dovecot 2.0 (which I will migrate to a 2.2 asap, but at 
>>> this point in time I need a fix). This is a dovecot 2.0 that came with Mac 
>>> OS X 10.8.5 Server 2.2.5.
>>> 
>>> Today, my spam/virus filtering (clamav) on the Server broke down. As a 
>>> result, all my meesages got the ***UNCHECKED*** tag added to each subject. 
>>> That was clearly unacceptable. SO, for the tim ebeing I have set that tag 
>>> to undef so the tag is no longer added. But I also wanted to repair the 
>>> messages that already ended up in dovecot 2.0
>>> 
>>> So, I did something simple: stopped all mail services on the server, went 
>>> into the dovecot mail store and edited the messages. I first tried with one 
>>> small ccount and it seemed OK. But now my mail client is experiencing 
>>> problems with the messages (cannot display) and I think I've been to 
>>> simplistic. I have for instance notedthat th esizeof the message is part of 
>>> the filename. So, I can change these of course, but probably I need to 
>>> change more.
>>> 
>>> Can someone enlighten me how I can repair the broken data store?
>>> 
>>> Thanks,
>>> 
>>> (Foolish) Gerben
>> 
>> I think your best bet is to delete dovecot.index* and run dovecot index -u 
>> username. Or revert all your changes. Depending what mail store you are 
>> using, this might lose any flags on your mails, such as \Read.
> 
> Thanks. Losing 300 flags and unread on thousands of emails was not a 
> preferred scenario.
> 
> I was able to repair by 
> - turning dovecote (and other mail services) off
> - find all message files that were changed in a certain period
> - check all their names against their file sizes (this found me the edited 
> ones)
> - returning the ***UNCHECKED*** string to the Subject lines making the file 
> sizes equal to the size as reported in the name of the file 
> 
> Which leaves me with something I would really like: change the subject line 
> of 5-10 messages in dovecot, without destroying everything.
> 
> I was thinking about the following scenario:
> - create a separate mailbox REPAIR within user X’s mail store (the INBOX, 
> btw, is named ‘cur’)
> - move all to be changed messages there using the mail client
> - kill the mail client
> - stop dovecot
> - edit the messages and change the names of the files so the S= W= parts are 
> in line with the new content. (I understand S, but what is W?)
> - run “devoid index -u user REPAIR”
> - start dovecot
> - start email client (potentially, reload the entire mail store for that user)
> 
> Would that work?

There was an easier solution. In my mail program I created a local mailbox, 
copied the messages there, edited them on disk, rebuilt the local mailbox and 
then moved them back to IMAP.

G

Re: Messed up dovecot mail store, need some repair advice

2016-10-22 Thread Gerben Wierda

> On 22 Oct 2016, at 19:04, Aki Tuomi  wrote:
> 
> 
>> On October 22, 2016 at 7:51 PM Gerben Wierda  wrote:
>> 
>> 
>> Hello folks,
>> 
>> I have an older dovecot 2.0 (which I will migrate to a 2.2 asap, but at this 
>> point in time I need a fix). This is a dovecot 2.0 that came with Mac OS X 
>> 10.8.5 Server 2.2.5.
>> 
>> Today, my spam/virus filtering (clamav) on the Server broke down. As a 
>> result, all my meesages got the ***UNCHECKED*** tag added to each subject. 
>> That was clearly unacceptable. SO, for the tim ebeing I have set that tag to 
>> undef so the tag is no longer added. But I also wanted to repair the 
>> messages that already ended up in dovecot 2.0
>> 
>> So, I did something simple: stopped all mail services on the server, went 
>> into the dovecot mail store and edited the messages. I first tried with one 
>> small ccount and it seemed OK. But now my mail client is experiencing 
>> problems with the messages (cannot display) and I think I've been to 
>> simplistic. I have for instance notedthat th esizeof the message is part of 
>> the filename. So, I can change these of course, but probably I need to 
>> change more.
>> 
>> Can someone enlighten me how I can repair the broken data store?
>> 
>> Thanks,
>> 
>> (Foolish) Gerben
> 
> I think your best bet is to delete dovecot.index* and run dovecot index -u 
> username. Or revert all your changes. Depending what mail store you are 
> using, this might lose any flags on your mails, such as \Read.

Thanks. Losing 300 flags and unread on thousands of emails was not a preferred 
scenario.

I was able to repair by 
- turning dovecote (and other mail services) off
- find all message files that were changed in a certain period
- check all their names against their file sizes (this found me the edited ones)
- returning the ***UNCHECKED*** string to the Subject lines making the file 
sizes equal to the size as reported in the name of the file 

Which leaves me with something I would really like: change the subject line of 
5-10 messages in dovecot, without destroying everything.

I was thinking about the following scenario:
- create a separate mailbox REPAIR within user X’s mail store (the INBOX, btw, 
is named ‘cur’)
- move all to be changed messages there using the mail client
- kill the mail client
- stop dovecot
- edit the messages and change the names of the files so the S= W= parts are in 
line with the new content. (I understand S, but what is W?)
- run “devoid index -u user REPAIR”
- start dovecot
- start email client (potentially, reload the entire mail store for that user)

Would that work?

G

Messed up dovecot mail store, need some repair advice

2016-10-22 Thread Gerben Wierda
Hello folks,

I have an older dovecot 2.0 (which I will migrate to a 2.2 asap, but at this 
point in time I need a fix). This is a dovecot 2.0 that came with Mac OS X 
10.8.5 Server 2.2.5.

Today, my spam/virus filtering (clamav) on the Server broke down. As a result, 
all my meesages got the ***UNCHECKED*** tag added to each subject. That was 
clearly unacceptable. SO, for the tim ebeing I have set that tag to undef so 
the tag is no longer added. But I also wanted to repair the messages that 
already ended up in dovecot 2.0

So, I did something simple: stopped all mail services on the server, went into 
the dovecot mail store and edited the messages. I first tried with one small 
ccount and it seemed OK. But now my mail client is experiencing problems with 
the messages (cannot display) and I think I've been to simplistic. I have for 
instance notedthat th esizeof the message is part of the filename. So, I can 
change these of course, but probably I need to change more.

Can someone enlighten me how I can repair the broken data store?

Thanks,

(Foolish) Gerben

Migrating users from a 2.0.19 to a 2.2.24 installation

2016-10-20 Thread Gerben Wierda
Hello,

I am currently still running an older dovecot (2.0.19apple1 on Mac OS X 10.8.5) 
and I want to migrate my users to a new server (macOS 10.12 with Server 5, 
which contains dovecot 2.2.24 (a82c823)).

Basically, I want to create a new server installation on the new server so I 
don't bring any junk over (new user accounts, with the same uid/gid (still need 
to figure that one out), but after I have done that I need to move the data 
over from the old instalation to the new.

Has anything changed in the formats between 2.0 and 2.2 that will stop me from 
doing this?

Thanks,

G