Re: mail-crypt long-term support? How to backup?

2023-07-29 Thread Aki Tuomi via dovecot 


 
 
  
    
  
  
   
On 30/07/2023 08:29 EEST eaerhaerhaehae aehraerhaeha  wrote:
   
   
 
   
   
 
   
   

 Hello everyone,


  


 1. I was wondering whether mail-crypt is intended to be supported in the long run.


  

   
  
  
    
  
  
   Yes. There are no plans to remove it and it is widely used.
  
  
   

 2. Furthermore, I was wondering about the best way to backup a whole email server with mail-crypt enabled.


  


 (3. how do i use dsync with mail-crypt?)

   
  
  
   If you are using per-user passwords, tar and rsync are probably best. dsync only works if it can decrypt users emails.
  
  
   

  


 Below are my thoughts so far. 


  


 - doveadm seems to work on each user individually, and it seems that the password is required for this? At least tachtler/dovecot-backup complains about missing the key and password to decrypt the mails, and so does "dsync -f -u a...@bc.de backup maildir:bc.de/a"

   
  
  
   Yep. As it needs to decrypt them mails.
  
  
   

 - I was going to just tar /var/mail, but it appears that maildirlock is deprecated and broken, so I would have to shut down dovecot temporarily, in order to maintain file/index integrity, which is not ideal.

   
  
  
    
  
  
   You can find a fixed version at https://github.com/dovecot/tools/blob/main/maildirlock.c
  
  
   

  


 - Running it in some kind of virtualized environment to be able to take atomic snapshots might work, but I read that the "uuid list", or whatever, is only updated "lazily". Would a shutdown even solve that?


  


 - I just saw that replication is going away. Maybe it would not have supported mail-crypt anyway.

   
  
  
    
  
  
   Not with per user passwords, no.
  
  
   

  


 Thanks for any input.

   
  
  
   Regards,
  
  
   Aki
  
 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

mail-crypt long-term support? How to backup?

2023-07-29 Thread eaerhaerhaehae aehraerhaeha 
Hello everyone,

 

1. I was wondering whether mail-crypt is intended to be supported in the long run.

 

2. Furthermore, I was wondering about the best way to backup a whole email server with mail-crypt enabled.

 

(3. how do i use dsync with mail-crypt?)

 

Below are my thoughts so far. 

 

- doveadm seems to work on each user individually, and it seems that the password is required for this? At least tachtler/dovecot-backup complains about missing the key and password to decrypt the mails, and so does "dsync -f -u a...@bc.de backup maildir:bc.de/a"

 

- I was going to just tar /var/mail, but it appears that maildirlock is deprecated and broken, so I would have to shut down dovecot temporarily, in order to maintain file/index integrity, which is not ideal.

 

- Running it in some kind of virtualized environment to be able to take atomic snapshots might work, but I read that the "uuid list", or whatever, is only updated "lazily". Would a shutdown even solve that?

 

- I just saw that replication is going away. Maybe it would not have supported mail-crypt anyway.

 

Thanks for any input.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Bug - doveadm backup out-of-memory kill/crash when no separators set

2023-07-29 Thread James Harrison 

Hi,

I hit a fun issue with doveadm when migrating from dovecot 2.2.36 
(1f10bfa63) to 2.3.19.1 (9b53102964) (CentOS 7 to Debian 12).


When running doveadm -v -D backup -R -u "user@name" tcp:localhost:1234, 
I found that the first sync would always work, but subsequent runs of 
the command would cause doveadm to reach a subfolder (Archives/2008 in 
the example below) and then silently mmap() increasing powers of 2 
before the OOM killer finally got it. Deleting the source folder caused 
it to remain stuck on the same folder, but for the deletion event.


I switched memory overcommit off to force a crash, and got a gdb backtrace:

#0  __pthread_kill_implementation (threadid=, 
signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x7786bd9f in __pthread_kill_internal (signo=6, 
threadid=) at ./nptl/pthread_kill.c:78
#2  0x7781cf32 in __GI_raise (sig=sig@entry=6) at 
../sysdeps/posix/raise.c:26

#3  0x77807472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x77b4ffae in default_fatal_finish (status=0, 
type=LOG_TYPE_PANIC) at ../lib/failures.c:465
#5  fatal_handler_real (ctx=, format=, 
args=) at ../lib/failures.c:477
#6  0x77bfa081 in default_fatal_handler (ctx=, 
format=, args=) at ../lib/failures.c:485
#7  0x77b5017c in i_panic (format=0x77c5d748 "data stack: 
Out of memory when allocating %zu bytes") at ../lib/failures.c:530
#8  0x77b4f67f in mem_block_alloc (min_size=min_size@entry=16) 
at ../lib/data-stack.c:386
#9  0x77bf8b60 in t_malloc_real (size=, 
permanent=) at ../lib/data-stack.c:492
#10 0x77c348f1 in t_strdup_until 
(start=start@entry=0x5565f440, end=end@entry=0x5565f448) at 
../lib/strfuncs.c:270
#11 0x555adc62 in convert_name_to_remote_sep 
(name=0x5565f440 "Archives/2008", tree=0x5564e188) at 
dsync/dsync-mailbox-tree.c:270
#12 dsync_mailbox_tree_build_name128_remotesep_hash 
(tree=0x5564e188) at dsync/dsync-mailbox-tree.c:315
#13 dsync_mailbox_tree_find_delete (tree=0x5564e188, 
del=0x556469f0) at dsync/dsync-mailbox-tree.c:405
#14 0x555a4195 in dsync_brain_mailbox_tree_add_delete 
(tree=0x5564e188, other_tree=0x5564f5f8, other_del=0x556469f0,
    node_r=0x7fffe350, status_r=0x7fffe348) at 
dsync/dsync-brain-mailbox-tree.c:504
#15 0x555a44cd in dsync_brain_recv_mailbox_tree_deletes 
(brain=0x5564b2d8) at dsync/dsync-brain-mailbox-tree.c:590
#16 0x555a5365 in dsync_brain_run_real 
(brain=brain@entry=0x5564b2d8, 
changed_r=changed_r@entry=0x7fffe453) at dsync/dsync-brain.c:709
#17 0x555a59f9 in dsync_brain_run (changed_r=0x7fffe453, 
brain=0x5564b2d8) at dsync/dsync-brain.c:752
#18 dsync_brain_run (changed_r=0x7fffe453, brain=0x5564b2d8) at 
dsync/dsync-brain.c:740

#19 dsync_brain_run_io (context=) at dsync/dsync-brain.c:113
#20 dsync_brain_run_io (context=0x5564b2d8) at dsync/dsync-brain.c:100
#21 0x555b23df in dsync_ibc_stream_input (ibc=0x55646720) at 
dsync/dsync-ibc-stream.c:232
#22 0x77c11cd9 in io_loop_call_io (io=0x556418f0) at 
../lib/ioloop.c:737
#23 0x77c13aa2 in io_loop_handler_run_internal 
(ioloop=ioloop@entry=0x5560bf40) at ../lib/ioloop-epoll.c:222
#24 0x77c13b50 in io_loop_handler_run 
(ioloop=ioloop@entry=0x5560bf40) at ../lib/ioloop.c:789
#25 0x77c13d10 in io_loop_run (ioloop=0x5560bf40) at 
../lib/ioloop.c:762
#26 0x5558b22e in cmd_dsync_run_remote (user=0x55637248) at 
./src/doveadm/doveadm-dsync.c:543
#27 cmd_dsync_run (_ctx=0x5561f288, user=0x55637248) at 
./src/doveadm/doveadm-dsync.c:750
#28 0x5558bd12 in doveadm_mail_next_user (ctx=0x5561f288, 
error_r=0x7fffe818) at ./src/doveadm/doveadm-mail.c:464
#29 0x5558cf45 in doveadm_mail_cmd_exec (wildcard_user=0x0, 
ctx=0x5561f288) at ./src/doveadm/doveadm-mail.c:659
#30 doveadm_cmd_ver2_to_mail_cmd_wrapper (cctx=0x7fffe970) at 
./src/doveadm/doveadm-mail.c:988
#31 0x55597622 in doveadm_cmd_run_ver2 (argc=5, 
argv=0x5560ba38, cctx=cctx@entry=0x7fffe970) at 
./src/doveadm/doveadm-cmd.c:465
#32 0x55597697 in doveadm_cmd_try_run_ver2 (cmd_name=out>, argc=, argv=, cctx=0x7fffe970)

    at ./src/doveadm/doveadm-cmd.c:363
#33 0x5557919a in main (argc=, argv=out>) at ./src/doveadm/doveadm.c:361


Neither server had a separator set in the inbox namespace. Having 
identified the convert_name_to_remote_sep function as the likely trigger 
point I set the separator to / on both ends, and doveadm backup now runs 
without issue.


Can provide config if helpful but I think probably irrelevant to this 
one - the namespace config is the default other than the separator item. 
May be worth mentioning that having the separator set can avoid this 
condition in the doveadm backup docs - this took me a while to figure out!


--
Cheers,
James Harrison

___