Re: doveadm sync out of memory
On Feb 17, 2015, at 10:51 AM, FUSTE Emmanuel emmanuel.fu...@thalesgroup.com wrote: Le 16/02/2015 20:40, Casey Stone a écrit : On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel emmanuel.fu...@thalesgroup.com wrote: Le 13/02/2015 16:19, Casey Stone a écrit : On Feb 5, 2015, at 10:39 PM, Casey Stone tcst...@caseystone.com wrote: Hello: I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined. Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command: doveadm sync -R tcps:mainserver.example.com:12345 Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user. The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug? Thanks for your help. No repsonses :-( Here is what it looks like when it crashes with an out of memory error: (start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1 many, many more brain messages (end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] - /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] - /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] - /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] - /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] - /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] - /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] - dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] - dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] - dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] - dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] - dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] - dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] - dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] - /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] - /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] - /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] - dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] - dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot:
Re: doveadm sync out of memory
Le 16/02/2015 20:40, Casey Stone a écrit : On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel emmanuel.fu...@thalesgroup.com wrote: Le 13/02/2015 16:19, Casey Stone a écrit : On Feb 5, 2015, at 10:39 PM, Casey Stone tcst...@caseystone.com wrote: Hello: I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined. Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command: doveadm sync -R tcps:mainserver.example.com:12345 Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user. The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug? Thanks for your help. No repsonses :-( Here is what it looks like when it crashes with an out of memory error: (start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1 many, many more brain messages (end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] - /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] - /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] - /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] - /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] - /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] - /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] - /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] - dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] - dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] - dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] - dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] - dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] - dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] - dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] - /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] - /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] - /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] - dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] - dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm
Re: doveadm sync out of memory
On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel emmanuel.fu...@thalesgroup.com
wrote:
Le 13/02/2015 16:19, Casey Stone a écrit :
On Feb 5, 2015, at 10:39 PM, Casey Stone tcst...@caseystone.com wrote:
Hello:
I've been looking forward to getting my mail server up to Dovecot 2.2+ to
be able to use the sync mechanism. I run my own mail server just for
myself, with a few different accounts, and want to keep a master and backup
server in sync.
I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot
2.2.9 (and Postfix). My setup is to use system users (userdb passwd /
passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n
if it seems necessary. I have not enabled any plugins (do I need the
replicator plugin active?) I have in my conf a doveadm_password defined.
Anyway, after setting up an ssl listener on the main machine and after
considerable struggles with SSL, I was able to run doveadm sync from the
backup server successfully for a small mailbox (around 78 MB) with this
command:
doveadm sync -R tcps:mainserver.example.com:12345
Since I run this command as the system user on the backup server (same
system users as main server) it 'just works' for the correct single user
with no further options required. My plan is to run a daily cron job to
sync once daily for each user.
The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server
on the remote (master) machine throws fatal error 83 Out of Memory. I
already raised vsz_limit to 512 MB. Problems probably arise with mailboxes
around 200 MB though I haven't tested specifically. So my question is, is
this expected and I will need to give my VM much more memory to be able to
use dovecot sync, or do I have something set wrong, or is it a bug?
Thanks for your help.
No repsonses :-(
Here is what it looks like when it crashes with an out of memory error:
(start of the run)
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug:
Effective uid=1002, gid=1002, home=/home/tcstone
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug:
Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug:
maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace :
Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out
state=send_mailbox_tree changed=1
many, many more brain messages
(end of the run)
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal:
pool_system_realloc(536870912): Out of memory
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -
/usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -
/usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -
/usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -
/usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -
/usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -
/usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -
/usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -
/usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05)
[0x7f9d1f6a0c05] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21)
[0x7f9d1f6a1881] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d)
[0x7f9d1f6a3a4d] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69)
[0x7f9d1f6a3d69] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d)
[0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf)
[0x7f9d20583e5f] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a)
[0x7f9d20583e8a] - dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -
dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -
dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -
dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -
dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -
dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -
dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7]
- /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -
dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -
dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb]
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master:
service(doveadm): child 13232 returned error 83 (Out of memory (service
doveadm { vsz_limit=512 MB }, you may
Re: doveadm sync out of memory
On Feb 5, 2015, at 10:39 PM, Casey Stone tcst...@caseystone.com wrote:
Hello:
I've been looking forward to getting my mail server up to Dovecot 2.2+ to be
able to use the sync mechanism. I run my own mail server just for myself,
with a few different accounts, and want to keep a master and backup server in
sync.
I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9
(and Postfix). My setup is to use system users (userdb passwd / passdb pam)
with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems
necessary. I have not enabled any plugins (do I need the replicator plugin
active?) I have in my conf a doveadm_password defined.
Anyway, after setting up an ssl listener on the main machine and after
considerable struggles with SSL, I was able to run doveadm sync from the
backup server successfully for a small mailbox (around 78 MB) with this
command:
doveadm sync -R tcps:mainserver.example.com:12345
Since I run this command as the system user on the backup server (same system
users as main server) it 'just works' for the correct single user with no
further options required. My plan is to run a daily cron job to sync once
daily for each user.
The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on
the remote (master) machine throws fatal error 83 Out of Memory. I already
raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200
MB though I haven't tested specifically. So my question is, is this expected
and I will need to give my VM much more memory to be able to use dovecot
sync, or do I have something set wrong, or is it a bug?
Thanks for your help.
No repsonses :-(
Here is what it looks like when it crashes with an out of memory error:
(start of the run)
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective
uid=1002, gid=1002, home=/home/tcstone
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace
inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++:
root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace :
Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out
state=send_mailbox_tree changed=1
many, many more brain messages
(end of the run)
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal:
pool_system_realloc(536870912): Out of memory
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -
/usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -
/usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -
/usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -
/usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -
/usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -
/usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -
/usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -
/usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05]
-
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21)
[0x7f9d1f6a1881] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d]
- /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69)
[0x7f9d1f6a3d69] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d)
[0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf)
[0x7f9d20583e5f] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a)
[0x7f9d20583e8a] - dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -
dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -
dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -
dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -
dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -
dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -
dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -
dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -
dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb]
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master:
service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm
{ vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1
environment to get core dump)
I haven't tested whether it is simply the effect of having DEBUG active that
kills it... A run requiring moving
Re: doveadm sync out of memory
Le 13/02/2015 16:19, Casey Stone a écrit :
On Feb 5, 2015, at 10:39 PM, Casey Stone tcst...@caseystone.com wrote:
Hello:
I've been looking forward to getting my mail server up to Dovecot 2.2+ to be
able to use the sync mechanism. I run my own mail server just for myself,
with a few different accounts, and want to keep a master and backup server
in sync.
I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot
2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb
pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it
seems necessary. I have not enabled any plugins (do I need the replicator
plugin active?) I have in my conf a doveadm_password defined.
Anyway, after setting up an ssl listener on the main machine and after
considerable struggles with SSL, I was able to run doveadm sync from the
backup server successfully for a small mailbox (around 78 MB) with this
command:
doveadm sync -R tcps:mainserver.example.com:12345
Since I run this command as the system user on the backup server (same
system users as main server) it 'just works' for the correct single user
with no further options required. My plan is to run a daily cron job to sync
once daily for each user.
The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server
on the remote (master) machine throws fatal error 83 Out of Memory. I
already raised vsz_limit to 512 MB. Problems probably arise with mailboxes
around 200 MB though I haven't tested specifically. So my question is, is
this expected and I will need to give my VM much more memory to be able to
use dovecot sync, or do I have something set wrong, or is it a bug?
Thanks for your help.
No repsonses :-(
Here is what it looks like when it crashes with an out of memory error:
(start of the run)
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective
uid=1002, gid=1002, home=/home/tcstone
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace
inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug:
maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace :
Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out
state=send_mailbox_tree changed=1
many, many more brain messages
(end of the run)
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal:
pool_system_realloc(536870912): Out of memory
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -
/usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -
/usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -
/usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -
/usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -
/usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -
/usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -
/usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -
/usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05]
-
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21)
[0x7f9d1f6a1881] -
/usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d]
- /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69)
[0x7f9d1f6a3d69] - /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d)
[0x7f9d20583d7d] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf)
[0x7f9d20583e5f] - /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a)
[0x7f9d20583e8a] - dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -
dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -
dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -
dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -
dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -
dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -
dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7]
- /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -
dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -
dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb]
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master:
service(doveadm): child 13232 returned error 83 (Out of memory (service
doveadm { vsz_limit=512 MB }, you may need to increase it) - set
DEBUG_OUTOFMEM=1 environment to get core dump)
I haven't tested whether
doveadm sync out of memory
Hello: I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined. Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command: doveadm sync -R tcps:mainserver.example.com:12345 Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user. The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug? Thanks for your help.
