[Dovecot] Possible mdbox corruption

[Daniel L. Miller]

I thought I had cleared out the corruption I had before - perhaps I was 
mistaken.  What steps should I take to help locate these issues?  
Currently using 2.1rc1.  I see the following errors in my logs, 
including out of memory and message size issues (at 15:30):

Jan  4 05:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7f6e17cbfd0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7f6e17cbfd56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f6e17c98d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7f6e17cd0310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7f6e17cbc965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7f6e17cbd0ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7f6e164b7292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7f6e164b7a97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7f6e166c4abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7f6e166c5561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7f6e166ca630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7510]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f6e17ccc0f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7f6e17ccd17f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7f6e17ccc098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f6e17cb9123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f6e1791cd8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7510]() [0x401d19]
Jan  4 05:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 06:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 06:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7f0ec1a57d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7f0ec1a57d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f0ec1a30d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7f0ec1a68310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7f0ec1a54965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7f0ec1a550ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7f0ec024f292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7f0ec024fa97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7f0ec045cabc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7f0ec045d561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7f0ec0462630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7510]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f0ec1a640f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7f0ec1a6517f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7f0ec1a64098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f0ec1a51123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f0ec16b4d8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7510]() [0x401d19]
Jan  4 06:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 06:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 11941 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 07:17:18 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 07:17:18 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7faed4e56d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7faed4e56d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7faed4e2fd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7faed4e67310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7faed4e53965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7faed4e540ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7faed364e292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7faed364ea97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7faed385babc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7faed385c561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7faed3861630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7510]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7faed4e630f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7faed4e6417f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7faed4e63098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7faed4e50123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7faed4ab3d8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7510]() [0x401d19]
Jan  4 07:17:18 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 07:17:18 bubba dovecot: master: Error: service(indexer-worker): 
child 13299 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 08:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 08:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7ffd84382d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7ffd84382d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7ffd8435bd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7ffd84393310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7ffd8437f965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7ffd843800ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7ffd82b7a292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7ffd82b7aa97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7ffd82d87abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7ffd82d88561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7ffd82d8d630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7510]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7ffd8438f0f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7ffd8439017f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7ffd8438f098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7ffd8437c123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7ffd83fdfd8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7510]() [0x401d19]
Jan  4 08:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 08:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 14413 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 09:17:19 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 09:17:19 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7fb701bf5d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7fb701bf5d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7fb701bced08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7fb701c06310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7fb701bf2965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7fb701bf30ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7fb7003ed292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7fb7003eda97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7fb7005faabc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7fb7005fb561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7fb700600630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7510]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7fb701c020f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7fb701c0317f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7fb701c02098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7fb701bef123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7fb701852d8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7510]() [0x401d19]
Jan  4 09:17:19 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 09:17:19 bubba dovecot: master: Error: service(indexer-worker): 
child 15486 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 10:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 10:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7f8dc590ed0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7f8dc590ed56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f8dc58e7d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7f8dc591f310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7f8dc590b965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7f8dc590c0ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7f8dc4106292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7f8dc4106a97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7f8dc4313abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7f8dc4314561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7f8dc4319630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7510]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f8dc591b0f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7f8dc591c17f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7f8dc591b098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f8dc5908123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7510](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f8dc556bd8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7510]() [0x401d19]
Jan  4 10:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 10:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 16472 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 11:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 11:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7ff619c1dd0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7ff619c1dd56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7ff619bf6d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7ff619c2e310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7ff619c1a965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7ff619c1b0ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7ff618415292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7ff618415a97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7ff618622abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7ff618623561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7ff618628630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7514]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7514]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7ff619c2a0f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7ff619c2b17f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7ff619c2a098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7ff619c17123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7514](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7ff61987ad8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7514]() [0x401d19]
Jan  4 11:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 11:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 17522 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 12:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 12:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7ffd988c1d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7ffd988c1d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7ffd9889ad08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7ffd988d2310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7ffd988be965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7ffd988bf0ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7ffd970b9292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7ffd970b9a97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7ffd972c6abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7ffd972c7561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7ffd972cc630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7515]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7515]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7ffd988ce0f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7ffd988cf17f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7ffd988ce098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7ffd988bb123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7515](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7ffd9851ed8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7515]() [0x401d19]
Jan  4 12:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 12:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 18498 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 13:17:16 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 13:17:16 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7f51b0163d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7f51b0163d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f51b013cd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7f51b0174310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7f51b0160965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7f51b01610ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7f51ae95b292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7f51ae95ba97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7f51aeb68abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7f51aeb69561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7f51aeb6e630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7515]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7515]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f51b01700f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7f51b017117f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7f51b0170098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f51b015d123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7515](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f51afdc0d8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7515]() [0x401d19]
Jan  4 13:17:16 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 13:17:16 bubba dovecot: master: Error: service(indexer-worker): 
child 19550 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 14:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 14:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7f423b546d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7f423b546d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f423b51fd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7f423b557310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7f423b543965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7f423b5440ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7f4239d3e292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7f4239d3ea97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7f4239f4babc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7f4239f4c561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7f4239f51630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7516]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7516]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f423b5530f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7f423b55417f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7f423b553098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f423b540123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7516](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f423b1a3d8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7516]() [0x401d19]
Jan  4 14:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 20638 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 14:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 15:17:17 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 15:17:17 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7f3ab5e51d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7f3ab5e51d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7f3ab5e2ad08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7f3ab5e62310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7f3ab5e4e965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7f3ab5e4f0ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7f3ab4649292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7f3ab4649a97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7f3ab4856abc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7f3ab4857561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7f3ab485c630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7517]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7517]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7f3ab5e5e0f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7f3ab5e5f17f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7f3ab5e5e098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f3ab5e4b123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7517](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f3ab5aaed8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7517]() [0x401d19]
Jan  4 15:17:17 bubba dovecot: master: Error: service(indexer-worker): 
child 21821 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 15:17:17 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com
Jan  4 15:30:48 bubba dovecot: imap(us...@domain.com): Error: Cached 
message size smaller than expected (822 < 1493)
Jan  4 15:30:48 bubba dovecot: imap(us...@domain.com): Error: Corrupted 
index cache file 
/var/mail/amfes.com/lmiller/mdbox/mailboxes/Sent/dbox-Mails/dovecot.index.cache: 
Broken physical size for mail UID 1786
Jan  4 15:30:48 bubba dovecot: imap(us...@domain.com): Error: 
read(/var/mail/amfes.com/lmiller/mdbox/storage/m.208) failed: 
Input/output error (FETCH for mailbox Sent UID 1786)
Jan  4 16:17:20 bubba dovecot: indexer-worker(us...@domain.com): Fatal: 
pool_system_realloc(134217728): Out of memory
Jan  4 16:17:20 bubba dovecot: indexer-worker(us...@domain.com): Error: 
Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) 
[0x7ffc91276d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) 
[0x7ffc91276d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7ffc9124fd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) 
[0x7ffc91287310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) 
[0x7ffc91273965] -> 
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) 
[0x7ffc912740ec] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) 
[0x7ffc8fa6e292] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) 
[0x7ffc8fa6ea97] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) 
[0x7ffc8fc7babc] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) 
[0x7ffc8fc7c561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) 
[0x7ffc8fc81630] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7517]() [0x40245f] -> dovecot/indexer-worker [us...@domain.com Sent 
- 5500/7517]() [0x4027dd] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) 
[0x7ffc912830f6] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) 
[0x7ffc9128417f] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) 
[0x7ffc91283098] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7ffc91270123] -> dovecot/indexer-worker [us...@domain.com Sent - 
5500/7517](main+0x109) [0x401f29] -> 
/lib/libc.so.6(__libc_start_main+0xfe) [0x7ffc90ed3d8e] -> 
dovecot/indexer-worker [us...@domain.com Sent - 5500/7517]() [0x401d19]
Jan  4 16:17:20 bubba dovecot: master: Error: service(indexer-worker): 
child 22927 returned error 83 (Out of memory (vsz_limit=256 MB, you may 
need to increase it))
Jan  4 16:17:20 bubba dovecot: indexer: Error: Indexer worker 
disconnected, discarding 1 requests for us...@domain.com

--
Daniel