We have a set of couchdb databases that we use to collect user information for
various purposes. I am inheriting this configuration from a predecessor and am
relatively new to couchdb.
Whenever we attempt to compact the databases, the server crashes without any
messages either in the couchdb log or the system logs. This is running in an
AWS instance with an EBS volume.
Experiments have shown that if the instance is configured with instance storage
(ephemeral storage that disappears when the instance disappears) then this
operation works properly. But we would like to use larger volumes and have
persistence.
When the instance is configured with an external EBS volume, then we see the
server crash described above.
I have searched the web for “couchdb compaction crash no log” and not found
anything helpful.
It seems like compacting while running should not be failing at all, much less
silently, so I am looking for insights to the problem, or solutions if such
exist.
Configuration and log info is below.
Any help would be appreciated.
Thanks,
Greg
---------------------------------------------------------
CouchDB version: 1.6.1
OS: RHEL 6.6
---------------------------------------------------------
Here is a directory of the databases as the time of the crash:
cat bad.couch.dbinfo.txt
total 15400740
12 -rw-r--r--. 1 couchdb couchdb 8297 Jan 20 16:31 _users.couch
16 -rw-r--r--. 1 couchdb couchdb 12393 Jan 20 16:33 _replicator.couch
21060 -rw-r--r--. 1 couchdb couchdb 21557368 Mar 7 11:57 biometrics.couch
781136 -rw-r--r--. 1 couchdb couchdb 799875192 Mar 7 12:00 fitness.couch
954244 -rw-r--r--. 1 couchdb couchdb 977137784 Mar 7 12:05 nutrition.couch
8419624 -rw-r--r--. 1 couchdb couchdb 8621678721 Mar 7 12:06 routine.couch
390796 -rw-r--r--. 1 couchdb couchdb 400167032 Mar 7 12:06 sleep.couch
217932 -rw-r--r--. 1 couchdb couchdb 223154296 Mar 7 12:06 weight.couch
4614884 -rw-r--r--. 1 couchdb couchdb 4725629060 Mar 7 12:06 trackers.couch
4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41
fitness.couch.compact
4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41
nutrition.couch.compact
4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41
routine.couch.compact
64 -rw-r--r--. 1 couchdb couchdb 61551 Mar 7 12:41 diabetes.couch
4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41
sleep.couch.compact
12 -rw-r--r--. 1 couchdb couchdb 8300 Mar 7 12:41
tobacco_cessation.couch
4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 users.couch
4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41
weight.couch.compact
152 -rw-r--r--. 1 couchdb couchdb 151797 Mar 7 12:42
trackers.couch.compact
784 -rw-r--r--. 1 couchdb couchdb 801865 Mar 7 12:42
biometrics.couch.compact
---------------------------------------------------------
Here is the contents of the log at the time of the crash:
[Mon, 07 Mar 2016 17:25:32 GMT] [info] [<0.31.0>] Apache CouchDB has started on
http://0.0.0.0:5984/
[Mon, 07 Mar 2016 17:25:32 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:33 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:35 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:25:37 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:37 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:38 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:38 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:40 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:25:45 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:25:50 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:50 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:50 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:25:52 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:52 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:54 GMT] [info] [<0.114.0>] 10.1.1.12 - - GET
/users/_changes?feed=continuous&style=all_docs&since=0&heartbeat=10000 200
[Mon, 07 Mar 2016 17:25:54 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:25:54 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:25:55 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:26:00 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:26:00 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/_active_tasks 200
[Mon, 07 Mar 2016 17:26:01 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET
/favicon.ico 200
[Mon, 07 Mar 2016 17:26:05 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
... [numerous GET /users/ 200 messages removed for brevity] ...
[Mon, 07 Mar 2016 17:41:51 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.152.0>] 127.0.0.1 - - GET /_all_dbs
200
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1157.0>] Starting compaction for db
"biometrics"
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.151.0>] 127.0.0.1 - - POST
/biometrics/_compact 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.150.0>] 127.0.0.1 - - POST
/biometrics/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1175.0>] Starting compaction for db
"diabetes"
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.198.0>] 127.0.0.1 - - POST
/diabetes/_compact 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.197.0>] 127.0.0.1 - - POST
/diabetes/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1193.0>] Starting compaction for db
"fitness"
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.118.0>] 127.0.0.1 - - POST
/fitness/_compact 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.119.0>] 127.0.0.1 - - POST
/fitness/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1211.0>] Starting compaction for db
"nutrition"
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.120.0>] 127.0.0.1 - - POST
/nutrition/_compact 202
[Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.121.0>] 127.0.0.1 - - POST
/nutrition/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1229.0>] Starting compaction for db
"routine"
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.122.0>] 127.0.0.1 - - POST
/routine/_compact 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.115.0>] 127.0.0.1 - - POST
/routine/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1175.0>] Compaction for db
"diabetes" completed.
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1254.0>] Starting compaction for db
"sleep"
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.116.0>] 127.0.0.1 - - POST
/sleep/_compact 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.117.0>] 127.0.0.1 - - POST
/sleep/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1272.0>] Starting compaction for db
"tobacco_cessation"
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.184.0>] 127.0.0.1 - - POST
/tobacco_cessation/_compact 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.183.0>] 127.0.0.1 - - POST
/tobacco_cessation/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1290.0>] Starting compaction for db
"trackers"
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.182.0>] 127.0.0.1 - - POST
/trackers/_compact 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1272.0>] Compaction for db
"tobacco_cessation" completed.
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1151.0>] 127.0.0.1 - - POST
/trackers/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.162.0>] Starting compaction for db
"users"
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1152.0>] 127.0.0.1 - - POST
/users/_compact 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1168.0>] 127.0.0.1 - - POST
/users/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.162.0>] Compaction for db "users"
completed.
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1329.0>] Starting compaction for db
"weight"
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1170.0>] 127.0.0.1 - - POST
/weight/_compact 202
[Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1187.0>] 127.0.0.1 - - POST
/weight/_view_cleanup 202
[Mon, 07 Mar 2016 17:41:56 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:42:01 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:42:06 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
[Mon, 07 Mar 2016 17:42:11 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ 200
--------------------------------------------------
