I just had a strange problem with an installation of couchdb. After restarting the couchdb service, it seems it can't access anymore to some of the dbs that were there before. The files of those dbs seems to be still present in the shards folder, but after the restart, it seems the system can't access them. Just before the restart, the dbs were there and working regularly, there was no delete operation. The couchdb configuration, at least to my knowledge, did not change since installation, and by my check seems to be the standard one. Couchdb is version 2.3.1.
I've also tried on a different machine copying one of our previous timed-backups (we do a daily snapshot of the whole couch db folders), but once restored seems to have the exact same problem: the couchdb service comes up, but the "documents" and "main" dbs are missing. I've no idea were to start debugging this, and if there is a way of recovering those dbs from the backup I've right now. Anyway, the logs during the restart are the following: [info] 2021-03-26T09:06:57.909241Z [email protected] <0.56.0> -------- alarm_handler: {set,{system_memory_high_watermark,[]}} [error] 2021-03-26T09:10:26.491449Z [email protected] <0.64.0> -------- gen_server memsup terminated with reason: {port_died,normal} last msg: {'EXIT',<0.65.0>,{port_died,normal}} state: [{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",7946051584},{"Total",8171896832}]}},{items,{"Worst Memory User",[{"Pid",<0.218.0>},{"Memory",7687816}]}}] extra: [] [info] 2021-03-26T09:10:32.894224Z [email protected] <0.9.0> -------- Application couch_log started on node '[email protected]' [info] 2021-03-26T09:10:32.899154Z [email protected] <0.9.0> -------- Application folsom started on node '[email protected]' [info] 2021-03-26T09:10:33.012320Z [email protected] <0.9.0> -------- Application couch_stats started on node '[email protected]' [info] 2021-03-26T09:10:33.012544Z [email protected] <0.9.0> -------- Application khash started on node '[email protected]' [info] 2021-03-26T09:10:33.026786Z [email protected] <0.9.0> -------- Application couch_event started on node '[email protected]' [info] 2021-03-26T09:10:33.027019Z [email protected] <0.9.0> -------- Application hyper started on node '[email protected]' [info] 2021-03-26T09:10:33.038401Z [email protected] <0.9.0> -------- Application ibrowse started on node '[email protected]' [info] 2021-03-26T09:10:33.048224Z [email protected] <0.9.0> -------- Application ioq started on node '[email protected]' [info] 2021-03-26T09:10:33.048468Z [email protected] <0.9.0> -------- Application mochiweb started on node '[email protected]' [info] 2021-03-26T09:10:33.064719Z [email protected] <0.212.0> -------- Apache CouchDB 2.3.1 is starting. [info] 2021-03-26T09:10:33.064980Z [email protected] <0.213.0> -------- Starting couch_sup [notice] 2021-03-26T09:10:33.082657Z [email protected] <0.96.0> -------- config: [features] pluggable-storage-engines set to true for reason nil [notice] 2021-03-26T09:10:33.097461Z [email protected] <0.96.0> -------- config: [admins] admin set to XXX for reason nil [info] 2021-03-26T09:10:33.258955Z [email protected] <0.212.0> -------- Apache CouchDB has started. Time to relax. [info] 2021-03-26T09:10:33.259081Z [email protected] <0.212.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/ [info] 2021-03-26T09:10:33.259276Z [email protected] <0.9.0> -------- Application couch started on node '[email protected]' [info] 2021-03-26T09:10:33.259725Z [email protected] <0.9.0> -------- Application ets_lru started on node '[email protected]' [notice] 2021-03-26T09:10:33.290376Z [email protected] <0.272.0> -------- rexi_server : started servers [notice] 2021-03-26T09:10:33.295464Z [email protected] <0.279.0> -------- rexi_buffer : started servers [info] 2021-03-26T09:10:33.295971Z [email protected] <0.9.0> -------- Application rexi started on node '[email protected]' [info] 2021-03-26T09:10:33.342338Z [email protected] <0.9.0> -------- Application mem3 started on node '[email protected]' [info] 2021-03-26T09:10:33.342560Z [email protected] <0.9.0> -------- Application fabric started on node '[email protected]' [info] 2021-03-26T09:10:33.403837Z [email protected] <0.9.0> -------- Application chttpd started on node '[email protected]' [info] 2021-03-26T09:10:33.436538Z [email protected] <0.9.0> -------- Application couch_index started on node '[email protected]' [info] 2021-03-26T09:10:33.437227Z [email protected] <0.9.0> -------- Application couch_mrview started on node '[email protected]' [info] 2021-03-26T09:10:33.437531Z [email protected] <0.9.0> -------- Application couch_plugins started on node '[email protected]' [notice] 2021-03-26T09:10:33.559220Z [email protected] <0.96.0> -------- config: [features] scheduler set to true for reason nil [info] 2021-03-26T09:10:33.620740Z [email protected] <0.9.0> -------- Application couch_replicator started on node '[email protected]' [info] 2021-03-26T09:10:33.649361Z [email protected] <0.9.0> -------- Application couch_peruser started on node '[email protected]' [info] 2021-03-26T09:10:33.703726Z [email protected] <0.9.0> -------- Application ddoc_cache started on node '[email protected]' [info] 2021-03-26T09:10:33.734466Z [email protected] <0.9.0> -------- Application global_changes started on node '[email protected]' [info] 2021-03-26T09:10:33.734529Z [email protected] <0.9.0> -------- Application jiffy started on node '[email protected]' [info] 2021-03-26T09:10:33.755574Z [email protected] <0.9.0> -------- Application mango started on node '[email protected]' [info] 2021-03-26T09:10:33.777359Z [email protected] <0.9.0> -------- Application setup started on node '[email protected]' [info] 2021-03-26T09:10:33.777609Z [email protected] <0.9.0> -------- Application snappy started on node '[email protected]' [notice] 2021-03-26T09:10:38.469290Z [email protected] <0.353.0> -------- couch_replicator_clustering : cluster stable [notice] 2021-03-26T09:10:38.478642Z [email protected] <0.405.0> -------- Started replicator db changes listener <0.511.0> [error] 2021-03-26T09:10:45.348964Z [email protected] emulator -------- Error in process <0.670.0> on node '[email protected]' with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"documents",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2021-03-26T09:10:45.349035Z [email protected] <0.260.0> -------- Could not get design docs for <<"shards/20000000-3fffffff/documents.1594915751">> error:{database_does_not_exist,[{mem3_shards,load_shards_from_db,"documents",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [notice] 2021-03-26T09:10:48.291445Z [email protected] <0.279.0> -------- rexi_buffer : cluster stable [notice] 2021-03-26T09:10:48.291610Z [email protected] <0.272.0> -------- rexi_server : cluster stable [error] 2021-03-26T09:11:06.416502Z [email protected] <0.260.0> -------- Could not get design docs for <<"shards/20000000-3fffffff/main.1594915751">> error:{database_does_not_exist,[{mem3_shards,load_shards_from_db,"main",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2021-03-26T09:11:06.416806Z [email protected] emulator -------- Error in process <0.1116.0> on node '[email protected]' with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"main",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]}
