sergey-safarov commented on issue #1073: CouchDB 3-node cluster (Windows) - multiple erlang errors URL: https://github.com/apache/couchdb/issues/1073#issuecomment-358870957 I installed CouchDB from [couchdb-2.1.1-2.el7.centos.x86_64.rpm](http://apache.bintray.com/couchdb-rpm/el7/x86_64) ``` [[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep rexi_server | head [error] 2018-01-18T18:57:41.932949Z [email protected] <0.12764.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.933954Z [email protected] <0.12760.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.933996Z [email protected] <0.12765.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934089Z [email protected] <0.12762.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934061Z [email protected] <0.12763.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934294Z [email protected] <0.12766.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934416Z [email protected] <0.12759.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.935568Z [email protected] <0.12761.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}] [error] 2018-01-18T18:58:25.372933Z [email protected] <0.23503.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:58:25.373010Z [email protected] <0.23509.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}] ``` **Other types of errors** File operation error ``` [[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "File operation error" | head [error] 2018-01-18T18:56:02.373572Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.377095Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.378314Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.381081Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.381934Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.384896Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.385662Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.388574Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.389582Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server. [error] 2018-01-18T18:56:02.392229Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server. ``` died with reason ``` [[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "died with reason" | head [info] 2018-01-18T18:56:11.273221Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,{{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},[{couch_db,start_link,3,[{file,"src/couch_db.erl"},{line,136}]},{couch_server,'-open_async/5-fun-0-',5,[{file,"src/couch_server.erl"},{line,310}]}]} [info] 2018-01-18T18:56:11.273522Z [email protected] <0.207.0> -------- db shards/a0000000-bfffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,{{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},[{couch_db,start_link,3,[{file,"src/couch_db.erl"},{line,136}]},{couch_server,'-open_async/5-fun-0-',5,[{file,"src/couch_server.erl"},{line,310}]}]} [info] 2018-01-18T18:56:11.273854Z [email protected] <0.207.0> -------- db shards/60000000-7fffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,{{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},[{couch_db,start_link,3,[{file,"src/couch_db.erl"},{line,136}]},{couch_server,'-open_async/5-fun-0-',5,[{file,"src/couch_server.erl"},{line,310}]}]} [info] 2018-01-18T18:56:11.532489Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} [info] 2018-01-18T18:57:05.184325Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/account/42/f7/db16e5f11dc86e01cbaa653cf6e3.1516301823 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,handle_info,2,[{file,"src/couch_db_updater.erl"},{line,338}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} [info] 2018-01-18T18:57:05.184600Z [email protected] <0.207.0> -------- db shards/e0000000-ffffffff/account/42/f7/db16e5f11dc86e01cbaa653cf6e3.1516301823 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,handle_info,2,[{file,"src/couch_db_updater.erl"},{line,338}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} [info] 2018-01-18T18:57:05.184843Z [email protected] <0.207.0> -------- db shards/60000000-7fffffff/account/42/f7/db16e5f11dc86e01cbaa653cf6e3.1516301823 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,handle_info,2,[{file,"src/couch_db_updater.erl"},{line,338}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} [info] 2018-01-18T19:01:13.389304Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/system_media.1516301568 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,398}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} [info] 2018-01-18T19:01:13.390153Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/tasks.1516301582 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,398}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} [info] 2018-01-18T19:01:13.391205Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/brand_provisioner.1516301513 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,398}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]} ``` neighbors exited with reason ``` [[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "neighbors exited with reason" | head [error] 2018-01-18T18:56:11.273943Z [email protected] <0.1669.2> -------- CRASH REPORT Process (<0.1669.2>) with 1 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:init/1(line:67) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.1655.2>], messages: [], links: [<0.1655.2>], dictionary: [{io_priority,{db_update,<<"shards/80000000-9fffffff/account/39/6e/a348ae...">>}},...], trap_exit: false, status: running, heap_size: 1598, stack_size: 27, reductions: 2610 [error] 2018-01-18T18:56:11.274595Z [email protected] <0.1671.2> -------- CRASH REPORT Process (<0.1671.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:init/1(line:67) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.1659.2>], messages: [], links: [<0.1659.2>], dictionary: [{io_priority,{db_update,<<"shards/60000000-7fffffff/account/39/6e/a348ae...">>}},...], trap_exit: false, status: running, heap_size: 1598, stack_size: 27, reductions: 2610 [error] 2018-01-18T18:56:11.532530Z [email protected] <0.1701.2> -------- CRASH REPORT Process (<0.1701.2>) with 2 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:init/1(line:67) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.1697.2>], messages: [], links: [<0.1697.2>], dictionary: [{io_priority,{db_update,<<"shards/80000000-9fffffff/account/39/6e/a348ae...">>}},...], trap_exit: false, status: running, heap_size: 1598, stack_size: 27, reductions: 1278 [error] 2018-01-18T18:57:00.102237Z [email protected] <0.4660.2> -------- CRASH REPORT Process (<0.4660.2>) with 0 neighbors exited with reason: {error,accept_failed} at mochiweb_acceptor:init/4(line:55) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {mochiweb_acceptor,init,['Argument__1','Argument__2',...]}, ancestors: [chttpd,chttpd_sup,<0.328.0>], messages: [], links: [<0.336.0>], dictionary: [], trap_exit: false, status: running, heap_size: 376, stack_size: 27, reductions: 267 [error] 2018-01-18T18:57:05.184346Z [email protected] <0.5458.2> -------- CRASH REPORT Process (<0.5458.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:handle_info/2(line:338) <= gen_server:try_dispatch/4(line:615) <= gen_server:handle_msg/5(line:681) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5444.2>], messages: [{'DOWN',#Ref<0.0.5.32558>,process,<0.5446.2>,normal}], links: [<0.207.0>], dictionary: [{io_priority,{db_update,<<"shards/80000000-9fffffff/account/42/f7/d...">>}},...], trap_exit: false, status: running, heap_size: 4185, stack_size: 27, reductions: 6779 [error] 2018-01-18T18:57:05.184669Z [email protected] <0.5455.2> -------- CRASH REPORT Process (<0.5455.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:handle_info/2(line:338) <= gen_server:try_dispatch/4(line:615) <= gen_server:handle_msg/5(line:681) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5439.2>], messages: [{'DOWN',#Ref<0.0.4.60859>,process,<0.5440.2>,normal}], links: [<0.207.0>], dictionary: [{io_priority,{db_update,<<"shards/e0000000-ffffffff/account/42/f7/d...">>}},...], trap_exit: false, status: running, heap_size: 4185, stack_size: 27, reductions: 6544 [error] 2018-01-18T18:57:05.184976Z [email protected] <0.5459.2> -------- CRASH REPORT Process (<0.5459.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:handle_info/2(line:338) <= gen_server:try_dispatch/4(line:615) <= gen_server:handle_msg/5(line:681) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5447.2>], messages: [{'DOWN',#Ref<0.0.2.68255>,process,<0.5448.2>,normal}], links: [<0.207.0>], dictionary: [{io_priority,{db_update,<<"shards/60000000-7fffffff/account/42/f7/d...">>}},...], trap_exit: false, status: running, heap_size: 4185, stack_size: 27, reductions: 7479 [error] 2018-01-18T19:01:03.968059Z [email protected] <0.27902.3> -------- CRASH REPORT Process (<0.27902.3>) with 2 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:398) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.27901.3>,<0.27900.3>], messages: [], links: [#Port<0.20470>,<0.27901.3>], dictionary: [{read_timestamp,{1516,302063,967577}}], trap_exit: false, status: running, heap_size: 610, stack_size: 27, reductions: 768 [error] 2018-01-18T19:01:03.981875Z [email protected] <0.214.0> -------- CRASH REPORT Process couch_index_server (<0.214.0>) with 1 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:398) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_index_server,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.201.0>], messages: [], links: [<0.216.0>,<0.210.0>], dictionary: [], trap_exit: true, status: running, heap_size: 4185, stack_size: 27, reductions: 584992 [error] 2018-01-18T19:01:05.849952Z [email protected] <0.32628.3> -------- CRASH REPORT Process (<0.32628.3>) with 2 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:398) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.32627.3>,<0.32626.3>], messages: [], links: [#Port<0.21996>,<0.32627.3>], dictionary: [{read_timestamp,{1516,302065,849433}}], trap_exit: false, status: running, heap_size: 610, stack_size: 27, reductions: 768 ``` mfa: fabric_rpc:all_docs ``` [[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "mfa: fabric_rpc:all_docs" | head [error] 2018-01-18T18:57:41.932949Z [email protected] <0.12764.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.933954Z [email protected] <0.12760.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.933996Z [email protected] <0.12765.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934089Z [email protected] <0.12762.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934061Z [email protected] <0.12763.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934294Z [email protected] <0.12766.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.934416Z [email protected] <0.12759.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:57:41.935568Z [email protected] <0.12761.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}] [error] 2018-01-18T18:58:25.372933Z [email protected] <0.23503.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}] [error] 2018-01-18T18:58:25.373010Z [email protected] <0.23509.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}] ``` too many open files ``` [[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "too many open files" | head [error] 2018-01-18T19:01:17.795630Z [email protected] <0.18880.4> -------- Could not open file ./data/shards/a0000000-bfffffff/system_media.1516301568.couch: too many open files [error] 2018-01-18T19:01:17.817397Z [email protected] <0.18939.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/69e3a57e087bac7b9667b7602970e56c.view: too many open files [error] 2018-01-18T19:01:17.817488Z [email protected] <0.18939.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/69e3a57e087bac7b9667b7602970e56c.view': too many open files [error] 2018-01-18T19:01:17.819229Z [email protected] <0.18945.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/665c8a92ba7daaf86a989aee308fbbe5.view: too many open files [error] 2018-01-18T19:01:17.819320Z [email protected] <0.18945.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/665c8a92ba7daaf86a989aee308fbbe5.view': too many open files [error] 2018-01-18T19:01:17.820790Z [email protected] <0.18951.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/3f703b0b07ced5a4c9804a63d7a15af9.view: too many open files [error] 2018-01-18T19:01:17.820882Z [email protected] <0.18951.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/3f703b0b07ced5a4c9804a63d7a15af9.view': too many open files [error] 2018-01-18T19:01:17.822353Z [email protected] <0.18957.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/ae24bf2461ecaaa8d6368a29badc5eb1.view: too many open files [error] 2018-01-18T19:01:17.822429Z [email protected] <0.18957.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/ae24bf2461ecaaa8d6368a29badc5eb1.view': too many open files [error] 2018-01-18T19:01:17.823900Z [email protected] <0.18963.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/f836e93763ffb225e3b1079db74acbbe.view: too many open files ``` I will configure to allow daemon open 999999 files and make test again
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
