idea-christian opened a new issue, #5544: URL: https://github.com/apache/couchdb/issues/5544
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ ) ## Description Our CouchDB server continues to crash every few hours, the log is more or less always the same as shown below. I do understand that we run into a system limit (and the system usage is very high) but I want to understand what exactly is the issue. I did already adjust the `[query_server_config]` os_process_limit (2000) and os_process_soft_limit (200) but I guess our server is simply not getting the load done in time? ``` [error] 2025-05-20T05:39:47.225573Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.225613Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.225741Z [email protected] emulator -------- Error in process <0.74769704.0> on node '[email protected]' with exit value: {system_limit,[{erlang,spawn_link,[erlang,apply,[#Fun<couch_mrview_updater.0.48133257>,[]]],[{error_info,#{module => erl_erts_errors}}]},{erlang,spawn_link,1,[]},{couch_mrview_updater,start_update,4,[{file,"src/couch_mrview_updater.erl"},{line,67}]},{couch_index_updater,'-update/3-fun-4-',8,[{file,"src/couch_index_updater.erl"},{line,173}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,559}]}]} [error] 2025-05-20T05:39:47.225809Z [email protected] emulator -------- Error in process <0.74769704.0> on node '[email protected]' with exit value: {system_limit,[{erlang,spawn_link,[erlang,apply,[#Fun<couch_mrview_updater.0.48133257>,[]]],[{error_info,#{module => erl_erts_errors}}]},{erlang,spawn_link,1,[]},{couch_mrview_updater,start_update,4,[{file,"src/couch_mrview_updater.erl"},{line,67}]},{couch_index_updater,'-update/3-fun-4-',8,[{file,"src/couch_index_updater.erl"},{line,173}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,559}]}]} [error] 2025-05-20T05:39:47.229093Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.229115Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.229125Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.229134Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.229142Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.229161Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.229305Z [email protected] emulator -------- Error in process <0.73767013.0> on node '[email protected]' with exit value: {system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.73767013.0>,[],gen,init_it,[gen_server,<0.73767013.0>,<0.73767013.0>,couch_work_queue,[{max_size,100000},{max_items,500}],[]]],[link,monitor]],[{error_info,#{module => erl_erts_errors}}]},{proc_lib,spawn_opt,4,[{file,"proc_lib.erl"},{line,192}]},{proc_lib,start_link,5,[{file,"proc_lib.erl"},{line,358}]},{couch_mrview_updater,start_update,4,[{file,"src/couch_mrview_updater.erl"},{line,26}]},{couch_index_updater,'-update/3-fun-4-',8,[{file,"src/couch_index_updater.erl"},{line,173}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,559}]}]} [error] 2025-05-20T05:39:47.229453Z [email protected] emulator -------- Error in process <0.73767013.0> on node '[email protected]' with exit value: {system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.73767013.0>,[],gen,init_it,[gen_server,<0.73767013.0>,<0.73767013.0>,couch_work_queue,[{max_size,100000},{max_items,500}],[]]],[link,monitor]],[{error_info,#{module => erl_erts_errors}}]},{proc_lib,spawn_opt,4,[{file,"proc_lib.erl"},{line,192}]},{proc_lib,start_link,5,[{file,"proc_lib.erl"},{line,358}]},{couch_mrview_updater,start_update,4,[{file,"src/couch_mrview_updater.erl"},{line,26}]},{couch_index_updater,'-update/3-fun-4-',8,[{file,"src/couch_index_updater.erl"},{line,173}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,559}]}]} [error] 2025-05-20T05:39:47.229895Z [email protected] <0.75491598.0> -------- CRASH REPORT Process (<0.75491598.0>) with 1 neighbors crashed with reason: system_limit at erlang:spawn_opt/4 <= proc_lib:spawn_opt/4(line:192) <= proc_lib:start_link/5(line:358) <= couch_file:open/2(line:72) <= couch_mrview_util:open_file/1(line:860) <= couch_mrview_index:open/2(line:123) <= couch_index:'-init/1-fun-0-'/3(line:76) <= couch_util:with_db/2(line:559); initial_call: {couch_index,init,['Argument__1']}, ancestors: [<0.75129833.0>], message_queue_len: 0, links: [<0.75129833.0>], dictionary: [{io_priority,{view_update,<<"shards/00000000-7fffffff/vessel_0945.17309...">>}}], trap_exit: false, status: running, heap_size: 1598, stack_size: 28, reductions: 1842 [error] 2025-05-20T05:39:47.230238Z [email protected] <0.75491598.0> -------- CRASH REPORT Process (<0.75491598.0>) with 1 neighbors crashed with reason: system_limit at erlang:spawn_opt/4 <= proc_lib:spawn_opt/4(line:192) <= proc_lib:start_link/5(line:358) <= couch_file:open/2(line:72) <= couch_mrview_util:open_file/1(line:860) <= couch_mrview_index:open/2(line:123) <= couch_index:'-init/1-fun-0-'/3(line:76) <= couch_util:with_db/2(line:559); initial_call: {couch_index,init,['Argument__1']}, ancestors: [<0.75129833.0>], message_queue_len: 0, links: [<0.75129833.0>], dictionary: [{io_priority,{view_update,<<"shards/00000000-7fffffff/vessel_0945.17309...">>}}], trap_exit: false, status: running, heap_size: 1598, stack_size: 28, reductions: 1842 [error] 2025-05-20T05:39:47.230551Z [email protected] emulator -------- Error in process <0.69857375.0> on node '[email protected]' with exit value: {{badmatch,{system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.75491598.0>,[<0.75129833.0>],gen,init_it,[gen_server,<0.75491598.0>,<0.75491598.0>,couch_file,{"./data/.shards/00000000-7fffffff/vessel_0945.1730977511_design/mrview/d1e62a1b1e90e57aa2bddcd0066e1db0.view",[nologifmissing],<0.75491598.0>,#Ref<0.934213240.2265710595.49597>},[]]],[link,monitor]],[{error_info,#{module => erl_erts_errors}}]},{proc_lib,spawn_opt,4,[{file,"proc_lib.erl"},{line,192}]},{proc_lib,start_link,5,[{file,"proc_lib.erl"},{line,358}]},{couch_file,open,2,[{file,"src/couch_file.erl"},{line,72}]},{couch_mrview_util,open_file,1,[{file,"src/couch_mrview_util.erl"},{line,860}]},{couch_mrview_index,open,2,[{file,"src/couch_mrview_index.erl"},{line,123}]},{couch_index,'-init/1-fun-0-',3,[{file,"src/couch_index.erl"},{line,76}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,559}]}]}},[{ken_server,update_ddoc_views,4,[{file,"src/ken_server.erl"},{line,404}]},{ken_server,update_ddoc_indexes,3,[{file, "src/ken_server.erl"},{line,318}]},{ken_server,'-update_db_indexes/2-fun-1-',4,[{file,"src/ken_server.erl"},{line,276}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,1599}]},{ken_server,update_db_indexes,2,[{file,"src/ken_server.erl"},{line,273}]}]} [error] 2025-05-20T05:39:47.230857Z [email protected] emulator -------- Error in process <0.69857375.0> on node '[email protected]' with exit value: {{badmatch,{system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.75491598.0>,[<0.75129833.0>],gen,init_it,[gen_server,<0.75491598.0>,<0.75491598.0>,couch_file,{"./data/.shards/00000000-7fffffff/vessel_0945.1730977511_design/mrview/d1e62a1b1e90e57aa2bddcd0066e1db0.view",[nologifmissing],<0.75491598.0>,#Ref<0.934213240.2265710595.49597>},[]]],[link,monitor]],[{error_info,#{module => erl_erts_errors}}]},{proc_lib,spawn_opt,4,[{file,"proc_lib.erl"},{line,192}]},{proc_lib,start_link,5,[{file,"proc_lib.erl"},{line,358}]},{couch_file,open,2,[{file,"src/couch_file.erl"},{line,72}]},{couch_mrview_util,open_file,1,[{file,"src/couch_mrview_util.erl"},{line,860}]},{couch_mrview_index,open,2,[{file,"src/couch_mrview_index.erl"},{line,123}]},{couch_index,'-init/1-fun-0-',3,[{file,"src/couch_index.erl"},{line,76}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,559}]}]}},[{ken_server,update_ddoc_views,4,[{file,"src/ken_server.erl"},{line,404}]},{ken_server,update_ddoc_indexes,3,[{file, "src/ken_server.erl"},{line,318}]},{ken_server,'-update_db_indexes/2-fun-1-',4,[{file,"src/ken_server.erl"},{line,276}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,1599}]},{ken_server,update_db_indexes,2,[{file,"src/ken_server.erl"},{line,273}]}]} [error] 2025-05-20T05:39:47.230888Z [email protected] emulator -------- Too many processes [error] 2025-05-20T05:39:47.230902Z [email protected] emulator -------- Too many processes [warning] 2025-05-20T05:39:47.233668Z [email protected] <0.368.0> -------- mem3_distribution : node [email protected] down, reason: net_kernel_terminated [error] 2025-05-20T05:39:47.233799Z [email protected] <0.38.0> -------- gen_server net_kernel terminated with reason: system_limit at erlang:spawn_opt/4 <= inet_tcp_dist:gen_setup/6(line:411) <= net_kernel:setup/5(line:1811) <= net_kernel:do_auto_connect_2/5(line:668) <= net_kernel:handle_info/2(line:974) <= gen_server:try_handle_info/3(line:1095) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) last msg: redacted state: {state,'[email protected]',longnames,{tick,<0.40.0>,60000,4},7000,sys_dist,#{},#{},#{},[{listen,#Port<0.4>,<0.39.0>,{net_address,{{127,0,0,1},42261},"127.0.0.1",tcp,inet},inet_tcp_dist}],[],0,#{},net_sup,#{}} extra: [] [error] 2025-05-20T05:39:47.234138Z [email protected] <0.38.0> -------- gen_server net_kernel terminated with reason: system_limit at erlang:spawn_opt/4 <= inet_tcp_dist:gen_setup/6(line:411) <= net_kernel:setup/5(line:1811) <= net_kernel:do_auto_connect_2/5(line:668) <= net_kernel:handle_info/2(line:974) <= gen_server:try_handle_info/3(line:1095) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) last msg: redacted state: {state,'[email protected]',longnames,{tick,<0.40.0>,60000,4},7000,sys_dist,#{},#{},#{},[{listen,#Port<0.4>,<0.39.0>,{net_address,{{127,0,0,1},42261},"127.0.0.1",tcp,inet},inet_tcp_dist}],[],0,#{},net_sup,#{}} extra: [] [error] 2025-05-20T05:39:47.234719Z [email protected] <0.349.0> -------- gen_server '[email protected]' terminated with reason: system_limit at erlang:spawn_opt/4 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:73) <= gen_server:try_handle_cast/3(line:1121) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) last msg: redacted state: {st,#Ref<0.934213240.2211053577.41460>,#Ref<0.934213240.2211053577.41461>,{[],[]},0,0} extra: [] [error] 2025-05-20T05:39:47.235065Z [email protected] <0.349.0> -------- gen_server '[email protected]' terminated with reason: system_limit at erlang:spawn_opt/4 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:73) <= gen_server:try_handle_cast/3(line:1121) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) last msg: redacted state: {st,#Ref<0.934213240.2211053577.41460>,#Ref<0.934213240.2211053577.41461>,{[],[]},0,0} extra: [] [error] 2025-05-20T05:39:47.235442Z [email protected] <0.38.0> -------- CRASH REPORT Process net_kernel (<0.38.0>) with 1 neighbors crashed with reason: system_limit at erlang:spawn_opt/4 <= inet_tcp_dist:gen_setup/6(line:411) <= net_kernel:setup/5(line:1811) <= net_kernel:do_auto_connect_2/5(line:668) <= net_kernel:handle_info/2(line:974) <= gen_server:try_handle_info/3(line:1095) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241); initial_call: {net_kernel,init,['Argument__1']}, ancestors: [net_sup,kernel_sup,<0.19.0>], message_queue_len: 1, links: [<0.40.0>,<0.35.0>], dictionary: [{longnames,true}], trap_exit: true, status: running, heap_size: 2586, stack_size: 28, reductions: 25290927 [error] 2025-05-20T05:39:47.235678Z [email protected] <0.38.0> -------- CRASH REPORT Process net_kernel (<0.38.0>) with 1 neighbors crashed with reason: system_limit at erlang:spawn_opt/4 <= inet_tcp_dist:gen_setup/6(line:411) <= net_kernel:setup/5(line:1811) <= net_kernel:do_auto_connect_2/5(line:668) <= net_kernel:handle_info/2(line:974) <= gen_server:try_handle_info/3(line:1095) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241); initial_call: {net_kernel,init,['Argument__1']}, ancestors: [net_sup,kernel_sup,<0.19.0>], message_queue_len: 1, links: [<0.40.0>,<0.35.0>], dictionary: [{longnames,true}], trap_exit: true, status: running, heap_size: 2586, stack_size: 28, reductions: 25290927 [error] 2025-05-20T05:39:47.235917Z [email protected] <0.349.0> -------- CRASH REPORT Process [email protected] (<0.349.0>) with 0 neighbors crashed with reason: system_limit at erlang:spawn_opt/4 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:73) <= gen_server:try_handle_cast/3(line:1121) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241); initial_call: {rexi_server,init,['Argument__1']}, ancestors: [rexi_server_sup,rexi_sup,<0.343.0>], message_queue_len: 67, links: [<0.346.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 332961192 [error] 2025-05-20T05:39:47.236087Z [email protected] <0.349.0> -------- CRASH REPORT Process [email protected] (<0.349.0>) with 0 neighbors crashed with reason: system_limit at erlang:spawn_opt/4 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:73) <= gen_server:try_handle_cast/3(line:1121) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241); initial_call: {rexi_server,init,['Argument__1']}, ancestors: [rexi_server_sup,rexi_sup,<0.343.0>], message_queue_len: 67, links: [<0.346.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 332961192 [error] 2025-05-20T05:39:47.236238Z [email protected] <0.346.0> -------- Supervisor rexi_server_sup had child '[email protected]' started with rexi_server:start_link('[email protected]') at <0.349.0> exit with reason system_limit at erlang:spawn_opt/4 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:73) <= gen_server:try_handle_cast/3(line:1121) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) in context child_terminated [error] 2025-05-20T05:39:47.236332Z [email protected] <0.346.0> -------- Supervisor rexi_server_sup had child '[email protected]' started with rexi_server:start_link('[email protected]') at <0.349.0> exit with reason system_limit at erlang:spawn_opt/4 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:73) <= gen_server:try_handle_cast/3(line:1121) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) in context child_terminated [error] 2025-05-20T05:39:47.236476Z [email protected] <0.35.0> -------- Supervisor net_sup had child net_kernel started with net_kernel:start_link(#{clean_halt => true,name => '[email protected]',name_domain => longnames,supervisor => net_sup}) at <0.38.0> exit with reason system_limit at erlang:spawn_opt/4 <= inet_tcp_dist:gen_setup/6(line:411) <= net_kernel:setup/5(line:1811) <= net_kernel:do_auto_connect_2/5(line:668) <= net_kernel:handle_info/2(line:974) <= gen_server:try_handle_info/3(line:1095) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) in context child_terminated [error] 2025-05-20T05:39:47.236638Z [email protected] <0.35.0> -------- Supervisor net_sup had child net_kernel started with net_kernel:start_link(#{clean_halt => true,name => '[email protected]',name_domain => longnames,supervisor => net_sup}) at <0.38.0> exit with reason system_limit at erlang:spawn_opt/4 <= inet_tcp_dist:gen_setup/6(line:411) <= net_kernel:setup/5(line:1811) <= net_kernel:do_auto_connect_2/5(line:668) <= net_kernel:handle_info/2(line:974) <= gen_server:try_handle_info/3(line:1095) <= gen_server:handle_msg/6(line:1183) <= proc_lib:init_p_do_apply/3(line:241) in context child_terminated [error] 2025-05-20T05:39:47.236709Z [email protected] <0.35.0> -------- Supervisor net_sup had child net_kernel started with net_kernel:start_link(#{clean_halt => true,name => '[email protected]',name_domain => longnames,supervisor => net_sup}) at <0.38.0> exit with reason reached_max_restart_intensity in context shutdown [error] 2025-05-20T05:39:47.236789Z [email protected] <0.35.0> -------- Supervisor net_sup had child net_kernel started with net_kernel:start_link(#{clean_halt => true,name => '[email protected]',name_domain => longnames,supervisor => net_sup}) at <0.38.0> exit with reason reached_max_restart_intensity in context shutdown [error] 2025-05-20T05:39:47.236829Z [email protected] <0.22.0> -------- Supervisor kernel_sup had child net_sup started with erl_distribution:start_link() at <0.35.0> exit with reason shutdown in context child_terminated [error] 2025-05-20T05:39:47.236863Z [email protected] <0.22.0> -------- Supervisor kernel_sup had child net_sup started with erl_distribution:start_link() at <0.35.0> exit with reason shutdown in context child_terminated [error] 2025-05-20T05:39:47.236946Z [email protected] <0.22.0> -------- Supervisor kernel_sup had child net_sup started with erl_distribution:start_link() at <0.35.0> exit with reason reached_max_restart_intensity in context shutdown [error] 2025-05-20T05:39:47.236982Z [email protected] <0.22.0> -------- Supervisor kernel_sup had child net_sup started with erl_distribution:start_link() at <0.35.0> exit with reason reached_max_restart_intensity in context shutdown [error] 2025-05-20T05:39:47.240982Z [email protected] <0.68837790.0> 059726d8e3 req_err(3318763005) internal_server_error : No DB shards could be opened. [<<"fabric_util:get_shard/4 L133">>,<<"fabric:get_security/2 L217">>,<<"chttpd_auth_request:db_authorization_check/1 L109">>,<<"chttpd_auth_request:authorize_request/1 L19">>,<<"chttpd:handle_req_after_auth/2 L428">>,<<"chttpd:process_request/1 L410">>,<<"chttpd:handle_request_int/1 L345">>,<<"mochiweb_http:headers/6 L140">>] [error] 2025-05-20T05:39:47.253594Z [email protected] <0.70309110.0> 2163077f12 req_err(3318763005) internal_server_error : No DB shards could be opened. [<<"fabric_util:get_shard/4 L133">>,<<"fabric:get_security/2 L217">>,<<"chttpd_auth_request:db_authorization_check/1 L109">>,<<"chttpd_auth_request:authorize_request/1 L19">>,<<"chttpd:handle_req_after_auth/2 L428">>,<<"chttpd:process_request/1 L410">>,<<"chttpd:handle_request_int/1 L345">>,<<"mochiweb_http:headers/6 L140">>] ``` [NOTE]: # ( Describe the problem you're encountering. ) [TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! ) ## Steps to Reproduce [NOTE]: # ( Include commands to reproduce, if possible. curl is preferred. ) ## Expected Behaviour [NOTE]: # ( Tell us what you expected to happen. ) ## Your Environment [TIP]: # ( Include as many relevant details about your environment as possible. ) [TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. ) * CouchDB version used: 3.5.0 * Browser name and version: NA * Operating system and version: Debian 12 ## Additional Context The server is a VM with 12 cors, and 62GB of ram -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
