[ https://issues.apache.org/jira/browse/COUCHDB-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160929#comment-13160929 ]
Benoit Chesneau commented on COUCHDB-1212: ------------------------------------------ @filipe I know it happen under heavy load, but having timeout are here to prevent any unexpected long task. It could solve this problem but what about unknown others. I"m not a fan about removing such guards because. This task shouldn't be infinite. and if this task is longer than expected maybe we should process it differently. @robert exactly. > Newly created user accounts cannot sign-in after _user database crashes > ------------------------------------------------------------------------ > > Key: COUCHDB-1212 > URL: https://issues.apache.org/jira/browse/COUCHDB-1212 > Project: CouchDB > Issue Type: Bug > Components: Database Core, HTTP Interface > Affects Versions: 1.0.2 > Environment: Ubuntu 10.10, Erlang R14B02 (erts-5.8.3) > Reporter: Jan van den Berg > Priority: Critical > Labels: _users, authentication > Attachments: couchdb-1212.patch > > > We have one (4,5 GB) couch database and we use the (default) _users database > to store user accounts for a website. Once a week we need to restart couchdb > because newly sign-up user accounts cannot login any more. They get a HTTP > statuscode 401 from the _session HTTP interface. We update, and compact the > database three times a day. > This is the a stacktrace I see in the couch database log prior to when these > issues occur. > ----------- couch.log --------------- > [Wed, 29 Jun 2011 22:02:46 GMT] [info] [<0.117.0>] Starting compaction for db > "fbm" > [Wed, 29 Jun 2011 22:02:46 GMT] [info] [<0.5753.79>] 127.0.0.1 - - 'POST' > /fbm/_compact 202 > [Wed, 29 Jun 2011 22:02:46 GMT] [info] [<0.5770.79>] 127.0.0.1 - - 'POST' > /fbm/_view_cleanup 202 > [Wed, 29 Jun 2011 22:10:19 GMT] [info] [<0.5773.79>] 86.9.246.184 - - 'GET' > /_session 200 > [Wed, 29 Jun 2011 22:24:39 GMT] [info] [<0.6236.79>] 85.28.105.161 - - 'GET' > /_session 200 > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.84.0>] ** Generic server > couch_server terminating > ** Last message in was {open,<<"fbm">>, > [{user_ctx,{user_ctx,null,[],undefined}}]} > ** When Server state == {server,"/opt/couchbase-server/var/lib/couchdb", > {re_pattern,0,0, > <<69,82,67,80,116,0,0,0,16,0,0,0,1,0,0,0,0,0, > 0,0,0,0,0,0,40,0,0,0,0,0,0,0,0,0,0,0,0,0,0, > 0,93,0,72,25,77,0,0,0,0,0,0,0,0,0,0,0,0,254, > 255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, > 77,0,0,0,0,16,171,255,3,0,0,0,128,254,255, > 255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,69,26, > 84,0,72,0>>}, > 100,2,"Sat, 18 Jun 2011 14:00:44 GMT"} > ** Reason for termination == > ** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.84.0>] {error_report,<0.31.0>, > {<0.84.0>,crash_report, > [[{initial_call,{couch_server,init,['Argument__1']}}, > {pid,<0.84.0>}, > {registered_name,couch_server}, > {error_info, > {exit, > {timeout, > {gen_server,call, > [<0.116.0>,{open_ref_count,<0.10417.79>}]}}, > [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, > {ancestors,[couch_primary_services,couch_server_sup,<0.32.0>]}, > {messages,[]}, > {links,[<0.91.0>,<0.483.0>,<0.116.0>,<0.79.0>]}, > {dictionary,[]}, > {trap_exit,true}, > {status,running}, > {heap_size,6765}, > {stack_size,24}, > {reductions,206710598}], > []]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.79.0>] {error_report,<0.31.0>, > {<0.79.0>,supervisor_report, > [{supervisor,{local,couch_primary_services}}, > {errorContext,child_terminated}, > {reason, > {timeout, > {gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}}, > {offender, > [{pid,<0.84.0>}, > {name,couch_server}, > {mfargs,{couch_server,sup_start_link,[]}}, > {restart_type,permanent}, > {shutdown,1000}, > {child_type,worker}]}]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.91.0>] ** Generic server <0.91.0> > terminating > ** Last message in was {'EXIT',<0.84.0>, > {timeout, > {gen_server,call, > [<0.116.0>, > {open_ref_count,<0.10417.79>}]}}} > ** When Server state == {db,<0.91.0>,<0.92.0>,nil,<<"1308405644393791">>, > <0.90.0>,<0.94.0>, > {db_header,5,91,0, > {378285,{30,9}}, > {380466,39}, > nil,0,nil,nil,1000}, > 91, > {btree,<0.90.0>, > {378285,{30,9}}, > #Fun<couch_db_updater.7.10053969>, > #Fun<couch_db_updater.8.35220795>, > #Fun<couch_btree.5.124754102>, > #Fun<couch_db_updater.9.107593676>}, > {btree,<0.90.0>, > {380466,39}, > #Fun<couch_db_updater.10.30996817>, > #Fun<couch_db_updater.11.96515267>, > #Fun<couch_btree.5.124754102>, > #Fun<couch_db_updater.12.117826253>}, > {btree,<0.90.0>,nil,#Fun<couch_btree.0.83553141>, > #Fun<couch_btree.1.30790806>, > #Fun<couch_btree.2.124754102>,nil}, > 91,<<"_users">>, > > "/opt/couchbase-server/var/lib/couchdb/_users.couch", > [#Fun<couch_doc.7.50754398>], > [],nil, > {user_ctx,null,[],undefined}, > nil,1000, > [before_header,after_header,on_file_open], > true} > ** Reason for termination == > ** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.91.0>] {error_report,<0.31.0>, > {<0.91.0>,crash_report, > [[{initial_call,{couch_db,init,['Argument__1']}}, > {pid,<0.91.0>}, > {registered_name,[]}, > {error_info, > {exit, > {timeout, > {gen_server,call, > [<0.116.0>,{open_ref_count,<0.10417.79>}]}}, > [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, > {ancestors,[<0.89.0>]}, > {messages,[]}, > {links,[]}, > {dictionary,[]}, > {trap_exit,true}, > {status,running}, > {heap_size,610}, > {stack_size,24}, > {reductions,8797798}], > []]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.300.0>] Shutting down view group > server, monitored db is closing. > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.10417.79>] Uncaught error in HTTP > request: {exit, > {{timeout, > {gen_server,call, > [<0.116.0>, > {open_ref_count,<0.10417.79>}]}}, > {gen_server,call, > [couch_server, > {open,<<"fbm">>, > [{user_ctx, > {user_ctx,null,[],undefined}}]}, > infinity]}}} > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.483.0>] ** Generic server > <0.483.0> terminating > ** Last message in was {'EXIT',<0.84.0>, > {timeout, > {gen_server,call, > [<0.116.0>, > {open_ref_count,<0.10417.79>}]}}} > ** When Server state == {db,<0.483.0>,<0.484.0>,nil,<<"1308405937993370">>, > <0.4643.19>,<0.4645.19>, > {db_header,5,890453,0, > {3279126950,{752003,0}}, > {3279118313,752003}, > {3279132318,[]}, > 0,nil,3279127184,1000}, > 890453, > {btree,<0.4643.19>, > {3279126950,{752003,0}}, > #Fun<couch_db_updater.7.10053969>, > #Fun<couch_db_updater.8.35220795>, > #Fun<couch_btree.5.124754102>, > #Fun<couch_db_updater.9.107593676>}, > {btree,<0.4643.19>, > {3279118313,752003}, > #Fun<couch_db_updater.10.30996817>, > #Fun<couch_db_updater.11.96515267>, > #Fun<couch_btree.5.124754102>, > #Fun<couch_db_updater.12.117826253>}, > {btree,<0.4643.19>, > {3279132318,[]}, > #Fun<couch_btree.0.83553141>, > #Fun<couch_btree.1.30790806>, > #Fun<couch_btree.2.124754102>,nil}, > 890453,<<"fbm_full">>, > > "/opt/couchbase-server/var/lib/couchdb/fbm_full.couch", > [#Fun<couch_doc.7.50754398>], > [{<<"admins">>, > {[{<<"names">>,[]}, > {<<"roles">>,[<<"import">>]}]}}, > {<<"readers">>, > {[{<<"names">>,[]},{<<"roles">>,[]}]}}], > 3279127184, > {user_ctx,null,[],undefined}, > nil,1000, > [before_header,after_header,on_file_open], > false} > ** Reason for termination == > ** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.483.0>] {error_report,<0.31.0>, > {<0.483.0>,crash_report, > [[{initial_call,{couch_db,init,['Argument__1']}}, > {pid,<0.483.0>}, > {registered_name,[]}, > {error_info, > {exit, > {timeout, > {gen_server,call, > [<0.116.0>,{open_ref_count,<0.10417.79>}]}}, > [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, > {ancestors,[<0.480.0>]}, > {messages,[]}, > {links,[]}, > {dictionary,[]}, > {trap_exit,true}, > {status,running}, > {heap_size,6765}, > {stack_size,24}, > {reductions,1389}], > []]}} > [Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.2984.19>] Shutting down view group > server, monitored db is closing. > [Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.10417.79>] Stacktrace: > [{gen_server,call,3}, > {couch_server,open,2}, > {couch_db,open,2}, > {couch_httpd_db,do_db_req,2}, > {couch_httpd,handle_request_int,5}, > {mochiweb_http,headers,5}, > {proc_lib,init_p_do_apply,3}] > ------- end -------- > Here's the log file of me signing in as an admin, creating a new user, and > trying to sign-in as the newly created user. > ------ couch.log ------- > [Fri, 01 Jul 2011 18:37:16 GMT] [info] [<0.20439.91>] 93.92.103.118 - - > 'POST' /_session 200 > [Fri, 01 Jul 2011 18:37:16 GMT] [info] [<0.20457.91>] checkpointing view > update at seq 91 for _users _design/_auth > [Fri, 01 Jul 2011 18:37:16 GMT] [info] [<0.20439.91>] 93.92.103.118 - - 'GET' > /_users/_design/_auth/_list/secure/users 200 > [Fri, 01 Jul 2011 18:38:35 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'PUT' > /_users/org.couchdb.user:exam...@mail.com 201 > [Fri, 01 Jul 2011 18:38:35 GMT] [info] [<0.20457.91>] checkpointing view > update at seq 92 for _users _design/_auth > [Fri, 01 Jul 2011 18:38:35 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'GET' > /_users/_design/_auth/_list/secure/users 200 > [Fri, 01 Jul 2011 18:38:47 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'GET' > /_users/_design/_auth/_list/secure/users?key=%22org.couchdb.user:exam...@mail.com%22 > 200 > [Fri, 01 Jul 2011 18:38:47 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'PUT' > /_users/org.couchdb.user:exam...@mail.com 201 > [Fri, 01 Jul 2011 18:39:00 GMT] [info] [<0.20547.91>] 93.92.103.118 - - 'GET' > /_session 200 > [Fri, 01 Jul 2011 18:39:01 GMT] [info] [<0.20547.91>] 93.92.103.118 - - 'GET' > /fbm/_design/api/_list/secure/competitions 200 > [Fri, 01 Jul 2011 18:39:12 GMT] [info] [<0.20547.91>] 93.92.103.118 - - > 'POST' /_session 401 > [Fri, 01 Jul 2011 18:39:22 GMT] [info] [<0.20547.91>] 93.92.103.118 - - > 'POST' /_session 401 > ------- end --------- > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira