Hi, The most likely explanation is there is a document that you update frequently that happens to land in the 80000000-9fffffff shard range.
Noting that you did not need to delete and replace the file, we strongly recommend against modifying database files directly, as compaction would have fixed this fragmentation issue for you as it writes out a new file and then swaps at the end. Compaction is an essential maintenance task so perhaps you should revise your schedule (or start compacting regularly if you have not been so far). -- Robert Samuel Newson [email protected] On Mon, 22 Apr 2019, at 22:37, Delfino Gustavo wrote: > (resending hopefully with corrected ascii table layout ) > > It turned out that my replications started failing shortly after my > last message. I was able to fix them, and I would like to share how. > > The problem was on my local server with a specific large database. As I > have had fragmentation issues in the past with a log file so I > suspected that this could be the problem. I used the contig.exe tool in > order to check for the number of fragments of the respective data file > (.couch) in each of the shards. This is what I found: > > Shard 1: "00000000-1fffffff" 330 fragments (~8GB) > > Shard 2: "20000000-3fffffff" 286 fragments (~6GB) > > Shard 3: "40000000-5fffffff" 333 fragments (~7GB) > > Shard 4: "60000000-7fffffff" 305 fragments (~6GB) > > Shard 5: "80000000-9fffffff" 100601 fragments (~10GB) > > Shard 6: "a0000000-bfffffff" 346 fragments (~8GB) > > Shard 7: "c0000000-dfffffff" 362 fragments (~10GB) > > Shard 8: "e0000000-ffffffff" 252 fragments (~5GB) > > I have no idea why the data file for shard 5 became so fragmented. The > contig.exe tool would not defragment it probably because it was too > large but I was able to drastically reduce the number of fragments by > shutting down CouchDB and replacing the file with a duplicate of > itself. Now the replication is working fine. > > Regards, > > Gustavo Delfino > > > -----Original Message----- > From: Delfino Gustavo <[email protected]> > Sent: Monday, April 22, 2019 2:05 PM > To: [email protected] > Subject: RE: CouchDB replication crashing > > Thank you Nick and Peter for your help. It is now back to normal. > > I had plenty of disk space on CouchDB database hard disks, but the > startup disk was 99.5% full. Besides dealing with this space situation > I ran the defragment tool as I know that it can also be an issue under > windows. Thankfully I'll be moving this DB to Linux soon. > > Regards, > > Gustavo Delfino > > -----Original Message----- > From: Nick Vatamaniuc <[email protected]> > Sent: Monday, April 22, 2019 12:19 PM > To: [email protected] > Subject: Re: CouchDB replication crashing > > Hi Gustavo, > > "no match of right hand value {error,eio}" looks like a failure to > write to a block device. Perhaps disks are full, or there is some kind > of throttling > (over-quota) issue? > > Replication will restart on failures. There is an exponential backoff > on repeated errors so if your replications crash too many times in a > row they'd be penalized and wait a bit. But I'd investigate why there > are these EIO errors happening first. > > Cheers, > -Nick > > On Mon, Apr 22, 2019 at 11:39 AM Delfino Gustavo <[email protected]> > wrote: > > > I am doing multimaster replication where each server is configured to > > push its changes to the other one. The log I sent was from the remote > > server trying to send the changes to my local server. As an experiment > > I instructed the local server to pull from the remote and that is not > > working either. This is a segment of the log in the local server: > > > > [error] 2019-04-22T13:39:06.972000Z couchdb@localhost <0.20502.2490> > > -------- CRASH REPORT Process (<0.20502.2490>) with 1 neighbors > > exited with reason: no match of right hand value {error,eio} at > > couch_bt_engine_stream:write/2(line:60) <= > > couch_stream:do_write/2(line:302) <= > > couch_stream:handle_call/3(line:278) > > <= gen_server:try_handle_call/4(line:615) <= > > gen_server:handle_msg/5(line:647) <= > > proc_lib:init_p_do_apply/3(line:247) > > at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_stream,init,['Argument__1']}, ancestors: [<0.27672.2480>], messages: > > [], links: [<0.27672.2480>], dictionary: > > [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims. > > 15...">>}}], > > trap_exit: false, status: running, heap_size: 987, stack_size: 27, > > reductions: 596 > > [error] 2019-04-22T13:39:07.081000Z couchdb@localhost <0.18062.2491> > > -------- Replicator, request PUT to "http:// > > nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad07bc57-1:results?new_edits=false" > > failed due to error {code,500} > > [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.4458.2476> > > -------- Replicator, request PUT to "http:// > > nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad0c5e1c-3:results?new_edits=false" > > failed due to error {code,500} > > [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490> > > -------- gen_server <0.22292.2490> terminated with reason: no match of > > right hand value {error,eio} at > > couch_bt_engine_stream:write/2(line:60) <= > > couch_stream:do_write/2(line:302) <= > > couch_stream:handle_call/3(line:278) > > <= gen_server:try_handle_call/4(line:615) <= > > gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247) > > last msg: close > > state: > > {stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.74448900 > > .31762>,[],[<<31, > > (...etc...),0>>],2186,4096,0,<<>>,<<>>,2186,#Fun<couch_stream.8.801299 > > 67>,#Fun<couch_stream.9.80129967>} > > [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490> > > -------- CRASH REPORT Process (<0.22292.2490>) with 1 neighbors > > exited with reason: no match of right hand value {error,eio} at > > couch_bt_engine_stream:write/2(line:60) <= > > couch_stream:do_write/2(line:302) <= > > couch_stream:handle_call/3(line:278) > > <= gen_server:try_handle_call/4(line:615) <= > > gen_server:handle_msg/5(line:647) <= > > proc_lib:init_p_do_apply/3(line:247) > > at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_stream,init,['Argument__1']}, ancestors: [<0.16555.2491>], messages: > > [], links: [<0.16555.2491>], dictionary: > > [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims. > > 15...">>}}], > > trap_exit: false, status: running, heap_size: 987, stack_size: 27, > > reductions: 676 > > [error] 2019-04-22T13:39:07.627000Z couchdb@localhost <0.13178.2487> > > -------- Replicator, request PUT to "http:// > > nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad043479-71:results?new_edits=false" > > failed due to error {code,500} > > [error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483> > > -------- gen_server <0.30019.2483> terminated with reason: no match of > > right hand value {error,eio} at > > couch_bt_engine_stream:write/2(line:60) <= > > couch_stream:do_write/2(line:302) <= > > couch_stream:handle_call/3(line:278) > > <= gen_server:try_handle_call/4(line:615) <= > > gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247) > > last msg: close > > state: > > {stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.88604680 > > .223648>,[],[<<31,(...etc...),0>>],2039,4096,0,<<>>,<<>>,2039,#Fun<cou > > ch_stream.8.80129967>,#Fun<couch_stream.9.80129967>} > > [error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483> > > -------- CRASH REPORT Process (<0.30019.2483>) with 1 neighbors > > exited with reason: no match of right hand value {error,eio} at > > couch_bt_engine_stream:write/2(line:60) <= > > couch_stream:do_write/2(line:302) <= > > couch_stream:handle_call/3(line:278) > > <= gen_server:try_handle_call/4(line:615) <= > > gen_server:handle_msg/5(line:647) <= > > proc_lib:init_p_do_apply/3(line:247) > > at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_stream,init,['Argument__1']}, ancestors: [<0.4711.2491>], messages: > > [], links: [<0.4711.2491>], dictionary: > > [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims. > > 151...">>}}], > > trap_exit: false, status: running, heap_size: 987, stack_size: 27, > > reductions: 676 > > > > I have now restarted the service and replication briefly resumed but > > it is now failing again. I'll continue to investigate. > > > > Regards, > > > > Gustavo Delfino > > > > > > -----Original Message----- > > From: Peter J Krawetzky <[email protected]> > > Sent: Sunday, April 21, 2019 3:08 PM > > To: [email protected] > > Subject: Re: CouchDB replication crashing > > > > Are the errors below on the primary or the replication server? > > PeterSent from my Verizon, Samsung Galaxy smartphone > > -------- Original message --------From: Delfino Gustavo < > > [email protected]> Date: 4/21/19 13:25 (GMT-05:00) To: > > [email protected] Subject: CouchDB replication crashing Hello, I > > have had two CouchDB servers replicating a couple of DBs for several > > months. One server is local and the other is in remote location. Both > > are running CouchDB 2.2.0 under Windows 7. Replication is failing and > > I know I can reactivate it by restarting the remote server but I would > > like to understand if there is anything I can do to prevent this > > situation in the first place.In previous CouchDB versions, I used to > > have this problem where the replication seems to be disconnected from > > the network while the DB continues to be accessible. It had been > > working very well under 2.2.0. I wonder if moving up to 2.3 would > > help.A representative segment of the log file is found > > below.Regards,Gustavo Delfino[error] 2019-04-21T12:38:21.805000Z > > couchdb@localhost <0.28098.851> -------- Replicator, request PUT to " > > http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false"; > > failed due to error {code,500}[error] 2019-04-21T12:38:24.020000Z > > couchdb@localhost <0.8055.852> -------- Replicator, request PUT to " > > http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false"; > > failed due to error {code,500}[error] 2019-04-21T12:38:28.497000Z > > couchdb@localhost <0.12289.852> -------- Replicator, request PUT to " > > http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false"; > > failed due to error {code,500}[error] 2019-04-21T12:38:51.819000Z > > couchdb@localhost <0.11004.852> -------- Replicator, request GET to " > > http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-211?revs=true&open_revs=%5B%221-4c79aa0880ec04131b40c06855b36ff4%22%5D&latest=true"; > > failed due to error {error,req_timedout}[error] > > 2019-04-21T12:38:51.819000Z couchdb@localhost <0.9866.852> -------- > > Replicator, request GET to " > > http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-15?atts_since=%5B%222-597b62c56ad573a21a63e62c1117856e%22%5D&revs=true&open_revs=%5B%223-5f29e1678a58784ee2e8aee46c57b29d%22%5D&latest=true"; > > failed due to error sel_conn_closed[error] 2019-04-21T12:38:52.240000Z > > couchdb@localhost <0.21284.852> -------- Replicator, request PUT to " > > http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false"; > > failed due to error {code,500}[error] 2019-04-21T12:39:01.678000Z > > couchdb@localhost <0.19310.852> -------- Replicator, request POST to " > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to > > error {code,500}[error] 2019-04-21T12:39:01.678000Z couchdb@localhost > > <0.9042.852> -------- gen_server <0.9042.852> terminated with reason: > > {process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}} last msg: {'EXIT',<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}} > > state: [{main_pid,<0.9322.852>},{loop,<0.6467.852>},{source," > > http://farawayhost.com:5984/jobhandlersims/"},{target,"; > > http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_ > > fetch,nil},{batch_size,268141}][error] > > 2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852> -------- > > CRASH REPORT Process (<0.9042.852>) with 1 neighbors exited with reason: > > {process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}} at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_replicator_worker,init,['Argument__1']}, ancestors: > > [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...] > > , > > messages: [], links: [<0.9322.852>,<0.6467.852>], dictionary: > > [{last_stats_report,{1555,850299,901000}}], trap_exit: true, status: > > running, heap_size: 6772, stack_size: 27, reductions: 393291[error] > > 2019-04-21T12:39:18.916000Z couchdb@localhost <0.9687.852> -------- > > Replicator, request POST to " > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to > > error {code,500}[error] 2019-04-21T12:39:18.916000Z couchdb@localhost > > <0.12947.851> -------- gen_server <0.12947.851> terminated with reason: > > {process_died,<0.9687.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}} last msg: {'EXIT',<0.9687.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}} > > state: [{main_pid,<0.9322.852>},{loop,<0.15756.851>},{source," > > http://farawayhost.com:5984/jobhandlersims/"},{target,"; > > http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_ > > fetch,nil},{batch_size,383624}][error] > > 2019-04-21T12:39:18.916000Z couchdb@localhost <0.12947.851> -------- > > CRASH REPORT Process (<0.12947.851>) with 1 neighbors exited with reason: > > {process_died,<0.9687.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}} at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_replicator_worker,init,['Argument__1']}, ancestors: > > [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...] > > , > > messages: [], links: [<0.9322.852>,<0.15756.851>], dictionary: > > [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status: > > running, heap_size: 6772, stack_size: 27, reductions: 552772[error] > > 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> -------- > > Replicator, request POST to " > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to > > error {code,500}[error] 2019-04-21T12:39:27.964000Z couchdb@localhost > > <0.10607.852> -------- gen_server <0.10607.852> terminated with reason: > > {http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}} at couch_replicator_httpc:report_error/4(line:363) <= > > couch_replicator_httpc:send_req/3(line:76) <= > > couch_replicator_worker:flush_docs/2(line:493) <= > > couch_replicator_worker:maybe_flush_docs/3(line:459) <= > > couch_replicator_worker:maybe_flush_docs/2(line:436) <= > > couch_replicator_worker:handle_call/3(line:124) <= > > gen_server:try_handle_call/4(line:615) <= > > gen_server:handle_msg/5(line:647) last msg: > > {batch_doc,{doc,<<"4bb0aa8269a879fdc95107973872dcfb-424">>,{3,[<<157,204,36,86,54,61,219,128,174,147,218,239,156,55,31,23>>,<<61,194,111,234,124,76,158,204,215,47,185,48,158,105,104,215>>,<<203,140,2,100,128,7,203,236,171,131,32,29,132,106,170,51>>]}, > > (...erlang json removed...) ,[],false,[]}} state: > > [{main_pid,<0.9322.852>},{loop,<0.1165.852>},{source," > > http://farawayhost.com:5984/jobhandlersims/"},{target,"; > > http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,20},{pending > > _fetch,{{<0.1165.852>,#Ref<0.0.23068677.99464>},{<<"4bb0aa8269a879fdc9 > > 5107973872dcfb-174">>,[{3,<<139,102,22,196,97,193,111,166,119,37,58,20 > > 6,57,73,182,130>>}],[{2,<<132,199,224,202,205,103,3,105,55,29,255,196, > > 200,63,170,152>>}]}}},{batch_size,523788}][error] > > 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> -------- > > CRASH REPORT Process (<0.10607.852>) with 15 neighbors exited with reason: > > {http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}} at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_replicator_worker,init,['Argument__1']}, ancestors: > > [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...] > > , > > messages: > > [{'EXIT',<0.24052.852>,normal},{'$gen_call',{<0.20083.852>,...},...},. > > ..], > > links: [<0.22451.852>,<0.24660.852>,<0.25927.852>,<0.27245.852>,...], > > dictionary: > > [{last_stats_report,{1555,850301,711000}},{ibrowse_stream_status,...}] > > , > > trap_exit: true, status: running, heap_size: 28690, stack_size: 27, > > reductions: 744275[error] 2019-04-21T12:39:27.995000Z > > couchdb@localhost <0.9322.852> -------- Worker <0.9042.852> died with > > reason: > > {process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> > > -------- Replication > > `6ccccb02a7483deae5c1c434bce9b376+continuous+create_target` (` > > http://farawayhost.com:5984/jobhandlersims/` > > <http://farawayhost.com:5984/jobhandlersims/> -> ` > > http://nearbyhost.com:5984/jobhandlersims/` > > <http://nearbyhost.com:5984/jobhandlersims/>) failed: > > {worker_died,<0.9042.852>, {process_died,<0.19310.852>, > > {http_request_failed,"POST", " > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs";, > > {error,{code,500}}}}}[error] 2019-04-21T12:39:27.995000Z > > couchdb@localhost <0.9322.852> -------- gen_server > > {couch_replicator_scheduler_job,{[54,99,99,99,99,98,48,50,97,55,52,56, > > 51,100,101,97,101,53,99,49,99,52,51,52,98,99,101,57,98,51,55,54],[43,9 > > 9,111,110,116,105,110,117,111,117,115,43,99,114,101,97,116,101,95,116, > > 97,114,103,101,116]}} > > terminated with reason: > > {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}} > > last msg: > > {'EXIT',<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}} > > state: > > [{rep_id,{"6ccccb02a7483deae5c1c434bce9b376","+continuous+create_target"}},{source," > > http://farawayhost.com:5984/jobhandlersims/"},{target,"; > > http://nearbyhost.com:5984/jobhandlersims/"},{db_name,<<"shards/400000 > > 00-5fffffff/_replicator.1526394302">>},{doc_id,<<"DE_to_US_for_jobhand > > lersims">>},{options,[{checkpoint_interval,30000},{connection_timeout, > > 30000},{continuous,true},{create_target,true},{http_connections,20},{r > > etries,5},{selector,{[{<<"$or">>,[{[{<<"type">>,{[{<<"$ne">>,<<"result > > s">>}]}}]},{[{<<"views">>,{[{<<"$gte">>,<<>>}]}}]},{[{<<"type">>,<<"re > > sults">>},{<<"requester_location">>,<<"US">>}]}]}]}},{socket_options,[ > > {keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batc > > h_size,500},{worker_processes,4}]},{session_id,<<"436547a63366ea84a38d > > 8aeb36dd2d88">>},{start_seq,{0,<<"3194311-g1AAAAJ7eJzLYWBg4MhgTmEQTM4v > > Tc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAM > > kke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsZqDDTMwsUizMDIg1bAEkGH1cMNUDCGGmRqk > > mBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceA > > Bi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQD5VMZk">>}},{sourc > > e_seq,<<"3229426-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJ > > TIkyf___z8riYHtnzQedUkKQDLJHqqUTQ6fUgeQ0nioUokz-JQmgJTWQ5WaLcSjNI8FSDI > > 0ACmg6vkg5R0PCSpfAFG-H6icNUKdoPIDEOX3QconXyCo_AFEOdjtdSVZAOiudpk">>},{ > > committed_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzM > > nILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1L > > Mzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZG > > gAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1 > > N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{current_through_seq > > ,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTI > > kyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB > > 5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMg > > HG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHE > > AOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{highest_seq_done,{8,<<"3204835-g > > 1AAAAI3eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge3 > > T91ygGLu5qWFqimEyNj14TEpSAJJJ9iDDkhhYf17Cp9QBpDQebi-3MNheAxOLNAsjA1LtT > > QAZVg-xl01zKR6leSxAkqEBSAFVz4fY3egIttsy0Sgx2dKCRLshBi6AGLgfbCCrVxPYQKO > > 0REsLM2OyDDwAMfA-xMB2C0ismBslpyaakmXgA4iB0OAuSgcbaJKUkmxkmYhNaxYAmZez- > > A">>}}][error] 2019-04-21T12:39:27.995000Z couchdb@localhost > > <0.9322.852> -------- CRASH REPORT Process (<0.9322.852>) with 3 > > neighbors exited with reason: > > {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}} at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: > > [couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: > > [{'EXIT',<0.12947.851>,{process_died,<0.9687.852>,...}},...], links: > > [<0.162.852>,<0.5422.852>,<0.10686.852>,<0.25534.850>,...], dictionary: > > [{pending_count_state,{{1555,850367,995000},9348}},{task_status_update > > ,...},...], > > trap_exit: true, status: running, heap_size: 10958, stack_size: 27, > > reductions: 63152[error] 2019-04-21T12:39:27.995000Z couchdb@localhost > > <0.10686.852> -------- gen_server <0.10686.852> terminated with reason: > > {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}} > > last msg: > > {'EXIT',<0.9322.852>,{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}}} > > state: [{main_pid,<0.9322.852>},{loop,<0.6805.852>},{source," > > http://farawayhost.com:5984/jobhandlersims/"},{target,"; > > http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,1},{pending_ > > fetch,nil},{batch_size,351185}][error] > > 2019-04-21T12:39:27.995000Z couchdb@localhost <0.10686.852> -------- > > CRASH REPORT Process (<0.10686.852>) with 2 neighbors exited with reason: > > {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}} at gen_server:terminate/7(line:812) <= > > proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_replicator_worker,init,['Argument__1']}, ancestors: > > [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...] > > , > > messages: [], links: [<0.6805.852>,<0.20253.851>], dictionary: > > [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status: > > running, heap_size: 2586, stack_size: 27, reductions: 496765[error] > > 2019-04-21T12:39:27.995000Z couchdb@localhost <0.385.0> -------- > > Supervisor couch_replicator_scheduler_sup had child undefined started > > with {couch_replicator_scheduler_job,start_link,undefined} at > > <0.9322.852> exit with reason > > {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST"," > > http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500 > > }}}}} in context child_terminated[error] 2019-04-21T12:40:13.313000Z > > couchdb@localhost <0.23845.851> 7ec5e9ec3e rexi_server: from: > > couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 > > exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.423000Z couchdb@localhost > > <0.19545.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost > > <0.26362.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost > > <0.25940.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost > > <0.25878.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost > > <0.25863.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost > > <0.28372.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost > > <0.29759.851> 7ec5e9ec3e > > rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:43:23.323000Z couchdb@localhost > > <0.16895.850> ed368f9947 > > rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li > > ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222} > > ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{ > > line,478}]}][error] 2019-04-21T12:43:23.339000Z couchdb@localhost > > <0.3976.852> ed368f9947 > > rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]}, > > {couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{lin > > e,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}] > > }][error] 2019-04-21T12:43:23.339000Z couchdb@localhost <0.3667.852> > > ed368f9947 > > rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: > > fabric_rpc:changes/3 exit:timeout > > [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang > > es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en > > gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]}, > > {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848 > > }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line, > > 775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]}, > > {couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{lin > > e,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}] > > }] > > >
