[I] mem3_sync repeatedly logs {{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}}} after replacing a cluster node [couchdb]

[via GitHub]

andres-mariano opened a new issue, #5612:
URL: https://github.com/apache/couchdb/issues/5612

   ## Description
   
   I’m seeing CouchDB’s internal mem3_sync process keep trying to push shards 
to a node that I’ve already replaced/removed from the cluster, and it logs this 
warning every sync cycle:
   This only happens in one of the three nodes in the cluster.
   ```
   [warning] 2025-08-01T00:37:24.341627Z couchdb@172.16.4.190 <0.323.0> 
-------- mem3_sync shards/80000000-ffffffff/conditions.1752004476 
couchdb@172.16.7.221 
{{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}},[{mem3_rpc,rexi_call,3,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,387}]},{mem3_rep,calculate_start_seq,3,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,497}]},{maps,map_1,3,[{file,[109,97,112,115,46,101,114,108]},{line,441}]},{maps,map,2,[{file,[109,97,112,115,46,101,114,108]},{line,426}]},{mem3_rep,calculate_start_seq_multi,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,483}]},{mem3_rep,repl,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,312}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,122}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line
 ,241}]}]}
   
{{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,387}]},{mem3_rep,calculate_start_seq,3,[{file,"src/mem3_rep.erl"},{line,497}]},{maps,map_1,3,[{file,"maps.erl"},{line,441}]},{maps,map,2,[{file,"maps.erl"},{line,426}]},{mem3_rep,calculate_start_seq_multi,1,[{file,"src/mem3_rep.erl"},{line,483}]},{mem3_rep,repl,1,[{file,"src/mem3_rep.erl"},{line,312}]},{mem3_rep,go,1,[{file,"src/mem3_rep.erl"},{line,122}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,"src/mem3_sync.erl"},{line,241}]}]}
   
{{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,387}]},{mem3_rep,calculate_start_seq,3,[{file,"src/mem3_rep.erl"},{line,497}]},{maps,map_1,3,[{file,"maps.erl"},{line,441}]},{maps,map,2,[{file,"maps.erl"},{line,426}]},{mem3_rep,calculate_start_seq_multi,1,[{file,"src/mem3_rep.erl"},{line,483}]},{mem3_rep,repl,1,[{file,"src/mem3_rep.erl"},{line,312}]},{mem3_rep,go,1,[{file,"src/mem3_rep.erl"},{line,122}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,"src/mem3_sync.erl"},{line,241}]}]}
   [warning] 2025-08-01T00:37:29.342756Z couchdb@172.16.4.190 <0.323.0> 
-------- mem3_sync shards/00000000-7fffffff/conditions.1752004476 
couchdb@172.16.7.221 
{{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}},[{mem3_rpc,rexi_call,3,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,387}]},{mem3_rep,calculate_start_seq,3,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,497}]},{maps,map_1,3,[{file,[109,97,112,115,46,101,114,108]},{line,441}]},{maps,map,2,[{file,[109,97,112,115,46,101,114,108]},{line,426}]},{mem3_rep,calculate_start_seq_multi,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,483}]},{mem3_rep,repl,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,312}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,122}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line
 ,241}]}]}
   
{{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,387}]},{mem3_rep,calculate_start_seq,3,[{file,"src/mem3_rep.erl"},{line,497}]},{maps,map_1,3,[{file,"maps.erl"},{line,441}]},{maps,map,2,[{file,"maps.erl"},{line,426}]},{mem3_rep,calculate_start_seq_multi,1,[{file,"src/mem3_rep.erl"},{line,483}]},{mem3_rep,repl,1,[{file,"src/mem3_rep.erl"},{line,312}]},{mem3_rep,go,1,[{file,"src/mem3_rep.erl"},{line,122}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,"src/mem3_sync.erl"},{line,241}]}]}
   
{{rexi_DOWN,{'couchdb@172.16.7.221',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,387}]},{mem3_rep,calculate_start_seq,3,[{file,"src/mem3_rep.erl"},{line,497}]},{maps,map_1,3,[{file,"maps.erl"},{line,441}]},{maps,map,2,[{file,"maps.erl"},{line,426}]},{mem3_rep,calculate_start_seq_multi,1,[{file,"src/mem3_rep.erl"},{line,483}]},{mem3_rep,repl,1,[{file,"src/mem3_rep.erl"},{line,312}]},{mem3_rep,go,1,[{file,"src/mem3_rep.erl"},{line,122}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,"src/mem3_sync.erl"},{line,241}]}]}
   ```
   
   I’ve verified:
        •       The node is gone (/_node/_local/_nodes/_all_docs no longer 
lists couchdb@172.16.7.221)
        •       The local /_dbs/conditions shard map on all three live nodes no 
longer contains that address in by_node or by_range
        •       Yet mem3_sync still resurrects a replication job to the dead IP 
on each cycle.
   
   **Dump of _dbs conditions document**
   As you can see the only references to the old node are 
`["remove","00000000-7fffffff","couchdb@172.16.7.221"],["remove","80000000-ffffffff","couchdb@172.16.7.221"]`.
 Could that be the issue? I couldn't find any other reference to `172.16.7.221`.
   ```
   
{"id":"conditions","key":"conditions","value":{"rev":"53-f44e2a99509849a25c788f3b3946cc57"},"doc":{"_id":"conditions","_rev":"53-f44e2a99509849a25c788f3b3946cc57","shard_suffix":[46,49,55,53,50,48,48,52,52,55,54],"changelog":[["add","00000000-7fffffff","couc...@conditionsdb1.dev"],["add","00000000-7fffffff","couc...@conditionsdb2.dev"],["add","00000000-7fffffff","couc...@conditionsdb3.dev"],["add","80000000-ffffffff","couc...@conditionsdb1.dev"],["add","80000000-ffffffff","couc...@conditionsdb2.dev"],["add","80000000-ffffffff","couc...@conditionsdb3.dev"],["add","00000000-7fffffff","couchdb@172.16.7.114"],["add","80000000-ffffffff","couchdb@172.16.7.114"],["remove","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couc...@conditionsdb1.dev"],["add","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","00000000-7fffffff","couchdb@172.16.7.114"],["add","80000000-ffffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couchdb@172.16.7
 
.114"],["add","00000000-7fffffff","couchdb@172.16.4.101"],["add","80000000-ffffffff","couchdb@172.16.4.101"],["remove","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couc...@conditionsdb2.dev"],["add","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","00000000-7fffffff","couchdb@172.16.4.101"],["add","80000000-ffffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couchdb@172.16.4.101"],["add","00000000-7fffffff","couchdb@172.16.6.55"],["add","80000000-ffffffff","couchdb@172.16.6.55"],["remove","00000000-7fffffff","couc...@conditionsdb3.dev"],["remove","80000000-ffffffff","couc...@conditionsdb3.dev"],["add","00000000-7fffffff","couc...@conditionsdb3.dev"],["remove","00000000-7fffffff","couchdb@172.16.6.55"],["add","80000000-ffffffff","couc...@conditionsdb3.dev"],["remove","80000000-ffffffff","couchdb@172.16.6.55"],["add","00000000-7fffffff","couchdb@172.16.6.136"],["add","80000000-ffffffff","couchdb@172.16.6.136"],["remove","0
 
0000000-7fffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couc...@conditionsdb2.dev"],["add","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","00000000-7fffffff","couchdb@172.16.6.136"],["add","80000000-ffffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couchdb@172.16.6.136"],["add","00000000-7fffffff","couchdb@172.16.4.135"],["add","80000000-ffffffff","couchdb@172.16.4.135"],["remove","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couc...@conditionsdb2.dev"],["add","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","00000000-7fffffff","couchdb@172.16.4.135"],["add","80000000-ffffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couchdb@172.16.4.135"],["add","00000000-7fffffff","couchdb@172.16.7.1"],["add","80000000-ffffffff","couchdb@172.16.7.1"],["remove","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couc...@conditionsdb2.dev"],["add","0000000
 
0-7fffffff","couc...@conditionsdb2.dev"],["remove","00000000-7fffffff","couchdb@172.16.7.1"],["add","80000000-ffffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couchdb@172.16.7.1"],["add","00000000-7fffffff","couchdb@172.16.5.204"],["add","80000000-ffffffff","couchdb@172.16.5.204"],["remove","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couc...@conditionsdb2.dev"],["add","00000000-7fffffff","couc...@conditionsdb2.dev"],["remove","00000000-7fffffff","couchdb@172.16.5.204"],["add","80000000-ffffffff","couc...@conditionsdb2.dev"],["remove","80000000-ffffffff","couchdb@172.16.5.204"],["add","00000000-7fffffff","couchdb@172.16.6.242"],["add","80000000-ffffffff","couchdb@172.16.6.242"],["remove","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couc...@conditionsdb1.dev"],["add","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","00000000-7fffffff","couchdb@172.16.6.242"],["add","80000000-ffffffff"
 
,"couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couchdb@172.16.6.242"],["add","00000000-7fffffff","couchdb@172.16.4.73"],["add","80000000-ffffffff","couchdb@172.16.4.73"],["remove","00000000-7fffffff","couc...@conditionsdb3.dev"],["remove","80000000-ffffffff","couc...@conditionsdb3.dev"],["add","00000000-7fffffff","couc...@conditionsdb3.dev"],["remove","00000000-7fffffff","couchdb@172.16.4.73"],["add","80000000-ffffffff","couc...@conditionsdb3.dev"],["remove","80000000-ffffffff","couchdb@172.16.4.73"],["add","00000000-7fffffff","couchdb@172.16.5.210"],["add","80000000-ffffffff","couchdb@172.16.5.210"],["remove","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couc...@conditionsdb1.dev"],["add","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","00000000-7fffffff","couchdb@172.16.5.210"],["add","80000000-ffffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couchdb@172.16.5.210"],["add","00000000-7fffffff","couchdb@1
 
72.16.5.87"],["add","80000000-ffffffff","couchdb@172.16.5.87"],["remove","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couc...@conditionsdb1.dev"],["add","00000000-7fffffff","couc...@conditionsdb1.dev"],["remove","00000000-7fffffff","couchdb@172.16.5.87"],["add","80000000-ffffffff","couc...@conditionsdb1.dev"],["remove","80000000-ffffffff","couchdb@172.16.5.87"],["remove","00000000-7fffffff","couchdb@172.16.4.190"],["add","00000000-7fffffff","couchdb@172.16.6.18"],["add","80000000-ffffffff","couchdb@172.16.6.18"],["remove","00000000-7fffffff","couchdb@172.16.7.221"],["remove","80000000-ffffffff","couchdb@172.16.7.221"],["add","00000000-7fffffff","couchdb@172.16.7.69"],["add","80000000-ffffffff","couchdb@172.16.7.69"],["remove","00000000-7fffffff","couchdb@172.16.4.30"],["remove","80000000-ffffffff","couchdb@172.16.4.30"],["add","00000000-7fffffff","couchdb@172.16.7.157"],["add","80000000-ffffffff","couchdb@172.16.7.157"],["remove","00000000-7fffffff
 
","couchdb@172.16.7.69"],["remove","80000000-ffffffff","couchdb@172.16.7.69"],["add","00000000-7fffffff","couchdb@172.16.7.224"],["add","80000000-ffffffff","couchdb@172.16.7.224"],["remove","00000000-7fffffff","couchdb@172.16.7.157"],["remove","80000000-ffffffff","couchdb@172.16.7.157"],["add","00000000-7fffffff","couchdb@172.16.6.243"],["add","80000000-ffffffff","couchdb@172.16.6.243"],["remove","00000000-7fffffff","couchdb@172.16.7.224"],["remove","80000000-ffffffff","couchdb@172.16.7.224"],["add","00000000-7fffffff","couchdb@172.16.7.241"],["add","80000000-ffffffff","couchdb@172.16.7.241"],["remove","00000000-7fffffff","couchdb@172.16.6.243"],["remove","80000000-ffffffff","couchdb@172.16.6.243"]],"by_node":{"couchdb@172.16.4.190":["00000000-7fffffff","80000000-ffffffff"],"couchdb@172.16.6.18":["00000000-7fffffff","80000000-ffffffff"],"couchdb@172.16.7.241":["00000000-7fffffff","80000000-ffffffff"]},"by_range":{"00000000-7fffffff":["couchdb@172.16.4.190","couchdb@172.16.6.18","cou
 
chdb@172.16.7.241"],"80000000-ffffffff":["couchdb@172.16.4.190","couchdb@172.16.6.18","couchdb@172.16.7.241"]},"props":{}}}
   
   ``` 
   
   ## Steps to Reproduce
   
   Not sure if it's easy to reproduce, but I created a cluster with three nodes 
and then replaced them with new nodes several times. I happened to find this 
issue after that.
   
   ## Expected Behaviour
   
   The node should not be trying to connect to the old node.
   
   ## Your Environment
   
   CouchDB version used: 3.5.0
   OS: Ubuntu 24.04 LTS
   
   Any thoughts? Thank you.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@couchdb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org