Christopher Malon created COUCHDB-3350:
------------------------------------------
Summary: emfile errors leave replication incomplete
Key: COUCHDB-3350
URL: https://issues.apache.org/jira/browse/COUCHDB-3350
Project: CouchDB
Issue Type: Bug
Components: Replication
Reporter: Christopher Malon
I replicated a database (continuously), but ended up with fewer documents on
the target than on the source. No conflicting activity (deletions, etc.) had
occurred on the target. No updates had happened on the source for hours.
However, the status made it look like the entire sequence of updates had been
processed successfully.
These problems occurred on many occasions when I triggered replications for a
lot of databases simultaneously, to load them onto a newly launched EC2
instance. The target databases would otherwise be functional, and a later
replication of one of the problematic databases to a differently named target
(presumably, with fewer open files at the same time) on the new instances would
copy the entire set of documents.
The incomplete replication could be seen only by comparing the doc_count on
source and target or trying to request one of the unreplicated documents. The
only reported sign of trouble was some errors of the form
{error,{conn_failed,{error,emfile}}}
logged to the console for some GET requests during the time of initial
replication, and messages about "restarting replication." By executing
ulimit -Sn 4096
(the limit was 1024 previously) prior to the initial replications, the problem
went away.
1. Here's the DB entry on the source machine, showing 12 documents:
{"db_name":"library","update_seq":"61-g1AAAAFTeJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rkQGPoiQFIJlkD1bHjE-dA0hdPFgdIz51CSB19WB1BnjU5bEASYYGIAVUOh-_mRC1CyBq9-P3D0TtAYja-1mJbATVPoCoBbqXKQsA-0Fvaw","sizes":{"file":181716,"external":11524,"active":60098},"purge_seq":0,"other":{"data_size":11524},"doc_del_count":0,"doc_count":12,"disk_size":181716,"disk_format_version":6,"data_size":60098,"compact_running":false,"instance_start_time":"0"}
2. Here's the DB entry on the target machine, showing 6 documents:
{"db_name":"library","update_seq":"6-g1AAAAFTeJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rkQGPoiQFIJlkD1bHhE-dA0hdPFgdIz51CSB19QTV5bEASYYGIAVUOh-_GyFqF0DU7idG7QGI2vvEqH0AUQvyfxYA1_dvNA","sizes":{"file":82337,"external":2282,"active":5874},"purge_seq":0,"other":{"data_size":2282},"doc_del_count":0,"doc_count":6,"disk_size":82337,"disk_format_version":6,"data_size":5874,"compact_running":false,"instance_start_time":"0"}
3. Here's _active_tasks for the task, converted to YAML for readability:
- changes_pending: 0
checkpoint_interval: 30000
checkpointed_source_seq: 61-g1AAAAJTeJyd0EsOgjAQBuAqxsfSE-gRKK08VnIT7UwhSBAWyl
pvojfRm-hNsLQkbAgRNtOkk__L5M8IIcvEkmSNRYmJhDArUGRJcblmajUVBDZVVaWJJchZfSwAucPQkW
RV5jKKT3kke-KwVRP2jWBpgdMAwcOuTJ8U1tKhkSZaYhS5x2GodKylWyPZWnJ9QW3KBkr5TE1yV4_CHu
1dMeyQ-c4o7Wm0V9u4F9setaM_GzfK2yifWplrxYeAcuGOuulrNN3X1PTFgXPqd-XSHxdwuSQ
continuous: !!perl/scalar:JSON::PP::Boolean 1
database: shards/00000000-1fffffff/_replicator.1489086006
doc_id: 172.16.100.222_library
doc_write_failures: 0
docs_read: 12
docs_written: 12
missing_revisions_found: 12
node: couchdb@localhost
pid: <0.5521.0>
replication_id: c60427215125bd97559d069f6fb3ddb4+continuous+create_target
revisions_checked: 12
source: http://172.16.100.222:5984/library/
source_seq:
61-g1AAAAJTeJyd0EsOgjAQBuAqxsfSE-gRKK08VnIT7UwhSBAWylpvojfRm-hNsLQkbAgRNtOkk__L5M8IIcvEkmSNRYmJhDArUGRJcblmajUVBDZVVaWJJchZfSwAucPQkWRV5jKKT3kke-KwVRP2jWBpgdMAwcOuTJ8U1tKhkSZaYhS5x2GodKylWyPZWnJ9QW3KBkr5TE1yV4_CHu1dMeyQ-c4o7Wm0V9u4F9setaM_GzfK2yifWplrxYeAcuGOuulrNN3X1PTFgXPqd-XSHxdwuSQ
started_on: 1489086008
target: http://localhost:5984/library/
through_seq:
61-g1AAAAJTeJyd0EsOgjAQBuAqxsfSE-gRKK08VnIT7UwhSBAWylpvojfRm-hNsLQkbAgRNtOkk__L5M8IIcvEkmSNRYmJhDArUGRJcblmajUVBDZVVaWJJchZfSwAucPQkWRV5jKKT3kke-KwVRP2jWBpgdMAwcOuTJ8U1tKhkSZaYhS5x2GodKylWyPZWnJ9QW3KBkr5TE1yV4_CHu1dMeyQ-c4o7Wm0V9u4F9setaM_GzfK2yifWplrxYeAcuGOuulrNN3X1PTFgXPqd-XSHxdwuSQ
type: replication
updated_on: 1489096815
user: peer
4. Here's the _replicator record for the task:
{"_id":"172.16.100.222_library","_rev":"2-8e6cf63bc167c7c7e4bd38242218572c","schema":1,"storejson":null,"source":"http://172.16.100.222:5984/library","target":"http://localhost:5984/library","create_target":true,"dont_storejson":1,"wholejson":{},"user_ctx":{"roles":["_admin"],"name":"peer"},"continuous":true,"owner":null,"_replication_state":"triggered","_replication_state_time":"2017-03-09T19:00:08+00:00","_replication_id":"c60427215125bd97559d069f6fb3ddb4"}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
