Hello! I'm running a load test on CouchDB. I have a cluster of 8 node.js servers writing to CouchDB. They write about 30000 documents per minute (500 per second). There are multiple concurrent requests form each server. There are no updates: documents are created and not modified.
I first tried CouchDB 1.1.1 from Debian 6.4 apt repo. After a few minutes, CouchDB starts freezing for a period of 1 to 3 seconds about every 10 seconds. It keeps this behaviour for some time and eventually it starts freezing more frequently and for longer periods. When the database has about 1.5 million documents, couchdb is freezing for more than 5 seconds each time. I then tried CouchDB 1.2, from build-couch. The freezes happen with it also, but the behavior is much worse: in less than one minute it's freezing for 5 seconds or more, and it spends more time not doing anything than working. When testing with 1.1.1 I was writing only to one database. With 1.2 I tried with one database and then with multiple databases but the problem was exactly the same. The documents have about 10 properties, only numbers or string and the strings are small (about 20 chars each). The document IDs are generated in the app and have the format <milliseconds since epoch>-<random 16 chars string> When CouchDB freezes, it's processor use (from top) goes to zero. It does not reply to read or write requests. The disk does not seem to be the problem as iostat reports near 0% utilization. CPU is mostly idle, and from the 16 GB of RAM, some of it is free and is not even used to cache disk. There are no error message in Couchdb log. I tried this in two different machines and the problem is the same in both. I did not change anything in the configuration files expect changing the database dir to use a RAID partition. Anyone has any idea of what the problem could be? Any help solving this issue is greatly appreciated. Thanks, Marco
