Re: CouchDB 1.0.2 errors under load

2011-03-25 Thread Pasi Eronen
I've finally managed to reproduce this problem in a test environment, and filed a bug report: https://issues.apache.org/jira/browse/COUCHDB-1100 Best regards, Pasi On Sat, Feb 26, 2011 at 11:19, Pasi Eronen p...@iki.fi wrote: BTW, it seems just changing ulimit -n did not work. At some point,

Re: CouchDB 1.0.2 errors under load

2011-02-26 Thread Pasi Eronen
BTW, it seems just changing ulimit -n did not work. At some point, the server just stopped responding completely (even the root URL just times out, and there's nothing in the error log). A quick googling found references to enabling kernel poll (at build time and on erl command line), and

CouchDB 1.0.2 errors under load

2011-02-25 Thread Pasi Eronen
Hi, I had a big batch job (inserting 10M+ documents and generating views for them) that ran just fine for about 6 hours, but then I got this error: [Thu, 24 Feb 2011 19:42:57 GMT] [error] [0.276.0] ** Generic server 0.276.0 terminating ** Last message in was delayed_commit ** When Server state

Re: CouchDB 1.0.2 errors under load

2011-02-25 Thread Paul Davis
On Fri, Feb 25, 2011 at 4:18 AM, Pasi Eronen p...@iki.fi wrote: Hi, I had a big batch job (inserting 10M+ documents and generating views for them) that ran just fine for about 6 hours, but then I got this error: [Thu, 24 Feb 2011 19:42:57 GMT] [error] [0.276.0] ** Generic server 0.276.0

Re: CouchDB 1.0.2 errors under load

2011-02-25 Thread Pasi Eronen
Thanks! I looked at /proc/(pid-of-beam.smp)/fd, and it seems CouchDB is indeed using a lot of file descriptors, and the number is growing. I restarted CouchDB (and my batch job), and after 40 minutes, it was already using 300 fds; an hour later, the figure was 600; half an hour later, almost 700.

Re: CouchDB 1.0.2 errors under load

2011-02-25 Thread Paul Davis
It sounds like your client is doing something funky with shutting down connections to the server. If I were to guess I would say that it looks like you're leaking sockets between a call to shutdown and a call to close. Though to say for certain I'd need to see what state both ends of the socket

Re: CouchDB 1.0.2 errors under load

2011-02-25 Thread Pasi Eronen
Yes, the client is doing something funky :-) Many of the requests it sends are POSTs with chunked transfer encoding, so it's being hit hard by issue COUCHDB-682. In other words, there's lot of closing and opening of TCP connections going on, and retrying requests that failed due to COUCHDB-682.

Re: CouchDB 1.0.2 errors under load

2011-02-25 Thread Paul J. Davis
On Feb 25, 2011, at 7:31 PM, Pasi Eronen p...@iki.fi wrote: Yes, the client is doing something funky :-) Many of the requests it sends are POSTs with chunked transfer encoding, so it's being hit hard by issue COUCHDB-682. In other words, there's lot of closing and opening of TCP