Hi Mark, Yeah, you know I thought about this as soon as I got on my bicycle to ride home. And remembered that while I did bump the limit on my QA setup I did not on the production setup. Oops, I'm sure that's it. I'll write back if that doesn't solve it.
Thanks for the quick response. -Anthony On Tue, Feb 01, 2011 at 04:22:34PM -0800, Mark Phillips wrote: > Hey Anthony, > > At first glance, it looks like you may have run into an open file > handle problem. Try increasing the ulimit to at least 1024. There is > more info on the wiki here: > > http://wiki.basho.com/Open-Files-Limit.html > > Give that a shot and let us know if the problem persists. > > Mark > > On Tue, Feb 1, 2011 at 4:08 PM, Anthony Molinaro > <[email protected]> wrote: > > Hi, > > > > I just set up a 4 node cluster mostly vanilla config with the exception > > that I specified 1024 partitions, and I'm using a multi-backend with one > > entry for bitcask and the default as bitcask (I plan to deploy a cache > > backend at some point). I have one bucket which stores a pretty small > > payload (key is 36 bytes, value is 36 bytes). > > > > Things ran fine under light load (~400 get, ~30 puts according to riak-admin > > status, so I think that's per minute). Suddenly several nodes (3/4) all > > shutdown within a few minutes of each other. They all seem to have > > errors like this > > > > =ERROR REPORT==== 1-Feb-2011::23:50:35 ===^M > > Failed to open lock file > > /var/lib/riak/bitcask/1156070631091827503657211635254091060470024765440/bitcask.write.lock: > > emfile > > > > then a state machine termination stacktrace. > > > > I had run cluster_info for all of them a few minutes before and the only > > machine which didn't crash was the one I ran cluster_info on. > > > > Not sure if that was the cause or not. Any ideas what could cause these > > errors? > > > > I can send more info if it would help, but wanted to get the conversation > > started before I head home. > > > > Thanks, > > > > -Anthony > > > > -- > > ------------------------------------------------------------------------ > > Anthony Molinaro <[email protected]> > > > > _______________________________________________ > > riak-users mailing list > > [email protected] > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > -- ------------------------------------------------------------------------ Anthony Molinaro <[email protected]> _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
