Re: Recommended riak configuration options for better performance

2014-06-05 Thread Alex Moore
Hi Naveen, You are running out of MR workers, you’ll either have to: a) Increase the worker limits on the current nodes (particularly map_js_vm_count and reduce_js_vm_count) b) Add more nodes (and thereby more workers) c) Do less MR work. d) Implement your MapReduce functions in Erlang to avoid

Re: Partition distribution between nodes

2014-06-05 Thread Luke Bakken
Hi Manu, Partition distribution is determined by the claim algorithm. In this case it more evenly distributes partitions in a from scratch scenario vs. adding nodes. There has been work to improve the algorithm that you can find here: https://github.com/basho/riak_core/pull/183 -- Luke Bakken

Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Alain Rodriguez
Hi all, I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to 1.4.9. The rest are running 1.4.0. Ever since I am seeing the upgraded node, riak01 consuming a significantly larger percent of CPU and the PUT times on it have gotten worse. htop indicicates one particular process

Re: Using the 'fold' keys threshold

2014-06-05 Thread Alex Moore
Hi Venkat, You can find those settings in our docs: http://docs.basho.com/riak/1.4.9/ops/advanced/backends/bitcask/#Configuring-Bitcask (search for “Fold Keys Threshold”). In Bitcask when we do range operations like “List Keys” or other operations that require us to fold over all the data,

Re:

2014-06-05 Thread Luke Bakken
Hi Gaurav, I believe you are running into this issue: https://github.com/basho/bitcask/issues/136 To resolve, shut down a node and remove all 0-byte data and 18-byte hint files. I would also recommend upgrading Riak at the same time as 1.4.8 and beyond include the fix for this issue. 1.4.9 is

Re: Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Shane McEwan
On 05/06/14 16:20, Alain Rodriguez wrote: Hi all, I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to 1.4.9. The rest are running 1.4.0. Ever since I am seeing the upgraded node, riak01 consuming a significantly larger percent of CPU and the PUT times on it have gotten

Re: Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Alain Rodriguez
Thanks for the quick reply and no I did not. Is this something I should be able to do now (stop, remove files, start again) or is it too late? How could I verify this is the issue? On Thu, Jun 5, 2014 at 8:42 AM, Shane McEwan sh...@mcewan.id.au wrote: On 05/06/14 16:20, Alain Rodriguez wrote:

Re: Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Alain Rodriguez
Actually I just noticed it is likely the AAE issue: 2014-06-05 14:53:47.587 [error] 0.16054.31 CRASH REPORT Process 0.16054.31 with 0 neighbours exited with reason: no match of right hand value {error,{db_open,IO error: lock

Re: Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Engel Sanchez
Hi Alain. I don't think you are seeing the AAE issue. The problem with upgrading from 1.4.4-1.4.7 to 1.4.8 was a broken hash function in those, which made the AAE trees incompatible. You should not have the same problem in 1.4.0. It seems that Erlang processes are repeatedly crashing and

Re: Upgraded riak 1.4.9 is pegging the CPU

2014-06-05 Thread Engel Sanchez
Alain, thanks for the logs you sent me on the side. I'm not yet sure what the root cause is, but I saw a lot of handoff activity and busy distributed port messages, which indicate the single TCP connection between two Erlang nodes is completely saturated. Since there is too much going on,

Cost of many PBC connections held open?

2014-06-05 Thread Toby Corkindale
Hi, Just looking for a bit of guidance on good practice when using Riak via the PBC interface. To keep request latency low, it's good to keep the connection open between requests, rather than making a new connection for just a few requests and then dropping it again, right? In the JVM client,