Riak nodes silently failing, possible problem with handoff_concurrency

2014-03-05 Thread Drew Goya
So I'm having a recurring problem where members of my cluster silently drop off. Their beam.smp process is doing about 1/10 the work of healthy nodes. This looks like it is related to handoff_concurrency. After seeing a large number of log messages like: An outbound handoff of partition

Re: Riak nodes silently failing, possible problem with handoff_concurrency

2014-03-05 Thread Brian Sparrow
Drew, That message is normal and is a result of handoff throttling done with the riak-admin transfer_limit setting. When you say nodes are silently dropping off what do you mean? Is the beam.smp process shutting down or are they just never finishing this handoff? Thanks, Brian -- Brian