Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Timo Gatsonides
I have a 7 node Riak cluster. Most nodes are on 1.4.10 and I am currently migrating the nodes from RedHat to Ubuntu (and Ext4 -> ZFS). One of the nodes is running Ubuntu 14 with ZFS for the data and has Riak 1.4.12 installed. On this node Riak is continuously reading and writing a lot of data.

Re: Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Doug Rohrer
Try running `riak-admin top` on the node to see what's going on: http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top This may give you some insight into what that node is doing. Doug > On May 22, 2015, at 9:

Re: Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Timo Gatsonides
> Try running `riak-admin top` on the node to see what's going on: > > http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top > > > This may give you some insight into what that node is doing. Thanks for the quic

Re: Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Doug Rohrer
Timo: The general layout of the information is pretty similar to Top (and described on the etop docs http://www.erlang.org/doc/man/etop.html and http://www.erlang.org/doc/apps/observer/etop_ug.html

Re: Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Engel Sanchez
Hi Timo. Did you check if there is active anti-entropy activity? [1] That could generate a lot of I/O in the background while building the trees with the data needed for automatic repair or running exchanges to verify replicas are up to date. [1] http://docs.basho.com/riak/latest/ops/advanced/aae

Re: Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Timo Gatsonides
AAE is disabled in the configuration file. The aae-status shows no AAE activity was ever performed. -Timo > > Hi Timo. Did you check if there is active anti-entropy activity? [1] That > could generate a lot of I/O in the background while building the trees with > the data needed for automa

Re: Riak 1.4.12 high I/O while almost idle

2015-05-22 Thread Timo Gatsonides
Unfortunately (not really…) the I/O has now stopped. Looking at the CPU usage and load graphs in New Relic it seems the high I/O stopped about two hours ago. I will see if I find anything unusual in the Riak logging around that time and post it. Otherwise I’m just happy that Riak is behaving nor