I had a cluster with 7 Riak nodes. They have multiple backends, a LevelDB on “fast disc” and a LevelDB on “big disc”. The “big disc" was filling up, amongst others because deletion of old data is not working properly (I’m still running 1.4.12, getting ready to switch to 2.x).
So I added a new node, number 8, with bigger discs and planned a migration of all data from one of the nodes (number 5) to the new node. However that has been running for almost a week now. I had to restart the node a few times because I hit a kernel BUG (seems like a race condition with heavy write activity, triggered after 10-20 hours, running ZFS … yes, ZFS on Linux however I want a snapshot before migrating to Riak 2.x). The directories on the NEW node actually hold more data then the OLD node, see “du -sm” output below: NEW: 434021 /bigdata/riak/bigleveldb/456719261665907161938651510223838443642478919680 456331 /bigdata/riak/bigleveldb/730750818665451459101842416358141509827966271488 413217 /bigdata/riak/bigleveldb/822094670998632891489572718402909198556462055424 409491 /bigdata/riak/bigleveldb/91343852333181432387730302044767688728495783936 OLD: 411476 /bigdata/riak/bigleveldb/456719261665907161938651510223838443642478919680 412018 /bigdata/riak/bigleveldb/730750818665451459101842416358141509827966271488 376401 /bigdata/riak/bigleveldb/822094670998632891489572718402909198556462055424 386021 /bigdata/riak/bigleveldb/91343852333181432387730302044767688728495783936 I’ve also tried changing “transfer-limit”, to see if things would move more quickly with 1, 2 or 4 and also to force a stop/start. But it looks like the ownership will never complete. Can someone please advise how to proceed? E.g. can I just mark node nr 5 as down with riak-admin since it looks like the new node already has all the data? And it will read-repair whatever it’s missing? (note that I’ve switched off AAE as that doesn’t seem to work well with this data as well). As a side note I know that I’m using values that are considered too large for typical Riak usage with values ranging 1-10Mb / value. Thanks, Timo
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
