From: Antonio Teixeira eagle.anto...@gmail.com
To: Matthew Von-Maszewski matth...@basho.com
Cc: riak-users riak-users@lists.basho.com
Subject: Re: Riak LevelDB Deletion Problem
Message-ID:
CAF+3j-tCr+CDD7PiWLm-QOaO69jWDzGyGhP8X9jsZyH=wsb...@mail.gmail.com
Content-Type: text/plain;
I had a cluster with 7 Riak nodes. They have multiple backends, a LevelDB on
“fast disc” and a LevelDB on “big disc”. The “big disc was filling up, amongst
others because deletion of old data is not working properly (I’m still running
1.4.12, getting ready to switch to 2.x).
So I added a new
I have a 7 node Riak cluster. Most nodes are on 1.4.10 and I am currently
migrating the nodes from RedHat to Ubuntu (and Ext4 - ZFS).
One of the nodes is running Ubuntu 14 with ZFS for the data and has Riak 1.4.12
installed. On this node Riak is continuously reading and writing a lot of data.
Try running `riak-admin top` on the node to see what's going on:
http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top
http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top
This may give you some insight into what that node is doing.
Thanks for the quick
AAE is disabled in the configuration file. The aae-status shows no AAE activity
was ever performed.
-Timo
Hi Timo. Did you check if there is active anti-entropy activity? [1] That
could generate a lot of I/O in the background while building the trees with
the data needed for automatic
Unfortunately (not really…) the I/O has now stopped. Looking at the CPU usage
and load graphs in New Relic it seems the high I/O stopped about two hours ago.
I will see if I find anything unusual in the Riak logging around that time and
post it. Otherwise I’m just happy that Riak is behaving
Hi,
I have recently added a 7th node to an existing cluster of 6 nodes. Standard
ring size. Riak 1.4.9, the new node is on Ubuntu 14, I used the Ubuntu 12
package.
It seems like the progress of transferring data is really, really slow. The
disks are continuously reading and writing data
Upgrading to 1.4.10 indeed solved this.
Thanks,
Timo
Thanks very much for your prompt response. I’ve upgraded and restarted Riak,
still getting the same errors. I’ll run the leveldb:repair on all partitions
first and then restart Riak.
-Timo
On 23 nov. 2014, at 14:46, Matthew
Date: Thu, 10 Apr 2014 14:42:23 +0100
From: Guido Medina guido.med...@temetra.com
To: Engel Sanchez en...@basho.com, Luke Bakken lbak...@basho.com
Cc: riak-users riak-users@lists.basho.com
Subject: Re: Rebuilding AAE hashes - small question
Message-ID: 53469fbf.1000...@temetra.com
...So I stopped AAE on all nodes (with riak attach), removed the AAE folders
on all the nodes. And then restarted them one-by-one, so they all started
with a clean AAE state. Then about a day later the cluster was finally in a
normal state.
I don't understand the difference between what
I also have 6 servers. Each server has about 2Tb of data. So maybe my
anti_entropy dir size is “normal”.
Kind regards,
Timo
p.s. I’m using the multi_backend as I have some data only in memory
(riak_kv_memory_backend); all data on disk is in riak_kv_eleveldb_backend.
Well, my anti-entropy
Since this is related to my earlier question: sorry to have let you wait,
Timo.
Kelly, the reason I brought up my original question was because my use case
involves delivering videos under load.
Suppose there is a cluster of 50 nodes with a replication value of three. Now
if a random
Related to the recent thread about Lowlevel Access to RiakCS objects I plan
to implement an extension to a Riak Object (in the Golang driver at
https://github.com/tpjg/goriakpbc) that will cover two use cases:
1) storing larger files directly in Riak (not Riak CS)
2) store growing files, e.g.
Hi Edgar,
please note that I’ve run into the same problem. And I did what you’re
describing: restarting the nodes one by one and removing the AAE folders.
However I am strongly suspecting that this didn’t solve it completely for me
since a few nodes kept thrashing the disk continuously. My
Is it possible to turn anti entropy on and off without stopping riak entirely?
Especially with the upgrade to 1.4.8 where it is recommended to delete the
existing anti_entropy directory it seems that after the upgrade Riak is reading
data continuously. Since it is not possible to limit the I/O
When you say Riak is reading data continuously, are you referring to the AAE
trees being rebuilt? I just want to confirm that you did remove the
anti_entropy directory, as not removing it can cause additional repair
operations to occur.
I did remove the anti_entropy directories on all (6)
I’m looking for advice what the best way forward would be with a failed drive.
I have a cluster of 6 machines. The data partition (not including the ring
file, only the LevelDB data) is on a separate drive. On one of the servers that
drive has crashed and is replaced by a new drive.
What
Thanks a lot, never saw that page! I’m definitely going to try that … as soon
as it’s after 22.00 so as not to disturb the users too much.
-Timo
Hi Timo,
Since you have the original ring from the node, all you should have to
do is replace the drive, bring Riak up using the same settings
I have upgraded two nodes in my cluster from 1.3.2 to 1.4.2. After the upgrade
it appears both started a compaction, as mentioned here:
https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md#leveldb-13-to-14-conversion.
In my case I think this may have also triggered compactions at level 5
Matthew, thanks very much for your feedback. I may bite the bullet and try your
suggested last resort method on one of the nodes. I'll let you know how that
goes. All nodes are 1.3 or newer.
As to your question: my key is a bit of both and is quite long for typical Riak
usage I guess. The
Excellent news. Thanks very much!
-Timo
On 25 sep. 2013, at 21:44, Matthew Von-Maszewski wrote:
Timo,
Your email again brought up the topic of fixing this issue within Basho's
leveldb. Previously there had always been a bigger problem and we did not
worry about tombstone (delete)
I'm running a database where the values are typically fairly large (1-3Mb). Now
I'm planning to remove older data (1 year ago) by moving it into Glacier and
only keeping some metadata in the database. I am currently planning to remove
the content of any K/V itself (object.raw_data in the Ruby
Thanks for the clarification Matthew. One thing remaining and I understand
there is no clear cut, 100% answer to this one. Your remark regarding old
values of a key only getting purged if they happen to participate in the same
compaction, can you please elaborate on that and specifically what
Christian and Jeremiah, thanks for your responses. See my comments inline below.
Date: Fri, 21 Sep 2012 10:46:36 +0100
From: Christian Dahlqvist christ...@whitenode.com
accounthttps://github.com/whitenode/riak_mapreduce_utilsand I also
submitted them to Basho Contrib yesterday for
Hello,
I have a question regarding the use of 2i. I've read some of the benefits (like
less writes when inserting data) of using 2i instead of links when modeling
data and it seems this is the preferred option nowadays when using Ripple
(Ruby).
I now have some data that keeps links (and links
As a follow-up, the node that was leaving is now stopped. The logging looks
kind of normal to me, but I notice that all handoffs went only to a single
other node. I have dumped the logging here:
http://pastebin.com/u6HYNjSZ
My cluster has nodes 1-6, node6 was already converted to 1.1 with
I have a 6 node cluster. One node is running 1.1 already, the other 5 are still
0.14.
I want not only to upgrade from 0.14 to 1.1 but also change backend from bit
cask to leveled, so I thought I would do a leave cluster, wait for handoff,
upgrade and change config, join cluster cycle for each
27 matches
Mail list logo
