Re:
Just a quick follow up, I did this exact approach and things went smoothly as expected. I had forgotten that stopping a node does not mark the node as down indicating that vnodes should be handed off. One last step I would add is that the log directory also needs to be changed in the vm.args file as that is where the erl_crash.dump file gets written (if the log dir is moved as well). Thanks, Jeremy On Fri, Aug 9, 2013 at 7:57 PM, Jeremiah Peschka jeremiah.pesc...@gmail.com wrote: You *should* be able to 1) stop the node 2) change platform_data_dir and ring_state_dir in the app.config file 3) move data to the new platform_data_dir 4) move ring state to the new ring_state_dir 5) restart the node This should avoid having to re-write all of your data one node at a time. --- Jeremiah Peschka - Founder, Brent Ozar Unlimited MCITP: SQL Server 2008, MVP Cloudera Certified Developer for Apache Hadoop On Fri, Aug 9, 2013 at 7:23 PM, Jeremy Ong jer...@quarkgames.com wrote: Hi Riak users, I'm wondering what the best approach to this is. The scenario is that I have mounted a new drive to the machine and want to have the node leverage that drive to save data as opposed to the mount point it is currently writing to. My current plan is to start a second instance of riak (by changing the name in the vm.args file) and having it join the cluster, followed by removing the previous node. Then to repeat this for all nodes for which the upgrade applies while monitoring the dashboard to see if the handoffs are occurring appropriately. Is there an easier way or better way, and are they any potential issues with this approach? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[no subject]
Hi Riak users, I'm wondering what the best approach to this is. The scenario is that I have mounted a new drive to the machine and want to have the node leverage that drive to save data as opposed to the mount point it is currently writing to. My current plan is to start a second instance of riak (by changing the name in the vm.args file) and having it join the cluster, followed by removing the previous node. Then to repeat this for all nodes for which the upgrade applies while monitoring the dashboard to see if the handoffs are occurring appropriately. Is there an easier way or better way, and are they any potential issues with this approach? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: erlang go boom
On the client you could extract the value_count of the objects you
read and just log them. Feel free to post code too, in particular, how
you are writing out updated values.
On Mon, Aug 5, 2013 at 9:20 PM, Paul Ingalls p...@fanzo.me wrote:
Interesting. I have sibling resolution code on the client side. Would
sibling explosion take out the entire cluster all at once? Within 5 minutes
of my last email, the rest of the cluster died.
Is there a way to quickly figure out whether the cluster is full of
siblings?
Paul Ingalls
Founder CEO Fanzo
p...@fanzo.me
@paulingalls
http://www.linkedin.com/in/paulingalls
On Aug 5, 2013, at 8:07 PM, Evan Vigil-McClanahan emcclana...@basho.com
wrote:
Given your leveldb settings, I think that compaction is an unlikely
culprit. But check this out:
2013-08-05 18:01:15.878 [info] 0.83.0@riak_core_sysmon_
handler:handle_event:92 monitor large_heap 0.14832.557
[{initial_call,{riak_kv_get_fsm,init,1}},{almost_current_function,{riak_object,encode_maybe_binary,1}},{message_queue_len,1}]
[{old_heap_block_size,0},{heap_block_size,116769640},{mbuf_size,0},{stack_size,52},{old_heap_size,0},{heap_size,81956791}]
That's a 78MB heap in encode object... Unless your objects are big, I
would suspect sibling explosion caused by rapid updates at w = 1.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Unit testing persistence
Be advised, avoiding mocking/stubbing and making your tests unit
tests are mutually exclusive. A unit test by definition should not
have any dependencies whatsoever (on other modules even, let alone a
database!).
On Wed, Jul 31, 2013 at 9:41 PM, Wagner Camarao wag...@crunchbase.com wrote:
Hi all ~
Great meetup today - looking forward to upgrading to 1.4
I had a question Mark suggested posting here, then we discussed with a few
other folks too: How do we unit / integration test persistence with riak?
Given a basic dev environment, e.g. running only one riak physical node
locally with all configs default, how do we surely read the data we just
wrote?
I have tried setting DW=all (Durable Write - as recommended for best
consistency in the financial example from the little riak book - section for
developers, more than N/R/W) and tried also using {delete_mode, keep} in
riak_kv app.config (since I truncate the buckets after each test suite), but
still, I get intermittent test failures as eventually the data isn't
available for reading right after writing.
Please note I'm trying to avoid mocking / stubbing as well as hacks like
keep trying to read until a certain timeout.
I'm looking ideally for a simple configuration or any known best practices.
Thanks
~W
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Erlang PB client
The X there indicates that it does not support connection pooling out of the box in contrast to the check. I'd look at poolboy (to use in conjunction with riakc_pb_socket) and riakpool (which pulls in riak-erlang-client as a dependency). On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin konstantin.ka...@gmail.com wrote: Looking at http://docs.basho.com/riak/latest/references/Client-Libraries/ I see that Erlang riak client should support Cluster connections/pools. But looking at Erlang riak client source code I would say that it doesn't support Cluster connections/pools out of box. And I have to develop my own code to have load balancing/connection pool. Is it right and the documentation is incorrect and needs to be updated? Thank you, Konstantin. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: question about riak siblings
Hi thanks, I was actually coming to the conclusion that to get this to work in my situation, I would need to embed a client side vector. On Thu, Jun 20, 2013 at 5:34 PM, Jordan West jw...@basho.com wrote: Jeremy, What you are running into sounds a lot like the problems described in [1]. Jordan [1] https://github.com/ricardobcl/Dotted-Version-Vectors On Tue, Jun 18, 2013 at 12:58 PM, Jeremy Ong jer...@quarkgames.comwrote: On Tue, Jun 18, 2013 at 12:57 PM, Jeremy Ong jer...@quarkgames.comwrote: Suppose I have two sources writing to the database, source A and source B. Next, suppose A and B both issue a write to the database so they spawn two siblings: [a1, b1]. Finally, suppose A writes another object using the metadata of the riak object returned when writing a2. This will lead to the following situation: Sorry typo. This should read: Finally, suppose A writes another object using the metadata of the riak object returned when writing a1 [a1, b1, a2]. Upon the next read, the client will be presented with these three siblings. Is there a publicly exposed way to establish a strict ordering between a1 and a2? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
question about riak siblings
Suppose I have two sources writing to the database, source A and source B. Next, suppose A and B both issue a write to the database so they spawn two siblings: [a1, b1]. Finally, suppose A writes another object using the metadata of the riak object returned when writing a2. This will lead to the following situation: [a1, b1, a2]. Upon the next read, the client will be presented with these three siblings. Is there a publicly exposed way to establish a strict ordering between a1 and a2? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: question about riak siblings
On Tue, Jun 18, 2013 at 12:57 PM, Jeremy Ong jer...@quarkgames.com wrote: Suppose I have two sources writing to the database, source A and source B. Next, suppose A and B both issue a write to the database so they spawn two siblings: [a1, b1]. Finally, suppose A writes another object using the metadata of the riak object returned when writing a2. This will lead to the following situation: Sorry typo. This should read: Finally, suppose A writes another object using the metadata of the riak object returned when writing a1 [a1, b1, a2]. Upon the next read, the client will be presented with these three siblings. Is there a publicly exposed way to establish a strict ordering between a1 and a2? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
First Write Wins
For automatic conflict resolution, often times, the easiest thing to do is impose a Last Write Wins policy. However, in certain circumstances, First Write Wins makes more sense. For example, suppose you are handling user creation. Creating a single user will populate values for two keys: User Id = User data Username = User Id A unique User Id is generated to accommodate future username changes. In the event that another user attempts to create an account simultaneously with the same username, a conflict will exist. The easiest thing for me to do here, is after the account is created, check the username to see if it points to the correct User Id. If so, return a success message. Otherwise, rollback changes to the first key. I'm OK with last write wins for now, but is there a way of imposing a simple first write wins policy without needing siblings and client conflict resolution? Cheers, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: First Write Wins
Thanks and yes, for the case of the user creation example I gave, I am setting pw=pr=quorum. Immutable values is a much better way of putting it, although I technically need deletion as well for this particular example. Cheers, Jeremy On Sun, May 19, 2013 at 7:24 PM, John Daily jda...@basho.com wrote: A few quick thoughts... We're tackling a very similar problem (perhaps even the identical problem) with user accounts in Riak CS via the Stanchion subsystem. I know that started out as a component running on a single node to avoid conflicts like this, but I believe it is evolving into a more sophisticated solution. This feels like a feature request for allowing keys to be created with immutable values. I would assume that topic has arisen before, but I'm not aware of when or whether it's something we've seriously considering implementing. Certainly one would have to be careful with sloppy quorums: in the event of a network partition, with default settings it would be possible to create identical keys on different parts of the cluster without an immediate conflict. You should make certain the primary vnodes are always required by specifying pw=quorum and pr=quorum (which would mean that attempting to write the same key in a partitioned scenario would fail unless a majority of the primary vnodes are reachable). Hopefully someone else who's given this some thought in the past will jump in; I'll make further inquiries tomorrow. -John Daily Technical Evangelist On May 19, 2013, at 7:29 PM, Jeremy Ong jer...@quarkgames.com wrote: For automatic conflict resolution, often times, the easiest thing to do is impose a Last Write Wins policy. However, in certain circumstances, First Write Wins makes more sense. For example, suppose you are handling user creation. Creating a single user will populate values for two keys: User Id = User data Username = User Id A unique User Id is generated to accommodate future username changes. In the event that another user attempts to create an account simultaneously with the same username, a conflict will exist. The easiest thing for me to do here, is after the account is created, check the username to see if it points to the correct User Id. If so, return a success message. Otherwise, rollback changes to the first key. I'm OK with last write wins for now, but is there a way of imposing a simple first write wins policy without needing siblings and client conflict resolution? Cheers, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: On siblings
Thanks all for your help. To perform my workflow properly, the
`return_head` option was precisely what I needed. Now I can do something
like
X = Initial Object
{ok, X1} = riakc_pb_socket(Pid, X, [return_head]).
X2 = riakc_obj:update_value(X1, NextValue).
{ok, X3} = riakc_pb_socket(Pid, X2, [return_head]).
And the above will generate no siblings. I was concerned more about
efficiency if using `return_body` but this works just fine.
Cheers,
Jeremy
On Wed, May 15, 2013 at 7:41 AM, Brian Roach ro...@basho.com wrote:
Jeremy -
As noted in the other replies, yes, you need to use 'return_body' to
get the new vector clock in order to avoid creating a sibling on a
subsequent write of the same key.
That said, you can supply the param 'return_head` in the proplist
along with `return_body` which will eliminate having the value
returned to you and get the vclock you need.
- Roach
On Wed, May 15, 2013 at 8:23 AM, John Daily jda...@basho.com wrote:
Thanks for the kind words, Jeremiah.
Jeremy, if you find anything that's wrong with that description of
sibling
behavior, please let me know. It's always possible I missed something
important.
-John
On Wednesday, May 15, 2013, Jeremiah Peschka wrote:
John Daily (@macintux) wrote a great blog post that covers sibling
behavior [1]
In short, though, because you're supplying an older vector clock, and
you
have allow_mult turned on, Riak makes the decision that since a vector
clock
is present that conflicts with what's already on disk a sibling should
be
created.
As I understand it, the only way to write into Riak and not get siblings
is to set allow_mult to false - even leaving out vector clocks will
lead to
siblings if allow_mult is true. Or so John Daily's chart claims.
[1]: http://basho.com/riaks-config-behaviors-part-2/
---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop
On Tue, May 14, 2013 at 10:48 PM, Jeremy Ong jer...@quarkgames.com
wrote:
To clarify, I am using the erlang client. From the looks of it, the
vector clock transition to the new value is opaque to the client so
the only
way to streamline this use case is to pass the `return_body` option
(My use
case is one read, many subsequent writes while updating in memory).
In this case however, I already have the value in memory, so it seems
inefficient to have to get the entire riakc_obj back when I really
just need
the metadata to construct the new object. Is this correct?
On Tue, May 14, 2013 at 9:06 PM, Jeremy Ong jer...@quarkgames.com
wrote:
Suppose I have an object X.
I make an update to X and store it as X1. I perform a put operation
using X1.
The same client then makes a modification to X1 and stores it as X2.
Then, I perform a put operation using X2.
This will create two siblings X1 and X2 if allow_mult is true. Is
there
any way I can avoid this? To me, the vector clock should have
incremented
once when transitioning from X to X1, then once more when
transitioning from
X1 to X2. This way, I shouldn't need to issue a get before I have to
perform
another write since my data is already in memory.
I probably am misunderstanding something about vector clocks. Does
anybody care to clarify this?
Thanks,
Jeremy
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
On siblings
Suppose I have an object X. I make an update to X and store it as X1. I perform a put operation using X1. The same client then makes a modification to X1 and stores it as X2. Then, I perform a put operation using X2. This will create two siblings X1 and X2 if allow_mult is true. Is there any way I can avoid this? To me, the vector clock should have incremented once when transitioning from X to X1, then once more when transitioning from X1 to X2. This way, I shouldn't need to issue a get before I have to perform another write since my data is already in memory. I probably am misunderstanding something about vector clocks. Does anybody care to clarify this? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: On siblings
To clarify, I am using the erlang client. From the looks of it, the vector clock transition to the new value is opaque to the client so the only way to streamline this use case is to pass the `return_body` option (My use case is one read, many subsequent writes while updating in memory). In this case however, I already have the value in memory, so it seems inefficient to have to get the entire riakc_obj back when I really just need the metadata to construct the new object. Is this correct? On Tue, May 14, 2013 at 9:06 PM, Jeremy Ong jer...@quarkgames.com wrote: Suppose I have an object X. I make an update to X and store it as X1. I perform a put operation using X1. The same client then makes a modification to X1 and stores it as X2. Then, I perform a put operation using X2. This will create two siblings X1 and X2 if allow_mult is true. Is there any way I can avoid this? To me, the vector clock should have incremented once when transitioning from X to X1, then once more when transitioning from X1 to X2. This way, I shouldn't need to issue a get before I have to perform another write since my data is already in memory. I probably am misunderstanding something about vector clocks. Does anybody care to clarify this? Thanks, Jeremy ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: ripple and presence property
From an implementation standpoint, see https://github.com/basho/ripple/blob/master/lib/ripple/validations/associated_validator.rb which does the passthrough to: http://guides.rubyonrails.org/active_record_validations_callbacks.html#presence I believe On Mon, Oct 8, 2012 at 11:13 AM, Sean Cribbs s...@basho.com wrote: That is a passthrough to validations, so it is equivalent to: property :email, String validates_presence_of :email On Mon, Oct 8, 2012 at 11:08 AM, Charlie Bowman charlesmbow...@gmail.com wrote: I can't seem to find any information on the :presence property of ripple. e.g. property :email, String, :presence = true I've also searched the source code for ripple, including tests and can't seem to find any info on what this actually does. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- Sean Cribbs s...@basho.com Software Engineer Basho Technologies, Inc. http://basho.com/ ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: 2i with the riakc client
Hi, to close this thread, I wrote a very brief blog post documenting basic
usage of secondary indices and get_index using riakc at
http://www.jeremyong.com/blog/2012/07/23/secondary-indices-in-riak-with-erlang/
It's obviously not geared toward expert users (I for one, am not an expert
user) but I'm throwing it out there in case somebody else struggles with
the same issues.
Cheers,
Jeremy
On Sun, Jul 22, 2012 at 2:05 PM, Jeremy Ong jer...@playmesh.com wrote:
Ah perfect that works thanks.
I saw the github issue asking for documentation on this. I feel woefully
unqualified to write this up, but would be happy to for the benefit of
other users running into similar issues as I.
Thanks,
Jeremy
On Sun, Jul 22, 2012 at 2:02 PM, Sean Cribbs s...@basho.com wrote:
Jeremy,
Your index field must have a _bin or _int suffix on it to signify
whether it is a binary or an integer. Add _bin to the field name you have
in your example, e.g.
MetaData = dict:store(index, [{afield_bin, avalue}],
dict:new())
On Sun, Jul 22, 2012 at 4:49 PM, Jeremy Ong jer...@playmesh.com wrote:
I am trying to store objects with indexed metadata but am running into
various difficulties (commands in bold).
*Obj = riakc_obj:new(foo, bar, zing).*
{riakc_obj,foo,bar,undefined,[],undefined,
zing}
*
*
*MetaData = dict:store(index, [{afield, avalue}], dict:new()).
*
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],
[[index,{afield,avalue}]],
[],[],[],[]}}}
*Obj1 = riakc_obj:update_metadata(Obj, MetaData).*
{riakc_obj,foo,bar,undefined,[],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[],[],[],[],[],[],[],[],[],[[...]],[],...}}},
zing}
*{ok, Pid} = riakc_pb_socket:start_link(127.0.0.1, 8087).*
{ok,0.103.0}
*riakc_pb_socket:put(Pid, Obj1).*
*
*
{error,{0,
{precommit_fail,[{unknown_field_type,\afield\}]}}}
I get this error when I try other things for the field name too. Any
ideas? I tried to look for documentation but nothing seems to be coming up.
Thanks,
Jeremy
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Sean Cribbs s...@basho.com
Software Engineer
Basho Technologies, Inc.
http://basho.com/
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.2 RC1 Release
Hi, looks awesome, which Erlang release is the recommended one for building this? jeremy On Fri, Jul 13, 2012 at 5:31 PM, Gordon Tillman gtill...@mezeo.com wrote: Jared this is great news. There are some supremely-useful additions and enhancements in this release! --gordon On Jul 13, 2012, at 19:16 , Jared Morrow wrote: Riak Users, Today we are really excited to make available Riak 1.2 release candidate 1. Riak 1.2 is a huge release for us and we hope everyone enjoys the new changes. So like the 1.0 and 1.1 release we are providing the release candidate hoping that everyone in the community who is willing to give it a try will download Riak, run it, and tell us what you think. Although the RC signifies that we consider it final release quality, we still ask that you hold off for 1.2 final release before putting into full production. So what has changed? Here is where you can find the current state of the release notes for 1.2. https://github.com/basho/riak/blob/1.2/RELEASE-NOTES.md The major features that have changed are listed, but some fixed bugs are missing. We will be continuing to fill out the release notes until the final release! Download here: http://basho.com/resources/downloads/ One thing you will notice is the downloads page has be reorganized a bit. Now each platform is under its own directory. Hopefully this will help remove some confusion when it comes to picking packages. Also, we've heard your pleas.. static links are now provided on the download page for those of you who deploy Riak via scripts. As always, our whole team thanks our wonderful community (You) for your dedication and love for Riak. Thanks, The Basho Team ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
