Re: Associating Riak CRDT Sets to Buckets / Keys via Erlang Client
This is awesome...! Many thanks Magnus - much appreciated...
Must have overlooked some of these details in my initial analysis but am
sure I have a very good starting point / details now!
Thanks again!
On Thu, Nov 17, 2016 at 7:48 AM, Magnus Kessler wrote:
> On 16 November 2016 at 17:40, Vikram Lalit wrote:
>
>> Hi - I am trying to leveraging CRDT sets to store chat messages that my
>> distributed Riak infrastructure would store. Given the intrinsic
>> conflict-resolution, I thought this might be more beneficial than me
>> putting together a merge implementation based on the causal context.
>>
>> However, my data model requires each chat message to be associated to
>> something like a post, hence I was thinking of having the post reference as
>> the bucket, and chat references as keys in that bucket. With of course the
>> bucket-type datasource equated to 'set'. Unfortunately though, from the
>> documentation, I'm not able to ascertain how to associate a created set
>> with an existing bucket and a new key reference if I use the Erlang client.
>> This seems possible for other languages but not for Erlang, with the Basho
>> doc mentioning "%% Sets in the Erlang client are opaque data structures
>> that collect operations as you mutate them. We will associate the data
>> structure
>> with a bucket type, bucket, and key later on.".
>>
>> Subsequent code only seems to fetch the set from the bucket / key but
>> where exactly is the allocation happening?
>>
>> {ok, SetX} = riakc_pb_socket:fetch_type(Pid, {<<"sets">>,<<"travel">>},
>> <<"cities">>).
>>
>> Perhaps I'm missing something, or is there a code snippet that I can
>> leverage?
>>
>> Thanks!
>>
>>
> Hi Vikram,
>
> Please have a look at the following snippet, that shows the complete set
> of operations used to update a CRDT set with the Erlang client:
>
> update_crdt_set(Server, BType, Bucket, Key, Val) ->
> T = unicode:characters_to_binary(BType),
> B = unicode:characters_to_binary(Bucket),
> K = unicode:characters_to_binary(Key),
>
> {ok, Pid} = riakc_pb_socket:start_link(Server, 8087),
>
> Set = case
> riakc_pb_socket:fetch_type(Pid, {T, B}, K)
> of
> {ok, O} -> O;
> {error, {notfound, set}} -> riakc_set:new()
> end,
>
> Set1 = riakc_set:add_element(unicode:characters_to_binary(Val),
> Set),
>
> {ok, {set, Vals, _Adds, _Dels, _Ctx}} =
> riakc_pb_socket:update_type(
> Pid, {T, B}, K, riakc_set:to_op(Set1), [return_body]),
> Vals.
>
> The set is updated with riakc_set:add_element/2, and sent back to the
> server with riakc_pb_socket:update_type/5, which in turn takes an
> argument returned from riakc_set:to_op/1.
>
> More information and samples can be found in the Riak documentation [0]
> and the Riak Erlang client API docs [1][2].
>
> Please let me know if this answered your question.
>
> Kind Regards,
>
> Magnus
>
> [0]: http://docs.basho.com/riak/kv/2.1.4/developing/data-types/sets/
> [1]: https://basho.github.io/riak-erlang-client/riakc_set.html
> [2]: https://basho.github.io/riak-erlang-client/riakc_pb_
> socket.html#update_type-5
>
> --
> Magnus Kessler
> Client Services Engineer
> Basho Technologies Limited
>
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Associating Riak CRDT Sets to Buckets / Keys via Erlang Client
Hi - I am trying to leveraging CRDT sets to store chat messages that my
distributed Riak infrastructure would store. Given the intrinsic
conflict-resolution, I thought this might be more beneficial than me
putting together a merge implementation based on the causal context.
However, my data model requires each chat message to be associated to
something like a post, hence I was thinking of having the post reference as
the bucket, and chat references as keys in that bucket. With of course the
bucket-type datasource equated to 'set'. Unfortunately though, from the
documentation, I'm not able to ascertain how to associate a created set
with an existing bucket and a new key reference if I use the Erlang client.
This seems possible for other languages but not for Erlang, with the Basho
doc mentioning "%% Sets in the Erlang client are opaque data structures
that collect operations as you mutate them. We will associate the data
structure
with a bucket type, bucket, and key later on.".
Subsequent code only seems to fetch the set from the bucket / key but where
exactly is the allocation happening?
{ok, SetX} = riakc_pb_socket:fetch_type(Pid, {<<"sets">>,<<"travel">>}, <<
"cities">>).
Perhaps I'm missing something, or is there a code snippet that I can
leverage?
Thanks!
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: High number of Riak buckets
Hiya Alexander, Thanks much indeed for the detailed note... very interesting insights... As you deduced, I actually omitted some pieces from my email for the sake of simplicity. I'm actually leveraging a transient / stateless chat server (ejabberd) wherein messages get delivered on live sessions / streams without the client having to do look-ups. So the storage in Riak is actually a post-facto delivery / archival rather than prior to the client receiving them. Hence determining the time key for the look-up isn't going to be an issue unless I run some analytics where I query all keys (which would be an issue as I now understand from your comments). There is of course the question of offline messages whose delivery would depend on look-ups, but ejabberd there uses the username (the offline storage is with the secondary index as well on leveldb) and hence the timestamp not being important. Riak TS sure looks promising there but I'll check further whether the change would be justified for only offline messages, or in case other use cases crop up... Makes sense on the listing all keys in a bucket being expensive though - let me see how I can model my data for that!!! Thanks again for your inputs... very informative... Cheers. Vikram On Fri, Sep 30, 2016 at 12:23 PM, Alexander Sicular wrote: > Hi Vikram, > > Bucket maximums aside, why are you modeling in this fashion? How will you > retrieve individual keys if you don't know the time stamp in advance? Do > you have a lookup somewhere else? Doable as lookup keys or crdts or other > systems. Are you relying on listing all keys in a bucket? Definitely don't > do that. > > Yes, there is a better way. Use Riak TS. Create a table with a composite > primary key of topic and time. You can then retrieve by topic equality and > time range. You can then cache those results in deterministic keys as > necessary. > > If you don't already know, Riak TS is basically (there are some notable > differences) Riak KV plus the time series data model. Riak TS makes all > sorts of time series oriented projects easier than modeling them against > KV. Oh, and you can also leverage KV buckets alongside TS (resource > limitations not withstanding.) > > Would love to hear more, > Alexander > > @siculars > http://siculars.posthaven.com > > Sent from my iRotaryPhone > > > On Sep 29, 2016, at 19:42, Vikram Lalit wrote: > > > > Hi - I am creating a messaging platform wherein am modeling each topic > to serve as a separate bucket. That means there can potentially be millions > of buckets, with each message from a user becoming a value on a distinct > timestamp key. > > > > My question is there any downside to modeling my data in such a manner? > Or can folks advise a better way of storing the same in Riak? > > > > Secondly, I would like to modify the default bucket properties (n_val) - > I understand that such 'custom' buckets have a higher performance overhead > due to the extra load on the gossip protocol. Is there a way the default > n_val of newly created buckets be changed so that even if I have the above > said high number of buckets, there is no performance degrade? Believe there > was such a config allowed in app.config but not sure that file is leveraged > any more after riak.conf was introduced. > > > > Thanks much. > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: High number of Riak buckets
Hi Luke - many thanks... actually I was planning to have different bucket types have a different n_val. Or I might end up doing so... the thinking being that I intend to start my production workloads with fewer replications, but as the system matures / stabilizes (and also increases in userbase!), I would want to increase n_val. In my testing that I had done a few weeks ago, each time I tried to increase the n_val of an existing bucket, I've found conflicting results (prior question here: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-July/018631.html) - perhaps due to read-repair taking time - not sure. Understood though from various Riak papers that decreasing n_val should not be done, but couldn't conclude yet as to why would increasing be an issue... So to avoid the scenario, I've been thinking that as the system criticality increases, I would create a new bucket (with a higher n_val) and then start pushing newer conversations on to that bucket. Still not sure how this would behave, but let me test further with bucket types as you suggest... Do let know please if there's something glaring I'm missing as am trying to clarify the thought-process to myself as well!!! Cheers. On Fri, Sep 30, 2016 at 12:07 PM, Luke Bakken wrote: > Hi Vikram, > > If all of your buckets use the same bucket type with your custom > n_val, there won't be a performance issue. Just be sure to set n_val > on the bucket type, and that all buckets are part of that bucket type. > > http://docs.basho.com/riak/kv/2.1.4/developing/usage/bucket-types/ > > -- > Luke Bakken > Engineer > lbak...@basho.com > > On Thu, Sep 29, 2016 at 4:42 PM, Vikram Lalit > wrote: > > Hi - I am creating a messaging platform wherein am modeling each topic to > > serve as a separate bucket. That means there can potentially be millions > of > > buckets, with each message from a user becoming a value on a distinct > > timestamp key. > > > > My question is there any downside to modeling my data in such a manner? > Or > > can folks advise a better way of storing the same in Riak? > > > > Secondly, I would like to modify the default bucket properties (n_val) - > I > > understand that such 'custom' buckets have a higher performance overhead > due > > to the extra load on the gossip protocol. Is there a way the default > n_val > > of newly created buckets be changed so that even if I have the above said > > high number of buckets, there is no performance degrade? Believe there > was > > such a config allowed in app.config but not sure that file is leveraged > any > > more after riak.conf was introduced. > > > > Thanks much. > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
High number of Riak buckets
Hi - I am creating a messaging platform wherein am modeling each topic to serve as a separate bucket. That means there can potentially be millions of buckets, with each message from a user becoming a value on a distinct timestamp key. My question is there any downside to modeling my data in such a manner? Or can folks advise a better way of storing the same in Riak? Secondly, I would like to modify the default bucket properties (n_val) - I understand that such 'custom' buckets have a higher performance overhead due to the extra load on the gossip protocol. Is there a way the default n_val of newly created buckets be changed so that even if I have the above said high number of buckets, there is no performance degrade? Believe there was such a config allowed in app.config but not sure that file is leveraged any more after riak.conf was introduced. Thanks much. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Playing with / understanding Riak configurations
Thanks Tom... On Wed, Jul 27, 2016 at 6:27 PM, Tom Santero wrote: > Vikram, > > I apologize, I initially just skimmed your question and thought you were > asking something entirely different. > > While increasing your N value is safe, decreasing it on a bucket with > pre-existing data, as you have, is not-recommended and the source of your > inconsistent results. > > Tom > > On Wed, Jul 27, 2016 at 4:18 PM, Vikram Lalit > wrote: > >> Thanks Tom... Yes I did read that but I couldn't deduce the outcome if n >> is decreased. John talks about data loss, but am actually observing a >> different result... perhaps am missing something! >> >> >> On Wed, Jul 27, 2016 at 6:11 PM, Tom Santero wrote: >> >>> Vikram, >>> >>> John Daily wrote a fantastic blog series that places your question in >>> context and then answers it. >>> >>> >>> http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ >>> >>> Tom >>> >>> On Wed, Jul 27, 2016 at 4:07 PM, Vikram Lalit >>> wrote: >>> >>>> Hi - I have a Riak node with n_val=3, r=2, w=2 and have just one >>>> key-object stored there-in. I'm trying to test various configurations to >>>> better understand the system and have the following observations - some >>>> dont seem to align with my understanding so far, so appreciate if someone >>>> can throw some light please... Thanks! >>>> >>>> 1. n=3, r=2, w=2: Base state, 1 key-value pair. >>>> >>>> 2. Change to n=2, r=2, w=2: When I query from my client, I randomly see >>>> 1 or 2 values being fetched. In fact, the number of keys fetched is 1 or 2, >>>> randomly changing each time the client queries the db. Ideally, I would >>>> have expected that if we reduce the n_val, there would be data loss from >>>> one of the vnodes. And that for this scenario, I would still expect only 1 >>>> (remaining) key-value pair to be read from the remaining two vnodes that >>>> has the data. Note that I dont intend to make such a change in production >>>> as cognizant of the recommendation to never decrease the value of n, but >>>> have done so only to test out the details. >>>> >>>> 3. Then change to n=2, r=1, w=1: I get the same alternating result as >>>> above, i.e. 1 or 2 values being fetched. >>>> >>>> 4. Then change to n=1, r=1, w=1: I get 3 key-value pairs, all >>>> identical, from the database. Again, are these all siblings? >>>> >>>> ___ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Playing with / understanding Riak configurations
Thanks Tom... Yes I did read that but I couldn't deduce the outcome if n is decreased. John talks about data loss, but am actually observing a different result... perhaps am missing something! On Wed, Jul 27, 2016 at 6:11 PM, Tom Santero wrote: > Vikram, > > John Daily wrote a fantastic blog series that places your question in > context and then answers it. > > > http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ > > Tom > > On Wed, Jul 27, 2016 at 4:07 PM, Vikram Lalit > wrote: > >> Hi - I have a Riak node with n_val=3, r=2, w=2 and have just one >> key-object stored there-in. I'm trying to test various configurations to >> better understand the system and have the following observations - some >> dont seem to align with my understanding so far, so appreciate if someone >> can throw some light please... Thanks! >> >> 1. n=3, r=2, w=2: Base state, 1 key-value pair. >> >> 2. Change to n=2, r=2, w=2: When I query from my client, I randomly see 1 >> or 2 values being fetched. In fact, the number of keys fetched is 1 or 2, >> randomly changing each time the client queries the db. Ideally, I would >> have expected that if we reduce the n_val, there would be data loss from >> one of the vnodes. And that for this scenario, I would still expect only 1 >> (remaining) key-value pair to be read from the remaining two vnodes that >> has the data. Note that I dont intend to make such a change in production >> as cognizant of the recommendation to never decrease the value of n, but >> have done so only to test out the details. >> >> 3. Then change to n=2, r=1, w=1: I get the same alternating result as >> above, i.e. 1 or 2 values being fetched. >> >> 4. Then change to n=1, r=1, w=1: I get 3 key-value pairs, all identical, >> from the database. Again, are these all siblings? >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Playing with / understanding Riak configurations
Hi - I have a Riak node with n_val=3, r=2, w=2 and have just one key-object stored there-in. I'm trying to test various configurations to better understand the system and have the following observations - some dont seem to align with my understanding so far, so appreciate if someone can throw some light please... Thanks! 1. n=3, r=2, w=2: Base state, 1 key-value pair. 2. Change to n=2, r=2, w=2: When I query from my client, I randomly see 1 or 2 values being fetched. In fact, the number of keys fetched is 1 or 2, randomly changing each time the client queries the db. Ideally, I would have expected that if we reduce the n_val, there would be data loss from one of the vnodes. And that for this scenario, I would still expect only 1 (remaining) key-value pair to be read from the remaining two vnodes that has the data. Note that I dont intend to make such a change in production as cognizant of the recommendation to never decrease the value of n, but have done so only to test out the details. 3. Then change to n=2, r=1, w=1: I get the same alternating result as above, i.e. 1 or 2 values being fetched. 4. Then change to n=1, r=1, w=1: I get 3 key-value pairs, all identical, from the database. Again, are these all siblings? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Recovering Riak data if it can no longer load in memory
Many thanks again, Matthew... this is very educative and helpful...! I'm now giving a shot with higher performance nodes... Thanks again! On Tue, Jul 12, 2016 at 4:18 PM, Matthew Von-Maszewski wrote: > You can further reduce memory used by leveldb with the following setting > in riak.conf: > > leveldb.threads = 5 > > The value "5" needs to be a prime number. The system defaults to 71. > Many Linux implementations will allocate 8Mbytes per thread for stack. So > bunches of threads lead to bunches of memory reserved for stack. That is > fine on servers with higher memory. But probably part of your problem on a > small memory machine. > > The thread count is high to promote parallelism across vnodes on the same > server, especially with "entropy = active". So again, this setting is > sacrificing performance to save memory. > > Matthew > > P.S. You really want 8 CPU cores, 4 as a dirt minimum. And review this > for more cpu performance info: > > https://github.com/basho/leveldb/wiki/riak-tuning-2 > > > > On Jul 12, 2016, at 4:04 PM, Vikram Lalit wrote: > > Thanks much Matthew. Yes the server is low-memory given only development > right now - I'm using an AWS micro instance, so 1 GB RAM and 1 vCPU. > > Thanks for the tip - let me try move the manifest file to a larger > instance and see how that works. More than reducing the memory footprint in > dev, my concern was more around reacting to a possible production scenario > where the db stops responding due to memory overload. Understood now that > moving to a larger instance should be possible. Thanks again. > > On Tue, Jul 12, 2016 at 12:26 PM, Matthew Von-Maszewski < > matth...@basho.com> wrote: > >> It would be helpful if you described the physical characteristics of the >> servers: memory size, logical cpu count, etc. >> >> Google created leveldb to be highly reliable in the face of crashes. If >> it is not restarting, that suggests to me that you have a low memory >> condition that is not able to load leveldb's MANIFEST file. That is easily >> fixed by moving the dataset to a machine with larger memory. >> >> There is also a special flag to reduce Riak's leveldb memory foot print >> during development work. The setting reduces the leveldb performance, but >> lets you run with less memory. >> >> In riak.conf, set: >> >> leveldb.limited_developer_mem = true >> >> Matthew >> >> >> > On Jul 12, 2016, at 11:56 AM, Vikram Lalit >> wrote: >> > >> > Hi - I've been testing a Riak cluster (of 3 nodes) with an ejabberd >> messaging cluster in front of it that writes data to the Riak nodes. Whilst >> load testing the platform (by creating 0.5 million ejabberd users via >> Tsung), I found that the Riak nodes suddenly crashed. My question is how do >> we recover from such a situation if it were to occur in production? >> > >> > To provide further context / details, the leveldb log files storing the >> data suddenly became too huge, thus making the AWS Riak instances not able >> to load them in memory anymore. So we get a core dump if 'riak start' is >> fired on those instances. I had an n_val = 2, and all 3 nodes went down >> almost simultaneously, so in such a scenario, we cannot even rely on a 2nd >> copy of the data. One way to of course prevent it in the first place would >> be to use auto-scaling, but I'm wondering is there a ex post facto / post >> the event recovery that can be performed in such a scenario? Is it possible >> to simply copy the leveldb data to a larger memory instance, or to curtail >> the data further to allow loading in the same instance? >> > >> > Appreciate if you can provide inputs - a tad concerned as to how we >> could recover from such a situation if it were to happen in production >> (apart from leveraging auto-scaling as a preventive measure). >> > >> > Thanks! >> > >> > ___ >> > riak-users mailing list >> > riak-users@lists.basho.com >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Recovering Riak data if it can no longer load in memory
Thanks much Matthew. Yes the server is low-memory given only development right now - I'm using an AWS micro instance, so 1 GB RAM and 1 vCPU. Thanks for the tip - let me try move the manifest file to a larger instance and see how that works. More than reducing the memory footprint in dev, my concern was more around reacting to a possible production scenario where the db stops responding due to memory overload. Understood now that moving to a larger instance should be possible. Thanks again. On Tue, Jul 12, 2016 at 12:26 PM, Matthew Von-Maszewski wrote: > It would be helpful if you described the physical characteristics of the > servers: memory size, logical cpu count, etc. > > Google created leveldb to be highly reliable in the face of crashes. If > it is not restarting, that suggests to me that you have a low memory > condition that is not able to load leveldb's MANIFEST file. That is easily > fixed by moving the dataset to a machine with larger memory. > > There is also a special flag to reduce Riak's leveldb memory foot print > during development work. The setting reduces the leveldb performance, but > lets you run with less memory. > > In riak.conf, set: > > leveldb.limited_developer_mem = true > > Matthew > > > > On Jul 12, 2016, at 11:56 AM, Vikram Lalit > wrote: > > > > Hi - I've been testing a Riak cluster (of 3 nodes) with an ejabberd > messaging cluster in front of it that writes data to the Riak nodes. Whilst > load testing the platform (by creating 0.5 million ejabberd users via > Tsung), I found that the Riak nodes suddenly crashed. My question is how do > we recover from such a situation if it were to occur in production? > > > > To provide further context / details, the leveldb log files storing the > data suddenly became too huge, thus making the AWS Riak instances not able > to load them in memory anymore. So we get a core dump if 'riak start' is > fired on those instances. I had an n_val = 2, and all 3 nodes went down > almost simultaneously, so in such a scenario, we cannot even rely on a 2nd > copy of the data. One way to of course prevent it in the first place would > be to use auto-scaling, but I'm wondering is there a ex post facto / post > the event recovery that can be performed in such a scenario? Is it possible > to simply copy the leveldb data to a larger memory instance, or to curtail > the data further to allow loading in the same instance? > > > > Appreciate if you can provide inputs - a tad concerned as to how we > could recover from such a situation if it were to happen in production > (apart from leveraging auto-scaling as a preventive measure). > > > > Thanks! > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Recovering Riak data if it can no longer load in memory
Hi - I've been testing a Riak cluster (of 3 nodes) with an ejabberd messaging cluster in front of it that writes data to the Riak nodes. Whilst load testing the platform (by creating 0.5 million ejabberd users via Tsung), I found that the Riak nodes suddenly crashed. My question is how do we recover from such a situation if it were to occur in production? To provide further context / details, the leveldb log files storing the data suddenly became too huge, thus making the AWS Riak instances not able to load them in memory anymore. So we get a core dump if 'riak start' is fired on those instances. I had an n_val = 2, and all 3 nodes went down almost simultaneously, so in such a scenario, we cannot even rely on a 2nd copy of the data. One way to of course prevent it in the first place would be to use auto-scaling, but I'm wondering is there a ex post facto / post the event recovery that can be performed in such a scenario? Is it possible to simply copy the leveldb data to a larger memory instance, or to curtail the data further to allow loading in the same instance? Appreciate if you can provide inputs - a tad concerned as to how we could recover from such a situation if it were to happen in production (apart from leveraging auto-scaling as a preventive measure). Thanks! ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Cluster Behavior - Clarification
Hi Alexander - thanks again for your inputs here. I believe the problem here was with the sloppy quorum coming in when I brought down 1 node. The failover node would get active and from what I understand from the documentation, since the failover did not have the key and could potentially respond faster than the fetch, I randomly failed to fetch any data till all the read repairs completed. Understand this much better now - many thanks once again... On Tue, May 24, 2016 at 4:24 PM, Alexander Sicular wrote: > Hi Vikram, > > If you're using the defaults, two of copies may be on the same > machine. When using the default values (ring_size=64, n_val=3) you are > not guaranteed copies on distinct physical machines. Implement a > back-off retry design pattern. aka, fail once, try again with r=1. > Also, a read will trigger a read repair operation which will then copy > your data n_val times to surviving members of the cluster. > > Have you tried that? > -Alexander > > Read these blog posts for more info: > > http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ > http://basho.com/posts/technical/riaks-config-behaviors-part-2/ > http://basho.com/posts/technical/riaks-config-behaviors-part-3/ > http://basho.com/posts/technical/riaks-config-behaviors-part-4/ > > On Tue, May 24, 2016 at 3:08 PM, Vikram Lalit > wrote: > > It's returning no object at all for the relevant key. That too is random > - > > every few calls it returns but then it doesn't. > > > > On May 24, 2016 4:06 PM, "Sargun Dhillon" wrote: > >> > >> What do you mean it's not returning? It's returning stale data? Or > >> it's erroring? > >> > >> On Tue, May 24, 2016 at 7:34 AM, Vikram Lalit > >> wrote: > >> > Hi - I'd appreciate if someone can opine on the below behavior of Riak > >> > that > >> > I am observing... is that expected, or something wrong in my set-up / > >> > understanding? > >> > > >> > To summarize, I have a 3-node Riak cluster (separate EC2 AWS > instances) > >> > with > >> > a separate chat server connecting to them. When I write data on the > Riak > >> > nodes, the process is successful and I can read all data correctly. > >> > However, > >> > as part of my testing, if I deliberately bring down one node (and then > >> > remove it from the cluster using riak-admin cluster force-remove / > plan > >> > / > >> > commit), the client API is not able to fetch all the written data. In > >> > fact, > >> > there is an alternation of success and failure which happens rather > >> > randomly. > >> > > >> > My initial suspicion was that it would be happening only during the > time > >> > the > >> > rebalancing is occurring (i.e. riak-admin ring-status is not fully > >> > settled) > >> > but I've seen this sporadic behavior post the same too. > >> > > >> > Does this have to do with the n and r values for the cluster and given > >> > that > >> > 1 node is down, the cluster does not succeed in returning results > >> > reliably? > >> > Also, does this mean that during the time a cluster is being > rebalanced > >> > (even incl. addition of new nodes), the results could be arbitrary - > >> > that > >> > doesn't sound correct to me? > >> > > >> > Appreciate if someone can throw some light here? Also, the HTTP API > >> > calls to > >> > retrieve and set the n / r / w values for a specific bucket - couldn't > >> > locate the same! > >> > > >> > Thanks much! > >> > Vikram > >> > > >> > > >> > ___ > >> > riak-users mailing list > >> > riak-users@lists.basho.com > >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >> > > > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Cluster Behavior - Clarification
Thanks again DeadZen... I believe it was indeed related to the sloppy quorum kicking in when one node was brought down. Playing with the node configs has helped me further test out the desired state... thanks again! On Tue, May 24, 2016 at 4:27 PM, DeadZen wrote: > Not to jump to conclusions, but this sounds like a quorum issue. > Such as your writes are not going to all three servers, but only to one or > two. > Handoffs would occur when nodes went down unexpectedly, it doesn't > ever attempt to fetch unwritten data, it fetches data from other > servers which were still up at the point the server went down.. > There are some configuration values you could experiment with, that > will ok a write only as long as its gone to a durable storage count.. > But keep in mind theres no guarantee vnodes are assigned to unique > servers.. > > > On Tue, May 24, 2016 at 4:08 PM, Vikram Lalit > wrote: > > It's returning no object at all for the relevant key. That too is random > - > > every few calls it returns but then it doesn't. > > > > On May 24, 2016 4:06 PM, "Sargun Dhillon" wrote: > >> > >> What do you mean it's not returning? It's returning stale data? Or > >> it's erroring? > >> > >> On Tue, May 24, 2016 at 7:34 AM, Vikram Lalit > >> wrote: > >> > Hi - I'd appreciate if someone can opine on the below behavior of Riak > >> > that > >> > I am observing... is that expected, or something wrong in my set-up / > >> > understanding? > >> > > >> > To summarize, I have a 3-node Riak cluster (separate EC2 AWS > instances) > >> > with > >> > a separate chat server connecting to them. When I write data on the > Riak > >> > nodes, the process is successful and I can read all data correctly. > >> > However, > >> > as part of my testing, if I deliberately bring down one node (and then > >> > remove it from the cluster using riak-admin cluster force-remove / > plan > >> > / > >> > commit), the client API is not able to fetch all the written data. In > >> > fact, > >> > there is an alternation of success and failure which happens rather > >> > randomly. > >> > > >> > My initial suspicion was that it would be happening only during the > time > >> > the > >> > rebalancing is occurring (i.e. riak-admin ring-status is not fully > >> > settled) > >> > but I've seen this sporadic behavior post the same too. > >> > > >> > Does this have to do with the n and r values for the cluster and given > >> > that > >> > 1 node is down, the cluster does not succeed in returning results > >> > reliably? > >> > Also, does this mean that during the time a cluster is being > rebalanced > >> > (even incl. addition of new nodes), the results could be arbitrary - > >> > that > >> > doesn't sound correct to me? > >> > > >> > Appreciate if someone can throw some light here? Also, the HTTP API > >> > calls to > >> > retrieve and set the n / r / w values for a specific bucket - couldn't > >> > locate the same! > >> > > >> > Thanks much! > >> > Vikram > >> > > >> > > >> > ___ > >> > riak-users mailing list > >> > riak-users@lists.basho.com > >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >> > > > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak not storing data in a round-robin manner across instances
Thanks very much Sam... much appreciated. On Fri, May 27, 2016 at 4:55 PM, Vikram Lalit wrote: > Thanks Alexander ... makes complete sense... > > On Fri, May 27, 2016 at 4:39 PM, Alexander Sicular > wrote: > >> There is a reason Basho's minimum production deployment recommendation is >> 5 machines. It is to ensure that, when operating with default settings, >> each replica of any key is stored on distinct physical nodes. It has to do >> with the allocation of virtual nodes to physical machines and replica sets. >> Replica sets are sets of virtual nodes, not physical nodes. >> >> When using 3 or 4 machines in your cluster reduce your n_val to 2. >> >> Cheers, Alexander >> >> On Friday, May 27, 2016, DeadZen wrote: >> >>> np, number of nodes and ring size play a lot into that. >>> as does your r,w settings. >>> might be fun to create a visualization one day ;) >>> >>> On Friday, May 27, 2016, Vikram Lalit wrote: >>> >>>> Ok got it thanks. >>>> On May 27, 2016 4:02 PM, "DeadZen" wrote: >>>> >>>>> reiterating my last email >>>>> >>>>> > theres no guarantee vnodes are assigned to unique servers.. >>>>> >>>>> referencing the docs >>>>> >>>>> > Nodes *attempt* to claim their partitions at intervals around the >>>>> ring such that there is an even distribution amongst the member nodes and >>>>> that no node is responsible for more than one replica of a key. >>>>> >>>>> >>>>> >>>>> On Friday, May 27, 2016, Vikram Lalit wrote: >>>>> >>>>>> I have a Riak cluster (of 3 nodes, with 64 partitions and n_val = 3) >>>>>> but I find that for some objects, their hosting partitions / vnodes are >>>>>> not >>>>>> spread out across the 3 nodes. In some cases, 2 of them are on 1 node and >>>>>> the third is on a second node. That runs contrary to my understanding >>>>>> (link >>>>>> here: http://docs.basho.com/riak/kv/2.1.4/learn/concepts/clusters/) >>>>>> that the data is spread out across partitions in such a way that the >>>>>> partitions are on different servers. Is there something I'm missing here >>>>>> please in terms of how Riak works? Thanks... >>>>>> >>>>>> >> >> -- >> >> >> Alexander Sicular >> Solutions Architect >> Basho Technologies >> 9175130679 >> @siculars >> >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak not storing data in a round-robin manner across instances
Thanks Alexander ... makes complete sense... On Fri, May 27, 2016 at 4:39 PM, Alexander Sicular wrote: > There is a reason Basho's minimum production deployment recommendation is > 5 machines. It is to ensure that, when operating with default settings, > each replica of any key is stored on distinct physical nodes. It has to do > with the allocation of virtual nodes to physical machines and replica sets. > Replica sets are sets of virtual nodes, not physical nodes. > > When using 3 or 4 machines in your cluster reduce your n_val to 2. > > Cheers, Alexander > > On Friday, May 27, 2016, DeadZen wrote: > >> np, number of nodes and ring size play a lot into that. >> as does your r,w settings. >> might be fun to create a visualization one day ;) >> >> On Friday, May 27, 2016, Vikram Lalit wrote: >> >>> Ok got it thanks. >>> On May 27, 2016 4:02 PM, "DeadZen" wrote: >>> >>>> reiterating my last email >>>> >>>> > theres no guarantee vnodes are assigned to unique servers.. >>>> >>>> referencing the docs >>>> >>>> > Nodes *attempt* to claim their partitions at intervals around the >>>> ring such that there is an even distribution amongst the member nodes and >>>> that no node is responsible for more than one replica of a key. >>>> >>>> >>>> >>>> On Friday, May 27, 2016, Vikram Lalit wrote: >>>> >>>>> I have a Riak cluster (of 3 nodes, with 64 partitions and n_val = 3) >>>>> but I find that for some objects, their hosting partitions / vnodes are >>>>> not >>>>> spread out across the 3 nodes. In some cases, 2 of them are on 1 node and >>>>> the third is on a second node. That runs contrary to my understanding >>>>> (link >>>>> here: http://docs.basho.com/riak/kv/2.1.4/learn/concepts/clusters/) >>>>> that the data is spread out across partitions in such a way that the >>>>> partitions are on different servers. Is there something I'm missing here >>>>> please in terms of how Riak works? Thanks... >>>>> >>>>> > > -- > > > Alexander Sicular > Solutions Architect > Basho Technologies > 9175130679 > @siculars > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak not storing data in a round-robin manner across instances
Yes am now beginning to realize the configs are much more important than I originally thought! Thanks again!!! On Fri, May 27, 2016 at 4:28 PM, DeadZen wrote: > np, number of nodes and ring size play a lot into that. > as does your r,w settings. > might be fun to create a visualization one day ;) > > On Friday, May 27, 2016, Vikram Lalit wrote: > >> Ok got it thanks. >> On May 27, 2016 4:02 PM, "DeadZen" wrote: >> >>> reiterating my last email >>> >>> > theres no guarantee vnodes are assigned to unique servers.. >>> >>> referencing the docs >>> >>> > Nodes *attempt* to claim their partitions at intervals around the >>> ring such that there is an even distribution amongst the member nodes and >>> that no node is responsible for more than one replica of a key. >>> >>> >>> >>> On Friday, May 27, 2016, Vikram Lalit wrote: >>> >>>> I have a Riak cluster (of 3 nodes, with 64 partitions and n_val = 3) >>>> but I find that for some objects, their hosting partitions / vnodes are not >>>> spread out across the 3 nodes. In some cases, 2 of them are on 1 node and >>>> the third is on a second node. That runs contrary to my understanding (link >>>> here: http://docs.basho.com/riak/kv/2.1.4/learn/concepts/clusters/) >>>> that the data is spread out across partitions in such a way that the >>>> partitions are on different servers. Is there something I'm missing here >>>> please in terms of how Riak works? Thanks... >>>> >>>> ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak not storing data in a round-robin manner across instances
Ok got it thanks. On May 27, 2016 4:02 PM, "DeadZen" wrote: > reiterating my last email > > > theres no guarantee vnodes are assigned to unique servers.. > > referencing the docs > > > Nodes *attempt* to claim their partitions at intervals around the ring > such that there is an even distribution amongst the member nodes and that > no node is responsible for more than one replica of a key. > > > > On Friday, May 27, 2016, Vikram Lalit wrote: > >> I have a Riak cluster (of 3 nodes, with 64 partitions and n_val = 3) but >> I find that for some objects, their hosting partitions / vnodes are not >> spread out across the 3 nodes. In some cases, 2 of them are on 1 node and >> the third is on a second node. That runs contrary to my understanding (link >> here: http://docs.basho.com/riak/kv/2.1.4/learn/concepts/clusters/) that >> the data is spread out across partitions in such a way that the partitions >> are on different servers. Is there something I'm missing here please in >> terms of how Riak works? Thanks... >> >> ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak not storing data in a round-robin manner across instances
I have a Riak cluster (of 3 nodes, with 64 partitions and n_val = 3) but I find that for some objects, their hosting partitions / vnodes are not spread out across the 3 nodes. In some cases, 2 of them are on 1 node and the third is on a second node. That runs contrary to my understanding (link here: http://docs.basho.com/riak/kv/2.1.4/learn/concepts/clusters/) that the data is spread out across partitions in such a way that the partitions are on different servers. Is there something I'm missing here please in terms of how Riak works? Thanks... ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Cluster Behavior - Clarification
It's returning no object at all for the relevant key. That too is random - every few calls it returns but then it doesn't. On May 24, 2016 4:06 PM, "Sargun Dhillon" wrote: > What do you mean it's not returning? It's returning stale data? Or > it's erroring? > > On Tue, May 24, 2016 at 7:34 AM, Vikram Lalit > wrote: > > Hi - I'd appreciate if someone can opine on the below behavior of Riak > that > > I am observing... is that expected, or something wrong in my set-up / > > understanding? > > > > To summarize, I have a 3-node Riak cluster (separate EC2 AWS instances) > with > > a separate chat server connecting to them. When I write data on the Riak > > nodes, the process is successful and I can read all data correctly. > However, > > as part of my testing, if I deliberately bring down one node (and then > > remove it from the cluster using riak-admin cluster force-remove / plan / > > commit), the client API is not able to fetch all the written data. In > fact, > > there is an alternation of success and failure which happens rather > > randomly. > > > > My initial suspicion was that it would be happening only during the time > the > > rebalancing is occurring (i.e. riak-admin ring-status is not fully > settled) > > but I've seen this sporadic behavior post the same too. > > > > Does this have to do with the n and r values for the cluster and given > that > > 1 node is down, the cluster does not succeed in returning results > reliably? > > Also, does this mean that during the time a cluster is being rebalanced > > (even incl. addition of new nodes), the results could be arbitrary - that > > doesn't sound correct to me? > > > > Appreciate if someone can throw some light here? Also, the HTTP API > calls to > > retrieve and set the n / r / w values for a specific bucket - couldn't > > locate the same! > > > > Thanks much! > > Vikram > > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Cluster Behavior - Clarification
Hi - I'd appreciate if someone can opine on the below behavior of Riak that I am observing... is that expected, or something wrong in my set-up / understanding? To summarize, I have a 3-node Riak cluster (separate EC2 AWS instances) with a separate chat server connecting to them. When I write data on the Riak nodes, the process is successful and I can read all data correctly. However, as part of my testing, if I deliberately bring down one node (and then remove it from the cluster using riak-admin cluster force-remove / plan / commit), the client API is not able to fetch all the written data. In fact, there is an alternation of success and failure which happens rather randomly. My initial suspicion was that it would be happening only during the time the rebalancing is occurring (i.e. riak-admin ring-status is not fully settled) but I've seen this sporadic behavior post the same too. Does this have to do with the n and r values for the cluster and given that 1 node is down, the cluster does not succeed in returning results reliably? Also, does this mean that during the time a cluster is being rebalanced (even incl. addition of new nodes), the results could be arbitrary - that doesn't sound correct to me? Appreciate if someone can throw some light here? Also, the HTTP API calls to retrieve and set the n / r / w values for a specific bucket - couldn't locate the same! Thanks much! Vikram ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
