Re: Waiting for riak to handoff ?
Here is some additional information, it seems the ownership handoff is waiting, see below: riak-admin ring-status == Claimant === Claimant: 'riak@oldnode1' Status: up Ring Ready: true == Ownership Handoff == Owner: riak@oldnode2 Next Owner: riak@oldnode3 Index: 68507889249886074290797726533575766546371837952 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 342539446249430371453988632667878832731859189760 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 616571003248974668617179538802181898917346541568 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 890602560248518965780370444936484965102833893376 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 1164634117248063262943561351070788031288321245184 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 1438665674247607560106752257205091097473808596992 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] --- Owner: riak@oldnode2 Next Owner: riak@oldnode1 Index: 159851741583067506678528028578343455274867621888 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 433883298582611803841718934712646521460354973696 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 707914855582156101004909840846949587645842325504 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 981946412581700398168100746981252653831329677312 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 125597796958124469533129165311720016817029120 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] --- == Unreachable Nodes == All nodes are up and reachable 2014-08-20 17:02 GMT+02:00 Marcel Koopman marcel.koop...@gmail.com: Hello, It seems that adding new nodes does not have any effect. Why it Riak waiting for transfers? When does it start handoff the new nodes? riak-admin member-status = Membership == Status RingPendingNode --- valid 25.0% 34.4%'riak@oldnode1' valid 25.0% 32.8%'riak@oldnode2' valid 50.0% 32.8%'riak@oldnode3 valid 0.0% 0.0%'riak@newnode1' valid 0.0% 0.0%'riak@newnode2' valid 0.0% 0.0%'riak@newnode3' valid 0.0% 0.0%'riak@newnode4' valid 0.0% 0.0%'riak@newnode5' --- Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 riak-admin transfers 'riak@newnode5' waiting to handoff 11 partitions 'riak@newnode4' waiting to handoff 11 partitions 'riak@newnode3' waiting to handoff 11 partitions 'riak@newnode2' waiting to handoff 11 partitions 'riak@newnode1' waiting to handoff 11 partitions 'riak@oldnode1' waiting to handoff 11 partitions 'riak@oldnode2' waiting to handoff 11 partitions Active Transfers: (nothing here) -- Met vriendelijke groet, Marcel Koopman -- Met vriendelijke groet, Marcel Koopman ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Waiting for riak to handoff ?
Hi Marcel - What version of Riak are you using? What is the output of riak-admin transfer-limit ? -- Luke Bakken CSE lbak...@basho.com On Wed, Aug 20, 2014 at 8:02 AM, Marcel Koopman marcel.koop...@gmail.com wrote: Hello, It seems that adding new nodes does not have any effect. Why it Riak waiting for transfers? When does it start handoff the new nodes? riak-admin member-status = Membership == Status RingPendingNode --- valid 25.0% 34.4%'riak@oldnode1' valid 25.0% 32.8%'riak@oldnode2' valid 50.0% 32.8%'riak@oldnode3 valid 0.0% 0.0%'riak@newnode1' valid 0.0% 0.0%'riak@newnode2' valid 0.0% 0.0%'riak@newnode3' valid 0.0% 0.0%'riak@newnode4' valid 0.0% 0.0%'riak@newnode5' --- Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 riak-admin transfers 'riak@newnode5' waiting to handoff 11 partitions 'riak@newnode4' waiting to handoff 11 partitions 'riak@newnode3' waiting to handoff 11 partitions 'riak@newnode2' waiting to handoff 11 partitions 'riak@newnode1' waiting to handoff 11 partitions 'riak@oldnode1' waiting to handoff 11 partitions 'riak@oldnode2' waiting to handoff 11 partitions Active Transfers: (nothing here) -- Met vriendelijke groet, Marcel Koopman ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Search VS other query systems
Re. Riak pipes. What's the latest regarding accessing the pipe framework? Haven't heard t much about it lately, admittedly haven't been listening t hard either. The thought would be to do stormish stream processing in situ. @siculars http://siculars.posthaven.com Sent from my iRotaryPhone On Aug 20, 2014, at 22:28, Sargun Dhillon sar...@sargun.me wrote: I second John's opinions. Generally, I would have have one key which is the secondary index, being an observe-remove OR-Set (or a relevant type for your application, be a register, g-set, or a plain old OR-set) pointing to back to the keys. Unfortunately, this mechanism can become quite unwieldy in when you have a term with a high cardinality. Now, moving onto the Twitter use case, you care a lot about the speed. With this strategy, if you're doing this from a client where you (1) read the 2i OR-set, and then (2) read the keys, that can be expensive as you have to read the entire 2i value back to the Riak client before reading any of the keys. An example, the hashtag #beiber, would have high cardinality would result in a super large value, and reading that back over the network would be less than awesome. Also, having to pass this value around disterl would be poor. Fortunately, the folks at Basho have invented riak_pipe. Riak_pipe is a method to allow running the read locally on the node the 2i lives on, and then streaming reads for all of those keys to the nodes that they live on, and then all back to the reader. It's actually the framework that Riak MR uses under the hood. Talk: https://vimeo.com/53910999 (there might be newer ones as well) Docs: https://github.com/basho/riak_pipe Also, to deal with high-cardinality values, there are a variety of work arounds, such as sharding the secondary index to some known set of keys, and doing a read across these list of keys. Also, you can postfix a nonce to the 2i-key, and ensure that they all end up on one node (custom hashing function), or a subset of nodes, and utilize leveldb's key iteration over a range to handle this. The general patterns I like for 2i atop Riak is to specialize Peter Bailis, from UC Berkeley's work for RAMPs. If you build the framework for this, it'll be all sorts of useful in the future. One warning is that there is no easy way to garbage collect in Riak today. Paper: http://www.bailis.org/papers/ramp-sigmod2014.pdf Talk: https://www.youtube.com/watch?v=_rAdJkAbGls None of these methods gracefully handle range queries. You can do clever things with your 2i to handle this, but it the Twitter use case wouldn't need ranges. On Wed, Aug 20, 2014 at 12:28 PM, John Daily jda...@basho.com wrote: I don't have benchmarks to discuss query performance for different tools at different sizes, but I'd like to point out that the ultimate search tool for Riak is to not search at all. Riak Search, 2i, MapReduce are all capable tools, but they don't scale nearly as well as straight key/value requests, and it is often possible to model your data around the latter. I covered this in https://basho.com/riak-development-anti-patterns/ and the next edition of Eric Redmond's Little Riak Book (http://littleriakbook.com) will have more discussion on the topic, but if at all possible, create your query results as reports as the data is ingested, instead of attempting to find it all later. -John On Wed, Aug 20, 2014 at 3:21 PM, Alex De la rosa alex.rosa@gmail.com wrote: Any thoughts about this? One thing it worries me about Riak Search is that if one index has several millions of object to search for maybe it becomes slow? 2i might be faster then? Thanks! Alex On Tue, Aug 19, 2014 at 8:47 AM, Alex De la rosa alex.rosa@gmail.com wrote: Hi there, I had been seeing lately Riak Search as an ultimate way to query Riak... and it seems recommended to use over MapReduce and even 2i... said so... should we try to always use Riak Search over the other systems? Is there any situation in which MapReduce could be a better approach than Riak Search? Same goes for 2i... I believe 2i is an optimal approach if you just want keys and know very well what are you looking for, but out of that, should Riak Search try to replace all 2i uses? Practical example: If you are twitter and want to get twits for the hashtag #Riak, what would be the best approach? 2i? Riak Search? MapReduce? Thanks! Alex ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com
Re: Riak Search VS other query systems
We have some prototypes for how to expand Pipe's capabilities thanks to Chris Meiklejohn. We have not exposed it directly and I honestly think it may be the fundamentally wrong level of abstraction to present to users. It could be a way to implement higher-level query/processing features, but that has not panned out and is low on our priorities at this time. On Thu, Aug 21, 2014 at 11:55 AM, Alexander Sicular sicul...@gmail.com wrote: Re. Riak pipes. What's the latest regarding accessing the pipe framework? Haven't heard t much about it lately, admittedly haven't been listening t hard either. The thought would be to do stormish stream processing in situ. @siculars http://siculars.posthaven.com Sent from my iRotaryPhone On Aug 20, 2014, at 22:28, Sargun Dhillon sar...@sargun.me wrote: I second John's opinions. Generally, I would have have one key which is the secondary index, being an observe-remove OR-Set (or a relevant type for your application, be a register, g-set, or a plain old OR-set) pointing to back to the keys. Unfortunately, this mechanism can become quite unwieldy in when you have a term with a high cardinality. Now, moving onto the Twitter use case, you care a lot about the speed. With this strategy, if you're doing this from a client where you (1) read the 2i OR-set, and then (2) read the keys, that can be expensive as you have to read the entire 2i value back to the Riak client before reading any of the keys. An example, the hashtag #beiber, would have high cardinality would result in a super large value, and reading that back over the network would be less than awesome. Also, having to pass this value around disterl would be poor. Fortunately, the folks at Basho have invented riak_pipe. Riak_pipe is a method to allow running the read locally on the node the 2i lives on, and then streaming reads for all of those keys to the nodes that they live on, and then all back to the reader. It's actually the framework that Riak MR uses under the hood. Talk: https://vimeo.com/53910999 (there might be newer ones as well) Docs: https://github.com/basho/riak_pipe Also, to deal with high-cardinality values, there are a variety of work arounds, such as sharding the secondary index to some known set of keys, and doing a read across these list of keys. Also, you can postfix a nonce to the 2i-key, and ensure that they all end up on one node (custom hashing function), or a subset of nodes, and utilize leveldb's key iteration over a range to handle this. The general patterns I like for 2i atop Riak is to specialize Peter Bailis, from UC Berkeley's work for RAMPs. If you build the framework for this, it'll be all sorts of useful in the future. One warning is that there is no easy way to garbage collect in Riak today. Paper: http://www.bailis.org/papers/ramp-sigmod2014.pdf Talk: https://www.youtube.com/watch?v=_rAdJkAbGls None of these methods gracefully handle range queries. You can do clever things with your 2i to handle this, but it the Twitter use case wouldn't need ranges. On Wed, Aug 20, 2014 at 12:28 PM, John Daily jda...@basho.com wrote: I don't have benchmarks to discuss query performance for different tools at different sizes, but I'd like to point out that the ultimate search tool for Riak is to not search at all. Riak Search, 2i, MapReduce are all capable tools, but they don't scale nearly as well as straight key/value requests, and it is often possible to model your data around the latter. I covered this in https://basho.com/riak-development-anti-patterns/ and the next edition of Eric Redmond's Little Riak Book (http://littleriakbook.com) will have more discussion on the topic, but if at all possible, create your query results as reports as the data is ingested, instead of attempting to find it all later. -John On Wed, Aug 20, 2014 at 3:21 PM, Alex De la rosa alex.rosa@gmail.com wrote: Any thoughts about this? One thing it worries me about Riak Search is that if one index has several millions of object to search for maybe it becomes slow? 2i might be faster then? Thanks! Alex On Tue, Aug 19, 2014 at 8:47 AM, Alex De la rosa alex.rosa@gmail.com wrote: Hi there, I had been seeing lately Riak Search as an ultimate way to query Riak... and it seems recommended to use over MapReduce and even 2i... said so... should we try to always use Riak Search over the other systems? Is there any situation in which MapReduce could be a better approach than Riak Search? Same goes for 2i... I believe 2i is an optimal approach if you just want keys and know very well what are you looking for, but out of that, should Riak Search try to replace all 2i uses? Practical example: If you are twitter and want to get twits for the hashtag #Riak, what would be the best approach? 2i? Riak Search? MapReduce? Thanks! Alex ___ riak-users mailing list
Streaming keys with the Erlang client
Hi guys, I was wondering if streaming the keys using 2i would be possible. I guess any way of limiting the number of keys returned would be good. According to the documentation there few options: http://docs.basho.com/riak/latest/dev/using/2i/#Examples - streaming the keys - pagination with max_results Is any of these available in the Erlang client? Thank you in advance, Istvan -- the sun shines for all ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Streaming keys with the Erlang client
Let me answer my own question, yes all of these are supported in the Erlang client. The function I was looking for is get_index_eq: http://basho.github.io/riak-erlang-client/riakc_pb_socket.html#get_index_eq-5 Regards, Istvan On Thu, Aug 21, 2014 at 10:32 AM, István lecc...@gmail.com wrote: Hi guys, I was wondering if streaming the keys using 2i would be possible. I guess any way of limiting the number of keys returned would be good. According to the documentation there few options: http://docs.basho.com/riak/latest/dev/using/2i/#Examples - streaming the keys - pagination with max_results Is any of these available in the Erlang client? Thank you in advance, Istvan -- the sun shines for all -- the sun shines for all ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Streaming keys with the Erlang client
István, For future reference, the Riak 2.0 docs have client-library-specific code samples. The *Querying* section of this doc has Erlang samples, including an example for how to stream: http://docs.basho.com/riak/2.0.0/dev/using/2i/#Querying Luc On Thu, Aug 21, 2014 at 10:38 AM, István lecc...@gmail.com wrote: Let me answer my own question, yes all of these are supported in the Erlang client. The function I was looking for is get_index_eq: http://basho.github.io/riak-erlang-client/riakc_pb_socket.html#get_index_eq-5 Regards, Istvan On Thu, Aug 21, 2014 at 10:32 AM, István lecc...@gmail.com wrote: Hi guys, I was wondering if streaming the keys using 2i would be possible. I guess any way of limiting the number of keys returned would be good. According to the documentation there few options: http://docs.basho.com/riak/latest/dev/using/2i/#Examples - streaming the keys - pagination with max_results Is any of these available in the Erlang client? Thank you in advance, Istvan -- the sun shines for all -- the sun shines for all ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Avoid siblings in data-type bucket
Hi there,
I was trying to create a bucket-type using the datatype MAP and it didn't
allow me to create it with allow_mult:false:
# riak-admin bucket-type create tp_images
'{props:{allow_mult:false,search_index:ix_images,datatype:map}}'Error
creating bucket type tp_images:Data Type buckets must be allow_mult=true
How to avoid siblings in this bucket then?
Thanks,Alex
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Avoid siblings in data-type bucket
Siblings are resolved automatically by Riak when using our data types, thus
the requirement that allow_mult=true.
-John
On Thu, Aug 21, 2014 at 3:17 PM, Alex De la rosa alex.rosa@gmail.com
wrote:
Hi there,
I was trying to create a bucket-type using the datatype MAP and it didn't
allow me to create it with allow_mult:false:
# riak-admin bucket-type create tp_images
'{props:{allow_mult:false,search_index:ix_images,datatype:map}}'Error
creating bucket type tp_images:Data Type buckets must be allow_mult=true
How to avoid siblings in this bucket then?
Thanks,Alex
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Avoid siblings in data-type bucket
Cool, thank you very much
Alex
On Thu, Aug 21, 2014 at 9:18 PM, John Daily jda...@basho.com wrote:
Siblings are resolved automatically by Riak when using our data types,
thus the requirement that allow_mult=true.
-John
On Thu, Aug 21, 2014 at 3:17 PM, Alex De la rosa alex.rosa@gmail.com
wrote:
Hi there,
I was trying to create a bucket-type using the datatype MAP and it didn't
allow me to create it with allow_mult:false:
# riak-admin bucket-type create tp_images
'{props:{allow_mult:false,search_index:ix_images,datatype:map}}'Error
creating bucket type tp_images:Data Type buckets must be allow_mult=true
How to avoid siblings in this bucket then?
Thanks,Alex
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
