Re: Docker support

[Sargun Dhillon]

I would discourage running Riak in Docker. If you use Docker in bridge
mode, then it becomes fairly difficult to deal with networking across
machines. If you run it in host mode, you run into issues with epmd in
the host network namespace. There are some workarounds to this, like
using third party networking solutions or swarm.

I'm curious though, for me to answer this question fully, what aspects
of Docker are you interested in using? Docker has evolved to be quite
the beast with dozens of different flags, levers, and switches. Which
ones are you planning on flipping?

On Wed, Jul 13, 2016 at 11:33 AM, Adam Winter
 wrote:
> Is a Docker-based deployment of Riak officially supported by Basho and
> encouraged?
>
> I see articles of how to configure Riak to run in Docker images, but no
> explicit mention of it within the list of supported platforms.
>
>
> Adam Winter
> 614-446-9033
> Delivery Lead
> Pillar Technology Group, LLC
> awin...@pillartechnology.com
>
> Interact with me on:
> LinkedIn | Twitter
>
>
>
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: How to cold (re)boot a cluster with already existing node data

[Sargun Dhillon]

Two suggestions:
1. Use Riak-EE, and have two rings. When you do an update, copy over one
ring to the other side after you do a "cold reboot"
2. Use the Riak Mesos Framework. Mesos is like K8s, but it has stateful
storage primitives. (Link: https://github.com/basho-labs/riak-mesos)

On Mon, Jun 6, 2016 at 10:37 AM, Jan-Philip Loos  wrote:

>
>
> On Mon, 6 Jun 2016 at 16:52 Alex Moore  wrote:
>
>> Hi Jan,
>>
>> When you update the Kubernates nodes, do you have to do them all at once
>> or can they be done in a rolling fashion (one after another)?
>>
>
> Thnaks for your reply,
>
> sadly this is not possible. Kubernetes with GKE just tears all nodes down,
> creating new nodes with new kubernets version and reschedule all services
> on these nodes. So after an upgrade, all riak nodes are stand-alone (when
> starting after deleting /var/lib/riak/ring)
>
> Greetings
>
> Jan
>
>
>> If you can do them rolling-wise, you should be able to:
>>
>> For each node, one at a time:
>> 1. Shut down Riak
>> 2. Shutdown/restart/upgrade Kubernates
>> 3. Start Riak
>> 4. Use `riak-admin force-replace` to rename the old node name to the new
>> node name
>> 5. Repeat on remaining nodes.
>>
>> This is covered in "Renaming Multi-node clusters
>> "
>> doc.
>>
>> As for your current predicament,  have you created any new
>> buckets/changed bucket props in the default namespace since you restarted?
>> Or have you only done regular operations since?
>>
>> Thanks,
>> Alex
>>
>>
>> On Mon, Jun 6, 2016 at 5:25 AM Jan-Philip Loos 
>> wrote:
>>
>>> Hi,
>>>
>>> we are using riak in a kuberentes cluster (on GKE). Sometimes it's
>>> necessary to reboot the complete cluster to update the kubernetes-nodes.
>>> This results in a complete shutdown of the riak cluster and the riak-nodes
>>> are rescheduled with a new IP. So how can I handle this situation? How can
>>> I form a new riak cluster out of the old nodes with new names?
>>>
>>> The /var/lib/riak directory is persisted. I had to delete the
>>> /var/lib/riak/ring folder otherwise "riak start" crashed with this message
>>> (but saved the old ring state in a tar):
>>>
>>> {"Kernel pid
 terminated",application_controller,"{application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,['
 riak@10.44.2.8
 ',[]],[{file,\"orddict.erl\"},{line,72}]},{riak_core_broadcast,init_peers,1,[{file,\"src/riak_core_broadcast.erl\"},{line,616}]},{riak_core_broadcast,start_link,0,[{file,\"src/riak_core_broadcast.erl\"},{line,116}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,310}]},{supervisor,start_children,3,[{file,\"supervisor.erl\"},{line,293}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,259}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}],{riak_core_app,start,[normal,[]]}}}"}
 Crash dump was written to: /var/log/riak/erl_crash.dump
 Kernel pid terminated (application_controller)
 ({application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,['
 riak@10.44.2.8',
>>>
>>>
>>> The I formed a new cluster via join & plan & commit.
>>>
>>> But now, I discovered a problems with incomplete and inconsistent
>>> partitions:
>>>
>>> *$ *curl -Ss "
>>> http://riak.default.svc.cluster.local:8098/buckets/users/keys?keys=true;
>>> | jq '.[] | length'
>>>
>>> 3064
>>>
>>> *$* curl -Ss "
>>> http://riak.default.svc.cluster.local:8098/buckets/users/keys?keys=true;
>>> | jq '.[] | length'
>>>
>>> 2987
>>>
>>> *$* curl -Ss "
>>> http://riak.default.svc.cluster.local:8098/buckets/users/keys?keys=true;
>>> | jq '.[] | length'
>>>
>>> 705
>>>
>>> *$* curl -Ss "
>>> http://riak.default.svc.cluster.local:8098/buckets/users/keys?keys=true;
>>> | jq '.[] | length'
>>> 3064
>>>
>>> Is there a way to fix this? I guess this is caused by the missing old
>>> ring-state?
>>>
>>> Greetings
>>>
>>> Jan
>>>
>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Cluster Behavior - Clarification

[Sargun Dhillon]

What do you mean it's not returning? It's returning stale data? Or
it's erroring?

On Tue, May 24, 2016 at 7:34 AM, Vikram Lalit  wrote:
> Hi - I'd appreciate if someone can opine on the below behavior of Riak that
> I am observing... is that expected, or something wrong in my set-up /
> understanding?
>
> To summarize, I have a 3-node Riak cluster (separate EC2 AWS instances) with
> a separate chat server connecting to them. When I write data on the Riak
> nodes, the process is successful and I can read all data correctly. However,
> as part of my testing, if I deliberately bring down one node (and then
> remove it from the cluster using riak-admin cluster force-remove / plan /
> commit), the client API is not able to fetch all the written data. In fact,
> there is an alternation of success and failure which happens rather
> randomly.
>
> My initial suspicion was that it would be happening only during the time the
> rebalancing is occurring (i.e. riak-admin ring-status is not fully settled)
> but I've seen this sporadic behavior post the same too.
>
> Does this have to do with the n and r values for the cluster and given that
> 1 node is down, the cluster does not succeed in returning results reliably?
> Also, does this mean that during the time a cluster is being rebalanced
> (even incl. addition of new nodes), the results could be arbitrary - that
> doesn't sound correct to me?
>
> Appreciate if someone can throw some light here? Also, the HTTP API calls to
> retrieve and set the n / r / w values for a specific bucket - couldn't
> locate the same!
>
> Thanks much!
> Vikram
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Questions about installing Stanchion for Riak CS

[Sargun Dhillon]

I never got around to it, but it should be pretty easy to glue Consul
to Stanchion to HaProxy. As far as gluing HAProxy to Consule -- that
should be pretty easy: https://github.com/hashicorp/consul-template.
And Stanchion with Consul lock:
https://www.consul.io/docs/commands/lock.html. It shouldn't take long.



On Mon, May 23, 2016 at 10:15 AM, Luke Bakken  wrote:
> Alex -
>
> You won't be able to create new users or buckets while Stanchion is
> offline. You would follow normal procedures to rebuild Riak KV on the
> crashed node, and in the meantime could bring up Stanchion on an
> existing node.
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
>
> On Mon, May 23, 2016 at 9:29 AM, Alex De la rosa
>  wrote:
>> Hi Luke,
>>
>> Ok, understood, what if I don't have a load balancer and the node having
>> Stanchion crashes? what will happen to the cluster and how to rebuild it?
>>
>> Thanks,
>> Alex
>>
>> On Mon, May 23, 2016 at 8:09 PM, Luke Bakken  wrote:
>>>
>>> Hi Alex,
>>>
>>> You should only have one active Stanchion process running in your
>>> cluster, since its purpose is to ensure consistent, ordered operations
>>> with regard to users and buckets. You can have a hot-backup if you
>>> configure a load balancer to proxy requests from the Riak CS
>>> processes.
>>> --
>>> Luke Bakken
>>> Engineer
>>> lbak...@basho.com
>>>
>>>
>>> On Sat, May 21, 2016 at 11:41 AM, Alex De la rosa
>>>  wrote:
>>> > Hi there,
>>> >
>>> > I'm creating a Riak CS cluster and I got some questions about the
>>> > following
>>> > sentence from the documentation:
>>> >
>>> > Riak KV and Riak CS must be installed on each node in your cluster.
>>> > Stanchion, however, needs to be installed on only one node.
>>> >
>>> > Is this statement saying that only 1 node can have Stanchion? Or can it
>>> > be
>>> > placed in more servers? like Riak KV and Riak CS must be in 5 out of
>>> > 5
>>> > nodes but Stanchion can be in 1 to 5 out of 5 nodes?
>>> >
>>> > If is referring that ONLY 1 out of 5 nodes can have Stanchion and the
>>> > other
>>> > 4 nodes are not allowed to have it installed, what happens if the
>>> > "master"
>>> > node that has Stanchion crashes?
>>> >
>>> > Thanks and Best Regards,
>>> > Alex
>>> >
>>> > ___
>>> > riak-users mailing list
>>> > riak-users@lists.basho.com
>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> >
>>
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak install on instance with multiple OTP's

[Sargun Dhillon]

How did you install OTP18? When dealing with Erlang, and multiple
installs of it, I might suggest using kerl
(https://github.com/kerl/kerl). It's an excellent tool for dealing
with the problem.


On Sat, May 21, 2016 at 12:01 PM, Robert Latko  wrote:
> Hi all,
>
> Quick question:
>
> I have an instance with OTP18 and I want to make it a Riak Node. I
> DL/install  the OTP16 patch 8 for Riak 2.1.4.  How then do I use make rel to
> use OTP16 instead of the default OTP18??
>
>
> Thanks in advance.
>
> Sincerely,
>
> Robert
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak DT refresh

[Sargun Dhillon]

Is the plan to keep using riak_dt_vclock? If so, I might contribute
some optimizations for large numbers of actor entries (1000s).

On Thu, Apr 28, 2016 at 12:55 AM, Russell Brown  wrote:
> Hi,
> Riak DT[1] is in need of some love. I know that some of you on this list 
> (Sargun, are you here? Heinz?) have expressed opinions on the work that needs 
> doing. Here is my short list, I would love to hear opinions on priority, and 
> any additions to this list:
>
> 1. merger smaller map branch
> 2. deltas
> 3. new data types (we have a range register and some and Multi-Value-Register 
> to add, any more?)
> 4. Internal state as records or maps (but not these messy tuples)
> 5. update to rebar3
> 6. update to latest erlang
>
> I’m pretty sure there is plenty more. Would greatly appreciate your feedback.
>
> Many thanks
>
> Russell
>
> [1] https://github.com/basho/riak_dt/
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak DT refresh

[Sargun Dhillon]

We're using riak_dt in anger in our product. We are already using it
with rebar3, and Erlang 18.3 through some super messy patches.

I would love to see a register that takes the logical clock, and
timestamp for resolution, rather than just a straightup timestamp. My
biggest ask though is delta-CRDTs. Delta-CRDTs with a decent
anti-entropy algorithm would allow our system to scale from 1000s of
keys to 100ks of keys.

Nonetheless, Riak_dt is an awesome library, and we're probably
planning on contributing back to it soonish, especially if any of the
other comments on this thread are things we need, and Basho doesn't.

On Thu, Apr 28, 2016 at 12:55 AM, Russell Brown  wrote:
> Hi,
> Riak DT[1] is in need of some love. I know that some of you on this list 
> (Sargun, are you here? Heinz?) have expressed opinions on the work that needs 
> doing. Here is my short list, I would love to hear opinions on priority, and 
> any additions to this list:
>
> 1. merger smaller map branch
> 2. deltas
> 3. new data types (we have a range register and some and Multi-Value-Register 
> to add, any more?)
> 4. Internal state as records or maps (but not these messy tuples)
> 5. update to rebar3
> 6. update to latest erlang
>
> I’m pretty sure there is plenty more. Would greatly appreciate your feedback.
>
> Many thanks
>
> Russell
>
> [1] https://github.com/basho/riak_dt/
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak_ensemble implementation

[Sargun Dhillon]

Riak_kv sits atop riak_ensemble, which is a multipaxos implementation.

On Tue, Mar 8, 2016 at 6:41 PM, Agung Laksono <agung.laks...@surya.ac.id> wrote:
> Thank you for answering the question.
>
> @Sargun: Right now, I'm interested in testing Riak-ensemble
> with a distributed model checker like SAMC. In the cluster membership
> in the documentation, I found steps to activate a cluster, add a node,
> create an ensemble. However, is there any example to implement those
> processes?
>
> And one more question: Does Riak-kv implement multi-paxos or paxos-like
> protocols?
>
> Actually, my target is so simple, I just need to be able to put some
> key-value by
> executing the multi-paxos.
>
> On Tue, Mar 8, 2016 at 11:39 PM, Sargun Dhillon <sar...@sargun.me> wrote:
>>
>> If you want to learn to use riak_ensemble the library, the
>> documentation that Joe put together (and others)
>> https://github.com/basho/riak_ensemble/blob/develop/doc/Readme.md
>>
>> There's real world usage of the code here:
>> -https://github.com/basho/riak_kv/blob/develop/src/riak_kv_ensembles.erl
>>
>> Basically as a user, you have to create the root ensemble, add all
>> your nodes to it, and then create your individual, "sub-ensembles".
>>
>> You need to implement a backend, and this is a rough sketch of one:
>>
>> https://github.com/basho/riak_ensemble/blob/develop/src/riak_ensemble_basic_backend.erl
>>
>> Due the design of Riak ensemble, and the fact that's it's effectively
>> a bunch of individual vertical Paxos group in conjunction with its
>> usage of single decree, over multi-decree paxos it has some
>> interesting fault-recovery semantics. An example of this is the
>> adjustable paranoia level -- it has some strong failure-recovery
>> semantics, so in the highest paranoia mode, if you have 5 nodes, and
>> one fails. When that node comes back, it will have to sync with 3/5 of
>> the other nodes to be considered good to go.
>>
>> Even with this, riak ensemble is a pretty easy, fun library to use.
>> The fact that it's a composition of single decree paxos groups it
>> makes it much simpler to understand, and you can implement SMR atop
>> it, like so: https://github.com/sargun/riak_ensemble/commits/sargun/redis
>> (see the few extra commits)
>>
>> On Tue, Mar 8, 2016 at 8:19 AM, Carlos González Florido
>> <carlosj...@gmail.com> wrote:
>> > You may have a look at nkbase[1] that includes riak_ensemble as one of
>> > it
>> > working models.
>> >
>> > [1]
>> >
>> > https://github.com/NetComposer/nkbase/blob/master/doc/strong_consistency.md
>> >
>> > Carlos González
>> > NetComposer
>> >
>> >
>> > On Mon, Mar 7, 2016 at 4:48 AM, Agung Laksono
>> > <agung.laks...@surya.ac.id>
>> > wrote:
>> >>
>> >> Hi Basho developer,
>> >>
>> >> I've seen the video that you guys present about rian_ensemble. I am
>> >> interesting to implement riak_ensemble in the production environment in
>> >> the
>> >> future. After reading the
>> >> documentation(https://github.com/basho/riak_ensemble/tree/develop/doc),
>> >> I
>> >> still couldn't understand how it's work and implement it. Could you
>> >> tell me
>> >> in a simple way to implement it in the single node to simulate multiple
>> >> nodes or 3 real nodes?
>> >>
>> >> So far, I can compile riak_ensemble via rebar and run
>> >> application:ensure_all_started(riak_ensemble). But I don't know what
>> >> should
>> >> I do after that(sent K/V, communicate to other nodes, etc)
>> >>
>> >>  Thank you in advance.
>> >>
>> >> --
>> >> Warm Regards,
>> >>
>> >> Agung Laksono
>> >>
>> >> ___
>> >> riak-users mailing list
>> >> riak-users@lists.basho.com
>> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >>
>> >
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>
>
>
> --
> Warm Regards,
>
> Agung Laksono

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak_ensemble implementation

[Sargun Dhillon]

If you want to learn to use riak_ensemble the library, the
documentation that Joe put together (and others)
https://github.com/basho/riak_ensemble/blob/develop/doc/Readme.md

There's real world usage of the code here:
-https://github.com/basho/riak_kv/blob/develop/src/riak_kv_ensembles.erl

Basically as a user, you have to create the root ensemble, add all
your nodes to it, and then create your individual, "sub-ensembles".

You need to implement a backend, and this is a rough sketch of one:
https://github.com/basho/riak_ensemble/blob/develop/src/riak_ensemble_basic_backend.erl

Due the design of Riak ensemble, and the fact that's it's effectively
a bunch of individual vertical Paxos group in conjunction with its
usage of single decree, over multi-decree paxos it has some
interesting fault-recovery semantics. An example of this is the
adjustable paranoia level -- it has some strong failure-recovery
semantics, so in the highest paranoia mode, if you have 5 nodes, and
one fails. When that node comes back, it will have to sync with 3/5 of
the other nodes to be considered good to go.

Even with this, riak ensemble is a pretty easy, fun library to use.
The fact that it's a composition of single decree paxos groups it
makes it much simpler to understand, and you can implement SMR atop
it, like so: https://github.com/sargun/riak_ensemble/commits/sargun/redis
(see the few extra commits)

On Tue, Mar 8, 2016 at 8:19 AM, Carlos González Florido
 wrote:
> You may have a look at nkbase[1] that includes riak_ensemble as one of it
> working models.
>
> [1]
> https://github.com/NetComposer/nkbase/blob/master/doc/strong_consistency.md
>
> Carlos González
> NetComposer
>
>
> On Mon, Mar 7, 2016 at 4:48 AM, Agung Laksono 
> wrote:
>>
>> Hi Basho developer,
>>
>> I've seen the video that you guys present about rian_ensemble. I am
>> interesting to implement riak_ensemble in the production environment in the
>> future. After reading the
>> documentation(https://github.com/basho/riak_ensemble/tree/develop/doc), I
>> still couldn't understand how it's work and implement it. Could you tell me
>> in a simple way to implement it in the single node to simulate multiple
>> nodes or 3 real nodes?
>>
>> So far, I can compile riak_ensemble via rebar and run
>> application:ensure_all_started(riak_ensemble). But I don't know what should
>> I do after that(sent K/V, communicate to other nodes, etc)
>>
>>  Thank you in advance.
>>
>> --
>> Warm Regards,
>>
>> Agung Laksono
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak_ensemble implementation

[Sargun Dhillon]

Can you tell us a little bit about the application? It might be easier to
use Riak kv rather than riak ensemble directly.

Sent from my iPhone

On Mar 8, 2016, at 06:54, Agung Laksono  wrote:

Hi Basho developer,

I've seen the video that you guys present about rian_ensemble. I am
interesting to implement riak_ensemble in the production environment in the
future. After reading the documentation(
https://github.com/basho/riak_ensemble/tree/develop/doc), I still couldn't
understand how it's work and implement it. Could you tell me in a simple
way to implement it in the single node to simulate multiple nodes or 3 real
nodes?

So far, I can compile riak_ensemble via rebar and run
application:ensure_all_started(riak_ensemble). But I don't know what should
I do after that(sent K/V, communicate to other nodes, etc)

 Thank you in advance.

-- 
Warm Regards,

Agung Laksono

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ring_creation_size & ring_partition_number error joining 2.1.3 node to 2.1.1 cluster

[Sargun Dhillon]

They're effectively equal

Sent from my iPhone

On Mar 6, 2016, at 17:32, Robert Latko  wrote:

Hi all,

Found the problem/solution.

With the first cluster, I set it up with a ring size of 64. Prior to
loading data. I stopped the node(s), changed the ring size to 256, then
restarted.

Therefore, if I want to add a new node, I FIRST must start the node with a
ring size of 64, then stop it, change the conf for a ring size of 256, then
start, then stage, plan, and commit.

I am still not sure the relation ship with ring size and ring partitions.

Sincerely,

Robert



On 03/06/2016 02:10 PM, Robert Latko wrote:

Hi All,

I've got a cluster in 2.1.1 and was wanting to join a 2.1.3 node to it

I am getting the following error:

# ./bin/riak-admin cluster join a-backend@10.0.0.199
Failed: a-backend@10.0.0.199 has a different ring_creation_size

So... for the new 2.1.3
# ./bin/riak-admin status | grep ring
ring_creation_size : 256
ring_members : ['node-template@10.0.0.54']
*ring_num_partitions : 256*
ring_ownership : <<"[{'node-template@10.0.0.54',256}]"
<[%7B%27node-template@10.0.0.54%27,256%7D]>>>
rings_reconciled : 0
rings_reconciled_total : 0

AND for node a-backend@10.0.0.199
# ./bin/riak-admin status | grep ring
ring_creation_size : 256
ring_members : ['a-backend@10.0.0.199',' 
b-backend@10.0.0.200',
*ring_num_partitions : 64*
ring_ownership : <<"[{'b-backend@10.0.0.200',13},\n
{'c-backend@10.0.0.201',13},\n {'d-backend@10.0.0.202',13},\n
{'e-backend@10.0.0.203',12},\n {'a-backend@10.0.0.199',13}]"
<[%7B%27b-backend@10.0.0.200%27,13%7D,%5Cn%7B%27c-backend@10.0.0.201%27,13%7D,%5Cn%7B%27d-backend@10.0.0.202%27,13%7D,%5Cn%7B%27e-backend@10.0.0.203%27,12%7D,%5Cn%7B%27a-backend@10.0.0.199%27,13%7D]>
>>
rings_reconciled : 0
rings_reconciled_total : 32

The 'ring_num_partitions' are different which leaves me to believe this is
the cause.

A.) How do I increase the 'ring_num_partitions' to 256 in the 2.1.1
cluster� **OR**
B.) How do I decrease the 'ring_num_partitions' to 64 in the 2.1.3 node?

Any ideas? recommendations?


Thanks in advance,


Robert









___
riak-users mailing
listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Continuous HTTP POSTs to Riak

[Sargun Dhillon]

So, if you pass back the vclock while pr=pw=quorum, and sloppy_quorum
= false you should get RYOW consistency.

On Thu, Mar 3, 2016 at 6:25 PM, Christopher Mancini  wrote:
> If you don't need strong consistency for all Riak requests, just certain
> ones, then explore the use of R and N vals that can be passed along with the
> request or on a bucket-type level.
>
> On Thu, Mar 3, 2016 at 9:23 PM Mark Schmidt  wrote:
>>
>> Hi Qiang,
>>
>>
>>
>> Other more skilled folks can correct me if I am wrong, but I believe you
>> can enforce strong consistency on your nodes:
>> http://docs.basho.com/riak/latest/theory/concepts/strong-consistency/
>>
>>
>>
>> If I recall, you will take a performance hit as all nodes must receive the
>> data.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> -  Mark Schmidt
>>
>>
>>
>> From: riak-users [mailto:riak-users-boun...@lists.basho.com] On Behalf Of
>> Qiang Cao
>> Sent: Thursday, March 03, 2016 6:13 PM
>> To: Dave King 
>> Cc: riak-users 
>> Subject: Re: Continuous HTTP POSTs to Riak
>>
>>
>>
>> Thanks, Dave! I'm aware of Riak's eventual consistency. I wonder how I can
>> get Riak work in the way as I need. I'm in a situation where I expect the
>> immediate GET reads the value I've just posted. And the GET request isn't
>> sent out until the POST is done.
>>
>>
>>
>> Thanks,
>>
>> -Qiang
>>
>>
>>
>> On Thu, Mar 3, 2016 at 6:53 PM, Dave King  wrote:
>>
>> You need to google 'Eventual consistency'.
>>
>>
>>
>> - Peace
>>
>> Dave
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 4:23 PM, Qiang Cao  wrote:
>>
>> Hi All,
>>
>>
>>
>> I use Riak as a KV store and send http requests to its REST API to update
>> entries. Instead of using curl, I use Apache HttpClient to send requests.
>>
>>
>>
>> One thing bothering me is that sometimes I find the entry isn't properly
>> updated after an HTTP POST/PUT. For example, if I do an HTTP GET on the same
>> key immediately after an HTTP POST, the chance is good that I get a stale
>> value. This happens when I continuously POST to a multi-node Riak, and some
>> of the posts target the same key.
>>
>>
>>
>> Also, I find I don't have this issue if I use a single-node Riak. So I
>> think something wasn't correct with my multi-node Riak cluster (I set
>> "allow_mult" to false).  Any thoughts?
>>
>>
>>
>> Thanks in advance!
>>
>>
>>
>> Qiang
>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> --
> Sincerely,
>
> Christopher Mancini
> -
>
> employee = {
> purpose: solve problems with code,
> phone:7164625591,
> email: cmanc...@basho.com,
> github:http://www.github.com/christophermancini
> }
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Reliability quastion

[Sargun Dhillon]

Given the tunables that Riak has, I would say that its reliability in
terms of consistency (as in ACID) are unparalleled. It offers a
variety of weak consistency options from eventual consistency, to
strong eventual consistency, to strong consistency*.

In terms of fault-tolerance, it is interesting. Riak hides faults in
funny ways where other databases would degrade. It instead does a
really good job of hiding faults. This can be problematic, because
often the signal for a database degrading can be "the website is
flakey".

* This is "experimental"

On Thu, Jan 14, 2016 at 12:07 AM, Russell Brown  wrote:
>
>
> On 14 Jan 2016, at 07:05, j.halif...@seznam.cz wrote:
>
> Hi All,
>
>
> I am a member of the community for about a year to understand reliability of
> Riak.
>
> What I can see here is that there are some problems which could prevent
> using
>
> it in a project which require high reliability. That's why I'd like to ask
> the following:
>
>
> Do you believe that Riak can be used in a project which requires high
> reliability?
>
> Yes!
>
>
> Thank you so much..
>
>
> BR,
>
> jh
>
>
>
>
> =
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: error on object retrieval: Cannot send another request while one is in progress

[Sargun Dhillon]

if there is a failure, and leader is re-elected, the epoch of the ensemble
changes, so you may have to refetch the object to get a new causal context
in order to perform another write. If this occurs, you should be able to
heal by doing another read, modify, write cycle.

On Wed, Sep 23, 2015 at 3:21 PM, Luke Bakken  wrote:

> Hristo,
>
> How are you communicating with Riak? Which client library is being used?
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
> On Wed, Sep 23, 2015 at 3:18 PM, Hristo Asenov
>  wrote:
> > Hello,
> >
> > I am using a strongly consistent bucket type. I need to simulate
> instances where a process can crash and recover. When a crash occurs, I
> cleanly disconnect and destroy the memory (through a signal handler).
> >
> > Sometimes, when process recovery occurs, performing a get on an object
> is successful, however a subsequent set with the Vector Clock from the
> previous get results in the error “Cannot send another request while one is
> in progress”. I found the error in riak_api project, file
> riak_api_pb_server.erl, but don’t really understand why the error gets
> thrown. Would someone be able to instruct me as to the cause of the error?
> I am using the current riak version from the main repository.
> >
> > - Hristo
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Running out of disk space

[Sargun Dhillon]

Are you using bitcask, or LevelDB? What version of Riak are you using?
Bitcask will lazily merge files in the background in order to reclaim
space. This is pretty aggressive, and it should show up pretty quickly. On
the other hand, LevelDB deletes files at compaction time. Compactions
aren't quite as aggressive as Bitcask, although in 2.X, some of this was
made better.

As to your other question, if you add a 6th node, ideally, it should
handoff vnodes and receive an equal share of the load as the remaining 5
nodes in your cluster, but disk usage may not look the same, as only live
data, and tombstones are moved over. The recommendation in this scenario is
to add nodes to your cluster. If handoffs don't occur quickly enough, it
may be wise to bump the handoff limits.

On Sun, Sep 13, 2015 at 7:40 PM, ender  wrote:

> I have a 5 node Riak cluster running on AWS. I noticed that disk usage is
> now at 80% for all nodes. I started deleting content but it doesn't appear
> to be making much of a dent. If I add a 6th node, will it start receiving
> all new content until it has approximately the same usage as the original 5
> nodes, or will they also continue to grow? What is the recommended approach
> in this situation?
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: distributed basho bench

[Sargun Dhillon]

So, the way it should work is pretty simple:
Run the command a la: ./basho_bench -N nodeA@1 0.0.1.123 -C
basho_bench_cookie 
(It's key that the IP address be the external IP, and not the internal IP
of the box, a la loopback)

In addition, nodeB must have the same version of Erlang installed as Node
A. So, typically, you should install ESL Erlang (my recommendation is R16,
https://www.erlang-solutions.com/downloads/download-erlang-otp) on both
boxes, and compile your own Basho bench.

-Thanks,
Sargun!

On Thu, Sep 10, 2015 at 9:15 AM, Jens V. Fischer 
wrote:

> Hi,
>
> I'm trying to use basho bench in distributed mode, that is to say to use
> more than one machine for load generation. The documentation and the
> "--join" param of the basho_bench seem to indicate that this should be
> possible. But so far I failed to set it up. Let's say I want to start basho
> bench on two nodes, lets call them Node A and B. What parameters to
> basho_bench are needed on A and B respectively?
>
> I also saw the previous mail exchange on the topic (How to run bash bench
> distributed, 18.5.2015). From there I gathered that I need to use something
> like the following (for Node A):
>
> Start command:
> ./basho_bench -N nodeA@127.0.0.1 -C basho_bench_cookie 
>
> Also I need in my config (on Node A):
> {remote_nodes, [{, 'nodeB'}]}.
> {distribute_work, true}.
>
> Do I need to do anything on Node B? Like starting basho_bench with
> "--join"? What would the command look like? What do I need to put into my
> config on Node B? In what order do I have to execute the commands on A and
> B?
>
> Additionally, as I am trying to use the distributed setup with a custom
> driver: Does distributed mode only works with certain drivers, i.e. does
> the driver need to support it? For what exactly is the passwordless SSH
> needed? Because I don't have SSH access on the particular cluster, I do
> however have access over TCP and are, for example, able to setup
> distributed Erlang between the nodes. Any chance to get basho_bench working
> in this setup?
>
> Any help is greatly appreciated. I'm glad to provide more information if
> needed.
>
> Best regards
> Jens
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Problem with Riak not thinking hostname is fully-qualified

[Sargun Dhillon]

Put a period at the end.

On Wed, Aug 12, 2015 at 8:52 PM, Toby Corkindale t...@dryft.net wrote:
 I'm trying to set up another cluster, and I'm hitting problems with Riak
 complaining that ** System running to use fully qualified hostnames **
 ** Hostname db04 is illegal **

 However, as far as I can see, the host does have a fully-qualified hostname:

 root@db04:/# hostname
 db04
 root@db04:/# hostname -f
 db04.dc.sdlocal.net

 /etc/hosts contains a line with both the short and fqdn names in it.

 Is there something else I should be checking?

 Cheers
 Toby

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: How to run bash bench distributed

[Sargun Dhillon]

When you start basho_bench, you must start Erlang in distributed mode,
which means you must set a node name, and a cookie.

So, it would look something like the following: ./basho_bench -N
foo@127.0.0.1 -C basho_bench examples/riakc_pb_distributed.config

Ensure that in examples/riakc_pb_distributed.config, remote nodes is
set to a list of nodese that passwordless SSH can connect to.

On Wed, Mar 18, 2015 at 3:25 PM, Mohamed Abd Elhafez
mohamed.abdelha...@gmail.com wrote:
 Hi guys,

 I am very new to Riak and I am trying to run some benchmarks for our cluster

 I saw this blog post about running basho bench distributed to be able to get
 high throughput.
 https://medium.com/@mustwin/benchmarking-riak-bfee93493419

 I also found the example config that can run in distributed mode, but when I
 try to run it:

 ./basho_bench examples/riakc_pb_distributed.config, I get the message:
 Basho bench not started in distributed mode, and distribute_work = true

 Is there some documentation no how to run basho bench distributed?


 Thanks,

 Mohamed


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak 1.4 vs riak 1.2

[Sargun Dhillon]

Are you using LevelDB? If so, it'd be impossible, and even unfair for
me to summarize all the work that's been done by MVM on LevelDB as
part of 1.4, and 2.0: https://github.com/basho/leveldb/wiki



On Tue, Feb 10, 2015 at 6:15 PM, siva Ram sivaram.pasupul...@gmail.com wrote:
 In terms of performance improvement  improvements , what is difference we
 need to see in riak 1.4 vs riak 1.2 ?

 Thanks
 Siva


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Strong consistent buckets ready after starting a cluster.

[Sargun Dhillon]

I really don't recommend doing this. Really, the cluster shouldn't be
changing often enough where the ensemble status is changing all that
often. You can do this by exploiting internal APIs that are accessible
over distributed Erlang in riak_ensemble. Though, connecting to the
cluster over distributed Erlang from your client is probably also less
than recommended because it adds extra complexity, and you're poking
at internal APIs that aren't meant to be exposed to the client.

If you want to know for your own sake, you can look at the output from
'riak-admin ensemble-status'

In order to do this programmatically, you'd want to hook into
riak_ensemble_manager, and interrogate it for the status of the
ensembles. I would start by looking at the code in
riak_kv/riak_kv_ensemble_console.erl to go down this route.

On Mon, Feb 2, 2015 at 3:35 AM, Zsolt Laky zsolt.l...@lamardan.com wrote:
 Hi All!

 I am using a dev cluster with three nodes on a Mac OS. After a boot, I start
 the nodes and they are ok immediately for the default buckets, but for the
 strong consistent type it took 10 minutes to get ready and perform as
 expected.

 Here is a small example where a get returns with different values and at the
 end it is ok and I can be sure the RIAK cluster is up and running for strong
 consistent buckets. At the very beginning of startup I got
 {error,insufficient_vnodes_available} replies but it is ok.

 My request for help is to find a way when I know the cluster is ready. I
 understand, RIAK needs time to sync all the nodes after startup, on the
 other hand I need to know when it gets ready for production and will give
 consistent answers. Is there a command in erlang to check it? I also attach
 part of the full log when I listed the keys and values in the buckets to see
 when would I get the correct result. It took about 10 minutes to get there.

 Thanks in advance for thoughts and suggestions, your help is highly
 appreciated.
 Kind regards,
 Zsolt

 ==
 Type:strong
 
 Bucket:Actors_msuser
 Key:{msuser_actor,\z2\,lamardan} Value:Error:{error,timeout}
 
 Bucket:Actors_mssession
 Key:{mssession_actor,\z2\,zschat,0} Value:Error:{error,timeout}
 ok
 ==
 Type:strong
 
 Bucket:Actors_msuser
 Key:{msuser_actor,\z2\,lamardan} Value:Error:{error,failed}
 
 Bucket:Actors_mssession
 Key:{mssession_actor,\z2\,zschat,0} Value:Error:{error,timeout}
 ==
 Type:strong
 
 Bucket:Actors_msuser
 Key:{msuser_actor,\z2\,lamardan} Value:Error:{error,failed}
 
 Bucket:Actors_mssession
 Key:{mssession_actor,\z2\,zschat,0} Value:Error:{error,failed}
 ==
 Type:strong
 
 Bucket:Actors_msuser
 ...
 Key:{msuser_actor,\z2\,lamardan} Value:Error:{error,notfound}
 
 Bucket:Actors_mssession
 Key:{mssession_actor,\z2\,zschat,0} Value:Error:{error,notfound}


 ---
 This email has been checked for viruses by Avast antivirus software.
 http://www.avast.com

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Adding nodes to cluster

[Sargun Dhillon]

1) Potentially re-enable AAE after migration. As your cluster gets
bigger, the likelihood of any node failing in the cluster goes up.
Replica divergence only becomes scarier in light of this. Losing data
!= awesome.

6) There shouldn't be any problems, but for safe measures you should
probably upgrade the old ones before the migration.



On Sat, Jan 24, 2015 at 1:31 PM, Edgar Veiga edgarmve...@gmail.com wrote:
 Sargun,

 Regarding 1) - AAE is disabled. We had a problems with it and there's a lot
 of threads here in the mailing list regarding this. AAE won't stop using
 more and more disk space and the only solution was disabling it! Since then
 the cluster has been pretty stable...

 Regarding 6) Can you or anyone in basho confirm that there won't be any
 problems using the latest (1.4.12) version of riak in the new nodes and only
 upgrading the old ones after this process is completed?

 Thanks a lot for the other tips, you've been very helpful!

 Best regards,
 Edgar

 On 24 January 2015 at 21:09, Sargun Dhillon sar...@sargun.me wrote:

 Several things:
 1) If you have data at rest that doesn't change, make sure you have
 AAE, and it's ran before your cluster is manipulated. Given that
 you're running at 85% space, I would be a little worried to turn it
 on, because you might run out of disk space. You can also pretty
 reasonably put the AAE trees on magnetic storage. AAE is nice in the
 sense that you _know_ your cluster is consistent at a point in time.

 2) Make sure you're getting SSDs of roughly the same quality. I've
 seen enterprise SSDs get higher and higher latency as time goes on,
 due to greater data protection features. We don't need any of that.
 Basho_bench is your friend if you have the time.

 3) Do it all in one go. This will enable handoffs more cleanly, and all at
 once.

 4) Do not add the new nodes to the load balancer until handoff is
 done. At least experimentally, latency increases slightly on the
 original cluster, but the target nodes have pretty awful latency.

 5) Start with a handoff_limit of 1. You can easily raise this. If
 things look good, you can increase it. We're not optimizing for the
 total time to handoff, we really should be optimizing for individual
 vnode handoff time.

 6) If you're using Leveldb, upgrade to the most recent version of Riak
 1.4. There have been some improvements. 1.4.9 made me happier. I think
 it's reasonable for the new nodes to start on 1.4.12, and the old
 nodes to be switched over later.

 7) Watch your network utilization. Keep your disk latency flat. Stop
 it if it spikes. Start from enabling one node with the lowest usage
 and see if it works.


 These are the things I can think of immediately.

 On Sat, Jan 24, 2015 at 12:42 PM, Alexander Sicular sicul...@gmail.com
 wrote:
  I would probably add them all in one go so you have one vnode migration
  plan that gets executed. What is your ring size? How much data are we
  talking about? It's not necessarily the number of keys but rather the total
  amount of data and how quickly that data can move en mass between machines.
 
  -Alexander
 
 
  @siculars
  http://siculars.posthaven.com
 
  Sent from my iRotaryPhone
 
  On Jan 24, 2015, at 15:37, Ed edgarmve...@gmail.com wrote:
 
  Hi everyone!
 
  I have a riak cluster, working in production for about one year, with
  the following characteristics:
  - Version 1.4.8
  - 6 nodes
  - leveldb backend
  - replication (n) = 3
  ~ 3 billion keys
 
  My ssd's are reaching 85% of capacity and we have decided to buy 6 more
  nodes to expand the cluster.
 
  Have you got any kind of advice on executing this operation or should I
  just follow the documentation on adding new nodes to a cluster?
 
  Best regards!
  Edgar
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Adding nodes to cluster

[Sargun Dhillon]

Several things:
1) If you have data at rest that doesn't change, make sure you have
AAE, and it's ran before your cluster is manipulated. Given that
you're running at 85% space, I would be a little worried to turn it
on, because you might run out of disk space. You can also pretty
reasonably put the AAE trees on magnetic storage. AAE is nice in the
sense that you _know_ your cluster is consistent at a point in time.

2) Make sure you're getting SSDs of roughly the same quality. I've
seen enterprise SSDs get higher and higher latency as time goes on,
due to greater data protection features. We don't need any of that.
Basho_bench is your friend if you have the time.

3) Do it all in one go. This will enable handoffs more cleanly, and all at once.

4) Do not add the new nodes to the load balancer until handoff is
done. At least experimentally, latency increases slightly on the
original cluster, but the target nodes have pretty awful latency.

5) Start with a handoff_limit of 1. You can easily raise this. If
things look good, you can increase it. We're not optimizing for the
total time to handoff, we really should be optimizing for individual
vnode handoff time.

6) If you're using Leveldb, upgrade to the most recent version of Riak
1.4. There have been some improvements. 1.4.9 made me happier. I think
it's reasonable for the new nodes to start on 1.4.12, and the old
nodes to be switched over later.

7) Watch your network utilization. Keep your disk latency flat. Stop
it if it spikes. Start from enabling one node with the lowest usage
and see if it works.


These are the things I can think of immediately.

On Sat, Jan 24, 2015 at 12:42 PM, Alexander Sicular sicul...@gmail.com wrote:
 I would probably add them all in one go so you have one vnode migration plan 
 that gets executed. What is your ring size? How much data are we talking 
 about? It's not necessarily the number of keys but rather the total amount of 
 data and how quickly that data can move en mass between machines.

 -Alexander


 @siculars
 http://siculars.posthaven.com

 Sent from my iRotaryPhone

 On Jan 24, 2015, at 15:37, Ed edgarmve...@gmail.com wrote:

 Hi everyone!

 I have a riak cluster, working in production for about one year, with the 
 following characteristics:
 - Version 1.4.8
 - 6 nodes
 - leveldb backend
 - replication (n) = 3
 ~ 3 billion keys

 My ssd's are reaching 85% of capacity and we have decided to buy 6 more 
 nodes to expand the cluster.

 Have you got any kind of advice on executing this operation or should I just 
 follow the documentation on adding new nodes to a cluster?

 Best regards!
 Edgar

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: erlang compile

[Sargun Dhillon]

So, you need to set add_paths as a runtime option, once you compile
the Erlang program into beam files (perhaps with a .app) so that Riak
can load your code. Have you already compiled your program with the
Erlang compiler that comes with your version of Riak? Which version of
Riak are you using? In 2.0, I believe you have to add it to the
advanced.config, as per: https://github.com/basho/riak/issues/566

On Fri, Jan 16, 2015 at 1:27 AM, Ildar Alishev ildaralis...@gmail.com wrote:
 Hello everyone!


 Would like to ask.

 Found on http://docs.basho.com/riak/latest/ops/advanced/install-custom-code/

 that i should find and add data in app.config

 ok, where i can find it

 i have this conf file in my rebar, but don’t know if it is right, maybe it
 should be created for riak somewhere???


 thank you!

 Ildar.

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Wiping Riak 2.0 data

[Sargun Dhillon]

Assuming your storage engine is bitcask:
1. Turn off all your Riak nodes
2. rm -rf --verbose /var/lib/riak/bitcask/*/*
3. rm -rf --verbose /var/lib/riak/anti_entropy/*/*
4. Turn Riak back on

This will preserve your buckets and bucket types in in cluster
metadata. You can also automate that, but it's a little more
complicated.

On Sun, Jan 4, 2015 at 4:25 AM, Cosmin Marginean cosmin...@gmail.com wrote:
 Hi,

 I’m running Riak 2.0.2 on Centos 6.5 installed from an RPM
 (http://s3.amazonaws.com/downloads.basho.com/riak/2.0/2.0.2/rhel/6/riak-2.0.2-1.el6.x86_64.rpm)

 I’m interested in wiping the entire storage (in order to re-run tests, etc).
 I have a script that traverses buckets/keys but I’m aware that this process
 is costly, plus I might be dealing with some eventual-consistency issues.

 Consequently, I would prefer to wipe the “data” directory entirely if that’s
 in any way possible, and re-start Riak. However, the structure seems to be
 scattered in several directories and is not entirely clear which one needs
 to be wiped for a “clean data reset” without corrupting anything:

 # ll /var/lib/riak/
 total 28
 drwxrwxr-x. 66 riak riak 4096 Jan  3 17:29 anti_entropy
 drwxrwxr-x. 66 riak riak 4096 Jan  3 17:29 bitcask
 drwxrwxr-x.  3 riak riak 4096 Jan  3 17:28 cluster_meta
 drwxrwxr-x.  2 riak riak 4096 Jan  3 20:07 generated.configs
 drwxrwxr-x.  2 riak riak 4096 Jan  3 17:29 kv_vnode
 drwxrwxr-x.  2 riak riak 4096 Jan  4 07:23 riak_kv_exchange_fsm
 drwxr-xr-x.  2 riak riak 4096 Jan  3 20:07 ring

 Thank you
 Cos

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Random timeouts on Riak

[Sargun Dhillon]

Several things:
1) I recommend you have a 5-node cluster:
http://basho.com/why-your-riak-cluster-should-have-at-least-five-nodes/
2) What version of Riak are you using?
3) What backend(s) are you using?
4) What's the size of your keyspace?
5) Are you actively rewriting keys, or writing keys to the cluster?
6) Do you know how much I/O the cluster is currently doing?

On Mon, Dec 29, 2014 at 2:51 AM, Jason Ryan jason.r...@trustev.com wrote:
 Hi,

 We are getting random timeouts from our application (60seconds) when we try
 to retrieve a key from our Riak cluster (4 nodes with a load balancer in
 front of them). Our application just uses the standard REST API to query
 Riak.

 We are pretty new to Riak - so would like to understand how best to debug
 this issue? Is there any good pointers on what to start with? This is our
 production cluster.

 Thanks,
 Jason


 This message is for the named person's use only. If you received this
 message in error, please immediately delete it and all copies and notify the
 sender. You must not, directly or indirectly, use, disclose, distribute,
 print, or copy any part of this message if you are not the intended
 recipient. Any views expressed in this message are those of the individual
 sender and not Trustev Ltd. Trustev is registered in Ireland No. 516425 and
 trades from 2100 Cork Airport Business Park, Cork, Ireland.


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Random timeouts on Riak

[Sargun Dhillon]

The bucket (type) that you're working with -- what are your
allow_mult, and last_write_wins settings?

On Mon, Dec 29, 2014 at 4:05 AM, Jason Ryan jason.r...@trustev.com wrote:
 It seems to move between 4 keys in particular, these keys are actually empty
 at the moment (i.e. an empty JSON document).

 CPU utilization is close to zero.

 Can't see anything in particular, bar the error message I just posted
 before.

 Jason


 On 29 December 2014 at 11:58, Ciprian Manea cipr...@basho.com wrote:

 Hi Jason,

 Are these random timeouts happening for only one key, or is common for
 more?

 What is the CPU utilisation in the cluster when you're experience these
 timeouts?

 Can you spot anything peculiar in your server's $ dmesg outputs? Any I/O
 errors there?


 Regards,
 Ciprian

 On Mon, Dec 29, 2014 at 1:55 PM, Sargun Dhillon sar...@sargun.me wrote:

 Several things:
 1) I recommend you have a 5-node cluster:
 http://basho.com/why-your-riak-cluster-should-have-at-least-five-nodes/
 2) What version of Riak are you using?
 3) What backend(s) are you using?
 4) What's the size of your keyspace?
 5) Are you actively rewriting keys, or writing keys to the cluster?
 6) Do you know how much I/O the cluster is currently doing?

 On Mon, Dec 29, 2014 at 2:51 AM, Jason Ryan jason.r...@trustev.com
 wrote:
  Hi,
 
  We are getting random timeouts from our application (60seconds) when
  we try
  to retrieve a key from our Riak cluster (4 nodes with a load balancer
  in
  front of them). Our application just uses the standard REST API to
  query
  Riak.
 
  We are pretty new to Riak - so would like to understand how best to
  debug
  this issue? Is there any good pointers on what to start with? This is
  our
  production cluster.
 
  Thanks,
  Jason
 
 
  This message is for the named person's use only. If you received this
  message in error, please immediately delete it and all copies and
  notify the
  sender. You must not, directly or indirectly, use, disclose,
  distribute,
  print, or copy any part of this message if you are not the intended
  recipient. Any views expressed in this message are those of the
  individual
  sender and not Trustev Ltd. Trustev is registered in Ireland No. 516425
  and
  trades from 2100 Cork Airport Business Park, Cork, Ireland.
 
 
  ___
  riak-users mailing list
  riak-users@lists.basho.com
  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




 This message is for the named person's use only. If you received this
 message in error, please immediately delete it and all copies and notify the
 sender. You must not, directly or indirectly, use, disclose, distribute,
 print, or copy any part of this message if you are not the intended
 recipient. Any views expressed in this message are those of the individual
 sender and not Trustev Ltd. Trustev is registered in Ireland No. 516425 and
 trades from 2100 Cork Airport Business Park, Cork, Ireland.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: problems with starting nodes

[Sargun Dhillon]

Can you post the files in the log directory on a github gist, and run
the command ./dev1/bin/riak console? In addition, run Riak as a
non-root user, with the max files limit bumped up as high as you can
set it.

On Wed, Dec 24, 2014 at 10:47 PM, Ildar Alishev ildaralis...@gmail.com wrote:
 Hello


 I have a problem starting nodes


 it says

 root@salo:/home/salohost/work/riak-2.0.2/dev# ./dev1/bin/riak start
 
  WARNING: ulimit -n is 1024; 65536 is the recommended minimum.
 
 riak failed to start within 15 seconds,
 see the output of 'riak console' for more information.
 If you want to wait longer, set the environment variable
 WAIT_FOR_ERLANG to the number of seconds to wait.

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Weird RIAK behavior

[Sargun Dhillon]

It sounds like allow_mult is off.
1. What's your last_write_wins set to?
2. Are the clocks on your nodes accurately synced to one another (if
they're all NTP peers from one another, what's the deltas?)? I know
drift is a common problem in virtual machines on VMWare -- or clocks
plain lying.
3. Are you using vector clocks for all operations?
4. Are you making concurrent updates?

Are you familiar with AP Riak, the consistency model, and conflict
resolution? There is some excellent documentation here:
http://docs.basho.com/riak/2.0.2/dev/using/conflict-resolution/



On Mon, Dec 22, 2014 at 7:15 PM, Claudio Cesar Sanchez Tejeda
demoncc...@gmail.com wrote:
 Hi,

 Sorry, I forgot to mention that it is RIAK 1.4.10. They are configured
 with multibackend. We are using, memory, bitcask and elevelDB
 backends.

 On the buckets where we are having issues, the siblings are disabled
 (allow_multi = false).

 Regards.

 On Mon, Dec 22, 2014 at 9:17 PM, Sargun Dhillon sar...@sargun.me wrote:
 What versions of Riak are you using? And are these CRDT sets?

 Sent from my iPhone

 On Dec 22, 2014, at 16:04, Claudio Cesar Sanchez Tejeda 
 demoncc...@gmail.com wrote:

 I'm a sysadmin and I managing 5 cluster of RIAK:

 - two of them are LXC containers on the same physical machine (3 nodes
 per cluster)
 - one of them are LXC containers located on different physical
 machines (6 nodes)
 - one of them are LXC containers located on different physical
 machines and XEN VMs (6 nodes)
 - and the last of them are VMware ESX VMs (3 nodes)

 Our application works correctly on the first four clusters, but it
 doesn't work as we expected on the last one.

 When we update a key and we retrieve this key in order to write it
 again, it has an old value (it doesn't have the first value that we
 wrote), for example:

 The key has: lalala
 We retrieve the key, and add lololo, so it should be lalala,lololo
 We retrieve the key again, and try to add lelele, so it should be now:
 lalala,lololo,lelele, but when we retrieve it again, we only have:
 lalala,lelele

 In the second write action, when we retrieve the key, we obtained a
 key with the old value. We set r, w, pr and rw to 3 to the REST
 requests, but it doesn't help.

 All the configuration files are very similiar and we don't have any
 major differences in the disk I/O and network performance of the nodes
 of the clusters.

 Did anyone have a similar issue?

 Regards.

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Weird RIAK behavior

[Sargun Dhillon]

What versions of Riak are you using? And are these CRDT sets?

Sent from my iPhone

 On Dec 22, 2014, at 16:04, Claudio Cesar Sanchez Tejeda 
 demoncc...@gmail.com wrote:

 I'm a sysadmin and I managing 5 cluster of RIAK:

 - two of them are LXC containers on the same physical machine (3 nodes
 per cluster)
 - one of them are LXC containers located on different physical
 machines (6 nodes)
 - one of them are LXC containers located on different physical
 machines and XEN VMs (6 nodes)
 - and the last of them are VMware ESX VMs (3 nodes)

 Our application works correctly on the first four clusters, but it
 doesn't work as we expected on the last one.

 When we update a key and we retrieve this key in order to write it
 again, it has an old value (it doesn't have the first value that we
 wrote), for example:

 The key has: lalala
 We retrieve the key, and add lololo, so it should be lalala,lololo
 We retrieve the key again, and try to add lelele, so it should be now:
 lalala,lololo,lelele, but when we retrieve it again, we only have:
 lalala,lelele

 In the second write action, when we retrieve the key, we obtained a
 key with the old value. We set r, w, pr and rw to 3 to the REST
 requests, but it doesn't help.

 All the configuration files are very similiar and we don't have any
 major differences in the disk I/O and network performance of the nodes
 of the clusters.

 Did anyone have a similar issue?

 Regards.

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak errors after node left the cluster

[Sargun Dhillon]

So, from my understanding, one of your servers was being replaced, so
you did a leave from the cluster, and it failed to commit, and then
another node failed, resulting in 3/4ish or 3/5ish of the ring being
up?
Did you down the failed node, or remove it from the cluster? What's
the current status of your ring?

Can you run:
riak-admin status (on each node)
riak-admin diag
riak-admin member-status
riak-admin ring-status
riak-admin vnode-status
riak-admin transfers
riak-admin transfer-limit

Can you put this all in a github gist: gist.github.com?
Also, are you running AAE?

You might want to temporarily turn off AAE, and transfers as you debug
the cluster, just to figure out what's going on with the cluster at
the moment.

On Thu, Nov 6, 2014 at 1:05 PM, Oleksiy Krivoshey oleks...@gmail.com wrote:
 Just got a new problem with Riak. Recently a hard drive has failed on one of
 Riak nodes so I had to shut it down. I'm running 4 nodes now and each 10
 minutes all of them start to fail with 'Error: {error,mailbox_overload}'
 until restarted. Can anyone from Basho please suggest a solution/ fix for
 this? My whole cluster is unusable with just 1 node failed.

 On 5 November 2014 00:11, Oleksiy Krivoshey oleks...@gmail.com wrote:

 There were also errors during initial handoff, here is a full console.log
 for that day: https://www.dropbox.com/s/o7zop181pvpxoa5/console.log?dl=0

 I actually replaced two nodes that day. First one went smoothly as it
 should. The second one resulted in the situation above. I replaced the first
 one and then the second after few hours.

 On 4 November 2014 20:44, Oleksiy Krivoshey oleks...@gmail.com wrote:

 Hi,

 I'm running a 5 node cluster (Riak 2.0.0) and I had to replace hardware
 on one of the servers. So I did a 'cluster leave', waited till the node
 exited, checked the ring status and members status, all was ok, with no
 pending changes. Then later after about 5 minutes every client connection to
 any of the 4 remaining nodes started to fail with

 [Error: {error,mailbox_overload}

 I have restarted one node after another and the error has gone. However I
 was still experiencing connectivity issues (timeouts) and riak error log is
 full of various errors even after I joined the 5th node back.

 Error are like:

 Failed to merge
 {[/var/lib/riak/bitcask_expire_1d/685078892498860742907977265335757665463718379520/1.bitcask.data]

 gen_fsm 0.818.0 in state active terminated with reason: bad record
 state in riak_kv_vnode:set_vnode_forwarding/2 line 991

 @riak_pipe_vnode:new_worker:826 Pipe worker startup failed:


 msg,7,[{file,gen_fsm.erl},{line,505}]},{proc_lib,init_p_do_apply,3,[{file,proc_lib.erl},{line,239}]}]
 2014-11-04 16:07:57.124 [error]
 0.11128.0@riak_core_handoff_sender:start_fold:279 hinted_handoff transfer
 of riak_kv_vnode from 'riak@10.0.1.1'
 353957427791078050502454920423474793822921162752 to 'riak@
 10.0.1.5' 353957427791078050502454920423474793822921162752 failed because
 of error:undef
 [{riak_core_format,human_size_fmt,[~.2f,588],[]},{riak_core_handoff_sender,start_fold,5,[{file,src/riak_core_han
 doff_sender.erl},{line,246}]}]

 The full error log file is available here:
 https://www.dropbox.com/s/3b8x3nqyego7lw3/error.log?dl=0

 There was no significant load on Riak so I would like to understand what
 caused so many errors?

 --
 Oleksiy




 --
 Oleksiy Krivoshey




 --
 Oleksiy Krivoshey

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Crypto with R16B02-basho5

[Sargun Dhillon]

How did you install R17? kerl? If so, it silently doesn't install
crypto if you don't have libssl-dev installed, see here:
https://github.com/evax/kerl/issues/13

On Wed, Nov 5, 2014 at 12:28 AM, Abd El-Fattah Mahran
abdoo.mah...@gmail.com wrote:

 Hi,

 I downloaded .deb package with R17 and it is working with no error at all.

 riak@oodba-debian:~$ /usr/sbin/riak start
 
  WARNING: ulimit -n is 1024; 65536 is the recommended minimum.
 
 riak@oodba-debian:~$ /usr/sbin/riak attach
 Remote Shell: Use Ctrl-C a to quit. q() or init:stop() will terminate the
 riak node.
 Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [async-threads:10]
 [kernel-poll:false]

 Eshell V5.10.3  (abort with ^G)
 (riak@127.0.0.1)1


 I wanted to compile from source and I can't while I'm having R17. I
 downloaded the recommend version for RIAK and i got crypto error

 erix@oodba-debian:~/riak$ erl
 Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [async-threads:10]
 [hipe] [kernel-poll:false]

 Eshell V5.10.3  (abort with ^G)
 1 application:start(crypto).
 ok
 2
 BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
(v)ersion (k)ill (D)b-tables (d)istribution
 ^Cerix@oodba-debian:~/riak$


 erix@oodba-debian:~/riak$ make all
 ./rebar get-deps
 Uncaught error in rebar_core: {'EXIT',
{undef,
 [{crypto,start,[],[]},
  {rebar,run_aux,2,[]},
  {rebar,main,1,[]},
  {escript,run,2,
   [{file,escript.erl},{line,747}]},
  {escript,start,1,
   [{file,escript.erl},{line,277}]},
  {init,start_it,1,[]},
  {init,start_em,1,[]}]}}
 make: *** [deps] Error 1
 erix@oodba-debian:~/riak$



 erix@oodba-debian:~$ lsb_release -a
 No LSB modules are available.
 Distributor ID: Debian
 Description:Debian GNU/Linux 7.7 (wheezy)
 Release:7.7
 Codename:   wheezy
 erix@oodba-debian:~$


 Thanks
 Best Regards,
 Abd El-Fattah Mahran
 http://www.linkedin.com/in/abdoomahran


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Creating a new object immediately after deleting one with the same key

[Sargun Dhillon]

Distributed deletion, and garbage collection is a really hard problem.
Riak has a couple different ways to do it talked about here:
http://docs.basho.com/riak/latest/ops/advanced/deletion/

The default mechanism tombstones the key, and then waits 3 seconds
after the write has reached stable state (on all the primary vnodes of
the preflist), and then blows it away. Although, this is configurable,
it's probably not what you want to change. You should pass the
deletedvclock=true, when doing the get, and then pass back the vclock
on write. There is more information in the linked doc.

On Thu, Oct 9, 2014 at 1:23 AM,  t...@fastmail.fm wrote:
 Hi all,

 While testing a simple go / protobuf driver I encountered following
 behaviour:
 After deleting an object and creating a new one with the same key
 immediately afterwards Riak creates a sibling of the new object. Trying to
 fetch an object with the key directly before creating the new one fails.

 If the timespan between delete and create is greater than ~3 seconds
 everything works as expected.

 Is this behaviour documented somewhere? Does the required minimal timespan
 depend on the cluster setup or is it configurable?
 I'm currently using Riak v2.0.0 on OS X locally, allow_mult is set to true.

 Best regards,
 Tilo Soenke

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Connecting to a single host vs balancing requests

[Sargun Dhillon]

When you do read / modify / writes, are you also planning on sending
the relevant read through one node only? In that case, your update
latency might suffer if the egress queues of your designated node get
backed up on writes, waiting for a very low cost read query.

You're more likely to get awkward load on riak1, and the vnodes that
that node hosts are going to suffer as a result. Although, you may not
see that in the common case where n=3, and r=quorum, (r=2), but if
another vnode in a preflist is degraded, it's likely to be
problematic.

There are potential performance benefits given some very specific
workloads where you can sacrifice a node, or alternatively you can
configure Riak not to put any vnodes on that node, but these are
advanced options that are not for the faint of heart.

My question would be why are you doing this?

P.S. The way you asked your question of the list is fine.

On Tue, Oct 7, 2014 at 2:09 AM, Geoff Garbers ge...@totalsend.com wrote:
 Hi all.

 Apologies if I'm not using the mailing list correctly - this is the
 first time I'm posting to a mailing list.

 We're in the process of redeveloping our systems using Riak, and will
 be using five nodes initially. Let's call these nodes riak1 through to
 riak5. Our read/write/delete distribution is weighted more towards
 reads than writes, with few deletes (if I have to hazard a guess, I
 would say 60/35/5 read/write/deletes).

 A technical decision has been made to ensure that all writes are sent
 to riak1, and all reads are done from riak2 through riak5. The read
 hosts are selected completely at random.

 Apart from the obvious write availability concern, is there any
 performance benefit or penalty to be had from writing to a specific
 node, and reading from the rest? I haven't been able to perform a test
 on this myself, so any input from the community would be appreciated
 for the time being.

 Thanks in advance,
 Geoff

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak for low latency/high write volumes

[Sargun Dhillon]

So, I've been doing some testing lately. Riak sounds like it'll meet
your usecases with some caveats. LevelDB on SSDs will handle change
often fairly well, as long as your disks have enough throughput, and
CPUs are large enough to handle the compaction. If you're in GCE, I
recommend using persistent SSDs only, and if you're in AWS, local SSDs
would be ideal. If your cluster can't keep up, you may see distributed
Erlang getting backed up, and that can be very problematic, as it can
raise latency for other transactions. This can be worked around by
building a Riak with the right amount of capacity. Other than that, I
don't find a workload with heavy updates too concerning.

For the second corpus of keys, I have some other concerns. It's
possible to do some of the stuff you're talking about with Riak 2.0.0
features, like strong-consistency. You could actually do RAMPs to get
what you're looking for without any external tools, or you could use
an external toolkit with Riak to get full serializability.

On Wed, Oct 1, 2014 at 2:20 AM, snacktime snackt...@gmail.com wrote:

 1) What are your requirements in terms of data durability?

 If we are talking write failures, then family 1 can have some loss in
 durability.   My caching layer has a write behind cache that already trades
 a few seconds of durability on node failure for the ability to buffer writes
 to the backend.  And for a rather significant part of the data in certain
 types of games that is acceptable.

 2) Do you have real hardware?

 Large games mostly yes.  Small games no.

 3) Do you the capability to use SSDs?

 Yes
 4) What's the size of your total keyspace? What about your working set?

 For larger games total keyspace would probably range from 10-40 million
 keys.   For a stable game with 10 million keys, I'd say an average might be
 around 200k active keys.  But a number of things can make that number swing
 up wildly over short periods of time.  Launch week, promotions, etc..  Also,
 it's worth noting that some values will increase in size over time as
 players collect more and more items.

 5) Do you need atomic, multi-key updates?

 Will always have a small set of stuff that requires this.

 Chris

 On Tue, Sep 30, 2014 at 11:53 PM, snacktime snackt...@gmail.com wrote:

 I'm going to be testing out Riak for use in an open source game server
 platform I created, and was looking for some tips on use and configuration.

 Read/write volumes as are normal for many games, is about 50/50.
 Consistent response times with low latency is a priority, as is not having
 eventually consistent get out of control like with what can happen with that
 'other' nosql db (ya another horror story not worth going into here).

 The data being saved would be a combination of json and binary.  It's
 protocol buffers and I provide the option to serialize to json or native
 protobuf.

 Currently all keys are compound.  A scope followed by a delimiter followed
 by the actual key.

 Usage patterns basically break down into two groups.  One is data that can
 change up to several times a second.  Local caching is available, but
 updates to the database every few seconds would be required.

 The other class of data does not change that often, and in some cases
 requires atomic updates.  This would be in game trading, etc..


 Any tips people have on getting the most out of Riak for this type of
 environment would be appreciated.

 Chris



 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak for low latency/high write volumes

[Sargun Dhillon]

So, from my understanding, there are two types of data that you have.
Family 1:
-Constantly changing
-Representable by CRDTs
-Can handle eventual consistency

Family 2:
-Rarely changing
-Need immediate consistency, and linearizability

With both key families, you care about tail latency.

Questions I have:
1) What are your requirements in terms of data durability?
2) Do you have real hardware?
3) Do you the capability to use SSDs?
4) What's the size of your total keyspace? What about your working set?
5) Do you need atomic, multi-key updates?

On Tue, Sep 30, 2014 at 11:53 PM, snacktime snackt...@gmail.com wrote:
 I'm going to be testing out Riak for use in an open source game server
 platform I created, and was looking for some tips on use and configuration.

 Read/write volumes as are normal for many games, is about 50/50.
 Consistent response times with low latency is a priority, as is not having
 eventually consistent get out of control like with what can happen with that
 'other' nosql db (ya another horror story not worth going into here).

 The data being saved would be a combination of json and binary.  It's
 protocol buffers and I provide the option to serialize to json or native
 protobuf.

 Currently all keys are compound.  A scope followed by a delimiter followed
 by the actual key.

 Usage patterns basically break down into two groups.  One is data that can
 change up to several times a second.  Local caching is available, but
 updates to the database every few seconds would be required.

 The other class of data does not change that often, and in some cases
 requires atomic updates.  This would be in game trading, etc..


 Any tips people have on getting the most out of Riak for this type of
 environment would be appreciated.

 Chris

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: replacing node results in error with diag

[Sargun Dhillon]

So, I don't have a ton of experience with Riaknostic, but taking a
casual glance at the source code, it appears that Riaknostic caches
some node-local data about the ring (see:
https://github.com/basho/riaknostic/blob/2.0.0/src/riaknostic_node.erl#L192-L208).
You should be able to unset this by attaching to a node riak attach
and running application:unset_env(riaknostic, local_stats). --
although, it'd be nice to get a dump of your local env first for
debugging purposes, you can get that via io:format(Local env: ~p~n,
[application:get_all_env(riaknostic)]). (including the period).

If that clears one node, you can do it on all of your nodes by issuing
rpc:multicall(application, unset_env, [riaknostic, local_stats]). on
one node.

On Tue, Sep 30, 2014 at 12:39 PM, Max Vernimmen
m.vernim...@comparegroup.eu wrote:
 Hi,



 Today I finished upgrading 2.0.0-pre20 to 2.0.0-1. Once that was done I did
 a node replace according to the instructions at
 http://docs.basho.com/riak/latest/ops/running/nodes/replacing/

 Once the replacing was done, our monitoring notified us about a problem with
 the cluster. Our monitoring does a ‘riak-admin diag’ and each of the nodes
 is now giving the output I’ve posted here:
 https://gist.github.com/anonymous/a313a07b0cd1da1c

 There is a node being referenced in the diag, which is the replaced node. It
 is no longer in the cluster. I confirmed the ring was settled and in the web
 interface of the cluster the replaced node is no longer listed neither is it
 in the `riak-admin status` output. Only a restart of the riak service on
 each of the nodes resolves the problem. Doing a restart on only one node
 fixes the diag status only for that node.



 To me it seems like there is some state left in the cluster nodes after a
 node is replaced, causing the `riak-admin diag` command to fail. Has anyone
 else seen this? Would this classify as a bug or did I simply do something
 wrong ? J



 Best regards,





 Max Vernimmen




 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: unable to delete certain keys in a bucket

[Sargun Dhillon]

So, if I'm interpreting your message correctly, you're executing the
following steps, with the following steps, in order?
1. Key delete, X - success
2. Key list - yields a set, which contains X
3. Key fetch, X - 404 / missing

I'd first ask why you're doing key listing. That's an anti-pattern.
You're going to get stale data that way. Alternatively, you could
probably set your delete_mode to immediate, and assuming the entire
preflist is available, it should result in key cleanup immediately.

If you want to simply reset your cluster, turn off all your Riak
nodes, and delete the bitcask directory, and the AAE directory. That
should clean up the cluster appropriately.

As far as your question if it's okay to delete an old backend - the
answer is yes. You should be able to delete /var/lib/riak/leveldb/* if
it's no longer in use.



On Mon, Sep 29, 2014 at 12:12 PM, Hristo Asenov
hristo.s.ase...@gmail.com wrote:
 I am using riak version 1.4.10 and it is in a cluster with two hosts. I am
 unable to get rid of keys left over from previous operations using simple
 delete operations on keys. When I list the keys for a bucket, it shows me
 the old keys, however if I try to retrieve the data associated with a key,
 no data is found. When I try to delete the key, it still persists.

 Is there a way to wipe the keys in the bucket so it starts from a clean
 slate? I don't care about any of the data in riak, but I would rather not
 have to reinstall everything again. What would happen If I just delete
 everything from /var/lib in both hosts?

 I also noticed that recently I did an update, and one of my clusters was
 using leveldb as a backend and another was using bitcask. I updated the
 configs so that both are using bitcask, but I am wondering if there is some
 stale data left over. In one of the clusters in /var/lib there is still a
 leveldb folder. Is it safe to delete it?

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: If I go to RICON 2014

[Sargun Dhillon]

Although, I'm not a Basho engineer, I work for a partner. I've
deployed Riak, and Riak-CS in production services. I promise you don't
have to stay in your corner if you want to talk about deploying, and
using Riak. You'll find me at the conference. There tends to be some
split of industry types, and academics. You'll have fun, I promise.

On Fri, Sep 19, 2014 at 3:04 AM, Chris Meiklejohn cmeiklej...@basho.com wrote:
 Hi Patrice,

 There will be plenty of Basho engineers there, including myself, who
 will be happy to answer any questions you may have.  We love to hear
 how people are using our product and want to help out in any way
 possible!

 - Chris


 On Fri, Sep 19, 2014 at 12:02 PM, Patrice Bruno
 patrice.br...@kuantic.com wrote:
 Hi,
 (my first post)
 if I go to RICON can I hope to meet people who can ask to my questions about
 our deployment and our usage of Riak, because I do not want to come to stay
 in my corner ?

 Best Regards
 Patrice Bruno
 Kuantic

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Logical Export/Cluster Migration

[Sargun Dhillon]

If you upgrade to 2.0, then there is a ring-resize feature built right in.
That might be a better approach to take.


On Tue, Sep 2, 2014 at 10:07 AM, Mark Rechler mrech...@bitly.com wrote:

 Hello All,

 What would be the best tool for moving data from one riak cluster to
 another?

 To give some context, we needed to change the ring_creation_size and
 build out a new cluster.
 Once the cluster was built I initially tried riak-data-migrator
 (https://github.com/basho-labs/riak-data-migrator) but that caused the
 load across the old cluster to skyrocket and the tool would time out
 before it could actually receive any data.

 I then attempted riak-admin backup/restore. The backup took ~ 2 days.
 The restore seemed much slower going ~ 10 megabytes/hour. Needing to
 fill 26gb/node this seemed impractical.

 Are there any other recommended ways to migrate data? We're using
 leveldb as the only back-end. Appreciate any help.

 Thanks,
 Mark

 --
 Mark Rechler
 Infrastructure Engineer, Bitly

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ID generation techniques

[Sargun Dhillon]

I have questions for your question.

1. What are you using your keys for? Do they get passed around in to
clients in Javascript? This is important because Javascript only
reliably implements IEEE 754 floating point, which is limited to 53
bits of precision.

2. What backend are you using? In Bitcask, your keyspace is costly
whereas in LevelDB, keys on disk are cheap. Additionally, having
k-sorted writes helps your story around levelDB compactions?

3. Is there any reason you want them K-sorted, other than to make
LevelDB compactions a bit nicer?

On Wed, Aug 20, 2014 at 10:23 AM, Shailesh Mangal
shailesh.man...@getzephyr.com wrote:
 Hi,

 I wanted to ask what are popular, scalable ID generation techniques for
 storing data in RIAK. Some ideas that we are toying with:

 ID Generation server (like Flikr ticket server) - Numeric
 Twitter's Snowflake like implementation- Alphanumeric
 Riak’s auto ID generation- Alphanumeric


 So far, we are thinking of going with option 2 as it doesn’t require node
 coordination, k-sorted. But we would prefer a uncoordinated numeric Unique
 Ids.

 - Shailesh

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ID generation techniques

[Sargun Dhillon]

My recommendation would be to use GUIDs. Since they are in hex, it's
more difficult to accidentally make the mistake of turning them into a
number due to the nature of Javascript's type system. They do not rely
on an external system, and it is fairly cheap to generate them. If you
want to them to be k-sorted-ish (no guarantees), prefix the timestamp.
Of course, due to the nature of clocks, this can end poorly.

Since you want to write your code to be idempotent, probably as Riak
is a AP, KV, Dynamo-store, you probably want to rely on client-side
key generation.

A heads up though! Buckets are not free if you use custom properties
on Riak 1.X. Buckets are cheap on Riak 2.0 thanks to the work that
Jordan West has done on the metadata system, but again, not free.

On Wed, Aug 20, 2014 at 3:14 PM, Shailesh Mangal
shailesh.man...@getzephyr.com wrote:
 Thanks Sargun,


 1. Keys do get exposed to client. We recently learnt about this limitation
 of javascript and we will convert the keys to String when they get to
 javascript.
 2. Considering Bitcask as storage choice due to its performance, but we
 are open for levelDB as well. BTW, We already have tenantId (UUID) in the
 bucket name.
 3. Coming from RDBMS background, we are used to seeing incremental Ids.
 K-sort is not a hard requirement but nice to have.

 On 8/20/14, 11:03 AM, Sargun Dhillon sar...@sargun.me wrote:

I have questions for your question.

1. What are you using your keys for? Do they get passed around in to
clients in Javascript? This is important because Javascript only
reliably implements IEEE 754 floating point, which is limited to 53
bits of precision.

2. What backend are you using? In Bitcask, your keyspace is costly
whereas in LevelDB, keys on disk are cheap. Additionally, having
k-sorted writes helps your story around levelDB compactions?

3. Is there any reason you want them K-sorted, other than to make
LevelDB compactions a bit nicer?

On Wed, Aug 20, 2014 at 10:23 AM, Shailesh Mangal
shailesh.man...@getzephyr.com wrote:
 Hi,

 I wanted to ask what are popular, scalable ID generation techniques for
 storing data in RIAK. Some ideas that we are toying with:

 ID Generation server (like Flikr ticket server) - Numeric
 Twitter's Snowflake like implementation- Alphanumeric
 Riak¹s auto ID generation- Alphanumeric


 So far, we are thinking of going with option 2 as it doesn¹t require
node
 coordination, k-sorted. But we would prefer a uncoordinated numeric
Unique
 Ids.

 - Shailesh

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak 2.0 - java client 'unavailable' exception

[Sargun Dhillon]

Is the requirement for having AAE enabled now removed for strong consistency?

On Mon, Jul 28, 2014 at 4:55 PM, Joseph Blomstedt j...@basho.com wrote:
 This means the consistency sub-system is not enabled/active. You can
 verify this with the output of `riak-admin ensemble-status`.

 To enable strong consistency you must:

 1) Set 'strong_consistency = on' in riak.conf.
 2) Have at least a 3 node cluster.

 You can address #2 by setting up 3+ local developer nodes as detailed
 in the 5 minute tutorial:
 http://docs.basho.com/riak/2.0.0/quickstart

 Alternatively, you can override the need for 3 nodes and use 1 node.

 To do that,

 1) Run 'riak attach' to attach to your Riak's node console
 2) Enter (including the period): riak_ensemble_manager:enable().
 3) Enter (including the period): riak_core_ring_manager:force_update().
 3) Detach from the console using: Ctrl-C a

 After either approach, re-check `riak-admin ensemble-status`. It may
 take up to a minute for the consistency sub-system to be enabled.

 If you haven't already, please take a look at the temporary (until we
 finish updating docs.basho.com) strong consistency related
 documentation (linked from the 2.0 RC1 release notes) here:
 https://github.com/basho/riak_ensemble/blob/wip/riak-2.0-user-docs/riak_consistent_user_docs.md

 Regards,
 Joe

 On Mon, Jul 28, 2014 at 3:05 PM, Jason W jason.w.p...@gmail.com wrote:
 Hi,

 I am trying out 2.0 w/ just one local node, created a strongly consistent
 bucket type.  But keep getting below exception.  If I just use the default
 bucket type, everything works fine.  Here is the bucket type detail with
 consistency bit on.

 young_vclock: 20
 w: quorum
 small_vclock: 50
 rw: quorum
 r: quorum
 pw: 0
 precommit: []
 pr: 0
 postcommit: []
 old_vclock: 86400
 notfound_ok: true
 n_val: 1
 linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}
 last_write_wins: false
 dw: quorum
 dvv_enabled: true
 chash_keyfun: {riak_core_util,chash_std_keyfun}
 big_vclock: 50
 basic_quorum: false
 allow_mult: true
 consistent: true
 active: true
 claimant: 'riak@0.0.0.0'

 Here is the java code

 ListString addresses = new LinkedListString();

 addresses.add(172.16.0.254);

 RiakClient  riakClient = RiakClient.newClient(addresses);

 try {

 Location wildeGeniusQuote = new Location(new
 Namespace(strongly_consistent2, sample), emp.getId());

 BinaryValue text =
 BinaryValue.create(objectMapper.writeValueAsBytes(sampleObj));

 RiakObject obj = new RiakObject()

 .setContentType(text/plain)

 .setValue(text);

 StoreValue store = new
 StoreValue.Builder(obj).withLocation(wildeGeniusQuote)

 .withOption(Option.ASIS, true)

 .withOption(Option.DW, new Quorum(1))

 .withOption(Option.IF_NONE_MATCH, true)

 .withOption(Option.IF_NOT_MODIFIED, true)

 .withOption(Option.PW, new Quorum(1))

 .withOption(Option.N_VAL, 1)

 .withOption(Option.RETURN_BODY, true)

 .withOption(Option.RETURN_HEAD, true)

 .withOption(Option.SLOPPY_QUORUM, true)

 .withOption(Option.TIMEOUT, 1000)

 .withOption(Option.W, new Quorum(1))

 .build();

 riakClient.execute(store);

 } catch (Exception e) {

 e.printStackTrace();

 return null;

 }

 Am I still missing something? Thanks.



 Caused by: com.basho.riak.client.core.netty.RiakResponseException:
 unavailable

 at
 com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:52)

 at
 io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)

 at
 io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:326)

 at
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:155)

 at
 io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:108)

 at
 io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)

 at
 io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:326)

 at
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)

 at
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116)

 at
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494)

 at
 io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461)

 at
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)

 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)

 at
 io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)

 ... 1 more


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




 --
 Joseph Blomstedt j...@basho.com
 Principal Engineer
 Basho Technologies, Inc.
 http://www.basho.com/

 ___
 riak-users mailing list
 

Re: Cluster security

[Sargun Dhillon]

You really should have some level of IP filtering to prevent people
from connecting directly to your BEAM / EPM instances, but even if
they do have the ability to make a TCP/IP connection, they have to
know the distributed Erlang cookie in order to connect. More on this:
http://www.erlang.org/doc/reference_manual/distributed.html - See,
section 13.7. You can actually use inet_tls for communication between
your VMs to ensure authenticity of clients, as well as
confidentiality.

On Mon, Jun 30, 2014 at 2:47 AM, Alex De la rosa
alex.rosa@gmail.com wrote:
 Hi there,

 Imaging the following IP configuration for Riak 2.0:

 # /etc/riak/riak.conf

 nodename = riak@111.222.333.1
 listener.http.internal = 127.0.0.1:8098
 listener.protobuf.internal = 127.0.0.1:8087
 listener.https.internal = 127.0.0.1:8098

 I put the server's real IP in the nodename so they can join/communicate with
 each other but then I limit any HTTP/PB communication to the localhost so
 nobody can mess with the node from outside (we assume I have an own Python
 API in each node doing the security and being a middleman between the
 cluster and the webapp).

 But how can I avoid 3rd-parties to just build a Riak server themselves and
 join my cluster without permission... they can freely join like this:

 # riak-admin join riak@111.222.333.1

 Of course, they will have to find out the IP address, but if they do, they
 can simply put a server themselves in the cluster and do whatever they want
 with it.

 Maybe a solution is creating subdomains on my domain just for the riak-ring
 in a way that is extremely hard to find out to be able to do the JOIN,
 something like: nodename = r...@rk001blahblahblah.mydomain.com

 Is this approach reasonable? Am I doing/thinking something very wrong? What
 would be the suggested way to prevent undesired JOINs?

 Cheers,
 Alex

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Consistent Riak (riak_ensemble) without Anti-entropy?

[Sargun Dhillon]

So, I noticed that if I don't have anti-entropy on, and I enable
strongly consistent Riak, it doesn't work. Specifically, what happens
is that riak_kv_ensembles sets up the ensembles, but the
riak_ensemble_peer never gets past to all_sync state. It appears that
this is because the riak_kv_ensemble_backend relies on anti-entropy to
perform an exchange before it comes up. See here, from
riak_kv/develop:


sync(Replies, State=#state{ensemble=_Ensemble, id=Id}) -
Peers0 = [{Idx, PeerId} || {PeerId={{kv,_PL,_N,Idx},_Node},_Reply}
- Replies],
Peers = orddict:from_list(Peers0),
{{kv, PL, N, Idx}, _} = Id,
IndexN = {PL,N},
%% Sort to remove duplicates when changing ownership / forwarded response
Siblings0 = lists:usort([I || {{{kv,_PL,_N,I},_Node},_Reply} - Replies]),
%% Just in case, remove self from list
Siblings = Siblings0 -- [Idx],

case local_partition(Idx) of
true -
T0 = erlang:now(),
Pid = self(),
spawn_link(fun() -
   wait_for_sync(Idx, IndexN, Pid, T0,
Siblings, Peers)
   end),
{async, State};
false -
{ok, State}
end.

wait_for_sync(Idx, IndexN, Pid, T0, Siblings, Peers) -
Exchanges = riak_kv_entropy_info:exchanges(Idx, IndexN),
Recent = [OtherIdx || {OtherIdx, T1, _} - Exchanges,
  T1  T0],
lager:info(~p/~p: Exchanges: ~p~nT0: ~p~nRecent: ~p~nSibs: ~p,
[Idx, IndexN, Exchanges, T0, Recent, Siblings]),
Need = length(Siblings),
Finished = length(Recent),
Local = local_partition(Idx),
Complete = ((Siblings -- Recent) =:= []),
if not Local -
lager:info(Partition ownership changed. No need to sync.),
riak_ensemble_backend:sync_complete(Pid, []);
   Complete -
lager:info(Complete ~b/~b :: ~p - ~p~n, [Finished,
Need, Idx, Pid]),
SyncPeers = [orddict:fetch(PeerIdx, Peers) || PeerIdx - Siblings],
riak_ensemble_backend:sync_complete(Pid, SyncPeers);
   true -
lager:info(Not yet ~b/~b :: ~p, [Finished, Need, Idx]),
timer:sleep(1),
wait_for_sync(Idx, IndexN, Pid, T0, Siblings, Peers)
end.


(I uncommented the debugging). If riak_kv_entropy_manager is not
enabled, then riak_kv_entropy_info:exchanges will always be empty. Can
we either (1) manually trigger AAE exchange upon noticing that strong
consistency is enabled (I imagine you can do this by setting the mode
to manual, and then queueing up the AAE jobs), (2) throw a warning to
the user saying that they should enable AAE.

-Sargun

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Consistent Riak (riak_ensemble) without Anti-entropy?

[Sargun Dhillon]

I've gone ahead and opened up an issue on the riak_kv github project:
https://github.com/basho/riak_kv/issues/959 - as it appears like this
is an actual bug. My preference would be that when
riak_kv_ensemble_backend is booting up, it initiates manual AAE sync,
so I don't have to wait for AAE exchange to occur, as on a real
cluster, this might take a while.

On Wed, May 28, 2014 at 3:24 AM, Sargun Dhillon sar...@sargun.me wrote:
 So, I noticed that if I don't have anti-entropy on, and I enable
 strongly consistent Riak, it doesn't work. Specifically, what happens
 is that riak_kv_ensembles sets up the ensembles, but the
 riak_ensemble_peer never gets past to all_sync state. It appears that
 this is because the riak_kv_ensemble_backend relies on anti-entropy to
 perform an exchange before it comes up. See here, from
 riak_kv/develop:


 sync(Replies, State=#state{ensemble=_Ensemble, id=Id}) -
 Peers0 = [{Idx, PeerId} || {PeerId={{kv,_PL,_N,Idx},_Node},_Reply}
 - Replies],
 Peers = orddict:from_list(Peers0),
 {{kv, PL, N, Idx}, _} = Id,
 IndexN = {PL,N},
 %% Sort to remove duplicates when changing ownership / forwarded response
 Siblings0 = lists:usort([I || {{{kv,_PL,_N,I},_Node},_Reply} - Replies]),
 %% Just in case, remove self from list
 Siblings = Siblings0 -- [Idx],

 case local_partition(Idx) of
 true -
 T0 = erlang:now(),
 Pid = self(),
 spawn_link(fun() -
wait_for_sync(Idx, IndexN, Pid, T0,
 Siblings, Peers)
end),
 {async, State};
 false -
 {ok, State}
 end.

 wait_for_sync(Idx, IndexN, Pid, T0, Siblings, Peers) -
 Exchanges = riak_kv_entropy_info:exchanges(Idx, IndexN),
 Recent = [OtherIdx || {OtherIdx, T1, _} - Exchanges,
   T1  T0],
 lager:info(~p/~p: Exchanges: ~p~nT0: ~p~nRecent: ~p~nSibs: ~p,
 [Idx, IndexN, Exchanges, T0, Recent, Siblings]),
 Need = length(Siblings),
 Finished = length(Recent),
 Local = local_partition(Idx),
 Complete = ((Siblings -- Recent) =:= []),
 if not Local -
 lager:info(Partition ownership changed. No need to sync.),
 riak_ensemble_backend:sync_complete(Pid, []);
Complete -
 lager:info(Complete ~b/~b :: ~p - ~p~n, [Finished,
 Need, Idx, Pid]),
 SyncPeers = [orddict:fetch(PeerIdx, Peers) || PeerIdx - 
 Siblings],
 riak_ensemble_backend:sync_complete(Pid, SyncPeers);
true -
 lager:info(Not yet ~b/~b :: ~p, [Finished, Need, Idx]),
 timer:sleep(1),
 wait_for_sync(Idx, IndexN, Pid, T0, Siblings, Peers)
 end.


 (I uncommented the debugging). If riak_kv_entropy_manager is not
 enabled, then riak_kv_entropy_info:exchanges will always be empty. Can
 we either (1) manually trigger AAE exchange upon noticing that strong
 consistency is enabled (I imagine you can do this by setting the mode
 to manual, and then queueing up the AAE jobs), (2) throw a warning to
 the user saying that they should enable AAE.

 -Sargun

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RIAK best practices for storage

[Sargun Dhillon]

How much deeper does your tree go? What's the average number of
children a node has? What is your query pattern (fetch a parent, and
all of its children?)?

On Fri, Apr 11, 2014 at 10:13 AM, Sapre, Meghna A
meghna.a.sa...@intel.com wrote:
 Hi all,

   Most of my data is in parent-child format (1:n).

 For read/write performance, is it better to store parents in a separate
 bucket and reference with IDs in children, or store complete copies of
 parents within children?



 Thanks,

 Meghna


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Randomizing Bitcask merges

[Sargun Dhillon]

If you can afford the disk space, and potential latency overheads, you
can make your bitcask merges less frequent. You can find a window when
your databases are going to be in low-usage, and have each node have a
separate window for bitcask merges, which (A) will latency-level
across your cluster, and (B) ensure that latency-leveling doesn't when
you don't want it to.

On Thu, Mar 13, 2014 at 10:49 AM, Matthew MacClary
maccl...@lifetime.oregonstate.edu wrote:
 I thought I would share part of one of our post install scripts with the
 community to get feedback. Our goal is to avoid big merges that affect the
 whole cluster at once. We set the merge trigger and merge threshold to the
 same value which should result in more frequent but smaller scoped merges.
 The trigger for each riak node is set randomly between 50% and 85% for dead
 bytes and fragmentation. We also chose to use largish values for the
 threshold because that dictates how much of our disk bandwidth we will
 waste, if you merge after 10% dead bytes - then your are wasting your
 time copying 90% good bytes and only recovering 10% of the slab's disk
 space.

 What we were seeing in testing is that soon after the cluster was installed,
 all the nodes were doing heavy merges at the same time. This created a very
 noticeable bad performance window. Over time the merges would start to
 spread out a little bit due to random factors. I should also point out that
 our work mix is basically 1:1:1:0 for write:read:delete:update   so merging
 happens a lot.

 We decided to make the merges smaller and start the cluster out with
 randomized merges. If you read with a consistency value of 1, then there is
 a good chance that if one replica is merging, then another replica vnode
 would not be part of a merge and could reply with normal latency.

 Let me know what you think! Is any of this reasoning faulty?

 # values used to compute values for app.config file
 NUMERATOR=2147483648   # matches max_file_size
 DENOMINATOR=100

 THRESH=`shuf -i 50-85 -n 1`
 #BYTES=$(expr $THRESH \* 2147483648 / 100 )
 BYTES=$(expr $THRESH \* $NUMERATOR / $DENOMINATOR )

 sed -ri s/(.*dead_bytes_merge_trigger,)\d*.*$/\1${BYTES}},/
 ${RIAK_CONFIG_STAGING}/app.config
 sed -ri s/(.*dead_bytes_threshold,)\d*.*$/\1${BYTES}},/
 ${RIAK_CONFIG_STAGING}/app.config

 sed -ri s/(.*frag_merge_trigger,)\d*.*$/\1${THRESH}},/
 ${RIAK_CONFIG_STAGING}/app.config
 sed -ri s/(.*frag_threshold,)\d*.*$/\1${THRESH}},/
 ${RIAK_CONFIG_STAGING}/app.config


 -Matt

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Custom hashing (partitioning) functions

[Sargun Dhillon]

Cool, I went ahead and created an issue a little while ago:
https://github.com/basho/riak_kv/issues/811

It's not so much of a blocker for us, because the product team working
with custom partitioners and Riak. It's just something that people
might want to keep an eye out in the future when using MR, or
riak_pipe directly.

As as side-note, is there information around querying Riak directly
with Riak_pipe; I imagine it has some of the same downsides as
mapreduce. We're primarily using it as a method to avoid the
read-modify-write overheads on writing to Riak, and multi-key,
partition-specific reads (utilizing targeted 2i, because we know the
coverage as we know the partition key).

Has anyone else here done this in production?

On Tue, Jan 21, 2014 at 7:25 AM, Sean Cribbs s...@basho.com wrote:
 Sargun,

 That might indeed be a bug. Would you file it on the issue tracker?
 https://github.com/basho/riak_kv/issues


 On Mon, Jan 20, 2014 at 3:08 PM, Sargun Dhillon sar...@sargun.me wrote:

 So, I don't know how many people are aware of if, but Riak supports
 custom hashing (partitioning) functions. It's exposed as a bucket
 property (chash_keyfun), in which you can deploy your own code to hash
 keys to ensure data locality to specific vnodes. This can come in
 handy when doing custom mapreduce jobs, riak_pipe, or riak_core
 applications.

 After playing around with this feature set quite a bit recently, it
 seems like there are some places that don't entirely respect this.
 Specifically, the place where I'm looking at right now is
 riak_kv_pipe_get, when try_preflist gets called (if try_partition
 falls through), it falls back to riak_core_util:chash_key (L188), as
 opposed to looking at the riak_pipe chashfun, or the bucket chashfun.
 What's the approach that most people have taken here with custom hash
 functions?

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




 --
 Sean Cribbs s...@basho.com
 Software Engineer
 Basho Technologies, Inc.
 http://basho.com/

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Search and Yokozuna Backup Strategy

[Sargun Dhillon]

Not to fork the thread too far from the topic being discussed, but is
there any possibility of opening up the API used for multidatacenter
replication? Specifically, the fullsync API? I imagine the code inside
riak_repl can also be used for an external node to connect and get a
full dump of a node's content using fullsync. Incremental backups
could potentially be taken by using the AAE strategy, by the backup
sink building a merkle tree of the data it has, and using that to
generate the keylist of deltas.

Unfortunately, riak_repl is not open source, so this is something
Basho would have to build.

On Thu, Jan 23, 2014 at 12:57 PM, Dave Martorana d...@flyclops.com wrote:
 I like that HyperDex provides direct backup support instead of simply
 suggesting a stop-filecopy-start-catchup scenario. Are there any plans at
 Basho to make backups a core function of Riak (or as a separate but included
 utility) - it would certainly be nice to have something Basho provides help
 ensure things are done properly each time, all the time.

 Cheers,

 Dave


 On Thu, Jan 23, 2014 at 1:42 PM, Joe Caswell jcasw...@basho.com wrote:

 Apologies, clicked send in the middle of an incomplete thought.  It should
 have read:

 Backing up the LevelDB data files while the node is stopped would remove
 the necessity of using the LevelDB repair process upon restoring to make the
 vnode self-consistent.

 From: Joe Caswell jcasw...@basho.com
 Date: Thursday, January 23, 2014 1:25 PM
 To: Sean McKibben grap...@graphex.com, Elias Levy
 fearsome.lucid...@gmail.com

 Cc: riak-users@lists.basho.com riak-users@lists.basho.com
 Subject: Re: Riak Search and Yokozuna Backup Strategy

 Backing up LevelDB data files can be accomplished while the node is
 running if the sst_x directories are backed up in numerical order.  The
 undesirable side effects of that could be duplicated data, inconsistent
 manifest, or incomplete writes, which necessitates running the leveldb
 repair process upon restoration for any vnode backed up while the node was
 running.  Since the data is initially written to the recovery log before
 being appended to level 0, and any compaction operation fully writes the
 data to its new location before removing it from its old location, if any of
 these operations are interrupted, the data can be completely recovered by
 leveldb repair.

 The only incomplete write that won't be recovered by the LevelDB repair
 process is the initial write to the recovery log, limiting exposure  to the
 key being actively written at the time of the snapshot/backup.  As long as 2
 vnodes in the same preflist are not backed up while simultaneously writing
 the same key to the recovery log (i.e. rolling backups are good), this key
 will be recovered by AAE/read repair after restoration.

 Backing up the LevelDB data files while the node is stopped would remove
 the necessity of repairing the

 Backing up Riak Search data, on the other hand, is a dicey proposition.
 There are 3 bits to riak search data: the document you store, the output of
 the extractor, and the merge index.

 When you put a document in key in a bucket with search
 enabled, Riak uses the pre-defined extractor to parse the document into
 terms, possibly flattening the structure, and stores the result in
 _rsid_bucket/key, which is used during update operations to
 remove stale entries before adding new ones, and would most likely be stored
 in a different vnode, possibly on a different node entirely.  The document
 id/link is inserted into the merge index entry for each term identified by
 the extractor, any or all of which may reside on different nodes.  Since the
 document, its index document, and the term indexes could not be guaranteed
 to be captured in any single backup operation, it is a very real probability
 that these would be out of sync in the event that a restore is required.

 If restore is only required for a single node, consistency could be
 restored by running a repair operation for each riak_kv vnode and
 riak_search vnode stored on the node, which would repair the data from other
 nodes in the cluster.  If more than one node is restored, it is quite likely
 that they both stored replicas of the same data, for some subset of the full
 data set.  The only way to ensure consistency is fully restored in the
 latter case is to reindex the data set.  This can be accomplished by reading
 and  rewriting all of the data, or by reindexing via MapReduce as suggested
 in this earlier mailing list post:
 http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-October/009861.html

 In either restore case, having a backup of the merge_index data files is
 not helpful, so there does not appear to be any point in backing them up.

 Joe Caswell
 From: Sean McKibben grap...@graphex.com
 Date: Tuesday, January 21, 2014 1:04 PM
 To: Elias Levy fearsome.lucid...@gmail.com
 Cc: riak-users@lists.basho.com riak-users@lists.basho.com
 Subject: Re: Riak Search and Yokozuna Backup 

Custom hashing (partitioning) functions

[Sargun Dhillon]

So, I don't know how many people are aware of if, but Riak supports
custom hashing (partitioning) functions. It's exposed as a bucket
property (chash_keyfun), in which you can deploy your own code to hash
keys to ensure data locality to specific vnodes. This can come in
handy when doing custom mapreduce jobs, riak_pipe, or riak_core
applications.

After playing around with this feature set quite a bit recently, it
seems like there are some places that don't entirely respect this.
Specifically, the place where I'm looking at right now is
riak_kv_pipe_get, when try_preflist gets called (if try_partition
falls through), it falls back to riak_core_util:chash_key (L188), as
opposed to looking at the riak_pipe chashfun, or the bucket chashfun.
What's the approach that most people have taken here with custom hash
functions?

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com