Re: File corruption after power cut

[Guido Medina]

Bryan* (misspelled your name), need a morning coffee :(

On 04/09/2019 10:29, Guido Medina wrote:
Thanks a lot Brian, I have deleted such directory and now the node 
started, let's see how it behaves.


Guido.

On 04/09/2019 10:19, Bryan Hunt wrote:
Easiest solution is to just delete the vnode - and let it recover 
from replicas - vnode directory will be 
844930634081928249586505293914101120738586001408


On 4 Sep 2019, at 10:01, Guido Medina <mailto:gmed...@temetra.com>> wrote:


Hi all,

We had a power cut which caused one of the nodes to corrupt one of 
the LevelDB files, after this that node doesn't even want to start, 
here is the error we are seeing:
2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 
Failed to start riak_kv_eleveldb_backend backend for index 
844930634081928249586505293914101120738586001408 error: 
{db_open,"Corruption: CURRENT file does not end with newline"}


Thanks in advance for your help ;-)
Guido.
___
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



Code Sync & Erlang Solutions Conferences

Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66scls>- 
Budapest: 20 September 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66scvd>- New 
York City: 01 October 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-08-22/69x6h1>- 
Berlin: 11 October 2019


RabbitMQ Summit 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sd8l>- 
London: 4 November 2019


Code Mesh LDN 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sd8x>- 
London: 7-8 November 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-08-22/69x6jc>- 
Bangalore: 14 November 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sdbs>- 
Amsterdam: 29 November 2019


Lambda Days 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sdcd>- 
Kraków: 13-14 February 2020


Code BEAM SF 
<https://www2.erlang-solutions.com/l/23452/2019-08-22/69x6cm>- San 
Francisco: 5-6 March 2020


Code BEAM STO - Stockholm: 28-29 May 2020

/Erlang Solutions cares about your data and privacy; please find all 
details about the basis for communicating with you and the way we 
process your data in our //Privacy Policy/ 
<https://www.erlang-solutions.com/privacy-policy.html>/.You can 
update your email preferences or opt-out from receiving Marketing 
emails here <http://www2.erlang-solutions.com/emailpreference>./






___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: File corruption after power cut

[Guido Medina]

Thanks a lot Brian, I have deleted such directory and now the node 
started, let's see how it behaves.


Guido.

On 04/09/2019 10:19, Bryan Hunt wrote:
Easiest solution is to just delete the vnode - and let it recover from 
replicas - vnode directory will be 
844930634081928249586505293914101120738586001408


On 4 Sep 2019, at 10:01, Guido Medina <mailto:gmed...@temetra.com>> wrote:


Hi all,

We had a power cut which caused one of the nodes to corrupt one of 
the LevelDB files, after this that node doesn't even want to start, 
here is the error we are seeing:
2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 
Failed to start riak_kv_eleveldb_backend backend for index 
844930634081928249586505293914101120738586001408 error: 
{db_open,"Corruption: CURRENT file does not end with newline"}


Thanks in advance for your help ;-)
Guido.
___
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



Code Sync & Erlang Solutions Conferences

Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66scls>- 
Budapest: 20 September 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66scvd>- New 
York City: 01 October 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-08-22/69x6h1>- Berlin: 
11 October 2019


RabbitMQ Summit 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sd8l>- London: 
4 November 2019


Code Mesh LDN 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sd8x>- London: 
7-8 November 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-08-22/69x6jc>- 
Bangalore: 14 November 2019


Code BEAM Lite 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sdbs>- 
Amsterdam: 29 November 2019


Lambda Days 
<https://www2.erlang-solutions.com/l/23452/2019-06-24/66sdcd>- Kraków: 
13-14 February 2020


Code BEAM SF 
<https://www2.erlang-solutions.com/l/23452/2019-08-22/69x6cm>- San 
Francisco: 5-6 March 2020


Code BEAM STO - Stockholm: 28-29 May 2020

/Erlang Solutions cares about your data and privacy; please find all 
details about the basis for communicating with you and the way we 
process your data in our //Privacy Policy/ 
<https://www.erlang-solutions.com/privacy-policy.html>/.You can update 
your email preferences or opt-out from receiving Marketing emails here 
<http://www2.erlang-solutions.com/emailpreference>./




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

File corruption after power cut

[Guido Medina]

Hi all,

We had a power cut which caused one of the nodes to corrupt one of the 
LevelDB files, after this that node doesn't even want to start, here is 
the error we are seeing:
2019-09-04 08:46:41.584 [error] <0.2329.0>@riak_kv_vnode:init:527 
Failed to start riak_kv_eleveldb_backend backend for index 
844930634081928249586505293914101120738586001408 error: 
{db_open,"Corruption: CURRENT file does not end with newline"}


Thanks in advance for your help ;-)
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Wants/Choose claim functions

[Guido Medina]

Hi Luke,

That old e-mail is what I was looking for, many thanks !!!
I had already found the FAQ

Guido.

On 19/02/2019 14:35, Luke Bakken wrote:

Hi Guido,

Searching google for "riak wants_claim_fun" turns up this document.
Please see the section "Q: A node left the cluster before handing off
all data. How can I resolve this?"

https://docs.basho.com/riak/kv/2.2.3/developing/faq/

This may be helpful as well -

http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/038435.html

On Tue, Feb 19, 2019 at 3:22 AM Guido Medina  wrote:

Hi,

Can someone please point me out to the guide explaining how to change wants and 
choose claim functions for Riak distribution % distribution?

We would like to set these permanently and trigger a cluster redistribution, I 
just can't find that documentation anymore, I was able to find this but I can't 
remember how to use it:

{wants_claim_fun, {riak_core_claim, wants_claim_v3}},
{choose_claim_fun, {riak_core_claim, choose_claim_v3}}


Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Wants/Choose claim functions

[Guido Medina]

Hi,

Can someone please point me out to the guide explaining how to change 
wants and choose claim functions for Riak distribution % distribution?


We would like to set these permanently and trigger a cluster 
redistribution, I just can't find that documentation anymore, I was able 
to find this but I can't remember how to use it:

|{wants_claim_fun,{riak_core_claim,wants_claim_v3}},{choose_claim_fun,{riak_core_claim,choose_claim_v3}}|


Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Guido Medina]

We could probably go the "replace" way, but maybe we take this as a good 
opportunity to increase our ring size, it is still something we are 
considering.


Resizing the cluster while operating daily is no joke, too much data and 
Riak becomes extremely slow when we have to add a new node, so it would 
either go the "replace" way or replicate and switch to the new cluster.


Thanks for all the answers ;-)
Guido.

On 01/02/2019 13:50, Nicholas Adams wrote:


If he has an extra node he can add then a replace would be much 
better. I provided this example under the assumption that he had no 
additional resources to play with.


Nicholas

*From:* riak-users  *On Behalf Of 
*Fred Dushin

*Sent:* 01 February 2019 22:47
*To:* riak-users@lists.basho.com
*Subject:* Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Wouldn't it be better to do a `riak-admin replace`?  Leave could be 
problematic if there are other nodes in the cluster that are 
under-provisioned (disk space, for example).  Plus a leave and add 
would move the data around the cluster twice, for each node in the 
cluster, whereas a replace would just move the data to the new node 
once, no?


-Fred



On Feb 1, 2019, at 8:32 AM, Nicholas Adams mailto:nicholas.ad...@tiot.jp>> wrote:

Hi Guido,

Although Martin would be the most qualified to comment on this, I
believe you should be able to do a slow migration.

 1. Choose target node.
 2. Make target node leave cluster as in a full “riak-admin
cluster leave”, commit and wait for transfers to finish.
 3. Set up target node with leveled and TicTac AAE.
 4. Have node rejoin cluster and wait for transfers to finish.
 5. Repeat with every single node in the cluster until all have
been done.

Unless you are using specific features restricted to your current
backend then Riak will usually put up with multiple backends in
the same cluster.

Failing that, I’d go with Bryan’s suggestion to use MDC to
replicate from your existing cluster to a separate cluster that is
using the leveled backend and TicTac AAE.

Either way, be sure to try in a dev environment first and only
proceed when you are happy with the process.

Best regards,

Nicholas

*From:*riak-users mailto:riak-users-boun...@lists.basho.com>>*On Behalf Of*Bryan Hunt
*Sent:*01 February 2019 19:22
*To:*Guido Medina mailto:gmed...@temetra.com>>
*Cc:*riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
*Subject:*Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

Replication would be the optimum solution - in theory anything can
be implemented - but it would be enormously painful in comparison
to simply standing up a new cluster.




    On 1 Feb 2019, at 10:14, Guido Medina mailto:gmed...@temetra.com>> wrote:

Hi all,

Nice work on the upcoming 2.9.0 release, I have a quick question:
Will it be possible to switch from the eleveldb to the new
leveled backend and Tictac AAE for an existing cluster?

In case it is not possible we are thinking to use the new
replication and move to a brand new cluster.

Kind regards,
Guido.


On 30/01/2019 15:23, Nicholas Adams wrote:

Dear All,

Following on from Martin’s email, the initial builds of
the KV 2.9.0rc1 packages are complete. Currently we are
limited to Redhat/CentOS 6 and 7 as well as Ubuntu Bionic.
We shall be adding more packages over the next few days.
Please seehttps://files.tiot.jp/riak/kv/2.9/2.9.0rc1/to
download.

Disclaimer: This is a release candidate. Please do not use
in production as we expect bugs and abnormal functionality
may occur in certain conditions. However, please test this
lots and place any issues on github so that they can be
fixed before the final release.

Best regards,

Nicholas

*From:*riak-users
<mailto:riak-users-boun...@lists.basho.com>*On Behalf
Of*Martin Sumner
*Sent:*30 January 2019 21:07
*To:*riak-users@lists.basho.com
<mailto:riak-users@lists.basho.com>
*Subject:*[ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for
Riak 2.9.0 for users to test in their own environment. The
release candidate is available to build from source here -
https://github.com/basho/riak/tree/riak-2.9.0rc1

There is only one significant change to Riak 2.2.6 for
users maintaining existing configuration settings.  The
release adds the `vnode_soft_limits` feature that aims to
reduce high percentile PUT latency, by chec

Re: [ANN] Riak 2.9.0 - Release Candidate 1 Available

[Guido Medina]

Hi all,

Nice work on the upcoming 2.9.0 release, I have a quick question:
Will it be possible to switch from the eleveldb to the new leveled 
backend and Tictac AAE for an existing cluster?


In case it is not possible we are thinking to use the new replication 
and move to a brand new cluster.


Kind regards,
Guido.

On 30/01/2019 15:23, Nicholas Adams wrote:


Dear All,

Following on from Martin’s email, the initial builds of the KV 
2.9.0rc1 packages are complete. Currently we are limited to 
Redhat/CentOS 6 and 7 as well as Ubuntu Bionic. We shall be adding 
more packages over the next few days. Please see 
https://files.tiot.jp/riak/kv/2.9/2.9.0rc1/ to download.


Disclaimer: This is a release candidate. Please do not use in 
production as we expect bugs and abnormal functionality may occur in 
certain conditions. However, please test this lots and place any 
issues on github so that they can be fixed before the final release.


Best regards,

Nicholas

*From:* riak-users  *On Behalf Of 
*Martin Sumner

*Sent:* 30 January 2019 21:07
*To:* riak-users@lists.basho.com
*Subject:* [ANN] Riak 2.9.0 - Release Candidate 1 Available

All,

There is now a publicly available release candidate for Riak 2.9.0 for 
users to test in their own environment.  The release candidate is 
available to build from source here - 
https://github.com/basho/riak/tree/riak-2.9.0rc1


There is only one significant change to Riak 2.2.6 for users 
maintaining existing configuration settings.  The release adds the 
`vnode_soft_limits` feature that aims to reduce high percentile PUT 
latency, by checking the outstanding work queue for a vnode before 
selecting it as a coordinator of a PUT.


With additional configuration, there are two major features added in 
riak 2.9.0:


- Support for the leveled backend (as an alternative to bitcask or 
eleveldb), to provide for improved throughput in some use cases and 
lower tail latency in some use cases - 
https://github.com/martinsumner/riak_testing_notes/blob/master/Release%202.9%20-%20Choosing%20a%20Backend.md;


- Support for a new anti-entropy mechanism (Tictac AAE) as an 
alternative for the existing anti-entropy mechanism (to provide both 
intra-cluster and inter-cluster data repair, with greater flexibility 
and reliability).


The release also adds support for:

- The riak_core node_worker_pool - which provides a mechanism for 
queueing background jobs and queries to control the resource consumed 
on the node by different queries.  No pre-existing features will use 
the node-worker_pool.


- AAE folds which allow for both cluster-wide AAE queries (e.g. 
produce a merkle tree representing all or a partial range of the 
cluster data), and administrative queries (e.g. discovering object 
sizes and sibling counts within the database depending on bucket name, 
key range and last modified date).  AAE folds depend on the Tictac AAE 
feature being activated.


- An interface to request re-replication of an object (where variances 
between clusters have been discovered).


Further details of the release, and the release plan in general can be 
found here - 
https://github.com/basho/riak/blob/develop-2.9/doc/Release%202.9%20Series%20-%20Overview.md.


It is hoped that there will be a short period of about 1 month before 
the release candidate will be converted into a formal release.  The 
period will allow for more testing by Riak users, and also there will 
be further enhanced testing of the new modules with the help of Quviq 
(http://www.quviq.com/).


There will follow additional releases under the 2.9 banner, targeted 
at enhancing both new and existing inter-cluster replication 
features.  In parallel to this, work will continue on Release 3.0 
which is intended to modernise the OTP/rebar platform used by Riak.


Thanks to all those who contributed to the release.  Apologies to 
those who have been kept waiting over the past few months as 
finalising the changes and completing the testing has dragged on.


Regards

Martin


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

The last command should have been like the following:

 * {ok, Ring} = riak_core_ring_manager:get_my_ring().
 * Partitions = [P || {P, 'r...@node4.domain.com'} <-
   riak_core_ring:all_owners(Ring)].
 * *[riak_kv_vnode:repair(P) || P <- Partitions].*

My bad, I think I copied the command from the wrong instructions before.

Guido.

On 01/06/18 09:37, Russell Brown wrote:

I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?


On 1 Jun 2018, at 09:35, Guido Medina  wrote:

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell



On 1 Jun 2018, at 09:07, Guido Medina 
  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:


(r...@node4.domain.com
)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1


The first two steps for the node repair executed fine:


{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '
r...@node4.domain.com
'} <- riak_core_ring:all_owners(Ring)].


We are running on 2.2.5
Guido.
___
riak-users mailing list

riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

What did I copied and pasted, sorry about that, it must be that it is 
Friday or something...my apologies.


On 01/06/18 09:37, Russell Brown wrote:

I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?


On 1 Jun 2018, at 09:35, Guido Medina  wrote:

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell



On 1 Jun 2018, at 09:07, Guido Medina 
  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:


(r...@node4.domain.com
)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1


The first two steps for the node repair executed fine:


{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '
r...@node4.domain.com
'} <- riak_core_ring:all_owners(Ring)].


We are running on 2.2.5
Guido.
___
riak-users mailing list

riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition


Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell


On 1 Jun 2018, at 09:07, Guido Medina  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:

(r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'r...@node4.domain.com'} <- 
riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Partitions repair isn't working anymore?

[Guido Medina]

Hi Russell,

I was repairing each node as specified in this guide 
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition


Guido.

On 01/06/18 09:16, Russell Brown wrote:

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that 
you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell


On 1 Jun 2018, at 09:07, Guido Medina  wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes 
out of 7 done), then we started getting this error:

(r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'r...@node4.domain.com'} <- 
riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Partitions repair isn't working anymore?

[Guido Medina]

Hi all,

We started the partitions repair a couple of weeks ago, so far so good 
(3 nodes out of 7 done), then we started getting this error:
(r...@node4.domain.com)3> [riak_search_vnode:repair(P) || P <- 
Partitions].

** exception error: undefined function riak_search_vnode:repair/1


The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'r...@node4.domain.com'} <- 
riak_core_ring:all_owners(Ring)].


We are running on 2.2.5
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Guido Medina]

Yes, but we haven't set back R = 2, we are currently with R = 3 until we 
resolve the issue.
I'm in the middle of writing something that will read every single key 
for the cluster using our relation DB IDs (each key has a counterpart on DB)


I'm guessing that 2i streaming won't help? we need to make sure we are 
fully consistent (for RW = 2) so I have to find a way to repair 
everything before turning R to 2.


After reading the object yes every request after was successful, but we 
even had a weird situation where after writing key with W = 2 we 
couldn't find with R = 2, not good for us.


Guido.

On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request?

Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?


On 18 May 2018, at 13:13, Russell Brown <russell.br...@icloud.com> wrote:

But why isn’t read repair “working”?


On 18 May 2018, at 11:07, Bryan Hunt <bryan.h...@erlang-solutions.com> wrote:

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.


On 18 May 2018, at 11:00, Bryan Hunt <bryan.h...@erlang-solutions.com> wrote:

Hi Guido,

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d 
'{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

 From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt


On 17 May 2018, at 15:42, Guido Medina <gmed...@temetra.com> wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

[Guido Medina]

A force replace done wrong causing some partitions to only have one copy.
And the cluster running extremely slow have forced us to disable AAE for 
now.


Guido.

On 18/05/18 13:13, Russell Brown wrote:

But why isn’t read repair “working”?


On 18 May 2018, at 11:07, Bryan Hunt <bryan.h...@erlang-solutions.com> wrote:

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.


On 18 May 2018, at 11:00, Bryan Hunt <bryan.h...@erlang-solutions.com> wrote:

Hi Guido,

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d 
'{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

 From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt


On 17 May 2018, at 15:42, Guido Medina <gmed...@temetra.com> wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

N = 3 and RW = 2 not finding some keys

[Guido Medina]

Hi all,

After some big rebalance of our cluster some keys are not found anymore 
unless we set R = 3, we had N = 3 and R = W = 2


Is there any sort of repair that would correct such situation for Riak 
2.2.3, this is really driving us nuts.


Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Java client question

[Guido Medina]

Also, a property annotated with a Riak annotation was kept in the Json 
only if it has been explicitly annotated with @JsonProperty or has not 
been annotated with a Riak property.


This was accomplished with the following code from the old client:


static {
RIAK_ANNOTATIONS.add(RiakKey.class);
RIAK_ANNOTATIONS.add(RiakIndex.class);
RIAK_ANNOTATIONS.add(RiakVClock.class);
RIAK_ANNOTATIONS.add(RiakTombstone.class);
RIAK_ANNOTATIONS.add(RiakUsermeta.class);
RIAK_ANNOTATIONS.add(RiakLinks.class);
}

private boolean keepProperty(BeanPropertyWriter beanPropertyWriter) {
if (beanPropertyWriter.getAnnotation(JsonProperty.class) != 
null) {

return true;
}
for (Class annotation : RIAK_ANNOTATIONS) {
if (beanPropertyWriter.getAnnotation(annotation) != null) {
return false;
}
}
return true;
}


So my other question is if this still holds true for the current Riak 
Java client 2.1.1?


On 22/06/17 09:49, Guido Medina wrote:

Hi,

I see now there is support for 2i which we needed in order to migrate 
to 2.x, there was another issue with the old client which forced us to 
modify the client, such issue was related to the following, let me put 
an example:


public class POJO {

  @RiakKey
  public String getKey() {
// generate our own key
  }

  @RiakIndex("some-index")
  @JsonIgnore
  public String getSomeIndex() {
// generate some index
  }

}

The following example should add "some-index" but ignore it as a Json 
property, and as you can see the Riak key is a pseudo (computed) 
property and has no setter, at some point Riak client 1.4.x required 
that the key be a property which IMHO is a very poor design to require 
the POJO to have some properties to hold internal Riak client values 
between calls.


I'm wondering what's the current POJO support as we have been stuck 
for years now with that old Riak client 1.4.x with our modifications, 
we are afraid that one day it will simply not be supported anymore by 
a newer Riak server.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak Java client question

[Guido Medina]

Hi,

I see now there is support for 2i which we needed in order to migrate to 
2.x, there was another issue with the old client which forced us to 
modify the client, such issue was related to the following, let me put 
an example:


public class POJO {

  @RiakKey
  public String getKey() {
// generate our own key
  }

  @RiakIndex("some-index")
  @JsonIgnore
  public String getSomeIndex() {
// generate some index
  }

}

The following example should add "some-index" but ignore it as a Json 
property, and as you can see the Riak key is a pseudo (computed) 
property and has no setter, at some point Riak client 1.4.x required 
that the key be a property which IMHO is a very poor design to require 
the POJO to have some properties to hold internal Riak client values 
between calls.


I'm wondering what's the current POJO support as we have been stuck for 
years now with that old Riak client 1.4.x with our modifications, we are 
afraid that one day it will simply not be supported anymore by a newer 
Riak server.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak issue after power outage

[Guido Medina]

Hi Fred,

This solved our issue, thanks a lot.

Guido.


On 01/06/17 13:26, Fred Dushin wrote:

There may be better ways, but what I would do is:

Stop Riak, blow away your ${platform_data_dir}/anti_entropy directory 
(and yz_anti_entropy, if you have search enabled), and restart.  This 
will force a rebuild of your hash trees.  They rebuild automatically 
anyway; this is just making them rebuild faster.


So it would recover .. eventually.

-Fred

On Jun 1, 2017, at 7:13 AM, Guido Medina <gmed...@temetra.com 
<mailto:gmed...@temetra.com>> wrote:


*Correction:* I replied to an old e-mail instead of creating a new 
one and forgot to change the subject.


Hi all,

My impression of Riak has always been that it would recover from 
anything but yesterday we had the worst happened, there was a power 
outage so all the servers in a Riak cluster went down, once they were 
back we have been having these constant so my questions are:


  * How can I recover from this?
  * Isn't Riak supposed to restore corrupted hashes from other nodes?


2017-06-01 12:07:43.795 [info] 
<0.659.0>@riak_kv_vnode:maybe_create_hashtrees:234 
riak_kv/1164634117248063262943561351070788031288321245184: unable to 
start index_hashtree: {error,{{badmatch,{error,{db_open,"Corruption: 
truncated record at end of 
file"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,712}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,565}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,308}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-06-01 12:07:45.457 [error] <0.27505.112> CRASH REPORT Process 
<0.27505.112> with 0 neighbours exited with reason: no match of 
right hand value {error,{db_open,"Corruption: truncated record at 
end of file"}} in hashtree:new_segment_store/2 line 725 in 
gen_server:init_it/6 line 328


This is not our production cluster but our development is now stuck 
because all the data we had there would take days to restore from 
another cluster, I hope there is a solvable problem.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak issue after power outage

[Guido Medina]

*Correction:* I replied to an old e-mail instead of creating a new one 
and forgot to change the subject.


Hi all,

My impression of Riak has always been that it would recover from 
anything but yesterday we had the worst happened, there was a power 
outage so all the servers in a Riak cluster went down, once they were 
back we have been having these constant so my questions are:


 * How can I recover from this?
 * Isn't Riak supposed to restore corrupted hashes from other nodes?

2017-06-01 12:07:43.795 [info] 
<0.659.0>@riak_kv_vnode:maybe_create_hashtrees:234 
riak_kv/1164634117248063262943561351070788031288321245184: unable to 
start index_hashtree: {error,{{badmatch,{error,{db_open,"Corruption: 
truncated record at end of 
file"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,712}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,565}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,308}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-06-01 12:07:45.457 [error] <0.27505.112> CRASH REPORT Process 
<0.27505.112> with 0 neighbours exited with reason: no match of right 
hand value {error,{db_open,"Corruption: truncated record at end of 
file"}} in hashtree:new_segment_store/2 line 725 in 
gen_server:init_it/6 line 328


This is not our production cluster but our development is now stuck 
because all the data we had there would take days to restore from 
another cluster, I hope there is a solvable problem.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Download link for basho-bench tool

[Guido Medina]

Hi all,

My impression of Riak has always been that it would recover from 
anything but yesterday we had the worst happened, there was a power 
outage so all the servers in a Riak cluster went down, once they were 
back we have been having these constant so my questions are:


 * How can I recover from this?
 * Isn't Riak supposed to restore corrupted hashes from other nodes?

2017-06-01 12:07:43.795 [info] 
<0.659.0>@riak_kv_vnode:maybe_create_hashtrees:234 
riak_kv/1164634117248063262943561351070788031288321245184: unable to 
start index_hashtree: {error,{{badmatch,{error,{db_open,"Corruption: 
truncated record at end of 
file"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,712}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,565}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,308}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-06-01 12:07:45.457 [error] <0.27505.112> CRASH REPORT Process 
<0.27505.112> with 0 neighbours exited with reason: no match of right 
hand value {error,{db_open,"Corruption: truncated record at end of 
file"}} in hashtree:new_segment_store/2 line 725 in 
gen_server:init_it/6 line 328


This is not our production cluster but our development is now stuck 
because all the data we had there would take days to restore from 
another cluster, I hope there is a solvable problem.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: speeding up bulk loading

[Guido Medina]

Hi Travis,

I have done similar things using the Java client but I will assume you 
have access to change certain settings at the C client, assuming you 
have RW = 2 and N =3, your client is returning to you once 2 writes are 
made but an asynchronous write is still pending which will eventually 
create lots of back pressure and overload your Riak nodes.


Concurrently speaking don't use more than 8 writer threads for this and 
set RW = N (in your case 3) for this tasks so that each write blocks, 
that way you won't hit a high back pressure.


Have some sort of blocking queue where you can only put up to (as an 
example) 500 tasks, 4 to 8 writer threads consuming from such queue, so 
here is a brief description of what I recommend:


 * RW = N (in this case 3) so that your write operation blocks and
   don't leave asynchronous tasks pending for the Riak cluster.
 * Don't exceed say, 8 writers (I have been there, more threads won't
   really help, sometimes 4 will just write faster than 8)
 * Have a bounded blocking queue (example max size = 500) where you
   schedule your tasks which the writer threads consume from.

Try that and see if that helps, I'm quite certain that at using such 
mechanism it will be consistent and your Riak nodes should never crash.


Best regards,

Guido.

On 06/09/16 16:42, Travis Kirstine wrote:


Thank Alexander, we’re using HAproxy

*From:*Alexander Sicular [mailto:sicul...@basho.com]
*Sent:* September-06-16 11:09 AM
*To:* Travis Kirstine 
*Cc:* Magnus Kessler ; riak-users@lists.basho.com
*Subject:* Re: speeding up bulk loading

Hi Travis,

I also want to confirm that you are spreading your load amongst all 
nodes in the cluster. You should be connecting your C client to Riak 
via a proxy like nginx/HAproxy/F5 [0]. The proxy will do a round 
robin/least connections distribution to all nodes in the cluster. This 
will greatly increase performance if you are not already doing it.


-alexander

[0] http://docs.basho.com/riak/kv/2.1.4/configuring/load-balancing-proxy/




Alexander Sicular

Solutions Architect

Basho Technologies
9175130679

@siculars

On Wed, Aug 31, 2016 at 10:41 AM, Travis Kirstine 
> wrote:


Magnus

Thanks for your reply.  We’re are using the riack C client library
for riak (https://github.com/trifork/riack) which is used within
an application called MapCache to store 256x256 px images with a
corresponding key within riak.  Currently we have 75 million
images to transfer from disk into riak which is being done
concurrently.  Periodically this transfer process will crash

Riak is setup using n=3 on 5 nodes with a leveldb backend.  Each
server has 45GB of memory and 16 cores with  standard hard
drives.  We made no significant modification to the riak.conf
except upping the leveldb.maximum_memory.percent to 70 and
tweeking the sysctl.conf as follows

vm.swappiness = 0

net.ipv4.tcp_max_syn_backlog = 4

net.core.somaxconn = 4

net.core.wmem_default = 8388608

net.core.rmem_default = 8388608

net.ipv4.tcp_sack = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_fin_timeout = 15

net.ipv4.tcp_keepalive_intvl = 30

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_moderate_rcvbuf = 1

# Increase the open file limit

# fs.file-max = 65536 # current setting

I have seen this error in the logs

2016-08-30 22:26:07.180 [error] <0.20777.512> CRASH REPORT Process
<0.20777.512> with 0 neighbours crashed with reason: no function
clause matching
webmachine_request:peer_from_peername({error,enotconn},

{webmachine_request,{wm_reqstate,#Port<0.2817336>,[],undefined,undefined,undefined,{wm_reqdata,'GET',...},...}})
line 150

Regards

*From:*Magnus Kessler [mailto:mkess...@basho.com
]
*Sent:* August-31-16 4:08 AM
*To:* Travis Kirstine >
*Cc:* riak-users@lists.basho.com 
*Subject:* Re: speeding up bulk loading

On 26 August 2016 at 22:20, Travis Kirstine
> wrote:

Is there any way to speed up bulk loading?  I wondering if I
should be tweeking the erlang, aae or other config options?

Hi Travis,

Excuse the late reply; your message had been stuck in the
moderation queue. Please consider subscribing to this list.

Without knowing more about how you perform bulk uploads, it's
difficult to recommend any changes. Are you using the HTTP REST
API or one of the client libraries, which use protocol buffers by
default? What concerns do you have about the upload performance?
Please let us know a bit more about your setup.

Kind Regards,

Riak Java client question

[Guido Medina]

Hi,

Are there any plans on releasing a Riak Java client with Netty-4.1.x?

The reasoning for this is that some projects like Vert.x 3.3.0 for 
example are already on Netty-4.1.x and AFAIK Netty's 4.1.x isn't just a 
drop in replacement for 4.0.x


Would it make sense to support another Riak Java client, say version 
2.1.x with Netty-4.1.x as a way to move forward?


Or maybe Riak 2.0.x works with Netty 4.1.x? but I doubt it.

Best regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak upgrade from 1.4.10 to 2.0.2 failed

[Guido Medina]

Hi,

Failed miserably trying to upgrade Riak 1.4.10 to 2.0.2 after created 
all the patch files to use the new riak.conf, I noticed that we don't 
need custom parameters any longer so the only thing we changed was the 
backend to be leveldb


Is it level DB max open files parameter needed any longer? I couldn't 
find it at the advanced config docs. Hence I'm not using it.


Any hints?

Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak upgrade from 1.4.10 to 2.0.2 failed

[Guido Medina]

Forgot to paste the error message:

   2014-11-13 14:16:06.524 [info] 0.7.0 Application mochiweb started
   on node nonode@nohost
   2014-11-13 14:16:06.528 [info] 0.7.0 Application webmachine
   started on node nonode@nohost
   2014-11-13 14:16:06.528 [info] 0.7.0 Application basho_stats
   started on node nonode@nohost
   2014-11-13 14:16:06.531 [info] 0.7.0 Application bitcask started
   on node nonode@nohost
   2014-11-13 14:16:06.531 [info] 0.7.0 Application eleveldb started
   on node nonode@nohost
   2014-11-13 14:16:06.531 [info] 0.7.0 Application pbkdf2 started on
   node nonode@nohost
   2014-11-13 14:16:06.531 [info] 0.7.0 Application poolboy started
   on node nonode@nohost
   2014-11-13 14:16:06.819 [error] 0.144.0 Supervisor riak_core_sup
   had child riak_core_broadcast started with
   riak_core_broadcast:start_link() at undefined exit with reason
   
{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['r...@x1.clonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',['r...@x6.clonmel.temetra.com',...]}]],...},...]}}
   in context start_error
   2014-11-13 14:16:06.819 [error] 0.142.0 CRASH REPORT Process
   0.142.0 with 0 neighbours exited with reason:
   
{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['r...@x1.clonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',...}]],...},...],...}
   in application_master:init/4 line 133
   2014-11-13 14:16:06.819 [info] 0.7.0 Application riak_core exited
   with reason:
   
{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['r...@x1.clonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',...}]],...},...],...}


On 13/11/14 14:32, Guido Medina wrote:

Hi,

Failed miserably trying to upgrade Riak 1.4.10 to 2.0.2 after created 
all the patch files to use the new riak.conf, I noticed that we don't 
need custom parameters any longer so the only thing we changed was the 
backend to be leveldb


Is it level DB max open files parameter needed any longer? I couldn't 
find it at the advanced config docs. Hence I'm not using it.


Any hints?

Regards,

Guido.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak upgrade from 1.4.10 to 2.0.2 failed

[Guido Medina]

Why is it Riak assuming the node is called nonode@nohost?

I didn't have to set that before and somehow my vm.args ended with the 
proper node name, in this case this one should be 
r...@x1.clonmel.temetra.com


Regards,

Guido.

On 13/11/14 14:34, Guido Medina wrote:

Forgot to paste the error message:

2014-11-13 14:16:06.524 [info] 0.7.0 Application mochiweb
started on node nonode@nohost
2014-11-13 14:16:06.528 [info] 0.7.0 Application webmachine
started on node nonode@nohost
2014-11-13 14:16:06.528 [info] 0.7.0 Application basho_stats
started on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application bitcask started
on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application eleveldb
started on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application pbkdf2 started
on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application poolboy started
on node nonode@nohost
2014-11-13 14:16:06.819 [error] 0.144.0 Supervisor riak_core_sup
had child riak_core_broadcast started with
riak_core_broadcast:start_link() at undefined exit with reason

{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['r...@x1.clonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',['r...@x6.clonmel.temetra.com',...]}]],...},...]}}
in context start_error
2014-11-13 14:16:06.819 [error] 0.142.0 CRASH REPORT Process
0.142.0 with 0 neighbours exited with reason:

{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['riak@x1.c

lonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',...}]],...},...],...}
in application_master:init/4 line 133
2014-11-13 14:16:06.819 [info] 0.7.0 Application riak_core
exited with reason:

{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['riak@x1.clonmel

.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',...}]],...},...],...}


On 13/11/14 14:32, Guido Medina wrote:

Hi,

Failed miserably trying to upgrade Riak 1.4.10 to 2.0.2 after created 
all the patch files to use the new riak.conf, I noticed that we don't 
need custom parameters any longer so the only thing we changed was 
the backend to be leveldb


Is it level DB max open files parameter needed any longer? I couldn't 
find it at the advanced config docs. Hence I'm not using it.


Any hints?

Regards,

Guido.




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak upgrade from 1.4.10 to 2.0.2 failed

[Guido Medina]

Disregard please, our patch file uses FQDN and it was pointing to the 
wrong file (vm.args)


Guido.

On 13/11/14 14:43, Guido Medina wrote:

Why is it Riak assuming the node is called nonode@nohost?

I didn't have to set that before and somehow my vm.args ended with the 
proper node name, in this case this one should be 
r...@x1.clonmel.temetra.com


Regards,

Guido.

On 13/11/14 14:34, Guido Medina wrote:

Forgot to paste the error message:

2014-11-13 14:16:06.524 [info] 0.7.0 Application mochiweb
started on node nonode@nohost
2014-11-13 14:16:06.528 [info] 0.7.0 Application webmachine
started on node nonode@nohost
2014-11-13 14:16:06.528 [info] 0.7.0 Application basho_stats
started on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application bitcask
started on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application eleveldb
started on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application pbkdf2 started
on node nonode@nohost
2014-11-13 14:16:06.531 [info] 0.7.0 Application poolboy
started on node nonode@nohost
2014-11-13 14:16:06.819 [error] 0.144.0 Supervisor
riak_core_sup had child riak_core_broadcast started with
riak_core_broadcast:start_link() at undefined exit with reason

{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['r...@x1.clonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',['r...@x6.clonmel.temetra.com',...]}]],...},...]}}
in context start_error
2014-11-13 14:16:06.819 [error] 0.142.0 CRASH REPORT Process
0.142.0 with 0 neighbours exited with reason:

{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['riak@x1.c

lonmel.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',...}]],...},...],...}
in application_master:init/4 line 133
2014-11-13 14:16:06.819 [info] 0.7.0 Application riak_core
exited with reason:

{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,[nonode@nohost,[{'r...@x1.clonmel.temetra.com',['r...@x2.clonmel.temetra.com','r...@x3.clonmel.temetra.com']},{'r...@x2.clonmel.temetra.com',['r...@x5.clonmel.temetra.com','r...@x6.clonmel.temetra.com']},{'r...@x3.clonmel.temetra.com',['riak@x1.clonmel

.temetra.com','r...@x2.clonmel.temetra.com']},{'r...@x5.clonmel.temetra.com',['r...@x3.clonmel.temetra.com','r...@x5.clonmel.temetra.com']},{'r...@x6.clonmel.temetra.com',...}]],...},...],...}


On 13/11/14 14:32, Guido Medina wrote:

Hi,

Failed miserably trying to upgrade Riak 1.4.10 to 2.0.2 after 
created all the patch files to use the new riak.conf, I noticed that 
we don't need custom parameters any longer so the only thing we 
changed was the backend to be leveldb


Is it level DB max open files parameter needed any longer? I 
couldn't find it at the advanced config docs. Hence I'm not using it.


Any hints?

Regards,

Guido.




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak API for Java

[Guido Medina]

You need to learn how to use Jackson, there is no other way around it, 
like you were told, you need a default constructor, or an annotated 
constructor to instruct Jackson how to instantiate your class, like the 
following example:



public class TestClass  {

  @JsonCreator
  public TestClass(
 @JsonProperty(property1) Type1 property1,
 @JsonProperty(property2) Type2 property2,
 @JsonProperty(property3) Type3 property3,
 @JsonProperty(property4) Type4 property4) {
...
...
  }
}


And here is an article about it, written by the Jackson author: 
http://www.cowtowncoder.com/blog/archives/2011/07/entry_457.html


P.D.: Did you take a look at the Exception when your code failed, 
Jackson exceptions can't be more clear...


Guido.

On 12/11/14 01:43, Ebbinge wrote:

Hello, I am trying to use the MapReduce function that comes along with the
Java API. I am using this code:

try{
 IRiakClient client = RiakFactory.pbcClient(192.168.0.41,
8087);
 Bucket myBucket = client.fetchBucket(Productos).execute();
//Bucket called Productos
 BucketMapReduce m = client.mapReduce(Productos);
 m.addMapPhase(new NamedJSFunction(Riak.mapValuesJson), true);
 MapReduceResult result = m.execute();
 //System.out.println(result.getResultRaw());
 CollectionProducto tmp = result.getResult(Producto.class);
 for (Producto p : tmp) {
 System.out.println(p.Nombre+\n);
 }

 client.shutdown();
 
 }
 
 catch(Exception Ex){

 System.out.println(Ex.getMessage());
 }

I have a Producto Class like this:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.util.StringTokenizer;
import javax.swing.JFileChooser;

/**
  *
  * @author edwinfernandez
  */
public class Producto {
 public String ID;
 public String Nombre;
 public String Descripcion;
 public String Vendedor;
 public String Url;
 public String Precio;
 
 public Producto(String id,String name,String desc,String seller,String

precio,String url){
 ID = id;
 Nombre = name;
 Descripcion = desc;
 Vendedor = seller;
 Precio = precio;
 Url = url;
 }
 public Producto(){
 
 }

}

I can't seem to make it work, I just need to mapreduce the description of
each product in order to get the top ten most used words to describe
products.

Thank you in advance.



--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Riak-API-for-Java-tp4032055.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak 2.x running with Riak Java client 1.4.x?

[Guido Medina]

Hi,

Is it possible to run the Riak Java client 1.4.x against Riak 2.x? At 
the moment we would have to do a major refactor in our application to 
support Riak Java client v2.x so I'm wondering if it is possible to 
first migrate to Riak v2.x before we start our refactor.


Best regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak key size estimation

[Guido Medina]

Hi,

Is there a way to estimate the size of a Riak Key using Riak Java client 
1.4.x, say from an instance of IRiakObject?


An approximation would be OK, I can probably get the Json's string and 
check its length but then there are 4 sizes to take into account right?


1. The Json string length.
2. The size of the binary object sent through the network using
   protocol buffers which maybe it is compressed?
3. The binary size before snappy compression?
4. The binary size after snappy compression?

In our case we are only interested in the size for LevelDB back-end, we 
want to be able to predict when keys are getting close to raise a big 
object warning at the Riak side and be able to do something before it 
can happen, say, split the keys, redesign etc.


Best regards,

Guido.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak VS Graph Databases

[Guido Medina]

Maybe what you are looking for is a combination of both, say, your KV 
data in Riak with a combination of background processes able to build 
the necessary searching graphs in Neo4J, in such way your data is secure 
in a Riak cluster and searchable on several Neo4J servers.


That's just an idea which might be not do-able, hope it helps,

Guido.

On 29/08/14 15:54, Alex De la rosa wrote:

Hi there,

For some time already I have in mind building a kind of social network 
myself. Is pretty ambitious project although it doesn't have in mind 
to be a new facebook; but still data will be quite big and complex.


I like Riak and I had been following since version 0.14, and new 
additions in Riak 2.0 seem to help a lot in how to model the data; 
although relationships will be unavoidable.


Some friends suggested me to use Graph Databases instead. How would 
Riak compare to Graph Databases for this use case? Is it doable to 
create a social network entirely from Riak? Or may not be recommended?


Thanks!
Alex


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak VS Graph Databases

[Guido Medina]

In a dream world my friend, we have Riak, PostgreSQL and Solr and might 
have to include a sort of query-able Big Table implementation in the 
future like Cassandra (we will try to avoid this last thing until we can't)


Your Graph DB will have trade off versus KV fetch in general, I don't 
think you have the tools you need in Riak to find the relationships per 
category nor the Riak advantages to do quick KV operations (data storage 
wise) nor cluster replication.


It won't be that simple without many trade off to build an homogeneous 
system.


Guido.

On 29/08/14 16:06, Alex De la rosa wrote:

Hi Guido,

This could be a solution; although I would try to do it in an 
homogeneous system where only one NoSQL DB would be around if possible :)


Thanks!
Alex


On Fri, Aug 29, 2014 at 5:03 PM, Guido Medina 
guido.med...@temetra.com mailto:guido.med...@temetra.com wrote:


Maybe what you are looking for is a combination of both, say, your
KV data in Riak with a combination of background processes able to
build the necessary searching graphs in Neo4J, in such way your
data is secure in a Riak cluster and searchable on several Neo4J
servers.

That's just an idea which might be not do-able, hope it helps,

Guido.


On 29/08/14 15:54, Alex De la rosa wrote:

Hi there,

For some time already I have in mind building a kind of social
network myself. Is pretty ambitious project although it doesn't
have in mind to be a new facebook; but still data will be quite
big and complex.

I like Riak and I had been following since version 0.14, and new
additions in Riak 2.0 seem to help a lot in how to model the
data; although relationships will be unavoidable.

Some friends suggested me to use Graph Databases instead. How
would Riak compare to Graph Databases for this use case? Is it
doable to create a social network entirely from Riak? Or may not
be recommended?

Thanks!
Alex


___
riak-users mailing list
riak-users@lists.basho.com  mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

repair-2i not working as expected?

[Guido Medina]

Hi,

We have had a problem now twice on Ubuntu 12.04 with MDM arrays (all 
nodes Raid 1 with 2x2TB disks) where once a disk fails 2i queries don't 
return the expected result, the only solution we have been able to apply 
is by replacing the affected node.


After doing an AAE clean up, repaired partitions and repaired 2i the 
problem still persists, would this be an indication of a problem with 
repair-2i not doing what is supposed to do?


Once a disk is gone, MDM says the array consist of only one disk and the 
server seems fine, keys are fine and no data is lost but it seems like 
2i once corrupted is never back to a correct state unless the 
problematic node is replaced.


We have had this problem with 1.4.7 (or 8) and now with 1.4.10

Any thoughts?

Best regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Australia data center

[Guido Medina]

Hi,

Does anyone know and can recommend a good -with price/value in mind- 
data center in Australia for hosting a Riak cluster? Our main data 
center is in Germany -http://www.hetzner.de- which is great, stable, 
fast and cheap but haven't had any luck finding something similar, and 
it has to be in Australia for data protection laws.


Best regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: repair-2i stops with bad argument in call to eleveldb:async_write

[Guido Medina]

Hi Simon,

There are some (maybe related) Level DB fixes in 1.4.9 and 1.4.10, I 
don't think there isn't any harm for you to do a rolling upgrade since 
nothing major changed, just bug fixes, here is the release notes' link 
for reference:


https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md

Best regards,

Guido.

On 29/07/14 09:35, Effenberg, Simon wrote:

Hi,

we have some issues with 2i queries like that:

seffenberg@kriak46-1:~$ while :; do curl -s 
localhost:8098/buckets/conversation/index/createdat_int/0/23182680 | ruby -rjson -e 
o = JSON.parse(STDIN.read); puts o['keys'].size; sleep 1; done

13853
13853
0
557
557
557
13853
0


...

So I tried to start a repair-2i first on one vnode/partition on one node
(which is quiet new in the cluster.. 2 weeks or so).

The command is failing with the following log entries:

seffenberg@kriak46-7:~$ sudo riak-admin repair-2i 
22835963083295358096932575511191922182123945984
Will repair 2i on these partitions:
 22835963083295358096932575511191922182123945984
Watch the logs for 2i repair progress reports
seffenberg@kriak46-7:~$ 2014-07-29 08:20:22.729 UTC [info] 
0.5929.1061@riak_kv_2i_aae:init:139 Starting 2i repair at speed 100 for 
partitions [22835963083295358096932575511191922182123945984]
2014-07-29 08:20:22.729 UTC [info] 
0.5930.1061@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition 
22835963083295358096932575511191922182123945984
2014-07-29 08:20:22.729 UTC [info] 
0.5930.1061@riak_kv_2i_aae:repair_partition:259 Repairing indexes in 
partition 22835963083295358096932575511191922182123945984
2014-07-29 08:20:22.740 UTC [info] 
0.5930.1061@riak_kv_2i_aae:create_index_data_db:324 Creating temporary 
database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
2014-07-29 08:20:22.751 UTC [info] 
0.5930.1061@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data 
for partition 22835963083295358096932575511191922182123945984
2014-07-29 08:25:22.752 UTC [info] 
0.5929.1061@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
 Total partitions: 1
 Finished partitions: 1
 Speed: 100
 Total 2i items scanned: 0
 Total tree objects: 0
 Total objects fixed: 0
With errors:
Partition: 22835963083295358096932575511191922182123945984
Error: index_scan_timeout


2014-07-29 08:25:22.752 UTC [error] 0.4711.1061 gen_server 0.4711.1061 terminated with reason: bad 
argument in call to eleveldb:async_write(#Ref0.0.10120.211816, , 
[{put,131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...,...}], 
[]) in eleveldb:write/3 line 155
2014-07-29 08:25:22.753 UTC [error] 0.4711.1061 CRASH REPORT Process 0.4711.1061 with 0 neighbours exited 
with reason: bad argument in call to eleveldb:async_write(#Ref0.0.10120.211816, , 
[{put,131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...,...}], 
[]) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
2014-07-29 08:25:22.753 UTC [error] 0.1031.0 Supervisor {0.1031.0,poolboy_sup} had child riak_core_vnode_worker 
started with {riak_core_vnode_worker,start_link,undefined} at 0.4711.1061 exit with reason bad argument in call to 
eleveldb:async_write(#Ref0.0.10120.211816, , 
[{put,131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...,...}], []) in 
eleveldb:write/3 line 155 in context child_terminated


Anything I can do about that? What's the issue here?

I'm using Riak 1.4.8 (.deb package).

Cheers
Simon
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak as log aggregator

[Guido Medina]

Hi Bryce,

There is very small chance a sessionID will repeat, let's assume that in 
theory it won't happen, that assume, there will be always a single 
writer per sessionID and such single writer will always happen to be the 
same writer, you want to lock using as locking key the session ID for 
two concurrent operations that might happen at the same time:


1. Add an entry to the sessionID and once it reaches the threshold+1.
2. Balance the key, meaning, get the last 2000 items and put them at
   the next sequence and at the current key keep only the last item.

If it wasn't because of balancing the keys the lock is not needed, and 
you don't want to balance while writing, you want add entries ASAP and 
them queue a request to redistribute/balance such sessionID.


If I were to give you a Java implementation of an entry, an entry should 
have a joda DateTime and a String and use DateTime for the hashcode() 
method and both for equals and DateTime for sorting, and instead of a 
List, use a sorted set so that unique and order are taken care of 
automatically.


The Session class then will have just equals and hashcode for the 
sessionID and have a SortedSetEntry


Hope that helps,

Guido.


On 25/04/14 00:02, Bryce Verdier wrote:
Thank you Jason and Guido for the quick responses and for helping me 
flesh out this idea.


Jason, Thank you for pointing out how you juggle a similar task, I'll 
look into the Set CDRT and hope that its able to get me somewhat close 
to what I was hoping for. One concern about your response that worries 
me though is this paragraph:


I like sorted JSON, but any data format that produces identical strings
would work.  If there is a chance of duplicate submissions into Riak,
you need to ensure the data format always produces identical output to
allow Riak to recognize and eliminate duplicates.

Would you mind elaborating on what you mean by that? In theory, this 
shouldn't be a problem as the sessionID's are unique (read: a very low 
probability of having a duplicate sessionID within a somewhat large 
period of time). But I just want to make sure I understand what you're 
saying there.


Warm regards,
Bryce


On 04/24/2014 01:13 AM, Guido Medina wrote:

Hi Bryce,

If each session ID is unique, even with multiple writers is unlikely 
for you to be writing to the same key at the same time from two 
different writers, that being the case, you could store each event as 
a JSON object into a JSON array and when your array reaches a 
threshold, say for a sake of an example; 2000, you could move those 
entries to another key, so your JSON will look like:


session12345 {
  next : 10,
  entries : [
 ...,...,...
  ]
}

session12345-10 {
  next : 9,
  entries : [
 ...,...,...
  ]
}

...
...
...

session12345-1 {
  entries : [
 ...,...,...
  ]
}

The reason while next is decrementing is because entries are sorted 
descending by time (newest events 1st) and balanced keys are inserted 
in between, so initially you have sesssion12345 and when you get to 
2001 events you have session12345 with next pointing to 1 (or 0 base) 
and sesssion12345-1, null if that session hasn't been balanced.


I'm using the hyphen as a convention, it can be any separator you want,

Hope that helps,

Guido.

On 24/04/14 04:29, Jason Campbell wrote:

Hi Bryce,

I have code that does something similar to this, and it works well.

In my case, the value is a JSON array, with a JSON object per event.

Siblings are easily resolved by merging the two arrays.

In Riak 2.0, sets using JSON-encoded strings would probably do this
automatically and more cleanly this manually resolving siblings.

I like sorted JSON, but any data format that produces identical strings
would work.  If there is a chance of duplicate submissions into Riak,
you need to ensure the data format always produces identical output to
allow Riak to recognize and eliminate duplicates.

The other thing I would worry about is how long-lived your sessions are
and how many events can they generate.  Riak starts having performance
issues over a few MB and you should probably consider another data
model at that point (maybe storing references instead of the data
itself).

Good luck with your project,
Jason

- Original Message -
From: Bryce bryceverd...@gmail.com
To: riak-users riak-users@lists.basho.com
Sent: Thursday, 24 April, 2014 1:22:12 PM
Subject: Riak as log aggregator

Hi All,

I'm interested in using Riak for a log aggregation project. These are
basically apache logs that I would like to correlate together based on
their session ID's. These session ID's would make sense as the key, but
its the value part of this that confuses me. There will be multiple
lines within these logs that have the same session ID, thus I will be
creating siblings. Now, is there a CRDT that will allow me to combine
all of these siblings into a single value or will I need to write my 
own
solution to do so? Any and all pointers are welcomed. Also, if Riak

Re: Riak as log aggregator

[Guido Medina]

Hi Bryce,

If each session ID is unique, even with multiple writers is unlikely for 
you to be writing to the same key at the same time from two different 
writers, that being the case, you could store each event as a JSON 
object into a JSON array and when your array reaches a threshold, say 
for a sake of an example; 2000, you could move those entries to another 
key, so your JSON will look like:


session12345 {
  next : 10,
  entries : [
 ...,...,...
  ]
}

session12345-10 {
  next : 9,
  entries : [
 ...,...,...
  ]
}

...
...
...

session12345-1 {
  entries : [
 ...,...,...
  ]
}

The reason while next is decrementing is because entries are sorted 
descending by time (newest events 1st) and balanced keys are inserted in 
between, so initially you have sesssion12345 and when you get to 2001 
events you have session12345 with next pointing to 1 (or 0 base) and 
sesssion12345-1, null if that session hasn't been balanced.


I'm using the hyphen as a convention, it can be any separator you want,

Hope that helps,

Guido.

On 24/04/14 04:29, Jason Campbell wrote:

Hi Bryce,

I have code that does something similar to this, and it works well.

In my case, the value is a JSON array, with a JSON object per event.

Siblings are easily resolved by merging the two arrays.

In Riak 2.0, sets using JSON-encoded strings would probably do this
automatically and more cleanly this manually resolving siblings.

I like sorted JSON, but any data format that produces identical strings
would work.  If there is a chance of duplicate submissions into Riak,
you need to ensure the data format always produces identical output to
allow Riak to recognize and eliminate duplicates.

The other thing I would worry about is how long-lived your sessions are
and how many events can they generate.  Riak starts having performance
issues over a few MB and you should probably consider another data
model at that point (maybe storing references instead of the data
itself).

Good luck with your project,
Jason

- Original Message -
From: Bryce bryceverd...@gmail.com
To: riak-users riak-users@lists.basho.com
Sent: Thursday, 24 April, 2014 1:22:12 PM
Subject: Riak as log aggregator

Hi All,

I'm interested in using Riak for a log aggregation project. These are
basically apache logs that I would like to correlate together based on
their session ID's. These session ID's would make sense as the key, but
its the value part of this that confuses me. There will be multiple
lines within these logs that have the same session ID, thus I will be
creating siblings. Now, is there a CRDT that will allow me to combine
all of these siblings into a single value or will I need to write my own
solution to do so? Any and all pointers are welcomed. Also, if Riak is a
bad fit for this, please let me know.

Warm regard,
Bryce


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Command line suggestions.

[Guido Medina]

Hi,

Base on the documentation and tricks I have seen to fix/repair stuff in 
Riak I would suggest the following approach:


   riak-admin [cluster] action (if cluster is specified then run the
   action at every node)

and I know most of the commands are designed like that but adding to 
them for specific actions will help admin to keep things tidy up:


 * riak-admin [cluster] repair-partitions - Instead of listing
   partitions per node via Erlang and running it at every node.
 * riak-admin [cluster] repair-2i - This is already part of riak-admin
   without the cluster level.
 * riak-admin [cluster] aae-cleanup - Stops AAE, deletes anti_entropy
   information and starts AAE, again; if cluster is specified, run it
   at every node.
 * riak-admin [cluster] aae-start | aae-stop - Handy when doing lots of
   hand-off, should be start-aae and stop-aae from the English
   perspective but then from the alphabetical perspective when
   documenting it can be confusing.

I do believe the 5 actions listed should all have a local vs cluster 
counterpart, don't get me wrong, Erlang is nice (I have plans to do deep 
Erlang stuff integrated into Riak and RabbitMQ for our application) but 
from the DevOps perspective, for scripting having each command line 
possible can save lot of time and avoid mistakes, like, what node am I, etc.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Choose/Wants claim functions.

[Guido Medina]

Hi,

What's the latest non-standard version of this function? v3 right? If 
Basho adds more versions to this, is this somewhere documented?


For our nodes standard choose/wants claim functions were doing a weird 
distribution so the numbers even out a bit better (just a bit better) by 
using v3, so it would be nice to know if improvements are done in this 
area and where they are being documented.


As of the latest 
http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/ 
both parameters have no default where my understanding is that the 
default for both is v2.


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Rebuilding AAE hashes - small question

[Guido Medina]

Thanks Engel,

That approach looks very accurate, I would only suggest to have a 
riak-admin cluster stop-aae and similar for start, for the dummies ;-)


Guido.

On 10/04/14 14:22, Engel Sanchez wrote:
Hey there. There are a couple of things to keep in mind when deleting 
invalid AAE trees from the 1.4.3-1.4.7 series after upgrading to 1.4.8:


* If AAE is disabled, you don't have to stop the node to delete the 
data in the anti_entropy directories
* If AAE is enabled, deleting the AAE data in a rolling manner may 
trigger an avalanche of read repairs between nodes with the bad trees 
and nodes with good trees as the data seems to diverge.


If your nodes are already up, with AAE enabled and with old incorrect 
trees in the mix, there is a better way.  You can dynamically disable 
AAE with some console commands. At that point, without stopping the 
nodes, you can delete all AAE data across the cluster.  At a 
convenient time, re-enable AAE.  I say convenient because all trees 
will start to rebuild, and that can be problematic in an overloaded 
cluster.  Doing this over the weekend might be a good idea unless your 
cluster can take the extra load.


To dynamically disable AAE from the Riak console, you can run this 
command:


 riak_core_util:rpc_every_member_ann(riak_kv_entropy_manager, 
disable, [], 6).


and enable with the similar:

 riak_core_util:rpc_every_member_ann(riak_kv_entropy_manager, enable, 
[], 6).


That last number is just a timeout for the RPC operation.  I hope this 
saves you some extra load on your clusters.



On Wed, Apr 9, 2014 at 11:02 AM, Luke Bakken lbak...@basho.com 
mailto:lbak...@basho.com wrote:


Hi Guido,

I specifically meant riak-admin transfers however using riak-admin
wait-for-service riak_kv riak@node is a good first step before
waiting for transfers.

Thanks!

--
Luke Bakken
CSE
lbak...@basho.com mailto:lbak...@basho.com


On Wed, Apr 9, 2014 at 7:54 AM, Guido Medina
guido.med...@temetra.com mailto:guido.med...@temetra.com wrote:

What do you mean by wait for handoff to finish?

Are you referring to wait for the service to be fully started?
i.e. riak-admin wait-for-service riak_kv riak@node

Or do you mean to check for riak-admin transfers on the
started node and wait until those handoffs/transfers are gone?

Guido.



On 09/04/14 15:46, Luke Bakken wrote:

Hi Guido,

That is the correct process. Be sure to use the rolling
restart procedure when restarting nodes (i.e. wait for
handoff to finish before moving on).

--
Luke Bakken
CSE
lbak...@basho.com mailto:lbak...@basho.com


On Wed, Apr 9, 2014 at 6:34 AM, Guido Medina
guido.med...@temetra.com mailto:guido.med...@temetra.com
wrote:

Hi,

If nodes are already upgraded to 1.4.8 (and they went all
the way from 1.4.0 to 1.4.8 including AAE buggy versions)

Will the following command (as root) on Ubuntu Servers 12.04:

riak stop; rm -Rf /var/lib/riak/anti_entropy/*; riak
start

executed on each node be enough to rebuild AAE hashes?

Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Choose/Wants claim functions.

[Guido Medina]

What would be a high ring size that would degrade performance for v3: 
128+? 256+?


I should had asked using the original response but I deleted it by accident.

Guido.

On 10/04/14 10:30, Guido Medina wrote:

Hi,

What's the latest non-standard version of this function? v3 right? If 
Basho adds more versions to this, is this somewhere documented?


For our nodes standard choose/wants claim functions were doing a weird 
distribution so the numbers even out a bit better (just a bit better) 
by using v3, so it would be nice to know if improvements are done in 
this area and where they are being documented.


As of the latest 
http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/ 
both parameters have no default where my understanding is that the 
default for both is v2.


Regards,

Guido.



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Rebuilding AAE hashes - small question

[Guido Medina]

Hi,

If nodes are already upgraded to 1.4.8 (and they went all the way from 
1.4.0 to 1.4.8 including AAE buggy versions)


Will the following command (as root) on Ubuntu Servers 12.04:

   riak stop; rm -Rf /var/lib/riak/anti_entropy/*; riak start

executed on each node be enough to rebuild AAE hashes?

Regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Rebuilding AAE hashes - small question

[Guido Medina]

What do you mean by wait for handoff to finish?

Are you referring to wait for the service to be fully started? i.e. 
riak-admin wait-for-service riak_kv riak@node


Or do you mean to check for riak-admin transfers on the started node 
and wait until those handoffs/transfers are gone?


Guido.


On 09/04/14 15:46, Luke Bakken wrote:

Hi Guido,

That is the correct process. Be sure to use the rolling restart 
procedure when restarting nodes (i.e. wait for handoff to finish 
before moving on).


--
Luke Bakken
CSE
lbak...@basho.com mailto:lbak...@basho.com


On Wed, Apr 9, 2014 at 6:34 AM, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:


Hi,

If nodes are already upgraded to 1.4.8 (and they went all the way
from 1.4.0 to 1.4.8 including AAE buggy versions)

Will the following command (as root) on Ubuntu Servers 12.04:

riak stop; rm -Rf /var/lib/riak/anti_entropy/*; riak start

executed on each node be enough to rebuild AAE hashes?

Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Provider

[Guido Medina]

Glad it helped, we have been using them for years, I would also 
recommend to use Ubuntu Server 12.04 LTS over there.


Regards,

Guido.

On 03/03/14 11:50, Massimiliano Ciancio wrote:

2014-02-21 15:44 GMT+01:00 Guido Medina guido.med...@temetra.com:

http://www.hetzner.de/en/
Really powerful servers, 500+ mbps inter server communication.

Thanks Guido,
I decided to use this solution.
5 servers (http://www.hetzner.de/hosting/produkte_rootserver/ex60)
directly connected to a private network by an 8-port gigabit switch at
an affordable price.
Thanks to all for the suggestions.
Massimiliano



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Provider

[Guido Medina]

http://www.hetzner.de/en/

Really powerful servers, 500+ mbps inter server communication.

Guido.

On 21/02/14 14:34, Massimiliano Ciancio wrote:

Hi all,
is there someone who have suggestions for a good, not so expensive,
European provider where to get 5 servers to install Riak?
Thanks
MC

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: RIAK API for Oracle

[Guido Medina]

The short answer is no, there is nothing that can fulfil your requirements

We developed something similar for PostgreSQL, we call it Hybrid DAOs, 
basically each POJO is annotated as Riak KV and also as Eclipselink JPA 
entity, I can only give you partial code and some hints.


You need a standard ORM, Eclipselink IMHO is perfect, faster, simpler 
and less cluttered than Hibernate, then using JPA interceptors you 
should be able to read/write your Object from Riak and the DB at the 
same time, finally, you need a cloning policy and use some hacks to make 
JPA write back, if you modify a non-mapped JPA property, it won't write 
back and hence won't call the interceptor, sample interceptor:


   public class JPA2RiakListenerK extends Comparable, T extends
   RiakKeyValueK, T
   {
  @PostLoad
  @SuppressWarnings(unchecked)
  public void onLoad(final T target) throws Exception
  {
final RiakKeyValueK, T
   
keyValue=getDAOFor(target.getClass()).fetchKeyValue(target.getKeyName(),target.getId());
if(keyValue != null){
  JPAQueryUtils.copyNotNullProperties(keyValue,target);
}
  }

  @PostPersist
  @PostUpdate
  @SuppressWarnings(unchecked)
  public void onPersistOrUpdate(final T target) throws Exception
  {
((RiakDAOK, T)
   getDAOFor(target.getClass())).saveKeyValue(target);
  }

  @PostRemove
  @SuppressWarnings(unchecked)
  public void onRemove(final T target) throws Exception
  {
   
getDAOFor(target.getClass()).deleteKeyValue(target.getKeyName(),target.getId());
  }
   }

Also, you need your mapped POJO's annotated with some clone policy, a 
clone policy will make sure non-mapped properties will be replicated, 
say, a column that is mapped to Riak but not mapped to your relational DB.


   @Entity
   @Table(name=foo)
   @EntityListeners(JPA2RiakListener.class)
   @CloneCopyPolicy(method=cloneThis,workingCopyMethod=cloneThis)
   public class Foo extends RiakKeyValueInteger, Foo {
   ...
   }

Sample cloneThis() method:

 public RiakKeyValueK, T cloneThis() throws Exception
 {
   @SuppressWarnings({unchecked})
   final RiakKeyValueK, T result=this.getClass().newInstance();
   JPAQueryUtils.copyNotNullProperties(this,result);
   return result;
 }

Hope that helps,

Guido.

On 05/02/14 10:06, Ajit Prasad(RMG00) wrote:


Hi,

We have a requirement for pulling Oracle data as JSON and putting the 
same in RIAK db. Didn't find any suitable RIAK API in RIAK docs. Pls. 
let me know whether any API or any third party tools which can perform 
the above activity.


Thanks and regards,

Ajit

Cell:+91-9980084384


~~Disclaimer~~~
Information contained and transmitted by this e-mail is confidential 
and proprietary to iGATE and its affiliates and is intended for use 
only by the recipient. If you are not the intended recipient, you are 
hereby notified that any dissemination, distribution, copying or use 
of this e-mail is strictly prohibited and you are requested to delete 
this e-mail immediately and notify the originator or 
mailad...@igate.com mailto:mailad...@igate.com. iGATE does not enter 
into any agreement with any party by e-mail. Any views expressed by an 
individual do not necessarily reflect the view of iGATE. iGATE is not 
responsible for the consequences of any actions taken on the basis of 
information provided, through this email. The contents of an 
attachment to this e-mail may contain software viruses, which could 
damage your own computer system. While iGATE has taken every 
reasonable precaution to minimise this risk, we cannot accept 
liability for any damage which you sustain as a result of software 
viruses. You should carry out your own virus checks before opening an 
attachment. To know more about iGATE please visit www.igate.com 
http://www.igate.com.
 




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: last_write_wins

[Guido Medina]

Hi Russell,

Thanks for your response, I understand most of it, I know LWW=true and 
allow_multi=true won't make any sense, but look at this scenario:


All of our buckets have allow_multi=false except for the one bucket we 
have for CRDT counters, our application requires certain some level of 
consistency so we have full control of our reads/writes using a fine 
grain locking mechanism combined with in-memory cache so in our case the 
LWW=true is what we would want?, now, we haven't touched this parameter 
so it is at its default value.


I'm assuming it will improve performance for our case, but, if we set 
LWW=true, will it affect the bucket(s) with allow_multi=true, is it safe 
to assume that if allow_multi=true LWW will be ignored? We only modify 
bucket properties using Riak Java client 1.4.x atm.


Also, about safety, LWW=true uses timestamp? and LWW=false uses vclock?, 
future of both?, should we leave it untouched? we don't really want to 
use something that could jeopardise our data consistency requirement 
even if it means better performance.


Hopefully I'm enriching the subject and not hijacking it,

Thanks,

Guido.

On 30/01/14 12:49, Russell Brown wrote:


On 30 Jan 2014, at 10:58, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:



Hi,

Now I'm curious too, according to 
http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/ 
the default value for Erlang property last_write_wins is false, now, 
if 95% of the buckets/keys have no siblings (or conflict resolution), 
does that mean that for such buckets last_write_wins is set to true, 
I'm wondering what's the effect (if any) if allow_multi on a bucket 
is false.


In other words; I could assume that:

  * If allow_multi is true then last_write_wins will be ignored
'cause vclock is needed for conflict resolution?
  * if allow_multi is false then last_write_wins is true?

They’re independant settings, but allow_mult=true + lww=true makes no 
sense (in reality, in the code, I’m pretty sure the lww=true will be 
applied.)


allow_mult=false+lww=false means at each vnode there is a 
read-before-write, and casually dominated values are dropped, while 
siblings values are made, but before we write to disk (or return to 
the user on get) we pick the sibling with the highest timestamp. This 
means that you get _one_ of the causally concurrent values, the one 
with the largest timestamp.


allow_mult=false+lww=false means that at the coordinating vnode we 
just increment whatever vclock the put has (probably none, right?) and 
write it to disk (no read of the local value first) and down stream at 
the replicas, the same thing, just store it. I need to check, but on a 
get, if there are siblings, just pick the highest timestamp.


I really think, for riak, 90% of the time, allow_mult=true is your 
best choice. John Daily did a truly exhaustive set of blog posts on 
this 
http://basho.com/understanding-riaks-configurable-behaviors-part-1/ I 
highly recommend it. If you data is always overwrite maybe LWW makes 
sense for you. If it is write once, read ever after LWW is perfect.


Cheers

Russell


Correct me if I'm wrong,

Again, we have a very similar scenarios, where we create/modify keys 
and we are certain we have the latest version so for us 
last_write_wins...


Regards,

Guido.

On 30/01/14 10:46, Russell Brown wrote:


On 30 Jan 2014, at 10:37, Edgar Veiga edgarmve...@gmail.com 
mailto:edgarmve...@gmail.com wrote:



Also,

Using last_write_wins = true, do I need to always send the vclock 
while on a PUT request? In the official documention it says that 
riak will look only at the timestamp of the requests.


Ok, from what you’ve said it sounds like you are always wanting to 
replace what is at a key with the new information you are putting. 
If that is the case, then you have the perfect use case for 
LWW=true. And indeed, you do not need to pass a vclock with your put 
request. And it sounds like there is no need for you to 
fetch-before-put since that is only to get context /resolve 
siblings. Curious about your use case if you can share more.


Cheers

Russell




Best regards,


On 29 January 2014 10:29, Edgar Veiga edgarmve...@gmail.com 
mailto:edgarmve...@gmail.com wrote:


Hi Russel,

No, it doesn't depend. It's always a new value.

Best regards


On 29 January 2014 10:10, Russell Brown russell.br...@me.com
mailto:russell.br...@me.com wrote:


On 29 Jan 2014, at 09:57, Edgar Veiga
edgarmve...@gmail.com mailto:edgarmve...@gmail.com wrote:


tl;dr

If I guarantee that the same key is only written with a 5
second interval, is last_write_wins=true profitable?


It depends. Does the value you write depend in anyway on
the value you read, or is it always that you are just
getting a totally new value that replaces what is in Riak
(regardless what is in Riak)?




On 27 January 2014 23:25

Copying counters from cluster A to B using Java client.

[Guido Medina]

Hi,

We are using Riak Java client 1.4.x and we want to copy all counters 
from cluster A to cluster B (all counters will be stored on a single to 
very few buckets), if I list the keys using special 2i bucket index and 
then treat each key as IRiakObject, will that be enough to copy 
counters, or will counter siblings stop me from doing that? Since at 
Riak Java client CounterObject is not an IRiakObject, it is instead an 
operation.


Comments at a working method:

*Source bucket:* Bucket from Riak client pointing to source cluster.
*Dest bucket:* Bucket from Riak client pointing to dest bucket.

  protected void copyOneItem(final Bucket sourceBucket, final Bucket 
destBucket, final String key) throws RiakRetryFailedException

  {
final IRiakObject riakObject=sourceBucket.fetch(key).execute();
if(riakObject!=null){
  destBucket.store(riakObject).withoutFetch().execute();
}
  }


Regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Copying counters from cluster A to B using Java client.

[Guido Medina]

Got it, do a counter fetch (CounterObject) from source which internally 
does the sibling resolution,  do a counter fetch from dest and increase 
that counter by source-dest (if dest is null then 0) :D


I wish I could setCounter(n) instead of increase (only in this case) to 
avoid extra fetch.


Guido.

On 29/01/14 11:44, Russell Brown wrote:

Oh damn, wait. You said 1.4.*. There might, therefore be siblings, do a counter 
increment before the copy to ensure siblings are resolved (if you can.) Or use 
RiakEE MDC.

On 29 Jan 2014, at 11:27, Guido Medina guido.med...@temetra.com wrote:


Hi,

We are using Riak Java client 1.4.x and we want to copy all counters from 
cluster A to cluster B (all counters will be stored on a single to very few 
buckets), if I list the keys using special 2i bucket index and then treat each 
key as IRiakObject, will that be enough to copy counters, or will counter 
siblings stop me from doing that? Since at Riak Java client CounterObject is 
not an IRiakObject, it is instead an operation.

Comments at a working method:

Source bucket: Bucket from Riak client pointing to source cluster.
Dest bucket: Bucket from Riak client pointing to dest bucket.

   protected void copyOneItem(final Bucket sourceBucket, final Bucket 
destBucket, final String key) throws RiakRetryFailedException
   {
 final IRiakObject riakObject=sourceBucket.fetch(key).execute();
 if(riakObject!=null){
   destBucket.store(riakObject).withoutFetch().execute();
 }
   }


Regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Java client counter object

[Guido Medina]

Hi,

Following up on this question, is there a way to list (not multi-fetch, 
just list) the existing counters in a bucket?


Regards,

Guido.

On 24/01/14 22:01, Dave Rusek wrote:

Guido,

I’m sorry, there are not currently any docs available. Rest assured 
they are a work in progress. In the meantime, please feel free to 
check-out the current pre-release [1]. The pre-release covers the low 
level API we have built on top of the PB interface. There is a start 
to a friendlier API in the develop branch of riak-java-client [2].


If you would like to get started with the low level API to begin 
getting a feel for Riak datatypes, I have included a few snippets that 
will hopefully help. The counters (and the data structures in general) 
are implemented as two separate operations: Fetch and Update. Updating 
a counter that does not exist will create it and initialize it to the 
given value. Fetching a counter that has not had a value stored will 
return an empty value.


While the low level API does not have convenience methods for 
getAndIncrement and the like, they can all be easily accomplished 
given fetch and update. It would clearly be easy for us to provide 
methods for these typical access patterns and in fact, it is something 
we are planning on doing.


RiakCluster cluster = …
BinaryValue bucket = …
BinaryValue key = …
BinaryValue type = ...

fetch():

DtFetchOperation get =
new DtFetchOperation.Builder(bucket, key)
.withBucketType(type)
.build();

DtFetchOperation.Response response = cluster.execute(get).get();

long value;
CrdtElement element = response.getCrdtElement();
if (crdtElement.isCounter()) {
value = element.getCounter().getValue();
} else {
...
}

update():

DtUpdateOperation update =
new DtUpdateOperation.Builder(bucket, key)
.withBucketType(type)
.withReturnBody(true)
.withOp(new CounterOp(1)) // increment by 1
.build();

DtUpdateOperation.Response response = cluster.execute(update).get();

long value;
CrdtElement element = response.getCrdtElement();
if (crdtElement.isCounter()) {
value = element.getCounter().getValue();
} else {
...
}

getAndIncrement();

long value = fetch();
update(1);

incrementAndGet():

long value = update(1); // make sure withReturnBody == true

[1] https://github.com/basho/riak-java-client
[2]https://github.com/basho/riak-java-client/tree/develop/src/main/java/com/basho/riak/client/operations

--
Dave Rusek
Basho Technologies
@davidjrusek

On January 24, 2014 at 8:41:07 AM, Guido Medina 
(guido.med...@temetra.com mailto://guido.med...@temetra.com) wrote:



Hi,

Is there any small doc that could explain its usage a little bit:

From the Java perspective it would be nice if it point out its 
counter part methods with Atomic Integer like:


  * How to create it? Does incrementing a counter will just create it
with zero as initial value and then increment it? What if you
need to know its value and then increment it before it is
created? (getAndIncrement)
  * get()
  * getAndIncrement()
  * incrementAndGet()

Regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak claim wants funs.

[Guido Medina]

Hi,

I'm trying to get different distributions on the ring and I used before 
the following Erlang code, which for Riak 1.4.7 is not re-triggering a 
re-calculation:


 * What is the latest algorithm version? v3? And is there a list of the
   last 2 or 3 versions? Sometimes depending on the keys one algorithm
   can fit better than the other (From what I have learned)

   *Changes algorithm:*

   rpc:multicall(application, set_env, [riak_core, wants_claim_fun,
   {riak_core_claim, wants_claim_v3}]).
   rpc:multicall(application, set_env, [riak_core, choose_claim_fun,
   {riak_core_claim, choose_claim_v3}]).

 * Not working in Riak 1.4.7, any Riak/Erlang expert know why?

   *Triggers a recalculation:*

   fun() -
   Node = node(),
   riak_core_ring_manager:ring_trans(
 fun(Ring, []) -
 case riak_core_ring:claimant(Ring) of
 Node -
 %% Set next to []
 NewRing1 = setelement(7, Ring, []),
 %% Increment ring version
 RVsn1 = element(11, NewRing1),
 RVsn2 = vclock:increment(Node, RVsn1),
 NewRing2 = setelement(11, NewRing1, RVsn2),
 {new_ring, NewRing2};
 _ -
 io:format(Failed. This node is not the
   claimant~n),
 ignore
 end
 end, []),
   ok
   end().


Regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak transfer limit.

[Guido Medina]

Hi,

What's a good value for transfer limit when re-arranging adding/removing 
nodes?

Or if there is a generic rule of thumb like physical nodes, processors, etc.

Once transfer is completed, is it a good practice to set it back to its 
default value or should the calculated (guessed?) transfer limit stay?


Regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Java Protobuf objects

[Guido Medina]

At Maven central repos it has yes:

https://repository.sonatype.org/index.html#nexus-search;quick~riak-pb

HTH,

Guido.

On 21/01/14 14:22, Jon Brisbin wrote:
Have the Riak Java Protobuf artifacts been updated to take advantage 
of Riak 2.0 features yet?


I'd like to work some more on getting the Riaktor (Riak async Java 
client based on Reactor [1]) up to Riak 2.0 functionality. Currently 
I'm using the Protobuf artifact for the actual objects I need to do 
the interactions. I'm handling all the network IO through 
Reactor/Netty and using the simple codecs in Reactor.


If not, can I just generate the Java objects I need from the original 
protobuf definitions?


[1] - https://github.com/reactor/reactor


Thanks!

Jon Brisbin | Reactor Project Lead
http://about.me/jbrisbin | @j_brisbin



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Erlang schedulers warning

[Guido Medina]

Hi,

We have this configuration at our vm.args for all of our 8 cores servers:

## Erlang scheduler tuning: 
https://github.com/basho/leveldb/wiki/Riak-tuning-1

*+S 4:4*


But at /var/log/riak/console.log we have the following warning, should 
we ignore it?


2014-01-13 22:15:35.246 [warning] 0.317.0 riak_kv_env: *Running 4 
schedulers for 8 cores, these should match*



Regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: high availability using riak Java client

[Guido Medina]

Hi Simon,

We use HA proxy for that matter, set up HA proxy to at localhost:8087 
and then your Riak Java client pointing to it, this is a sample HA proxy 
config: https://gist.github.com/gburd/1507077


HTH,

Guido.

On 09/12/13 12:05, Shimon Benattar wrote:

Hi Riak users,

we are using the Riak Java client and need to support high availability

Currently I am connection the pbc to a specific node in our cluster,

I would like to pass a list of IP's so that in case the connect fails 
on one we will try the operation again on the other node (Also that 
way the work distribution is shared by all the nodes on the cluster)


Thanks,

Shimon


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: high availability using riak Java client

[Guido Medina]

I meant Hi Shimon, sorry for the wrong spelling.

Guido.

On 09/12/13 12:12, Guido Medina wrote:

Hi Simon,

We use HA proxy for that matter, set up HA proxy to at localhost:8087 
and then your Riak Java client pointing to it, this is a sample HA 
proxy config: https://gist.github.com/gburd/1507077


HTH,

Guido.

On 09/12/13 12:05, Shimon Benattar wrote:

Hi Riak users,

we are using the Riak Java client and need to support high availability

Currently I am connection the pbc to a specific node in our cluster,

I would like to pass a list of IP's so that in case the connect fails 
on one we will try the operation again on the other node (Also that 
way the work distribution is shared by all the nodes on the cluster)


Thanks,

Shimon


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Timeout on Streaming all keys

[Guido Medina]

Hi Shimon,

Did you try streaming the 2i bucket index and then doing your job per 
key basis? I sent you a code snippet the other day.


It should work fine regardless of how many keys you have in your bucket, 
it is equivalent to the section:


http://docs.basho.com/riak/latest/dev/using/2i/ - Look for Retrieve all 
Bucket Keys via $bucket Index


Except that the code I sent you uses streaming API.

Guido.

On 09/12/13 14:10, Luke Bakken wrote:

Hi Shimon,

There is a tech preview of 2.0.0 available here that includes
Yokozuna: http://docs.basho.com/riak/2.0.0pre5/downloads/

However, since then there have been updates to Yokozuna. You can build
from the latest sources using the instructions here:

https://github.com/basho/yokozuna/blob/develop/docs/INSTALL.md#source-package

You will find related YZ docs in that docs directory as well.

--
Luke Bakken
CSE
lbak...@basho.com


On Mon, Dec 9, 2013 at 4:11 AM, Shimon Benattar shim...@gmail.com wrote:

Hi Luke, I'd love to have your insights on our problem.

About setting the timeout, we are anyway considering to upgrade to version 2.0 
so maybe I'll try that. is there an official release data?
Does the basic installation already include installation of Yokozuna? is there 
some guide on how to work woth Yokozuna?

Thanks,

Shimon


On 5 December 2013 19:48, Luke Bakken lbak...@basho.com wrote:

Hi Shimon,

I spoke with a member of the Java client team and the timeout is set to 60 
seconds. In the 2.0 version of the client this will be user-settable.

To make sure there's not a crash or other issue causing the long list keys, feel free to 
run riak-debug on nodes in your cluster and provide the generated archives. 
We can discuss that off-list.

--
Luke Bakken
CSE
lbak...@basho.com


On Wed, Dec 4, 2013 at 12:43 AM, Shimon Benattar shim...@gmail.com wrote:

Hi Luke,
We're using v1.4

On Dec 3, 2013 6:19 PM, Luke Bakken lbak...@basho.com wrote:

Hi Shimon,

What version of Riak and the Java client are you using?

Thanks
--
Luke Bakken
CSE
lbak...@basho.com


On Thu, Nov 28, 2013 at 4:00 AM, Shimon Benattar shim...@gmail.com wrote:

Hi Riak users,



Unfortunately for me I need to implement the get all keys method in our system 
(I know it is not recommended)



I am developing classes with the Java Api and using the streaming option as 
following



StreamingOperationString keys = riakBucket.keys();

while (keys.hasNext()) {

final String next = keys.next();

out.write(next);

out.newLine();

}

riakBucket.keys().cancel();



My problem here is that for a simple class I inserted in Riak, after ~4.7 
million keys I am getting a timeout from Riak.

Is there any way to configure the read operation timeout or to somehow catch 
the timeout and continue to receive the next keys (Can I somehow bulk read the 
keys)?



Thanks,



Shimon




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Timeout on Streaming all keys

[Guido Medina]

Assuming your backend is on LevelDB, have you try streaming the special 
bucket 2i index? We stream millions of keys using specific 2i indexes, 
if you didn't create any 2i index on your bucket, you can still fetch 
that special key, and per key you can fetch it one by one while 
iterating over your 2i index.


Sample code (Needs Riak Java client 1.4+ and Riak 1.4+):

/for (IndexEntry entry : 
bucket.fetchIndex(BucketIndex.index).executeStreaming()) {//

//
   // Then you fetch the key like this://
//   IRiakObject obj=bucket.fetch(entry.getObjectKey()).execute();//
//
   // Or you could do the fetch and process operation using an 
ExecutorService.//

//  FutureWhatever future=executorService.submit(new CallableWhatever//
// public Whatever call(){
///IRiakObject obj=bucket.fetch(entry.getObjectKey()).execute();//
...
return new Whatever OR null if Void
/ }
//);//
//}/

HTH,

Guido.

On 05/12/13 18:46, Alexander Sicular wrote:
Shimon, chevreh, re-architect your app. Make like a key that has 
pointers to all of a given days keys or something like that and 
iterate over those.


If the sign says don't feed the animals, don't feed the animals.

Shalom v simcha,
-Alexander Sicular

@siculars

On Nov 28, 2013, at 7:00 AM, Shimon Benattar shim...@gmail.com 
mailto:shim...@gmail.com wrote:



Hi Riak users,


Unfortunately for me I need to implement the get all keys method in 
our system (I know it is not recommended)



I am developing classes with the Java Api and using the streaming 
option as following



StreamingOperationString keys = riakBucket.keys();

while (keys.hasNext()) {

final String next = keys.next();

out.write(next);

out.newLine();

}

riakBucket.keys().cancel();


My problem here is that for a simple class I inserted in Riak, after 
~4.7 million keys I am getting a timeout from Riak.


Is there any way to configure the read operation timeout or to 
somehow catch the timeout and continue to receive the next keys (Can 
I somehow bulk read the keys)?



Thanks,


Shimon



___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java Client with POJOs

[Guido Medina]

Hi Michael,

I'm quite sure lately annotating the Riak key method was enabled at the 
client but I'm not 100% of its usage yet (I haven't used it), in the 
meantime for testing you can annotate the property with @RiakKey inside 
your POJO, that should do, then just treat that property as any other 
property (generate its getter and setter)


/public class Account {//

///*@RiakKey*/
//private String accountId;//
//...//
//...//
//...//
//}/

HTH,

Guido.

On 11/11/13 22:06, Michael Guymon wrote:
So I had to add @JsonProperty(metadata) for the @RiakUsermeta to 
appear in the serialized json being processed by the Reduce phase.  I 
have been using ejsLog('/tmp/map_reduce.log', 
JSON.stringify(values)); to see what is being passed in.


One last question, the field with @RiakKey is always null. I added the 
@JsonProperty(id) and it is null in the serialized json as well. Is 
there a step I am missing in the store to populate the @RiakKey?


thanks,
Michael

On 11/11/2013 04:12 PM, Brian Roach wrote:
If your mapping function you simply add a qualifier to detect 
tombstones;


if (values[i].metadata['X-Riak-Deleted'] == 'true')

- Roach

On Mon, Nov 11, 2013 at 1:59 PM, Michael Guymon
mich...@tobedevoured.com wrote:
Ahh, yes, now that makes sense. I see with @RiakUsermeta or 
@RiakTombstone
it is possible to filter the results of the MapReduce for 
tombstones. Is it

possible to add a phase to reduce the tombstones instead of manually
filtering the final result?

thanks,
Michael


On 11/11/2013 03:16 PM, Brian Roach wrote:

Michael -

You have something stored in that bucket that isn't the JSON you're
expecting when you run your second map/reduce. As I mentioned, there's
nothing special about how the Java client works; it just serializes
the POJO instance using Jackson.

My suggestion would be using curl / your browser (or the Java client)
and seeing what that is; listing the keys and checking the contents.

I notice you're using the .withoutFetch() option when storing;
that's guaranteed to create a sibling if you have allow_multi=true set
in the bucket. If that's the case then that behavior is expected; both
versions are stored in Riak.

Also worth noting is that if you're recently deleted something
(explicitly via a delete operation) it's very likely to get a
tombstone pass to map/reduce.  If you're doing explicit deletes from
Riak you need to check the object metadata for the
X-Riak-Deleted header being true, and then ignore that object in
your map function.

- Roach


On Mon, Nov 11, 2013 at 12:46 PM, Michael Guymon
mich...@tobedevoured.com wrote:

Hi Roach,

Thanks for taking a moment to give me a hand with this. Let me try 
and be

a
bit more clear on what I am trying to figure out. My first steps 
are a

Class
Account:

public class Account implements Serializable {
  private String email;
}

Storing the account via

myBucket.store(key, account).withoutFetch().execute();

then retrieving it with a map reduce using JS, along the lines of:

var accounts = [];
for( i=0; ivalues.length;i++) {
if ( values[i].email == 't...@test.net ) {
  accounts.push(values[i]);
}
}
return accounts

works as expected.

Now I updated the Class Account to have the name property:

public class Account implements Serializable {
  private String name;
  private String email;
}

and storing with data to the same bucket, for the same key and 
attempting

to
Map Reduce for name I get a

{phase:1,error:[{\lineno\,1},{\message\,\TypeError: 


values[i].name is

undefined\},{\source\,\unknown\}],input:null,type:null,stack:null}. 



If I change the bucket to a new one, the Map Reduce runs successfully
without the above error.

This is Riak 1.4.2 running on Ubuntu 13.04

thanks,
Michael


On 11/11/2013 02:32 PM, Brian Roach wrote:

Hi Michael,

I'm somewhat confused by your question; map/reduce doesn't really 
have

anything to do with your Java POJO/class.

When using the Riak Java client and storing a POJO, the default
converter (JSONConverter)  uses the Jackson JSON library and converts
the instance of your POJO into a JSON string and stores it in Riak.

If you change that POJO class and store more things, the resulting
JSON is obviously going to be different (in your case having an
additional field named minty).

When doing Map/Reduce, whatever JavaScript or Erlang functions you
provide are executing in Riak and being given the data stored in Riak
(the JSON you stored); they have no connection to Java.

Can you expand on  Now the map reduce fails for that the new
property with what exactly the problem is? It sounds like you have a
problem with your JavaScript or Erlang function(s).

Thanks!
- Roach


On Mon, Nov 11, 2013 at 12:07 PM, Michael Guymon
mich...@tobedevoured.com wrote:

Hello,

I have a (hopefully dumb) question about working with the Java 
client and

POJOs. I justed started tinkering with Riak and have created a simple
Account POJO and happily crammed it into a bucket test1 and 

Java client blocked at shutdown?

[Guido Medina]

Hi,

We are tracing some threads at our webapp which are stopping Tomcat from 
shooting down properly, one of them seems to be related with Riak Java 
client, here is the repeating stack trace once all services have been 
stopped properly:


*Thread Name:* riak-stream-timeout-thread
*State:* in Object.wait()
*Java Stack trace:*
at java.lang.Object.wait(Native Method)
- waiting on [0x0004bb4001a0] (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked [0x0004bb4001a0] (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)


Thanks for the help,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client blocked at shutdown?

[Guido Medina]

Sorry, I meant stopping Tomcat from shutting down properly...I must 
have been thinking of some FPS night game.


On 05/11/13 13:04, Guido Medina wrote:

Hi,

We are tracing some threads at our webapp which are stopping Tomcat 
from shooting down properly, one of them seems to be related with Riak 
Java client, here is the repeating stack trace once all services have 
been stopped properly:


*Thread Name:* riak-stream-timeout-thread
*State:* in Object.wait()
*Java Stack trace:*
at java.lang.Object.wait(Native Method)
- waiting on [0x0004bb4001a0] (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked [0x0004bb4001a0] (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)


Thanks for the help,

Guido.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client blocked at shutdown?

[Guido Medina]

That's done already, I'm looking at the source now, not sure of the 
following timer needs to be cancelled when Riak client shutdown method 
is called:



public abstract class RiakStreamClientT implements IterableT {

static Timer TIMER = new Timer(riak-stream-timeout-thread, true);
...
...
}

Guido.

On 05/11/13 13:29, Konstantin Kalin wrote:
You need to call shutdown method of Riak client when you are stopping 
your application.


Thank you,
Konstantin.

On Nov 5, 2013, at 5:06, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:


Sorry, I meant stopping Tomcat from shutting down properly...I must 
have been thinking of some FPS night game.


On 05/11/13 13:04, Guido Medina wrote:

Hi,

We are tracing some threads at our webapp which are stopping Tomcat 
from shooting down properly, one of them seems to be related with 
Riak Java client, here is the repeating stack trace once all 
services have been stopped properly:


*Thread Name:* riak-stream-timeout-thread
*State:* in Object.wait()
*Java Stack trace:*
at java.lang.Object.wait(Native Method)
- waiting on [0x0004bb4001a0] (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked [0x0004bb4001a0] (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)


Thanks for the help,

Guido.


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client blocked at shutdown?

[Guido Medina]

Which I would suggest to make it non-daemon and each periodic task be 
registered into an internal and elastic ThreadPoolExecutor using 
corePoolSize, maxPoolSize and LinkedBlockingQueue for parallel tasks not 
bound to Riak client main thread pool.


That why cancelling such pool will cancel all scheduled tasks.

Guido.

On 05/11/13 13:31, Guido Medina wrote:
That's done already, I'm looking at the source now, not sure of the 
following timer needs to be cancelled when Riak client shutdown method 
is called:



public abstract class RiakStreamClientT implements IterableT {

static Timer TIMER = new Timer(riak-stream-timeout-thread, true);
...
...
}

Guido.

On 05/11/13 13:29, Konstantin Kalin wrote:
You need to call shutdown method of Riak client when you are stopping 
your application.


Thank you,
Konstantin.

On Nov 5, 2013, at 5:06, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:


Sorry, I meant stopping Tomcat from shutting down properly...I 
must have been thinking of some FPS night game.


On 05/11/13 13:04, Guido Medina wrote:

Hi,

We are tracing some threads at our webapp which are stopping Tomcat 
from shooting down properly, one of them seems to be related with 
Riak Java client, here is the repeating stack trace once all 
services have been stopped properly:


*Thread Name:* riak-stream-timeout-thread
*State:* in Object.wait()
*Java Stack trace:*
at java.lang.Object.wait(Native Method)
- waiting on [0x0004bb4001a0] (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked [0x0004bb4001a0] (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)


Thanks for the help,

Guido.


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client blocked at shutdown?

[Guido Medina]

You may be right, we are still investigating other 2 threads, was worth 
to add this one to the list just in case, daemon threads by contract 
should go down with no issues when JDK is going down.


Guido.

On 05/11/13 13:59, Konstantin Kalin wrote:


Strange. If you call shutdown of Riak client it shouldn't stop Tomcat 
shutdown anymore. This is that I learn from source code. I called the 
method in Servlet listener and never had issues after that. Before I 
had similar behavior like you have.


Thank you,
Konstantin.

On Nov 5, 2013 5:31 AM, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:


That's done already, I'm looking at the source now, not sure of
the following timer needs to be cancelled when Riak client
shutdown method is called:


public abstract class RiakStreamClientT implements IterableT {

static Timer TIMER = new Timer(riak-stream-timeout-thread,
true);
...
...
}

Guido.

On 05/11/13 13:29, Konstantin Kalin wrote:

You need to call shutdown method of Riak client when you are
stopping your application.

Thank you,
Konstantin.

On Nov 5, 2013, at 5:06, Guido Medina guido.med...@temetra.com
mailto:guido.med...@temetra.com wrote:


Sorry, I meant stopping Tomcat from shutting down properly...I
must have been thinking of some FPS night game.

On 05/11/13 13:04, Guido Medina wrote:

Hi,

We are tracing some threads at our webapp which are stopping
Tomcat from shooting down properly, one of them seems to be
related with Riak Java client, here is the repeating stack
trace once all services have been stopped properly:

*Thread Name:* riak-stream-timeout-thread
*State:* in Object.wait()
*Java Stack trace:*
at java.lang.Object.wait(Native Method)
- waiting on [0x0004bb4001a0] (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked [0x0004bb4001a0] (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)


Thanks for the help,

Guido.


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak on SAN

[Guido Medina]

And for ZFS? I wouldn't recommend it, after Riak 1.4 snappy LevelDB 
compression does a nice job, why take the risk of yet another not so 
enterprise ready compression algorithms.


I could be wrong though,

Guido.

On 03/10/13 12:11, Guido Medina wrote:
I have heard some SAN's horrors stories too, Riak nodes are so cheap 
that I don't see the point in even having any mirror on the node, here 
my points:


 1. Erlang interprocess communication brings some network usage, why
yet another network usage on replicating the data? If the whole
idea of Riak is have your data replicated in different nodes.
 2. If a node goes down or die for whatever reason, bring up another
node and rebuild it.
 3. If you want to really replicate your cluster Riak offers the
enterprise replication which I'm quite sure will be less expensive
than a SAN and will warranty to have your cluster ready to go
somewhere else as a backup.
 4. I would even go further, SSDs are so cheap and Riak nodes are so
cheap now adays that I would even build a cluster using RAID 0 or
RAID 5 SSDs (yes, no mirror with RAID 1, if too afraid, RAID 5),
that will have a great impact on performance. Again, if something
goes wrong with 1 node, refer to point 2.

SANs and all those legacy backup and replication IMHO are meant for 
other products, like an Oracle money eater DB server.


HTH,

Guido.

On 03/10/13 12:00, Brian Akins wrote:

So, call me naive, but couldn't ZFS be used as Heinze suggested?

I have some SAN horror stories - both operationally and from an 
economic perspective.



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak on SAN

[Guido Medina]

If using LevelDB backend, LevelDB has a nice compression (snappy), 
including CRC checks and all sort of data corruption checks, I have read 
on this mail list people that has required to disable snappy compression 
because it renders ZFS useless (not much to compress after that)


Hence, it is kind of related to using ZFS or not, if you go for ZFS 
whatever variant you will have to support two sub-systems, if you let 
LevelDB snappy compression on, you won't have to worry about it.


As for backup, Basho provides a sort of cluster-to-cluster replication 
tool, we built our own in Java, making backups per storage on every node 
won't make much sense due to CAP/distributed nature, replicating the 
keys to another cluster is what will make sense.


Hope that helps and is understandable,

Guido.

On 03/10/13 13:54, Pedram Nimreezi wrote:
Not sure what ZFS has to do with snappy compression, as it's a file 
system not a compression algorithm..
feature wise, ZFS is quite possibly the most enterprise file system 
around, including advanced data corruption prevention and remote 
backing up..


This would be a viable option in BSD/Solaris environments, at least 
for making snapshots.

Might make a nice write up for the Basho blog..

Backups for riak I think require a bit more consideration then file 
system snapshot send,
and should include provisions for transferring data to smaller/larger 
clusters, transfer

ring ownerships properly, etc.


On Thu, Oct 3, 2013 at 7:15 AM, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:


And for ZFS? I wouldn't recommend it, after Riak 1.4 snappy
LevelDB compression does a nice job, why take the risk of yet
another not so enterprise ready compression algorithms.

I could be wrong though,

Guido.


On 03/10/13 12:11, Guido Medina wrote:

I have heard some SAN's horrors stories too, Riak nodes are so
cheap that I don't see the point in even having any mirror on the
node, here my points:

 1. Erlang interprocess communication brings some network usage,
why yet another network usage on replicating the data? If the
whole idea of Riak is have your data replicated in different
nodes.
 2. If a node goes down or die for whatever reason, bring up
another node and rebuild it.
 3. If you want to really replicate your cluster Riak offers the
enterprise replication which I'm quite sure will be less
expensive than a SAN and will warranty to have your cluster
ready to go somewhere else as a backup.
 4. I would even go further, SSDs are so cheap and Riak nodes are
so cheap now adays that I would even build a cluster using
RAID 0 or RAID 5 SSDs (yes, no mirror with RAID 1, if too
afraid, RAID 5), that will have a great impact on
performance. Again, if something goes wrong with 1 node,
refer to point 2.

SANs and all those legacy backup and replication IMHO are meant
for other products, like an Oracle money eater DB server.

HTH,

Guido.

On 03/10/13 12:00, Brian Akins wrote:

So, call me naive, but couldn't ZFS be used as Heinze suggested?

I have some SAN horror stories - both operationally and from an
economic perspective.


___
riak-users mailing list
riak-users@lists.basho.com  mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
/* Sincerely
--
Pedram Nimreezi - Chief Technology Officer  */

// The hardest part of design … is keeping features out. - Donald Norman




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak on SAN

[Guido Medina]

Some users might avoid the ZFS overheads, remember we are on a KV world 
where to read/write many keys you will have to do so concurrently, say 
there is less than 1% chances for things things going wrong with 1 a 
server belonging to a Riak cluster, if building a Riak server is cheap, 
would you pay the overhead price?


It is a choice, you might risk it and remove such overheads which in 
theory is not risky at all because Riak will replicate your keys in 
different nodes, for example, on a 5 servers cluster, 2 servers will 
have to die simultaneously for you to lose data (Someone correct me if 
I'm wrong in this part)


All I'm trying to say is that such over data protection might not be 
necessary at all with a distributed and high available NoSQL DB like Riak.


Guido.

On 03/10/13 15:11, Heinz Nikolaus Gies wrote:

Hi Guido,
I don’t see how snappy compression renders ZFS useless, you might do 
some things twice like crcing but it also protects on different 
layers. While the ZFS crc protects data on the disks the in app crc 
could protect the data ‘all’ the way up, compression wise you might 
not even turn on ZFS compression and even if you do, you could still 
get a higher ratio given that ZFS will use compression over the entire 
volume not ‘just’ the data in the DB.


That said there is a lot more to ZFS then compression and CRC ;) like 
snapshots, cloning, ARC ^^



On 03 Oct 2013, at 9:56, Guido Medina guido.med...@temetra.com 
mailto:guido.med...@temetra.com wrote:


If using LevelDB backend, LevelDB has a nice compression (snappy), 
including CRC checks and all sort of data corruption checks, I have 
read on this mail list people that has required to disable snappy 
compression because it renders ZFS useless (not much to compress 
after that)


Hence, it is kind of related to using ZFS or not, if you go for ZFS 
whatever variant you will have to support two sub-systems, if you let 
LevelDB snappy compression on, you won't have to worry about it.


As for backup, Basho provides a sort of cluster-to-cluster 
replication tool, we built our own in Java, making backups per 
storage on every node won't make much sense due to CAP/distributed 
nature, replicating the keys to another cluster is what will make sense.


Hope that helps and is understandable,

Guido.

On 03/10/13 13:54, Pedram Nimreezi wrote:
Not sure what ZFS has to do with snappy compression, as it's a file 
system not a compression algorithm..
feature wise, ZFS is quite possibly the most enterprise file system 
around, including advanced data corruption prevention and remote 
backing up..


This would be a viable option in BSD/Solaris environments, at least 
for making snapshots.

Might make a nice write up for the Basho blog..

Backups for riak I think require a bit more consideration then file 
system snapshot send,
and should include provisions for transferring data to 
smaller/larger clusters, transfer

ring ownerships properly, etc.


On Thu, Oct 3, 2013 at 7:15 AM, Guido Medina 
guido.med...@temetra.com mailto:guido.med...@temetra.com wrote:


And for ZFS? I wouldn't recommend it, after Riak 1.4 snappy
LevelDB compression does a nice job, why take the risk of yet
another not so enterprise ready compression algorithms.

I could be wrong though,

Guido.


On 03/10/13 12:11, Guido Medina wrote:

I have heard some SAN's horrors stories too, Riak nodes are so
cheap that I don't see the point in even having any mirror on
the node, here my points:

 1. Erlang interprocess communication brings some network
usage, why yet another network usage on replicating the
data? If the whole idea of Riak is have your data
replicated in different nodes.
 2. If a node goes down or die for whatever reason, bring up
another node and rebuild it.
 3. If you want to really replicate your cluster Riak offers
the enterprise replication which I'm quite sure will be
less expensive than a SAN and will warranty to have your
cluster ready to go somewhere else as a backup.
 4. I would even go further, SSDs are so cheap and Riak nodes
are so cheap now adays that I would even build a cluster
using RAID 0 or RAID 5 SSDs (yes, no mirror with RAID 1, if
too afraid, RAID 5), that will have a great impact on
performance. Again, if something goes wrong with 1 node,
refer to point 2.

SANs and all those legacy backup and replication IMHO are
meant for other products, like an Oracle money eater DB server.

HTH,

Guido.

On 03/10/13 12:00, Brian Akins wrote:

So, call me naive, but couldn't ZFS be used as Heinze suggested?

I have some SAN horror stories - both operationally and from
an economic perspective.


___
riak-users mailing list
riak-users@lists.basho.com  mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo

Riak Java client isKeyPresent function?

[Guido Medina]

Hi,

Is there a way to quick check if a key is present without fetching it 
using the Riak Java client? It would be nice to have one for quick 
checks without fetching the key:


/interface Bucket {//
//   //
//   public boolean isKeyPresent(String key);//
//   //
//}/


Of course, that would have to be supported via PB client without 
actually fetching the key (If current implementation makes it possible)


Thanks,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Ownership question.

[Guido Medina]

Morning,

Is there a way to determine what nodes a key belong to? I'm guessing 
that the hash of a key will be computed using the bucket name and key 
combined, I'm having some issues with some writes and would like to see 
if there is a pattern, knowing what nodes are involved will help me a lot.


Thanks,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

LevelDB tuning questions.

[Guido Medina]

Hi,

I'm trying to tune our Riak cluster using 
http://docs.basho.com/riak/latest/ops/advanced/backends/leveldb/#Parameter-Planning 
but I'm still lost on how to use the calculation results, here are my 
questions:


1. Does this calculator
   
https://github.com/basho/basho_docs/raw/master/source/data/LevelDB1.2MemModel_calculator.xls
   applies to Riak 1.4.2?
2. How do I verify my running configuration? (I tried riak-admin status
   but somehow I can't relate the calculation with the obtained status,
   etc.)
3. And finally, based on the calculator, how do I use the results, what
   pertinent variables am I supposed to set in app.config in the
   LevelDB section?


Thanks,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak Java client bug?

[Guido Medina]

Hi,

Streaming 2i indexes is not timing out, even though the client is 
configured to timeout, this coincidentally is causing the writes to fail 
(or or the opposite?), is there anything elemental that could lock (I 
know the locking concept in Erlang is out of the equation so LevelDB?) 
something in Riak while trying to stream a 2i index?


Basically, our cluster copier which runs every two minutes once it gets 
to this state it never exists (no timeout) and our app just starts slow 
writing (over a minute to write a key)


Not sure what's going on.

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Java client bug?

[Guido Medina]

Like this: withConnectionTimeoutMillis(5000).build();

Guido.

On 25/09/13 18:08, Brian Roach wrote:

Guido -

When you say the client is configured to time out do you mean you're
using PB and you set the SO_TIMEOUT on the socket via the
PBClientConfig's withRequestTimeoutMillis()?

- Roach

On Wed, Sep 25, 2013 at 5:54 AM, Guido Medina guido.med...@temetra.com wrote:

Hi,

Streaming 2i indexes is not timing out, even though the client is configured
to timeout, this coincidentally is causing the writes to fail (or or the
opposite?), is there anything elemental that could lock (I know the
locking concept in Erlang is out of the equation so LevelDB?) something in
Riak while trying to stream a 2i index?

Basically, our cluster copier which runs every two minutes once it gets to
this state it never exists (no timeout) and our app just starts slow writing
(over a minute to write a key)

Not sure what's going on.

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Java client bug?

[Guido Medina]

To ease the situation we have done some redesign to avoid locks 
contention (Internal to our app), cause we were writing too often in 
very short time to the same key (no siblings), so it might be a 
combination of LevelDB+AAE+2i streaming, tomorrow hopefully things are 
back to normal, the crash.log just reports about nodes timed-out and 
their removal, the timing of such situation happening coincidentally 
matches such conditions, we tried playing with the quorum but that 
didn't help so the best we could come up was to have a finer grain set 
of locks.


We will set our cluster copy to use the with request timeout parameter 
to avoid the been stalled.


Regarding to your comment I guess there is a need to improve 2i+LevelDB 
weird situations.


Thanks for the help and let's see how things move in the future for 
LevelDB and Riak.


Guido.

On 25/09/13 19:21, Brian Roach wrote:

That option is for the connection timeout (i.e. when the connection
pool makes a new connection to Riak).

You can set the read timeout on the socket with the aforementioned
withRequestTimeoutMillis()

The default is the Java default, which is to say it'll block on the
socket read until either there's data to read or the remote side
closes the socket. That would at least get the client unstuck.

This, however, doesn't explain/solve the real issue you're describing
which is that Riak is hanging up and not sending data.

Someone else will need to chime in on that - are you seeing anything
in the Riak logs?

- Roach

On Wed, Sep 25, 2013 at 12:11 PM, Guido Medina guido.med...@temetra.com wrote:

Like this: withConnectionTimeoutMillis(5000).build();

Guido.


On 25/09/13 18:08, Brian Roach wrote:

Guido -

When you say the client is configured to time out do you mean you're
using PB and you set the SO_TIMEOUT on the socket via the
PBClientConfig's withRequestTimeoutMillis()?

- Roach

On Wed, Sep 25, 2013 at 5:54 AM, Guido Medina guido.med...@temetra.com
wrote:

Hi,

Streaming 2i indexes is not timing out, even though the client is
configured
to timeout, this coincidentally is causing the writes to fail (or or the
opposite?), is there anything elemental that could lock (I know the
locking concept in Erlang is out of the equation so LevelDB?) something
in
Riak while trying to stream a 2i index?

Basically, our cluster copier which runs every two minutes once it gets
to
this state it never exists (no timeout) and our app just starts slow
writing
(over a minute to write a key)

Not sure what's going on.

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Very high CPU usage by bean.smp

[Guido Medina]

Jared,

Is it possible to elaborate more on meet me in the middle 
settings/scenarios?, let me explain, let's say the quorum is configured 
with low values, say, R=W=1 and N=3, doesn't that add more work to AAE 
background process? Could there be ways to sacrifice some client 
performance with let's say, in the middle kind of generic for most 
scenario settings to have the client wait a bit longer (R=W=2 or similar 
combinations) and AAE to work a bit less?


Just thoughts though, I leave to the experts the deep explanation of this.

HTH,

Guido.

On 24/09/13 13:41, Jared Morrow wrote:



Interesting. Someone knows what this entropy thing is doing
exactly? Can
it be switched off by default maybe?


Here is a description of AAE when it was released in Riak 1.3 
http://basho.com/introducing-riak-1-3/   You can turn off AAE in your 
config files if you want it off.  With Riak (and distributed systems 
in general) there are always tradeoffs for any feature.  Riak more 
than others tends to pick 'safety' for the default case.  AAE protects 
your data's integrity even when not being read for a long time.  It is 
a very important feature, but we understand it does not fit everyone's 
needs, so we provide the option to turn it off.


Hope that helps,
Jared




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak control on HTTP (not HTTPS)

[Guido Medina]

Hi,

Is it possible to have Riak control running at HTTP port on localhost? 
Assuming security is provided by SSH tunnels.
If so what is needed to be done at app.config? I enabled Riak control 
but it is redirecting me to HTTPS.


Thanks,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: use Siblings to implement a message queue?

[Guido Medina]

Alex,

RabbitMQ which is a good high performer, developed in Erlang and scales 
just as Riak.


The old saying, the right tool for the right job, I like how fast Riak 
is fetching/storing key values on a distributed environment, I don't 
like Riak for queues, is it because it wasn't designed for that? CRDT 
like CAS operations on structures (to mention some for Java) like 
LinkedBlockingQueue and ConcurrentLinkedQueue should be painful enough 
to develop in a CRDT environment, cause now you would be talking of 
guarding states of few within the same structure; say AtomicReference of 
several variables to warranty a non-blocking CAS operation, in this case 
heap and tail of the queue (Not so true for LinkedBlockingQueue)


And performance of several CRDT for high performance queue I don't 
actually think is going to be good.


If I were to use Riak for my big data environment and would like to 
match Riak on a distributed queueing system then I would use like I 
said, RabbitMQ.


Guido.

On 12/09/13 00:29, Alex Rice wrote:

Hi I'm very new to Riak. The allow_mult = true /  Siblings feature is
very interesting. Could it be used to implement a high performance
collection like a Queue or Set, in a lock free manner? The Riak docs
make it sound like allow_mult is mainly for confict resolution and
degenerate cases, rather than a feature which a data structure could
be designed around.  Does anyone have any links to share, or thoughts
about this broader issue?
Thanks much!
Alex

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: use Siblings to implement a message queue?

[Guido Medina]

Hi Alex,

I get what you mean, I would say; if you think your project will succeed 
don't think small, RabbitMQ and Riak can be pretty easy to understand 
and manage once you play with them, don't be scare to try, high demand 
systems usually have several subsystems to get their tasks done quickly 
like:


1. Distributed data.
2. Distributed queues.
3. Text indexing/search.

With Riak+Yokozuna you cover 1 and 3 which is a pretty good combination, 
as a side note, maybe, just maybe, if you divide and conquer you may 
be able to remove the Set concept and just work with CRDT or siblings, 
let me explain briefly:


CRDT would warranty your distributed CAS operations per your designed 
key, instead of treating the keys as a Set, treat them independently and 
each CRDT key to manage its own states, that way you get the atomicity 
you want without worrying about locking or distributed concurrency in a 
collection of items since each key is unique and each would control its 
own state, or let each key have siblings and conflict resolution, as 
long as you don't create too many siblings per key in a very small 
amount of time.


The the only show stopper for CRDT is if you need to do operations in 
more than one atomically, then you will have to use siblings, even for 
siblings, your key approach would be to try to operate atomically in the 
minimum amount of states as possible so that you can avoid using Sets or 
Queues.


I hope I have expressed my thoughts well since explaining all this can 
get complicated really quick, or become a really long talk.


Hope that helps,

Guido.

On 12/09/13 17:45, Alex Rice wrote:

Hey all, thanks for the feedback, this is interesting!

re: Those CRDT white papers look pretty complicated, I will definitely
will leave that to you hardcore mathematicians and computer scientists
:)
re: Riak 2.0, I see there are git branches for 2.0 related stuff in
Riak and in corrugatediron, I am guessing it's still too rough around
the edges for a newbie like me to be messing around with?
re: RabbitMQ I had heard of it but am hesitant to add another
daemon/system dependency. But I am  definitely considering your
advice, Guido!

Also for this particular project (a multiplayer game) I may be able to
get away with Sets, don't actually really need proper Queues. Actually
I'm not sure. Just thinking out loud here.

For a Set implementation in Riak, I am thinking of modeling it after
this blog post, I cannot find which at the moment, which described how
to implement a lock-free Set in Redis. Redis is not distributed, but I
think the data structure is still relevant:

* To get around the eventual-consistency issue, a prerequisite would
be to always use R=1 and W=all. This will optimize it for reads
because there will be a process polling Riak to remove messages out of
the Set.
*  In the Redis implementation, they used the atomic APPEND operation
to append entries into the set, as a list but with notation to add or
remove, prefixed with + or -. e.g.
  +itemA
  +itemB
  -itemA
  +itemC
  = yields { itemB, itemC }
* Obviously Riak doesn't have atomic appends, however setting a bucket
to allow_mult = true, and always Put-ing with null vector clock, then
the Siblings feature should be equivalent. Instead of a value being
appended to, the Siblings would be populated with +itemA, +itemB,
-itemA, +itemC, etc.
* Any process reading the Set is also be responsible for collating and
Put-ing the condensed set back, after it reaches some size limit or I
think they called it 'fragmentation'.  This condense and re-Put
operation would be the tricky part, and would need to use vector clock
and ensure there are 0 siblings when finished. But it should be
possible? It seems like this is an uber-simplified form of a CRDT data
structure?

Thanks,
Alex













On Thu, Sep 12, 2013 at 2:12 AM, Guido Medina guido.med...@temetra.com wrote:

Alex,

RabbitMQ which is a good high performer, developed in Erlang and scales just
as Riak.

The old saying, the right tool for the right job, I like how fast Riak is
fetching/storing key values on a distributed environment, I don't like Riak
for queues, is it because it wasn't designed for that? CRDT like CAS
operations on structures (to mention some for Java) like LinkedBlockingQueue
and ConcurrentLinkedQueue should be painful enough to develop in a CRDT
environment, cause now you would be talking of guarding states of few within
the same structure; say AtomicReference of several variables to warranty a
non-blocking CAS operation, in this case heap and tail of the queue (Not so
true for LinkedBlockingQueue)

And performance of several CRDT for high performance queue I don't actually
think is going to be good.

If I were to use Riak for my big data environment and would like to match
Riak on a distributed queueing system then I would use like I said,
RabbitMQ.

Guido.


On 12/09/13 00:29, Alex Rice wrote:

Hi I'm very new to Riak. The allow_mult = true /  Siblings feature is
very

Re: Including @RiakKey and @RiakIndexes in the stored JSON

[Guido Medina]

Create a pseudo getters for the 2i indexes, valid return types are:
String, Long (And Integer) and Set of any of the mentioned, the benefit 
of this is the fact that your 2i indexes are not actual properties, they 
are meant to be computation of something, example:


/public class Postcode {//

//@//RiakKey//
//String postcode//
//...//
//...//
//
//@RiakIndex(sector)//
//@JsonIgnore//
//public String getSector() {//
//return postcode.substring(0,postcode.length() - 4//);//
//}//

//}//
/
In this example we create a 2i index for the sector but don't store it 
in the Riak JSON.


HTH,

Guido.

On 22/08/13 16:32, mex wrote:

If I declare @RiakKey and 2i indexes (@RiakIndex) for some fields in my
Item class, then those fields will not be displayed when querying the
record over the browser (ie. http://localhost/riak/myBucket/myKey1).

I have tried adding the annotation @JSONInclude but does not seem to change
the behaviour. Is there a way around it?

Thanks



--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Including-RiakKey-and-RiakIndexes-in-the-stored-JSON-tp4028933.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: vm.args change for 15% to 80% improvement in leveldb

[Guido Medina]

For the following information should it be +S 4:4 or +S 4:8?

root@somehost# lshw -C processor
  *-cpu
   description: CPU
   product: Intel(R) Core(TM) i7 CPU 930  @ 2.80GHz
   vendor: Intel Corp.
   physical id: 4
   bus info: cpu@0
   version: Intel(R) Core(TM) i7 CPU 930  @ 2.80GHz
   serial: To Be Filled By O.E.M.
   slot: CPU 1
   size: 1600MHz
   capacity: 1600MHz
   width: 64 bits
   clock: 133MHz
   capabilities: x86-64 fpu fpu_exception wp vme de pse tsc msr pae 
mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr 
sse sse2 ss ht tm pbe syscall nx rdtscp constant_tsc arch_perfmon pebs 
bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor 
ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida 
dtherm tpr_shadow vnmi flexpriority ept vpid cpufreq

   configuration: cores=4 enabledcores=4 threads=8

Thanks,

Guido.

On 14/08/13 01:38, Matthew Von-Maszewski wrote:

** The following is copied from Basho's leveldb wiki page:

https://github.com/basho/leveldb/wiki/Riak-tuning-1



Summary:

leveldb has a higher read and write throughput in Riak if the Erlang scheduler 
count is limited to half the number of CPU cores. Tests have demonstrated 
improvements of 15% to 80% greater throughput.

The scheduler limit is set in the vm.args file:

+S x:x

where x is the number of schedulers Erlang may use. Erlang's default value of x is 
the total number of CPUs in the system. For Riak installations using leveldb, the recommendation is to set 
x to half the number of CPUs. Virtual environments are not yet tested.

Example: for 24 CPU system

+S 12:12

Discussion:

We have tested a limited number of CPU configurations and customer loads. In all cases, 
there is a performance increase when the +S option is added to the vm.args file to reduce 
the number of Erlang schedulers. The working hypothesis is that the Erlang schedulers 
perform enough busy wait work that they always create context switch away 
from leveldb when leveldb is actually the only system task with real work.

The tests included 8 CPU (no hyper threading, physical cores only) and 24 CPU 
(12 physical cores with hyper threading) systems. All were 64bit Intel 
platforms. Generalized findings:

• servers running higher number of vnodes (64) had larger performance 
gains than those with fewer (8)
• servers running SSD arrays had larger performance gains than those 
running SATA arrays
• Get and Write operations showed performance gains, 2i query 
operations (leveldb iterators) were unchanged
• Not recommended for servers with less than 8 CPUs (go no lower than 
+S 4:4)

Performance improvements were as high as 80% over extended, heavily loaded 
intervals on servers with SSD arrays and 64 vnodes. No test resulted in worse 
performance due to the addition of +S x:x.

The +S x:x configuration change does not have to be implemented simultaneously 
to an entire Riak cluster. The change may be applied to a single server for 
verification. Steps: update the vm.args file, then restart the Riak node. 
Erlang command line changes to schedules were ineffective.

This configuration change has been running in at least one large, 
multi-datacenter production environment for several months.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: vm.args change for 15% to 80% improvement in leveldb

[Guido Medina]

Hi Matthew,

It is a bit confusing, cause let's say +S C:T is defined as:

C = Physical cores?
T = Total threads or total - physical cores?

Is it a sum of or the physical cores and total of threads? That's the 
confusing part, let's say you have a server with 8 physical cores with 
no hyper threading, so total threads is also 8, would that still be +S 
4:4, +S 8:8 or +S 8:0


Thanks,

Guido.

On 14/08/13 15:41, Matthew Von-Maszewski wrote:

threads=8 is the key phrase … +S 4:4

On Aug 14, 2013, at 10:04 AM, Guido Medina guido.med...@temetra.com wrote:


For the following information should it be +S 4:4 or +S 4:8?

root@somehost# lshw -C processor
  *-cpu
   description: CPU
   product: Intel(R) Core(TM) i7 CPU 930  @ 2.80GHz
   vendor: Intel Corp.
   physical id: 4
   bus info: cpu@0
   version: Intel(R) Core(TM) i7 CPU 930  @ 2.80GHz
   serial: To Be Filled By O.E.M.
   slot: CPU 1
   size: 1600MHz
   capacity: 1600MHz
   width: 64 bits
   clock: 133MHz
   capabilities: x86-64 fpu fpu_exception wp vme de pse tsc msr pae mce cx8 
apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht 
tm pbe syscall nx rdtscp constant_tsc arch_perfmon pebs bts rep_good nopl 
xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 
cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi 
flexpriority ept vpid cpufreq
   configuration: cores=4 enabledcores=4 threads=8

Thanks,

Guido.

On 14/08/13 01:38, Matthew Von-Maszewski wrote:

** The following is copied from Basho's leveldb wiki page:

https://github.com/basho/leveldb/wiki/Riak-tuning-1



Summary:

leveldb has a higher read and write throughput in Riak if the Erlang scheduler 
count is limited to half the number of CPU cores. Tests have demonstrated 
improvements of 15% to 80% greater throughput.

The scheduler limit is set in the vm.args file:

+S x:x

where x is the number of schedulers Erlang may use. Erlang's default value of x is 
the total number of CPUs in the system. For Riak installations using leveldb, the recommendation is to set 
x to half the number of CPUs. Virtual environments are not yet tested.

Example: for 24 CPU system

+S 12:12

Discussion:

We have tested a limited number of CPU configurations and customer loads. In all cases, 
there is a performance increase when the +S option is added to the vm.args file to reduce 
the number of Erlang schedulers. The working hypothesis is that the Erlang schedulers 
perform enough busy wait work that they always create context switch away 
from leveldb when leveldb is actually the only system task with real work.

The tests included 8 CPU (no hyper threading, physical cores only) and 24 CPU 
(12 physical cores with hyper threading) systems. All were 64bit Intel 
platforms. Generalized findings:

• servers running higher number of vnodes (64) had larger performance 
gains than those with fewer (8)
• servers running SSD arrays had larger performance gains than those 
running SATA arrays
• Get and Write operations showed performance gains, 2i query 
operations (leveldb iterators) were unchanged
• Not recommended for servers with less than 8 CPUs (go no lower than 
+S 4:4)

Performance improvements were as high as 80% over extended, heavily loaded 
intervals on servers with SSD arrays and 64 vnodes. No test resulted in worse 
performance due to the addition of +S x:x.

The +S x:x configuration change does not have to be implemented simultaneously 
to an entire Riak cluster. The change may be applied to a single server for 
verification. Steps: update the vm.args file, then restart the Riak node. 
Erlang command line changes to schedules were ineffective.

This configuration change has been running in at least one large, 
multi-datacenter production environment for several months.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Java Client - 2i indexes streaming

[Guido Medina]

Hi,

I have to say it is nice, we started using it today and it seems to 
leave a very low CPU and memory footprint at both cluster and 
application using the client, now I have a couple of questions:


1. This one is probably part of Riak 1.4.x but won't hurt to ask: Will
   reduce identity (to count keys) MapReduce job use a streamed 2i
   input instead?
2. If I decide to stream index and keys (withReturnKeyAndIndex), how do
   I get a Collection/List of IRiakObject(s)?


Thanks,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: different versions in upgrade

[Guido Medina]

Same here, except that Riak 1.3.2 did that for me automatically. As 
Jeremiah mentioned, you should go first to 1.3.2 on all nodes, per node 
the first time Riak starts it will take some time upgrading the 2i 
indexes storage format, if you see any weirdness then execute 
riak-admin reformat-indexes as soon as you upgrade a node, 1 by 1.


Before you even start read the release notes for each major version 
besides the rolling upgrade doc:


*Riak 1.3.2:* https://github.com/basho/riak/blob/1.3/RELEASE-NOTES.md
*Riak 1.4.1:* https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md

HTH,

Guido.

On 13/08/13 13:41, Bhuwan Chawla wrote:

Having done a similar upgrade, a gotcha to keep in mind:


Note for Secondary Index users
If you use Riak's Secondary Indexes and are upgrading from a version 
prior to Riak version 1.3.1, you need to reformat the indexes using 
the riak-admin reformat-indexes command




On Tue, Aug 13, 2013 at 8:36 AM, Jeremiah Peschka 
jeremiah.pesc...@gmail.com mailto:jeremiah.pesc...@gmail.com wrote:


From
http://docs.basho.com/riak/latest/ops/running/rolling-upgrades/ it
looks like you should upgrade to 1.3.2 and then 1.4.1

Depending on how badly you need the extra capacity, it would
probably be better to start by upgrading all nodes and then adding
the new one.

-- 
Jeremiah Peschka - Founder, Brent Ozar Unlimited

MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop

On Aug 13, 2013, at 5:06 AM, Louis-Philippe Perron
lpper...@gmail.com mailto:lpper...@gmail.com wrote:


Hi riak peoples,
I'm in the process of adding a new node to a aging (1 node)
cluster.  I would like to know what would be the prefered
incrementing upgrade to get all my nodes on the latest riak
version.  The best scenario would also have the least downtime.
 The old node is at riak version 1.2.1.

My actual plan is:

- install riak 1.4.1 on the new node
- add the new 1.4.1 node to the old 1.2.1 cluster.
- bring the 1.2.1 node offline
- upgrade the 1.2.1 node to 1.4.1
- put the upgraded node back online

will this work?
thanks!


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: different versions in upgrade

[Guido Medina]

Also, in theory if you have at least 5 nodes in the cluster one node 
down at a time doesn't stop your cluster from working properly.


You could do the following node by node which I have done several times:

1. Stop Riak on the upgrading node and in another node mark the
   upgrading node as down (riak-admin down riak@upgrading-node)
2. Upgrade Riak on that node to version 1.3.2, start it up and wait
   till it is completely operative (riak_kv is up and all transfers are
   finished, check by typing riak-admin transfers)
3. For Riak 1.3.2 once the step is over, type riak-admin
   reformat-indexes and tail -f /var/log/riak/console.log which should
   be done really fast if there isn't anything to fix.
4. Do 1 to 3 per node.
5. Do 1 and 2 but for for Riak 1.4.1.


HTH,

Guido.

On 13/08/13 13:50, Guido Medina wrote:
Same here, except that Riak 1.3.2 did that for me automatically. As 
Jeremiah mentioned, you should go first to 1.3.2 on all nodes, per 
node the first time Riak starts it will take some time upgrading the 
2i indexes storage format, if you see any weirdness then execute 
riak-admin reformat-indexes as soon as you upgrade a node, 1 by 1.


Before you even start read the release notes for each major version 
besides the rolling upgrade doc:


*Riak 1.3.2:* https://github.com/basho/riak/blob/1.3/RELEASE-NOTES.md
*Riak 1.4.1:* https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md

HTH,

Guido.

On 13/08/13 13:41, Bhuwan Chawla wrote:

Having done a similar upgrade, a gotcha to keep in mind:


Note for Secondary Index users
If you use Riak's Secondary Indexes and are upgrading from a version 
prior to Riak version 1.3.1, you need to reformat the indexes using 
the riak-admin reformat-indexes command




On Tue, Aug 13, 2013 at 8:36 AM, Jeremiah Peschka 
jeremiah.pesc...@gmail.com mailto:jeremiah.pesc...@gmail.com wrote:


From
http://docs.basho.com/riak/latest/ops/running/rolling-upgrades/ it looks
like you should upgrade to 1.3.2 and then 1.4.1

Depending on how badly you need the extra capacity, it would
probably be better to start by upgrading all nodes and then
adding the new one.

-- 
Jeremiah Peschka - Founder, Brent Ozar Unlimited

MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop

On Aug 13, 2013, at 5:06 AM, Louis-Philippe Perron
lpper...@gmail.com mailto:lpper...@gmail.com wrote:


Hi riak peoples,
I'm in the process of adding a new node to a aging (1 node)
cluster.  I would like to know what would be the prefered
incrementing upgrade to get all my nodes on the latest riak
version.  The best scenario would also have the least downtime.
 The old node is at riak version 1.2.1.

My actual plan is:

- install riak 1.4.1 on the new node
- add the new 1.4.1 node to the old 1.2.1 cluster.
- bring the 1.2.1 node offline
- upgrade the 1.2.1 node to 1.4.1
- put the upgraded node back online

will this work?
thanks!


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client - conflict resolver on both fetch() and store()?

[Guido Medina]

Hi Matt,

Like Sean said, you should have a mutator if you are dealing with 
conflict resolution in domain objects; a good side effect of using a 
mutator is that Riak Java client will fetch-modify-write so your 
conflict resolver will be called once(?), if you don't use mutators, you 
get the effect you are describing(?) or in other words, you have to 
treat the operations as non-atomic and do things twice.


There are two interfaces for mutations: *Mutation* and 
*ConditionalStoreMutation*, the 2nd interface will write only if the 
object was actually mutated, you must return true or false to state if 
it was mutated or not, which can be helpful if you are mutating an 
object and you discover the change you are requesting to make was 
already in place, then to save I/O, siblings creation and all implied on 
a write operation you decide not to write back.


Mutation and conflict resolution are two separate concerns, but if you 
specify a mutator and a conflict resolver, conflict resolution will 
happen after the object is fetched and it is ready to be modified, which 
will emulate an atomic operation if you use a domain object.


If you use a raw RiakObject, you must fetch, resolve the conflicts and 
on the write operation pass the VClock which is not a trivial nor easy 
to understand in code.


HTH,

Guido.


On 11/08/13 03:32, Sean Cribbs wrote:
I'm sure Roach will correct me if I'm off-base, but I believe the 
store operation does a fetch and resolve before writing. I think the 
ideal way to do that is to create a MutationT (T being your POJO) as 
well, in which case it's less of a store and more of a 
fetch-modify-write. The way to skip the fetch/modify is to use the 
withoutFetch() option on the operation builder.



On Sat, Aug 10, 2013 at 6:50 PM, Matt Painter m...@deity.co.nz 
mailto:m...@deity.co.nz wrote:


Hi,

I've just rolled up my sleeves and have started to make my
application more robust with conflict resolution.

I am currently using a @RiakVClock in my POJO (I need to think
more about whether the read/modify/write approach is preferable or
whether I'd have to rearchitect things).

I read in the Riak Handbook the recommendation that conflicts are
best resolved on read -  not write - however the example App.java
snipping on the Storing data in Riak

https://github.com/basho/riak-java-client/wiki/Storing-data-in-riak#appjava 
page
in the Java client's doco uses a resolver on both the store() and
fetch() operations.

Indeed, if I don't specify my conflict resolver in my store(),
things blow up (in my unit test, mind - I'm still getting my head
around the whole area so my test may be a bit contrived).

However when I use it in both places, my conflicts are being
resolved twice. Is this anticipated?

My store is:


bucket.store(record).returnBody(true).withoutFetch().withResolver(myConflictResolver);

and my fetch is:

bucket.fetch(id,Record.class).withResolver(myConflictResolver).execute();

The order of operations in my test is:

 1. Store new record
 2. Fetch the record as firstRecord
 3. Fetch the record as secondRecord
 4. Modify a field on firstRecord and secondRecord
 5. Save firstRecord
 6. Save secondRecord- this invokes my resolver with two siblings
 7. Read record - this also invokes my resolver with the two siblings

Am I missing something? Or is this what's supposed to happen? I'm
not too worried - the double-handling is hardly that intensive -
but I'm keen to get it right.

Thanks in advance,
Matt

___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
Sean Cribbs s...@basho.com mailto:s...@basho.com
Software Engineer
Basho Technologies, Inc.
http://basho.com/


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client - conflict resolver on both fetch() and store()?

[Guido Medina]

Hi Brian,

I probably asked a similar question before, let's say you have an 
in-memory cache and a single writer (I know, not the best distributed 
design), if you do the following, take into account that we use 
mutations but we have no siblings enabled:


/bucket.store(record).returnBody(true).withoutFetch().withMutation(myMutation);/

Will it do the following?

1. Not fetch the object from Riak?
2. Mutate the passed object?
3. Return the mutated object?

I know I can probably go and see the source code, so to save sometime 
don't worry about the specifics, a yes no with a very short reason of 
why not will suffice.


Thanks,

Guido.


On 11/08/13 18:11, Brian Roach wrote:

Matt,

The original design of StoreObject (which is what Bucket.store()
returns) was that it would encapsulate the entire read/modify/write
cycle in a very Java-y / enterprise-y way. This is why it takes a
Resolver and a Mutator; it does a fetch, resolves conflicts, passes
the resolved object to the Mutator, then stores the result of the
mutation to Riak.

Several users put in requests to make the fetch/resolve portion of
that optional as they had a workflow where that wasn't ideal and
didn't wanted to store a previously fetched value without fetching it
again. This is why the 'withoutFetch()' method was introduced along
with the @RiakVClock annotation.

When using withoutFetch() no fetch is performed, and no conflict
resolution occurs. Any ConflictResolver you pass in is simply not used
/ ignored ... except possibly if you're using returnBody()

Your code here:

bucket.store(record).returnBody(true).withoutFetch().withResolver(myConflictResolver);

is not doing a fetch or conflict resolution before storing your data.
It's just storing `record` in Riak. If that POJO has a vclock from a
previous fetch available via a @RiakVClock annotated field it will be
used. Otherwise, you're doing a store without a vclock.

I suspect where your confusion is stemming from is that you've also
specified 'returnBody()' and you're creating a sibling in that store
operation. When that's the case the body is going to be multiple
objects (all the siblings) which require resolution as
StoreObject.execute() only returns a single object back to the caller.
The same Resolver used if you had done the pre-fetch is employed. If
you haven't passed in a Resolver then the DefaultResolver is used
which ... isn't really a resolver - it simply passes through an
object if there's only one, or throws an exception if there's multiple
(siblings) present.

Thanks,
- Roach




On Sun, Aug 11, 2013 at 5:41 AM, Guido Medina guido.med...@temetra.com wrote:

Hi Matt,

Like Sean said, you should have a mutator if you are dealing with conflict
resolution in domain objects; a good side effect of using a mutator is that
Riak Java client will fetch-modify-write so your conflict resolver will be
called once(?), if you don't use mutators, you get the effect you are
describing(?) or in other words, you have to treat the operations as
non-atomic and do things twice.

There are two interfaces for mutations: Mutation and
ConditionalStoreMutation, the 2nd interface will write only if the object
was actually mutated, you must return true or false to state if it was
mutated or not, which can be helpful if you are mutating an object and you
discover the change you are requesting to make was already in place, then to
save I/O, siblings creation and all implied on a write operation you decide
not to write back.

Mutation and conflict resolution are two separate concerns, but if you
specify a mutator and a conflict resolver, conflict resolution will happen
after the object is fetched and it is ready to be modified, which will
emulate an atomic operation if you use a domain object.

If you use a raw RiakObject, you must fetch, resolve the conflicts and on
the write operation pass the VClock which is not a trivial nor easy to
understand in code.

HTH,

Guido.



On 11/08/13 03:32, Sean Cribbs wrote:

I'm sure Roach will correct me if I'm off-base, but I believe the store
operation does a fetch and resolve before writing. I think the ideal way to
do that is to create a MutationT (T being your POJO) as well, in which
case it's less of a store and more of a fetch-modify-write. The way to
skip the fetch/modify is to use the withoutFetch() option on the operation
builder.


On Sat, Aug 10, 2013 at 6:50 PM, Matt Painter m...@deity.co.nz wrote:

Hi,

I've just rolled up my sleeves and have started to make my application
more robust with conflict resolution.

I am currently using a @RiakVClock in my POJO (I need to think more about
whether the read/modify/write approach is preferable or whether I'd have to
rearchitect things).

I read in the Riak Handbook the recommendation that conflicts are best
resolved on read -  not write - however the example App.java snipping on the
Storing data in Riak page in the Java client's doco uses a resolver on both
the store() and fetch() operations.

Indeed, if I don't

Re: Java client - conflict resolver on both fetch() and store()?

[Guido Medina]

Brian,

In StoreObject's execute() method, this condition, is it a bug or intended?

...
...
if (!doNotFetch) {
resolved = fetchObject.execute();
vclock = fetchObject.getVClock();
}
...
...

My reasoning is: if do not fetch then shouldn't the resolved object be 
the one passed? I'm doing some tests and if I do store a mutation 
returning the body without fetching, I get a new mutated object and not 
the one I passed + mutation. So I'm wondering if that was the original 
intention.


Thanks,

Guido.

On 11/08/13 18:49, Guido Medina wrote:

Hi Brian,

I probably asked a similar question before, let's say you have an 
in-memory cache and a single writer (I know, not the best distributed 
design), if you do the following, take into account that we use 
mutations but we have no siblings enabled:


/bucket.store(record).returnBody(true).withoutFetch().withMutation(myMutation);/

Will it do the following?

 1. Not fetch the object from Riak?
 2. Mutate the passed object?
 3. Return the mutated object?

I know I can probably go and see the source code, so to save sometime 
don't worry about the specifics, a yes no with a very short reason of 
why not will suffice.


Thanks,

Guido.


On 11/08/13 18:11, Brian Roach wrote:

Matt,

The original design of StoreObject (which is what Bucket.store()
returns) was that it would encapsulate the entire read/modify/write
cycle in a very Java-y / enterprise-y way. This is why it takes a
Resolver and a Mutator; it does a fetch, resolves conflicts, passes
the resolved object to the Mutator, then stores the result of the
mutation to Riak.

Several users put in requests to make the fetch/resolve portion of
that optional as they had a workflow where that wasn't ideal and
didn't wanted to store a previously fetched value without fetching it
again. This is why the 'withoutFetch()' method was introduced along
with the @RiakVClock annotation.

When using withoutFetch() no fetch is performed, and no conflict
resolution occurs. Any ConflictResolver you pass in is simply not used
/ ignored ... except possibly if you're using returnBody()

Your code here:

bucket.store(record).returnBody(true).withoutFetch().withResolver(myConflictResolver);

is not doing a fetch or conflict resolution before storing your data.
It's just storing `record` in Riak. If that POJO has a vclock from a
previous fetch available via a @RiakVClock annotated field it will be
used. Otherwise, you're doing a store without a vclock.

I suspect where your confusion is stemming from is that you've also
specified 'returnBody()' and you're creating a sibling in that store
operation. When that's the case the body is going to be multiple
objects (all the siblings) which require resolution as
StoreObject.execute() only returns a single object back to the caller.
The same Resolver used if you had done the pre-fetch is employed. If
you haven't passed in a Resolver then the DefaultResolver is used
which ... isn't really a resolver - it simply passes through an
object if there's only one, or throws an exception if there's multiple
(siblings) present.

Thanks,
- Roach




On Sun, Aug 11, 2013 at 5:41 AM, Guido Medinaguido.med...@temetra.com  wrote:

Hi Matt,

Like Sean said, you should have a mutator if you are dealing with conflict
resolution in domain objects; a good side effect of using a mutator is that
Riak Java client will fetch-modify-write so your conflict resolver will be
called once(?), if you don't use mutators, you get the effect you are
describing(?) or in other words, you have to treat the operations as
non-atomic and do things twice.

There are two interfaces for mutations: Mutation and
ConditionalStoreMutation, the 2nd interface will write only if the object
was actually mutated, you must return true or false to state if it was
mutated or not, which can be helpful if you are mutating an object and you
discover the change you are requesting to make was already in place, then to
save I/O, siblings creation and all implied on a write operation you decide
not to write back.

Mutation and conflict resolution are two separate concerns, but if you
specify a mutator and a conflict resolver, conflict resolution will happen
after the object is fetched and it is ready to be modified, which will
emulate an atomic operation if you use a domain object.

If you use a raw RiakObject, you must fetch, resolve the conflicts and on
the write operation pass the VClock which is not a trivial nor easy to
understand in code.

HTH,

Guido.



On 11/08/13 03:32, Sean Cribbs wrote:

I'm sure Roach will correct me if I'm off-base, but I believe the store
operation does a fetch and resolve before writing. I think the ideal way to
do that is to create a MutationT (T being your POJO) as well, in which
case it's less of a store and more of a fetch-modify-write. The way to
skip the fetch/modify is to use the withoutFetch() option on the operation
builder.


On Sat, Aug 10

Re: Java client - conflict resolver on both fetch() and store()?

[Guido Medina]

I hate it too but my point still stands, if DO NOT FETCH, what's the 
target object the mutation should work with? Isn't it the passed object 
instead?


Anyway, I sent a pull request which hopefully applies a better 
semantics: https://github.com/basho/riak-java-client/pull/271


Thanks,

Guido.

On 11/08/13 20:45, YN wrote:

Guido,

In this case it appears that fetching is enabled i.e. if (!doNotFetch) 
i.e. if NOT doNotFetch... so basically doNotFetch = false (fetching is 
true / enabled).


I hate the double negative cases since it's easy to get confused / 
miss the logic that was intended.

YN shared this with you.
Re: Java client - conflict resolver on both fetch() and store()? 
http://permalink.gmane.org/gmane.comp.db.riak.user/12099
Via gmane.comp.db.riak.user 
http://blog.gmane.org/gmane.comp.db.riak.user

Brian,

In StoreObject's execute() method, this condition, is it a bug or intended?

  ...
  ...
  if (!doNotFetch) {
  resolved = fetchObject.execute();
  vclock = fetchObject.getVClock();
  }
  ...
  ...

My reasoning is: if do not fetch then shouldn't the resolved object be
the one passed? I'm doing some tests and if I do store a mutation
returning the body without fetching, I get a new mutated object and not
the one I passed + mutation. So I'm wondering if that was the original
intention.

Thanks,

Guido.

On 11/08/13 18:49, Guido Medina wrote:

___
riak-users mailing list
riak-users at lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Sent from InoReader http://www.inoreader.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Java client - without fetch and mutation?

[Guido Medina]

Hi Brian,

*/New thread for this, sorry for the hijacking./*

Yes, without fetch should be indeed used without mutation or conflict 
resolution, originally we had mutations and siblings, but our 
application ended up creating too many siblings and made Riak fail 
miserably, so we disable the siblings and left the mutation in place 
cause it is nice for separation of concerns and a good idiom for 
fetch-modify-write.


We have two types of objects, the ones that are created and never 
modified and others which contain historical data, and we use Riak 
because the AP part of the CAP theorem but we have to deliver consistent 
data so we have a sort of DAO pattern which fetch from cache if present 
or Riak, then locks using a well written in-memory locking mechanism, 
applies mutation to cached object.


For performance reasons and let me say this, for single reader/writer it 
provides a very high performance cause we read and/or write optionally 
using withoutFetch() and/or ConditionalStoreMutationT


All that said, the little sense of withoutFetch() and mutation got 
some value for us, I guess it is a mutation side effect, having all the 
logic to verify that an object has changed or not, locking mechanism and 
caching would be too difficult and confusing without the current Riak 
Java client mutation mechanism, we have been using it for over a year 
now and we haven't seen any weirdness in it, not even once, I mean the 
weirdness related with concurrent applications.


Hope with this long explanation (sorry about that) you now connect the 
pieces of our pull requests,


Let me know your thoughts,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Nailing down the bucket design?

[Guido Medina]

As a 2nd thought, you could have a key per player on the player's bucket 
and a key with the collection of units per player on the unit's bucket.


Guido.

On 07/08/13 15:52, Guido Medina wrote:
Whats the size of each unit JSON wise?, if it is too small, you could 
have the player's units inside a single key as a collection, that way 
when you fetch a player your key will contain the units and you could 
play around with mutations/locking of such player's key. And also, it 
will leverage your multi-threading code making each player thread your 
multi-threading pattern (Removing the need of a thread pool to fetch 
units)


The drawback of such design is that each player's key would be fetched 
as a whole just to add/remove a unit. If you have a single 
reader/writer (Server), you could add some in-memory cache (For 
example, in Java you could use Google's Guava framework; LoadingCacheT)


Maybe I proposed something with way out of proportions, I think I 
would need more details.


Anyway, HTH,

Guido.

On 07/08/13 10:28, Maksymilian Strzelecki wrote:
Hi. I've read somewhere around the Internet that Riak benefits its 
performence when there are more buckets and not massive amount of 
keys in them. Now, if this is true I'm asking for an advice on how to 
go about my data model.


I've started off with one bucket called accounts_units. Every player 
has his set of units in-game (like 10-20) and they all have their 
individual keys in there. Since knowing what your units are is pretty 
important thing I would be doing a lot of queries on that bucket. 
Then I thought I could create individual buckets for every player and 
his units. Though it would create A LOT of buckets with little keys. 
Would that be better? What do you think?


Thanks for your time,
Max.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Many-To-Many relationships

[Guido Medina]

Hi Massimiliano,

I think your design is very thorough, I wouldn't worry about the 
cardinality of such index but its per index size (how many keys a single 
2i index will return?) , in that case think of 2i as yet another 
relational DB (LevelDB), you should test it with many keys and check its 
performance, but from my perspective I see nothing wrong about it.


If that isn't good enough and what you need is a graph oriented search, 
the alternative could be Neo4J, though it is not as high available and 
scalable as Riak (In theory), and yes, you are right, for the 
many-to-many combinations you are talking about you will have to avoid 
MapReduce and use a well designed 2i, or use a MapReduce using a 2i 
result as input (As long as the list of keys per 2i index isn't too long)


You might have to do a couple of 2i queries to get your many-to-many 
list, then fetch key by key either single threaded or multi-threaded.


Hope that helps,

Guido.

On 04/08/13 19:43, Massimiliano Ciancio wrote:

Hi all,
I'm new to Riak and I'm evaluating its use in a project in which I've
to map some many-to-many relationship with millions of link.
That is, imagine you have objects of type X={x_0,x_1,...,x_n} and
objects of type Y={y_0,y_1,...,y_m} that are related. Xs are connected
to millions of Ys and viceversa.
If there wasn't a so high number I could put the relation inside the
objects X as a list of related Ys and viceversa.
But with this number I'll have a very big data field associated to each object.
I'm thinking that I can create a bucket relation X-Y in which I can
put objects with key x_i-y_j with value, i.e., True menaing that
x_i is related to y_j. In this way it will be simple to know if x_i is
related to y_j or viceversa: if the object with key x_i-y_j exists
then they are related. Ok. But now, how can I extract all the Ys
related to x_i or, viceversa, the Xs related to y_j?
Using map-reduce or similar will imply analizyng all the objects in the cluster.
I can use a secondary index on the objects of the bucket relation
X-Y. But I imagine it could give performance problems because of the
hundreds of millions of items in it.
So I thought of the following use of 2i: when I write a new relation
x_i-y_j I can add it to two secondary indexes, one named x_i-index
with value y_j and the other y_j-index with value x_i. In this way
I'll not have one big index but many (millions!) of smaller indexes.
Suppose I want all the Ys related to x_i: I can ask Riak for the
elements in the index x_i-index. In my mind this will be a not so
heavy query...
Now my questions are: is this approach viable? How many million of
small indexes impact on the riak's performances? Are there best
practices to do this?
Thanks in advance
Massimiliano

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Reuse of Buckets (Java Client)

[Guido Medina]

Yes, it is thread safe, you can treat them as singleton instances per 
bucket, the following order is the kind of the general usage pattern:


 * Fetch bucket.
 * Optional: If exists verify it has your application values (N value, etc)
 * If doesn't exist create it with your settings.
 * Cache it as a singleton instance (You could create a /final
   MapString, Bucket buckets=new HashMap()/) and re-use it in your
   application; assuming your initialization is not lazy, or if it is
   use proper thread safety initialization.

Hope that helps,

Guido.

On 31/07/13 08:45, Nico Huysamen wrote:
Is the Bucket class reusable and thread-safe? I.e. can I create my 
Bucket objects during instantiation of client class, and then reuse 
the same bucket for all operations for the application lifetime? Or 
should buckets be re-created for each request?


Thanks


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Querying multiple 2i via Intersections

[Guido Medina]

Rohman,

I think the reason for this is that the cluster will have to do the 
whole intersection in memory, 2i only provides queries for 1 single 
index and then return that result to either the client streaming or not, 
intersection indeed will require a MapReduce job to get a hold of both 
lists and intersect it, honestly, I wish that was possible, I think 
that's why I'm learning Erlang at the moment, we all wanted join like 
functionality with 2i to be honest but I don't think that is going to 
happen before version 1.5 if ever.


If you noticed, Riak 1.4 added 2i streaming which IMHO is like moving 
farer away from handling results in memory, if you get the point of what 
I mean. Join 2i queries in Riak will be an issue for a long time.


HTH,

Guido.

On 27/07/13 14:20, Antonio Rohman Fernandez wrote:

Hi there,

I didn't fully check the new release notes yet, but I guess that Riak 1.4 
doesn't have the ability to query multiple 2i at the same time... how difficult 
may be to implement a multiple 2i query via their intersecting items in their 
arrays? Maybe we could have this in future versions?

I normally do this with scripts before doing a MapReduce, if the data I want to 
look for is having 2 indexes conditions, I get the array of items of each 2i 
query, then get only the items found on both arrays and use it as a MapReduce 
input list.

However, done in my app's side is a bit costly as Riak needs to give me the 
full items arrays and then I need to process them to find repeated keys.

I really wish for Riak to be able to query multiple 2i soon as it will help a 
lot.

Merci,
Rohman

Sent from my iPhone
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Querying multiple 2i via Intersections

[Guido Medina]

Are you saying that you can join two 2i indexes? Let's say you have a 2i 
named date and another named category so you could get a list of 
modified objects on a date and for a specific category with Riak Search?


Guido.

On 27/07/13 16:32, Jeremiah Peschka wrote:
Both Riak Search (currently) and Yokozuna (future) provide the ability 
to perform joins of multiple indices on the same bucket objects.


I find it best to think of 2i as an extension of what Riak KV already 
does well - fast key based lookups.


Search and Yokozuna provide complex query semantics.

---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Sat, Jul 27, 2013 at 11:16 AM, Guido Medina 
guido.med...@temetra.com mailto:guido.med...@temetra.com wrote:


Rohman,

I think the reason for this is that the cluster will have to do
the whole intersection in memory, 2i only provides queries for 1
single index and then return that result to either the client
streaming or not, intersection indeed will require a MapReduce job
to get a hold of both lists and intersect it, honestly, I wish
that was possible, I think that's why I'm learning Erlang at the
moment, we all wanted join like functionality with 2i to be
honest but I don't think that is going to happen before version
1.5 if ever.

If you noticed, Riak 1.4 added 2i streaming which IMHO is like
moving farer away from handling results in memory, if you get the
point of what I mean. Join 2i queries in Riak will be an issue for
a long time.

HTH,

Guido.


On 27/07/13 14:20, Antonio Rohman Fernandez wrote:

Hi there,

I didn't fully check the new release notes yet, but I guess
that Riak 1.4 doesn't have the ability to query multiple 2i at
the same time... how difficult may be to implement a multiple
2i query via their intersecting items in their arrays? Maybe
we could have this in future versions?

I normally do this with scripts before doing a MapReduce, if
the data I want to look for is having 2 indexes conditions, I
get the array of items of each 2i query, then get only the
items found on both arrays and use it as a MapReduce input list.

However, done in my app's side is a bit costly as Riak needs
to give me the full items arrays and then I need to process
them to find repeated keys.

I really wish for Riak to be able to query multiple 2i soon as
it will help a lot.

Merci,
Rohman

Sent from my iPhone
___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Querying multiple 2i via Intersections

[Guido Medina]

I get you now, I was confused by the fact that he asked explicitly about 
2i, so when you mentioned indexes I thought you were referring to 2i, I 
would categorize Riak Search and Yokozuna as an alternative due to extra 
effort, schema definition and post-commits hooks, so for the straight 
question he asked the answer would be no, the alternatives, those you 
mentioned.


Guido.

On 27/07/13 17:12, Jeremiah Peschka wrote:
No, I never said or hinted that you could join 2i indices using Riak 
Search.


You can, however, query against multiple indexed fields with Riak 
Search. Here's what that would look like in CorrugatedIron:


varreq=newRiakSearchRequest
{
Query=newRiakFluentSearch(riak_search_bucket,date)
};

req.Query.Search(2012-01-03)
.And(category, sandwiches);

varresult=Client.Search(req);


This becomes a Riak Search query on the riak_search_bucket with the 
syntax date:2012-01-03 AND category:sandwiches


The trick with Riak Search is that the objects you're saving have to 
be readable by Riak Search. If you can't encode field names in a way 
that is expected by Search, you'll need to specify a search schema so 
that Riak Search knows which data types to index.


See Using Search[1] and Search Schema[2] for implementation details

[1]: http://docs.basho.com/riak/latest/dev/using/search/
[2]: http://docs.basho.com/riak/latest/dev/advanced/search-schema/

---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Sat, Jul 27, 2013 at 11:44 AM, Guido Medina 
guido.med...@temetra.com mailto:guido.med...@temetra.com wrote:


Are you saying that you can join two 2i indexes? Let's say you
have a 2i named date and another named category so you could
get a list of modified objects on a date and for a specific
category with Riak Search?

Guido.


On 27/07/13 16:32, Jeremiah Peschka wrote:

Both Riak Search (currently) and Yokozuna (future) provide the
ability to perform joins of multiple indices on the same bucket
objects.

I find it best to think of 2i as an extension of what Riak KV
already does well - fast key based lookups.

Search and Yokozuna provide complex query semantics.

---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Sat, Jul 27, 2013 at 11:16 AM, Guido Medina
guido.med...@temetra.com mailto:guido.med...@temetra.com wrote:

Rohman,

I think the reason for this is that the cluster will have to
do the whole intersection in memory, 2i only provides queries
for 1 single index and then return that result to either the
client streaming or not, intersection indeed will require a
MapReduce job to get a hold of both lists and intersect it,
honestly, I wish that was possible, I think that's why I'm
learning Erlang at the moment, we all wanted join like
functionality with 2i to be honest but I don't think that is
going to happen before version 1.5 if ever.

If you noticed, Riak 1.4 added 2i streaming which IMHO is
like moving farer away from handling results in memory, if
you get the point of what I mean. Join 2i queries in Riak
will be an issue for a long time.

HTH,

Guido.


On 27/07/13 14:20, Antonio Rohman Fernandez wrote:

Hi there,

I didn't fully check the new release notes yet, but I
guess that Riak 1.4 doesn't have the ability to query
multiple 2i at the same time... how difficult may be to
implement a multiple 2i query via their intersecting
items in their arrays? Maybe we could have this in future
versions?

I normally do this with scripts before doing a MapReduce,
if the data I want to look for is having 2 indexes
conditions, I get the array of items of each 2i query,
then get only the items found on both arrays and use it
as a MapReduce input list.

However, done in my app's side is a bit costly as Riak
needs to give me the full items arrays and then I need to
process them to find repeated keys.

I really wish for Riak to be able to query multiple 2i
soon as it will help a lot.

Merci,
Rohman

Sent from my iPhone
___
riak-users mailing list
riak-users@lists.basho.com
mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak

Java client stats question

[Guido Medina]

Hi,

Is there a way to get the JSON stats via PBC? This is how we are doing 
it now, we would like to get rid of any HTTP call, currently, this is 
the only call being made to HTTP:


/private void collectNodeInfo(final PBClientConfig clientConfig)//
//  {//
//...//
//RiakClusterStats stats=null;//
//try{//
//  stats=new RiakClusterStats();//
//  HttpClient client=new DefaultHttpClient();//
*HttpGet g=new HttpGet(http://; + clientConfig.getHost() + 
:8098/stats);*//

//  HttpResponse resonse=client.execute(g);//
//  JSONObject statsMap;//
//  InputStream contentStream=null;//
//  try{//
//contentStream=resonse.getEntity().getContent();//
//JSONTokener tok=new JSONTokener(contentStream);//
//statsMap=new JSONObject(tok);//
//stats.addNode(clientConfig.getHost(),statsMap);//
//  } finally{//
//if(contentStream != null){//
//  contentStream.close();//
//}//
//  }//
//} catch(Exception e){//
//  log.error(Huh? Exception when ,e);//
//}//
//lastClusterStats=stats;//
//  }/


Kind regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java client stats question

[Guido Medina]

Hi Brian,

We have a custom (and very small) admin page that shows Riak KV version, 
connected nodes and ring members, here is how it looks:


riak_kv_version 1.4.0
connected_nodes riak@node2
riak@node3
riak@node4
riak@node5
ring_membersriak@node1
riak@node2
riak@node3
riak@node4
riak@node5
sys_system_architecture x86_64-unknown-linux-gnu


That also tell us where the running app is working against, if dev or 
prod cluster cause nodes will be different. We also show some buckets 
information but that is currently on PBC.


Guido.


On 25/07/13 16:18, Brian Roach wrote:

Guido -

Right now, no.

We've been having some internal discussions around that topic and
whether it's really a client library operation or not.

How are you using stats? Is it for a monitoring app or ... ?

Thanks,
Brian Roach

On Thu, Jul 25, 2013 at 4:25 AM, Guido Medina guido.med...@temetra.com wrote:

Hi,

Is there a way to get the JSON stats via PBC? This is how we are doing it
now, we would like to get rid of any HTTP call, currently, this is the only
call being made to HTTP:

   private void collectNodeInfo(final PBClientConfig clientConfig)
   {
 ...
 RiakClusterStats stats=null;
 try{
   stats=new RiakClusterStats();
   HttpClient client=new DefaultHttpClient();
   HttpGet g=new HttpGet(http://; + clientConfig.getHost() +
:8098/stats);
   HttpResponse resonse=client.execute(g);
   JSONObject statsMap;
   InputStream contentStream=null;
   try{
 contentStream=resonse.getEntity().getContent();
 JSONTokener tok=new JSONTokener(contentStream);
 statsMap=new JSONObject(tok);
 stats.addNode(clientConfig.getHost(),statsMap);
   } finally{
 if(contentStream != null){
   contentStream.close();
 }
   }
 } catch(Exception e){
   log.error(Huh? Exception when ,e);
 }
 lastClusterStats=stats;
   }


Kind regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak_kv_memory_backend in production

[Guido Medina]

Since the data he is requiring to store is only transient, would it 
make sense to set N=2 for performance? Or will N=2 have the opposite 
effect due to amount of nodes having such replica?


Guido.

On 18/07/13 16:15, Jared Morrow wrote:

Kumar,

We have a few customers who use the memory backend.  The first example 
I could find (with the help of our CSE team) uses the memory backend 
on 8 machines with 12gb of ram each.


I know you are just testing right now, but we'd suggest using 5 node 
minimum.  With N=3 on a 3-node cluster you could be writing multiple 
replicas to the same machine.


Good luck in your testing,
-Jared




On Thu, Jul 18, 2013 at 8:38 AM, kpandey kumar.pan...@gmail.com 
mailto:kumar.pan...@gmail.com wrote:


Are there known production installation of riak that uses
riak_kv_memory_backend.  We have a need to store transient data
just in
memory ( never hitting persistent store). I'm testing riak on aws
with 3
node cluster and looks good so far.   Just wanted to find out what
kind of
setup people are using in production.

Thanks
Kumar



--
View this message in context:

http://riak-users.197444.n3.nabble.com/riak-kv-memory-backend-in-production-tp4028393.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak_kv_memory_backend in production

[Guido Medina]

Forgot to mention, with N=2 should he be able to have only 4 nodes and 
focus on RAM per node rather than 5?


I know is not recommended but shouldn't N=2 reduce the minimum 
recommended nodes to 4?


Guido.

On 18/07/13 16:21, Guido Medina wrote:
Since the data he is requiring to store is only transient, would it 
make sense to set N=2 for performance? Or will N=2 have the opposite 
effect due to amount of nodes having such replica?


Guido.

On 18/07/13 16:15, Jared Morrow wrote:

Kumar,

We have a few customers who use the memory backend.  The first 
example I could find (with the help of our CSE team) uses the memory 
backend on 8 machines with 12gb of ram each.


I know you are just testing right now, but we'd suggest using 5 node 
minimum.  With N=3 on a 3-node cluster you could be writing multiple 
replicas to the same machine.


Good luck in your testing,
-Jared




On Thu, Jul 18, 2013 at 8:38 AM, kpandey kumar.pan...@gmail.com 
mailto:kumar.pan...@gmail.com wrote:


Are there known production installation of riak that uses
riak_kv_memory_backend.  We have a need to store transient data
just in
memory ( never hitting persistent store). I'm testing riak on aws
with 3
node cluster and looks good so far.   Just wanted to find out
what kind of
setup people are using in production.

Thanks
Kumar



--
View this message in context:

http://riak-users.197444.n3.nabble.com/riak-kv-memory-backend-in-production-tp4028393.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com mailto:riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: TCP recv timeout and handoffs almost all the time

[Guido Medina]

If what you are describing is happening for 1.4, type riak-admin diag 
and see the new recommended kernel parameters, also, on vm.args 
uncomment the +zdbbl 32768 parameter, since what you are describing is 
similar to what happened to us when we upgraded to 1.4.


HTH,

Guido.

On 18/07/13 19:21, Simon Effenberg wrote:

Hi @list,

I see sometimes logs talking about hinted_handoff transfer of .. failed because of 
TCP recv timeout.
Also riak-admin transfers shows me many handoffs (is it possible to give some insights about 
how many handoffs happened through riak-admin status?).

- Is it a normal behavior to have up to 30 handoffs from/to different nodes?
- How can I get down to the problem with the TCP recv timeout? I'm not sure if 
this is a network problem or if the other node is too slow. The load is ok on 
the machines (some IOwait but not 100%). Maybe interfering with AAE?

Here the log information about the TCP recv timeout. But that is not that often 
but handoffs happens really often:

2013-07-18 16:22:05.654 UTC [error] 
0.28933.14@riak_core_handoff_sender:start_fold:216 hinted_handoff transfer of 
riak_kv_vnode from 'riak@10.46.109.207' 
1118962191081472546749696200048404186924073353216 to 'riak@10.46.109.205' 
1118962191081472546749696200048404186924073353216 failed because of TCP recv timeout
2013-07-18 16:22:05.673 UTC [error] 
0.202.0@riak_core_handoff_manager:handle_info:282 An outbound handoff of 
partition riak_kv_vnode 1118962191081472546749696200048404186924073353216 was 
terminated for reason: {shutdown,timeout}


Thanks in advance
Simon

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: TCP recv timeout and handoffs almost all the time

[Guido Medina]

Follow the white rabbit:

http://docs.basho.com/riak/latest/cookbooks/Linux-Performance-Tuning/

Most recommended parameters are on that link.

HTH,

Guido.

On 18/07/13 19:48, Simon Effenberg wrote:

Sounds like zdbbl.. I'm running 1.3.1 but it started after added 6 more
nodes to the previously 12 node cluster. So maybe it is because of a 18
node cluster?

I'll try the zdbbl stuff. Any other hint would be cool (if the new
kernel parameters are also good for 1.3.1.. could you provide them?).

Cheers
Simon

On Thu, 18 Jul 2013 19:34:18 +0100
Guido Medina guido.med...@temetra.com wrote:


If what you are describing is happening for 1.4, type riak-admin diag
and see the new recommended kernel parameters, also, on vm.args
uncomment the +zdbbl 32768 parameter, since what you are describing is
similar to what happened to us when we upgraded to 1.4.

HTH,

Guido.

On 18/07/13 19:21, Simon Effenberg wrote:

Hi @list,

I see sometimes logs talking about hinted_handoff transfer of .. failed because of 
TCP recv timeout.
Also riak-admin transfers shows me many handoffs (is it possible to give some insights about 
how many handoffs happened through riak-admin status?).

- Is it a normal behavior to have up to 30 handoffs from/to different nodes?
- How can I get down to the problem with the TCP recv timeout? I'm not sure if 
this is a network problem or if the other node is too slow. The load is ok on 
the machines (some IOwait but not 100%). Maybe interfering with AAE?

Here the log information about the TCP recv timeout. But that is not that often 
but handoffs happens really often:

2013-07-18 16:22:05.654 UTC [error] 
0.28933.14@riak_core_handoff_sender:start_fold:216 hinted_handoff transfer of 
riak_kv_vnode from 'riak@10.46.109.207' 
1118962191081472546749696200048404186924073353216 to 'riak@10.46.109.205' 
1118962191081472546749696200048404186924073353216 failed because of TCP recv timeout
2013-07-18 16:22:05.673 UTC [error] 
0.202.0@riak_core_handoff_manager:handle_info:282 An outbound handoff of 
partition riak_kv_vnode 1118962191081472546749696200048404186924073353216 was 
terminated for reason: {shutdown,timeout}


Thanks in advance
Simon

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

+zdbbl recommended value

[Guido Medina]

Hi,

We had an issue in 1.4 where 2i operations were timing out, after going 
through support we were suggested to use +zdbbl 16384, on the 
Configuring Riak Docs it is strongly suggested (unless doc need to be 
re-phrased) it should be higher:


*Source:* 
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/


At the end of such doc there is the following fragment and we would like 
to know if there is a kind of all situations balanced value and the 
implications:



 Performance  Capacity settings

It is strongly recommended that the following values be set in the Riak 
|vm.args| configuration file, which is located in the |/etc/riak| or 
|/opt/riak/etc| folder.


## This setting is not present in default Riak installations, so
## it should be added.  In some cases, a value of 128000 may be
## appropriate.
+zdbbl96000

## This setting is present in default Riak installations, so
## its value should be edited.
-env ERL_MAX_PORTS16384



Kind regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com