Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Matthew Dawson 
On July 15, 2013 02:16:47 PM Gabriel Littman wrote:
> Hi,
> 
> Another posibility is to create a new bucket for each test run.
> 
> Gabe
> 
Also a good idea, but not easily implemented against my current library.  I've 
got a solution in using random addition to keys that works for now.

My only problem is that this doesn't solve my underlying issues regarding keys 
disappearing.  Is the only way to ensure keys don't disappear is to never 
reuse a key?  I'd understand if this was caused by two random processes 
interacting, but in this case there is a significant (>1s) time between two 
runs where keys overlap.

Thanks for all the help,
-- 
Matthew

signature.asc
Description: This is a digitally signed message part.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Toby Corkindale 

On 16/07/13 07:16, Gabriel Littman wrote:

Another posibility is to create a new bucket for each test run.


+1

I think using a unique bucket name is definitely the only sane way to go.
It also prevents you from having weird issues if you run two tests 
simultaneously (as can easily happen once you have multiple developers 
and automated integration testing).



On Mon, Jul 15, 2013 at 12:06 PM, Matthew Dawson mailto:matt...@mjdsystems.ca>> wrote:

Hi Seth,
On July 14, 2013 02:25:02 PM Seth Bunce wrote:
 > I had a similar problem. I mitigated the problem by appending a
 > current timestamp (in nanoseconds) to the keys I'm using in my tests.
 > This way I don't have to worry about waiting for Riak to reap
 > tombstones after 3 seconds and I don't have to use
 > delete_mode=immediate. I still do a list keys + delete before each
 > test.
I think this is what I'm going to have to do.  I tried all the different
delete_mode settings, but none of them seemed to help.  Thanks for the
suggestion!



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Erlang PB client

2013-07-15 Thread Igor Karymov 
Mark, Dave, thanks you for your help.
Also any feedback and pull requests are welcome.


2013/7/16 Igor Karymov 

> We have two different unlinked riak clusters.
> One used for statistic aggregation across the system. This one heavily
> abuse riak secondary indexes range query feature, so we have to grow it
> with caution.
> Another one used as our main storage. Both clusters have different
> backends, different nodes count, and occasionally different riak version.
> Our library handle this use case.
> Also we implemented keep alive pings to each riak node inside each cluster
> (one special keep_alive connection to each node, that periodically call pb
> client ping function).
> This mechanism allow us handle gracefully downs and ups of riak nodes
>  inside one cluster (if a particular node of the cluster stops responding,
> the pool is temporarily stops route requests to it).
> With this small update external load balancer is no needed anymore.
>
>
> 2013/7/16 Mark Phillips 
>
>> Hi Igor,
>>
>>
>> On Mon, Jul 15, 2013 at 3:00 PM, Igor Karymov  wrote:
>>
>>> Unfortunately this library is not included into "community projects"
>>> wiki page list and i have no idea how-to fix this. Maybe somebody from
>>> basho guys can help us.
>>>
>>>
>>
>> I just went ahead and opened a pull request to add it the docs [1]. Feel
>> free to do so in the future when you've got something to add.
>>
>>
>> > Also our library provide interface wrapper and multicluster
>> configuration.
>>
>> Can you elaborate on how the multicluster configuration works and how
>> you are using it in production?
>>
>> Best,
>>
>> Mark
>>
>>
>> [1] https://github.com/basho/basho_docs/pull/489
>>
>>
>>>
>>> 2013/7/16 Konstantin Kalin 
>>>
 Thank you for sharing it. I built similar application already (pool +
 multi-node connections) after I had got confirmation that there is no
 "official way" :) Lucky it wasn't hard at all and laid down into existing
 prototype nicely (mostly because it's in-house app).

 Thank you,
 Konstantin.


 On Mon, Jul 15, 2013 at 2:43 PM, Igor Karymov wrote:

> Hi. Maybe this will be usefull for you:
> https://github.com/unisontech/uriak_pool
>
> We was have to write our own solution, because poolboy was not be able
> handle riak nodes ups and downs gracefully.
> Also our library provide interface wrapper and multicluster
> configuration.
>
>
> 2013/7/11 Sean Cribbs 
>
>> We intend to overhaul the Erlang client soon. In the meantime, Riak
>> CS has done exactly what Jeremy has suggested. Look here:
>> https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl
>>
>>
>> On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin <
>> konstantin.ka...@gmail.com> wrote:
>>
>>> Oops… was looking at wrong column. Sorry and thanks for the advice.
>>>
>>> Thank you,
>>> Konstantin.
>>>
>>> On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:
>>>
>>> The X there indicates that it does not support connection pooling
>>> out of the box in contrast to the check. I'd look at poolboy (to use in
>>> conjunction with riakc_pb_socket) and riakpool (which pulls in
>>> riak-erlang-client as a dependency).
>>>
>>>
>>> On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin <
>>> konstantin.ka...@gmail.com> wrote:
>>>
 Looking at
 http://docs.basho.com/riak/latest/references/Client-Libraries/ I
 see that Erlang riak client should support Cluster connections/pools. 
 But
 looking at Erlang riak client source code I would say that it doesn't
 support Cluster connections/pools out of box. And I have to develop my 
 own
 code to have load balancing/connection pool. Is it right and the
 documentation is incorrect and needs to be updated?

 Thank you,
 Konstantin.

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


>>>
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>> --
>> Sean Cribbs 
>> Software Engineer
>> Basho Technologies, Inc.
>> http://basho.com/
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

>>>
>>> __

Re: Erlang PB client

2013-07-15 Thread Igor Karymov 
We have two different unlinked riak clusters.
One used for statistic aggregation across the system. This one heavily
abuse riak secondary indexes range query feature, so we have to grow it
with caution.
Another one used as our main storage. Both clusters have different
backends, different nodes count, and occasionally different riak version.
Our library handle this use case.
Also we implemented keep alive pings to each riak node inside each cluster
(one special keep_alive connection to each node, that periodically call pb
client ping function).
This mechanism allow us handle gracefully downs and ups of riak nodes
 inside one cluster (if a particular node of the cluster stops responding,
the pool is temporarily stops route requests to it).
With this small update external load balancer is no needed anymore.


2013/7/16 Mark Phillips 

> Hi Igor,
>
>
> On Mon, Jul 15, 2013 at 3:00 PM, Igor Karymov  wrote:
>
>> Unfortunately this library is not included into "community projects" wiki
>> page list and i have no idea how-to fix this. Maybe somebody from basho
>> guys can help us.
>>
>>
>
> I just went ahead and opened a pull request to add it the docs [1]. Feel
> free to do so in the future when you've got something to add.
>
>
> > Also our library provide interface wrapper and multicluster
> configuration.
>
> Can you elaborate on how the multicluster configuration works and how you
> are using it in production?
>
> Best,
>
> Mark
>
>
> [1] https://github.com/basho/basho_docs/pull/489
>
>
>>
>> 2013/7/16 Konstantin Kalin 
>>
>>> Thank you for sharing it. I built similar application already (pool +
>>> multi-node connections) after I had got confirmation that there is no
>>> "official way" :) Lucky it wasn't hard at all and laid down into existing
>>> prototype nicely (mostly because it's in-house app).
>>>
>>> Thank you,
>>> Konstantin.
>>>
>>>
>>> On Mon, Jul 15, 2013 at 2:43 PM, Igor Karymov wrote:
>>>
 Hi. Maybe this will be usefull for you:
 https://github.com/unisontech/uriak_pool

 We was have to write our own solution, because poolboy was not be able
 handle riak nodes ups and downs gracefully.
 Also our library provide interface wrapper and multicluster
 configuration.


 2013/7/11 Sean Cribbs 

> We intend to overhaul the Erlang client soon. In the meantime, Riak CS
> has done exactly what Jeremy has suggested. Look here:
> https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl
>
>
> On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin <
> konstantin.ka...@gmail.com> wrote:
>
>> Oops… was looking at wrong column. Sorry and thanks for the advice.
>>
>> Thank you,
>> Konstantin.
>>
>> On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:
>>
>> The X there indicates that it does not support connection pooling out
>> of the box in contrast to the check. I'd look at poolboy (to use in
>> conjunction with riakc_pb_socket) and riakpool (which pulls in
>> riak-erlang-client as a dependency).
>>
>>
>> On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin <
>> konstantin.ka...@gmail.com> wrote:
>>
>>> Looking at
>>> http://docs.basho.com/riak/latest/references/Client-Libraries/ I
>>> see that Erlang riak client should support Cluster connections/pools. 
>>> But
>>> looking at Erlang riak client source code I would say that it doesn't
>>> support Cluster connections/pools out of box. And I have to develop my 
>>> own
>>> code to have load balancing/connection pool. Is it right and the
>>> documentation is incorrect and needs to be updated?
>>>
>>> Thank you,
>>> Konstantin.
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> Sean Cribbs 
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


>>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http

Re: Erlang PB client

2013-07-15 Thread Dave Parfitt 
Hello Igor -

  You can submit pull requests against the docs repo:
https://github.com/basho/basho_docs 

Cheers,
Dave

On Jul 15, 2013, at 6:00 PM, Igor Karymov  wrote:

> Unfortunately this library is not included into "community projects" wiki 
> page list and i have no idea how-to fix this. Maybe somebody from basho guys 
> can help us.
> 
> 
> 2013/7/16 Konstantin Kalin 
>> Thank you for sharing it. I built similar application already (pool + 
>> multi-node connections) after I had got confirmation that there is no 
>> "official way" :) Lucky it wasn't hard at all and laid down into existing 
>> prototype nicely (mostly because it's in-house app). 
>> 
>> Thank you,
>> Konstantin.  
>> 
>> 
>> On Mon, Jul 15, 2013 at 2:43 PM, Igor Karymov  wrote:
>>> Hi. Maybe this will be usefull for you:
>>> https://github.com/unisontech/uriak_pool
>>> 
>>> We was have to write our own solution, because poolboy was not be able 
>>> handle riak nodes ups and downs gracefully.
>>> Also our library provide interface wrapper and multicluster configuration.
>>> 
>>> 
>>> 2013/7/11 Sean Cribbs 
 We intend to overhaul the Erlang client soon. In the meantime, Riak CS has 
 done exactly what Jeremy has suggested. Look here: 
 https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl
 
 
 On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin 
  wrote:
> Oops… was looking at wrong column. Sorry and thanks for the advice. 
> 
> Thank you,
> Konstantin. 
> 
> On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:
> 
>> The X there indicates that it does not support connection pooling out of 
>> the box in contrast to the check. I'd look at poolboy (to use in 
>> conjunction with riakc_pb_socket) and riakpool (which pulls in 
>> riak-erlang-client as a dependency).
>> 
>> 
>> On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin 
>>  wrote:
>>> Looking at 
>>> http://docs.basho.com/riak/latest/references/Client-Libraries/ I see 
>>> that Erlang riak client should support Cluster connections/pools. But 
>>> looking at Erlang riak client source code I would say that it doesn't 
>>> support Cluster connections/pools out of box. And I have to develop my 
>>> own code to have load balancing/connection pool. Is it right and the 
>>> documentation is incorrect and needs to be updated? 
>>> 
>>> Thank you,
>>> Konstantin. 
>>> 
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 
 
 
 -- 
 Sean Cribbs 
 Software Engineer
 Basho Technologies, Inc.
 http://basho.com/
 
 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> 
>>> 
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Erlang PB client

2013-07-15 Thread Mark Phillips 
Hi Igor,


On Mon, Jul 15, 2013 at 3:00 PM, Igor Karymov  wrote:

> Unfortunately this library is not included into "community projects" wiki
> page list and i have no idea how-to fix this. Maybe somebody from basho
> guys can help us.
>
>

I just went ahead and opened a pull request to add it the docs [1]. Feel
free to do so in the future when you've got something to add.


> Also our library provide interface wrapper and multicluster configuration.

Can you elaborate on how the multicluster configuration works and how you
are using it in production?

Best,

Mark


[1] https://github.com/basho/basho_docs/pull/489


>
> 2013/7/16 Konstantin Kalin 
>
>> Thank you for sharing it. I built similar application already (pool +
>> multi-node connections) after I had got confirmation that there is no
>> "official way" :) Lucky it wasn't hard at all and laid down into existing
>> prototype nicely (mostly because it's in-house app).
>>
>> Thank you,
>> Konstantin.
>>
>>
>> On Mon, Jul 15, 2013 at 2:43 PM, Igor Karymov  wrote:
>>
>>> Hi. Maybe this will be usefull for you:
>>> https://github.com/unisontech/uriak_pool
>>>
>>> We was have to write our own solution, because poolboy was not be able
>>> handle riak nodes ups and downs gracefully.
>>> Also our library provide interface wrapper and multicluster
>>> configuration.
>>>
>>>
>>> 2013/7/11 Sean Cribbs 
>>>
 We intend to overhaul the Erlang client soon. In the meantime, Riak CS
 has done exactly what Jeremy has suggested. Look here:
 https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl


 On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin <
 konstantin.ka...@gmail.com> wrote:

> Oops… was looking at wrong column. Sorry and thanks for the advice.
>
> Thank you,
> Konstantin.
>
> On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:
>
> The X there indicates that it does not support connection pooling out
> of the box in contrast to the check. I'd look at poolboy (to use in
> conjunction with riakc_pb_socket) and riakpool (which pulls in
> riak-erlang-client as a dependency).
>
>
> On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin <
> konstantin.ka...@gmail.com> wrote:
>
>> Looking at
>> http://docs.basho.com/riak/latest/references/Client-Libraries/ I see
>> that Erlang riak client should support Cluster connections/pools. But
>> looking at Erlang riak client source code I would say that it doesn't
>> support Cluster connections/pools out of box. And I have to develop my 
>> own
>> code to have load balancing/connection pool. Is it right and the
>> documentation is incorrect and needs to be updated?
>>
>> Thank you,
>> Konstantin.
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>


 --
 Sean Cribbs 
 Software Engineer
 Basho Technologies, Inc.
 http://basho.com/

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Jeremiah Peschka 
This is what we do for CorrugatedIron integration testing. Test buckets
typically have a test name + UUID. Which makes it interesting when I try to
verify data via curl while I'm debugging. But it also keeps me from
polluting my testing buckets with the output of other failed tests.

---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Mon, Jul 15, 2013 at 2:16 PM, Gabriel Littman  wrote:

> Hi,
>
> Another posibility is to create a new bucket for each test run.
>
> Gabe
>
> On Mon, Jul 15, 2013 at 12:06 PM, Matthew Dawson wrote:
>
>> Hi Seth,
>> On July 14, 2013 02:25:02 PM Seth Bunce wrote:
>> > I had a similar problem. I mitigated the problem by appending a
>> > current timestamp (in nanoseconds) to the keys I'm using in my tests.
>> > This way I don't have to worry about waiting for Riak to reap
>> > tombstones after 3 seconds and I don't have to use
>> > delete_mode=immediate. I still do a list keys + delete before each
>> > test.
>> I think this is what I'm going to have to do.  I tried all the different
>> delete_mode settings, but none of them seemed to help.  Thanks for the
>> suggestion!
>> --
>> Matthew
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Erlang PB client

2013-07-15 Thread Igor Karymov 
Unfortunately this library is not included into "community projects" wiki
page list and i have no idea how-to fix this. Maybe somebody from basho
guys can help us.


2013/7/16 Konstantin Kalin 

> Thank you for sharing it. I built similar application already (pool +
> multi-node connections) after I had got confirmation that there is no
> "official way" :) Lucky it wasn't hard at all and laid down into existing
> prototype nicely (mostly because it's in-house app).
>
> Thank you,
> Konstantin.
>
>
> On Mon, Jul 15, 2013 at 2:43 PM, Igor Karymov  wrote:
>
>> Hi. Maybe this will be usefull for you:
>> https://github.com/unisontech/uriak_pool
>>
>> We was have to write our own solution, because poolboy was not be able
>> handle riak nodes ups and downs gracefully.
>> Also our library provide interface wrapper and multicluster configuration.
>>
>>
>> 2013/7/11 Sean Cribbs 
>>
>>> We intend to overhaul the Erlang client soon. In the meantime, Riak CS
>>> has done exactly what Jeremy has suggested. Look here:
>>> https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl
>>>
>>>
>>> On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin <
>>> konstantin.ka...@gmail.com> wrote:
>>>
 Oops… was looking at wrong column. Sorry and thanks for the advice.

 Thank you,
 Konstantin.

 On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:

 The X there indicates that it does not support connection pooling out
 of the box in contrast to the check. I'd look at poolboy (to use in
 conjunction with riakc_pb_socket) and riakpool (which pulls in
 riak-erlang-client as a dependency).


 On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin <
 konstantin.ka...@gmail.com> wrote:

> Looking at
> http://docs.basho.com/riak/latest/references/Client-Libraries/ I see
> that Erlang riak client should support Cluster connections/pools. But
> looking at Erlang riak client source code I would say that it doesn't
> support Cluster connections/pools out of box. And I have to develop my own
> code to have load balancing/connection pool. Is it right and the
> documentation is incorrect and needs to be updated?
>
> Thank you,
> Konstantin.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


>>>
>>>
>>> --
>>> Sean Cribbs 
>>> Software Engineer
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Erlang PB client

2013-07-15 Thread Konstantin Kalin 
Thank you for sharing it. I built similar application already (pool +
multi-node connections) after I had got confirmation that there is no
"official way" :) Lucky it wasn't hard at all and laid down into existing
prototype nicely (mostly because it's in-house app).

Thank you,
Konstantin.


On Mon, Jul 15, 2013 at 2:43 PM, Igor Karymov  wrote:

> Hi. Maybe this will be usefull for you:
> https://github.com/unisontech/uriak_pool
>
> We was have to write our own solution, because poolboy was not be able
> handle riak nodes ups and downs gracefully.
> Also our library provide interface wrapper and multicluster configuration.
>
>
> 2013/7/11 Sean Cribbs 
>
>> We intend to overhaul the Erlang client soon. In the meantime, Riak CS
>> has done exactly what Jeremy has suggested. Look here:
>> https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl
>>
>>
>> On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin <
>> konstantin.ka...@gmail.com> wrote:
>>
>>> Oops… was looking at wrong column. Sorry and thanks for the advice.
>>>
>>> Thank you,
>>> Konstantin.
>>>
>>> On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:
>>>
>>> The X there indicates that it does not support connection pooling out of
>>> the box in contrast to the check. I'd look at poolboy (to use in
>>> conjunction with riakc_pb_socket) and riakpool (which pulls in
>>> riak-erlang-client as a dependency).
>>>
>>>
>>> On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin <
>>> konstantin.ka...@gmail.com> wrote:
>>>
 Looking at
 http://docs.basho.com/riak/latest/references/Client-Libraries/ I see
 that Erlang riak client should support Cluster connections/pools. But
 looking at Erlang riak client source code I would say that it doesn't
 support Cluster connections/pools out of box. And I have to develop my own
 code to have load balancing/connection pool. Is it right and the
 documentation is incorrect and needs to be updated?

 Thank you,
 Konstantin.

 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


>>>
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>> --
>> Sean Cribbs 
>> Software Engineer
>> Basho Technologies, Inc.
>> http://basho.com/
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Erlang PB client

2013-07-15 Thread Igor Karymov 
Hi. Maybe this will be usefull for you:
https://github.com/unisontech/uriak_pool

We was have to write our own solution, because poolboy was not be able
handle riak nodes ups and downs gracefully.
Also our library provide interface wrapper and multicluster configuration.


2013/7/11 Sean Cribbs 

> We intend to overhaul the Erlang client soon. In the meantime, Riak CS has
> done exactly what Jeremy has suggested. Look here:
> https://github.com/basho/riak_cs/blob/develop/src/riak_cs_riakc_pool_worker.erl
>
>
> On Wed, Jul 10, 2013 at 5:46 PM, Konstantin Kalin <
> konstantin.ka...@gmail.com> wrote:
>
>> Oops… was looking at wrong column. Sorry and thanks for the advice.
>>
>> Thank you,
>> Konstantin.
>>
>> On Jul 10, 2013, at 3:41 PM, Jeremy Ong wrote:
>>
>> The X there indicates that it does not support connection pooling out of
>> the box in contrast to the check. I'd look at poolboy (to use in
>> conjunction with riakc_pb_socket) and riakpool (which pulls in
>> riak-erlang-client as a dependency).
>>
>>
>> On Wed, Jul 10, 2013 at 3:33 PM, Konstantin Kalin <
>> konstantin.ka...@gmail.com> wrote:
>>
>>> Looking at
>>> http://docs.basho.com/riak/latest/references/Client-Libraries/ I see
>>> that Erlang riak client should support Cluster connections/pools. But
>>> looking at Erlang riak client source code I would say that it doesn't
>>> support Cluster connections/pools out of box. And I have to develop my own
>>> code to have load balancing/connection pool. Is it right and the
>>> documentation is incorrect and needs to be updated?
>>>
>>> Thank you,
>>> Konstantin.
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> Sean Cribbs 
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Gabriel Littman 
Hi,

Another posibility is to create a new bucket for each test run.

Gabe

On Mon, Jul 15, 2013 at 12:06 PM, Matthew Dawson wrote:

> Hi Seth,
> On July 14, 2013 02:25:02 PM Seth Bunce wrote:
> > I had a similar problem. I mitigated the problem by appending a
> > current timestamp (in nanoseconds) to the keys I'm using in my tests.
> > This way I don't have to worry about waiting for Riak to reap
> > tombstones after 3 seconds and I don't have to use
> > delete_mode=immediate. I still do a list keys + delete before each
> > test.
> I think this is what I'm going to have to do.  I tried all the different
> delete_mode settings, but none of them seemed to help.  Thanks for the
> suggestion!
> --
> Matthew
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Yokozuna kv write timeouts on 1.4 (yz-merge-1.4.0)

2013-07-15 Thread Jeremiah Peschka 
I've tested this using the PBC interface and a build from the source branch
as well.

The timeout occurs waiting for any response from protobufs waiting for
message size.

---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Mon, Jul 15, 2013 at 1:00 PM, Dave Martorana  wrote:

> Hi everyone. First post, if I leave anything out just let me know.
>
> I have been using vagrant in testing Yokozuna with 1.3.0 (the official
> 0.7.0 “release") and it runs swimmingly. When 1.4 was released and someone
> pointed me to the YZ integration branch, I decided to give it a go.
>
> I realize that YZ probably doesn’t support 1.4 yet, but here are my
> experiences.
>
> - Installs fine
> - Using default stagedevrel with 5 node setup
> - Without yz enabled in app.config, kv accepts writes and reads
> - With yz enabled on dev1 and nowhere else, kv accepts writes and reads,
> creates yz index, associates index with bucket, does not index content
> - With yz enabled on 4/5 nodes, kv stops accepting writes (timeout)
>
> Ex:
>
> (env)➜  curl -v -H 'content-type: text/plain' -XPUT '
> http://localhost:10018/buckets/players/keys/name' -d "Ryan Zezeski"
> * Adding handle: conn: 0x7f995a804000
> * Adding handle: send: 0
> * Adding handle: recv: 0
> * Curl_addHandleToPipeline: length: 1
> * - Conn 0 (0x7f995a804000) send_pipe: 1, recv_pipe: 0
> * About to connect() to localhost port 10018 (#0)
> *   Trying 127.0.0.1...
> * Connected to localhost (127.0.0.1) port 10018 (#0)
> > PUT /buckets/players/keys/name HTTP/1.1
> > User-Agent: curl/7.30.0
> > Host: localhost:10018
> > Accept: */*
> > content-type: text/plain
> > Content-Length: 12
> >
> * upload completely sent off: 12 out of 12 bytes
> < HTTP/1.1 503 Service Unavailable
> < Vary: Accept-Encoding
> * Server MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue) is
> not blacklisted
> < Server: MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue)
> < Date: Mon, 15 Jul 2013 19:54:50 GMT
> < Content-Type: text/plain
> < Content-Length: 18
> <
> request timed out
> * Connection #0 to host localhost left intact
>
> Here are my Vagrant file:
>
> https://gist.github.com/themartorana/460a52bb3f840010ecde
>
> and build script for the server:
>
> https://gist.github.com/themartorana/e2e0126c01b8ef01cc53
>
> Hope this helps.
>
>  Dave
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Yokozuna kv write timeouts on 1.4 (yz-merge-1.4.0)

2013-07-15 Thread Dave Martorana 
Hi everyone. First post, if I leave anything out just let me know.

I have been using vagrant in testing Yokozuna with 1.3.0 (the official
0.7.0 “release") and it runs swimmingly. When 1.4 was released and someone
pointed me to the YZ integration branch, I decided to give it a go.

I realize that YZ probably doesn’t support 1.4 yet, but here are my
experiences.

- Installs fine
- Using default stagedevrel with 5 node setup
- Without yz enabled in app.config, kv accepts writes and reads
- With yz enabled on dev1 and nowhere else, kv accepts writes and reads,
creates yz index, associates index with bucket, does not index content
- With yz enabled on 4/5 nodes, kv stops accepting writes (timeout)

Ex:

(env)➜  curl -v -H 'content-type: text/plain' -XPUT '
http://localhost:10018/buckets/players/keys/name' -d "Ryan Zezeski"
* Adding handle: conn: 0x7f995a804000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f995a804000) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 10018 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 10018 (#0)
> PUT /buckets/players/keys/name HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:10018
> Accept: */*
> content-type: text/plain
> Content-Length: 12
>
* upload completely sent off: 12 out of 12 bytes
< HTTP/1.1 503 Service Unavailable
< Vary: Accept-Encoding
* Server MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue) is not
blacklisted
< Server: MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue)
< Date: Mon, 15 Jul 2013 19:54:50 GMT
< Content-Type: text/plain
< Content-Length: 18
<
request timed out
* Connection #0 to host localhost left intact

Here are my Vagrant file:

https://gist.github.com/themartorana/460a52bb3f840010ecde

and build script for the server:

https://gist.github.com/themartorana/e2e0126c01b8ef01cc53

Hope this helps.

Dave
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

[ANN] Riak Community Survey

2013-07-15 Thread Hector Castro 
Hi everyone,

We'd sincerely appreciate it if you could take some time to complete
our second quarterly Riak community survey:

http://riak.polldaddy.com/s/community-survey

Same as last time, all contributors to the survey will be provided
Basho swag, as well as discounted RICON tickets. One lucky contributor
will be selected to receive a free ticket to RICON West!

If you have any questions or concerns, please don't hesitate to hit up
.

--
Hector

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Matthew Dawson 
Hi Seth,
On July 14, 2013 02:25:02 PM Seth Bunce wrote:
> I had a similar problem. I mitigated the problem by appending a
> current timestamp (in nanoseconds) to the keys I'm using in my tests.
> This way I don't have to worry about waiting for Riak to reap
> tombstones after 3 seconds and I don't have to use
> delete_mode=immediate. I still do a list keys + delete before each
> test.
I think this is what I'm going to have to do.  I tried all the different 
delete_mode settings, but none of them seemed to help.  Thanks for the 
suggestion!
-- 
Matthew

signature.asc
Description: This is a digitally signed message part.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Matthew Dawson 
On July 15, 2013 09:55:05 AM Andrew Thompson wrote:
> See this old post:
> 
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004601
> .html
> 
> Effectively, you should be doing a GET before ANY PUT, and using the
> deletedvclock option if you're using PB or checking for the
> X-Riak-VClock header on the 404 response. If you get back a tombstone,
> you should be doing the new PUT with that tombstone so you don't
> accidentally create a sibling with that tombstone.
> 
> Alternatively, turn off allow_mult=false, so you can resolve the
> siblings yourself.
> 
> Andrew
Hi Andrew,

Thanks for the response!  I read through that mailing list thread, and tried 
doing as you said by fetching the object before storing the newly created 
value.  However, I'm still seeing objects that are disappearing.

I'm running with all my buckets having allow_mult set to true, so I'm not sure 
if that would have fixed the issue.  I already resolve conflicts on all 
fetches, so I'm pretty sure I'm not just seeing a single tombstone and 
assuming the data is dead based upon that.
-- 
Matthew

signature.asc
Description: This is a digitally signed message part.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Connection timeout issues

2013-07-15 Thread Y N 


It's unlikely to be that problem since I don't have any test data in my system 
yet. My "batch functions" are just running in the background doing what they 
normally do (i.e. running a mapreduce as well as querying all keys from a 
specific bucket). However, since there is no actual data yet, I'm surprised I'm 
running into a timeout issue.



 From: Jeremiah Peschka 
To: Y N  
Cc: "riak-users@lists.basho.com"  
Sent: Monday, July 15, 2013 6:33 AM
Subject: Re: Connection timeout issues
 


Riak 1.4 introduced a time out setting that defaults to 60,000 seconds. If you 
need to read for longer periods of time, you'll need to increase the timeout. 
You can set it on the messages that are being sent in to Riak. 

Is this, perhaps, what you're running into?


---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Sun, Jul 14, 2013 at 10:16 PM, Y N  wrote:


>
>I don't know if this is a client (Java 1.1.1 client) or server issue
>
>
>I recently upgraded to 1.4 and I am now seeing timeout issues when my app is 
>trying to read data either via mapreduce or getting all keys for a bucket. 
>Currently my server has no data (this is my test server and I wiped everything 
>for the upgrade).
>
>
>The client is using the PB interface.
>
>
>Is this a known issue?
>
>
>I did not have this problem on 1.3 and when running the 1.1.0 client.
>
>
>Thanks.
>___
>riak-users mailing list
>riak-users@lists.basho.com
>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: +zdbbl recommended value

2013-07-15 Thread Guido Medina 

Thanks for the quick answers Jared & Kelly,

+zdbbl 16384 is working ok for our production cluster (5 nodes), but for 
development for some large 2i (a list of keys of 4m+) I get some 
time-outs so I asked the question, cause there are memory implications, 
I will try 32768 at our dev cluster (4 nodes) and leave production alone 
until needed.


Regards,

Guido.

On 15/07/13 15:58, Jared Morrow wrote:

Guido,

So higher +zdbbl numbers will allow for larger message buffers, but at 
the expense of memory usage.  We've actually added better wording in 
the default vm.args file for 1.4+ 
https://github.com/basho/riak/blob/master/rel/files/vm.args#L40


So I guess the best answer is trial & error.  We added 32MB commented 
out in the vm.args, but it can go much higher than that (around 2gb is 
the true max).  I think most people have found setting it to 32768 
helps without too much memory usage penalty.


-Jared




On Mon, Jul 15, 2013 at 8:38 AM, Guido Medina 
mailto:guido.med...@temetra.com>> wrote:


Hi,

We had an issue in 1.4 where 2i operations were timing out, after
going through support we were suggested to use "+zdbbl 16384", on
the "Configuring Riak Docs" it is strongly suggested (unless doc
need to be re-phrased) it should be higher:

*Source:*

http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/

At the end of such doc there is the following fragment and we
would like to know if there is a kind of all situations balanced
value and the implications:


  Performance & Capacity settings

It is strongly recommended that the following values be set in the
Riak |vm.args| configuration file, which is located in the
|/etc/riak| or |/opt/riak/etc| folder.

## This setting is not present in default Riak installations, so
## it should be added.  In some cases, a value of 128000 may be
## appropriate.
+zdbbl96000

## This setting is present in default Riak installations, so
## its value should be edited.
-env ERL_MAX_PORTS16384



Kind regards,

Guido.

___
riak-users mailing list
riak-users@lists.basho.com 
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: +zdbbl recommended value

2013-07-15 Thread Jared Morrow 
Guido,

So higher +zdbbl numbers will allow for larger message buffers, but at the
expense of memory usage.  We've actually added better wording in the
default vm.args file for 1.4+
https://github.com/basho/riak/blob/master/rel/files/vm.args#L40

So I guess the best answer is trial & error.  We added 32MB commented out
in the vm.args, but it can go much higher than that (around 2gb is the true
max).  I think most people have found setting it to 32768 helps without too
much memory usage penalty.

-Jared




On Mon, Jul 15, 2013 at 8:38 AM, Guido Medina wrote:

>  Hi,
>
> We had an issue in 1.4 where 2i operations were timing out, after going
> through support we were suggested to use "+zdbbl 16384", on the
> "Configuring Riak Docs" it is strongly suggested (unless doc need to be
> re-phrased) it should be higher:
>
> *Source:*
> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/
>
> At the end of such doc there is the following fragment and we would like
> to know if there is a kind of all situations balanced value and the
> implications:
> Performance & Capacity settings
>
> It is strongly recommended that the following values be set in the Riak
> vm.args configuration file, which is located in the /etc/riak or
> /opt/riak/etc folder.
>
> ## This setting is not present in default Riak installations, so## it should 
> be added.  In some cases, a value of 128000 may be## appropriate.
> +zdbbl 96000
> ## This setting is present in default Riak installations, so## its value 
> should be edited.
> -env ERL_MAX_PORTS 16384
>
>
>
> Kind regards,
>
> Guido.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: +zdbbl recommended value

2013-07-15 Thread Kelly McLaughlin 
Hi Guido. The docs section you referenced is for RiakCS. We do recommend a
substantially higher value for zdbbl when using RiakCS because the object
data being stored is divided into 1MB chunks which is still a relatively
large object size for Riak to handle. Riak used by itself may not need the
value to be that large, but it may benefit depending on the size of objects
you are storing. If you do decide to increase the value beyond 16384, just
proceed with caution and make sure you have enough memory to support the
increased setting.

Kelly


On Mon, Jul 15, 2013 at 8:38 AM, Guido Medina wrote:

>  Hi,
>
> We had an issue in 1.4 where 2i operations were timing out, after going
> through support we were suggested to use "+zdbbl 16384", on the
> "Configuring Riak Docs" it is strongly suggested (unless doc need to be
> re-phrased) it should be higher:
>
> *Source:*
> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/
>
> At the end of such doc there is the following fragment and we would like
> to know if there is a kind of all situations balanced value and the
> implications:
> Performance & Capacity settings
>
> It is strongly recommended that the following values be set in the Riak
> vm.args configuration file, which is located in the /etc/riak or
> /opt/riak/etc folder.
>
> ## This setting is not present in default Riak installations, so## it should 
> be added.  In some cases, a value of 128000 may be## appropriate.
> +zdbbl 96000
> ## This setting is present in default Riak installations, so## its value 
> should be edited.
> -env ERL_MAX_PORTS 16384
>
>
>
> Kind regards,
>
> Guido.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

+zdbbl recommended value

2013-07-15 Thread Guido Medina 

Hi,

We had an issue in 1.4 where 2i operations were timing out, after going 
through support we were suggested to use "+zdbbl 16384", on the 
"Configuring Riak Docs" it is strongly suggested (unless doc need to be 
re-phrased) it should be higher:


*Source:* 
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/


At the end of such doc there is the following fragment and we would like 
to know if there is a kind of all situations balanced value and the 
implications:



 Performance & Capacity settings

It is strongly recommended that the following values be set in the Riak 
|vm.args| configuration file, which is located in the |/etc/riak| or 
|/opt/riak/etc| folder.


## This setting is not present in default Riak installations, so
## it should be added.  In some cases, a value of 128000 may be
## appropriate.
+zdbbl96000

## This setting is present in default Riak installations, so
## its value should be edited.
-env ERL_MAX_PORTS16384



Kind regards,

Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riak-admin diag output

2013-07-15 Thread Daniel Iwan 
Thanks Jared

I'm aware of limitations of 3-node cluster. If I understand it correctly
there are some corner cases where certain copies for some vnodes can land
on the same physical node. But I would
assume there is no case where all 3 copies (for N=3) should land on the
same physical node. Hence I'm thinking 1-node failure means we are still
safe.

riak-admin diag output seems bit strange to me showing particular vnode
twice. Any reason for that?
Also, where can I find description of algorithm which vnodes are assigned
to which physical node? I think there was blog post about it...

Daniel
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Quickly deleting + recreating item in Riak deletes new item

2013-07-15 Thread Andrew Thompson 
See this old post:

http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004601.html

Effectively, you should be doing a GET before ANY PUT, and using the
deletedvclock option if you're using PB or checking for the
X-Riak-VClock header on the 404 response. If you get back a tombstone,
you should be doing the new PUT with that tombstone so you don't
accidentally create a sibling with that tombstone.

Alternatively, turn off allow_mult=false, so you can resolve the
siblings yourself.

Andrew

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: mapreduce timeout

2013-07-15 Thread Christian Dahlqvist 
Hi Deyan,

When running mapreduce jobs, reduce phases often end up being the bottleneck. 
This is especially true when all input data needs to be gathered on the 
coordinating node before it can be executed, as is the case if the 
reduce_phase_only_1 flag is enabled. Having this flag set will cause the 
mapreduce job to not scale very well.

Depending on your exact requirements, it may be worthwhile considering 
gathering the histogram data periodically, e.g. per hour and/or day. These 
aggregates can then be stored in separate buckets with a key that describes the 
content, e.g. __ . Once this has been done, you can 
efficiently retrieve a limited number of objects that cover the period you want 
to get statistics for directly through the descriptive keys, and process these 
in the application layer. Even though this periodically requires a bit more 
work, it will most likely be much more efficient at query time and scale better.

Best regards,

Christian


On 14 Jul 2013, at 12:16, Deyan Dyankov  wrote:

> Hi everyone,
> 
> first time here. Thanks in advance.
> 
> I am experiencing issues with MapReduce and it seems to timeout after a 
> certain volume data threshold is reached. The reducer is only one and here is 
> the mapreduce initiation script:
> #!/usr/bin/env ruby
> […]
> @client = Riak::Client.new(
>   :nodes => [
> {:host => 'db1', :pb_port => 8087, :http_port => 8098},
> {:host => 'db2', :pb_port => 8087, :http_port => 8098},
> {:host => 'db3', :pb_port => 8087, :http_port => 8098}
>   ],
>   :protocol => 'pbc'
> )
> 
> start_key = "#{cust}:#{setup}:#{start_time}"
> end_key = "#{cust}:#{setup}:#{end_time}"
> 
> result = Riak::MapReduce.new(@client).
>   index(bucket_name, index_name, start_key..end_key).
>   map('map95th').
>   reduce('reduce95th', :arg => { 'reduce_phase_only_1' => true }, :keep => 
> true).
>   run()
> 
> puts result
> 
> The following is the code for the map95th and reduce95th javascript functions:
> function map95th(v, keyData, arg) {
>   var key_elements = v['key'].split(':');
>   var cust = key_elements[0];
>   var setup = key_elements[1];
>   var sid = key_elements[2];
>   var ts = key_elements[3];
> 
>   var result_key = cust + ':' + setup + ':' + ts;
>   var obj = {}
>   var obj_data = Riak.mapValuesJson(v)[0];
> 
>   obj_data['bps'] = (obj_data['rx_bytes'] + obj_data['tx_bytes']) / 60;
>   return_val = obj_data['bps'];
>   return [ return_val ];
> }
> 
> // if used, this must be a single reducer! Call from Ruby like this:
> //  reduce('reduce95th', :arg => { 'reduce_phase_only_1' => true }, :keep => 
> true).
> function reduce95th(values) {
>   var sorted = values.sort(function(a,b) { return a - b; });
>   var pct = sorted.length / 100;
>   var element_95th = pct * 95;
>   element_95th = parseInt(element_95th, 10) + 1;
> 
>   return [ sorted[element_95th] ];
> }
> 
> 
> 
> Now here is the interesting part. The MR goes through one record per minute. 
> If I run it for a period of less than ~20 days, it executes. Otherwise, it 
> times out:
> [deyandyankov@azobook ~/repos/loshko/mapreduce/ruby (master)]$
> [deyandyankov@azobook ~/repos/loshko/mapreduce/ruby (master)]$ ./95h.rb 
> yellingtone default $((`date +%s` - 20 * 86400)) `date +%s`
> 125581.516
> [deyandyankov@azobook ~/repos/loshko/mapreduce/ruby (master)]$ ./95h.rb 
> yellingtone default $((`date +%s` - 30 * 86400)) `date +%s`
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client/beefcake_protobuffs_backend.rb:182:in
>  `decode_response': Expected success from Riak but received 0. 
> {"phase":1,"error":"timeout","input":null,"type":null,"stack":null} 
> (Riak::ProtobuffsFailedRequest)
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client/beefcake_protobuffs_backend.rb:116:in
>  `mapred'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client.rb:325:in
>  `block in mapred'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client.rb:435:in
>  `block in recover_from'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/innertube-1.0.2/lib/innertube.rb:127:in
>  `take'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client.rb:433:in
>  `recover_from'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client.rb:379:in
>  `protobuffs'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client.rb:133:in
>  `backend'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/client.rb:324:in
>  `mapred'
>   from 
> /Users/deyandyankov/.rvm/gems/ruby-1.9.3-p392/gems/riak-client-1.1.1/lib/riak/map_reduce.rb:217:in
>  `run'
>   from ./95h.rb:29:in `'
> [deyandyankov@azobook ~/repos/loshko/mapreduce/ruby (master)]$
> 
> The records being processed look

Re: Connection timeout issues

2013-07-15 Thread Jeremiah Peschka 
Riak 1.4 introduced a time out setting that defaults to 60,000 seconds. If
you need to read for longer periods of time, you'll need to increase the
timeout. You can set it on the messages that are being sent in to Riak.

Is this, perhaps, what you're running into?

---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop


On Sun, Jul 14, 2013 at 10:16 PM, Y N  wrote:

>
> I don't know if this is a client (Java 1.1.1 client) or server issue
>
> I recently upgraded to 1.4 and I am now seeing timeout issues when my app
> is trying to read data either via mapreduce or getting all keys for a
> bucket. Currently my server has no data (this is my test server and I wiped
> everything for the upgrade).
>
> The client is using the PB interface.
>
> Is this a known issue?
>
> I did not have this problem on 1.3 and when running the 1.1.0 client.
>
> Thanks.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com