date:20180518

RE: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Nicholas Adams

Sorry for being late to the party but have you tried repairing all partitions? 
https://docs.basho.com/riak/kv/2.1.4/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

This forces every single piece of information to be read and thus causes read 
repair to kick in for any partitions that register less than 3 copies.

Best regards,

Nicholas

From: riak-users  On Behalf Of Guido Medina
Sent: 18 May 2018 22:52
To: riak-users 
Subject: Re: N = 3 and RW = 2 not finding some keys

Yes, but we haven't set back R = 2, we are currently with R = 3 until we 
resolve the issue.
I'm in the middle of writing something that will read every single key for the 
cluster using our relation DB IDs (each key has a counterpart on DB)

I'm guessing that 2i streaming won't help? we need to make sure we are fully 
consistent (for RW = 2) so I have to find a way to repair everything before 
turning R to 2.

After reading the object yes every request after was successful, but we even 
had a weird situation where after writing key with W = 2 we couldn't find with 
R = 2, not good for us.

Guido.
On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request?



Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?



On 18 May 2018, at 13:13, Russell Brown 
 wrote:



But why isn’t read repair “working”?



On 18 May 2018, at 11:07, Bryan Hunt 
 wrote:



Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.



On 18 May 2018, at 11:00, Bryan Hunt 
 wrote:



Hi Guido,



You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.



I.e



curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
application/json" -d '{"props":{"notfound_ok": false}}'



This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.



https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok



From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.



Best Regards,



Bryan Hunt



On 17 May 2018, at 15:42, Guido Medina 
 wrote:



Hi all,



After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2



Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.



Any help will be truly appreciated.



Kind regards,

Guido.

___

riak-users mailing list

riak-users@lists.basho.com

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





___

riak-users mailing list

riak-users@lists.basho.com

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com







___

riak-users mailing list

riak-users@lists.basho.com

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Guido Medina

Yes, but we haven't set back R = 2, we are currently with R = 3 until we 
resolve the issue.
I'm in the middle of writing something that will read every single key 
for the cluster using our relation DB IDs (each key has a counterpart on DB)


I'm guessing that 2i streaming won't help? we need to make sure we are 
fully consistent (for RW = 2) so I have to find a way to repair 
everything before turning R to 2.


After reading the object yes every request after was successful, but we 
even had a weird situation where after writing key with W = 2 we 
couldn't find with R = 2, not good for us.


Guido.

On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request?

Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?


On 18 May 2018, at 13:13, Russell Brown  wrote:

But why isn’t read repair “working”?


On 18 May 2018, at 11:07, Bryan Hunt  wrote:

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate.


On 18 May 2018, at 11:00, Bryan Hunt  wrote:

Hi Guido,

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d 
'{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

 From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt


On 17 May 2018, at 15:42, Guido Medina  wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless 
we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, 
this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Bryan Hunt

Russell, Good question. I’m guessing they are iterating, and requesting a 
different object for each request? 

Guido, given the behaviour you initially described, before applying the 
configuration I suggested - did you receive a successful response upon 
subsequent requests for the same object ?

> On 18 May 2018, at 13:13, Russell Brown  wrote:
> 
> But why isn’t read repair “working”?
> 
>> On 18 May 2018, at 11:07, Bryan Hunt  wrote:
>> 
>> Of course, AAE will eventually repair the missing object replicas but it 
>> seems like you need something more immediate. 
>> 
>>> On 18 May 2018, at 11:00, Bryan Hunt  
>>> wrote:
>>> 
>>> Hi Guido, 
>>> 
>>> You should attempt to change the bucket property ‘notfound_ok’ from the 
>>> default of ‘true' to ‘false'.
>>> 
>>> I.e 
>>> 
>>> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
>>> application/json" -d '{"props":{"notfound_ok": false}}'
>>> 
>>> This makes GET operations for non-existent keys slower as it forces an 
>>> internal GET for each of the three copies.
>>> 
>>> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>>> 
>>> From what you describe, it sounds like only a single copy (out of the 
>>> original three), somehow remain present in your cluster.
>>> 
>>> Best Regards,
>>> 
>>> Bryan Hunt
>>> 
 On 17 May 2018, at 15:42, Guido Medina  wrote:
 
 Hi all,
 
 After some big rebalance of our cluster some keys are not found anymore 
 unless we set R = 3, we had N = 3 and R = W = 2
 
 Is there any sort of repair that would correct such situation for Riak 
 2.2.3, this is really driving us nuts.
 
 Any help will be truly appreciated.
 
 Kind regards,
 Guido.
 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> 
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Guido Medina

A force replace done wrong causing some partitions to only have one copy.
And the cluster running extremely slow have forced us to disable AAE for
now.

Guido.

On 18/05/18 13:13, Russell Brown wrote:

But why isn’t read repair “working”?

On 18 May 2018, at 11:07, Bryan Hunt wrote:

Of course, AAE will eventually repair the missing object replicas but it seems
like you need something more immediate.

On 18 May 2018, at 11:00, Bryan Hunt wrote:

Hi Guido,

You should attempt to change the bucket property ‘notfound_ok’ from the default
of ‘true' to ‘false'.

I.e

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d
'{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

From what you describe, it sounds like only a single copy (out of the original
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

On 17 May 2018, at 15:42, Guido Medina wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless
we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3,
this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Russell Brown

But why isn’t read repair “working”?

> On 18 May 2018, at 11:07, Bryan Hunt  wrote:
> 
> Of course, AAE will eventually repair the missing object replicas but it 
> seems like you need something more immediate. 
> 
>> On 18 May 2018, at 11:00, Bryan Hunt  wrote:
>> 
>> Hi Guido, 
>> 
>> You should attempt to change the bucket property ‘notfound_ok’ from the 
>> default of ‘true' to ‘false'.
>> 
>> I.e 
>> 
>> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
>> application/json" -d '{"props":{"notfound_ok": false}}'
>> 
>> This makes GET operations for non-existent keys slower as it forces an 
>> internal GET for each of the three copies.
>> 
>> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>> 
>> From what you describe, it sounds like only a single copy (out of the 
>> original three), somehow remain present in your cluster.
>> 
>> Best Regards,
>> 
>> Bryan Hunt
>> 
>>> On 17 May 2018, at 15:42, Guido Medina  wrote:
>>> 
>>> Hi all,
>>> 
>>> After some big rebalance of our cluster some keys are not found anymore 
>>> unless we set R = 3, we had N = 3 and R = W = 2
>>> 
>>> Is there any sort of repair that would correct such situation for Riak 
>>> 2.2.3, this is really driving us nuts.
>>> 
>>> Any help will be truly appreciated.
>>> 
>>> Kind regards,
>>> Guido.
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Bryan Hunt

Of course, AAE will eventually repair the missing object replicas but it seems 
like you need something more immediate. 

> On 18 May 2018, at 11:00, Bryan Hunt  wrote:
> 
> Hi Guido, 
> 
> You should attempt to change the bucket property ‘notfound_ok’ from the 
> default of ‘true' to ‘false'.
> 
> I.e 
> 
> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
> application/json" -d '{"props":{"notfound_ok": false}}'
> 
> This makes GET operations for non-existent keys slower as it forces an 
> internal GET for each of the three copies.
> 
> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>  
> 
> 
> From what you describe, it sounds like only a single copy (out of the 
> original three), somehow remain present in your cluster.
> 
> Best Regards,
> 
> Bryan Hunt
> 
>> On 17 May 2018, at 15:42, Guido Medina > > wrote:
>> 
>> Hi all,
>> 
>> After some big rebalance of our cluster some keys are not found anymore 
>> unless we set R = 3, we had N = 3 and R = W = 2
>> 
>> Is there any sort of repair that would correct such situation for Riak 
>> 2.2.3, this is really driving us nuts.
>> 
>> Any help will be truly appreciated.
>> 
>> Kind regards,
>> Guido.
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com 
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: N = 3 and RW = 2 not finding some keys

2018-05-18 Thread Bryan Hunt

Hi Guido, 

You should attempt to change the bucket property ‘notfound_ok’ from the default 
of ‘true' to ‘false'.

I.e 

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: 
application/json" -d '{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal 
GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
 


From what you describe, it sounds like only a single copy (out of the original 
three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

> On 17 May 2018, at 15:42, Guido Medina  wrote:
> 
> Hi all,
> 
> After some big rebalance of our cluster some keys are not found anymore 
> unless we set R = 3, we had N = 3 and R = W = 2
> 
> Is there any sort of repair that would correct such situation for Riak 2.2.3, 
> this is really driving us nuts.
> 
> Any help will be truly appreciated.
> 
> Kind regards,
> Guido.
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: N = 3 and RW = 2 not finding some keys

Re: N = 3 and RW = 2 not finding some keys

Re: N = 3 and RW = 2 not finding some keys

Re: N = 3 and RW = 2 not finding some keys

Re: N = 3 and RW = 2 not finding some keys

Re: N = 3 and RW = 2 not finding some keys

Re: N = 3 and RW = 2 not finding some keys

7 matches

Site Navigation

Mail list logo

Footer information