date:20130822

Re: 2i, search or something else for most efficient "range" lookup

2013-08-22 Thread Y N



Hi Kresten,

Thanks for the response... I was thinking about this for a bit... and unless 
I'm missing something (or don't quite understand how 2i or search works), I 
don't think 2i or search will solve this problem. I may need to fall back to 
basic M/R for this.

Correct me if I'm wrong, but 2i won't work since range queries essentially 
allow you to find objects / documents where a single indexed field falls within 
a specified range. I don't believe there is a way to find objects where a 
specified value falls within a single instance of an objects 2 indexed fields 
(at least not in the way that standard 2i works). I am not sure if having the 2 
fields (startVal and endVal) be indexed is somehow more optimal for M/R than if 
the field were not indexed (could someone from Basho comment on this)?

As for search, from what I can see, it seems to be purely for text based 
searching and won't quite suite this particular use case which involves a 
somewhat range type of binary query.





 From: Kresten Krab Thorup 
To: Y N  
Cc: Jon Meredith ; "riak-users@lists.basho.com" 
 
Sent: Thursday, August 22, 2013 2:43 PM
Subject: Re: 2i, search or something else for most efficient "range" lookup
 

How about using a 2i range query with pagination using max_results=1?  Such 
results are ordered, so it's effectively an open ended query-for-first after N, 
if you choose a large end value for the range.

Kresten

Sent from my iPhone

On 22/08/2013, at 22.38, "Y N" mailto:yug...@yahoo.com>> 
wrote:


Hi Jon,

Thanks for your response. Unfortunately it is not as simple as that... in this 
particular case, I have a large dataset of predefined objects (with ranges 
already defined). I then need to perform the lookups given specific values.

Unfortunately the ranges are not uniform e.g. domain object 1 may be from 
1000-2000, and domain object 2 may be from 2001 to 2002 etc. The ranges do not 
overlap, but the fact that they aren't uniform makes this problem
slightly more difficult (i.e. I cannot clamp / round my value down since I 
don't have a fixed / uniform difference. In some cases the differences between 
startval and endval may be 0 or 1 or as high as 50 million).


From: Jon Meredith mailto:jmered...@basho.com>>
To: Y N mailto:yug...@yahoo.com>>
Cc: "riak-users@lists.basho.com" 
mailto:riak-users@lists.basho.com>>
Sent: Monday, August 19, 2013 5:34 AM
Subject: Re: 2i, search or something else for most efficient "range" lookup

How can you guarantee it's only a single object you'll find?

If you know an event happens every X minutes but don't know the exact 
start/stop times, you could truncate the startVal/endVal to the level of 
precision you need, hit that directly first and probe either side if you don't 
find it.

If I went with a design like that I'd make it so that I could store an array of 
readings under the key to tolerate timing issues, or to allow you to update 
your collection rate.

Jon




On Fri, Aug 16, 2013 at 6:00 PM, Y N 
mailto:yug...@yahoo.com>> wrote:
Hi,

I have a question regarding the most efficient way to perform a search / lookup 
for something. This isn't a typical range lookup (where you want to find all 
objects given a specific range). In this case I want to find
a specific object where a lookup value falls within that objects range.

My use case is as follows I have the following domain object:

MyObj
startVal (long / number)
endVal (long / number)
. (other data)

I need to perform many lookups where I need to find a specific domain object 
given a long value that falls within startVal & endVal (inclusive).

What would be the most efficient way to do this in Riak? I can't think of a way 
I could do this via directly lookup, so that only leaves me with 2i and search 
as possible options.
They seem a bit like overkill, since in this case I am only interested in 
getting back a single object... but there doesn't see to be any other way to do 
this.

Would 2i or search be the most efficient / least impactful way to perform this 
type of search?

Thanks.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 




--
Jon Meredith
VP, Engineering
Basho Technologies, Inc.
jmered...@basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak-CS , S3cmd and A single Node outage on a 3 node cluster

2013-08-22 Thread Kelly McLaughlin

Idan,Actually in the case you described of using a 3 node Riak cluster with n_val of 2 the behavior you see makes perfect sense. When using only three nodes Riak does not guarantee that all replicas of an object will be on distinct physical nodes. So if you have one node down you can hit a case where both of the replicas of an object stored in Riak live on the downed node. This explains the reason you see occasional failure to retrieve an object with s3cmd, but most of the time it works just fine. It is for this reason we strongly recommend people to use at least 5 nodes in their production deployments. For testing, using at least use 4 nodes in the cluster will likely reduce the chances of this situation occurring versus using only 3.Also just be very cautious about reducing the n_val to 2. You will save on storage space doing that, but there is a trade-off to be made in availability and performance in doing so. Hope that helps shed some light. Kelly On August 22, 2013 at 10:12:06 AM, Idan Shinberg (idan.shinb...@idomoo.com) wrote: Thanks for the aidThis is a testing environemt . no-one accesses it aside of mePrior to creating the objects via riak-cs , I set the cluster to n_val 2 for the default bucket props .
On that empty cluster , i started creating objects . ( around 500) .Then I stopped and killed one of the nodes . Then the issues mentioned above were seen.still doesn't make any sense...
Regards,Idan Shinberg
System Architect
Idomoo Ltd.
Mob +972.54.562.2072email idan.shinb...@idomoo.com
web www.idomoo.com

<>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: 2i, search or something else for most efficient "range" lookup

2013-08-22 Thread Kresten Krab Thorup

How about using a 2i range query with pagination using max_results=1?  Such 
results are ordered, so it's effectively an open ended query-for-first after N, 
if you choose a large end value for the range.

Kresten

Sent from my iPhone

On 22/08/2013, at 22.38, "Y N" mailto:yug...@yahoo.com>> 
wrote:


Hi Jon,

Thanks for your response. Unfortunately it is not as simple as that... in this 
particular case, I have a large dataset of predefined objects (with ranges 
already defined). I then need to perform the lookups given specific values.

Unfortunately the ranges are not uniform e.g. domain object 1 may be from 
1000-2000, and domain object 2 may be from 2001 to 2002 etc. The ranges do not 
overlap, but the fact that they aren't uniform makes this problem
slightly more difficult (i.e. I cannot clamp / round my value down since I 
don't have a fixed / uniform difference. In some cases the differences between 
startval and endval may be 0 or 1 or as high as 50 million).


From: Jon Meredith mailto:jmered...@basho.com>>
To: Y N mailto:yug...@yahoo.com>>
Cc: "riak-users@lists.basho.com" 
mailto:riak-users@lists.basho.com>>
Sent: Monday, August 19, 2013 5:34 AM
Subject: Re: 2i, search or something else for most efficient "range" lookup

How can you guarantee it's only a single object you'll find?

If you know an event happens every X minutes but don't know the exact 
start/stop times, you could truncate the startVal/endVal to the level of 
precision you need, hit that directly first and probe either side if you don't 
find it.

If I went with a design like that I'd make it so that I could store an array of 
readings under the key to tolerate timing issues, or to allow you to update 
your collection rate.

Jon




On Fri, Aug 16, 2013 at 6:00 PM, Y N 
mailto:yug...@yahoo.com>> wrote:
Hi,

I have a question regarding the most efficient way to perform a search / lookup 
for something. This isn't a typical range lookup (where you want to find all 
objects given a specific range). In this case I want to find
a specific object where a lookup value falls within that objects range.

My use case is as follows I have the following domain object:

MyObj
startVal (long / number)
endVal (long / number)
. (other data)

I need to perform many lookups where I need to find a specific domain object 
given a long value that falls within startVal & endVal (inclusive).

What would be the most efficient way to do this in Riak? I can't think of a way 
I could do this via directly lookup, so that only leaves me with 2i and search 
as possible options.
They seem a bit like overkill, since in this case I am only interested in 
getting back a single object... but there doesn't see to be any other way to do 
this.

Would 2i or search be the most efficient / least impactful way to perform this 
type of search?

Thanks.


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
Jon Meredith
VP, Engineering
Basho Technologies, Inc.
jmered...@basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: 2i, search or something else for most efficient "range" lookup

2013-08-22 Thread Y N

Hi Jon,

Thanks for your response. Unfortunately it is not as simple as that... in this 
particular case, I have a large dataset of predefined objects (with ranges 
already defined). I then need to perform the lookups given specific values.

Unfortunately the ranges are not uniform e.g. domain object 1 may be from 
1000-2000, and domain object 2 may be from 2001 to 2002 etc. The ranges do not 
overlap, but the fact that they aren't uniform makes this problem
slightly more difficult (i.e. I cannot clamp / round my value down since I 
don't have a fixed / uniform difference. In some cases the differences between 
startval and endval may be 0 or 1 or as high as 50 million).

 From: Jon Meredith 
To: Y N  
Cc: "riak-users@lists.basho.com"  
Sent: Monday, August 19, 2013 5:34 AM
Subject: Re: 2i, search or something else for most efficient "range" lookup

How can you guarantee it's only a single object you'll find? 

If you know an event happens every X minutes but don't know the exact 
start/stop times, you could truncate the startVal/endVal to the level of 
precision you need, hit that directly first and probe either side if you don't 
find it.

If I went with a design like that I'd make it so that I could store an array of 
readings under the key to tolerate timing issues, or to allow you to update 
your collection rate.

Jon

On Fri, Aug 16, 2013 at 6:00 PM, Y N  wrote:

Hi,
>
>
>I have a question regarding the most efficient way to perform a search / 
>lookup for something. This isn't a typical range lookup (where you want to 
>find all objects given a specific range). In this case I want to find
>a specific object where a lookup value falls within that objects range.
>
>
>My use case is as follows I have the following domain object:
>
>
>MyObj
>startVal (long / number)
>endVal (long / number)
>. (other data)
>
>
>I need to perform many lookups where I need to find a specific domain object 
>given a long value that falls within startVal & endVal (inclusive).
>
>
>What would be the most efficient way to do this in Riak? I can't think of a 
>way I could do this via directly lookup, so that only leaves me with 2i and 
>search as possible options. 
>They seem a bit like overkill, since in this case I am only interested in 
>getting back a single object... but there doesn't see to be any other way to 
>do this.
>
>
>Would 2i or search be the most efficient / least impactful way to perform this 
>type of search?
>
>
>
>Thanks.
>
>
>___
>riak-users mailing list
>riak-users@lists.basho.com
>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

-- 
Jon Meredith
VP, Engineering
Basho Technologies, Inc.

jmered...@basho.com___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak CS reduced redundancy

2013-08-22 Thread Andrew Stone

Hi Thomas,

Yes you can change the n_val in the default bucket properties of the riak
cluster. That's how CS determines how many replicas to store. However,
please keep in mind that reducing the replicas can reduce your availability.

-Andrew


On Wed, Aug 21, 2013 at 5:27 AM, Thomas Dunham  wrote:

> Hi all,
> Is it possible to adjust the number of replicas Riak-CS keeps of every
> object (I'd like to only keep two copies)?
>
> I've read the documentation and the default number of replicas seems to be
> three, but there doesn't seem to be any config in Riak-CS to change it. If
> I changed the n_val in the default bucket properties of the Riak cluster,
> would this have the effect of changing the number of replicas?
>
> Here's the only comment I could find in the docs about how many replicas
> are kept. The language implies it's possible to change the number, but I
> can find no reference to any config settings in the open source or
> enterprise versions:
> In Riak CS, large objects are broken into blocks and streamed to the
> underlying Riak cluster on write, where they are replicated for high
> availability (3 replicas by default). (
> http://docs.basho.com/riakcs/latest/cookbooks/MDC-Overview/).
>
>
> Thanks,
> Tom
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Including @RiakKey and @RiakIndexes in the stored JSON

2013-08-22 Thread Guido Medina


Create a pseudo getters for the 2i indexes, valid return types are:
String, Long (And Integer) and Set of any of the mentioned, the benefit 
of this is the fact that your 2i indexes are not actual properties, they 
are meant to be computation of something, example:


/public class Postcode {//

//@//RiakKey//
//String postcode//
//...//
//...//
//
//@RiakIndex("sector")//
//@JsonIgnore//
//public String getSector() {//
//return postcode.substring(0,postcode.length() - 4//);//
//}//

//}//
/
In this example we create a 2i index for the sector but don't store it 
in the Riak JSON.


HTH,

Guido.

On 22/08/13 16:32, mex wrote:

If I declare @RiakKey and 2i indexes (@RiakIndex) for some fields in my
"Item" class, then those fields will not be displayed when querying the
record over the browser (ie. http://localhost/riak/myBucket/myKey1).

I have tried adding the annotation @JSONInclude but does not seem to change
the behaviour. Is there a way around it?

Thanks



--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Including-RiakKey-and-RiakIndexes-in-the-stored-JSON-tp4028933.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Including @RiakKey and @RiakIndexes in the stored JSON

2013-08-22 Thread Brian Roach

No; they are explicitly excluded from serialization because they're metadata.

The 2i indexes are returned in the http headers, and the key you kinda
already have to know to make the query in the first place.

Thanks,
Roach

On Thu, Aug 22, 2013 at 9:32 AM, mex  wrote:
> If I declare @RiakKey and 2i indexes (@RiakIndex) for some fields in my
> "Item" class, then those fields will not be displayed when querying the
> record over the browser (ie. http://localhost/riak/myBucket/myKey1).
>
> I have tried adding the annotation @JSONInclude but does not seem to change
> the behaviour. Is there a way around it?
>
> Thanks
>
>
>
> --
> View this message in context: 
> http://riak-users.197444.n3.nabble.com/Including-RiakKey-and-RiakIndexes-in-the-stored-JSON-tp4028933.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Including @RiakKey and @RiakIndexes in the stored JSON

2013-08-22 Thread mex

If I declare @RiakKey and 2i indexes (@RiakIndex) for some fields in my
"Item" class, then those fields will not be displayed when querying the
record over the browser (ie. http://localhost/riak/myBucket/myKey1).

I have tried adding the annotation @JSONInclude but does not seem to change
the behaviour. Is there a way around it?

Thanks



--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Including-RiakKey-and-RiakIndexes-in-the-stored-JSON-tp4028933.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Repeated key exchange between 2 nodes

2013-08-22 Thread Daniil Churikov

Hello dear riak list.

Could you clarify my observation.
Recently we had an issue with corrupted data on riak. Instance was powered
off and after it became available we found some bad_crc errors in logs. At
that moment we had 1.2.1-1 riak nodes in our cluster. After some googling we
found an issue on github and decided to upgrade riak to 1.4.1 to resolve
this issue (https://github.com/basho/riak_kv/pull/385).
Everything went perfectly smooth, all nodes were upgraded.
As expected some messages regarded to active anti-entropy background work
has appeared in logs.
Since that date ~ two weeks had passed, but periodically I see in logs such
messages:

2013-08-22 01:12:17.411 [info]
<0.5234.552>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during a
ctive anti-entropy exchange of
{890602560248518965780370444936484965102833893376,3} between {8906025602485
18965780370444936484965102833893376,'riak@10.0.1.190'} and
{8963115510193428053046035888142829456483648798
72,'riak@10.0.1.191'}   
  
2013-08-22 01:33:17.576 [info]
<0.24544.552>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during 
active anti-entropy exchange of
{1210306043414653979137426502093171875652569137152,3} between {12103060434
14653979137426502093171875652569137152,'riak@10.0.1.190'} and
{1216015034185477818661659645970969856198100
123648,'riak@10.0.1.191'}   
  
2013-08-22 02:48:18.035 [info]
<0.19539.554>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during 
active anti-entropy exchange of
{890602560248518965780370444936484965102833893376,3} between {890602560248
518965780370444936484965102833893376,'riak@10.0.1.190'} and
{896311551019342805304603588814282945648364879
872,'riak@10.0.1.191'}  
  
2013-08-22 03:09:18.232 [info]
<0.32041.554>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during 
active anti-entropy exchange of
{1210306043414653979137426502093171875652569137152,3} between {12103060434
14653979137426502093171875652569137152,'riak@10.0.1.190'} and
{1216015034185477818661659645970969856198100
123648,'riak@10.0.1.191'}   
  
2013-08-22 04:24:18.588 [info]
<0.18936.556>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during 
active anti-entropy exchange of
{890602560248518965780370444936484965102833893376,3} between {890602560248
518965780370444936484965102833893376,'riak@10.0.1.190'} and
{896311551019342805304603588814282945648364879
872,'riak@10.0.1.191'}  
  
2013-08-22 04:45:18.738 [info]
<0.31665.556>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during 
active anti-entropy exchange of
{1210306043414653979137426502093171875652569137152,3} between {12103060434
14653979137426502093171875652569137152,'riak@10.0.1.190'} and
{1216015034185477818661659645970969856198100
123648,'riak@10.0.1.191'}   
  

This is strange to me, because there was no other outages or any network
problems since that date. And one more interesting thing is that this
partition numbers (I believe?) are same all the time.



--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Repeated-key-exchange-between-2-nodes-tp4028932.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: 2i, search or something else for most efficient "range" lookup

Re: Riak-CS , S3cmd and A single Node outage on a 3 node cluster

Re: 2i, search or something else for most efficient "range" lookup

Re: 2i, search or something else for most efficient "range" lookup

Re: Riak CS reduced redundancy

Re: Including @RiakKey and @RiakIndexes in the stored JSON

Re: Including @RiakKey and @RiakIndexes in the stored JSON

Including @RiakKey and @RiakIndexes in the stored JSON

Repeated key exchange between 2 nodes

9 matches

Site Navigation

Mail list logo

Footer information