Ok, I'm not exactly sure what will be most helpful to you, but I am
attaching an example of a Key/Value.
What you see in the file, we would have thousands of those. Probably would
get to at least a million or so KV's like that.
The bucket is search enabled, so all those JSON fields are going to be
indexed.
However, we are primarily search filtering on the systemId, indexId, &
fullText fields in the value.
We currently have a 6 node cluster with each server having approx 16 cores
& in most cases at least 10 GB (20 GB free on most servers probably) free
memory with all the process running (including Riak). Computing power
should not be an issue period!
Let me know what use that would be useful to provide.
On Wed, Nov 20, 2013 at 12:46 PM, Todd Tyree <t...@basho.com> wrote:
> Hi Roger,
>
> Sorry, meant to reply to the mailing list, but accidentally replied
> directly to you.
>
> Before I can say whether or not secondary indexes are suitable, I need to
> know more about your data, access patterns and query patterns.
>
> Can you share this information with me here? What kind of data is being
> searched and how frequently is it updated?
>
> Best,
> Todd
>
>
> On Wed, Nov 20, 2013 at 5:05 PM, Roger Diller <
> ro...@flexrentalsolutions.com> wrote:
>
>> Do you have any other suggestions on how we can find data in real time
>> from a bucket (without moving to 2.0)? What about secondary indexes?
>>
>>
>> On Wed, Nov 20, 2013 at 11:59 AM, Todd Tyree <t...@basho.com> wrote:
>>
>>> Hi Roger,
>>>
>>> You are essentially correct in that map reduce was never designed as a
>>> realtime query tool.
>>>
>>> However, we do have a solution in a technology preview release stage
>>> that may solve this problem for you: Yokozuna [0]. It is a tight
>>> integration of Riak and Solr and brings the best of both technologies
>>> together.
>>>
>>> It is currently scheduled for release as part of Riak 2.0, but you can
>>> clone the repo and build it now if you would like. Just be aware it is
>>> still undergoing development and the API may be subject to change before
>>> the final release.
>>>
>>> [0] https://github.com/basho/yokozuna
>>>
>>> Best,
>>> Todd
>>>
>>
>
> On Wed, Nov 20, 2013 at 4:45 PM, Roger Diller <
> ro...@flexrentalsolutions.com> wrote:
>
>> I could dig up all our nitty gritty Riak details but I don't think that
>> will help really.
>>
>> The point I think is this: Using search map reduce is not a viable way to
>> do real time search queries. Especially ones that may have 2000+ plus
>> results each. Couple that with search requests coming in every few seconds
>> from 300+ customer app instances and you literally bring Riak to it's
>> knees.
>>
>> Not that Riak is the problem really, it's just we are using it in a way
>> it was not designed for. In essence, we are using Riak as a search engine
>> for our application data. Correct me if I'm wrong but Riak is more for
>> storing large amounts of KV data, but not really for finding that data in a
>> search sense.
>>
>> Am I missing something here? Is there a viable way for doing real time
>> search queries on a bucket with 1 million keys?
>>
>>
>> On Mon, Nov 18, 2013 at 5:29 PM, Alexander Sicular <sicul...@gmail.com>wrote:
>>
>>> More info please...
>>>
>>> Version
>>> Current config
>>> Hardware
>>> Data size
>>> Search Schema
>>> Etc.
>>>
>>> But I would probably say that your search is returning too many keys to
>>> your mr. More inline.
>>>
>>> @siculars
>>> http://siculars.posthaven.com
>>>
>>> Sent from my iRotaryPhone
>>>
>>> On Nov 18, 2013, at 13:59, Roger Diller <ro...@flexrentalsolutions.com>
>>> wrote:
>>>
>>> Using the Riak Java client, I am executing a search map reduce like this:
>>>
>>> MapReduceResult result = riakClient.mapReduce(SEARCH_BUCKET,
>>> search).execute();
>>>
>>>
>>> ^is this part a typo. Cause otherwise it looks like you do a s>mr, set
>>> the search and then another s>mr.
>>>
>>>
>>> String search = "systemId:" + systemName + " AND indexId:" + indexId;
>>>
>>> MapReduceResult result = riakClient.mapReduce(SEARCH_BUCKET,
>>> search).execute();
>>>
>>> This worked fine when the bucket contained a few thousand keys. Now that
>>> we have far more data stored in the bucket (at least 250K keys), it's
>>> throwing this generic error:
>>>
>>> com.basho.riak.client.RiakException: java.io.IOException:
>>> {"error":"map_reduce_error"}
>>>
>>> We've also noticed that storing new key/values in the bucket has slowed
>>> WAY down.
>>>
>>> Any idea what's going on?
>>>
>>>
>>> Your data set is incorrectly sized to your production config.
>>>
>>> Are there limitations to Search Map Reduce?
>>>
>>>
>>> Certainly
>>>
>>> Are there configuration options that need changed?
>>>
>>>
>>> Possibly
>>>
>>> Any help would be greatly appreciated.
>>>
>>>
>>> --
>>> Roger Diller
>>> Flex Rental Solutions, LLC
>>> Email: ro...@flexrentalsolutions.com
>>> Skype: rogerdiller
>>> Time Zone: Eastern Time
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>> --
>> Roger Diller
>> Flex Rental Solutions, LLC
>> Email: ro...@flexrentalsolutions.com
>> Skype: rogerdiller
>> Time Zone: Eastern Time
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> *Todd Tyree*
> Client Services Engineer Basho <http://www.basho.com/>
>
> mobile: +44(0)7861 220 182
> web: www.basho.com
> github: tatyree <http://github.com/tatyree>
>
--
Roger Diller
Flex Rental Solutions, LLC
Email: ro...@flexrentalsolutions.com
Skype: rogerdiller
Time Zone: Eastern Time
Key:
qamaster:23e51460-519b-11df-81b5-1231390c71d1:04741560-0754-11e2-bed8-00e08175e43e
Value:
'{"systemId":"qamaster","indexId":"23e51460-519b-11df-81b5-1231390c71d1","documentId":"04741560-0754-11e2-bed8-00e08175e43e","deleted":"false","closed":"false","fieldMap":{"element.parentElement.name":{"fieldId":"element.parentElement.name","fieldValue":"Subrental
1","fieldType":"STRING"},"elementName":{"fieldId":"elementName","fieldValue":"Subrental
1","fieldType":"STRING"},"element.resolvedStartDate.SHORT_DATE":{"fieldId":"element.resolvedStartDate.SHORT_DATE","fieldValue":"7/16/2012","fieldType":"STRING"},"element.name":{"fieldId":"element.name","fieldValue":"Subrental
1","fieldType":"STRING"},"element.client.company":{"fieldId":"element.client.company","fieldValue":"","fieldType":"STRING"},"parentCaption":{"fieldId":"parentCaption","fieldValue":"Subrental
1
(ABWR0)","fieldType":"STRING"},"locationId":{"fieldId":"locationId","fieldValue":"9cd7a7c6-1648-102d-9c2d-00e081c0bae2","fieldType":"STRING"},"element.status.name":{"fieldId":"element.status.name","fieldValue":"Shelved","fieldType":"STRING"},"plannedStartDate":{"fieldId":"plannedStartDate","fieldValue":1342483260000,"fieldType":"DATE"},"pickupLocationId":{"fieldId":"pickupLocationId","fieldValue":"9cd7a7c6-1648-102d-9c2d-00e081c0bae2","fieldType":"STRING"},"deleted":{"fieldId":"deleted","fieldValue":"false","fieldType":"STRING"},"definitionName":{"fieldId":"definitionName","fieldValue":"Manifest","fieldType":"STRING"},"element.resolvedEndDate.SHORT_DATE":{"fieldId":"element.resolvedEndDate.SHORT_DATE","fieldValue":"7/17/2012","fieldType":"STRING"},"pickupLocation":{"fieldId":"pickupLocation","fieldValue":"Flex
-
Utah","fieldType":"STRING"},"element.client.associatedUser.name":{"fieldId":"element.client.associatedUser.name","fieldValue":"","fieldType":"STRING"},"personResponsibleId":{"fieldId":"personResponsibleId","fieldValue":"67c4e24a-c7c5-11e0-be4c-1231390c71d1","fieldType":"STRING"},"plannedEndDate":{"fieldId":"plannedEndDate","fieldValue":1342546200000,"fieldType":"DATE"},"locationString":{"fieldId":"locationString","fieldValue":"Flex
-
Utah","fieldType":"STRING"},"closed":{"fieldId":"closed","fieldValue":"false","fieldType":"STRING"},"element.documentNumber":{"fieldId":"element.documentNumber","fieldValue":"ABWR0","fieldType":"STRING"},"displayCaption":{"fieldId":"displayCaption","fieldValue":"Subrental
1
(ABWR0)","fieldType":"STRING"},"definitionId":{"fieldId":"definitionId","fieldValue":"23e51460-519b-11df-81b5-1231390c71d1","fieldType":"STRING"}},"fullText":"subrental
1 subrental 1 7/16/2012 subrental 1 subrental 1 (abwr0)
9cd7a7c6-1648-102d-9c2d-00e081c0bae2 shelved
9cd7a7c6-1648-102d-9c2d-00e081c0bae2 false manifest 7/17/2012 flex - utah
67c4e24a-c7c5-11e0-be4c-1231390c71d1 flex - utah false abwr0 subrental 1
(abwr0) 23e51460-519b-11df-81b5-1231390c71d1 "}'
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
