[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-14 Thread Tony
If the number of entities in the kind is reasonably small, you could do: keys = Kind.all(keys_only=True).fetch(MAX_ENTITIES) Pick x keys out of that list, and then use db.get() to fetch them. Otherwise, johnP's suggestion is probably best, but I don't see why you need the extra overhead of pick

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-14 Thread Bennomatic
I was just thinking this last night when I wasn't at my computer. Darn you for publishing my good idea first! :) It doesn't bring the number of queries down to 1, but it does bring them down to 2, at least on the request side. Of course, it also means one extra transactions every time a new enti

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-12 Thread johnP
> Dont know quite how would implement a fifo buffer on AppEngine, would > almost certainly use the datastore, so is going to be expensive... A possibility is to maintain an index, which is a pickled list of entity keys, stored as a blob somewhere. You'll need to maintain the index by adding/remo

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-11 Thread Barry Hunter
On 11/07/2009, Devel63 wrote: > > Barry, I understand your objections below, but do you have a better > approach? no, I dont. The way in my first reply is the best I can think off. Not totally uniform, but dont think any 'random' solution is going to be, unless you keep the data uptodate - whi

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-11 Thread Devel63
Barry, I understand your objections below, but do you have a better approach? Assigning random numbers to entities is guaranteed to be worse. If you are worried about an entity being deleted and opening a gap in the sequence, imagine the thousand-fold gaps you will see with random ID generation

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-10 Thread Barry Hunter
On 10/07/2009, Devel63 wrote: > > The best way is to assign a one-up counter to each record as you > create it, then call random.randint(1,max_counter) to determine the > desired record. > > To retrieve multiple random entities in a query, do a filter('IN ', > [my random nums]). doent work

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-10 Thread Jason (Google)
No, IDs are not guaranteed contiguous. In my sample application, my IDs routinely jump from 1, 2, 3, 1000 for a lot of my kinds, so you can't count on continuity. I generally like the custom ID solution based on a sharded counter. This should guarantee continuity if you use a transaction... see htt

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-10 Thread Aleem Mawani
I'm not sure the ID's are guranteed continous nor do they start at some starting point (0 or 1). Assigning my own id's seems to turn into a more difficult problem of creating an IdGenerator. - Aleem On Fri, Jul 10, 2009 at 3:32 PM, Julian Namaro wrote: > > You should try to generate a list of N

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-10 Thread Julian Namaro
You should try to generate a list of N random keys for your entity because you can fetch that with only one datastore call(and no index). Aren't google-generated numeric IDs guaranted continuous? If not you can yourself assign alphanumeric keys that are continuous (or generated by a formula that y

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-10 Thread Devel63
The best way is to assign a one-up counter to each record as you create it, then call random.randint(1,max_counter) to determine the desired record. To retrieve multiple random entities in a query, do a filter('IN ', [my random nums]). Note that behind the scenes this generates multiple queries,

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-10 Thread Wooble
Highly non-optimal solution: have a cron job assign new random numbers to your entities often enough to simulate randomness. Even just re- assigning numbers to entities that have been previously selected might work. This involves a lot more CPU as you'd be doing writes, but shifts the work from

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-09 Thread Barry Hunter
Your results wont be uniform anyway http://groups.google.com/group/google-appengine/msg/a6b952a4b7a59562?hl=en On 09/07/2009, aloo wrote: > > Because it won't be uniformly random. > > If the random ordering is: A B C D E F G H I J K L M and I want to > select 5 random records, I'll always

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-09 Thread aloo
Because it won't be uniformly random. If the random ordering is: A B C D E F G H I J K L M and I want to select 5 random records, I'll always get similarly grouped records (i.e. I'll never get A and M in the same query). On Jul 9, 1:42 pm, Barry Hunter wrote: > Why cant you do "> rand ORDER BY

[google-appengine] Re: Querying for N random records on Appengine datastore

2009-07-09 Thread Barry Hunter
Why cant you do "> rand ORDER BY asc LIMIT 5" ??? Yes you will get consecutive records, but as the random number you meant to have stored is random, the records are not consecutive in reality (only in the index). You might even find it not that much more expensive (but benchmark on live!) to sa