I forgot to mention the most efficient solution:
GAE team provides a way to have composite indexes on unindexed properties.
This way the problem is solved quite elegantly:
Popularity.all(keys_only=True).order('-hourly').ancestor(db.Key.from_path('_DoesNotMatter',
'eu/Sweden')).fetch(N)
For global queries use:
Popularity.all(keys_only=True).order('-hourly').ancestor(db.Key.from_path('_DoesNotMatter',
'world')).fetch(N)
This is even more efficient than my hacky solution with the zillion models
since it would require just a single composite index per property for a
total of 5 indexes per write/update.
- alkis
2010/4/6 Ikai L (Google) <ika...@google.com>
> Well, I'm glad you've done your homework. Have you gone through our
> datastore article series yet?
>
> http://code.google.com/appengine/articles/datastore/overview.html
>
> <http://code.google.com/appengine/articles/datastore/overview.html>Let us
> know how this approach works for you. Could be a very interesting blog post.
>
> 2010/4/6 Alkis Evlogimenos ('Αλκης Ευλογημένος) <evlogime...@gmail.com>
>
> I want to build a popularity metric on a tree of entities. Say my
>> popularity model looks like this:
>>
>> class Popularity(db.Model):
>> hourly = db.FloatProperty()
>> daily = db.FloatProperty()
>> weekly = db.FloatProperty()
>> monthly = db.FloatProperty()
>> yearly = db.FloatProperty()
>>
>> Now say I want to aggregate the downloads of a set of files. If I want to
>> do that globally that's easy. Keyname is the filename (or some unique id for
>> each file if names are not unique) and update the counter on each download.
>> When I want the most popular download in the last hour I can query for it
>> easily:
>>
>> Popularity.all(keys_only=True).order('-hourly').fetch(N) for the first N
>> filenames.
>>
>> So far so good, each property has 2 indices (which is overkill I only want
>> only 1).
>>
>> Now lets make the problem a bit more interesting. I want to aggregate
>> globally, but also on each region, country and city. So I need to setup a
>> "tree" of popularity metrics and update them individually. The keys are set
>> as a filenames so I cannot have the same key for all popularities. But I can
>> have an ancestor with keyname being the path down the tree (the entity of
>> the ancestor need not be realized in this case since it is irrelevant
>> anyway). Key for the 'eu' counter will be db.Key.from_path('_DoesNotMatter',
>> 'eu', 'Popularity', <filename>). For eu-Sweden it will
>> be db.Key.from_path('_DoesNotMatter', 'eu/Sweden', 'Popularity', <filename>)
>> and so on. Now I can query for the most popular downloads in Sweden like so:
>>
>> Popularity.all(keys_only=True).order('-hourly').ancestor(db.Key.from_path('_DoesNotMatter',
>> 'eu/Sweden')).fetch(N)
>>
>> But this will require a composite index for (ancestor, hourly, desc). For
>> all hourly, daily, ... queries I will need 5 composite indexes on top of the
>> 10 existing ones for a total of 15 indexes per write/update.
>>
>> The question is how to reduce the number of indexes required to do the
>> above. Since the model name of the popularity class does not matter at all,
>> I came up with the following:
>>
>> eu popularity: db.Key.from_path('Popularity_eu', <filename>)
>> Sweden popularity: db.Key.from_path('Popularity_eu_Sweden', <filename>)
>> global popularity: db.Key.from_path('Popularity', <filename>)
>>
>> This way I can use the following query for top N downloads in Sweden:
>>
>> db.Query('_'.join(['Popularity', 'eu', 'Sweden'],
>> keys_only=True).order('-hourly').fetch(N)
>>
>> This will still require 10 indexes (5 useless ones but still better than
>> 15) to implement but the number of models will be quite large (order of
>> cities in the world).
>>
>> - alkis
>>
>>
>> 2010/4/6 Nick Johnson (Google) <nick.john...@google.com>
>>
>>> Hi Alkis,
>>>
>>>
>>> 2010/4/5 Alkis Evlogimenos ('Αλκης Ευλογημένος) <evlogime...@gmail.com>
>>>
>>> Is there a limit in the number of model kinds per application? I am
>>>> considering a design where the number of models is going to be in the order
>>>> of 500k. I understand the model viewer in the admin console will be
>>>> completely unusable but other than that is there going to be a problem in
>>>> general?
>>>
>>>
>>> No. Other than the admin console datastore viewer likely timing out,
>>> there's no problem with doing this - though I can't think of a reason why
>>> you would actually want to do this, though.
>>>
>>> -Nick
>>>
>>>
>>>>
>>>> If that won't work, is there a plan to make unindexed properties able to
>>>> participate in composite indexes? There are cases where you only want an
>>>> (ancestor, property, descending) index and you do not care about the base
>>>> (property, ascending) and (property, descending) indexes but the current
>>>> design/api forces you to have at least 3 indexes where only 1 would just
>>>> work.
>>>>
>>>> - alkis
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To post to this group, send email to google-appeng...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>
>>>
>>>
>>>
>>> --
>>> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd.
>>> :: Registered in Dublin, Ireland, Registration Number: 368047
>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
>>> Number: 368047
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appeng...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appeng...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> http://googleappengine.blogspot.com | http://twitter.com/app_engine
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
