[google-appengine] Re: Retrieving Large Query Sets Containing Inequality Operators
perhaps your question could be 'how can i do xyz given the 1000 result limit?'
basically can your task be done some other way? perhaps you can
calculate something at write time. eg if its a sum or count you want.
the limitations sometimes requires thinking outside the box!
On 19/04/2009, Chris wrote:
>
> I have the model:
>
> class Record(db.Model):
> date = db.DateProperty()
>
> Over 1000 Record instances may contain the same date value. The
> instances are also not created in the order of their dates.
>
> I'm trying to query all instances whose date is within a given range.
> I need to be able to query sets larger than the default max of 1000.
>
> I was thinking of doing something like:
>
> Record.all().filter('date >', startDate).filter('date <', endDate)
>
> But this would limit me at most to the first 1000 records. How would I
> get the next 1000 records, maintaining the range query? I can't use
> the last date value from the previous query as the startDate for the
> next query, since there's no guarantee that the date value has
> changed. For example, I could have 2000 instances with the date
> 2008-1-1, so the first and second query sets have identical filter
> parameters.
>
> Intuitively, I could accomplish this by simply filtering on the
> previous key value (e.g. .filter('key >', lastKeyValue)).
> Unfortunately, App Engine currently prohibits filtering with
> inequality operators on more than one property.
>
> Am I correct in understanding that there's essentially no way to query
> all instances within a specific date range, due to this limitation?
>
> This issue keeps coming up for me, and has been a huge deal-breaker in
> a lot of potential projects I've been working on for App Engine. Any
> advice on how I could work around it would be appreciated.
>
>
> Chris
> >
>
--
Barry
- www.nearby.org.uk - www.geograph.org.uk -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Retrieving Large Query Sets Containing Inequality Operators
CHeck this:
http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt?gda=7FbhWXcAAACSStSWrftt07H4FK2Rtvurv5Qi9dmYyYZMEnTZCjCsYQQgER4RQV57mxjvIzAWBZmQ3TeCdbqm30Qz_AwgYlIpRbcWRj3jGGBm-fgbnPJIYc4-hXRRmo3Xgj6KgtSLBeZ45alvcyXc30EbEX-RNDZveV4duv6pDMGhhhZdjQlNAw
On 4月19日, 下午12时32分, Chris wrote:
> I have the model:
>
> class Record(db.Model):
> date = db.DateProperty()
>
> Over 1000 Record instances may contain the same date value. The
> instances are also not created in the order of their dates.
>
> I'm trying to query all instances whose date is within a given range.
> I need to be able to query sets larger than the default max of 1000.
>
> I was thinking of doing something like:
>
> Record.all().filter('date >', startDate).filter('date <', endDate)
>
> But this would limit me at most to the first 1000 records. How would I
> get the next 1000 records, maintaining the range query? I can't use
> the last date value from the previous query as the startDate for the
> next query, since there's no guarantee that the date value has
> changed. For example, I could have 2000 instances with the date
> 2008-1-1, so the first and second query sets have identical filter
> parameters.
>
> Intuitively, I could accomplish this by simply filtering on the
> previous key value (e.g. .filter('key >', lastKeyValue)).
> Unfortunately, App Engine currently prohibits filtering with
> inequality operators on more than one property.
>
> Am I correct in understanding that there's essentially no way to query
> all instances within a specific date range, due to this limitation?
>
> This issue keeps coming up for me, and has been a huge deal-breaker in
> a lot of potential projects I've been working on for App Engine. Any
> advice on how I could work around it would be appreciated.
>
> Chris
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Retrieving Large Query Sets Containing Inequality Operators
Thanks, but I don't see how that's relevant. Like I said, the ordering
of an object's key has no correlation to an object's date.
I'm not trying to simply page. That's trivial using the method listed
in your link. What I'm trying to do is page through a set of objects
limited by a date range.
Chris
2009/4/19 秦锋 :
>
> CHeck this:
> http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt?gda=7FbhWXcAAACSStSWrftt07H4FK2Rtvurv5Qi9dmYyYZMEnTZCjCsYQQgER4RQV57mxjvIzAWBZmQ3TeCdbqm30Qz_AwgYlIpRbcWRj3jGGBm-fgbnPJIYc4-hXRRmo3Xgj6KgtSLBeZ45alvcyXc30EbEX-RNDZveV4duv6pDMGhhhZdjQlNAw
>
> On 4月19日, 下午12时32分, Chris wrote:
>> I have the model:
>>
>> class Record(db.Model):
>> date = db.DateProperty()
>>
>> Over 1000 Record instances may contain the same date value. The
>> instances are also not created in the order of their dates.
>>
>> I'm trying to query all instances whose date is within a given range.
>> I need to be able to query sets larger than the default max of 1000.
>>
>> I was thinking of doing something like:
>>
>> Record.all().filter('date >', startDate).filter('date <', endDate)
>>
>> But this would limit me at most to the first 1000 records. How would I
>> get the next 1000 records, maintaining the range query? I can't use
>> the last date value from the previous query as the startDate for the
>> next query, since there's no guarantee that the date value has
>> changed. For example, I could have 2000 instances with the date
>> 2008-1-1, so the first and second query sets have identical filter
>> parameters.
>>
>> Intuitively, I could accomplish this by simply filtering on the
>> previous key value (e.g. .filter('key >', lastKeyValue)).
>> Unfortunately, App Engine currently prohibits filtering with
>> inequality operators on more than one property.
>>
>> Am I correct in understanding that there's essentially no way to query
>> all instances within a specific date range, due to this limitation?
>>
>> This issue keeps coming up for me, and has been a huge deal-breaker in
>> a lot of potential projects I've been working on for App Engine. Any
>> advice on how I could work around it would be appreciated.
>>
>> Chris
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Retrieving Large Query Sets Containing Inequality Operators
Hi Chris,
You can do this by adding an extra attribute to the object that
contains the date and a sequence number.
The extra attribute is an IntegerProperty. It has arbitrary precision,
unlimited bit length.
If you store (long(datetime.toordinal(myRecord.date))<<128)+long(sequenceNum)
you can store a lot of entities on one date, 2**128. If you set
sequenceNum starting at 1 you can use your query that sets
startDate = (long(datetime.toordinal(someDate1))<<128)+long(0)
endDate = (long(datetime.toordinal(someDate2))<<128)+long(0)
query = Record.all().filter('dateEx >', startDate).filter('dateEx <', endDate)
if you fetch 100 objects use the value of dateEx of the last object
retrieved as the new startDate for the next batch.
When you store a new entity:
get the object with the highest value of dateEx with the same
ordinal value,
add 1L and set as the dateEx value of the new entity
This should be done in a transaction or some kind of "Critical
section" to ensure unique sequence numbers
2009/4/20 Chris Spencer :
>
> Thanks, but I don't see how that's relevant. Like I said, the ordering
> of an object's key has no correlation to an object's date.
>
> I'm not trying to simply page. That's trivial using the method listed
> in your link. What I'm trying to do is page through a set of objects
> limited by a date range.
>
> Chris
>
> 2009/4/19 秦锋 :
>>
>> CHeck this:
>> http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt?gda=7FbhWXcAAACSStSWrftt07H4FK2Rtvurv5Qi9dmYyYZMEnTZCjCsYQQgER4RQV57mxjvIzAWBZmQ3TeCdbqm30Qz_AwgYlIpRbcWRj3jGGBm-fgbnPJIYc4-hXRRmo3Xgj6KgtSLBeZ45alvcyXc30EbEX-RNDZveV4duv6pDMGhhhZdjQlNAw
>>
>> On 4月19日, 下午12时32分, Chris wrote:
>>> I have the model:
>>>
>>> class Record(db.Model):
>>> date = db.DateProperty()
>>>
>>> Over 1000 Record instances may contain the same date value. The
>>> instances are also not created in the order of their dates.
>>>
>>> I'm trying to query all instances whose date is within a given range.
>>> I need to be able to query sets larger than the default max of 1000.
>>>
>>> I was thinking of doing something like:
>>>
>>> Record.all().filter('date >', startDate).filter('date <', endDate)
>>>
>>> But this would limit me at most to the first 1000 records. How would I
>>> get the next 1000 records, maintaining the range query? I can't use
>>> the last date value from the previous query as the startDate for the
>>> next query, since there's no guarantee that the date value has
>>> changed. For example, I could have 2000 instances with the date
>>> 2008-1-1, so the first and second query sets have identical filter
>>> parameters.
>>>
>>> Intuitively, I could accomplish this by simply filtering on the
>>> previous key value (e.g. .filter('key >', lastKeyValue)).
>>> Unfortunately, App Engine currently prohibits filtering with
>>> inequality operators on more than one property.
>>>
>>> Am I correct in understanding that there's essentially no way to query
>>> all instances within a specific date range, due to this limitation?
>>>
>>> This issue keeps coming up for me, and has been a huge deal-breaker in
>>> a lot of potential projects I've been working on for App Engine. Any
>>> advice on how I could work around it would be appreciated.
>>>
>>> Chris
>> >
>>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Retrieving Large Query Sets Containing Inequality Operators
Thanks, that's a good thought. I agree that it should be done in a
transaction. However, I don't believe App Engine allows you to run
queries in a transaction.
Chris
2009/4/20 djidjadji :
>
> Hi Chris,
>
> You can do this by adding an extra attribute to the object that
> contains the date and a sequence number.
>
> The extra attribute is an IntegerProperty. It has arbitrary precision,
> unlimited bit length.
> If you store (long(datetime.toordinal(myRecord.date))<<128)+long(sequenceNum)
> you can store a lot of entities on one date, 2**128. If you set
> sequenceNum starting at 1 you can use your query that sets
>
> startDate = (long(datetime.toordinal(someDate1))<<128)+long(0)
> endDate = (long(datetime.toordinal(someDate2))<<128)+long(0)
> query = Record.all().filter('dateEx >', startDate).filter('dateEx <', endDate)
>
> if you fetch 100 objects use the value of dateEx of the last object
> retrieved as the new startDate for the next batch.
>
> When you store a new entity:
>get the object with the highest value of dateEx with the same
> ordinal value,
>add 1L and set as the dateEx value of the new entity
> This should be done in a transaction or some kind of "Critical
> section" to ensure unique sequence numbers
>
> 2009/4/20 Chris Spencer :
>>
>> Thanks, but I don't see how that's relevant. Like I said, the ordering
>> of an object's key has no correlation to an object's date.
>>
>> I'm not trying to simply page. That's trivial using the method listed
>> in your link. What I'm trying to do is page through a set of objects
>> limited by a date range.
>>
>> Chris
>>
>> 2009/4/19 秦锋 :
>>>
>>> CHeck this:
>>> http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt?gda=7FbhWXcAAACSStSWrftt07H4FK2Rtvurv5Qi9dmYyYZMEnTZCjCsYQQgER4RQV57mxjvIzAWBZmQ3TeCdbqm30Qz_AwgYlIpRbcWRj3jGGBm-fgbnPJIYc4-hXRRmo3Xgj6KgtSLBeZ45alvcyXc30EbEX-RNDZveV4duv6pDMGhhhZdjQlNAw
>>>
>>> On 4月19日, 下午12时32分, Chris wrote:
I have the model:
class Record(db.Model):
date = db.DateProperty()
Over 1000 Record instances may contain the same date value. The
instances are also not created in the order of their dates.
I'm trying to query all instances whose date is within a given range.
I need to be able to query sets larger than the default max of 1000.
I was thinking of doing something like:
Record.all().filter('date >', startDate).filter('date <', endDate)
But this would limit me at most to the first 1000 records. How would I
get the next 1000 records, maintaining the range query? I can't use
the last date value from the previous query as the startDate for the
next query, since there's no guarantee that the date value has
changed. For example, I could have 2000 instances with the date
2008-1-1, so the first and second query sets have identical filter
parameters.
Intuitively, I could accomplish this by simply filtering on the
previous key value (e.g. .filter('key >', lastKeyValue)).
Unfortunately, App Engine currently prohibits filtering with
inequality operators on more than one property.
Am I correct in understanding that there's essentially no way to query
all instances within a specific date range, due to this limitation?
This issue keeps coming up for me, and has been a huge deal-breaker in
a lot of potential projects I've been working on for App Engine. Any
advice on how I could work around it would be appreciated.
Chris
>>> >
>>>
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Retrieving Large Query Sets Containing Inequality Operators
You can use a unique counter for all the entities. 2**128 is a lot.
You can use a sharded counter to generate the unique id's, it uses a
transaction.
2009/4/20 Chris Spencer :
>
> Thanks, that's a good thought. I agree that it should be done in a
> transaction. However, I don't believe App Engine allows you to run
> queries in a transaction.
>
> Chris
>
> 2009/4/20 djidjadji :
>>
>> Hi Chris,
>>
>> You can do this by adding an extra attribute to the object that
>> contains the date and a sequence number.
>>
>> The extra attribute is an IntegerProperty. It has arbitrary precision,
>> unlimited bit length.
>> If you store
>> (long(datetime.toordinal(myRecord.date))<<128)+long(sequenceNum)
>> you can store a lot of entities on one date, 2**128. If you set
>> sequenceNum starting at 1 you can use your query that sets
>>
>> startDate = (long(datetime.toordinal(someDate1))<<128)+long(0)
>> endDate = (long(datetime.toordinal(someDate2))<<128)+long(0)
>> query = Record.all().filter('dateEx >', startDate).filter('dateEx <',
>> endDate)
>>
>> if you fetch 100 objects use the value of dateEx of the last object
>> retrieved as the new startDate for the next batch.
>>
>> When you store a new entity:
>>get the object with the highest value of dateEx with the same
>> ordinal value,
>>add 1L and set as the dateEx value of the new entity
>> This should be done in a transaction or some kind of "Critical
>> section" to ensure unique sequence numbers
>>
>> 2009/4/20 Chris Spencer :
>>>
>>> Thanks, but I don't see how that's relevant. Like I said, the ordering
>>> of an object's key has no correlation to an object's date.
>>>
>>> I'm not trying to simply page. That's trivial using the method listed
>>> in your link. What I'm trying to do is page through a set of objects
>>> limited by a date range.
>>>
>>> Chris
>>>
>>> 2009/4/19 秦锋 :
CHeck this:
http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt?gda=7FbhWXcAAACSStSWrftt07H4FK2Rtvurv5Qi9dmYyYZMEnTZCjCsYQQgER4RQV57mxjvIzAWBZmQ3TeCdbqm30Qz_AwgYlIpRbcWRj3jGGBm-fgbnPJIYc4-hXRRmo3Xgj6KgtSLBeZ45alvcyXc30EbEX-RNDZveV4duv6pDMGhhhZdjQlNAw
On 4月19日, 下午12时32分, Chris wrote:
> I have the model:
>
> class Record(db.Model):
> date = db.DateProperty()
>
> Over 1000 Record instances may contain the same date value. The
> instances are also not created in the order of their dates.
>
> I'm trying to query all instances whose date is within a given range.
> I need to be able to query sets larger than the default max of 1000.
>
> I was thinking of doing something like:
>
> Record.all().filter('date >', startDate).filter('date <', endDate)
>
> But this would limit me at most to the first 1000 records. How would I
> get the next 1000 records, maintaining the range query? I can't use
> the last date value from the previous query as the startDate for the
> next query, since there's no guarantee that the date value has
> changed. For example, I could have 2000 instances with the date
> 2008-1-1, so the first and second query sets have identical filter
> parameters.
>
> Intuitively, I could accomplish this by simply filtering on the
> previous key value (e.g. .filter('key >', lastKeyValue)).
> Unfortunately, App Engine currently prohibits filtering with
> inequality operators on more than one property.
>
> Am I correct in understanding that there's essentially no way to query
> all instances within a specific date range, due to this limitation?
>
> This issue keeps coming up for me, and has been a huge deal-breaker in
> a lot of potential projects I've been working on for App Engine. Any
> advice on how I could work around it would be appreciated.
>
> Chris
>
>>
>> >
>>
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
