Ah...Got it. Thanks a bunch Nick. Manny On Wed, Feb 17, 2010 at 9:12 PM, Nick Johnson (Google) < nick.john...@google.com> wrote:
> Hi Manny, > > On Wed, Feb 17, 2010 at 6:57 AM, Manny S <manny.m...@gmail.com> wrote: > >> Ryan, >> I just made the pagination code work using cursors. One thing I noticed in >> the documentation is this: >> >> >> - A query with a cursor does not always work as expected when the >> query uses an inequality filter or a sort order on a property with >> multiple >> values. Logic that de-duplicates the multivalued property for the query >> does >> not persist between queries, and may return a result more than once. >> >> >> I am sorting the results of the query (simple query that returns data >> below a threshold value which is a long) by post date. The sort order is >> post date and might have duplicate values. Does this mean that the cursors >> will return duplicates? >> > > No, that's not (quite) what that bullet point means. An example may help. > Suppose you have three entities: > > BlogPost(key_name="one", tags=["abacus", "dog"]) > BlogPost(key_name="two", tags=["banana", "elephant"]) > BlogPost(key_name=three", tags=["cat", "zebra"]) > > And a query like this: > > q = BlogPost.all().order("tags") > > The index used for this query will look something like this: > > (BlogPost, tags, abacus) -> Key(BlogPost, one) > (BlogPost, tags, banana) -> Key(BlogPost, two) > (BlogPost, tags, cat) -> Key(BlogPost, three) > (BlogPost, tags, dog) -> Key(BlogPost, one) > (BlogPost, tags, elephant) -> Key(BlogPost, two) > (BlogPost, tags, zebra) -> Key(BlogPost, three) > > Calling q.fetch(2) will scan over that index to return the first two > entries and return them - entities "one" and "two". Calling q.fetch(3) will > return all three posts, in that order, but without duplicates, as the query > infrastructure keeps a list of previously seen keys to avoid returning > duplicates. > > If, however, you call q.fetch(2), then get a cursor with q.cursor(), and > later re-use it again: > > q.fetch(2) # Returns "one", "two" > cursor = q.cursor() > q = BlogPost.all().order("tags").with_cursor(cursor) > q.fetch(2) # Returns "three", "one" > > The resumed fetch returns "one" again, because the cursor doesn't contain > enough information to know what results were previously returned. > > As you can see, this only happens if you're using multiply valued > properties (ListProperties), and sorting or doing an inequality filter on > them. > > -Nick Johnson > > >> >> Manny >> >> On Tue, Feb 16, 2010 at 4:01 PM, Manny S <manny.m...@gmail.com> wrote: >> >>> Ryan - I will check out query cursors. Took a quick look and I think >>> this would solve my problems. Thanks. >>> Jeff - thanks for the input. I have included a createdDate and a >>> modifiedDate in all my entities. >>> >>> manny >>> >>> On Tue, Feb 16, 2010 at 12:11 AM, Jeff Schnitzer <j...@infohazard.org>wrote: >>> >>>> You will likely find it good practice to keep a createdDate (and >>>> possibly a modifiedDate) in all of your entities. Makes debugging >>>> easier, and if you're looking for this kind of pagination, there you >>>> go. Trying to abuse the key system for these purposes seems a bit >>>> hazardous. >>>> >>>> That said, you can always generate your own String name as part of the >>>> key. This is not a trivial task given the distributed nature of >>>> appengine. >>>> >>>> Jeff >>>> >>>> On Sat, Feb 13, 2010 at 8:12 AM, Manny S <manny.m...@gmail.com> wrote: >>>> > Hi Tim, >>>> > Thanks again Tim. A naive question - if I use the datastore generated >>>> keys >>>> > and sort by the keys desc (for pagination) the key value of new data >>>> > inserted will be greater than that of previously entered data (if I do >>>> a >>>> > keytostring and a string compare?). >>>> > In other words does the datastore generate keys in a order or at >>>> random. I >>>> > did some experiments and I found that pagination works according to >>>> > date/time of insert if I just sort keys by descending but just wanted >>>> to >>>> > make sure. >>>> > Regards, >>>> > Manny >>>> > On Sat, Feb 13, 2010 at 1:17 PM, Tim Hoffman <zutes...@gmail.com> >>>> wrote: >>>> >> >>>> >> Hi Manny >>>> >> >>>> >> Understood. >>>> >> >>>> >> You know you can let the system generate the entity identifiers for >>>> >> you, and pagination will >>>> >> just work (by __key__) any way. So you don't have to the key_name or >>>> >> id at all >>>> >> if you don't care what the key is. >>>> >> >>>> >> Rgds >>>> >> >>>> >> T >>>> >> >>>> >> On Feb 13, 1:00 am, Manny S <manny.m...@gmail.com> wrote: >>>> >> > Thanks a bunch Tim for your inputs, >>>> >> > >>>> >> > My rationale for adding the date to the appstore generated key is >>>> to >>>> >> > make >>>> >> > pagination easier. I would do pagination on the key and not add a >>>> >> > separate >>>> >> > column for that. (Pagination by date alone will also not solve my >>>> >> > problem >>>> >> > since it can have duplicates and hence couple it with the key) >>>> >> > >>>> >> > I understand fetching by keys is much faster. Though I don't see a >>>> >> > scenario >>>> >> > where I would have to do that now I would like to architect my app >>>> where >>>> >> > that would be possible. However, I do not have anything unique in >>>> my >>>> >> > record >>>> >> > with which I can set the key. It just contains city name, locality >>>> >> > details >>>> >> > and a series of other fields all of which could have duplicates. >>>> Any >>>> >> > ideas >>>> >> > as to how I can generate unique ids from these or any general >>>> pointers >>>> >> > towards generating unique Ids from data where the data itself does >>>> not >>>> >> > have >>>> >> > a unique field? >>>> >> > >>>> >> > Manny >>>> >> > >>>> >> > On Thu, Feb 11, 2010 at 4:49 PM, Tim Hoffman <zutes...@gmail.com> >>>> wrote: >>>> >> > > Hi Manny >>>> >> > >>>> >> > > Do you really want to do that for a key. One if the big >>>> advantages of >>>> >> > > creating your own keys >>>> >> > > is being able to explicitly get entities by key (1 or more with >>>> >> > > db.get(list_of_keys) which is much >>>> >> > > quicker than a gql or filter. Making your keys include dates >>>> mean you >>>> >> > > will be unlikely to >>>> >> > > guess/know what the keys are in advance. >>>> >> > >>>> >> > > This of course may not be useful for what you are doing, but >>>> worth >>>> >> > > keeping in mind. >>>> >> > >>>> >> > > Rgds >>>> >> > >>>> >> > > T >>>> >> > >>>> >> > > On Feb 11, 2:12 pm, Manny S <manny.m...@gmail.com> wrote: >>>> >> > > > Hi Ikai, >>>> >> > >>>> >> > > > I did read the documentation and now I have my data structures >>>> in >>>> >> > > > place. >>>> >> > > One >>>> >> > > > thing I wanted to do and that was not clear from my previous >>>> post >>>> >> > > > was to >>>> >> > > > append a app generated string (not unique) as a prefix to a >>>> >> > > > datastore >>>> >> > > > generated key. For instance, I want to generate a key that has >>>> the >>>> >> > > > date >>>> >> > > (of >>>> >> > > > record creation) as a prefix to the datastore generated unique >>>> key. >>>> >> > > > Is >>>> >> > > > there a way to do this? I do not want my application to >>>> generate >>>> >> > > > unique >>>> >> > > Ids. >>>> >> > >>>> >> > > > From reading through the literature so far, I am guessing that >>>> will >>>> >> > > > not >>>> >> > > be >>>> >> > > > possible since the datastore keys are generated only at the >>>> time >>>> >> > > > when the >>>> >> > > > objects are being made persistent. >>>> >> > >>>> >> > > > Manny >>>> >> > >>>> >> > > > On Wed, Feb 10, 2010 at 3:30 AM, Ikai L (Google) < >>>> ika...@google.com> >>>> >> > > wrote: >>>> >> > >>>> >> > > > > Have you read our documentation on KeyFactory? >>>> >> > >>>> >> > > > >>>> >> > > > > > >>>> http://code.google.com/appengine/docs/java/datastore/relationships.html >>>> >> > >>>> >> > > > > < >>>> >> > > >>>> http://code.google.com/appengine/docs/java/datastore/relationships.html >>>> >> > > >I'd >>>> >> > > > > try to understand what's going on there. It sounds like >>>> you're >>>> >> > > > > doing it >>>> >> > > the >>>> >> > > > > right way, but it's up to you to benchmark and find the best >>>> >> > > > > approach >>>> >> > > for >>>> >> > > > > what works for you. The usage characteristics of your >>>> application >>>> >> > > should >>>> >> > > > > determine the way your store your data. >>>> >> > >>>> >> > > > > On Wed, Feb 3, 2010 at 3:42 AM, Manny S < >>>> manny.m...@gmail.com> >>>> >> > > > > wrote: >>>> >> > >>>> >> > > > >> Ikai, >>>> >> > > > >> Based on your inputs I created two data classes that have a >>>> >> > > unidirectional >>>> >> > > > >> one-to-one relationship >>>> >> > > > >> Now, I have two data classes simpledata and detailscol. >>>> >> > > > >> simpledata contains fields A, B, C (and a Key field) >>>> >> > > > >> detailscol just contains field D. >>>> >> > >>>> >> > > > >> simpledata imports detailscol that contains field D (and a >>>> Key >>>> >> > > > >> field). >>>> >> > > It >>>> >> > > > >> also contains an accessor for the detailscol. >>>> >> > > > >> Code: >>>> >> > > > >> simpledata sdata = new simpledata(A,B,C); >>>> >> > > > >> sdata.setKey(null); >>>> >> > > > >> detailscol obj = new detailscol(D); >>>> >> > > > >> sdata.setD(obj); >>>> >> > >>>> >> > > > >> The keys are generated by the application and then I make >>>> the >>>> >> > > > >> data >>>> >> > > > >> persistent. >>>> >> > >>>> >> > > > >> Now, I display just the data in simpledata and if the user >>>> clicks >>>> >> > > > >> on a >>>> >> > > > >> details link I get the data stored in detailscol >>>> >> > > > >> To get to that data I just do >>>> >> > >>>> >> > > > >> detailscol d = sdata.getDetails(); >>>> >> > >>>> >> > > > >> Two questions: >>>> >> > >>>> >> > > > >> 1) Is this the right approach? >>>> >> > >>>> >> > > > >> 2) If I want to get the child data using just the parent >>>> keyhow >>>> >> > > > >> do I >>>> >> > > go >>>> >> > > > >> about it? >>>> >> > >>>> >> > > > >> E.g, user clicks details and I use some AJAX to redirect to >>>> a >>>> >> > > different >>>> >> > > > >> servlet with just parent key as a parameter (since I don't >>>> access >>>> >> > > > >> the >>>> >> > > child >>>> >> > > > >> object yet). I get the parent key using >>>> >> > > > >> KeyFactory.keyToString(sdata.getKey()); >>>> >> > >>>> >> > > > >> Now, that I have the parent's key should I do a >>>> getObjectbyID on >>>> >> > > > >> the >>>> >> > > > >> parent data again using this and then get the child using >>>> the >>>> >> > > > >> accessor >>>> >> > > > >> method or is there a direct way to construct the child key >>>> and >>>> >> > > > >> get to >>>> >> > > the >>>> >> > > > >> child data. >>>> >> > >>>> >> > > > >> Due to the nature of my application I would like to have the >>>> key >>>> >> > > generated >>>> >> > > > >> automatically (using setKey(null)). >>>> >> > >>>> >> > > > >> Apologies for the confusion in advance :) >>>> >> > >>>> >> > > > >> Manny >>>> >> > >>>> >> > > > >> On Sat, Jan 30, 2010 at 12:16 AM, Ikai L (Google) >>>> >> > > > >> <ika...@google.com >>>> >> > > >wrote: >>>> >> > >>>> >> > > > >>> Hi Manny, >>>> >> > >>>> >> > > > >>> A few things to first remember - App Engine's datastore is >>>> not a >>>> >> > > > >>> database, but a distributed key value store with additional >>>> >> > > > >>> features. >>>> >> > > Thus, >>>> >> > > > >>> we should be careful not to frame our thinking in terms of >>>> RDBMS >>>> >> > > schemas. >>>> >> > > > >>> For this reason, I like to avoid using database terminology >>>> that >>>> >> > > > >>> can >>>> >> > > > >>> confound the design process like "table" or "column". App >>>> Engine >>>> >> > > stores >>>> >> > > > >>> objects serialized ("entities") and indexes on the values. >>>> It'd >>>> >> > > > >>> be >>>> >> > > similar >>>> >> > > > >>> to an approach of creating a MySQL table with a String ID >>>> and a >>>> >> > > > >>> blob >>>> >> > > value, >>>> >> > > > >>> storing serialized Objects in the blob column, or using >>>> Memcache >>>> >> > > > >>> and >>>> >> > > storing >>>> >> > > > >>> JSON values. >>>> >> > >>>> >> > > > >>> When you retrieve a single value from the key value store, >>>> we >>>> >> > > > >>> have to >>>> >> > > > >>> retrieve everything at once. In most scenarios, unlike SQL >>>> >> > > > >>> databases >>>> >> > > you may >>>> >> > > > >>> be used to, retrieving large binary or text data does not >>>> add >>>> >> > > > >>> serious >>>> >> > > > >>> overhead. Of course, this changes if you start storing data >>>> on >>>> >> > > > >>> the >>>> >> > > scale of >>>> >> > > > >>> 1mb and are retrieving it unnecessarily. How large is the >>>> data >>>> >> > > > >>> you >>>> >> > > are >>>> >> > > > >>> retrieving? >>>> >> > >>>> >> > > > >>> Here's the way I would model your scenario if I was >>>> positive the >>>> >> > > > >>> text/binary field had a 1:1 relationship with the parent >>>> class: >>>> >> > >>>> >> > > > >>> * on your main entity, define the properties. >>>> >> > > > >>> * define a new entity with a text/binary field, and encode >>>> the >>>> >> > > > >>> parent >>>> >> > > key >>>> >> > > > >>> information in this key such that generating the key for >>>> this >>>> >> > > > >>> child >>>> >> > > field is >>>> >> > > > >>> very cheap. KeyFactory.stringToKey and >>>> KeyFactory.keyToString >>>> >> > > > >>> are >>>> >> > > crucial >>>> >> > > > >>> here. Read more about them here: >>>> >> > >>>> >> > >>>> >> > > > >>>> http://code.google.com/appengine/docs/java/javadoc/com/google/appengi.. >>>> .. >>>> >> > > > >>> You can call your child property >>>> "parent_id:additional_info" or >>>> >> > > whatever >>>> >> > > > >>> makes sense to you. >>>> >> > >>>> >> > > > >>> Robert's solution of using a child key is basically just a >>>> >> > > > >>> variation >>>> >> > > on >>>> >> > > > >>> this, as parent key information is encoded in a child key. >>>> >> > >>>> >> > > > >>> A lot of this stuff can be a bit different to get used to. >>>> I >>>> >> > > > >>> suggest >>>> >> > > > >>> becoming familiar with keys and how they are used in App >>>> Engine: >>>> >> > >>>> >> > > > >>> Basic documentation about relationships: >>>> >> > >>>> >> > > >>>> http://code.google.com/appengine/docs/java/datastore/relationships.html >>>> >> > > > >>> A more advanced article: >>>> >> > > > >>> >>>> http://code.google.com/appengine/articles/storage_breakdown.html >>>> >> > >>>> >> > > > >>> On Thu, Jan 28, 2010 at 10:28 PM, Manny S >>>> >> > > > >>> <manny.m...@gmail.com >>>> >> > > >wrote: >>>> >> > >>>> >> > > > >>>> Hi All, >>>> >> > >>>> >> > > > >>>> First off, thanks for your time. A quick noob question on >>>> the >>>> >> > > > >>>> right >>>> >> > > way >>>> >> > > > >>>> to model data. >>>> >> > >>>> >> > > > >>>> I have a table with four columns A,B,C, D. D - the fourth >>>> is >>>> >> > > > >>>> of >>>> >> > > type >>>> >> > > > >>>> text (contains quite a bit of data). >>>> >> > >>>> >> > > > >>>> I wanted to ensure that the contents of the details column >>>> 'D' >>>> >> > > > >>>> is >>>> >> > > not >>>> >> > > > >>>> fetched during a query. A sample scenario >>>> >> > > > >>>> User does a search. Sees Columns A,B,C. If they need more >>>> >> > > > >>>> details >>>> >> > > for >>>> >> > > > >>>> that particular record Click on a link that fetches D for >>>> that >>>> >> > > particular >>>> >> > > > >>>> record. >>>> >> > >>>> >> > > > >>>> So I tried to do something like - Select A, B, C from >>>> >> > > > >>>> tablename. >>>> >> > >>>> >> > > > >>>> I found from the documentation that the GQL query returns >>>> full >>>> >> > > > >>>> data >>>> >> > > > >>>> objects and so all queries start with SELECT *. Is this >>>> true >>>> >> > > > >>>> for >>>> >> > > JDOQL on >>>> >> > > > >>>> the datastore as well? Does this mean everytime I query >>>> the >>>> >> > > > >>>> data >>>> >> > > store its >>>> >> > > > >>>> going to return all columns consuming bandwidth? >>>> >> > >>>> >> > > > >>>> Also since I want the content of COlumn D to be fetched on >>>> >> > > subsequent >>>> >> > > > >>>> user action so should I instead create two tables one with >>>> >> > >>>> >> > > > >>>> ID_TB1, A, B, C >>>> >> > >>>> >> > > > >>>> and the other one with >>>> >> > >>>> >> > > > >>>> ID, ID_TB1, D? >>>> >> > >>>> >> > > > >>>> Manny >>>> >> > >>>> >> > > > >>>> -- >>>> >> > > > >>>> You received this message because you are subscribed to >>>> the >>>> >> > > > >>>> Google >>>> >> > > > >>>> Groups "Google App Engine" group. >>>> >> > > > >>>> To post to this group, send email to >>>> >> > > google-appeng...@googlegroups.com. >>>> >> > > > >>>> To unsubscribe from this group, send email to >>>> >> > > > >>>> >>>> >> > > > >>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com> >>>> > >>>> >> > > >>>> >> > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com> >>>> <google-appengine%252bunsubscr...@googlegroups.com<google-appengine%25252bunsubscr...@googlegroups.com> >>>> > >>>> >> > >>>> >> > > > >>>> . >>>> >> > > > >>>> For more options, visit this group at >>>> >> > > > >>>>http://groups.google.com/group/google-appengine?hl=en. >>>> >> > >>>> >> > > > >>> -- >>>> >> > > > >>> Ikai Lan >>>> >> > > > >>> Developer Programs Engineer, Google App Engine >>>> >> > > > >>>http://googleappengine.blogspot.com| >>>> http://twitter.com/app_engine >>>> >> > >>>> >> > > > >>> -- >>>> >> > > > >>> You received this message because you are subscribed to >>>> the >>>> >> > > > >>> Google >>>> >> > > > >>> Groups "Google App Engine" group. >>>> >> > > > >>> To post to this group, send email to >>>> >> > > google-appeng...@googlegroups.com. >>>> >> > > > >>> To unsubscribe from this group, send email to >>>> >> > > > >>> >>>> >> > > > >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com> >>>> > >>>> >> > > >>>> >> > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com> >>>> <google-appengine%252bunsubscr...@googlegroups.com<google-appengine%25252bunsubscr...@googlegroups.com> >>>> > >>>> >> > >>>> >> > > > >>> . >>>> >> > > > >>> For more options, visit this group at >>>> >> > > > >>>http://groups.google.com/group/google-appengine?hl=en. >>>> >> > >>>> >> > > > >> -- >>>> >> > > > >> You received this message because you are subscribed to the >>>> >> > > > >> Google >>>> >> > > Groups >>>> >> > > > >> "Google App Engine" group. >>>> >> > > > >> To post to this group, send >>>> >> > >>>> >> > ... >>>> >> > >>>> >> > read more ยป >>>> >> >>>> >> -- >>>> >> You received this message because you are subscribed to the Google >>>> Groups >>>> >> "Google App Engine" group. >>>> >> To post to this group, send email to >>>> google-appeng...@googlegroups.com. >>>> >> To unsubscribe from this group, send email to >>>> >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> . >>>> >> For more options, visit this group at >>>> >> http://groups.google.com/group/google-appengine?hl=en. >>>> >> >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups >>>> > "Google App Engine" group. >>>> > To post to this group, send email to >>>> google-appeng...@googlegroups.com. >>>> > To unsubscribe from this group, send email to >>>> > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> . >>>> > For more options, visit this group at >>>> > http://groups.google.com/group/google-appengine?hl=en. >>>> > >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to google-appeng...@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> . >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.