Re: [google-appengine] Re: Transaction across entities in different groups
If you want to get multiple objects with a list of ids and it only works if they have the same parent then it is a bug, if the parent parameter of Model.get_by_id([1,2,3],parent=None) is None. The code of Model.get_by_id(ids) only constructs a list of keys with db.Key.from_path and calls db.get(keys) The having same parent is only needed if you specify the parent parameter of the Model.get_by_id([1,2,3],parent=aparent) call. There is a small type in the doc for Model.get_by_id(ids) : http://code.google.com/appengine/docs/python/datastore/modelclass.html#Model_get_by_id If ids is a string representing one name, should be If ids is an integer representing one id, And the doc string for Model.get_by_id(ids) refers to a parameter key_names, that should be ids 2010/1/30 dburns drrnb...@gmail.com: You're right, it's not entirely clear why Model.get_by_id(ids) requires all entities to have the same parent, while Model.get(keys) doesn't have that requirement. Presumably some implementation detail. But whatever the reason, using Model.get(keys) is a good idea. I'll give that a shot when I get a chance. Thanks for the idea! On Jan 29, 4:06 pm, Danny Tuppeny da...@tuppeny.com wrote: I don't know if I was drunk when I sent this message, but I don't think it makes any sense at all. If you're calling get() on SS, it's quite clear what the kind is! On Jan 28, 7:20 am, Danny Tuppeny da...@tuppeny.com wrote: On Jan 24, 12:29 am, dburns drrnb...@gmail.com wrote: Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Can you use SS.get(keys) instead? I think the reason that get_by_id requires the parent is because IDs are not full keys (and presumably an ID can be duplicated for different kinds). If you create Keys for them (you'll need to know their Kind), you could call SS.get(keys) and get them all in one go. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
On Jan 30, 12:44 am, dburns drrnb...@gmail.com wrote: You're right, it's not entirely clear why Model.get_by_id(ids) requires all entities to have the same parent, while Model.get(keys) doesn't have that requirement. Presumably some implementation detail. An entity's parent is encoded in it's key. The parent can be any other entity, of whatever type. When you call Model.get_by_ids() it needs to construct the keys. It know the entity type, and you pass it the id. How does it know what arbitrary parent the entity may or may not have? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
I don't know if I was drunk when I sent this message, but I don't think it makes any sense at all. If you're calling get() on SS, it's quite clear what the kind is! On Jan 28, 7:20 am, Danny Tuppeny da...@tuppeny.com wrote: On Jan 24, 12:29 am, dburns drrnb...@gmail.com wrote: Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Can you use SS.get(keys) instead? I think the reason that get_by_id requires the parent is because IDs are not full keys (and presumably an ID can be duplicated for different kinds). If you create Keys for them (you'll need to know their Kind), you could call SS.get(keys) and get them all in one go. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
You're right, it's not entirely clear why Model.get_by_id(ids) requires all entities to have the same parent, while Model.get(keys) doesn't have that requirement. Presumably some implementation detail. But whatever the reason, using Model.get(keys) is a good idea. I'll give that a shot when I get a chance. Thanks for the idea! On Jan 29, 4:06 pm, Danny Tuppeny da...@tuppeny.com wrote: I don't know if I was drunk when I sent this message, but I don't think it makes any sense at all. If you're calling get() on SS, it's quite clear what the kind is! On Jan 28, 7:20 am, Danny Tuppeny da...@tuppeny.com wrote: On Jan 24, 12:29 am, dburns drrnb...@gmail.com wrote: Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Can you use SS.get(keys) instead? I think the reason that get_by_id requires the parent is because IDs are not full keys (and presumably an ID can be duplicated for different kinds). If you create Keys for them (you'll need to know their Kind), you could call SS.get(keys) and get them all in one go. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
On Jan 24, 12:29 am, dburns drrnb...@gmail.com wrote: Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Can you use SS.get(keys) instead? I think the reason that get_by_id requires the parent is because IDs are not full keys (and presumably an ID can be duplicated for different kinds). If you create Keys for them (you'll need to know their Kind), you could call SS.get(keys) and get them all in one go. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
Thanks Robert. The basic techniques I'm aware of for paging with a query object are either: 1) Order by key name (not useful for me since the order looks quasi- random; also you need to represent the starting key for the next page somehow, and it looks ugly in an url). 2) Sort by an indexed property that's invented for the purpose. An option perhaps, but probably not worth it for me to have the cost of another indexed property (both CPU time and size). Thanks for that blog link. It's exactly on the subject I'm talking about. Having read through it, it looks like the answer to my question is, possible, but not easy. I'm leaning toward simply accepting that the data could get out of sync on very rare occasions, and to make the system self-repairing or at least resilient to bogus entries. It may seem crude but it's just photos we're talking about here, not a bank account. You asked about query time vs. size of returned elements. Some quick experimentation seems to indicate that the gap between the two methods is close to linear with the number of elements returned, meaning there isn't simply a fixed overhead for the query. I'm talking about API CPU time. The real time seems to fluctuate enough that it's hard to say. Thank you both for your input. On Jan 24, 12:42 am, Robert Kluin robert.kl...@gmail.com wrote: You can page using the reference query. Since it is a query object, just be sure that you order in a stable way, then you can use one of the methods suggested in the app engine articles to page. Have you checked to see if as the size of the list property grows get_by_id()'s performance gets closer to the query's performance? Otherwise you might consider implementing something like Nick's distributed transactions:http://blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine Robert On Sat, Jan 23, 2010 at 9:14 PM, dburns drrnb...@gmail.com wrote: Thanks for the reply. Using a ReferenceProperty from the SS to the User does at first sound ideal, but the problem is that the back- reference that this gives you is actually a Query object. So that would essentially put me back to where I was initially (as I described in the second-last paragraph). Using the Query seems to be a bit slower than fetching all the objects by a list of ids, but the more important difference is that the list of ids lends itself to paging (I simply chop the list of ids into groups of 20 or whatever). One other optimization I get from using lists of ids (fav_ids and owned_ids) is that I can optimize the two get_by_id calls into a single call by concatenating the lists into one, then splitting the returned list since I know how many are favourites and how many are owned. There are a lot of things I like about the current set-up I have. The only thing I don't like is the lack of atomic operations. If I can't find a way out, I'll try to make the code more resilient to corrupted data just in case it ever happens. I appreciate the feedback. On Jan 23, 9:51 pm, Wooble geoffsp...@gmail.com wrote: Why not use a ReferenceProperty pointing to a User in the SS model instead of an unindexed StringProperty? The User model can then use the backreference collection to get a list of photos owned by the user. I don't see a problem with a simple ListProperty of favorites, although making this a list of db.Keys instead of a list of integer IDs is probably preferable, because if you do decide to make photos children of another entity the IDs won't be globally unique while the db.Keys still will be. On Jan 23, 7:29 pm, dburns drrnb...@gmail.com wrote: I'd appreciate any insight into the best design for this problem. I have a photo-sharing app. I want people to be able to mark a photo as a favourite, just as YouTube does with videos. I have two kinds: Snapshots and Users. Given a user, I need to be able to get a list of their favourites, and a list of the snapshots they created. I need the inverse too, i.e. given a snapshot, I need to know who made it. The problem I have is ensuring consistency across these two kinds, but I don't think I can put them into the same entity group. First, here they are: # The snapshot class. class SS(db.Model): owner = db.StringProperty(indexed=False) # Who created this (user id). #Other data about the snapshot here # The user class (the key_name is the user id) class User(db.Model): owned_ids = db.ListProperty(int, indexed=False) # IDs of owned snapshots (i.e. created by this user) fav_ids = db.ListProperty(int, indexed=False) # IDs of favourite snapshots The issue is that when a user either creates or deletes a snapshot, there's a potential for those to get out of sync if an exception happens just at the wrong moment (e.g. a snapshot could exist where the creating user doesn't have it in the owned_ids list). Making
[google-appengine] Re: Transaction across entities in different groups
Why not use a ReferenceProperty pointing to a User in the SS model instead of an unindexed StringProperty? The User model can then use the backreference collection to get a list of photos owned by the user. I don't see a problem with a simple ListProperty of favorites, although making this a list of db.Keys instead of a list of integer IDs is probably preferable, because if you do decide to make photos children of another entity the IDs won't be globally unique while the db.Keys still will be. On Jan 23, 7:29 pm, dburns drrnb...@gmail.com wrote: I'd appreciate any insight into the best design for this problem. I have a photo-sharing app. I want people to be able to mark a photo as a favourite, just as YouTube does with videos. I have two kinds: Snapshots and Users. Given a user, I need to be able to get a list of their favourites, and a list of the snapshots they created. I need the inverse too, i.e. given a snapshot, I need to know who made it. The problem I have is ensuring consistency across these two kinds, but I don't think I can put them into the same entity group. First, here they are: # The snapshot class. class SS(db.Model): owner = db.StringProperty(indexed=False) # Who created this (user id). #Other data about the snapshot here # The user class (the key_name is the user id) class User(db.Model): owned_ids = db.ListProperty(int, indexed=False) # IDs of owned snapshots (i.e. created by this user) fav_ids = db.ListProperty(int, indexed=False) # IDs of favourite snapshots The issue is that when a user either creates or deletes a snapshot, there's a potential for those to get out of sync if an exception happens just at the wrong moment (e.g. a snapshot could exist where the creating user doesn't have it in the owned_ids list). Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Originally I had no owned_ids in User, and did a query to find that user's snapshots (owner in SS was indexed). But that was slow and didn't lend itself to paging. So I switched to get_by_id. Documentation note: run_in_transaction athttp://code.google.com/appengine/docs/python/datastore/functions.html doesn't mention the restriction that entities have to be in the same group. I discovered it by seeing the exception, then read up in more detail elsewhere. I naturally started there so it probably should be mentioned. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
Thanks for the reply. Using a ReferenceProperty from the SS to the User does at first sound ideal, but the problem is that the back- reference that this gives you is actually a Query object. So that would essentially put me back to where I was initially (as I described in the second-last paragraph). Using the Query seems to be a bit slower than fetching all the objects by a list of ids, but the more important difference is that the list of ids lends itself to paging (I simply chop the list of ids into groups of 20 or whatever). One other optimization I get from using lists of ids (fav_ids and owned_ids) is that I can optimize the two get_by_id calls into a single call by concatenating the lists into one, then splitting the returned list since I know how many are favourites and how many are owned. There are a lot of things I like about the current set-up I have. The only thing I don't like is the lack of atomic operations. If I can't find a way out, I'll try to make the code more resilient to corrupted data just in case it ever happens. I appreciate the feedback. On Jan 23, 9:51 pm, Wooble geoffsp...@gmail.com wrote: Why not use a ReferenceProperty pointing to a User in the SS model instead of an unindexed StringProperty? The User model can then use the backreference collection to get a list of photos owned by the user. I don't see a problem with a simple ListProperty of favorites, although making this a list of db.Keys instead of a list of integer IDs is probably preferable, because if you do decide to make photos children of another entity the IDs won't be globally unique while the db.Keys still will be. On Jan 23, 7:29 pm, dburns drrnb...@gmail.com wrote: I'd appreciate any insight into the best design for this problem. I have a photo-sharing app. I want people to be able to mark a photo as a favourite, just as YouTube does with videos. I have two kinds: Snapshots and Users. Given a user, I need to be able to get a list of their favourites, and a list of the snapshots they created. I need the inverse too, i.e. given a snapshot, I need to know who made it. The problem I have is ensuring consistency across these two kinds, but I don't think I can put them into the same entity group. First, here they are: # The snapshot class. class SS(db.Model): owner = db.StringProperty(indexed=False) # Who created this (user id). #Other data about the snapshot here # The user class (the key_name is the user id) class User(db.Model): owned_ids = db.ListProperty(int, indexed=False) # IDs of owned snapshots (i.e. created by this user) fav_ids = db.ListProperty(int, indexed=False) # IDs of favourite snapshots The issue is that when a user either creates or deletes a snapshot, there's a potential for those to get out of sync if an exception happens just at the wrong moment (e.g. a snapshot could exist where the creating user doesn't have it in the owned_ids list). Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Originally I had no owned_ids in User, and did a query to find that user's snapshots (owner in SS was indexed). But that was slow and didn't lend itself to paging. So I switched to get_by_id. Documentation note: run_in_transaction athttp://code.google.com/appengine/docs/python/datastore/functions.html doesn't mention the restriction that entities have to be in the same group. I discovered it by seeing the exception, then read up in more detail elsewhere. I naturally started there so it probably should be mentioned. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transaction across entities in different groups
Thanks for the reply. Using a ReferenceProperty from the SS to the User does at first sound ideal, but the problem is that the back- reference that this gives you is actually a Query object. So that would essentially put me back to where I was initially (as I described in the second-last paragraph). Using the Query seems to be a bit slower than fetching all the objects by a list of ids, but the more important difference is that the list of ids lends itself to paging (I simply chop the list of ids into groups of 20 or whatever). One other optimization I get from using lists of ids (fav_ids and owned_ids) is that I can optimize the two get_by_id calls into a single call by concatenating the lists into one, then splitting the returned list since I know how many are favourites and how many are owned. There are a lot of things I like about the current set-up I have. The only thing I don't like is the lack of atomic operations. If I can't find a way out, I'll try to make the code more resilient to corrupted data just in case it ever happens. I appreciate the feedback. On Jan 23, 9:51 pm, Wooble geoffsp...@gmail.com wrote: Why not use a ReferenceProperty pointing to a User in the SS model instead of an unindexed StringProperty? The User model can then use the backreference collection to get a list of photos owned by the user. I don't see a problem with a simple ListProperty of favorites, although making this a list of db.Keys instead of a list of integer IDs is probably preferable, because if you do decide to make photos children of another entity the IDs won't be globally unique while the db.Keys still will be. On Jan 23, 7:29 pm, dburns drrnb...@gmail.com wrote: I'd appreciate any insight into the best design for this problem. I have a photo-sharing app. I want people to be able to mark a photo as a favourite, just as YouTube does with videos. I have two kinds: Snapshots and Users. Given a user, I need to be able to get a list of their favourites, and a list of the snapshots they created. I need the inverse too, i.e. given a snapshot, I need to know who made it. The problem I have is ensuring consistency across these two kinds, but I don't think I can put them into the same entity group. First, here they are: # The snapshot class. class SS(db.Model): owner = db.StringProperty(indexed=False) # Who created this (user id). #Other data about the snapshot here # The user class (the key_name is the user id) class User(db.Model): owned_ids = db.ListProperty(int, indexed=False) # IDs of owned snapshots (i.e. created by this user) fav_ids = db.ListProperty(int, indexed=False) # IDs of favourite snapshots The issue is that when a user either creates or deletes a snapshot, there's a potential for those to get out of sync if an exception happens just at the wrong moment (e.g. a snapshot could exist where the creating user doesn't have it in the owned_ids list). Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Originally I had no owned_ids in User, and did a query to find that user's snapshots (owner in SS was indexed). But that was slow and didn't lend itself to paging. So I switched to get_by_id. Documentation note: run_in_transaction athttp://code.google.com/appengine/docs/python/datastore/functions.html doesn't mention the restriction that entities have to be in the same group. I discovered it by seeing the exception, then read up in more detail elsewhere. I naturally started there so it probably should be mentioned. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Transaction across entities in different groups
You can page using the reference query. Since it is a query object, just be sure that you order in a stable way, then you can use one of the methods suggested in the app engine articles to page. Have you checked to see if as the size of the list property grows get_by_id()'s performance gets closer to the query's performance? Otherwise you might consider implementing something like Nick's distributed transactions: http://blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine Robert On Sat, Jan 23, 2010 at 9:14 PM, dburns drrnb...@gmail.com wrote: Thanks for the reply. Using a ReferenceProperty from the SS to the User does at first sound ideal, but the problem is that the back- reference that this gives you is actually a Query object. So that would essentially put me back to where I was initially (as I described in the second-last paragraph). Using the Query seems to be a bit slower than fetching all the objects by a list of ids, but the more important difference is that the list of ids lends itself to paging (I simply chop the list of ids into groups of 20 or whatever). One other optimization I get from using lists of ids (fav_ids and owned_ids) is that I can optimize the two get_by_id calls into a single call by concatenating the lists into one, then splitting the returned list since I know how many are favourites and how many are owned. There are a lot of things I like about the current set-up I have. The only thing I don't like is the lack of atomic operations. If I can't find a way out, I'll try to make the code more resilient to corrupted data just in case it ever happens. I appreciate the feedback. On Jan 23, 9:51 pm, Wooble geoffsp...@gmail.com wrote: Why not use a ReferenceProperty pointing to a User in the SS model instead of an unindexed StringProperty? The User model can then use the backreference collection to get a list of photos owned by the user. I don't see a problem with a simple ListProperty of favorites, although making this a list of db.Keys instead of a list of integer IDs is probably preferable, because if you do decide to make photos children of another entity the IDs won't be globally unique while the db.Keys still will be. On Jan 23, 7:29 pm, dburns drrnb...@gmail.com wrote: I'd appreciate any insight into the best design for this problem. I have a photo-sharing app. I want people to be able to mark a photo as a favourite, just as YouTube does with videos. I have two kinds: Snapshots and Users. Given a user, I need to be able to get a list of their favourites, and a list of the snapshots they created. I need the inverse too, i.e. given a snapshot, I need to know who made it. The problem I have is ensuring consistency across these two kinds, but I don't think I can put them into the same entity group. First, here they are: # The snapshot class. class SS(db.Model): owner = db.StringProperty(indexed=False) # Who created this (user id). #Other data about the snapshot here # The user class (the key_name is the user id) class User(db.Model): owned_ids = db.ListProperty(int, indexed=False) # IDs of owned snapshots (i.e. created by this user) fav_ids = db.ListProperty(int, indexed=False) # IDs of favourite snapshots The issue is that when a user either creates or deletes a snapshot, there's a potential for those to get out of sync if an exception happens just at the wrong moment (e.g. a snapshot could exist where the creating user doesn't have it in the owned_ids list). Making a User instance the parent of a SS (snapshot) instance seems like a natural fit, except then I can't fetch all the favourites via: favs = SS.get_by_id(user.fav_ids). The reason is that all parents have to be the same to use SS.get_by_id (according tohttp://code.google.com/appengine/docs/python/datastore/modelclass.htm...), but those favourites may have been created by various users (hence the parents would be different). Originally I had no owned_ids in User, and did a query to find that user's snapshots (owner in SS was indexed). But that was slow and didn't lend itself to paging. So I switched to get_by_id. Documentation note: run_in_transaction athttp://code.google.com/appengine/docs/python/datastore/functions.html doesn't mention the restriction that entities have to be in the same group. I discovered it by seeing the exception, then read up in more detail elsewhere. I naturally started there so it probably should be mentioned. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message
