[google-appengine] Re: Datastore design question
It sounds like the list property can't be appended to without a transaction? If so, it seems it'll be better to create two tables, one for Users and one for Items. Just to answer the above questions... I expect many reads of the data and few writes. I also expect the normal case to have few items (1-2 on avg) and it won't change often. The key to the Items table would be user_id-item_name. Are prefix scans in GQL efficient? Also, in terms of data I'm storing about the items, I have ~5-10 fields of metadata. For now I'm assuming that the right thing to do here is to split the data between tables. Even if it seems to be an unlikely problem now, I don't want to put myself in a situation where I'm limited by the QPS that transactions can support. Thanks, Phil On Jun 27, 11:17 pm, Robert Kluin robert.kl...@gmail.com wrote: The answer to this question depends on several other questions too. For example: 1) how many items are you expecting? 2) how often do a users items change, and how do they get changed (the user or something else)? 3) what type of querying / reporting do you need to be able to do? 4) what type of information would you need to store about each users item? Robert On Sun, Jun 27, 2010 at 9:59 PM, Tom Wu service.g2...@gmail.com wrote: Hi Luís, If you have a User model, with a items heavy Property like TextProperty which is not necessary for every query. Which db structure is better ? Best Regards Tom Wu 2010/6/28 Luís Marques luismarq...@gmail.com Hello Phil, Anyone correct me if I'm wrong but you use one of the following options, among others. You can have a User model, with a items ListProperty. The pros: if you have the key for the user (e.g. you can use the email), then with a simple get (faster than a query) you can retrieve the entity including the items. Writing a new object is transactional. The cons: you must always get a set the entire user entity. The entity might get large if there are a lot of items. If you use custom indexes with the items property the index might get large quickly, especially if you use more than one ListProperty (exploding indexes). You can have a User model, and a UserItem whose parent is the User, making it part of a transaction group. Cons: you have limited QPS writes to the transaction group (about 1-10 QPS), but that's the same limitation you have to a standalone User with items. You have to query (seek) the items, although I wouldn't call that a problem without profile data. Pros: you can write individual UserItems. You can have transactions for the user and the items s/he has. You can have a User model and a UserItem with a User key reference. Cons: you can no longer have a transaction depending on the user and the items s/he has. Pros: you can write individual items without write collisions to the User. You can have a User and a flat list (e.g. text, non-indexed) of the corresponding items. Cons: limits the queries you can make, have to update all items at once. Pros: better index performance. Etc. I hope it was helpful and correct? Best regards, Luís -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
On Jun 29, 8:41 am, Phil phil.a.mcdonn...@gmail.com wrote: It sounds like the list property can't be appended to without a transaction? If so, it seems it'll be better to create two tables, one for Users and one for Items. The list property can be appended without an explicit transaction. It just might not be wise, since between a read of the entity and the write with the appended list another request can update the entity, and that update would be lost. The entity write itself is atomic though. You won't have some of the properties written from the original request and some written from the other concurrent request. So just be careful of race conditions, if you can be sure there aren't concurrent requests or that they are idempotent then I guess you should be OK without a transaction. Just to answer the above questions... I expect many reads of the data and few writes. I also expect the normal case to have few items (1-2 on avg) and it won't change often. The key to the Items table would be user_id-item_name. Are prefix scans in GQL efficient? Also, in terms of data I'm storing about the items, I have ~5-10 fields of metadata. If you know the key then that's the fastest way to retrieve the entity, instead of using a query. I don't know the Java syntax by heart, but it's easy to find. For now I'm assuming that the right thing to do here is to split the data between tables. Even if it seems to be an unlikely problem now, I don't want to put myself in a situation where I'm limited by the QPS that transactions can support. Someone more experienced might be able to elucidate, but on your scenario I don't know what you'd gain with that, other than smaller RPC requests for writing to the user items (at least performance-wise, but it might be a cleaner design). If you have to make a transaction then the User and UserItems must be part of a transaction group (items have the user as a parent), so the transaction locks out the entire group anyway. That's the same as locking out a group made of a single entity, a user with a list of items property. See if you can avoid transactions with your design, if you ever run into write performance problems. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore design question
The answer to this question depends on several other questions too. For example: 1) how many items are you expecting? 2) how often do a users items change, and how do they get changed (the user or something else)? 3) what type of querying / reporting do you need to be able to do? 4) what type of information would you need to store about each users item? Robert On Sun, Jun 27, 2010 at 9:59 PM, Tom Wu service.g2...@gmail.com wrote: Hi Luís, If you have a User model, with a items heavy Property like TextProperty which is not necessary for every query. Which db structure is better ? Best Regards Tom Wu 2010/6/28 Luís Marques luismarq...@gmail.com Hello Phil, Anyone correct me if I'm wrong but you use one of the following options, among others. You can have a User model, with a items ListProperty. The pros: if you have the key for the user (e.g. you can use the email), then with a simple get (faster than a query) you can retrieve the entity including the items. Writing a new object is transactional. The cons: you must always get a set the entire user entity. The entity might get large if there are a lot of items. If you use custom indexes with the items property the index might get large quickly, especially if you use more than one ListProperty (exploding indexes). You can have a User model, and a UserItem whose parent is the User, making it part of a transaction group. Cons: you have limited QPS writes to the transaction group (about 1-10 QPS), but that's the same limitation you have to a standalone User with items. You have to query (seek) the items, although I wouldn't call that a problem without profile data. Pros: you can write individual UserItems. You can have transactions for the user and the items s/he has. You can have a User model and a UserItem with a User key reference. Cons: you can no longer have a transaction depending on the user and the items s/he has. Pros: you can write individual items without write collisions to the User. You can have a User and a flat list (e.g. text, non-indexed) of the corresponding items. Cons: limits the queries you can make, have to update all items at once. Pros: better index performance. Etc. I hope it was helpful and correct? Best regards, Luís -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore design question
i got alot of user meta data, i save them on another db model called
usermeta which is referencekey to the user model key. under the usermeta i
write duplicated data from the user table too such as user key id as string
and much more. i usually pull the usermeta only and use the user model just
for authentication keeping it clean and to the purpose.
-
Cheers,
public static function AlonCarmel() {
//Contact me
var email = 'a...@aloncarmel.me';
var twitter = '@aloncarmel';
var web = 'http://aloncarmel.me';
var phone = '+972-54-4860380';
}
* If you received an unsolicited email from by mistake that wasn't of your
matter please delete immediately. All E-mail sent from Alon Carmel is
copyrighted to Alon Carmel 2008. Any details revealed in e-mails sent by
Alon Carmel are owned by the Author only. Any attempt to duplicate or
imitate any of the Content is prohibited under copyright law 2008.
On Mon, Jun 28, 2010 at 9:17 AM, Robert Kluin robert.kl...@gmail.comwrote:
The answer to this question depends on several other questions too.
For example:
1) how many items are you expecting?
2) how often do a users items change, and how do they get changed
(the user or something else)?
3) what type of querying / reporting do you need to be able to do?
4) what type of information would you need to store about each users item?
Robert
On Sun, Jun 27, 2010 at 9:59 PM, Tom Wu service.g2...@gmail.com wrote:
Hi Luís,
If you have a User model, with a items heavy Property like
TextProperty
which is not necessary for every query.
Which db structure is better ?
Best Regards
Tom Wu
2010/6/28 Luís Marques luismarq...@gmail.com
Hello Phil,
Anyone correct me if I'm wrong but you use one of the following
options, among others.
You can have a User model, with a items ListProperty. The pros: if
you have the key for the user (e.g. you can use the email), then with
a simple get (faster than a query) you can retrieve the entity
including the items. Writing a new object is transactional. The cons:
you must always get a set the entire user entity. The entity might get
large if there are a lot of items. If you use custom indexes with the
items property the index might get large quickly, especially if you
use more than one ListProperty (exploding indexes).
You can have a User model, and a UserItem whose parent is the
User, making it part of a transaction group. Cons: you have limited
QPS writes to the transaction group (about 1-10 QPS), but that's the
same limitation you have to a standalone User with items. You have to
query (seek) the items, although I wouldn't call that a problem
without profile data. Pros: you can write individual UserItems. You
can have transactions for the user and the items s/he has.
You can have a User model and a UserItem with a User key
reference. Cons: you can no longer have a transaction depending on the
user and the items s/he has. Pros: you can write individual items
without write collisions to the User.
You can have a User and a flat list (e.g. text, non-indexed) of the
corresponding items. Cons: limits the queries you can make, have to
update all items at once. Pros: better index performance.
Etc. I hope it was helpful and correct?
Best regards,
Luís
--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
You have to read thoroughly how to code for the one-to-many in bigtable implementation. Instead of fluffing all the content in one column, use that column to hold an object which fluffs the given objects into it. (Object of Objects, a list in python, a collection in Java.) When code, you can say, my_object.that_list. (After getting the my_object through GQL ) Which will give you no worry over queries per second and the likes. The whole operation would be atomic, as we should group the objects in subject with 'parent-child' concepts. But Phil, You may have to forget RDBMS if you wish to use Bigtable and vice-versa. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
( Personally I believe Google is on fire on Oracle which is the Information company for 30 years and is entirely newish in its Data design strategies. The objects will be created on the fly, as opposed to SQL, which takes a huge time on the name of normalization bullshit, and the modifications to table structures happen on the fly with expando, as opposed to RDBMS, which needs a Database Mgr's approval cycle in the industry, etc ) Business wise, one should be conscious of shifting technologies, and if people are more towards RDBMS, they should consider the Oracle's In- Memory Times Ten-series databases and force their RDBMS vendors to implement these on-the fly operations. Because precautions should be taken before the knowledge migration by the techies happen. Once Appengine folks release their SQL implementation and the RDBMS people ( Including Oracle ) migrate their data to appengine servers, You all lose the flexibility of data privacy and locations of your data servers. Instead of falling pray to that dependency, other RDBMS vendors must opt to implement their RDBMS in today's languages ( V V H L Languages ) right from the heart, so that one would be confident of supporting them over the following years. ( Inviting criticism, comments on my view. ) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
Hello Phil, Anyone correct me if I'm wrong but you use one of the following options, among others. You can have a User model, with a items ListProperty. The pros: if you have the key for the user (e.g. you can use the email), then with a simple get (faster than a query) you can retrieve the entity including the items. Writing a new object is transactional. The cons: you must always get a set the entire user entity. The entity might get large if there are a lot of items. If you use custom indexes with the items property the index might get large quickly, especially if you use more than one ListProperty (exploding indexes). You can have a User model, and a UserItem whose parent is the User, making it part of a transaction group. Cons: you have limited QPS writes to the transaction group (about 1-10 QPS), but that's the same limitation you have to a standalone User with items. You have to query (seek) the items, although I wouldn't call that a problem without profile data. Pros: you can write individual UserItems. You can have transactions for the user and the items s/he has. You can have a User model and a UserItem with a User key reference. Cons: you can no longer have a transaction depending on the user and the items s/he has. Pros: you can write individual items without write collisions to the User. You can have a User and a flat list (e.g. text, non-indexed) of the corresponding items. Cons: limits the queries you can make, have to update all items at once. Pros: better index performance. Etc. I hope it was helpful and correct? Best regards, Luís -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore design question
Hi Luís, If you have a User model, with a items heavy Property like TextProperty which is not necessary for every query. Which db structure is better ? Best Regards Tom Wu 2010/6/28 Luís Marques luismarq...@gmail.com Hello Phil, Anyone correct me if I'm wrong but you use one of the following options, among others. You can have a User model, with a items ListProperty. The pros: if you have the key for the user (e.g. you can use the email), then with a simple get (faster than a query) you can retrieve the entity including the items. Writing a new object is transactional. The cons: you must always get a set the entire user entity. The entity might get large if there are a lot of items. If you use custom indexes with the items property the index might get large quickly, especially if you use more than one ListProperty (exploding indexes). You can have a User model, and a UserItem whose parent is the User, making it part of a transaction group. Cons: you have limited QPS writes to the transaction group (about 1-10 QPS), but that's the same limitation you have to a standalone User with items. You have to query (seek) the items, although I wouldn't call that a problem without profile data. Pros: you can write individual UserItems. You can have transactions for the user and the items s/he has. You can have a User model and a UserItem with a User key reference. Cons: you can no longer have a transaction depending on the user and the items s/he has. Pros: you can write individual items without write collisions to the User. You can have a User and a flat list (e.g. text, non-indexed) of the corresponding items. Cons: limits the queries you can make, have to update all items at once. Pros: better index performance. Etc. I hope it was helpful and correct? Best regards, Luís -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: datastore design for hierarchical data.
I agree with this approach. It works well with tree data containing fewer than 1000 nodes. It's also the amount of entities that appengine can fetch in one query. Here's some example code which I'm using for a forum, it may help you get started: http://gist.github.com/370861 It's running at: http://www.caterpi.com/forum On Apr 18, 8:03 am, Tim Hoffman zutes...@gmail.com wrote: Here is the approach I would probably take unless you have many hundreds of child tasks. I would keep a list of keys of all children of each project/task. (You can then do a db.get and get all immediate children with single call) and each child has a reference to it's parent. In addition in each item keep a list of all parents keys back to the root. That way you can search for all children in part of the tree/subtree (This approach is especially useful if you have multiple child types) as back ref sets are kind specific. Keep a running count of complete children at each level, rather than trying to calculate the current state in a single big function, (keep these subsidiary values up to date through tasks) T On Apr 18, 8:29 pm, gops patelgo...@gmail.com wrote: hi , i am designing a simple project based to do list. the idea is to define tasks under project ( no workflow - just task is completed or not is required. ) in a hirarchial way. i.e. each task has multiple task and that task may have other multiple task. a project can be said to be completed if all task under that project are completed. , i tought of using refrenceproeperty to create hirarchy , but could not figure out easy way ( which do not take more than 30 seconds to find all the children of a project and check weather it is completed or not ) . to detect if project is complete or not. how to design database for such job ? and also , if i need to copy the project in order to define another project , how to copy hierarchical data ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: datastore design for hierarchical data.
one idea is to store both , reference value of project as well as reference value of parent task in a child task. ( or referencelist property ?? -- in that case , is it guranteed in appengine to preserve list sequence ?? ) On Apr 18, 5:29 pm, gops patelgo...@gmail.com wrote: hi , i am designing a simple project based to do list. the idea is to define tasks under project ( no workflow - just task is completed or not is required. ) in a hirarchial way. i.e. each task has multiple task and that task may have other multiple task. a project can be said to be completed if all task under that project are completed. , i tought of using refrenceproeperty to create hirarchy , but could not figure out easy way ( which do not take more than 30 seconds to find all the children of a project and check weather it is completed or not ) . to detect if project is complete or not. how to design database for such job ? and also , if i need to copy the project in order to define another project , how to copy hierarchical data ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: datastore design for hierarchical data.
Here is the approach I would probably take unless you have many hundreds of child tasks. I would keep a list of keys of all children of each project/task. (You can then do a db.get and get all immediate children with single call) and each child has a reference to it's parent. In addition in each item keep a list of all parents keys back to the root. That way you can search for all children in part of the tree/subtree (This approach is especially useful if you have multiple child types) as back ref sets are kind specific. Keep a running count of complete children at each level, rather than trying to calculate the current state in a single big function, (keep these subsidiary values up to date through tasks) T On Apr 18, 8:29 pm, gops patelgo...@gmail.com wrote: hi , i am designing a simple project based to do list. the idea is to define tasks under project ( no workflow - just task is completed or not is required. ) in a hirarchial way. i.e. each task has multiple task and that task may have other multiple task. a project can be said to be completed if all task under that project are completed. , i tought of using refrenceproeperty to create hirarchy , but could not figure out easy way ( which do not take more than 30 seconds to find all the children of a project and check weather it is completed or not ) . to detect if project is complete or not. how to design database for such job ? and also , if i need to copy the project in order to define another project , how to copy hierarchical data ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Design
Dear Mark, I think, if you want to get an answer clarifying your problem, you need to formulate your answer in a way that is better understandable. Instead of impying that your 'cluster around' is needed, perhaps you should ask if it is of any advantage to force Google's datastore to save data 'clustered' around something you call parent. Regards, Holger --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore Design
On Thu, Jul 30, 2009 at 3:34 AM, Mark Jones mark0...@gmail.com wrote: I'm using the AppEnginePatch to get a fairly useful Django implementation. The problem I've run into is that parent/child/ ancestor relationships aren't well supported. I've found a less than pretty workaround for the moment. The link to this discussion is http://groups.google.com/group/app-engine-patch/tree/browse_frm/thread/d1399dee0fe6bd08/1fcac449684ce388?rnum=1_done=%2Fgroup%2Fapp-engine-patch%2Fbrowse_frm%2Fthread%2Fd1399dee0fe6bd08%3F#doc_1fcac449684ce388 , but here are my design thoughts. Account: (Parent) one per user Item: (Child) 1000 or so per user I think that to do this optimally in the appengine with 100,000,000 rows and 100,000 users is to cluster the data around the account. So that the Account is the parent to the Items on a per user basis. Account is the parent/ancestor of Items that belong to that account. Is this just over design on my part? Will ReferenceProperties work just as well? There's no significant performance advantage to using parent entities. Athough the data is stored together, that doesn't translate to a significant performance increase when fetching them, and because writes to an Entity Group are serialized (to ensure transactional consistency), it can actually decrease write throughput. Only use entity groups if you need transactional consistency across multiple entities in the group. I'm thinking about this from the stand point of wanting to see all the Items from Sam's account. I don't want to search for Sam's items across 100,000,000 rows, I want to open up Sam's account by key, then search across the 1000 rows that belong to Sam. Assuming your query is indexed, finding items that belong to Sam and match some other criteria is no less efficient using ReferenceProperties than it would be using Parent relationships. -Nick Johnson Could someone that actually understands the insides of the Datastore tell me I'm right or wrong? -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore design
Just to clarify one point, 'IN' actully uses multiple queries under the hood, so a single 'IN' query does not really save you anything over doing the many queries yourself. Ie you still used a quota of 30 datastore calls. 2) i doubt is really going to scale as memcache might get flushed or entries removed at anypoint, so cant use it for data. Would still need to use the datastore to store such data. and would be doing it for all even if not needed which wastes resources. Without storeing the friend list yourself, think you are limited to 1) although you could use basically 2) to populate a StringList of friends in the datastore - just making sure to refresh it every 24 hours? 2009/6/9 Hugh hugh.phe...@gmail.com: I'm looking for some high level ideas on how to design a datastore for a Facebook application. In this app a user can collect widgets. I'd like to be able to show a user lists of the widgets owned by their friends - something like: q = db.GqlQuery(SELECT * FROM Widget WHERE owner IN ( friendlist ) and other filter criteria ) The problem with this approach is that friendlist can only have 30 entries. Since the Facebook TOS don't allow you to store the friend list (for more than 24 hours anyway), it seems like the options here are pretty limited. Some of the ideas I've been considering are: 1) divide into 30 friend chunks make (potentially) lots of datastore calls. 2) Combine #1 with some background processing (cron or queues when available) that do the datastore calls populate memcahed with the results. 3) Use a clever design that allows you to formulate the query in such a way that you don't hit 30 entry limit on in clauses Any thoughts? -- Barry - www.nearby.org.uk - www.geograph.org.uk - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
