[google-appengine] Re: Datastore design question
On Jun 29, 8:41 am, Phil wrote: > It sounds like the list property can't be appended to without a > transaction? If so, it seems it'll be better to create two tables, > one for Users and one for Items. The list property can be appended without an explicit transaction. It just might not be wise, since between a read of the entity and the write with the appended list another request can update the entity, and that update would be lost. The entity write itself is atomic though. You won't have some of the properties written from the original request and some written from the other concurrent request. So just be careful of race conditions, if you can be sure there aren't concurrent requests or that they are idempotent then I guess you should be OK without a transaction. > Just to answer the above questions... I expect many reads of the data > and few writes. I also expect the normal case to have few items (1-2 > on avg) and it won't change often. The key to the Items table would > be user_id-item_name. Are prefix scans in GQL efficient? Also, in > terms of data I'm storing about the items, I have ~5-10 fields of > metadata. If you know the key then that's the fastest way to retrieve the entity, instead of using a query. I don't know the Java syntax by heart, but it's easy to find. > For now I'm assuming that the right thing to do here is to split the > data between tables. Even if it seems to be an unlikely problem now, > I don't want to put myself in a situation where I'm limited by the QPS > that transactions can support. Someone more experienced might be able to elucidate, but on your scenario I don't know what you'd gain with that, other than smaller RPC requests for writing to the user items (at least performance-wise, but it might be a cleaner design). If you have to make a transaction then the User and UserItems must be part of a transaction group (items have the user as a parent), so the transaction locks out the entire group anyway. That's the same as locking out a group made of a single entity, a user with a list of items property. See if you can avoid transactions with your design, if you ever run into write performance problems. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
It sounds like the list property can't be appended to without a transaction? If so, it seems it'll be better to create two tables, one for Users and one for Items. Just to answer the above questions... I expect many reads of the data and few writes. I also expect the normal case to have few items (1-2 on avg) and it won't change often. The key to the Items table would be user_id-item_name. Are prefix scans in GQL efficient? Also, in terms of data I'm storing about the items, I have ~5-10 fields of metadata. For now I'm assuming that the right thing to do here is to split the data between tables. Even if it seems to be an unlikely problem now, I don't want to put myself in a situation where I'm limited by the QPS that transactions can support. Thanks, Phil On Jun 27, 11:17 pm, Robert Kluin wrote: > The answer to this question depends on several other questions too. > For example: > 1) how many items are you expecting? > 2) how often do a users items change, and how do they get changed > (the user or something else)? > 3) what type of querying / reporting do you need to be able to do? > 4) what type of information would you need to store about each users item? > > Robert > > > > On Sun, Jun 27, 2010 at 9:59 PM, Tom Wu wrote: > > Hi Luís, > > > If you have a "User" model, with a "items" heavy Property like TextProperty > > which is not necessary for every query. > > Which db structure is better ? > > > Best Regards > > Tom Wu > > > 2010/6/28 Luís Marques > > >> Hello Phil, > > >> Anyone correct me if I'm wrong but you use one of the following > >> options, among others. > > >> You can have a "User" model, with a "items" ListProperty. The pros: if > >> you have the key for the user (e.g. you can use the email), then with > >> a simple get (faster than a query) you can retrieve the entity > >> including the items. Writing a new object is transactional. The cons: > >> you must always get a set the entire user entity. The entity might get > >> large if there are a lot of items. If you use custom indexes with the > >> "items" property the index might get large quickly, especially if you > >> use more than one ListProperty (exploding indexes). > > >> You can have a "User" model, and a "UserItem" whose parent is the > >> User, making it part of a transaction group. Cons: you have limited > >> QPS writes to the transaction group (about 1-10 QPS), but that's the > >> same limitation you have to a standalone User with items. You have to > >> query (seek) the items, although I wouldn't call that a problem > >> without profile data. Pros: you can write individual UserItems. You > >> can have transactions for the user and the items s/he has. > > >> You can have a "User" model and a "UserItem" with a User key > >> reference. Cons: you can no longer have a transaction depending on the > >> user and the items s/he has. Pros: you can write individual items > >> without write collisions to the User. > > >> You can have a "User" and a flat list (e.g. text, non-indexed) of the > >> corresponding items. Cons: limits the queries you can make, have to > >> update all items at once. Pros: better index performance. > > >> Etc. I hope it was helpful and correct? > > >> Best regards, > >> Luís > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to google-appeng...@googlegroups.com. > >> To unsubscribe from this group, send email to > >> google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
( Personally I believe Google is on fire on Oracle which is the Information company for 30 years and is entirely newish in its Data design strategies. The objects will be created on the fly, as opposed to SQL, which takes a huge time on the name of normalization bullshit, and the modifications to table structures happen on the fly with expando, as opposed to RDBMS, which needs a Database Mgr's approval cycle in the industry, etc ) Business wise, one should be conscious of shifting technologies, and if people are more towards RDBMS, they should consider the Oracle's In- Memory Times Ten-series databases and force their RDBMS vendors to implement these on-the fly operations. Because precautions should be taken before the knowledge migration by the techies happen. Once Appengine folks release their SQL implementation and the RDBMS people ( Including Oracle ) migrate their data to appengine servers, You all lose the flexibility of data privacy and locations of your data servers. Instead of falling pray to that dependency, other RDBMS vendors must opt to implement their RDBMS in today's languages ( V V H L Languages ) right from the heart, so that one would be confident of supporting them over the following years. ( Inviting criticism, comments on my view. ) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
You have to read thoroughly how to code for the one-to-many in bigtable implementation. Instead of fluffing all the content in one column, use that column to hold an object which fluffs the given objects into it. (Object of Objects, a list in python, a collection in Java.) When code, you can say, "my_object.that_list". (After getting the my_object through GQL ) Which will give you no worry over queries per second and the likes. The whole operation would be atomic, as we should group the objects in subject with 'parent-child' concepts. But Phil, You may have to forget RDBMS if you wish to use Bigtable and vice-versa. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore design question
i got alot of user meta data, i save them on another db model called
usermeta which is referencekey to the user model key. under the usermeta i
write duplicated data from the user table too such as user key id as string
and much more. i usually pull the usermeta only and use the user model just
for authentication keeping it clean and to the purpose.
-
Cheers,
public static function AlonCarmel() {
//Contact me
var email = 'a...@aloncarmel.me';
var twitter = '@aloncarmel';
var web = 'http://aloncarmel.me';
var phone = '+972-54-4860380';
}
* If you received an unsolicited email from by mistake that wasn't of your
matter please delete immediately. All E-mail sent from Alon Carmel is
copyrighted to Alon Carmel 2008. Any details revealed in e-mails sent by
Alon Carmel are owned by the Author only. Any attempt to duplicate or
imitate any of the Content is prohibited under copyright law 2008.
On Mon, Jun 28, 2010 at 9:17 AM, Robert Kluin wrote:
> The answer to this question depends on several other questions too.
> For example:
> 1) how many items are you expecting?
> 2) how often do a users items change, and how do they get changed
> (the user or something else)?
> 3) what type of querying / reporting do you need to be able to do?
> 4) what type of information would you need to store about each users item?
>
> Robert
>
>
>
>
>
>
>
> On Sun, Jun 27, 2010 at 9:59 PM, Tom Wu wrote:
> > Hi Luís,
> >
> > If you have a "User" model, with a "items" heavy Property like
> TextProperty
> > which is not necessary for every query.
> > Which db structure is better ?
> >
> > Best Regards
> > Tom Wu
> >
> >
> > 2010/6/28 Luís Marques
> >>
> >> Hello Phil,
> >>
> >> Anyone correct me if I'm wrong but you use one of the following
> >> options, among others.
> >>
> >> You can have a "User" model, with a "items" ListProperty. The pros: if
> >> you have the key for the user (e.g. you can use the email), then with
> >> a simple get (faster than a query) you can retrieve the entity
> >> including the items. Writing a new object is transactional. The cons:
> >> you must always get a set the entire user entity. The entity might get
> >> large if there are a lot of items. If you use custom indexes with the
> >> "items" property the index might get large quickly, especially if you
> >> use more than one ListProperty (exploding indexes).
> >>
> >> You can have a "User" model, and a "UserItem" whose parent is the
> >> User, making it part of a transaction group. Cons: you have limited
> >> QPS writes to the transaction group (about 1-10 QPS), but that's the
> >> same limitation you have to a standalone User with items. You have to
> >> query (seek) the items, although I wouldn't call that a problem
> >> without profile data. Pros: you can write individual UserItems. You
> >> can have transactions for the user and the items s/he has.
> >>
> >> You can have a "User" model and a "UserItem" with a User key
> >> reference. Cons: you can no longer have a transaction depending on the
> >> user and the items s/he has. Pros: you can write individual items
> >> without write collisions to the User.
> >>
> >> You can have a "User" and a flat list (e.g. text, non-indexed) of the
> >> corresponding items. Cons: limits the queries you can make, have to
> >> update all items at once. Pros: better index performance.
> >>
> >> Etc. I hope it was helpful and correct?
> >>
> >> Best regards,
> >> Luís
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appeng...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com
> .
> >> For more options, visit this group at
> >> http://groups.google.com/group/google-appengine?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appeng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com
> .
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore design question
The answer to this question depends on several other questions too. For example: 1) how many items are you expecting? 2) how often do a users items change, and how do they get changed (the user or something else)? 3) what type of querying / reporting do you need to be able to do? 4) what type of information would you need to store about each users item? Robert On Sun, Jun 27, 2010 at 9:59 PM, Tom Wu wrote: > Hi Luís, > > If you have a "User" model, with a "items" heavy Property like TextProperty > which is not necessary for every query. > Which db structure is better ? > > Best Regards > Tom Wu > > > 2010/6/28 Luís Marques >> >> Hello Phil, >> >> Anyone correct me if I'm wrong but you use one of the following >> options, among others. >> >> You can have a "User" model, with a "items" ListProperty. The pros: if >> you have the key for the user (e.g. you can use the email), then with >> a simple get (faster than a query) you can retrieve the entity >> including the items. Writing a new object is transactional. The cons: >> you must always get a set the entire user entity. The entity might get >> large if there are a lot of items. If you use custom indexes with the >> "items" property the index might get large quickly, especially if you >> use more than one ListProperty (exploding indexes). >> >> You can have a "User" model, and a "UserItem" whose parent is the >> User, making it part of a transaction group. Cons: you have limited >> QPS writes to the transaction group (about 1-10 QPS), but that's the >> same limitation you have to a standalone User with items. You have to >> query (seek) the items, although I wouldn't call that a problem >> without profile data. Pros: you can write individual UserItems. You >> can have transactions for the user and the items s/he has. >> >> You can have a "User" model and a "UserItem" with a User key >> reference. Cons: you can no longer have a transaction depending on the >> user and the items s/he has. Pros: you can write individual items >> without write collisions to the User. >> >> You can have a "User" and a flat list (e.g. text, non-indexed) of the >> corresponding items. Cons: limits the queries you can make, have to >> update all items at once. Pros: better index performance. >> >> Etc. I hope it was helpful and correct? >> >> Best regards, >> Luís >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore design question
Hi Luís, If you have a "User" model, with a "items" heavy Property like TextProperty which is not necessary for every query. Which db structure is better ? Best Regards Tom Wu 2010/6/28 Luís Marques > Hello Phil, > > Anyone correct me if I'm wrong but you use one of the following > options, among others. > > You can have a "User" model, with a "items" ListProperty. The pros: if > you have the key for the user (e.g. you can use the email), then with > a simple get (faster than a query) you can retrieve the entity > including the items. Writing a new object is transactional. The cons: > you must always get a set the entire user entity. The entity might get > large if there are a lot of items. If you use custom indexes with the > "items" property the index might get large quickly, especially if you > use more than one ListProperty (exploding indexes). > > You can have a "User" model, and a "UserItem" whose parent is the > User, making it part of a transaction group. Cons: you have limited > QPS writes to the transaction group (about 1-10 QPS), but that's the > same limitation you have to a standalone User with items. You have to > query (seek) the items, although I wouldn't call that a problem > without profile data. Pros: you can write individual UserItems. You > can have transactions for the user and the items s/he has. > > You can have a "User" model and a "UserItem" with a User key > reference. Cons: you can no longer have a transaction depending on the > user and the items s/he has. Pros: you can write individual items > without write collisions to the User. > > You can have a "User" and a flat list (e.g. text, non-indexed) of the > corresponding items. Cons: limits the queries you can make, have to > update all items at once. Pros: better index performance. > > Etc. I hope it was helpful and correct? > > Best regards, > Luís > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore design question
Hello Phil, Anyone correct me if I'm wrong but you use one of the following options, among others. You can have a "User" model, with a "items" ListProperty. The pros: if you have the key for the user (e.g. you can use the email), then with a simple get (faster than a query) you can retrieve the entity including the items. Writing a new object is transactional. The cons: you must always get a set the entire user entity. The entity might get large if there are a lot of items. If you use custom indexes with the "items" property the index might get large quickly, especially if you use more than one ListProperty (exploding indexes). You can have a "User" model, and a "UserItem" whose parent is the User, making it part of a transaction group. Cons: you have limited QPS writes to the transaction group (about 1-10 QPS), but that's the same limitation you have to a standalone User with items. You have to query (seek) the items, although I wouldn't call that a problem without profile data. Pros: you can write individual UserItems. You can have transactions for the user and the items s/he has. You can have a "User" model and a "UserItem" with a User key reference. Cons: you can no longer have a transaction depending on the user and the items s/he has. Pros: you can write individual items without write collisions to the User. You can have a "User" and a flat list (e.g. text, non-indexed) of the corresponding items. Cons: limits the queries you can make, have to update all items at once. Pros: better index performance. Etc. I hope it was helpful and correct? Best regards, Luís -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: datastore design for hierarchical data.
I agree with this approach. It works well with tree data containing fewer than 1000 nodes. It's also the amount of entities that appengine can fetch in one query. Here's some example code which I'm using for a forum, it may help you get started: http://gist.github.com/370861 It's running at: http://www.caterpi.com/forum On Apr 18, 8:03 am, Tim Hoffman wrote: > Here is the approach I would probably take unless you have many > hundreds of child tasks. > > I would keep a list of keys of all children of each project/task. > (You can then do a db.get and get all immediate children with single > call) > and each child has a reference to it's parent. In addition in each > item keep a list of all parents keys back to the root. That way you > can search for all children > in part of the tree/subtree > > (This approach is especially useful if you have multiple child types) > as back ref sets > are kind specific. > > Keep a running count of complete children at each level, rather than > trying to calculate the current state in a single big function, (keep > these subsidiary values up to date through tasks) > > T > > On Apr 18, 8:29 pm, gops wrote: > > > > > hi , i am designing a simple project based to do list. the idea is to > > define tasks under project ( no workflow - just "task is completed" or > > not is required. ) in a hirarchial way. i.e. each task has multiple > > task and that task may have other multiple task. a project can be said > > to be completed if all task under that project are completed. , i > > tought of using refrenceproeperty to create hirarchy , but could not > > figure out easy way ( which do not take more than 30 seconds to find > > all the children of a project and check weather it is completed or > > not ) . to detect if project is complete or not. how to design > > database for such job ? and also , if i need to copy the project in > > order to define another project , how to copy hierarchical data ? > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: datastore design for hierarchical data.
Here is the approach I would probably take unless you have many hundreds of child tasks. I would keep a list of keys of all children of each project/task. (You can then do a db.get and get all immediate children with single call) and each child has a reference to it's parent. In addition in each item keep a list of all parents keys back to the root. That way you can search for all children in part of the tree/subtree (This approach is especially useful if you have multiple child types) as back ref sets are kind specific. Keep a running count of complete children at each level, rather than trying to calculate the current state in a single big function, (keep these subsidiary values up to date through tasks) T On Apr 18, 8:29 pm, gops wrote: > hi , i am designing a simple project based to do list. the idea is to > define tasks under project ( no workflow - just "task is completed" or > not is required. ) in a hirarchial way. i.e. each task has multiple > task and that task may have other multiple task. a project can be said > to be completed if all task under that project are completed. , i > tought of using refrenceproeperty to create hirarchy , but could not > figure out easy way ( which do not take more than 30 seconds to find > all the children of a project and check weather it is completed or > not ) . to detect if project is complete or not. how to design > database for such job ? and also , if i need to copy the project in > order to define another project , how to copy hierarchical data ? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: datastore design for hierarchical data.
one idea is to store both , reference value of project as well as reference value of parent task in a child task. ( or referencelist property ?? -- in that case , is it guranteed in appengine to preserve list sequence ?? ) On Apr 18, 5:29 pm, gops wrote: > hi , i am designing a simple project based to do list. the idea is to > define tasks under project ( no workflow - just "task is completed" or > not is required. ) in a hirarchial way. i.e. each task has multiple > task and that task may have other multiple task. a project can be said > to be completed if all task under that project are completed. , i > tought of using refrenceproeperty to create hirarchy , but could not > figure out easy way ( which do not take more than 30 seconds to find > all the children of a project and check weather it is completed or > not ) . to detect if project is complete or not. how to design > database for such job ? and also , if i need to copy the project in > order to define another project , how to copy hierarchical data ? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Design
On Thu, Jul 30, 2009 at 3:34 AM, Mark Jones wrote: > > I'm using the AppEnginePatch to get a fairly useful Django > implementation. The problem I've run into is that parent/child/ > ancestor relationships aren't well supported. I've found a less than > pretty workaround for the moment. > > The link to this discussion is > > http://groups.google.com/group/app-engine-patch/tree/browse_frm/thread/d1399dee0fe6bd08/1fcac449684ce388?rnum=1&_done=%2Fgroup%2Fapp-engine-patch%2Fbrowse_frm%2Fthread%2Fd1399dee0fe6bd08%3F#doc_1fcac449684ce388 > , > but here are my design thoughts. > > Account: (Parent) one per user > Item: (Child) 1000 or so per user > > I think that to do this optimally in the appengine with 100,000,000 > rows and 100,000 users is to cluster the data around the account. So > that the Account is the parent to the Items on a per user basis. > Account is the parent/ancestor of Items that belong to that account. > > Is this just over design on my part? Will ReferenceProperties work > just as well? There's no significant performance advantage to using parent entities. Athough the data is stored together, that doesn't translate to a significant performance increase when fetching them, and because writes to an Entity Group are serialized (to ensure transactional consistency), it can actually decrease write throughput. Only use entity groups if you need transactional consistency across multiple entities in the group. > > > I'm thinking about this from the stand point of wanting to see all the > Items from Sam's account. I don't want to search for Sam's items > across 100,000,000 rows, I want to open up Sam's account by key, then > search across the 1000 rows that belong to Sam. Assuming your query is indexed, finding items that belong to Sam and match some other criteria is no less efficient using ReferenceProperties than it would be using Parent relationships. -Nick Johnson > > > Could someone that actually understands the insides of the Datastore > tell me I'm right or wrong? > > > > > -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore Design
Dear Mark, I think, if you want to get an answer clarifying your problem, you need to formulate your answer in a way that is better understandable. Instead of impying that your 'cluster around' is needed, perhaps you should ask if it is of any advantage to force Google's datastore to save data 'clustered' around something you call parent. Regards, Holger --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore design
Just to clarify one point, 'IN' actully uses multiple queries under
the hood, so a single 'IN' query does not really save you anything
over doing the many queries yourself. Ie you still used a quota of 30
datastore calls.
2) i doubt is really going to scale as memcache might get flushed or
entries removed at anypoint, so cant use it for data. Would still need
to use the datastore to store such data. and would be doing it for all
even if not needed which wastes resources.
Without storeing the friend list yourself, think you are limited to 1)
although you could use basically 2) to populate a StringList of
friends in the datastore - just making sure to refresh it every 24
hours?
2009/6/9 Hugh :
>
> I'm looking for some high level ideas on how to design a datastore for
> a Facebook application.
>
> In this app a user can collect widgets.
> I'd like to be able to show a user lists of the widgets owned by their
> friends - something like:
>
> q = db.GqlQuery("SELECT * FROM Widget WHERE owner IN ( )
> and ")
>
> The problem with this approach is that friendlist can only have 30
> entries.
>
> Since the Facebook TOS don't allow you to store the friend list (for
> more than 24 hours anyway), it seems like the options here are pretty
> limited.
>
> Some of the ideas I've been considering are:
> 1) divide into 30 friend chunks & make (potentially) lots of datastore
> calls.
> 2) Combine #1 with some background processing (cron or queues when
> available) that do the datastore calls &
> populate memcahed with the results.
> 3) Use a clever design that allows you to formulate the query in such
> a way that you don't hit 30 entry limit on "in" clauses
>
> Any thoughts?
> >
>
--
Barry
- www.nearby.org.uk - www.geograph.org.uk -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
