I think your missing the point - you dont need a million rows if s fry reads a book - you don't send that data to every follower. you need one row and then 200 string properties listing his followers 200x5000 so that they get the activity he read the book. Fan out does not mean you send the same message to all receivers by dding a row of the same message. it means you store the index of each user who owns the message and store the message once.
I still think what you are doing is no different to a social app - your model is simple - follow someone and we will tell you what book they are reading or have read. We will also make suggestions as to what books you might like based on that data - "its an activity stream" and as such when modeled that way its simple to build and scale for 100 or 1 million users - twitter, facebook all do the same thing - not with books with people and photos and so on. this solution also gives you the activity stream which may be useful to users in your app - like s.fry has started reading.... Of course that's optional functionality you can add or not add later when you become the next amazon - my point is the pipeline and data is in place for that to happen all you will do is dream new ways of showing the data to your users in a way they will find useful as your model progresses. As i said in previous posts the activity stream makes it easy to query what you need as s fry will have 1 million users - but its the - actor - me that will be asking the question and to me s fry is just one person i am following making the query a simple big table read - its e-relevant to me and in this model that their are another million me's following s fry. Further more s fry himself will be just another me following probably 10 people. Don't be turned away by a solution that is historically modeled around social sites - your business model is and I'm guessing selling books - but your model is to socialise them around the books your users read - a social connection - consider using the social model rather than trying to re-invent the wheel or attempt to invent a square one. I hope this is helpful :-) Regards Martin Webb The information contained in this email is confidential and may contain proprietary information. It is meant solely for the intended recipient. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted in reliance on this, is prohibited and may be unlawful. No liability or responsibility is accepted if information or data is, for whatever reason corrupted or does not reach its intended recipient. No warranty is given that this email is free of viruses. The views expressed in this email are, unless otherwise stated, those of the author ________________________________ From: stefoid <stevenmath...@yahoo.com> To: Google App Engine <google-appengine@googlegroups.com> Sent: Fri, 9 July, 2010 2:34:47 Subject: [google-appengine] Re: list-property, many to many, 5000 indexes...HELP! Hi, actually I think you and Martin are on the same page, more or less. The size of the table you suggest is kind of scary (think 10 million users, each with 50 friends, each having read 50 books = 25 billion rows) And if super-star user who has been befriended a million times reads a book, thats a million rows that need to be added at the click of a button). And Im not even sure if it is technically possible for datastore. This table has two multi-valued properties for filtering: tags (0-4) and friends (0-?) and a property for sorting (numfriends). So I guess it needs some kind of composite index across those three properties, which, to keep from exploding, would have to have friends capped in the hundreds. can anyone confirm that, or is ti technically impossible for other reasons ? cheers On Jul 8, 2:20 pm, Stephen Johnson <onepagewo...@gmail.com> wrote: > Personally, I would try attacking this problem from a different > direction. I'm assuming that if Sally is John's friend then John is > also Sally's friend. So, let's suppose that Sally (key S1) reads the > book Great Expectations (key E1) and has friends John (key J1) and > Mike (key M1). I would add a table that looks something like: > > userKey > tag > bookKey > numFriends > friendsList > > and add an index on userKey, tag, bookKey, numFriends > > So, when Sally indicates she has read Great Expectations (E1). I would > add (or modify if entries for these users/book combinations already > exist) two entries to this table (one entry for each of her friends), > so it would look like: > > userKey tag bookKey numFriends friendsList > --------------------------------------------------------------- > J1 historical+drama E1 1 Sally > M1 historical+drama M1 1 Sally > > Now let's say Ken (K1) is a friend of John's but not Mike's and he > reads Great Expections, so table would look like: > > userKey tag bookKey numFriends friendsList > --------------------------------------------------------------- > J1 historical+drama E1 2 S1,K1 > M1 historical+drama M1 1 S1 > > And finally, let's say Betty (B1) is a friend of Johns and reads Moby > Dick (D1) so table would be: > > userKey tag bookKey numFriends friendsList > --------------------------------------------------------------- > J1 historical+drama E1 2 S1,K1 > J1 historical+drama D1 1 B1 > M1 historical+drama M1 1 S1 > > Now, querying this table should be fast and efficient along with > sorted properly. > > Well, this is the way I'd do it. Hope it helps. > Steve -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
