Make that January 6th, 2010. :) - Jason
On Jan 14, 5:30 pm, "Jason (Google)" <apija...@google.com> wrote: > Last Wednesday, the App Engine team hosted the latest session of its > bimonthly IRC office hours. A transcript of the session and a summary > of the topics covered is provided below. The next session will take > place next Wednesday, January 20th from 9:00-10:00 a.m. PST in the > #appengine channel on irc.freenode.net. > > - Jason > > --SUMMARY----------------------------------------------------------- > - Q: Is there any work underway to speed up the index building > process? A: Indexes do not take a set amount of time to build; rather, > the building duration is dependent in part on the amount of traffic > hitting the cluster and the number of other applications that are > building indexes during the same period. But we're always making > optimizations where we can to make it as efficient as possible. > [7:01-7:02, 7:05-7:07, 7:10-7:11, 7:22, 7:24, 7:26, 7:29-7:30, 7:32] > > - Q: What is the best way to do aggregate reporting on App Engine > data? A: Trying to retrieve and then store all data in memory is not a > scalable approach -- it may work for small datasets but will increase > latency and start to fail as the size of the data grows. Ultimately > App Engine's datastore is not built for this type of data mining. If > you can't find any reasonable way to split the data into smaller > chunks which can be processed individually and aggregated using the > task queue API, you may consider downloading the data using the bulk > download utility and doing the data mining and report generation on > your local box. [7:13-7:20] > > - Q: What should be done in order to ensure an application performs > well when it scales from several users to many hundreds or thousands? > A: If you expect your application to receive a lot of requests, you > should first perform a load test with the expected amount of traffic > before deploying; all billing-enabled apps can handle up to 500 QPS > and you can request higher quota if necessary. You can determine how > well your application will function by using a load testing utility to > ramp up traffic from 0 to the expected level and hold this level for a > period. Inspect your "Number of Quota Denials/Second" graph (should be > a steady 0) and "Milliseconds/Request" (latency) graph to guage > whether any changes need to be made before you go live. [7:25-7:27, > 7:30] > > - Q: In lieu of a true multi-tenant solution, what's the best way to > model data for an application that will be used by multiple clients/ > customers? A: This depends on the specific needs of your customers and > the size of your application. If you want to use a single application > ID, you can either separate your kinds by client (e.g. > Users_customer1, Users_customer2) or just use a single kind (Users) > with an extra property indicating which client that entity is > associated with. For larger applications, you may consider deploying a > new application per client, but first you'll need to request an > exemption to clause 4.4 of the App Engine TOS using the form > athttp://code.google.com/support/bin/request.py?contact_type=AppEngineM.... > [7:32, 7:34-7:35, 7:39-7:40] > > - Discussion on HTTPS support for non-appspot.com domains [7:37, 7:39, > 7:43-7:44] > > - Q: What's the best way to extend the list of accepted property > types, e.g. adding a decimal data type? A: Check these > resources:http://googleappengine.blogspot.com/2009/07/writing-custom-property-c... > andhttp://www.redredred.com.au/money-database-property-for-google-app-en... > [7:45, 7:49-7:50] > > - Q: What kind of write load can App Engine handle? A: You should > model your data so as to avoid writing to the same entity or entity > group more than once per second on average, but the write throughput > to distinct entity groups is high enough that most developers > shouldn't need to worry. For more on avoiding datastore contention > which can limit write throughput, > seehttp://code.google.com/appengine/articles/scaling/contention.html > [7:50, 7:52, 7:54-7:55, 7:58, 8:00-8:01] > > - Discussion of the "preview release" designation; note that apart > from the "preview release" status, App Engine has graduated from > Google Code Labs and thus any kind of formal turndown wouldn't take > place until 3 years past any deprecation announcement. Therefore, > should App Engine be deprecated in the future, which isn't planned, > hosted apps won't suddenly be disabled. [7:50-7:54] > > --FULL TRANSCRIPT--------------------------------------------------- > [7:00pm] daskalou: Let's begin... > [7:01pm] Jason_Google: Hi Everyone. Welcome to the first Chat Time of > the new year! I'll be here for the next hour to answer any App Engine > questions. > [7:01pm] daskalou: Do I have the correct time? Is it 7PM PST now? > [7:01pm] daskalou: sweet > [7:01pm] Jason_Google: Yes indeed. > [7:01pm] daskalou: Q: Any plans on speeding up index building? > [7:02pm] Jason_Google: Index building is always something we're > working on, optimizing the work of the index task workers and so > forth. I don't know of any specific measures being done right now, > only that optimizations are added every few releases or so that should > help. What kind of index building delays are you experiencing? > [7:05pm] daskalou: Currently, I am not, I've just been reading all the > complaints about it > [7:06pm] daskalou: Q: Can we get a more defined window of time that an > app is kept in the app cache? > [7:06pm] chilts: Jason_Google: you know how Google Labs Short Link > service has a config page in the Google Apps interface, is there any > way we can do that with an AppEngine service? > [7:07pm] chilts: > ie.https://www.google.com/a/cpanel/example.org/LabServiceSettings?appId=... > [7:07pm] Jason_Google: Currently, as far as I know, index building is > dependent on the size of the data of the apps with indexes also being > built, so indexes do occasionally take longer to build when larger > apps are also having their indexes built -- there's only a limited > number of index workers that run. But like I said, we're continually > looking into ways of optimizing. > [7:07pm] ryan_google: daskalou: good q. unfortunately no, not really. > it depends on overall cluster-wide traffic weather and patterns; it's > not a fixed period of time. > [7:08pm] SnapABug: Q: Is there a way using the XMPP service to send > XMPP states such as <COMPOSING/>, <PAUSED/>, <INACTIVE/> etc... > [7:09pm] SnapABug: (using Java) > [7:09pm] ryan_google: SnapABug: i don't know that there are any xmpp > experts here, but i expect the answer is no > [7:10pm] Jason_Google: chilts: I haven't looked into the Google Labs > Short Link service actually. I need to explore it a bit more before I > can answer your question fully. But can you go into more detail about > what you're looking for, e.g. a way of creating short links for App > Engine apps running on a certain Google Apps domain? > [7:10pm] Wesley_google: also as i understand it, it's about how many > "units of work" are required to build your indices, regardless of > size. if you use up your internal "quota" of resources necessary to > build all your indices, then it will "pause" until there is some > quota available > [7:10pm] SnapABug: OK. I could only find ways to update the <BODY>. > That's what I was afraid of. > [7:10pm] SnapABug: Thanks > [7:10pm] ryan_google: SnapABug: feel free to file a feature request! > [7:10pm] SnapABug: Will do. > [7:10pm] chilts: Jason_Google: yep, I can see that I can configure my > AppEngine app at but it's only the web address and > 'Disable':https://www.google.com/a/cpanel/chilts.org/AppEngineServiceSettings?a... > [7:11pm] chilts: I was wondering if I can do other config there ... > e.g. what colour do you want your CSS (or proper admin questions) > [7:11pm] Wesley_google: this will make it *seem* like it's taking a > long time to build your indices but in reality, they're paused on > purpose > [7:12pm] chilts: for example it has a link to further settings on the > app itself > [7:12pm] Jason_Google: chilts: I see. I believe the customization > options are limited for App Engine apps but this is something we're > looking to improve significantly in the coming year. Please file a > feature request for specific options you'd like to see and we'll > forward those along to the proper teams. > [7:12pm] stephan__: newbie q: what's the protocol here? ask away, or > get in line somehow? > [7:12pm] chilts: hmm, maybe I just do that and have the settings in > the app itself > [7:12pm] chilts: Jason_Google: thanks ... I will > [7:12pm] Jason_Google: stephan__: Ask away! > [7:13pm] chilts: I guess I meant app config and maybe I just do that > in the app itself ... thought there was more to the Google Labs ones > [7:13pm] stephan__: ok then... here's my catch 22 - i need to do > aggregate reporting on data. queries don't support aggregate clauses, > so my solution is to pull things into memory and query using josql. > [7:13pm] Jason_Google: chilts: Currently, I believe you have to do it > in the app itself, but we may be able to add more options down the > line, so your specific ideas may help there. > [7:14pm] chilts: awesome > [7:14pm] ryan_google: stephan__: the short answer is that the app > engine datastore isn't designed for reporting, data mining, business > intelligence, ad hoc queries, etc > [7:14pm] ryan_google: pulling data into memory will work but only with > small datasets > [7:15pm] stephan__: so far so good. but.. (1) because of the 30 second > limit, i need to do this in batches -- querying the data store in > sets. but.. (2) sometimes GAE will spin up a new JVM, meaning > everything i've build up in memory is no longer there, so i have to > start over. > [7:15pm] ryan_google: right. basically, don't expect much success with > that approach. > [7:15pm] ryan_google: generally we'd encourage you to d/l the data > with eg. the bulk loader, put it into a data warehouse, and do the > data mining and reports there > [7:15pm] stephan__: all i would need, however, is a way to keep a > user's session sticky to a given jvm > [7:16pm] ryan_google: right. one approach would be to let users "kick > off" a report, add a task to a task queue, split up the work into > manageable chunks, give each chunk to a task, then collect the results > and compile them and mark the report as ready to view > [7:17pm] ryan_google: and provide a reports page that shows in > progress and completed reports > [7:17pm] stephan__: but i still need for all the data to be in the > memory of one jvm at some point. if i use task queus, that's a sure > fire way to cause a lot more jvms to spin up > [7:18pm] ryan_google: true. again, you'd need to be able to split up > the data into chunks. if you really can't do that, honestly your only > option is probably to d/l it all and process it offline. > [7:18pm] ryan_google: i'd be surprised if you couldn't shard it > though, mapreduce style > [7:19pm] ryan_google: btw we're working on support for aggregate data > processing. it'll be mapreduce style, not sql query style, but it'll > still be something. > [7:19pm] stephan__: perhaps. are their any plans to support any sort > of batch mode where a given process can run for more than 30 seconds? > [7:19pm] stephan__: i'd pay for this, as i'm sure others would as well > [7:19pm] ryan_google: that'd help you, but soon after that you'd run > into the memory limit > [7:20pm] ryan_google: honestly for most nontrivial datasets, you'll > hit the limit of memory on a given machine pretty quickly > [7:20pm] salsakran: ryan: is there a timeframe for map/reduce queries > on the datastore? > [7:20pm] ryan_google: even if you use your own machine > [7:20pm] stephan__: ya, i'm slowly coming to that conclusion, > unfortunately. disapointing however... > [7:21pm] ryan_google: salsakran: sorry, no. ETAs for software more > than, say, 6 months out tend to be kind of meaningless anyway (imho). > we tend to use agile processes, so we have a rough idea, but nothing > very precise or far out. > [7:21pm] salsakran: heh. so > 6 months? > [7:21pm] Wesley_google: "maybe" > [7:22pm] ryan_google: salsakran: we don't know. > [7:22pm] stephan__: fwiw, i'm a product manager at salesforce.com, so > i understand why these limits need to be enforced to "protect the > server" for everyone else. still, you should be thinking about some > kind of server poll for batch operations in a way that wouldn't impact > other people. as i said, this is something i'm sure people would pay > for. > [7:22pm] stephan__: server POOL i meant > [7:22pm] ryan_google: it's on the roadmap, > though,http://code.google.com/appengine/docs/roadmap.html, which is a strong > sign that we're prioritizing it and hope to get it out soon. > [7:22pm] daskalou: Wesley_google: re: index building and using up your > "index building quota" - how big is this quota? > [7:23pm] salsakran: ryan_google: ahh ... so that's what "Support for > mapping operations across datasets" meant > [7:23pm] ryan_google: stephan: we definitely are! it just sounds like > it will be a very different model from what you're asking for, where > you load the entire data set into memory. we make sure that anything > we build scales, and that doesn't really. > [7:23pm] Wesley_google: stephan__: can't you do the same thing with > force.com? > [7:24pm] Wesley_google: daskalou: this is an internal number that i'm > not aware of, but just know that there is a built-in limitation in > there, so if things seem like they're taking too long, they may be > paused > [7:24pm] SnapABug: Q: We experienced a lot of Datastore Timeout > Exceptions impacting our customers today. > [7:24pm] SnapABug: The status page was at "investigating" earlier but > is all green now. Do you have an explanation for the Query latency > problem? > [7:25pm] ryan_google: SAB: when? > [7:25pm] stephan__: haha -- i'm a freaking product manager on the > platform. i started a contract gig before i started there that i did > on GAE. interesting, i chose GAE over SF because of portability > concerns -- i know that if it came to it i could take my java code and > run it on a rackspace server if need be. SF seemed like lockin. that > said, i'm now thinking of maybe moving the reporting part to SF, as > the force.com is a lot more mature in that re > [7:25pm] salsakran: what's the current protocol for making sure an app > doesn't get throttled when launched to the public? going from 3-5 devs > hitting it to opening it up to the masses? > [7:26pm] > ryan_google:http://code.google.com/status/appengine/detail/datastore/2010/01/06#a... > does show spikes, but occasional spikes that aren't too high or > widespread do happen > [7:26pm] ryan_google: salsakran: what do you mean by throttled? > [7:26pm] SnapABug: ryan_ggole: starting 12 hours ago. > [7:26pm] daskalou: Wesley_google: If I add 100 new indexes but have an > empty Datastore, my index building times will still have to "wait in > line" due to other apps' indexes being built, even though there my app > has no data in the Datastore? > [7:27pm] SnapABug: The problem peaked about 4 hours ago. > [7:27pm] salsakran: ryan_google: hiting quota and/or having requests > dropped from a sudden spike in traffic > [7:27pm] ryan_google: SAB: how long did they last? > [7:27pm] Jason_Google: salsakran: You're more than welcome to run a > load test before you formally launch your app. > [7:27pm] ryan_google: ah > [7:27pm] salsakran: ryan_google: I recall there being reports of this > before, just wondered what the state of that was > [7:27pm] SnapABug: most of 9 hours. > [7:28pm] ryan_google: SAB: i don't have an answer for your app > specifically. we are working on a few things that will further > increase datastore isolation, though, which could help with things > like this > [7:28pm] ryan_google: do you know roughly what error rate you're > seeing? > [7:28pm] SnapABug: You can see it there > too:http://code.google.com/status/appengine/detail/datastore/2010/01/06#a... > [7:29pm] SnapABug: The error is: Uncaught exception from servlet > com.google.appengine.api.datastore.DatastoreTimeoutException > [7:29pm] ryan_google: fair enough > [7:29pm] ryan_google: i don't have an answer right now, but i'll look > into it > [7:29pm] Wesley_google: daskalou: i don't believe that your index- > building has to "wait in line" behind other apps. your building > requires the same amount of "units of work" regardless of other apps > [7:29pm] SnapABug: ryan_google: Thank you. It is much appreciated. > [7:29pm] Wesley_google: if your datastore is empty, i would imagine > that it shouldn't take long to index your "data" > [7:30pm] ryan_google: salsakran: load testing is always a good idea. > also, you generally want to enable billing if you're at all worried > about quota > [7:30pm] daskalou: Q: When do you guys expect to release (1) Mapping > operations support, and (2) Cursor support? (I know they're on the > roadmap, just wondering "how close" you guys) > [7:30pm] ryan_google: wes is half right. index building isn't strictly > a queue, but it is affected by how many other indices are building at > the same time > [7:30pm] Jason_Google: salsakran: In general, if you're concerned > about a high-scale launch, you should a) enable billing for your app > which automatically increases several key quotas and b) do a load test > at the projected level of QPS to make sure your latency numbers stay > reasonable. This has been a model that other apps have followed with > success in the past. > [7:30pm] daskalou: are* > [7:31pm] daskalou: Wesley_google: ok, thanks > [7:31pm] ryan_google: daskalou: cursors, in either the next release or > the one after that. mapping, see earlier in the conversation. the > answer is we don't know. > [7:31pm] Jason_Google: daskalou: Currently, cursor support is slated > for the next release. You can see references to it in the Javadocs, > etc. > [7:31pm] Jason_Google: Things could always change of course, but it's > coming up soon. > [7:31pm] daskalou: when is the "next release"? > [7:32pm] Wesley_google: yes, ryan, that's what i meant! jason > mentioned earlier that there are index "workers" and altho there isn't > a queue, their number is finite > [7:32pm] rcopenjr: Hey Guys, I'm new to GAE and appreciate your time > here. I have a quick question as I begin to > [7:32pm] rcopenjr: design models for the datastore (Python). I'm > building an application that will host multiple customer > [7:32pm] rcopenjr: who I will need to be able to store data for. For > example I will have a contact model that will > [7:32pm] rcopenjr: contain person information that will be segregated > by company - Each company owns their own > [7:32pm] rcopenjr: data. QUESTION: Given performance constraints is > it better to create separate models for each > [7:32pm] rcopenjr: new customer, i.e. contact_company1, > contact_company2, or one contact model with a property > [7:32pm] rcopenjr: for company????? > [7:32pm] ryan_google: daskalou: soon, probably within a month, but we > don't provide exact ETAs for releases > [7:32pm] salsakran: jason_google: thanks ... that's exactly what I > wanted to know > [7:32pm] daskalou: ryan_google: cool > [7:32pm] Jason_Google: daskalou: Hopefully before the end of the > month. We're shooting for quick releases (usually at least one per > month) if you've been following the release history. > [7:34pm] ryan_google: rcopenjr: we'd need more info. data modelling > q's tend to be somewhat involved and don't work as well for developer > chats like these. there are lots of similar discussions on the google > group, though. you might consider search the archives and/or post > there. > [7:34pm] andywocky: rcopenjr: are trying to emulate multiple databases > on Datastore? Is that even possible? > [7:35pm] Jason_Google: rcopenjr: There are several models you can > follow depending on your goals. For larger applications, you may want > to have one app ID per customer. For smaller apps, you can get away > with using one kind per customer or even using a single kind and > storing the customer's ID as a property which you can query on. > Segregating the data is easiest when it comes to querying and data > modeling, but harder for app maintenance. > [7:36pm] chilts: whilst we're in the thick of the chat, I'd like to > say thanks to all you Googlers for the BlobStore ... it's awesome ... > nice and simple > [7:36pm] chilts: easy to make it work (Python) > [7:36pm] rcopenjr: Thanks Jason.... > [7:37pm] SnapABug: One last Q: Do you plan on providing a way to > redirect https traffic from something > likehttps://www.snapabug.comtohttps://snapabug.appspot.com? > [7:37pm] Jason_Google: chilts: Glad you're finding it useful. We'll be > making it even better in forthcoming releases. > [7:37pm] SnapABug: (in this case, we already have the CNAMEwww.snapabug.com > mapped ghs.google.com, but only http is handled) > [7:37pm] Wesley_google: chilts: glad you're enjoying it! > [7:37pm] Jason_Google: SnapABug: Hopefully, we'll be able to start > supporting HTTPS on non-appspot.com domains before it comes to that > point. For now, you have to manage this redirect yourself. > [7:39pm] silassewell: Q: I was wondering if there were any plans to > allow the use of the blobstore via the fetch service? > [7:39pm] SnapABug: HTTPS support would be awesome. > [7:39pm] rcopenjr: Jason_Google: A follow up question - is there a > determining factor for a large application - > [7:39pm] SnapABug: Is it on the roadmap? > [7:39pm] rcopenjr: number of entities, kinds, etc. > [7:40pm] ryan_google: rcopenjr: determining factor for performance? ie > latency, scalability, ...? > [7:40pm] daskalou: Q: Besides for transactions, is there any other > reason why we should put entities into entity groups? > [7:40pm] ryan_google: the short answer is no, number of entities and > kinds generally don't affect performance of individual entities. > [7:40pm] ryan_google: daskalou: the short answer is no > [7:40pm] ryan_google: er, sorry, performance of individual > *operations* > [7:40pm] rcopenjr: ryan_google: performance > [7:40pm] rcopenjr: Thanks..... > [7:42pm] andywocky: Q: I'm confused about the lifecycle of a handler. > If I map two endpoints in app.yaml, say to handler1.py and > handler2.py, are those handlers ever persisted in memory or cached? > For instance, will each handler run (including imports) for each > request, or less frequently? > [7:43pm] Jason_Google: SnapABug: Not yet. I'm not close enough to this > piece to comment with any authority, but I believe that support for > HTTPS on non-appspot.com domains, when it happens, requires us to take > advantage of browser support for SNI, so older browsers will > necessarily be left behind here. This is one reason why we haven't > enabled this yet -- we're still waiting for broader adoption. But we > recognize that a lot of developers want this, so we're looking into > prioritizing it higher. > [7:44pm] Wesley_google: SAB: here's more regarding what jason was > saying about non-appspot HTTS support and > SNI:http://googleappengine.blogspot.com/2008/10/announcing-https-support-... > [7:44pm] SnapABug: Jason_Google: Thanks for the detailed answer. Will > be looking forward for this feature release. > [7:44pm] Jason_Google: silassewell: Can you elaborate on your request? > I probably can't confirm one way or the other, but I can follow up for > you. Also, feel free to file feature requests in the issue tracker so > other developers can vote them > up:http://code.google.com/p/googleappengine/issues/list > [7:45pm] ryan_google: andywocky: have you > readhttp://code.google.com/appengine/docs/python/runtime.html#App_Caching > ? > [7:45pm] Wesley_google: (look towards the end of the blogpost) > [7:45pm] daskalou: Q: What's the best way to handle putting a decimal > data type into the Datastore? > [7:46pm] SnapABug: Wesley_Google: OK. Cool. > [7:46pm] Jason_Google: daskalou: There's actually a thread about this > in the groups, I believe. Nick Johnson wrote something about it. Let > me try to find it... > [7:47pm] silassewell: Jason_Google: I created a reverse proxy which > sites on app engine. Right now the fetch service and datastore is > currently limited to 1mb, so allowing fetch to retrieve objects larger > than 1mb and storing them in the blobstore would be nice (I'm also > assuming you do some optimizations when serving blobstore objects > which may be more efficient than datastore + memcache). > [7:47pm] enigmus_: Q: Are there news > onhttp://code.google.com/p/googleappengine/issues/detail?id=2481 > ? "Index inconsistency: queries not returning all results" > [7:47pm] ryan_google: enigmus: one of my pet projects! > [7:47pm] ryan_google: we hate that > [7:48pm] daskalou: Jason_Google: thanks > [7:48pm] ryan_google: we (the datastore team) have been pushing hard > on that, and we're making solid progress. i can't say that we've fully > fixed it, but we think we're close. > [7:48pm] ryan_google: one thing to note is that we're focusing on > finding and eliminating the root causes first, since if we just go > through and fix the data without fixing those bugs, they'll just cause > more inconsistency > [7:49pm] ryan_google: but naturally we fully expect to fix both the bug > (s) and the existing data > [7:49pm] Jason_Google: daskalou: So far, I've found > this:http://googleappengine.blogspot.com/2009/07/writing-custom-property-c... > [7:49pm] enigmus_: ryan_google: ok, thanks > [7:49pm] Brandon\wcr: I've been sort of out of it, have cursors been > implemented yet? > [7:50pm] Jason_Google: Brandon\wcr: Coming up in the next release or > two. > [7:50pm] daskalou: Jason_Google: thanks, I found this > too:http://www.redredred.com.au/money-database-property-for-google-app-en... > [7:50pm] salsakran: what kind of write load into the datastore is > possible by an app? > [7:50pm] Jason_Google: daskalou: Yep, that links to the same blog > post. > [7:50pm] daskalou: Q: When do you expect GAE to come out of "preview > release"? > [7:50pm] daskalou: Jason_Google: lol, cool > [7:51pm] ryan_google: daskalou: we'd need to define the difference > between preview and non-preview first. > [7:51pm] andywocky: ryan_google: thanks for the pointer. Am I > interpreting correctly that imports are cached, but that imports for > each handler (handler1 vs handler2) are independent? > [7:51pm] andywocky: ... meaning that handler1 has its own namespace, > separate from handler2? > [7:51pm] ryan_google: out of curiosity, does it mean something > specific to you? for example, do you have stakeholders who won't use a > "preview" product because of that term specifically? > [7:52pm] daskalou: ryan_google: What do you expect the difference to > be? > [7:52pm] Brandon\wcr: Q: Is there any timeline for task queue quota > increases? > [7:52pm] daskalou: ryan_google: Yes, that's pretty much it > [7:52pm] Jason_Google: salsakran: You can write data pretty quickly, > but you'll run into problems when you try to write the same entity or > write to the same entity group too quickly. > [7:52pm] chilts: Q. one concern of mine is if Google ever decide to > pull the plug on AppEngine? I guess there are no guarantees this will > never happen? > [7:52pm] salsakran: specifically: if we're writing scrobbler type > application, and split by user_id and shard the counters in question, > what sort of throughput is reasonable to hope for? > [7:52pm] chilts: I mean, if we have all these things running on it, > then it goes away, that'll be sad > [7:53pm] Jason_Google: Brandon\wcr: Which quotas specifically? > [7:53pm] daskalou: chilts: Good question > [7:53pm] Wesley_google: daskalou: your post does point to nick's > blogpost but the blogger in this case focused on a limitation of using > strings to store numeric values with and provided his own solution > [7:53pm] Brandon\wcr: Jason_Google: oops! nevermind, just saw it was > increased already to 1mill > [7:53pm] ryan_google: daskalou: it's basically for managing > expectations, as you'd expect. re your stakeholders, that's sad. i > guess i'd encourage you to point them to actual data (status site for > uptime, over 100k apps and 250k developers, more than 250M pageviews > per day, apps in the high 10s of millions of users, etc.) > [7:54pm] salsakran: do you happen to know what kind of write > throughput I can expect per entity? > [7:54pm] Jason_Google: chilts: App Engine isn't in labs, so we do have > a formal policy in place to support all apps running on it for a > certain period, 3 years I think, from any announcement of deprecation, > which isn't planned. > [7:54pm] ryan_google: chilts: naturally we don't expect that. you're > right though that it's a risk with almost anything you use that's run > by someone else. > [7:54pm] ryan_google: aha, jason has a much more specific answer. > ignore me. > [7:54pm] chilts: awesome > [7:54pm] chilts: 3yrs sounds plenty in that event > [7:55pm] ryan_google: salsakran: one answer is, don't expect more than > 1-10qps of writes per entity group > [7:55pm] chilts: it's good to know these things, esp. if you base a > business around someone elses > [7:55pm] ryan_google: definitely! > [7:55pm] daskalou: Wesley_google and ryan_google: thanks > [7:56pm] ryan_google: np! glad to help > [7:56pm] Wesley_google: same here... glad to help users out!! > [7:56pm] andywocky: ryan_google: is my assumption about cached imports > in handlers correct? > [7:56pm] chilts: yeah, thanks guys (I figure these chats take it out > of you and make you tired) > [7:56pm] SnapABug: Yep. Thank you guys! > [7:56pm] ryan_google: andywocky: right, sorry. i'm not an expert on > the python runtime so i'm thinking about it. > [7:56pm] Wesley_google: i'm also open to taking any Python-specific > questions too for those of you who are newbies > [7:57pm] daskalou: Yeah Python newbie here > [7:57pm] daskalou: Wesley_google: Do you have experience with AEP? > [7:57pm] Wesley_google: speaking of which, today is the last day for > earlybird registration to the largest Python conference in the world > happening next month in ATL > [7:57pm] chilts: Wesley_google: I had an interesting one the other day > with BlobStore ... but I think that might be a question for the group > (since I'll need a few points to describe it) > [7:57pm] ryan_google: andywocky: when you say independent namespaces, > do you mean that the global vars are kept separate? > [7:58pm] Wesley_google: daskalou: nope... what's AEP? > [7:58pm] ryan_google: actually, regardless, it's probably over my > head. wesley might know, otherwise you could try the google-appengine- > python group > [7:58pm] Jason_Google: salsakran: Did Ryan answer your question? > Writing a lot of data isn't a problem, only when you're writing too > much to the same entity group.Some strategies about how to avoid > this:http://code.google.com/appengine/articles/scaling/contention.html > [7:58pm] salsakran: jason_google: yeah ... the 1-10 q/s was a useful > number > [7:58pm] Wesley_google: andywocky: each handler has its own namespace, > just like independent modules in "regular" Python > [7:59pm] andywocky: wesley_google: thanks! > [7:59pm] Wesley_google: sure > [7:59pm] daskalou: Wesley_google: app-engine-patch - Django with > support for admin and other things the built in Django doesn't support > [7:59pm] chilts: Jason_Google: so my thoughts there are, if I have > maybe 100 entities (no particular entity group), writing them all > should be no problem (even if they land on the same datastore server)? > [7:59pm] Wesley_google: ah, i figured as much > [8:00pm] Jason_Google: salsakran: Yeah, it varies, but you will likely > see some level of timeouts if you start writing more than 1 QPS to a > given entity group with any consistency. For that reason, we recommend > you keep your entities as small as possible and for items like > counters where you expect to update every request, you can shard your > data using a technique mentioned in that article. > [8:00pm] chilts: this is a theoretical thought (I don't have 100 > entities to write) > [8:00pm] Wesley_google: unfortunately when i worked on the Django book > back in 2008, only the Django Helper for App Engine was out. AEP just > came out when our book was published, so i didn't get a chance to > experiment with it > [8:00pm] daskalou: Wesley_google: Re: my thread on the > groups:http://groups.google.com/group/google-appengine/browse_thread/thread/... > Do you know if it's difficult to "share" models between the two > frameworks? > [8:01pm] Jason_Google: chilts: Yes, this is right. There does come a > point (e.g. several thousand QPS) when there are other considerations, > but in general, you should be able to write many entities in separate > entity groups in parallel. > [8:01pm] chilts: sweet > [8:02pm] chilts: I must admit, sometimes I wonder about how much > hardware you guys have there, and whilst you might not be at liberty > to say, a blog post would be awesome (how many memcache instances, > datastore servers, frontends etc) > [8:02pm] chilts: would make an interesting post > [8:02pm] Wesley_google: daskalou: yes, i think i skimmed your post. i > don't know the answer but my intuition is that it isn't cached > [8:02pm] Wesley_google: or rather, shared > [8:03pm] daskalou: Wesley_google: oh... damn > [8:03pm] Jason_Google: Alright then, the office hour is officially > over. Feel free to keep the discussion going though. We'll be back in > two weeks, Wednesday, January 20th, from 9:00 to 10:00 a.m. PST, and > I'll post a transcript of this session to the groups as soon as I can. > Thanks for the company. > [8:03pm] ryan_google: chilts: agreed! it's actually probably a lot > smaller than you think...but still, yeah, don't hold your breath. > [8:03pm] chilts: thanks Jason_Google > [8:03pm] chilts: ryan_google: heh, awesome > [8:03pm] chilts: but interesting
-- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
