Last Wednesday, the App Engine team hosted the latest session of its bimonthly IRC office hours. A transcript of the session and a summary of the topics covered is provided below. The next session will take place next Wednesday, January 20th from 9:00-10:00 a.m. PST in the #appengine channel on irc.freenode.net.
- Jason --SUMMARY----------------------------------------------------------- - Q: Is there any work underway to speed up the index building process? A: Indexes do not take a set amount of time to build; rather, the building duration is dependent in part on the amount of traffic hitting the cluster and the number of other applications that are building indexes during the same period. But we're always making optimizations where we can to make it as efficient as possible. [7:01-7:02, 7:05-7:07, 7:10-7:11, 7:22, 7:24, 7:26, 7:29-7:30, 7:32] - Q: What is the best way to do aggregate reporting on App Engine data? A: Trying to retrieve and then store all data in memory is not a scalable approach -- it may work for small datasets but will increase latency and start to fail as the size of the data grows. Ultimately App Engine's datastore is not built for this type of data mining. If you can't find any reasonable way to split the data into smaller chunks which can be processed individually and aggregated using the task queue API, you may consider downloading the data using the bulk download utility and doing the data mining and report generation on your local box. [7:13-7:20] - Q: What should be done in order to ensure an application performs well when it scales from several users to many hundreds or thousands? A: If you expect your application to receive a lot of requests, you should first perform a load test with the expected amount of traffic before deploying; all billing-enabled apps can handle up to 500 QPS and you can request higher quota if necessary. You can determine how well your application will function by using a load testing utility to ramp up traffic from 0 to the expected level and hold this level for a period. Inspect your "Number of Quota Denials/Second" graph (should be a steady 0) and "Milliseconds/Request" (latency) graph to guage whether any changes need to be made before you go live. [7:25-7:27, 7:30] - Q: In lieu of a true multi-tenant solution, what's the best way to model data for an application that will be used by multiple clients/ customers? A: This depends on the specific needs of your customers and the size of your application. If you want to use a single application ID, you can either separate your kinds by client (e.g. Users_customer1, Users_customer2) or just use a single kind (Users) with an extra property indicating which client that entity is associated with. For larger applications, you may consider deploying a new application per client, but first you'll need to request an exemption to clause 4.4 of the App Engine TOS using the form at http://code.google.com/support/bin/request.py?contact_type=AppEngineMultiInstanceExceptionRequest. [7:32, 7:34-7:35, 7:39-7:40] - Discussion on HTTPS support for non-appspot.com domains [7:37, 7:39, 7:43-7:44] - Q: What's the best way to extend the list of accepted property types, e.g. adding a decimal data type? A: Check these resources: http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html and http://www.redredred.com.au/money-database-property-for-google-app-engine/ [7:45, 7:49-7:50] - Q: What kind of write load can App Engine handle? A: You should model your data so as to avoid writing to the same entity or entity group more than once per second on average, but the write throughput to distinct entity groups is high enough that most developers shouldn't need to worry. For more on avoiding datastore contention which can limit write throughput, see http://code.google.com/appengine/articles/scaling/contention.html [7:50, 7:52, 7:54-7:55, 7:58, 8:00-8:01] - Discussion of the "preview release" designation; note that apart from the "preview release" status, App Engine has graduated from Google Code Labs and thus any kind of formal turndown wouldn't take place until 3 years past any deprecation announcement. Therefore, should App Engine be deprecated in the future, which isn't planned, hosted apps won't suddenly be disabled. [7:50-7:54] --FULL TRANSCRIPT--------------------------------------------------- [7:00pm] daskalou: Let's begin... [7:01pm] Jason_Google: Hi Everyone. Welcome to the first Chat Time of the new year! I'll be here for the next hour to answer any App Engine questions. [7:01pm] daskalou: Do I have the correct time? Is it 7PM PST now? [7:01pm] daskalou: sweet [7:01pm] Jason_Google: Yes indeed. [7:01pm] daskalou: Q: Any plans on speeding up index building? [7:02pm] Jason_Google: Index building is always something we're working on, optimizing the work of the index task workers and so forth. I don't know of any specific measures being done right now, only that optimizations are added every few releases or so that should help. What kind of index building delays are you experiencing? [7:05pm] daskalou: Currently, I am not, I've just been reading all the complaints about it [7:06pm] daskalou: Q: Can we get a more defined window of time that an app is kept in the app cache? [7:06pm] chilts: Jason_Google: you know how Google Labs Short Link service has a config page in the Google Apps interface, is there any way we can do that with an AppEngine service? [7:07pm] chilts: ie. https://www.google.com/a/cpanel/example.org/LabServiceSettings?appId=shortlinks [7:07pm] Jason_Google: Currently, as far as I know, index building is dependent on the size of the data of the apps with indexes also being built, so indexes do occasionally take longer to build when larger apps are also having their indexes built -- there's only a limited number of index workers that run. But like I said, we're continually looking into ways of optimizing. [7:07pm] ryan_google: daskalou: good q. unfortunately no, not really. it depends on overall cluster-wide traffic weather and patterns; it's not a fixed period of time. [7:08pm] SnapABug: Q: Is there a way using the XMPP service to send XMPP states such as <COMPOSING/>, <PAUSED/>, <INACTIVE/> etc... [7:09pm] SnapABug: (using Java) [7:09pm] ryan_google: SnapABug: i don't know that there are any xmpp experts here, but i expect the answer is no [7:10pm] Jason_Google: chilts: I haven't looked into the Google Labs Short Link service actually. I need to explore it a bit more before I can answer your question fully. But can you go into more detail about what you're looking for, e.g. a way of creating short links for App Engine apps running on a certain Google Apps domain? [7:10pm] Wesley_google: also as i understand it, it's about how many "units of work" are required to build your indices, regardless of size. if you use up your internal "quota" of resources necessary to build all your indices, then it will "pause" until there is some quota available [7:10pm] SnapABug: OK. I could only find ways to update the <BODY>. That's what I was afraid of. [7:10pm] SnapABug: Thanks [7:10pm] ryan_google: SnapABug: feel free to file a feature request! [7:10pm] SnapABug: Will do. [7:10pm] chilts: Jason_Google: yep, I can see that I can configure my AppEngine app at but it's only the web address and 'Disable': https://www.google.com/a/cpanel/chilts.org/AppEngineServiceSettings?appId=website-chilts-org [7:11pm] chilts: I was wondering if I can do other config there ... e.g. what colour do you want your CSS (or proper admin questions) [7:11pm] Wesley_google: this will make it *seem* like it's taking a long time to build your indices but in reality, they're paused on purpose [7:12pm] chilts: for example it has a link to further settings on the app itself [7:12pm] Jason_Google: chilts: I see. I believe the customization options are limited for App Engine apps but this is something we're looking to improve significantly in the coming year. Please file a feature request for specific options you'd like to see and we'll forward those along to the proper teams. [7:12pm] stephan__: newbie q: what's the protocol here? ask away, or get in line somehow? [7:12pm] chilts: hmm, maybe I just do that and have the settings in the app itself [7:12pm] chilts: Jason_Google: thanks ... I will [7:12pm] Jason_Google: stephan__: Ask away! [7:13pm] chilts: I guess I meant app config and maybe I just do that in the app itself ... thought there was more to the Google Labs ones [7:13pm] stephan__: ok then... here's my catch 22 - i need to do aggregate reporting on data. queries don't support aggregate clauses, so my solution is to pull things into memory and query using josql. [7:13pm] Jason_Google: chilts: Currently, I believe you have to do it in the app itself, but we may be able to add more options down the line, so your specific ideas may help there. [7:14pm] chilts: awesome [7:14pm] ryan_google: stephan__: the short answer is that the app engine datastore isn't designed for reporting, data mining, business intelligence, ad hoc queries, etc [7:14pm] ryan_google: pulling data into memory will work but only with small datasets [7:15pm] stephan__: so far so good. but.. (1) because of the 30 second limit, i need to do this in batches -- querying the data store in sets. but.. (2) sometimes GAE will spin up a new JVM, meaning everything i've build up in memory is no longer there, so i have to start over. [7:15pm] ryan_google: right. basically, don't expect much success with that approach. [7:15pm] ryan_google: generally we'd encourage you to d/l the data with eg. the bulk loader, put it into a data warehouse, and do the data mining and reports there [7:15pm] stephan__: all i would need, however, is a way to keep a user's session sticky to a given jvm [7:16pm] ryan_google: right. one approach would be to let users "kick off" a report, add a task to a task queue, split up the work into manageable chunks, give each chunk to a task, then collect the results and compile them and mark the report as ready to view [7:17pm] ryan_google: and provide a reports page that shows in progress and completed reports [7:17pm] stephan__: but i still need for all the data to be in the memory of one jvm at some point. if i use task queus, that's a sure fire way to cause a lot more jvms to spin up [7:18pm] ryan_google: true. again, you'd need to be able to split up the data into chunks. if you really can't do that, honestly your only option is probably to d/l it all and process it offline. [7:18pm] ryan_google: i'd be surprised if you couldn't shard it though, mapreduce style [7:19pm] ryan_google: btw we're working on support for aggregate data processing. it'll be mapreduce style, not sql query style, but it'll still be something. [7:19pm] stephan__: perhaps. are their any plans to support any sort of batch mode where a given process can run for more than 30 seconds? [7:19pm] stephan__: i'd pay for this, as i'm sure others would as well [7:19pm] ryan_google: that'd help you, but soon after that you'd run into the memory limit [7:20pm] ryan_google: honestly for most nontrivial datasets, you'll hit the limit of memory on a given machine pretty quickly [7:20pm] salsakran: ryan: is there a timeframe for map/reduce queries on the datastore? [7:20pm] ryan_google: even if you use your own machine [7:20pm] stephan__: ya, i'm slowly coming to that conclusion, unfortunately. disapointing however... [7:21pm] ryan_google: salsakran: sorry, no. ETAs for software more than, say, 6 months out tend to be kind of meaningless anyway (imho). we tend to use agile processes, so we have a rough idea, but nothing very precise or far out. [7:21pm] salsakran: heh. so > 6 months? [7:21pm] Wesley_google: "maybe" [7:22pm] ryan_google: salsakran: we don't know. [7:22pm] stephan__: fwiw, i'm a product manager at salesforce.com, so i understand why these limits need to be enforced to "protect the server" for everyone else. still, you should be thinking about some kind of server poll for batch operations in a way that wouldn't impact other people. as i said, this is something i'm sure people would pay for. [7:22pm] stephan__: server POOL i meant [7:22pm] ryan_google: it's on the roadmap, though, http://code.google.com/appengine/docs/roadmap.html , which is a strong sign that we're prioritizing it and hope to get it out soon. [7:22pm] daskalou: Wesley_google: re: index building and using up your "index building quota" - how big is this quota? [7:23pm] salsakran: ryan_google: ahh ... so that's what "Support for mapping operations across datasets" meant [7:23pm] ryan_google: stephan: we definitely are! it just sounds like it will be a very different model from what you're asking for, where you load the entire data set into memory. we make sure that anything we build scales, and that doesn't really. [7:23pm] Wesley_google: stephan__: can't you do the same thing with force.com? [7:24pm] Wesley_google: daskalou: this is an internal number that i'm not aware of, but just know that there is a built-in limitation in there, so if things seem like they're taking too long, they may be paused [7:24pm] SnapABug: Q: We experienced a lot of Datastore Timeout Exceptions impacting our customers today. [7:24pm] SnapABug: The status page was at "investigating" earlier but is all green now. Do you have an explanation for the Query latency problem? [7:25pm] ryan_google: SAB: when? [7:25pm] stephan__: haha -- i'm a freaking product manager on the platform. i started a contract gig before i started there that i did on GAE. interesting, i chose GAE over SF because of portability concerns -- i know that if it came to it i could take my java code and run it on a rackspace server if need be. SF seemed like lockin. that said, i'm now thinking of maybe moving the reporting part to SF, as the force.com is a lot more mature in that re [7:25pm] salsakran: what's the current protocol for making sure an app doesn't get throttled when launched to the public? going from 3-5 devs hitting it to opening it up to the masses? [7:26pm] ryan_google: http://code.google.com/status/appengine/detail/datastore/2010/01/06#ae-trust-detail-datastore-get-latency does show spikes, but occasional spikes that aren't too high or widespread do happen [7:26pm] ryan_google: salsakran: what do you mean by throttled? [7:26pm] SnapABug: ryan_ggole: starting 12 hours ago. [7:26pm] daskalou: Wesley_google: If I add 100 new indexes but have an empty Datastore, my index building times will still have to "wait in line" due to other apps' indexes being built, even though there my app has no data in the Datastore? [7:27pm] SnapABug: The problem peaked about 4 hours ago. [7:27pm] salsakran: ryan_google: hiting quota and/or having requests dropped from a sudden spike in traffic [7:27pm] ryan_google: SAB: how long did they last? [7:27pm] Jason_Google: salsakran: You're more than welcome to run a load test before you formally launch your app. [7:27pm] ryan_google: ah [7:27pm] salsakran: ryan_google: I recall there being reports of this before, just wondered what the state of that was [7:27pm] SnapABug: most of 9 hours. [7:28pm] ryan_google: SAB: i don't have an answer for your app specifically. we are working on a few things that will further increase datastore isolation, though, which could help with things like this [7:28pm] ryan_google: do you know roughly what error rate you're seeing? [7:28pm] SnapABug: You can see it there too: http://code.google.com/status/appengine/detail/datastore/2010/01/06#ae-trust-detail-datastore-query-latency [7:29pm] SnapABug: The error is: Uncaught exception from servlet com.google.appengine.api.datastore.DatastoreTimeoutException [7:29pm] ryan_google: fair enough [7:29pm] ryan_google: i don't have an answer right now, but i'll look into it [7:29pm] Wesley_google: daskalou: i don't believe that your index- building has to "wait in line" behind other apps. your building requires the same amount of "units of work" regardless of other apps [7:29pm] SnapABug: ryan_google: Thank you. It is much appreciated. [7:29pm] Wesley_google: if your datastore is empty, i would imagine that it shouldn't take long to index your "data" [7:30pm] ryan_google: salsakran: load testing is always a good idea. also, you generally want to enable billing if you're at all worried about quota [7:30pm] daskalou: Q: When do you guys expect to release (1) Mapping operations support, and (2) Cursor support? (I know they're on the roadmap, just wondering "how close" you guys) [7:30pm] ryan_google: wes is half right. index building isn't strictly a queue, but it is affected by how many other indices are building at the same time [7:30pm] Jason_Google: salsakran: In general, if you're concerned about a high-scale launch, you should a) enable billing for your app which automatically increases several key quotas and b) do a load test at the projected level of QPS to make sure your latency numbers stay reasonable. This has been a model that other apps have followed with success in the past. [7:30pm] daskalou: are* [7:31pm] daskalou: Wesley_google: ok, thanks [7:31pm] ryan_google: daskalou: cursors, in either the next release or the one after that. mapping, see earlier in the conversation. the answer is we don't know. [7:31pm] Jason_Google: daskalou: Currently, cursor support is slated for the next release. You can see references to it in the Javadocs, etc. [7:31pm] Jason_Google: Things could always change of course, but it's coming up soon. [7:31pm] daskalou: when is the "next release"? [7:32pm] Wesley_google: yes, ryan, that's what i meant! jason mentioned earlier that there are index "workers" and altho there isn't a queue, their number is finite [7:32pm] rcopenjr: Hey Guys, I'm new to GAE and appreciate your time here. I have a quick question as I begin to [7:32pm] rcopenjr: design models for the datastore (Python). I'm building an application that will host multiple customer [7:32pm] rcopenjr: who I will need to be able to store data for. For example I will have a contact model that will [7:32pm] rcopenjr: contain person information that will be segregated by company - Each company owns their own [7:32pm] rcopenjr: data. QUESTION: Given performance constraints is it better to create separate models for each [7:32pm] rcopenjr: new customer, i.e. contact_company1, contact_company2, or one contact model with a property [7:32pm] rcopenjr: for company????? [7:32pm] ryan_google: daskalou: soon, probably within a month, but we don't provide exact ETAs for releases [7:32pm] salsakran: jason_google: thanks ... that's exactly what I wanted to know [7:32pm] daskalou: ryan_google: cool [7:32pm] Jason_Google: daskalou: Hopefully before the end of the month. We're shooting for quick releases (usually at least one per month) if you've been following the release history. [7:34pm] ryan_google: rcopenjr: we'd need more info. data modelling q's tend to be somewhat involved and don't work as well for developer chats like these. there are lots of similar discussions on the google group, though. you might consider search the archives and/or post there. [7:34pm] andywocky: rcopenjr: are trying to emulate multiple databases on Datastore? Is that even possible? [7:35pm] Jason_Google: rcopenjr: There are several models you can follow depending on your goals. For larger applications, you may want to have one app ID per customer. For smaller apps, you can get away with using one kind per customer or even using a single kind and storing the customer's ID as a property which you can query on. Segregating the data is easiest when it comes to querying and data modeling, but harder for app maintenance. [7:36pm] chilts: whilst we're in the thick of the chat, I'd like to say thanks to all you Googlers for the BlobStore ... it's awesome ... nice and simple [7:36pm] chilts: easy to make it work (Python) [7:36pm] rcopenjr: Thanks Jason.... [7:37pm] SnapABug: One last Q: Do you plan on providing a way to redirect https traffic from something like https://www.snapabug.com to https://snapabug.appspot.com ? [7:37pm] Jason_Google: chilts: Glad you're finding it useful. We'll be making it even better in forthcoming releases. [7:37pm] SnapABug: (in this case, we already have the CNAME www.snapabug.com mapped ghs.google.com, but only http is handled) [7:37pm] Wesley_google: chilts: glad you're enjoying it! [7:37pm] Jason_Google: SnapABug: Hopefully, we'll be able to start supporting HTTPS on non-appspot.com domains before it comes to that point. For now, you have to manage this redirect yourself. [7:39pm] silassewell: Q: I was wondering if there were any plans to allow the use of the blobstore via the fetch service? [7:39pm] SnapABug: HTTPS support would be awesome. [7:39pm] rcopenjr: Jason_Google: A follow up question - is there a determining factor for a large application - [7:39pm] SnapABug: Is it on the roadmap? [7:39pm] rcopenjr: number of entities, kinds, etc. [7:40pm] ryan_google: rcopenjr: determining factor for performance? ie latency, scalability, ...? [7:40pm] daskalou: Q: Besides for transactions, is there any other reason why we should put entities into entity groups? [7:40pm] ryan_google: the short answer is no, number of entities and kinds generally don't affect performance of individual entities. [7:40pm] ryan_google: daskalou: the short answer is no [7:40pm] ryan_google: er, sorry, performance of individual *operations* [7:40pm] rcopenjr: ryan_google: performance [7:40pm] rcopenjr: Thanks..... [7:42pm] andywocky: Q: I'm confused about the lifecycle of a handler. If I map two endpoints in app.yaml, say to handler1.py and handler2.py, are those handlers ever persisted in memory or cached? For instance, will each handler run (including imports) for each request, or less frequently? [7:43pm] Jason_Google: SnapABug: Not yet. I'm not close enough to this piece to comment with any authority, but I believe that support for HTTPS on non-appspot.com domains, when it happens, requires us to take advantage of browser support for SNI, so older browsers will necessarily be left behind here. This is one reason why we haven't enabled this yet -- we're still waiting for broader adoption. But we recognize that a lot of developers want this, so we're looking into prioritizing it higher. [7:44pm] Wesley_google: SAB: here's more regarding what jason was saying about non-appspot HTTS support and SNI: http://googleappengine.blogspot.com/2008/10/announcing-https-support-for-appspotcom.html [7:44pm] SnapABug: Jason_Google: Thanks for the detailed answer. Will be looking forward for this feature release. [7:44pm] Jason_Google: silassewell: Can you elaborate on your request? I probably can't confirm one way or the other, but I can follow up for you. Also, feel free to file feature requests in the issue tracker so other developers can vote them up: http://code.google.com/p/googleappengine/issues/list [7:45pm] ryan_google: andywocky: have you read http://code.google.com/appengine/docs/python/runtime.html#App_Caching ? [7:45pm] Wesley_google: (look towards the end of the blogpost) [7:45pm] daskalou: Q: What's the best way to handle putting a decimal data type into the Datastore? [7:46pm] SnapABug: Wesley_Google: OK. Cool. [7:46pm] Jason_Google: daskalou: There's actually a thread about this in the groups, I believe. Nick Johnson wrote something about it. Let me try to find it... [7:47pm] silassewell: Jason_Google: I created a reverse proxy which sites on app engine. Right now the fetch service and datastore is currently limited to 1mb, so allowing fetch to retrieve objects larger than 1mb and storing them in the blobstore would be nice (I'm also assuming you do some optimizations when serving blobstore objects which may be more efficient than datastore + memcache). [7:47pm] enigmus_: Q: Are there news on http://code.google.com/p/googleappengine/issues/detail?id=2481 ? "Index inconsistency: queries not returning all results" [7:47pm] ryan_google: enigmus: one of my pet projects! [7:47pm] ryan_google: we hate that [7:48pm] daskalou: Jason_Google: thanks [7:48pm] ryan_google: we (the datastore team) have been pushing hard on that, and we're making solid progress. i can't say that we've fully fixed it, but we think we're close. [7:48pm] ryan_google: one thing to note is that we're focusing on finding and eliminating the root causes first, since if we just go through and fix the data without fixing those bugs, they'll just cause more inconsistency [7:49pm] ryan_google: but naturally we fully expect to fix both the bug (s) and the existing data [7:49pm] Jason_Google: daskalou: So far, I've found this: http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html [7:49pm] enigmus_: ryan_google: ok, thanks [7:49pm] Brandon\wcr: I've been sort of out of it, have cursors been implemented yet? [7:50pm] Jason_Google: Brandon\wcr: Coming up in the next release or two. [7:50pm] daskalou: Jason_Google: thanks, I found this too: http://www.redredred.com.au/money-database-property-for-google-app-engine/ [7:50pm] salsakran: what kind of write load into the datastore is possible by an app? [7:50pm] Jason_Google: daskalou: Yep, that links to the same blog post. [7:50pm] daskalou: Q: When do you expect GAE to come out of "preview release"? [7:50pm] daskalou: Jason_Google: lol, cool [7:51pm] ryan_google: daskalou: we'd need to define the difference between preview and non-preview first. [7:51pm] andywocky: ryan_google: thanks for the pointer. Am I interpreting correctly that imports are cached, but that imports for each handler (handler1 vs handler2) are independent? [7:51pm] andywocky: ... meaning that handler1 has its own namespace, separate from handler2? [7:51pm] ryan_google: out of curiosity, does it mean something specific to you? for example, do you have stakeholders who won't use a "preview" product because of that term specifically? [7:52pm] daskalou: ryan_google: What do you expect the difference to be? [7:52pm] Brandon\wcr: Q: Is there any timeline for task queue quota increases? [7:52pm] daskalou: ryan_google: Yes, that's pretty much it [7:52pm] Jason_Google: salsakran: You can write data pretty quickly, but you'll run into problems when you try to write the same entity or write to the same entity group too quickly. [7:52pm] chilts: Q. one concern of mine is if Google ever decide to pull the plug on AppEngine? I guess there are no guarantees this will never happen? [7:52pm] salsakran: specifically: if we're writing scrobbler type application, and split by user_id and shard the counters in question, what sort of throughput is reasonable to hope for? [7:52pm] chilts: I mean, if we have all these things running on it, then it goes away, that'll be sad [7:53pm] Jason_Google: Brandon\wcr: Which quotas specifically? [7:53pm] daskalou: chilts: Good question [7:53pm] Wesley_google: daskalou: your post does point to nick's blogpost but the blogger in this case focused on a limitation of using strings to store numeric values with and provided his own solution [7:53pm] Brandon\wcr: Jason_Google: oops! nevermind, just saw it was increased already to 1mill [7:53pm] ryan_google: daskalou: it's basically for managing expectations, as you'd expect. re your stakeholders, that's sad. i guess i'd encourage you to point them to actual data (status site for uptime, over 100k apps and 250k developers, more than 250M pageviews per day, apps in the high 10s of millions of users, etc.) [7:54pm] salsakran: do you happen to know what kind of write throughput I can expect per entity? [7:54pm] Jason_Google: chilts: App Engine isn't in labs, so we do have a formal policy in place to support all apps running on it for a certain period, 3 years I think, from any announcement of deprecation, which isn't planned. [7:54pm] ryan_google: chilts: naturally we don't expect that. you're right though that it's a risk with almost anything you use that's run by someone else. [7:54pm] ryan_google: aha, jason has a much more specific answer. ignore me. [7:54pm] chilts: awesome [7:54pm] chilts: 3yrs sounds plenty in that event [7:55pm] ryan_google: salsakran: one answer is, don't expect more than 1-10qps of writes per entity group [7:55pm] chilts: it's good to know these things, esp. if you base a business around someone elses [7:55pm] ryan_google: definitely! [7:55pm] daskalou: Wesley_google and ryan_google: thanks [7:56pm] ryan_google: np! glad to help [7:56pm] Wesley_google: same here... glad to help users out!! [7:56pm] andywocky: ryan_google: is my assumption about cached imports in handlers correct? [7:56pm] chilts: yeah, thanks guys (I figure these chats take it out of you and make you tired) [7:56pm] SnapABug: Yep. Thank you guys! [7:56pm] ryan_google: andywocky: right, sorry. i'm not an expert on the python runtime so i'm thinking about it. [7:56pm] Wesley_google: i'm also open to taking any Python-specific questions too for those of you who are newbies [7:57pm] daskalou: Yeah Python newbie here [7:57pm] daskalou: Wesley_google: Do you have experience with AEP? [7:57pm] Wesley_google: speaking of which, today is the last day for earlybird registration to the largest Python conference in the world happening next month in ATL [7:57pm] chilts: Wesley_google: I had an interesting one the other day with BlobStore ... but I think that might be a question for the group (since I'll need a few points to describe it) [7:57pm] ryan_google: andywocky: when you say independent namespaces, do you mean that the global vars are kept separate? [7:58pm] Wesley_google: daskalou: nope... what's AEP? [7:58pm] ryan_google: actually, regardless, it's probably over my head. wesley might know, otherwise you could try the google-appengine- python group [7:58pm] Jason_Google: salsakran: Did Ryan answer your question? Writing a lot of data isn't a problem, only when you're writing too much to the same entity group.Some strategies about how to avoid this: http://code.google.com/appengine/articles/scaling/contention.html [7:58pm] salsakran: jason_google: yeah ... the 1-10 q/s was a useful number [7:58pm] Wesley_google: andywocky: each handler has its own namespace, just like independent modules in "regular" Python [7:59pm] andywocky: wesley_google: thanks! [7:59pm] Wesley_google: sure [7:59pm] daskalou: Wesley_google: app-engine-patch - Django with support for admin and other things the built in Django doesn't support [7:59pm] chilts: Jason_Google: so my thoughts there are, if I have maybe 100 entities (no particular entity group), writing them all should be no problem (even if they land on the same datastore server)? [7:59pm] Wesley_google: ah, i figured as much [8:00pm] Jason_Google: salsakran: Yeah, it varies, but you will likely see some level of timeouts if you start writing more than 1 QPS to a given entity group with any consistency. For that reason, we recommend you keep your entities as small as possible and for items like counters where you expect to update every request, you can shard your data using a technique mentioned in that article. [8:00pm] chilts: this is a theoretical thought (I don't have 100 entities to write) [8:00pm] Wesley_google: unfortunately when i worked on the Django book back in 2008, only the Django Helper for App Engine was out. AEP just came out when our book was published, so i didn't get a chance to experiment with it [8:00pm] daskalou: Wesley_google: Re: my thread on the groups: http://groups.google.com/group/google-appengine/browse_thread/thread/44d2d67cb1f4f735 Do you know if it's difficult to "share" models between the two frameworks? [8:01pm] Jason_Google: chilts: Yes, this is right. There does come a point (e.g. several thousand QPS) when there are other considerations, but in general, you should be able to write many entities in separate entity groups in parallel. [8:01pm] chilts: sweet [8:02pm] chilts: I must admit, sometimes I wonder about how much hardware you guys have there, and whilst you might not be at liberty to say, a blog post would be awesome (how many memcache instances, datastore servers, frontends etc) [8:02pm] chilts: would make an interesting post [8:02pm] Wesley_google: daskalou: yes, i think i skimmed your post. i don't know the answer but my intuition is that it isn't cached [8:02pm] Wesley_google: or rather, shared [8:03pm] daskalou: Wesley_google: oh... damn [8:03pm] Jason_Google: Alright then, the office hour is officially over. Feel free to keep the discussion going though. We'll be back in two weeks, Wednesday, January 20th, from 9:00 to 10:00 a.m. PST, and I'll post a transcript of this session to the groups as soon as I can. Thanks for the company. [8:03pm] ryan_google: chilts: agreed! it's actually probably a lot smaller than you think...but still, yeah, don't hold your breath. [8:03pm] chilts: thanks Jason_Google [8:03pm] chilts: ryan_google: heh, awesome [8:03pm] chilts: but interesting
-- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
