This past Wednesday, the App Engine team hosted the latest session of its bimonthly IRC office hours. A transcript of the session and a summary of the topics covered is provided below. The next session will take place on Wednesday, August 5th from 7:00-8:00 p.m. PDT in the #appengine channel on irc.freenode.net.
--SUMMARY----------------------------------------------------------- Q: Are there any plans to add support for JDO's syntax for unowned relationships? A: The documentation does indicate that JDO's syntax for unowned relationships may be supported in a future release. Since there are other ways of specifying unowned relationships, this isn't the highest priority, but it may appear down the line. [9:03, 9:15, 9:18] Q: Currently, there doesn't seem to be a way to execute tasks as a particular user -- will this be added? A: There are several workarounds available. e.g. When enqueueing a new Task, you can take the ACSID cookie from the incoming request and add it as a Cookie: header to the Task -- this should work as long as the cookie doesn't expire in between the two requests. If you're using a Google Data API, you may also be able to pass the auth token in the payload of the task and then read it from the request when the task is started. Lastly, if you just need the user's email, you can use os.environ['USER_EMAIL']. [9:06-9:08, 9:10, 9:15] Since the local development server can only service one request at a time, you may experience some issues sending multiple requests simultaneously, including dropped static resources. It may help to limit your browser to one connection per site for testing (e.g. through about:config in FireFox). [9:12-9:13, 9:16, 9:19, 9:20, 9:21-9:22, 9:25] Q: Will there be any MapReduce service added to GAE? A: Yes! "Support for mapping operations across datasets" was recently added to the roadmap. [9:17, 9:19] Q: What is the status of full text search? A: Native support is still a ways off, but there are a number of third-party libraries that may help. Many are being updated to use the new task queue to build index entities. [9:20-9:22] Q: Will there be an SLA for paid services? A: In time, an SLA and premium support options may be made available for App Engine. As of now, we generally offer the same support for all users -- we check the support groups daily and have forms set up for requesting more quota, reporting billing issues, etc. [9:26, 9:27, 9:30] Q: Any news on the multi-tenancy front? A: Support for multi-tenant applications is in process, but for now, you can get part of the way there by using hooks in the datastore to namespace all of the entities for a particular user/tenant. [9:28, 9:30] Q: Any plans to support a "firewall" or offer increased protection against DDOS attacks? A: It's not on the current roadmap, but a more robust DDOS API may be made available in the future so you can more easily block certain IPs and perform other related actions. If this is important to you, star the issue in the tracker: http://code.google.com/p/googleappengine/issues/detail?id=644 [9:31, 9:34] Q: Can the task queue API be used as a substitute for sharded counters? e.g. Instead of updating the counter every request, can I instead add a task to the queue that updates the counter when it's executed? A: The obvious limitation is the limit on the number of tasks that you can add to a queue per day -- 10,000. Aside from this, yes, it could work as a substitute for sharding. The counter reads may not be accurate to-the-second, but since you can control the rate at which the incrementor Task writes the value to the datastore, you can effectively bypass the write contention that occurs when writing to a single entity too often. [9:41, 9:47, 9:50-9:53] --FULL TRANSCRIPT--------------------------------------------------- [09:00am] scudder_google: Hi all, welcome to another installment of our twice monthly chat time [09:01am] theboxfactory__: Good morning [09:01am] scudder_google: From Google thus far we have Jason_Google and myself, and more will likely join as we go along [09:01am] scudder_google: so if you have questions for us, comments, suggestions, feel free. Don't hold back [09:02am] hansamann: when can we expect task api for java? [09:03am] max-oizo: Hi google team! My first question to Max Ross/or ORM team - when we expect a support unowned relations in JDO? I naturally want to approximate date (month/3 month/6 month/year)? [09:04am] max-oizo: And to java team - thanks for sdk 1.2.2 [09:04am] theboxfactory__: Python: You now support Django 1.0, but when can we expect or indeed can we expect a more appengine specific version, closer to what the appengine patch current implements [09:04am] scudder_google: hansamann: we usually don't get very specific on release dates for specific features, as acurately predicting is just too difficult, so all I can say is "soon" [09:04am] Beruk: Hello! How is the XMPP api coming along? Can you share what areas this API will cover? (e.g. will it be limited to sending/receiving instant messages, or will I be able to implement a XMPP component hosted on GAE) [09:05am] hansamann: thx. another question. how can you best prepare your app for tasks once available? [09:06am] Wooble: speaking of django 1.0, is it possible to use the new version of just the templates? [09:06am] scudder_google: theboxfactory__: I would say if appengine patch is working for you, then by all means continue with it. What kinds of things specifically were you hoping to see? [09:06am] pedromorais78: Hi - about the task API - how about "run as user" tasks? Currently, in production are run without a user logged in. [09:06am] _matt_: you mean so you could like, send an e-mail from that user account? [09:07am] pedromorais78: actually, so I can do some gdata calls (using the token store) [09:07am] scudder_google: pedromorais78: I recall seeing this dicsussed in the app engine discussion group and it sounds like a great feature request [09:07am] _matt_: yeah, pro-that. [09:08am] scudder_google: I recommend looking for it in the issue tracker, or filing on if this isn't in there yet [09:08am] pedromorais78: btw: right now, I'm working around it by using os.environ['USER_EMAIL'] = (determine the user's email) [09:08am] _matt_: scudder, are you an official rep of any sort? [09:08am] theboxfactory__: scudder: I suppose ideally, I am hoping to see something closer to a complete application template as a starting point. So rather than joining the dots ourselves, we have a definitive way to use a specific framework. [09:08am] scudder_google: pedromorais78: for your specific case, it might be possible to send the auth token in the payload of the task, then the token could be read from the body, and set in the gdata client [09:09am] max-oizo: What about my question? If we haven't any plans about it I will try to write my own implementation [09:10am] pedromorais78: scudder_google: instead of the token, I can determine the user, and use os.environ['USER_EMAIL'] to "fool" the get current user calls inside the token store [09:10am] dw: i posted a question to google-appengine-python regarding pausing task queues, but received no reply. specifically, setting the rate to 0 doesn't seem to pause immediately. [09:10am] Jason_Google: max-oizo: Can you go into more detail about what you want? There is SOME support for unowned relationships right now. [09:12am] scudder_google: dw: yes I recall seeing that thread but I haven't been able to reproduce yet, either Nick or I will have to get back to you on that one. Great question. [09:12am] _matt_: Anyone have any info on modifying devappserver be more multi-request friendly? (so random images don't drop out of my local page renderings). I started down what felt like a rabbit hole yesterday (with a threading mixin to the BaseHTTPServer), and aborted. [09:13am] dw: scudder_google: i'll try and follow up with more details (since i need to unpause-and-probably-repause the queue shortly ) [09:13am] _matt_: appears each request does a complete recheck of everything for modification, and that uses a dictionary which isn't concurrency friendly...wasn't sure it I keep running into things like that the deeper I got. [09:13am] Beruk: In case my question was lost in the flow of questions, here it is again: How is the XMPP api coming along? Can you share what areas this API will cover? (e.g. will it be limited to sending/receiving instant messages, or will I be able to implement a XMPP component hosted on GAE) [09:14am] Jason_Google: _matt_: I haven't encountered this before. If you can come up with a reproducible test case, please file a new issue in the issue tracker. [09:15am] scudder_google: pedromorais78: the gdata client's token store relies on get_current_user which in turn uses os.environ ['USER_EMAIL'] so yes I think this should work though I haven't tried it [09:15am] Jason_Google: Beruk: The XMPP API is coming along nicely, but I don't have any extra details for you, unfortunately. [09:15am] max-oizo: I mean JDO's syntax. From the documentation - ... JDO's syntax for unowned relationships may be supported in a future release [09:16am] pedromorais78: Also - not expecting concrete dates or anything - any news about efficient paging/efficient datastore text search/data analysis services? [09:16am] _matt_: It's appears interrmittent based on load... webkit activity monitor shows the particular requests as 'connection reset', while local dev_appserver logs show those requests at HTTP 200 OK.. [09:16am] scudder_google: Beruk: XMPP is still in work and I was curious what you meant by implementing an XMPP component [09:17am] _matt_: (re: dropped resources/images in local dev_appserver) has anyone else encountered this? [09:17am] Beruk: Jason_google: the XMPP api was originally slated for June in the previous roadmap, but now it is listed among "features on deck" with no timeframe. I know you don't want to commit to a date, but can we expect XMPP soon? [09:17am] randomnick123123: Hi Google - Will there be any mapreduce service added to GAE? [09:18am] randomnick123123: And will the search be improved and how? are we likely to see decent freetext search ? [09:18am] Jason_Google: max-oizo: Ah, I see. I'm not sure where this is at... it's probably not the top priority since there are other ways of using unowned relationships. [09:19am] scudder_google: randomnick123123: we plan to have something along the lines of map reduce, we just added it to our roadmap [09:19am] scudder_google: http://code.google.com/appengine/docs/roadmap.html [09:19am] scudder_google: "Support for mapping operations across datasets" [09:19am] _matt_: (re: dropped resources/images) also happens in firefox intermittently, so pretty sure it's not the browser. happens on my business partners machine also, so that leaves our code, or dev_appserver. [09:20am] rsaccon_: there are here about 90% roadmap questions, so let me add mine: what can we expect next for the Java platform (not asking for a date) [09:20am] Beruk: scudder_google: implementing an XMPP service as an external component to an existing XMPP server (e.g. ejabberd or OpenFire) [09:20am] randomnick123123: scudder_google: ok great thanks - what about search? [09:20am] _matt_: (re: dropped resources/images), so far, only seen for static resources... [09:21am] Jason_Google: Beruk: It's on it's way. I'm not sure about the timeline, but the feature is in active development. [09:21am] _matt_: Does the 'dev_appserver.py can only serve one request at a time' restriction seem to apply to this problem? [09:21am] scudder_google: randomnick123123: full text search is likely a bit further out, but I've seen several libraries in the discussion groups that should be able to get you most of what you need [09:22am] randomnick123123: Is there/ will there be any kind of SLA for paid services? [09:22am] Jason_Google: _matt_: It does sound related. [09:22am] scudder_google: often these involve building index entities using the task queue [09:22am] Beruk: Jason_Google: would it be possible to post more details about what you will be offering in the XMPP api, even if the API isn't ready for release? I am implementing a solution based on GAE and XMPP right now, and I need to figure out how my GAE services will interact with the XMPP server... [09:23am] Beruk: Jason_Google: or, to put it differently, will you be offering a complete XMPP server as part of GAE, or simply XMPP client APIs? Will you allow persistent connections from GAE to an XMPP server? [09:24am] rsynnott: some sort of message queue-based thing could perhaps work.... [09:25am] _matt_: Jason_Google: thanks. Only curious thing. No one else has mentioned this problem? [09:25am] Wooble: _matt_: I suggest limiting your browser to 1 connection per site for testing, might help. [09:25am] _matt_: Wooble: good idea I'll try that in ff about:config today. [09:26am] randomnick123123: Jason_google: Is there/ will there be any kind of SLA for paid services? [09:26am] stevenmbp: dear google employees: why is google so absolutely impossible to ever contact about anything, ever? absolutely no contact links are available no matter how far i dig. all i can do is post to a public mailing list and hope a google employee sees it [09:26am] greg: How can I specify the heap size in Java? [09:26am] stevenmbp: it's extremely frustrating. that is all. [09:27am] Jason_Google: randomnick123123: I'm sure an SLA will be available eventually. [09:27am] Kevin_Noonan: Hi Googlers, do you plan to provide task queues for the Java runtime? Why is there no mention on the list of "features on deck"? [09:27am] _matt_: stevenmbp: aren't you talking to them right now? ... ok I know that's not exactly what you mean [09:28am] stevenmbp: to clarify, i dont thnk any of you suck and this isnt personal. [09:28am] randomnick123123: Jason_google: is there any support available currently to businesses who want to base their business on this platform? [09:28am] stevenmbp: and to be fair, ive been using google's products for 9 whole years before the first time i ever wanted to contact you guys [09:28am] stevenmbp: but when i did, you were nowhere to be found. that is all. [09:28am] pedromorais78: roadmap/multi tenancy: in one google io session (i think) someone mentioned that multi-tenancy is "not a solved problem". Any news on that front? [09:28am] scudder_google: Beruk: XMPP features like stanzas, IQ, etc. are all on our roadmap for XMPP, we are aiming for full support beyond just chat messages [09:28am] Wooble: not waiting for a response is a good way to feel like you can't contact them [09:29am] Beruk: scudder_google: that sounds great! Will I be able to maintain a persistent connection between GAE and XMPP server? To implement a bot on GAE, for example. [09:30am] scudder_google: pedromorais78: multitenancy is in process as well. You can get part of the way there today by using hooks in the datastore (for example) to namespace all of the entities into a particular tenant [09:30am] Jason_Google: randomnick123123: In general, we offer the same support for all users -- we check the support groups daily and have forms set up for requesting more quota, reporting abuse and billing issues, etc. [09:30am] scudder_google: pedromorais78: let me find the article for you [09:30am] scudder_google: http://code.google.com/appengine/articles/hooks.html [09:31am] _matt_: thanks for the input/ideas all. gotta run [09:31am] max-oizo: The second very important question - have you any plans to support "firewall"? Now exists big problem with DDOS. When a business of "malware" and "antispyware" ended, a botnets switched to the to organize DDOS attacks. As example? In ICQ I get spam every day with offers DDOS attacks. Some of my full legal projects repeatedly subjected to attacks. On my own servers i just set firewall in "deny all" for some time, but in distributed AppEngine this attacks de [09:32am] pedromorais78: scudder_google: thanks for the tip [09:33am] scudder_google: Wooble: yeah, it's too bad stevenmbp signed off, I was going to tell him my email address, not that it is difficult to find [09:33am] mark0978: Datastore Question: Trying to figure out how to get indexes to cluster around things. Basically, I'm wanting indices to be around Account/Year So I'm using parents, Account is Parent of Year is Parent of Item. Is this the right way to get that kind of clustering? [09:33am] greg: heap? heap? I'm getting an OutOfMemoryException, and I'm not using JDO or anything like that. Can I increase the heap size? [09:34am] Jason_Google: max-oizo: It's not on the current roadmap, but I know the team wants to implement better DDOS services in the future. So you can block certain IPs and so on. [09:34am] Jason_Google: greg: I've seen a solution posted in the group. Have you searched there by any chance? [09:35am] greg: I have been searching, but most of the advice has to do with various persistence mechanisms, which I am not using. I'm still looking, though. [09:35am] greg: Can one specify command-line arguments to the 'java' executable? [09:35am] stevenmbp: i appreciate your concern, but this is more of a complaint againt google itself than an immeduate request for contact... its just wrong in principle that one has to randomly run into a google employee (like i am now) in order to contact google... [09:36am] Jason_Google: greg: This thread is the one I was thinking of. Let me know if you're looking for something else: http://groups.google.com/group/google-appengine-java/browse_thread/thread/031d8875b6c8f1a7 [09:37am] greg: I mean OutOfMemoryError, not Exception, of course. [09:37am] greg: Oh, I see -- thanks Jason_Google. [09:38am] scudder_google: stevenmbp: glad you're back, and thank you for letting us know. I agree that this is something we could really improve on. [09:38am] stevenmbp: thanks. [09:38am] the_tommy_: If we want to develop an app to tight deadlines a paid for priority support with direct access to google experts would be good - along with SLA [09:38am] stevenmbp: id appreciate it if this was passed on somehow to a supervisor (but hopefully not like a game of telephone) [09:39am] scudder_google: stevenmbp: Ideally, what are you looking for? Often times people want to talk to someone on the phone. Would an email address be better than a mailing list? [09:39am] dickyj: question on task queue api, how many queues can be created and how many can be executed concurrently? [09:39am] dw: dickyj: i believe concurrency is based on the max http reqs/second your app can serve. i may be wrong tho, new to task queues myself [09:40am] stevenmbp: yes. [09:40am] scudder_google: mark0978: I think we would need a lot more detail to be able to give good advice on entity design. This is something that often works better in the group. Where paragraphs of text aren't chopped up by interleaving conversation [09:40am] stevenmbp: an email address like cont...@google.com [09:40am] Jason_Google: the_tommy: Yes, this has been brought up before, and we're definitely aware of the needs out there. [09:40am] dickyj: ok thanks [09:41am] dennis_tw: python task queues: i'm trying to figure out if i should use them in my code. in the unlikely case that i hit the current limit of 10k/day, will my code stop working? if this case, can i get the limit increased by 10x? the current limit is much smaller than would be needed for a busy website that uses the queues as a critical part of reducing latency. (ps: amazon SQS queues are unlimited) [09:41am] dickyj: if I have a database row with high contention, instead of sharded counter, can I use task queue? [09:42am] hansamann: will the jpa documentation be updated and expanded soon? [09:42am] stevenmbp: i dont think this is the appropriate channel for this discussion (ironically, since there is none) [09:42am] Beruk: scudder_google: Will I be able to maintain a persistent connection between GAE and XMPP server? To implement a bot on GAE, for example. [09:42am] stevenmbp: so im going to leave. PM me or join #giantrobot for continuation, if wanted [09:44am] jblank_: Having problems doing re.sub where the 'from' text includes one or more \ns [09:44am] max-oizo: Jason_Google: I hope to include it in this roadmap. Really, i'm afraid (and not only i) transfer projects without DDOS protection to the AppEngine. [09:44am] max-oizo: This problem with DDOS mass revealed only recently and turned into a nightmare. [09:44am] max-oizo: As far as I know, the price of DDOS fell dozens of times, and it became public benefit competitors (and botnets grow, and prices for their services will fall). [09:44am] max-oizo: I very much ask to think about this problem. [09:44am] scudder_google: Beruk: implementing a bot is something that we certainly plan to allow. As far as a persistant connection I can't recall at the moment, so I'd have to look into it. [09:45am] Jason_Google: max-oizo: Understood. [09:45am] Beruk: scudder_google: well, to implement a bot I need to be connected to the XMPP server, so I'd assume that persistent connections will be allowed... [09:45am] Wooble: apparent DDOS protection has appeared a few times, when people have shouted about their own bots being blocked for being bots. [09:46am] max-oizo: Jason_Google: Thank you very match. [09:46am] Jason_Google: dennis_tw: Please file a quota increase request. You can find a link to the form in the FAQ. [09:47am] Jason_Google: dickyj: Good question. I'd have to run some tests to see how well this would work in practice. [09:47am] Jason_Google: dicky: Obviously, when we wrote the original sharded counter example, the task queue API wasn't available yet. [09:47am] dickyj: Jason_Google: I just think the sharded counter is such a hack! [09:48am] Beruk: dickyj & Jason_Google: out of curiosity, how would I implement a counter with the task queue? [09:48am] Jason_Google: dicky: I prefer to think of it as a workaround. [09:49am] dennis_tw: jason_google: well, i'm in the design phase. seems like it's in google best interest to allow large growth in task q usage so gae can test them under stress. no guarantees that it will work under stress, but could be allowed [09:49am] Jason_Google: Beruk: See http://code.google.com/appengine/articles/sharding_counters.html for the sharded solution that dickyj was referring to. [09:50am] dickyj: jason_google: question, if I use task queue, how do I ensure that the transaction does not fail, should I be creating a task queue for each table row then? [09:50am] max-oizo: What's the reason in anormal urlfetch (from http://code.google.com/status/appengine/) ? [09:50am] Beruk: Jason_Google: I know about it, I am using sharded counters in my app! I just don't see how the task queue could be used to solve the same problem... [09:51am] scudder_google: Beruk: I think the idea is that you could insert a task to increment the counter instead of changing it directly so you would defer the write. [09:51am] Jason_Google: dennis_tw: I don't know how flexible this particular quota is, but if you file a request for additional quota, I'll be able to look into it in more detail. [09:51am] _sid: my question is neither for python nor for java..just want to ask when you guys are going to support Php? [09:52am] scudder_google: Beruk: you could also limit the rate that the incrementor task queue would run at, so as to minimize collisions [09:52am] dickyj: Beruk: agree, with task queue, write is deferred [09:52am] jlivni: Beruk: Jason_Google perhaps the io session at http://code.google.com/events/io/sessions/OfflineProcessingAppEngine.html around minute 40 would be of interest for TQ related sharded counters [09:52am] Jason_Google: dickyj: In the event of a failure, the task should be re-pushed to the queue automatically. [09:52am] Beruk: scudder_google: indeed, but then counter reads wouldn't be accurate... [09:53am] Beruk: jlivni: Thanks, I'll watch the session! [09:53am] dickyj: yes, it is not a replacement for sharded counters [09:53am] jlivni: Beruk: yeah, the counter would not be accurate until the TQ is empty [09:53am] scudder_google: Beruk: yes, so this works for cerain use cases, but not too the second accurate counts [09:53am] scudder_google: *to the second [09:54am] jlivni: http://code.google.com/events/io/sessions/AppEngineNittyGritty.html around minute 27 describes a sharded counter w/memcache in front to vastly minimize datastore transactions [09:54am] dickyj: so, should I just create one task queue and push everything there [09:54am] jlivni: our app we do lots of counting, and use this w/some success [09:54am] dickyj: what are the benefits of multiple task queues? [09:54am] Jason_Google: _sid: We're not ready to confirm any other runtimes. At present, we offer SDKs for Python and Java, and there are no other runtimes on the current roadmap. [09:54am] jlivni: if i rewrote it i'd add in TQ inbetween memcache and the datastore write [09:55am] jlivni: er, around minute 29 for that last video [09:56am] scudder_google: dickyj: one of the benefits of having different queues is that each can have it's own settings [09:57am] Beruk: Do you have some sort of private beta for upcoming features, such as XMPP? [09:57am] max-oizo: I'm trying to repeat the question What's the cause of anormal behavior urlfetch (in appengine status)? [09:58am] dickyj: scudder_google: do you have a sample use case for that? [09:58am] scudder_google: dickyj: the rate and bucket size can be different for each queue, see http://code.google.com/appengine/docs/python/config/queue.html [09:59am] Jason_Google: max-oizo: It looks like there was an abnormal number of errors in fetching google.com, which is the measure of how the URL Fetch service is doing. [09:59am] Jason_Google: http://code.google.com/status/appengine/detail/urlfetch/2009/07/15#ae-trust-detail-urlfetch-internal-search-error_rate [10:00am] Jason_Google: Since the AWS S3 error rate is low, this could be related to google.com and not the actual URL Fetch service. [10:01am] Beruk: (repeat) Do you have some sort of private beta for upcoming features, such as XMPP? [10:01am] _sid: Jason_Google: alright. Any plan to support of Atom publishing just like you guys provide GData for various services? [10:01am] max-oizo: [10:01am] max-oizo: By the way, please correct the documentation - http://code.google.com/p/googleappengine/issues/detail?id=1766#c7 [10:01am] dw: _sid: um. what's to stop you from doing that today? [10:02am] Jason_Google: Beruk: No. All testing is done internally. [10:02am] _sid: dw: i am just unware whether such support is already provided.. [10:02am] Jason_Google: max-oizo: Thanks for the note. We'll correct this soon. [10:02am] max-oizo: It seems to me that this is the case. According to the release dates Python SDK converges [10:03am] Beruk: Jason_Google: Ok, thanks. [10:03am] dw: _sid: atom collections are just xml documents, which python supports. the atom protocol operates on http, which python supports. [10:03am] dickyj: scudder_google: thanks [10:03am] max-oizo: Jason_Google: your welcome [10:04am] dickyj: looking forward to the task queue api in java [10:04am] Beruk: Looking forward to the XMPP api in Python! [10:04am] _sid: dw: I actually meant sort of wrapper for Google's DB engine to store/retrieve data rather thn accessing native methods.. in this way one will not need to ask for stuff like php support or any other support [10:04am] Jason_Google: Alright, that concludes today's Chat Time. We'll be back Wednesday, August 5th from 7:00 to 8:00 p.m. Thanks for the great chat! [10:04am] dw: thanks Jason_Google [10:04am] dickyj: thanks [10:05am] pedromorais78: thanks [10:05am] Beruk: Thanks to you googlers for your time, and keep up the great work with App Engine! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
