This past Wednesday, the App Engine team hosted the latest session of its bimonthly IRC office hours. A transcript of the session and a summary of the topics covered is provided below. The next session will take place on Wednesday, September 2nd from 7:00-8:00 p.m. PDT in the #appengine channel on irc.freenode.net.
--SUMMARY----------------------------------------------------------- - To check support for a Java library or framework with Google App Engine for Java, see the "Will it play in App Engine" wiki page at http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine. If you have tested other libraries, please leave a comment on that page. [9:06] - Q: Are there any current plans to better integrate Google Web Toolkit (GWT) and App Engine for Python, specifically around user management and data exchange? A: For now, most of the work going into better integrating GWT and App Engine is in Java, although this does not imply that you can't use GWT with App Engine for Python -- many developers currently do. If you want to write your own tools to improve user management with App Engine for Python, you should do so without worry that they may be replaced down the line. [9:06-9:07, 9:15, 9:24] - Q: Are there any plans to support other currencies (e.g. the Euro) or implement threshold billing so users aren't billed until they accrue a certain minimum charge? For very small charges, the tax can make up 90% or more of the total charge -- see http://code.google.com/p/googleappengine/issues/detail?id=1999. A: We are aware that there are various issues with the current billing setup, and we are looking into ways of mitigating these issues. [9:09-9:10, 9:13-9:15] - Q: What is the TRUSTED_IP_REQUEST variable that appears when you inspect os.environ in http://shell.appspot.com? A: It doesn't represent a new API in the works or any change in behavior on how we treat user data -- it's simply the result of a configuration option we use for internal testing which was accidentally enabled for this app. - Q: What is the most effective way to continually fetch feeds for many (e.g. > 1,000) users? A: Setting up a cron job to run every minute and create hundreds of new tasks likely won't scale very well. A better solution is something akin to PubSubHubbub (http:// code.google.com/p/pubsubhubbub/). [9:10-9:11, 9:13] - Q: What paths are there for moving an application off of App Engine? A: The Java runtime in particular was designed to take full advantage of existing Java standards including the servlet API, JDO/JPA, JavaMail, JCache, etc. while the Python runtime supports Django, one of the most popular frameworks available. We also have a script for easily exporting data out of App Engine and we're working to make it even more robust. [9:17-9:18] - App Engine is designed to serve end-user web applications as opposed to running large, lengthy data processing operations in the background. If you want to do background processing today, you can look into ways of splitting your data up and use the task queue, repeatedly adding new tasks to the queue, each running for 30 seconds, until the entire operation is done. This is particularly useful for cleanup operations. You can also look into remote_api, available today, and the forthcoming support for "mapping operations across datasets" listed on the roadmap. [9:22-9:25, 9:28] - Datastore queries vs. gets: In general, direct fetches (e.g. getting an entity by ID/key name) will be up to twice as fast as issuing a query, but in practice, this doesn't necessarily result in a huge gain, so don't worry if your data model isn't designed to support direct gets. There are ways of building your model to reduce the need for queries at the expense of code maintainability. [9:24-9:25, 9:35, 9:37, 9:39, 9:41, 9:45-9:46, 9:48] - Q: What will the upcoming cursor support enable? A: You will be able to do effective paging, but you won't be able to page backward or have random access. Cursor support is tentatively being targeted for 1.2.6. [9:26] - Q: Are we ever going to see support for maintaining sort order in multi-valued (list) properties and querying against the highest and lowest value of a sorted list? A: Likely no -- in general, supporting truly correct sort orders on multi-valued properties requires exponential space, and exponential time at write time, depending on the number of combinations of property values. [9:26, 9:28, 9:31-9:35, 9:38, 9:40-9:42, 9:44] - In general, the more index rows you have to write, the more likely you are to hit an unavailable tablet and error. using indexed=False is very powerful and definitely recommended, at least if you're sure you won't need to query on that property in the future. [9:29-9:31] - Q: When are we likely to see HTTPS support for non-appspot.com domains? A: We're still waiting for large-scale SNI adoption. [9:40-9:41] - Q: When are we likely to see support for static IPs? A: Allocating static IP addresses for an application would be difficult due to the distributed nature of App Engine, and this won't be available as an option in the near term. If you would be willing to pay for the feature, a better bet is to rent your own servers to act as proxies. [9:42-9:45] - Q: Is there any work being done on an API so we can test for capability issues before hitting an exception due to maintenance, etc? A: See the google.appengine.api.capabilities library. Unfortunately, it's not well documented right now. [9:48, 9:55-9:56] - Discussion on efficiently handling permutations of filters without causing exploding indexes using multiple boolean properties. [9:49-9:53, 9:59-10:01] - It's possible today to build an App Engine application that can be added to multiple domains running Google Apps. Just check the Host header to determine which domain is calling the application. There is no automated way to add a domain/mapping for an application, but this is straightforward for a domain administrator. [9:57-10:00] --FULL TRANSCRIPT--------------------------------------------------- [9:01am] scudder: Hi all, I'd like to kick off our hour long chat session. From Google we have nickjohnson, ryan_google, and myself. More will likely join as we go. [9:01am] scudder: I also see Wooble, one of our gurus here in the channel [9:02am] Beruk: Hello, is there any progress on the XMPP api? Any ETA? [9:02am] guest4: do you support icefaces? [9:02am] scudder: Beruk: we don't generally give specifics for when new features will be available [9:03am] scudder: Beruk: we are making progress on XMPP :) [9:03am] samgecko: can I solve this using tasks? let's say I have 5,000 users and I need to check the xml feed of a service they subscribe to, but I need to do it every minute. Can I have a cron that runs every minute spawn off let's say 100 tasks (assuming each task can process 50 feeds in under 30 seconds) [9:03am] Beruk: scudder: can you share some of its features? [9:03am] maxoizo: Hi google team! I have recently noticed in os.environ (from the shell.appspot.com) a variable "TRUSTED_IP_REQUEST". Is this a part of the firewall API (in future)? [9:04am] kickingvegas: hi all - [9:05am] scudder: guest4: for specific libraries and frameworks, we usually rely on the library maintainers and open source community to work with us to get a library to work within the restrictions of our runtime sandbox [9:05am] Wooble: samgecko: I can't imagine that doing any task for each of your users every minute is going to scale well. [9:06am] scudder: we attempt to track which frameworks people have found to work and which have reported known issues here: [9:06am] kickingvegas: wondering what plans are there to integrate GWT with Python AppEngine with respect to user management and data exchange [9:06am] scudder: http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine [9:06am] Keyframe2: appteam: what do you as a platform specialists prefer running on appengine - java or python, what would you use internally if you used app engine for an application and why? [9:07am] Jason_Google: kickingvegas: We are working to make GWT and App Engine play better together, but these are mostly concentrated on the Java front. [9:07am] samgecko: wooble:it's app engine .. why would there be problems scaling it? [9:08am] jurajv: google guys - have you had a chance to look at: http://groups.google.com/group/google-appengine/browse_frm/thread/b41b0e1ebbd43a59 -- is it realistic to expect this could get implemented at some point? [9:08am] digitalsanctum: appteam: what options are there available for near-realtime data pub/sub in GAE for Java? [9:08am] guest4: scudder: I have heard that if an icefaces app has threading disabled and sets the synchronousUpdate param to true then it may possibly work in GAE [9:09am] Wooble: samgecko: app engine makes it easy to write code that will scale well. that doesn't mean you can't design apps that run on it to scale horribly. Doing N/50 urlfetches for N users isn't going to scale. [9:09am] Jason_Google: Keyframe2: It depends on taste more than anything. I use Java for my internal projects, others use Python. [9:09am] psykidellic: So I am trying to create a UserProfile() class in addition to the user model to keep my data. When I save it with key, user.user_id(), GAE gives BadValueError as key cannot start with numbers. [9:09am] maxoizo: I'll try to repeat my question :) I have recently noticed in os.environ (from the shell.appspot.com) a variable "TRUSTED_IP_REQUEST". Is this a part of the firewall API (in future)? [9:09am] sylvain_: Hi all, I follow the [17:58] ibolomo question about monthly charge. [9:09am] psykidellic: and only user_id() is supposed to be unique. What do you guys suggest? [9:09am] rdayal: Hello all. [9:09am] sylvain_: I wrote a topic about taxes : http://groups.google.com/group/google-appengine/browse_thread/thread/b77805e2a405ae2d# [9:10am] sylvain_: In order to reduce the taxes part (for my example 90% of the price are taxes) , any plan to add new currency (Euro,...) or not be charged below a certain amount, etc, ? [9:10am] samgecko: Wooble:so I guess that's my question to the app engine team .. what would I use if I have X number of users and I need to check an xml feed that they subscribe to every minute? [9:10am] DocSavage: psykidellic: Just append a prefix like 'PR' to your key. [9:11am] psykidellic: DocSavage: I was actually doing it. Just wanted to be 100% sure. I am just adding my appname to it. [9:11am] samgecko: that's why I thought of using tasks .. eventually I'll hit my limit of tasks per minute that I can spawn [9:11am] Jason_Google: maxoizo: Off hand, I'm not sure what this variable is for, but I don't think that's necessarily a part of any forthcoming DOS/firewall API. Remember that we can only confirm features that are on the product roadmap, and these are the features in active development today. [9:11am] kickingvegas: Jason_Google: thanks for the answer, but does that imply that if I really want to use GWT, then I should start rebuilding my AppEngine site in Java? [9:11am] DocSavage: How is the Task API working out? Is the team going to promote it out of the lab? [9:12am] Jason_Google: sylvain_: There are no current plans to support other currency or billing systems. Please file a request in the public tracker. [9:12am] DocSavage: kickingvegas: Or use pyjamas [9:13am] Wooble: sylvain_: I'm guessing taxes aren't the real issue there; I find it hard to believe any country is assessing an 1100% tax on a service. [9:13am] scudder: maxoizo: I'm not sure that I'll be able to look into this during the char time, just wanted to let you know that your question hasn't been lost [9:13am] Jason_Google: samgecko: Have you seen this? : http://code.google.com/p/pubsubhubbub/ [9:13am] stbain: Wooble: Never underestimate the power of stupid people in large groups. That especially applies when it comes to banks and governments with regards to taxation. [9:13am] sylvain_: @wooble : check my topic : it is particularly true for very small charge [9:14am] scudder: I can follow up with you on this later, it might be best if you could post this to the discussion group, or I can email you, whichever you prefer [9:14am] kickingvegas: DocSavage: all due respect but looked at pyjamas and it's nowhere near as baked as GWT [9:14am] Wooble: well, if the bank's requring a $1 minimum charge that's not really a taxation issue. [9:14am] sylvain_: Yes, but if the charge was montly, it will reduce a lot the price [9:15am] DocSavage: kickingvegas: I'm using Extjs, which I think is better than GWT on many fronts. But it's javascript. [9:15am] maxoizo: 2Jason_google, scudder: thanks [9:15am] Jason_Google: kickingvegas: You can use GWT with App Engine for Python and many applications do. All of the current integration efforts that I'm aware of are targeting Java, but that doesn't mean that you can't use it with Python. [9:15am] jurajv: [Java] is the DatastoreService.put(List<Entity>) working as a batch put? [9:15am] sylvain_: or if checkout do not charge below a certain amont, or if there was a credit system, etc,... [9:15am] samgecko: Jason_Google: interesting, I'll check it out in more detail ... I like the "near-instant" .. does it scale? [9:15am] sylvain_: Anyway, thank you Jason, I will fill a request ASAP [9:16am] scudder: DocSavage: yes at some point the Tasks API will move out of labs, we're still gathering a list of changes that we're planning to make now that we've had more usage [9:16am] Jason_Google: jurajv: Yes, I believe so. [9:16am] Jason_Google: samgecko: Yes, as far as I know. [9:16am] jurajv: [Java] what is the runtime overhead of DatastoreServiceFactory.getDatastoreService()? Is it recommended to cache the DatastoreService on a static field, similarly to the PersistenceManagerFactory in JDO? (even though PMF is at a higher level logically) [9:16am] DocSavage: scudder: For Task API, might I suggest adding method to check # or at least presence of a task in a queue? [9:17am] guest4: can your app have obfusticated code? [9:17am] scudder: guest4: sure [9:17am] Wooble: guest4: well, there's no perl runtime yet... ;) [9:17am] scudder: haha [9:17am] Keyframe2: I know this might sound vague and probably will be without an answer - but what paths are there for if you want to take off your app off the app engine. It seems like you are completely tied to it. Will there, in the future, be some kind of OSS to host your own appengine appication? [9:18am] kickingvegas: Jason_Google: understood; however I'm disinclined to reinvent the wheel or use a project that will (or would) be superceded by Google since both GWT and AppEngine come from the same company; [9:18am] Jason_Google: jurajv: It is not necessarily recommended and I've never done it in my apps. That said, I don't know the actual overhead incurred. Typically, I actually make very few calls to this since I use Memcache in front of anything, which is the best practice. [9:18am] scudder: Keyframe2: if you are using the Java runtime and stick to standard APIs (jcache, JDO/JPA, etc.) then you can move any of several servlet containers [9:18am] DocSavage: Keyframe2: take a look at Appscale: http://code.google.com/p/appscale/ [9:18am] samgecko: ahh, pubsubhubbub is not going to work, the publisher in this case does not have an option to update a server each time a feed is changed [9:18am] Jason_Google: Keyframe2: There is a basic data export API which is only going to get better and fuller featured in the next releases. [9:19am] scudder: Keyframe2: if you are using Python, you may be able to migrate if you write all in Django, and yes as DocSavage mentions there are a few people I'm aware of who are creating App Engine compatible severs [9:19am] maxoizo: 2scudder: i prefering email - m...@oizo.biz [9:19am] cringer: ~maxoizo (pt|en) -> 2scudder i prefering email - m...@oizo.biz [9:19am] jurajv: before retrying a (retriable) datastore operation - is it a good thing to wait, let's say a 1000 ms before the retry? could this actually delay other requests already queued for the same VM instance? [9:19am] Keyframe2: excellent answers - thanks! [9:20am] scudder: maxoizo: sounds good, I'll follow up [9:20am] Jason_Google: kickingvegas: What do you mean by superceded? [9:20am] maxoizo: scudder: ok, thanks! [9:22am] Jason_Google: samgecko: Pubsubhubbub was developed by a Googler. He's not in the chat right now, but if you want to open a discussion on its capabilities, I'll let him know so he can talk with you more about its capabilities. He might also be able to help you with your original problem since he works on the Task Queue API. [9:22am] Keyframe2: one more question - about process execution time limit to 30 seconds, is there a guide/best practices how to work with that kind of limitation? Breaking stuff into chunks that run for <30 seconds seems impractical if you are doing, for example, calculating something, scrapping or pulling other resources outside of your appengine [9:22am] ryan_google: i'm late to the party, but psykidellic: we may remove the "key names must start with a letter" req't in the future [9:22am] samgecko: Jason, that would be great .. I'll msg you my details [9:23am] ryan_google: keyframe2: agreed! app engine is designed primarily to serve end user webapps, not long-running data processing within a single process or thread [9:24am] kickingvegas: Jason_Google: superceded may be the wrong word; deprecated or replaced may be more accurate; for example if somebody wrote a 3rd party library to do user management between GWT and AppEngine for Python and then Google decided to build that library themselves. [9:24am] psykidellic: ryan_google: Thanks. [9:24am] ryan_google: likewise if you need to load or pre-process a large amount of data to serve a single end user request [9:24am] DocSavage: Query vs Gets. My impression is that there's a big hit both in speed and error rate when you deal with queries that have to deal with the index. If so, let's say you have a User entity that you want to access through several properties (siteid, email, etc.) Would it be better time/error-rate-wise to create a child of User entity with key name = queryable property? [9:24am] scudder: Keyframe2 depending on what you are trying to do, a problem solving approach like map-reduce could accomplish calculations, aggregations, things along those lines [9:24am] samgecko: doh, Jason left [9:24am] ryan_google: having said that, there's lots of support and best practice discussions on splitting work up into smaller chunks usable with e.g. task queue [9:24am] psykidellic: So correct me if I am wrong, when I do a filter like: filter('mylist = ', somvalue) - internally it means the python IN i.e. elem in list: [9:24am] ryan_google: agreed, what scudder said [9:24am] Keyframe2: ok, thanks [9:25am] ryan_google: docsavage: gets will generally be roughly twice as fast as queries [9:25am] ryan_google: (with a wide margin of error) [9:25am] ryan_google: which is noticeable but not crucial [9:25am] Wooble: "Support for mapping operations across datasets" looks promising in the roadmap... [9:25am] DocSavage: Query vs Gets: The above would mean you ran a get on a key name followed by get on parent, instead of a query on the key name targeting a property. [9:25am] ryan_google: if it's reasonable within your data model to get instead of query, go for it, but don't tie yourself in knots. [9:25am] Keyframe2: seems a bit impractical if you want to do any houskeeping of the data in the background though [9:25am] moraes_: ooh chat time... [9:26am] ryan_google: psykidellic: correct [9:26am] moraes_: anybody asked about cursors already? what they will be? pagination will be easier with them? [9:26am] ryan_google: moraes: if your use case is "next page," and not random access or going backward, then yes, cursors will do that [9:26am] jurajv: http://groups.google.com/group/google-appengine/browse_frm/thread/b41b0e1ebbd43a59 -- is it realistic to expect this could get implemented at some point? [9:26am] ryan_google: we're tentatively targeting them in 1.2.6 [9:27am] ryanov: hello google engineers [9:27am] moraes_: cursors possibly in 1.2.6, you mean? [9:27am] ryan_google: jurajv: i'd need to spend some more time thinking about it [9:28am] scudder: Keyframe2: I think cleanup is one of the places where task queues really come in handy, you can grab ~30 entities or so, perform checks, then add a new task with a marker (key or something) to say which next 30 entities to grab [9:28am] jurajv: ryan_google: ok np [9:28am] cringer: ~jurajv (pl|en) -> ryan_google: ok np [9:28am] humanhistory: hey all [9:28am] ryan_google: but in general supporting truly correct sort orders on multi-valued properties requires exponential space, and exponential time at write time, depending on the number of combinations of property values :/ [9:28am] ryan_google: so, realistically, no [9:29am] ryan_google: moraes: yes [9:29am] jurajv: ryan_google: ok thanks [9:29am] moraes_: cool. tx for the info. [9:29am] humanhistory: im sorry if its been asked before, but are large file uploads (roadmapped, i know) closer to "really soon", "not too distant future" or "more like 6 months"? [9:29am] Wooble: Keyframe2: the remote API is also useful for doing stuff that takes a long time. [9:29am] DocSavage: ryan_google: Have you seen Dave Westwood's stats regarding his error rate? He's got a 0.00375% rate, which seems far lower than others (http://files.meetup.com/1135107/ BuddyPoke_AppEngine_Stats.pdf). He's mainly reducing indexed entities for puts. [9:30am] Keyframe2: Wooble, thanks for info [9:30am] nickjohnson: DocSavage: More complex operations are more likely to timeout than simple ones, so that's a good strategy [9:30am] ryan_google: jurajv, re retries, it depends largely on what caused the original error. backoff never hurts, but it may help more or less depending on what's going on. if you have hot entity(ies) or query(ies), then definitely. if it's transient timeouts due to e.g. bigtable tablet unavailability, backoff won't matter. [9:31am] ryan_google: docsavage: i haven't read that, no, but it's compelling. agreed, in general, the more index rows you have to write, the more likely you are to hit an unavailable tablet and error. using indexed=False is very powerful and definitely recommended, at least if you're *sure* you won't need to query on that property in the future [9:31am] DocSavage: It raises the question whether we should be structuring some models to be index-free or is the error/timeout rate more or less a linear function of the # of indexed properties? [9:31am] jurajv: ryan_google: re prev. issue: altough i'm not sure what is meant by 'truly correct' --> the current ones that consider smallest and lowest value of the list property would perfectly do --- the only additional requirement would be to not drop the sort order if equality and inequality filters are used on the list property [9:31am] jurajv: the current ones --> current sorting [9:31am] ryan_google: (docsavage: jinx :P) [9:31am] Jason_Google: humanhistory: "not too distant future" sounds about right. [9:32am] nickjohnson: jurajv: I believe his point is that you can't filter and sort on different elements of a list without requiring indexing it twice - thus, exponential space [9:32am] humanhistory: Jason_Google: faaaaantastic [9:33am] ryan_google: nick, it's actually the same property in this case [9:33am] nickjohnson: DocSavage: As ryan_google said, the more index entries you have to update, the higher the chance one will be on an unavailable template [9:33am] nickjohnson: ryan_google: Right, but different elements from that property's list, no? [9:33am] nickjohnson: eg, filter where foo=bar, order by the smallest element in the 'foo' list [9:34am] jurajv: nickjohnson: a list property should only have 2 entries in the index used for sorting - the smallest and the highest value; then an additional index with all it's properties is required for filtering --- if i'm not mistaken [9:35am] ryan_google: nickjohnson, jurajv: acknowledged, and thinking. keep discussing in the meantime. :P [9:35am] DocSavage: ryan_google: So back to the query vs get question. It seems more compelling to move queryable properties into child entities where the keyname = the property value. Instead of one query, you would then do a get on key name followed by parent, for queries where you expect 0 or 1 returns. [9:35am] nickjohnson: jurajv: It doesn't really work like that. The query has to be able to select a contiguous range of an index, and I don't think that's possible with filtering on a different listproperty element than is being sorted on [9:36am] DocSavage: ryan_google: It wins on error rate and should be comparable or better on time, right? [9:37am] ryan_google: docsavage: sure. again, though, i'm reluctant to push too strongly on optimizations like this in general, though, since a) often query vs get is nowhere near the bottleneck - premature optimization and all - b) they tend to turn your data model into spaghetti, which causes more and more pain as you get farther into development. [9:38am] jurajv: nickjohnson: yes, the index would be required 2 or more times.. once for sorting and then once for each equality filter - ok maybe if you guys look at that more closely later, you'll find some way how do to it cheap - it would be a killer feature, if possible to implement cheaper than custom index in the same situation [9:39am] moraes_: i saw that spaghetti and moved back [9:39am] dfabulich: Hi, I'm late so excuse me if anybody has already asked my question. I have a question about issue 1269, with 227 stars. (unique static ip per app) It's related to issue 792 (SSL/HTTPS Support on Google Apps domains, 244 stars) and issue 777 (Officially Support Naked Domains for GAE Apps, 303 stars). [9:39am] dan_google: DocSavage: Why not just put the key of the related entity on a property? Assuming you're fetching the first entity in the first place, you can have the key without putting the entities in the same entity group unnecessarily. [9:40am] nickjohnson: jurajv: But indexing a listproperty twice leads to exploding indexes [9:40am] ryan_google: jurajv: that's the key point. as soon as you include... [9:40am] ryan_google: yup. nick beat me to it. [9:40am] nickjohnson: In theory you could only include the smallest or largest value as you suggest, but App Engine doesn't know you're going to use that index only for sorting on, and not filtering [9:40am] dfabulich: Obviously, everyone's favorite question about filed bugs is "when are you going to address this?" But I have a subtler question. Am I right in thinking that you guys don't directly have the ability to address issue 1269? I get the feeling that putting more stars on this issue is barking up the wrong tree; it's the wrong department. [9:41am] ryan_google: dfabulich: correct. so far we've been waiting for SNI adoption, which is largely out of our control. [9:41am] Jason_Google: dfabulich: Regarding SSL on Google Apps domains, this is an issue that comes up consistently. We're limited by browser capabilities, but it's something we definitely want to add support for soon. Same with 777. [9:41am] Jason_Google: soon == eventually :) [9:41am] dfabulich: Obviously both of those issues could be addressed by addressing 1269. [9:41am] dfabulich: And SNI is never coming to IE6. :-) [9:41am] cringer: MSIE stinks. [9:41am] DocSavage: dan_google: Not sure what you mean. If you're trying to find some entity that has a certain property value, you can either query of it or do a get on some other Model that uses that property value as a key name. How were you planning to get the matching entity for a property value? [9:41am] ryan_google: nickjohnson: right, not to mention that when entities are inserted, deleted, or changed, we'd not only need to add their index rows, but determine which old index row(s) to remove or recreate [9:42am] ryan_google: which depends on all of the other relevant entities, which we don't know at write time [9:42am] scudder: dfabulich: static IPs would be pretty difficult due to the way that our system works [9:42am] nickjohnson: dfabulich: Assuming assigning static IPs to apps was easy, bear in mind that with a globally distributed project like app engine, you would either need _many_ IPs per app, or you'd lose all the benefits of having a distributed system in the first place. [9:42am] dfabulich: scudder: Yes, I understand that. Of course, I think some customers would be willing to pay extra for it, which may accomodate for that problem. [9:43am] scudder: I'm not saying it is impossible, and we're aware that quite a few people want this, but I doubt this would be soon [9:43am] dan_google: DocSavage: sorry I'm late to the conversation, I was asking why your suggested solution of using ancestors to make a key of a related entity available was preferable to putting the key on a property. [9:43am] dfabulich: So, the real question here is 1269. What can we do to make this service available at extra charge? This is a business decision, really, not a code decision. It seems like just putting stars on the issue and waiting patiently won't really get this addressed. [9:43am] maxoizo: Long wanted to ask - I see in roadmap "Alerting system for exceptions in your application" - what does it mean? I'm searching the answer on the forums, but nothing found [9:43am] ryan_google: nickjohnson, dfabulich: i'll second that. when scudder says pretty difficult, read almost impossible. (again, due to the way google infrastructure works internally.) [9:43am] nickjohnson: The other option is to drive IPv6 opportunity, since IPv6 addresses are pretty trivial to give away ;) [9:43am] nickjohnson: (Technically and politically, not necessarially practically, of course) [9:44am] dfabulich: ryan_google: Surely if I'm willing to pay for the service, you guys could set up a simple proxy on a static IP. [9:44am] nickjohnson: s/opportunity/adoption/. Don't know how that came out like that [9:44am] jurajv: basically i was coming from this line in the docs: "One important caveat is queries with both an equality filter and a sort order on a multi-valued property. In those queries, the sort order is disregarded." <-- so I thought, ok, what if the sort order was not disregarded:) [9:44am] ryan_google: dfabulich: as a one off, yes, with *significant* effort. but we couldn't scale it. [9:44am] ryan_google: jurajv: totally! [9:44am] ryan_google: if the indexes are set up right, you can do really cool things at query time [9:44am] dfabulich: ryan_google: So name your price! :-) [9:45am] ryan_google: dfabulich: if only we were a consulting organization... :P [9:45am] nickjohnson: dfabulich: If you're prepared to pay any price - why not rent your own servers to act as proxies? :) [9:45am] DocSavage: dan_google: my suggestion is a way to optimize searches for unique property values by using gets on key names vs queries, so no indexes are used, hence speed and better error rate. At the cost of spaghetti models (per ryan's remark) :-) [9:45am] ryan_google: jurajv: the difficulty is almost always figuring out how to make writes update the indexes to *keep* them in the interesting, useful state. often there's no efficient, low latency way. [9:45am] dan_google: DocSavage: ah, caught up now, thanks :) [9:45am] dan_google: DocSavage: so basically emulating BigTable's prefix searches using ancestors [9:46am] dan_google: DocSavage: (which is how BigTable finds ancestors in the first place) [9:46am] jurajv: ryan_google: yeah one needs to pre-compute a lot to have an efficient app, i'm still learning that [9:47am] DocSavage: dan_google: We App Engine users aren't allowed to soil BigTable directly, so I wouldn't know about that :-) [9:47am] dan_google: DocSavage: you can soil it readily enough. :) [9:47am] ryan_google: jurajv: at least the way we do it, yes [9:47am] nickjohnson: DocSavage: You probably wouldn't want to - the datastore adds a lot of value on top of Plain Old Bigtable. :) [9:47am] eighty_: question for a googler: on os x, eclipse 3.5, google eclipse plugin, i create a new web application and try to run it. i get this error: http://utilitybase.com/paste/16789 [9:48am] ryan_google: dan, docsavage: +1 to that. the "unique property => optimize queries to gets" feature is definitely doable in an end user library [9:48am] rdayal: eighty_: Checking it out now.. [9:48am] DocSavage: Question on CapabilityDisabledError handling: Is there any work being done on an API so we can test for Capability issues *before* hitting an exception due to maintenance, etc? [9:48am] eighty_: rdayal: thanks. i'm using GWT trunk which might be a factor. [9:48am] nickjohnson: DocSavage: You can - see google.appengine.api.capabilities [9:48am] ryan_google: and re implementing features in userland vs in the datastore itself, i'm going to blatantly plug a post i recently made to the group: http://groups.google.com/group/google-appengine/browse_thread/thread/6e1e8f9321f53775/8469e1cffdfa828c#8469e1cffdfa828c [9:49am] rdayal: eighty_: That is exactly the problem. There is an issue right now where xerces was bundled in with GWT trunk, which causes a problem with app engine. A CL is pending right now that should fix this issue. Let me find it. [9:49am] nickjohnson: ^ Good post, by the way. :) [9:49am] eighty_: rdayal: yay! thanks. :) [9:49am] jurajv: my biggest problem is how to handle/index/pre-store permutations of multiple filters I need to be able to test on, ie. User.roleA, User.roleB, User.[roleA+roleB] -- the last one is what causes the problem.. same problem as exploding indexes really [9:49am] cringer: Astronomers now believe that it's not a black hole at the center of the galaxy, but MSIE. [9:49am] xp_prg: ok hi, is this the getting help hour here? [9:50am] scudder: xp_prg: yes it is :) [9:50am] jurajv: maybe if app engine could do a bit-mask equality filters [9:50am] xp_prg: ok! [9:50am] jurajv: not equality, logical [9:50am] nickjohnson: jurajv: It does merge joins, if you're prepared to forgo sorting [9:50am] ryan_google: jurajv: interesting, i was talking about that with just the other day [9:50am] ryan_google: yup, what nick said [9:50am] jurajv: nickjohnson: that's the thing, i need the sorting :( [9:50am] xp_prg: I am in desperate need of help, I need to know how to read a private rsa key and use it to decrypt something with pycrypto, can you please assist me? [9:50am] Anarki: Hey app engine team :). QQ, we're planning on using GAE-J and i had a question from my research. I know you've probably answered this a 1000 times, but how do I make a query to return me more that 2000 records? [9:50am] ryan_google: storing multiple boolean values and using multiple equals filters will work [9:51am] rdayal: eighty_: The CL has not been committed as yet. If you want, update trunk to r5921 (the xerces dependency was added in r5922). That version should work for you. [9:51am] ryan_google: and if you enumerate each of the booleans (annoying and not always feasible, i know), then you can sort too [9:51am] xp_prg: I know I need to use pycrypto.RSA.construct [9:51am] eighty_: rdayal: SGTM. thanks again! :) [9:51am] rdayal: eighty_: From what I hear, the patch should be committed today. [9:51am] jurajv: ryan_google: that would require custom (and expensive) index again [9:51am] xp_prg: so who can hep me? [9:51am] eighty_: rdayal: sweet. [9:51am] scudder: xp_prg: I think I recall a discussion group thread on this, did you ask in the group recently? [9:51am] ryan_google: anarki: you can iterate over your query results instead of using fetch() [9:51am] xp_prg: yes [9:52am] ryan_google: in general, though, if you need thousands of query results per end user request, you probably need to rethink what you're doing [9:52am] ryan_google: jurajv: really? it should be just one composite index row, right? [9:52am] Anarki: ryan_google: there isn't a fetch() in Java right? [9:52am] nickjohnson: jurajv: No, the point is that if you use only equality filters you _don't_ need a custom index [9:52am] guest4: can you restrict the visibility of an app on GAE as private, in order to test the app before it goes public? [9:52am] eighty_: Anarki: http://code.google.com/appengine/articles/paging.html [9:52am] DocSavage: nickjohnson: google.appengine.api.capabilities -- didn't see that before. Was it always hidden in plain sight? [9:52am] scudder: xp_prg: ah OK, would it be alright if I follow up there, often with issues like this it helps to see some of your code, and that is especially difficult on IRC ;-) [9:53am] ryan_google: jurajv, nick: right, but if you sort, you should still be able to use a composite index, as long as you use many separate boolean properties and not a list property of booleans [9:53am] nickjohnson: DocSavage: It's only been there a couple of months or so. It's not documented (yet!), though [9:53am] xp_prg: ok, but can we do that now, I would be happy to pastebin what I have if you want [9:53am] scudder: yes, that would be great [9:53am] xp_prg: ok let me get it going one sec [9:53am] Anarki: ryan_google: well, ok, fair enough...let me rephrase...how would i get the 2001th record if the max limit is 1000 and the max offset is 1000? [9:53am] ryan_google: anarki: i believe the "low level" java datastore api has fetch. jdo/jpa probably too, but i'm not familiar with them. [9:53am] rdayal: eighty_: There's no external thread for this, but watch the GWT commits and you should see it go by. [9:54am] nickjohnson: xp_prg: Have you tried asking on pycrypto- related groups? [9:54am] xp_prg: yes [9:54am] eighty_: rdayal: k [9:54am] ryan_google: anarki: see the paging article eighty plugged. we also plan to release built-in paging soon too. [9:54am] ryan_google: but that's just paging, not random access [9:55am] moraes_: omg! built-in paging! [9:55am] moraes_: weee time! [9:55am] ryan_google: in return i expect a pony [9:55am] moraes_: weee! [9:55am] Anarki: heh [9:55am] DocSavage: nickjohnson: That api should be publicized. Would be particularly interested in elaboration of how CapabilitySet works: 'images', 'datastore_v3'... 'memcached'? [9:55am] ryan_google: i'm holding the paging code hostage until i get it [9:55am] ryan_google: pink please. with ruffles. [9:55am] eighty_: singing my pretty pony theme song to ryan_google [9:55am] dfabulich: Can I request a status update on bug http://code.google.com/p/googleappengine/issues/detail?id=1227 (Can't set 512 character cookies)? [9:55am] nickjohnson: DocSavage: Those are all service names, defined in the SDK - check out the appropriate *_stub modules [9:56am] DocSavage: I'd also like to plug issue 1811 so we can test capability disablings: http://code.google.com/p/googleappengine/issues/detail?id=1811 [9:56am] nickjohnson: DocSavage: That's something that could be fairly easily implemented with a stub such as I demonstrated in here the other day. Perhaps file a bug against the App Engine testing library for them to support it too? [9:57am] ryan_google: dfabulich: that should have been fixed in 1.2.4 [9:57am] xp_prg: http://pastebin.com/d2d89032f [9:57am] ryan_google: mind retrying? [9:57am] xp_prg: ok there it is [9:57am] ryan_google: we'll update the issue [9:57am] xp_prg: any help is very, very, very appreciated [9:57am] xp_prg: I am just looking for a pure python implementation to read in the key and decrypt with it [9:57am] koen_: Hey Nick/Ryan, any plans to make appengine instances listen to multiple domains at once, so we can offer a service people can run with their own domain names? [9:58am] moraes_: koen_: that's possible alreadu through google apps [9:58am] moraes_: *already [9:58am] dfabulich: ryan_google: Do I need to download anything to retry, or should the JSP "just work" if I refresh it? [9:58am] guest4: http://jira.icefaces.org/browse/ICE-4394 fyi [9:58am] kickingvegas: thanks everybody! [9:59am] koen_: moraes_: yeah but I want to create them automatically as we expect quite some clients [9:59am] nickjohnson: What moraes_ said. You simply add an app to multiple Apps domains, and check the Host header [9:59am] xp_prg: so is there a way to do with keycscar? [9:59am] xp_prg: I am looking for any solution at all for this in gae [9:59am] koen_: nickjohnson: is there a way to add them in an automated way? [9:59am] nickjohnson: You don't need to create an app for each; just use one instance of the app in multiple places. The only issue is with using Google Accounts for authentication [9:59am] jurajv: ryan_google: re boolean composite index: i've defined a 10 filter + 1 value index, but i could not store an entity using that index (too many indexed props) --- i will retry with booleans, as i tried with strings first [9:59am] nickjohnson: koen_: To add the mappings? Not that I'm aware of [10:00am] ryan_google: dfabulich: it looks like the bug was in production, not the sdk, so it should "just work" [10:00am] nickjohnson: But it's a very straightforward practice for your end users -and you can't do it without their credentials anyway [10:00am] ryan_google: jurajv: property type won't matter [10:00am] jcgregorio: koen_: No API available for that at this time [10:00am] ryan_google: but if any of those were list properties, that will definitely matter [10:00am] Jason_Google: xp_prg: This article has something that might be relevant: http://wiki.opensocial.org/index.php?title=Building_an_OpenSocial_App_with_Google_App_Engine [10:00am] koen_: so if the end user uses google apps they can link their owen domain to my appengine instance? [10:01am] jurajv: ryan_google: but good to know that it should be less expensive than a list property impl. - i thought it's about the same [10:01am] rdayal: Ok, I'm heading out. See you guys later! [10:01am] Jason_Google: xp_prg: See the bottom section on signing requests. It also links to a Python crypto library that is compatible with App Engine. [10:01am] jurajv: ryan_google: f asc, f asc, f asc, v asc vs. f(list prop) asc, v asc (list prop with 3 vals max) [10:01am] jcgregorio: koen_: yes [10:01am] dankles: you need to add each domain to your appengine app (click "Versions" in the panel) first tho [10:01am] koen_: cool, thanks [10:01am] eighty_: thanks rdayal! [10:01am] xp_prg: so nobody knows how to simply do this though? [10:01am] nickjohnson: koen_: correct [10:02am] rdayal: eighty_: anytime [10:02am] nickjohnson: dankles: No, that's not correct. [10:02am] nickjohnson: You don't need to add a domain in your app's control panel in order for someone using Apps to add your app. [10:02am] DocSavage: Any estimate on next python SDK release and likely next features? [10:02am] dankles: nickjohnson: no? that's how we are doing it & i thought we followed the instructions on the app engine site. [10:03am] Anarki: eighty_, ryan_google: sigh the page is for python : (...any idea for java? [10:03am] nickjohnson: dankles: There's two ways to add an app to a domain - either through your app's control panel, or through the Google Apps control panel [10:03am] Wooble: dankles: there are 2 ways to do it, one starting from App Engine and one starting from Google Apps... [10:03am] ryan_google: docsavage: definitely, we've started on the 1.2.5 cycle already [10:03am] nickjohnson: Both have the same result [10:03am] eighty_: Anarki: same principles should apply for java, no? [10:03am] Jason_Google: OK Everyone, that ends today's official Chat Time session. Feel free to continue chatting though. :) We'll be in here again two Wednesdays from now, from 7:00-8:00 p.m. PDT on September 2nd. [10:03am] scudder: well, it's a few minutes past the hour, so I'd like to call our hour long chat session to a close [10:03am] eighty_: thanks GAE team! [10:04am] Anarki: eighty_: will they? b'coz there's a lot more things in python that in the java environment [10:04am] Anarki: thanks GAE guys :) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
