Re: [google-appengine] Is it possible to get only some properties with NDB?
Entities are stored and retrieved as a single unit by the underlying datastore layer, so NDB, like all other APIs to it, retrieves and stores only complete entities. If you want to retrieve only parts, you'll have to split your model up into separate models, instead. Note that entities are limited to 1MB each, too, so 4 1MB properties are not possible. -Nick On Wed, Mar 28, 2012 at 9:51 PM, Kaan Soral kaanso...@gmail.com wrote: Lets say I have a model and it has numerous properties, with A,B,C,D properties, all being 1MB's, so total size is roughly ~4MB's, and other properties being insignificantly small. Is it possible to only fetch certain properties with entities? - For example - as it is - if I fetched 100 entities, it would require 400MB's of RAM [too much]. (I don't need A,B,C,D for that query result) I am trying to figure out If I should separate some properties to * different_models* to ease querying, or does NDB solve this problem with some built-in functions. (I have checked the documentation, not so deeply, couldn't find anything). I think discovering-learning functionalities like these are very important while designing a system, so I am trying to learn NDB in depth, for this reason any tips regarding NDB are also appreciated. (For example, what problems does ndb solve that are unsolvable or unpractical using db ) Thanks in Advance -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/V_wp9a9MTAUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Idle instances do not turn off automatically. Please help
What is your setting for min idle instances? -Nick On Sun, Nov 27, 2011 at 10:47 AM, sb shaneb...@gmail.com wrote: It doesn't behave this way at all. It just fires up a new instance once the resident instance get overloaded. The resident instance goes idle. The new instance then handles all the traffic, and when that gets overloaded another instance is created, while the original resident instance is just sitting their completely idle. So I am getting billed for three instances one of which is doing nothing and never will do anything until I manually intervene. On Nov 26, 6:20 pm, Nick Johnson nickjohn...@google.com wrote: On Sun, Nov 27, 2011 at 3:16 AM, WallyDD shaneb...@gmail.com wrote: Hi Nick, Thanks for responding and looking into this. See my response below; On Nov 24, 8:57 pm, Nick Johnson nickjohn...@google.com wrote: On Thu, Nov 24, 2011 at 5:48 PM, WallyDD shaneb...@gmail.com wrote: I am having something of similar problem with instances not turning off. This isn't a problem - you're not being charged for those instances. You'll only be charged if demand requires sending traffic to them, in which case you've been saved the overhead of starting up a new instance. I am very much being charged for these instances. The resident instances do nothing and stay idle while other instances serve. Not entirely sure if it is related to your issue. If you've specified a 'min idle instances' greater than 0, then this is behaving as documented. The point of requesting idle instances is to handle sudden increases in traffic volume while more instances are being spun up in the background; naturally this means that they have to remain idle while waiting for a traffic spike that will require them. The second instance fires up when traffic overloads the resident instance(s). The new dynamic instance(s) then stay on, permanently. So I get charged for both instances, one of which does nothing. You set your min idle instances to 1; therefore, 1 instance will be idle, in order to handle traffic spikes. If you don't want this behaviour, set min idle instances to 0. -Nick Johnson If it is behaving as documented, which part of the documentation should I be looking at? - sb -Nick Johnson -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Idle instances do not turn off automatically. Please help
On Sun, Nov 27, 2011 at 3:16 AM, WallyDD shaneb...@gmail.com wrote: Hi Nick, Thanks for responding and looking into this. See my response below; On Nov 24, 8:57 pm, Nick Johnson nickjohn...@google.com wrote: On Thu, Nov 24, 2011 at 5:48 PM, WallyDD shaneb...@gmail.com wrote: I am having something of similar problem with instances not turning off. This isn't a problem - you're not being charged for those instances. You'll only be charged if demand requires sending traffic to them, in which case you've been saved the overhead of starting up a new instance. I am very much being charged for these instances. The resident instances do nothing and stay idle while other instances serve. Not entirely sure if it is related to your issue. If you've specified a 'min idle instances' greater than 0, then this is behaving as documented. The point of requesting idle instances is to handle sudden increases in traffic volume while more instances are being spun up in the background; naturally this means that they have to remain idle while waiting for a traffic spike that will require them. The second instance fires up when traffic overloads the resident instance(s). The new dynamic instance(s) then stay on, permanently. So I get charged for both instances, one of which does nothing. You set your min idle instances to 1; therefore, 1 instance will be idle, in order to handle traffic spikes. If you don't want this behaviour, set min idle instances to 0. -Nick Johnson If it is behaving as documented, which part of the documentation should I be looking at? - sb -Nick Johnson -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Potential Feature Request Memcache Allocation and Delayed Writes
Hi Brandon, On Thu, Nov 24, 2011 at 8:05 PM, Brandon Wirtz drak...@digerat.com wrote: Realizing my app is different than from most everybody elses…. ** ** I got to thinking about the thread where we were talking about reading keys from memcache. I know all the reasons this is a bad idea. But I got to thinking what if it wasn’t? ** ** My app is just a big optimized cache, but I rely on the 3 tiers of storage to make it all work and do so quickly. The sum of all the data for the day is about 2 gigs. In a virtual machine environment I would typically allocate a bunch of ram and every so often dump that to longer term storage, but since most my caching is measured in minutes, some is in days, and the longest I ever care about data is a month… the only reason I need long term storage is so that when the memory gets reset, or a new “instance” comes online that I don’t have a 100% miss rate. The system you describe won't evict data unless you explicitly choose to do so - you are in full control of cache eviction. This is not the case for memcache, where memcache may choose to evict items without notice. I don't think your proposal really applies to memcache, since by the time you do your 'dump', some indeterminate amount of your original data will have already been evicted from the cache. A much better approach if you don't mind data loss is a write-behind cache, where you schedule a task to write the data to the datastore after each time you update it in memcache. -Nick Johnson ** ** Why can’t I do that with Memcache? Allocate 2 gigs, populate it with data only on a version change. Once a day take all the values and dump them back to datastore so that if the world ends that I don’t have to start from nothing. (maybe only write all the values that have an expiration so many hours away) Since Backends share Memcache this “long” operation could be a scheduled task and execute in the background. ** ** In my case this would save a lot of cycles since my writes are Local Memory, MemCache, Datastore. And I do so with every piece of data because I can’t count on getting a hit from Memory or MemCache because of their volatility. ** ** But if I had a set amount of Memcache I wouldn’t need to worry, it wouldn’t be volatile, and Google Could charge me for the resource. Doesn’t even have to be perfectly non-volatile because even if I only “back-up” 75% of the data that’s fine it is just a cache. ** ** ** ** ** ** ** ** ** ** -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Idle instances do not turn off automatically. Please help
On Thu, Nov 24, 2011 at 5:48 PM, WallyDD shaneb...@gmail.com wrote: I am having something of similar problem with instances not turning off. This isn't a problem - you're not being charged for those instances. You'll only be charged if demand requires sending traffic to them, in which case you've been saved the overhead of starting up a new instance. The resident instances do nothing and stay idle while other instances serve. Not entirely sure if it is related to your issue. If you've specified a 'min idle instances' greater than 0, then this is behaving as documented. The point of requesting idle instances is to handle sudden increases in traffic volume while more instances are being spun up in the background; naturally this means that they have to remain idle while waiting for a traffic spike that will require them. -Nick Johnson I get charged for mine :( On Nov 23, 6:30 am, Ice13ill andrei.fifi...@gmail.com wrote: I deployed yesterday a version of my app for testing purposes. I checked the number of instances on that version today and I noticed that there were 5 instances active, each one with 24h of age (aprox.). But I didn't do any tests in the last 15-16 hours. I also checked if there the number of requests changed in the last 15-20 minutes and it didn't I also noticed that on my default version, i have about 5 instances with 1h20m each... lets say a total of about 6.5 h for the last 1.5 hours (real time). But my dashboard shows a total of 4.6 hours. (the counter was reset 3.5 hours ago, relative to the time when i'm writing this post). Until today, everything was fine. I set the number of idle instances to 1 (max idle instances, about a week and a half ago) and on my default version were 2, or 3 maximum instances active. It's like the instance scheduler is not working correctly. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] datastore write operations quota mismatch
Hi Jia Jun, Writing one datastore entity may require many writes. Alfred gives a good summary of this here: https://groups.google.com/d/msg/google-appengine/mjnSqQWOfqU/cgPVeHbrR8oJ -Nick Johnson On Fri, Nov 18, 2011 at 11:43 PM, Jia Jun Tan jiajun@gmail.com wrote: My app had been running beyond the 50k free datastore put limit these few days. So i examined the numbers today. Using appstat and the dashboard, i manually calculated the number of writes for 4 hours since the quotas resetted. I had 4 paths that generated datastore_v3.Put: 22(writes)*41(requests) = 902 4(writes)*36(requests) = 144 2(writes)*41(requests) = 82 12(writes)*30(requests) = 360 this totals to 1488 datastore write operations, which is a far cry from the 0.03 of 0.05 Million Ops that the dashboard says i used. is there a bug somewhere in the system? Regards, Jia Jun -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2zWmuWPQbyAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Multithreading question
Hi Richard, Consider using NDB, and its coroutine-based parallelism. It's pretty much exactly built for this situation: if you have multiple coroutines, each one will be run until it attempts to wait on an RPC (such as a datastore or memcache operation), then the requests will all be automatically batched together and executed as efficiently as possible. Thus, you get the benefits of get_multi, without the complexity of restructuring your code for it. -Nick Johnson On Wed, Nov 23, 2011 at 7:18 AM, Richard Arrano rickarr...@gmail.comwrote: Hello, Quick question regarding multithreading in Python 2.7: I have some requests that call 2-3 functions that call the memcache in each function. It would be possible but quite complicated to just use get_multi, and I was wondering if I could simply put each function into a thread and run the 2-3 threads to achieve some parallelism. Would this work or am I misunderstood about what we can and cannot do with regards to multithreading in 2.7? Thanks, Richard -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Using RAM instead of datastore - any limits?
Weak references may not work as you expect in Python. Python uses both reference counting and garbage collection; if the reference count of an object goes to 0, it will be freed immediately, instead of waiting for garbage collection. As a result, your cache may well be empty most or all of the time. -Nick Johnson On Tue, Nov 22, 2011 at 12:46 AM, Joshua Smith joshuaesm...@charter.netwrote: When building in-memory caches, it's typical to use a weak reference system (like a WeakHashMap in Java) so you don't have to rely on heuristics for how much memory you should use. Googling around, I found a thing called WeakValueDictionary in python. Anyone here have experience using one of these in GAE? On Nov 20, 2011, at 11:26 PM, Brandon Wirtz wrote: You get an amount of ram close to but not always equal to 128M or ram PER instance. ** ** Python 2.7 uses more memory for Hello World, and less for most operations. They both use the same for storing things like Data Caches. ** ** You can use the local instance in addition to data store. Not instead. My apps waterfall from edge cache to instance memory to memcache to datastore Use all the ram you can, it is free. Don’t count on it being there, don’t over use it, and stick to the API’s and Libraries for accessing it, or the world will end violently. No, I won’t share code for doing this it is our biggest selling point. ** ** image001.jpg ** ** -Original Message- From: google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] On Behalf Of JH Sent: Sunday, November 20, 2011 5:17 PM To: Google App Engine Subject: [google-appengine] Re: Using RAM instead of datastore - any limits? ** ** You get 128 megs of ram for front end instances. Also, so far my experience says that py 2.7 uses quite a bit more ram just to run hello world. ** ** On Nov 20, 5:58 pm, ThePiachu thepia...@gmail.com wrote: My application relies on accessing a lot of simple stored data and displaying it. I'm considering storing all data in the RAM of the application in order not to have problems with datastore access quotas, but I'm not sure if there are any limits of how much data can*** * be stored this way. Is there any limit on how much data can one store*** * in say, a vector in RAM? ** ** -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com.*** * To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. ** ** -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Want my money back, What should I do?
On Tue, Nov 22, 2011 at 4:49 AM, reco r...@nex9.com wrote: hi brandon, we are on app engine since day one. its a really nice platform even though we had a lot of issues through out the past with outages and and the apps not behaving as they should. I understand that you guys need to make money with this platform. I think this is great. Actually i love you guys to make money so there will be future enhancements, fixes, To clarify, Brandon does not work for Google, and does not speak on behalf of Google. -Nick Johnson the issue we see though atm its that you are forcing us to fast to move on. running an app on 2.5 even on HR is from business standpoints not worth what you are charting for. i am sure things will change with python 2.7 and multithreading. the SDK to work on this is out there since around 2 weeks. we are in the process of moving to a multithreaded version of our app which will hopefully result in less instances used. i suggest that you guys should expand the 50% discount for another 2-3 months. dec 1st is just to early. you get this from a very happy GAE developer. best, christof -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] List of Memcache Items
No, there's no API for this. If you're trying to track all memcache keys, you're probably Doing It Wrong. -Nick Johnson On Wed, Nov 23, 2011 at 3:44 AM, James Gilliam jimgill...@gmail.com wrote: Any way to get a list of the memcache keys without having to track them yourself? Thanks you -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Want my money back, What should I do?
On Wed, Nov 23, 2011 at 11:51 AM, Brandon Wirtz drak...@digerat.com wrote: I hope I have never said anything to imply I work for Google. I have tried for so long to make sure that never happens. Not that I've seen; I was responding to reco's post where he seemed to assume that you did, and I wanted to clarify. -Nick Johnson ** ** I specifically don’t use a Gmail address or anything to associate me with Google. I may go back to putting my sig on the list so that there is less confusion. ** ** ** ** *Brandon Wirtz *BlackWaterOps: President / Lead Mercenary [image: Description: http://www.linkedin.com/img/signature/bg_slate_385x42.jpg] *Work:* 510-992-6548 *Toll Free:* 866-400-4536 *IM:* drak...@gmail.com (Google Talk) *Skype:* drakegreene BlackWater Ops http://www.blackwaterops.com/ ** ** ** ** ** ** ** ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *Kaan Soral *Sent:* Sunday, November 20, 2011 9:46 AM *To:* google-appengine@googlegroups.com *Subject:* [google-appengine] Want my money back, What should I do? ** ** As soon as the new billing rolled out, I made the necessary changes to switch to 2.7 with concurrency and launched my new version. But because of the bugs I had to roll back to a version with 2.5 on it. (Here is a fatal bug: http://code.google.com/p/googleappengine/issues/detail?id=6282 ) (As a side problem: I already modified my code, and I don't want to roll back my code [too many modifications], for this reason I froze my app to the version with python 2.5, so this situation froze my development and the ability to make changes to the live version too ) Before new billing I was paying ~40$s a day, now I am paying ~90$s. so until 1.6.1 gets launched I will pay ~50$s extra (maybe less, maybe more) So I feel like an idiot now Should I submit a billing ticket for this, any advices? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/PlIMAtqo3IsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. image001.jpg
Re: [google-appengine] Query returning about 2000 results?
Hi Kyle, It sounds like you generally fetch a schematic all at once. Rather than storing them as separate datastore objects, why not serialize them all as a single schematic object, which you fetch and store as a single unit? -Nick Johnson On Tue, Nov 22, 2011 at 6:58 AM, Kyle Anderson kyleanderso...@gmail.comwrote: I run an application which manages large grid schematics. A schematic has about 2000 grid elements (transformers, power lines, voltage regulators, etc). There are about 40 different types of grid elements, which are mostly set in stone. Each grid element has about 40 attribute (voltage rating, install date, etc). The attributes are not necessarily fixed (attributes could be added semi-frequently). Here is are some example rows for a single schematic (schematicId=1) which contains a Transformer, a PowerLine, and a Regulator ElementId, SchematicId, ElementType, ElementAttributes Example rows: 1, 1, Transformer, voltage5/voltageinstall_date5/1/2011/ install_date 2, 1, PowerLine, voltage5/voltageinstall_date5/1/2011/ install_date 3, 1, Regulator, rating5KW/ratinginstall_date5/1/2011/ install_date When the user wants to a load a schematic, I simply query this table for all entries with the associated schematicId (this returns about 2000 rows). The problem is that this process takes about 2 minutes. (even though the data is only about 10MB). Suggestions for improvement? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Git Deploy
Hi Roch, I presume you're referring to Git's ability to send patches and only transfer file modifications. In fact, App Engine's deploy process already does this; whenever you deploy, only files that have never been seen before are uploaded. For instance, if you were to download a popular open-source app, modify the configuration file, and upload your app, likely only the configuration file and your app's configuration would be uploaded when you do this. App Engine doesn't upload diffs in the way git does, but this isn't practical given that App Engine can't rely on having other versions of the code to base a diff on. -Nick Johnson On Thu, Nov 17, 2011 at 10:10 PM, Roch Delsalle rdelsa...@gmail.com wrote: Hi, I would like to know why Google App Engine isn't proposing an alternative deploy workflow using Git. That would save a lot of bandwidth. Roch -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime
Hi Waleed, It's impossible to say what the cause of the latency might be without more details. You say you spent a lot of time investigating it; can you show us some of the data you gathered? Did you use appstats? -Nick Johnson On Wed, Nov 16, 2011 at 11:37 AM, Waleed Abdulla wal...@ninua.com wrote: I just switched one of my apps to Python 2.7 and the average latency went up from 125ms to about 1.3 seconds (about 10x). My app is 'not' CPU-bound. It has a mix of different use patterns (some requests use urlfetch, others load data from datastore or memcache). That variety should make it a good candidate to benefit from multi-threading, but that doesn't seem to be working as expected. While the number of instances went down from 9 instances to 6, the 10X increase in latency is not acceptable. I spent a lot of time investigating it, and finally changed threadsafe to false and that fixed the problem (and, of course, raised my instance count to it's original level). Like the original poster, I was hoping that multithreading will reduce my instance cost, but that plan failed. Am I doing something wrong? Or is multi-threading not ready for live traffic yet? Waleed app id: bitpixelshr On Tue, Nov 15, 2011 at 3:10 AM, Jeff Schnitzer j...@infohazard.orgwrote: Dunno about the Python library, but the standard Java jBCrypt library uses a random salt per-user by default. I'm also very suspicious of this idea that the attacker doesn't have the source code. Python is trivially easy to decompile. Also, where do you keep your source code? In Github? Opens up a whole new set of attack vectors, including disgruntled Github employees. Jeff On Mon, Nov 14, 2011 at 10:52 PM, Brandon Wirtz drak...@digerat.comwrote: Nick, ** ** I agree, that my threat model assumes they didn’t get my source code. That “Somebody else’s problem” works under the assumption people are going to get my data, not my source code because I don’t ever write my own DB server code I am stuck using someone else’s which means the vulnerability that I am most likely to face is that somebody else’s screw up will be where my problem lies. ** ** Granted this is a better strategy if you are running compiled code, since my code lives on the Google Server I’m at the mercy of Google’s Security, where as if I were running compiled code it would be less likely someone would get the code. ** ** I would say that unique salt per user, is a good thing. The most common way to attack a large password database is to look at the most common entries and compare against the most common passwords from other sources. If you know the 15 most used passwords and the 15 most often occurring database results you are a long ways towards knowing what those 15 values are and calculating the salt. You aren’t crunching millions of combinations you are crunching 1000’s and once you have the salt, you take your already deciphered list of the most common passwords and you calculate the top 5k using bcrypt and you now have about 50% of the data in fewer than 10k operations. ** ** Compare that with my scenario. You have data. You don’t have the source code. The UserID or other “spoiler” is in every salt so the reoccurrence of a hash doesn’t correspond to a duplicate password, and now the computation is nearly impossible even if you have the source code, because you have to calculate every value for every user anyway. ** ** Would Brcypt(Pass+UserID+Salt) be the best? Yes. But MD5(Pass+UserID+Salt) is going to still going to be orders of magnitude more difficult than Bcrypt(Pass+salt), because I can’t use knowledge of frequency tables to predict likely outcomes or detect duplicate passwords. ** ** -Brandon ** ** ** ** ** ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson *Sent:* Monday, November 14, 2011 6:21 PM *To:* google-appengine@googlegroups.com *Subject:* Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime ** ** Hi Brandon, ** ** What you say is fine if your threat model only includes script kiddies who don't have your source code. If either of those is not true - you have an adversary with some level of independent skill, or your source code is compromised - any method that relies on obscurity for its security will fare very poorly. ** ** One thing to bear in mind is that if your app is ever compromised, your password database and/or source may be posted publicly; at that point, you no longer have to worry about just the initial attacker, but anyone with sufficient motivation. ** ** Of course, using federated login like OpenID or the Users API obviates the need to store passwords at all, making it Someone Else's Problem. :) ** ** -Nick On Tue, Nov 15, 2011 at 1:07 PM, Brandon Wirtz drak
Re: [google-appengine] Question about Write Ops cost for updating an entity with changed properties
Hi Alex, The datastore compares old values to new; there's no way for the SDK to mark properties as modified or not. -Nick Johnson On Tue, Nov 15, 2011 at 9:49 AM, Alex Epshteyn alexander.epsht...@gmail.com wrote: Alfred wrote an excellent explanation of how write ops are charged ( http://groups.google.com/group/google-appengine/msg/ca47eb7678d50372? ). My question is how does the the datastore determine whether a property has actually changed - does it compare the new value to the saved value or does the python db.Model class mark a property as changed every time it's assigned (even if the values are the same). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime
No! Please, please don't do this. Obscurity is no substitute for security. 1) Bcrypt or similar is not 'overkill' no matter who you are. Users reuse passwords, and they're entitled to the best protection you can reasonably provide them. 2) Bcrypt is not there to protect against online attacks, it's there to protect against offline attacks, where an attacker obtains your hashed and salted passwords. 3) Doing something weird is security through obscurity. Do not base your security on your attacker not knowing what you did. Really, really don't just concatenate salts to the beginning or end of the password. 4) Both MD5 and SHA1 are merkle-damgard construction hashes ( http://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction). As a result, the concatenation of several hashes is no more secure than the most secure of the individual hashes. -Nick Johnson On Sun, Nov 13, 2011 at 2:58 PM, Brandon Wirtz drak...@digerat.com wrote: Unless you are protecting Medical records bcrypt is overkill if you do some reasonably smart things like Failed logins from IP 9 Or, if you just do something weird to the password BEFORE you SHA it. Like interleave the user name in the password, Salt1 + UpSaEsRsNwAoMrEd + Salt2 Or Pick 2 Hash's SHA(pass) + Md5(pass) Don't want to store all that string length? Odd Characters from Sha(Pass+salt) + Even Characters from MD5(Pass+Salt) Uniqueness of the method is more important than the method. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Brian Quinlan Sent: Saturday, November 12, 2011 6:58 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime Hi Pol, On Sun, Nov 13, 2011 at 1:48 PM, Pol p...@everpix.net wrote: Hi, Since switching to 2.7 runtime, logging in to http://www.everpix.com went from about a second to anywhere from 15s to 60s. I tracked it down to this single password checking line: from bcrypt import bcrypt bcrypt.hashpw(password, self.password_hash) == self.password_hash What value are you using for threadsafe in your app.yaml? How large is self.password_hash? Cheers, Brian This comes from a native Python implementation of the py-bcrypt package from http://www.mindrot.org/projects/py-bcrypt/; grabbed from here: https://github.com/erlichmen/py-bcrypt. So what's happening here and how can we fix this? Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Can I use set in Appengine?
You don't need to import Sets in Python 2.5. -Nick On Tue, Nov 15, 2011 at 2:51 AM, Tim Hoffman zutes...@gmail.com wrote: Hi If your using Python 2.5 then you need to import the sets module. from sets import Set Set([1,2,1,2,4]) Set([1, 2, 4]) This code will also work in 2.7 Though in 2.7 you have the set type builtin. Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jbZddnsDuQgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime
Hi Brandon, What you say is fine if your threat model only includes script kiddies who don't have your source code. If either of those is not true - you have an adversary with some level of independent skill, or your source code is compromised - any method that relies on obscurity for its security will fare very poorly. One thing to bear in mind is that if your app is ever compromised, your password database and/or source may be posted publicly; at that point, you no longer have to worry about just the initial attacker, but anyone with sufficient motivation. Of course, using federated login like OpenID or the Users API obviates the need to store passwords at all, making it Someone Else's Problem. :) -Nick On Tue, Nov 15, 2011 at 1:07 PM, Brandon Wirtz drak...@digerat.com wrote: If I know your salt, I can “De-Hash” bcrypts faster than I can any of the “weird” combinations. Because there are libraries for doing so on ATI cards. ** ** If you do something weird a script kiddie can’t just pull code off the web and attack it. ** ** You want to see who can offline crack a set of 1M users? Your bcrypt list vs my “Weird” You don’t even have to give me the salt I’ll have 10k of those cracked in the first 72 hours. 10 to 1 odds you won’t get through mine without my source code in my life time. ** ** -Brandon Wirtz ** ** PS I don’t usually do the “trust me I’m far more evil” but FBI, Homeland Security, and the CIA have been to my doorstep for things I have defeated, documented, or built to keep from being defeated. The first time I was in 3 rd grade. ** ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson *Sent:* Monday, November 14, 2011 3:56 PM *To:* google-appengine@googlegroups.com *Subject:* Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime ** ** No! Please, please don't do this. Obscurity is no substitute for security. ** ** 1) Bcrypt or similar is not 'overkill' no matter who you are. Users reuse passwords, and they're entitled to the best protection you can reasonably provide them. 2) Bcrypt is not there to protect against online attacks, it's there to protect against offline attacks, where an attacker obtains your hashed and salted passwords. 3) Doing something weird is security through obscurity. Do not base your security on your attacker not knowing what you did. Really, really don't just concatenate salts to the beginning or end of the password. 4) Both MD5 and SHA1 are merkle-damgard construction hashes ( http://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction). As a result, the concatenation of several hashes is no more secure than the most secure of the individual hashes. ** ** -Nick Johnson ** ** On Sun, Nov 13, 2011 at 2:58 PM, Brandon Wirtz drak...@digerat.com wrote: Unless you are protecting Medical records bcrypt is overkill if you do some reasonably smart things like Failed logins from IP 9 Or, if you just do something weird to the password BEFORE you SHA it. Like interleave the user name in the password, Salt1 + UpSaEsRsNwAoMrEd + Salt2 Or Pick 2 Hash's SHA(pass) + Md5(pass) Don't want to store all that string length? Odd Characters from Sha(Pass+salt) + Even Characters from MD5(Pass+Salt) Uniqueness of the method is more important than the method. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Brian Quinlan Sent: Saturday, November 12, 2011 6:58 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime Hi Pol, On Sun, Nov 13, 2011 at 1:48 PM, Pol p...@everpix.net wrote: Hi, Since switching to 2.7 runtime, logging in to http://www.everpix.com went from about a second to anywhere from 15s to 60s. I tracked it down to this single password checking line: from bcrypt import bcrypt bcrypt.hashpw(password, self.password_hash) == self.password_hash What value are you using for threadsafe in your app.yaml? How large is self.password_hash? Cheers, Brian This comes from a native Python implementation of the py-bcrypt package from http://www.mindrot.org/projects/py-bcrypt/; grabbed from here: https://github.com/erlichmen/py-bcrypt. So what's happening here and how can we fix this? Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed
Re: [google-appengine] how to override skip_files on local dev server
Hi Andreas, The dev_appserver attempts to emulate the production environment in every way possible; we don't document each case explicitly because it's the default behaviour. -Nick Johnson On Thu, Nov 10, 2011 at 12:51 PM, andreas schmid a.schmi...@gmail.comwrote: nick i understand what you are trying to say but i still think such things should be documented here http://code.google.com/appengine/docs/python/config/appconfig.html#Skipping_Files where the only thing said in the docs about the consequences of the skip_files configuration is: The skip_files element specifies which files in the application directory are not to be uploaded to App Engine. The value is either a regular expression, or a list of regular expressions. Any filename that matches any of the regular expression is omitted from the list of files to upload when the application is uploaded. this could avoid developers to spend hours to figure out whats wrong in their code. andreas On Nov 9, 2011, at 7:46 PM, Nick Johnson wrote: On Thu, Nov 10, 2011 at 2:53 AM, andreas schmid a.schmi...@gmail.comwrote: i know its not an issue but what is the reason for blocking access to local files? in my case i have a dev environment and whats going to be deployed and i don't see the reason why i have to deploy those files too or switch back and forth in the skip_files section. no docs say that adding stuff to the skip_files section will affect access to those files. The dev_appserver attempts to emulate the production environment as closely as possible. skipped files aren't available in production, so it makes them unavailable in dev. If you need access to files that aren't uploaded, you should write a script that runs outside the dev_appserver, rather than trying to work around the restrictions, which are designed to help you test your app in as realistic an environment as possible. -Nick Johnson On Nov 9, 2011, at 10:19 AM, Jacob Gur wrote: Now that GAE has increased file deployment limit from 3,000 to 10,000, this should not be an issue for most people. On Wed, Nov 9, 2011 at 10:16 AM, andreas schmid a.schmi...@gmail.comwrote: have the same issue but i use different folders depending on local development or deployed app. i think the sdk should not block access to files. skip files should be there to avoid deploying those files and thats it. On Jul 15, 2011, at 4:42 PM, Robert Kluin wrote: Hey Jscob, It sounds like you're either dangerously close to the limit or have a frightening number of js files. Perhaps you should write a build script to handle making the mods you need or removing the files? Robert On Friday, July 15, 2011, Jacob G jacobs...@gmail.com wrote: To clarify, I don't want these file uploaded at all to GAE, because I'll go over 3000 file limit. On Jul 15, 3:17 pm, Jacob G jacobs...@gmail.com wrote: skip_files seems to block access on local dev server. How do override that? I need it for minifying javascript, but having access to original files on dev server for debugging. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- -- Robert Kluin Ezox Systems, LLC -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google
Re: [google-appengine] how to override skip_files on local dev server
On Thu, Nov 10, 2011 at 2:53 AM, andreas schmid a.schmi...@gmail.comwrote: i know its not an issue but what is the reason for blocking access to local files? in my case i have a dev environment and whats going to be deployed and i don't see the reason why i have to deploy those files too or switch back and forth in the skip_files section. no docs say that adding stuff to the skip_files section will affect access to those files. The dev_appserver attempts to emulate the production environment as closely as possible. skipped files aren't available in production, so it makes them unavailable in dev. If you need access to files that aren't uploaded, you should write a script that runs outside the dev_appserver, rather than trying to work around the restrictions, which are designed to help you test your app in as realistic an environment as possible. -Nick Johnson On Nov 9, 2011, at 10:19 AM, Jacob Gur wrote: Now that GAE has increased file deployment limit from 3,000 to 10,000, this should not be an issue for most people. On Wed, Nov 9, 2011 at 10:16 AM, andreas schmid a.schmi...@gmail.comwrote: have the same issue but i use different folders depending on local development or deployed app. i think the sdk should not block access to files. skip files should be there to avoid deploying those files and thats it. On Jul 15, 2011, at 4:42 PM, Robert Kluin wrote: Hey Jscob, It sounds like you're either dangerously close to the limit or have a frightening number of js files. Perhaps you should write a build script to handle making the mods you need or removing the files? Robert On Friday, July 15, 2011, Jacob G jacobs...@gmail.com wrote: To clarify, I don't want these file uploaded at all to GAE, because I'll go over 3000 file limit. On Jul 15, 3:17 pm, Jacob G jacobs...@gmail.com wrote: skip_files seems to block access on local dev server. How do override that? I need it for minifying javascript, but having access to original files on dev server for debugging. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- -- Robert Kluin Ezox Systems, LLC -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] memcache quota limits
Hi Vlad, Like all APIs, memcache has 'sanity limits', which can be lifted on an app-by-app basis if your app encounters them. Nothing has changed in this respect with the new billing. -Nick Johnson On Wed, Nov 9, 2011 at 11:07 AM, vlad vlad.troyan...@gmail.com wrote: I would like to get a confirmation. It appears that under new billing memcache API limits are lifted for all apps. In effect memcache use is unlimited. Is that correct? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/hcKwTosNRPMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Flushing async requests on development server for unit testing
Hi Lenny, You should always wait for async requests to finish, including in production. If you don't explicitly wait yourself, the appserver will wait for them to complete before returning the response to the user anyway. -Nick Johnson On Wed, Nov 9, 2011 at 7:49 AM, Lenny Rachitsky lenny...@gmail.com wrote: I'm finding that unit testing async based requests is rather tricky, as the put() often doesn't finish by the time the unit test is run. Is there any way to do a system-wide flush of all async requests on the development server? The only other solution I have is to add if branch that waits for the results to finish if it's on the dev server. Any advice would be much appreciated. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] app.yaml skip_files blocks access to local resources
Hi Andreas, The appserver deliberately does its best to emulate the production environment; this includes prohibiting access to files your app will not have in production. If you want to access files that won't be available in production, you should probably write a separate script and run it outside the dev_appserver. -Nick Johnson On Wed, Nov 9, 2011 at 6:43 AM, andreas schmid a.schmi...@gmail.com wrote: hi, in my app i need local resources which i user only locally while developing and don't want to upload them on deploy. thats why i have this specific folder listed in skip_files. but since now (probably one of the latest SDK releases) this makes the SDK ignore those files, which is pretty uncool. skip_files should only avoid to upload those files, not block the access to it. is there any reason why this is blocking access to those files? thx -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] copying data from appengine datastore admin to my dev server. failed with status 401
This is unlikely to work, and you should never ever expose your dev_appserver publicly. Consider making a small reusable test dataset you can easily load, rather than relying on a complete dump of production data, which won't scale. -Nick On Wed, Nov 9, 2011 at 4:31 PM, Mariano Benitez marianob...@gmail.comwrote: Hello, I am trying to copy data from my appengine to my dev server, instead of doing the normal bulk download/upload, I tried the datastore admin copy function. I've setup the dev server public ip so it can be seen from outside. When I try to run it from appengine, it fails with status 401, not authorized. Is it possible to use this anyway? seems like a nice way to populate my dev database. Thanks, -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/eJsdLLfmtkwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Memcaching BlobProperty
Hi Melissa, It's likely your data is simply being evicted as not worth the memory it takes up - you can fit a lot of smaller memcache values in the space a blob takes up. Consider just setting the correct caching headers in your response, instead. This will result in frontends, proxies and browsers caching your content, without using up precious memcache space. -Nick Johnson On Fri, Nov 4, 2011 at 3:32 PM, Melissa Hong carra.mel...@gmail.com wrote: Hi, I just installed appstats and noticed that I'm hitting the database for images all the time instead of the cache. In my image server I pull the image data from the blobstore to serve and use memcache to store the blob data for later use. Using logging, I've determined that it is setting the memcache with the image data, but the next time I go to grab this image memcache returns None. My theories are: 1. Memcache doesn't like BlobProperty values 2. My image data is 1MB (however I've wrapped my memcache in a try/ catch block so I would have thought it would be throwing errors if this was the case...) Any idea on why memcache seems to be failing? Thanks :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] NO? App Engine Chat time?!
Hi John, Sorry for the no-show! We're all very busy travelling around to GDDs, building demos, and so forth, and I think it just slipped off our calendars for once. Rest assured we should be back to our regular schedule for the next chat. -Nick On Thu, Nov 3, 2011 at 1:29 PM, John jwb...@gmail.com wrote: It's the first Wednesday, but there were no appengine folks on irc:// irc.freenode.net/#appengine Is this venue defunct? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/pPbna_wSz6sJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Retrieving the index position of an entity
Good post, David! Just one thing to add - there is in fact an existing library that already implements this: http://code.google.com/p/google-app-engine-ranklist/ -Nick Johnson On Thu, Nov 3, 2011 at 12:54 AM, David Whittaker dpwhitta...@gmail.comwrote: I doubt BigTable keeps track of index positions... it seems from its architecture (distributed sorted list of keys) that it would require updating every entity in the table on every insert to keep track of each entity's position. So, the question becomes: do you really need this information, or can your application be restructured to use the index key itself for whatever purpose you have in mind for the index key's position? I can think of at least one example where you might really need this information: a ranking system that ranks a huge list of entities based on where they fall in the sorted list of some key (probably a numerical strength value calculated from the rest of the entity's properties). Let's say the entities represent Users on some competitive flash gaming site, and JoeNewUser, who has been a member of the site for 3 weeks and played 734 games of Tower Defense, wants to know what his rank is. Joe's rank is 73,248, but we don't want to have to look at the 73,247 people ahead of him (sorted by TotalScore DESC) to find that out. If your actual use case has nothing in common with this, let me know, but I think it at least parallels the general gist of what you are asking for. What you are looking for is a B-Tree, with the addition of subtree size in the subtree pointers: http://en.wikipedia.org/wiki/B-tree The data model for that might look something like this: class BTree(Model): totalScores = ListProperty(int) #a sorted list totalScores, up to some limit long user = ListProperty(Key(User)) #same length as totalScores, points to the User the score belongs to pointers = ListProperty(Key(BTree)) #pointers to child Nodes subTreeCount = ListProperty(int) #number of keys in all child Nodes below this pointer subTreeSum = ListProperty(int) #a prefix sum (running total) of subTreeCount +1 for each totalScore So it's really just a bunch of list properties. You set how big you want the list to get (bigger is better - fewer datastore calls for lookups - but keep in mind the 1 MB limit). Follow the procedures on the wiki page linked above, but on insert, add one to the subTreeCount for the each pointer you follow (and 1 to every subTreeSum from there to the right on each node). To find the index of an item you are searching for, keep track of the subTreeSum of all the subtrees to the left of the subtree you are following, plus the nodes you pass along the way, and add them all up to get the index of the final node. Don't forget to binary search the totalScores list - there's no need to look at every item in it, since it's already sorted. You may have to do some linear searching at the end if several users have the same totalScore, but you can avoid that by using fixed point numbers and appending the UserID to the key to ensure uniqueness while still sorting correctly (lexicographically). Split operations are relatively simple too, since you have all the data you need to populate the new subTreeCounts and subTreeSums... it's just bookkeeping. Looking at the complexity of this, if you have 100 keys per BTree node, and a million entities, you can find the one you are looking for and its index in 3 sequential datastore calls, while comparing to 8 keys per node. 1000 keys per page gets you down to 2 datastore lookups and 10 comparisons per page, but you start spending more and more CPU time serializing and deserializing the data in and out of the datastore, so you'll have to play around with it to find the Goldilocks number. Good luck, and let me know if you need a more detailed explanation (though it will probably take a few days). David On Nov 1, 3:18 am, de Witte jcreator.xi...@gmail.com wrote: Hi, I'm looking for the following: Is it possible to retrieve the position of an entity in an index table? Of course, within going through the entire index by retrieving keys. Ayy tipes would be welcome -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http
Re: [google-appengine] wildcard host mappings to ghs.google.com
Hi Jason, You don't need to change your registrar - just your DNS host. I believe zoneedit support wildcard CNAMEs. -Nick On Sat, Oct 29, 2011 at 3:46 AM, Jason Collins jason.a.coll...@gmail.comwrote: When you map a custom domain to App Engine, you use a CNAME entry that points to some-host.your-domain.com. Our products often use a dynamic, programmatic hostname, e.g., [dynamic-host].our-domain.com. Google Apps supports a wildcard host mapping, which is great. However, our domain registrar (GoDaddy) does not support a wildcard CNAME, they only support a wildcard A record. So, we've hacked it to use an IP address that ghs.google.com resolves to, knowing this is fragile. And, of course, one of the IP address that we were using for ghs.google.com changed sometime overnight and our sites broke. Does anyone know of a registrar that supports wildcard host matches on CNAME records. It seems that GoDaddy.com and register.com do _not_. Or, any other solutions? (I definitely don't want to front this with my own proxy.) Thanks, j -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Announcing SSL for Custom Domains Trusted Tester Program
On Fri, Oct 21, 2011 at 7:33 AM, Brandon Wirtz drak...@digerat.com wrote: IE5/IE6 will say page cannot be displayed and will never connect. For this reason you should encourage users to arrive at a non-HTTPs version of the page, do browser detection and display an Upgrade your browser notification, then use the login to take them to the secure version of the site. Doing this will make your users vulnerable to man-in-the-middle attacks: an attacker could intercept the HTTP request and send back HTTP responses, with no redirect to HTTPS. How plausible this is depends on the nature of your app, naturally. -Nick Johnson ** ** ** ** ** ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *Nick *Sent:* Thursday, October 20, 2011 1:13 PM *To:* google-appengine@googlegroups.com *Subject:* [google-appengine] Re: Announcing SSL for Custom Domains Trusted Tester Program ** ** What happens when a non-supported browser attempts to access https://www.my-sercure-appengine-app.com? Does it redirect to http:// or show an error dialog? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/DUTj6iVJ49gJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Useful to abandon any hierarchical Entity Groups?
Hi Mos, The short answer is that you should use entity groups if and only if you need transactionality for the entities in that group. If you don't, put them in separate entity groups. If you have a firm requirement for transactionality but you can't put the relevant entities in the same group (for instance because that would put everything in one group, or you need a high write rate per-group), you will need to employ a different approach, such as redesigning your model or using global transactions. -Nick Johnson On Sun, Oct 23, 2011 at 6:16 AM, Mos mosa...@googlemail.com wrote: I'm still in a design phase of a new App Engine application and I'm wondering if it makes sense to go *without* using any Entity Groups. My application's data-model is similar to a discussion-forum. It has Postings, Comments, Users, Ratings and so on. If the application becomes successful, there will be significant write operations on Postings, Comments and Ratings. If I would model hierarchical *Entity-Groups* (e.g. starting from a User-Root), I would have the following *advantages*: +1: Transactional behavior during writes +2: Queries are strongly consistent +3: For hierachical data-structures, fast ancestor queries are available But *Entity-Groups* introduce also some *disadvantages*: -1: Limitation in write throughput (in worst case just 1 write per second on an entity group) -2: Inflexible and a more complex data-model, which makes it harder to add or re-factor relationships (forces owned-relations all over your application) *I think all together the disadvantages outweigh the advantages for a typical web application like mine.* The limitation for writes seems to be a significant disadvantage. Ok, you can work around this with sharding but this makes you data-model even more complex. And an hierarchical data-model which is technically motivated and not related to your domain seems to be a bad idea. On the other hand transactions during write operations are often not necessary (the last one wins). Hierarchical searches are not needed most of the time. The only issue I'm not sure about is the advantage of Queries are strongly consistent ? What does it mean if they are not strongly consistent? Is this just an issue for a small time frame when new data are written? Once the data is stored and replicated this shouldn't be an issue at all, or? I'm happy about feedback! Thanks Mos -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: high replication datastore limited to 1 write per second?!
Hi Scott, It's impossible to say without more details - what are you trying to achieve, what is your model design, and why do you think this will be a problem? It'd probably be better if you started a new thread - here or on Stack Overflow - with more details. -Nick Johnson On Mon, Oct 24, 2011 at 10:26 PM, Scott Lewis scott.le...@gmail.com wrote: I'm writing a multiplayer game using App Engine. What are ways I can work around the 1 write per second within an entity group? I don't know when a message acknowledgement will return from a client, and with the unreliable nature of the memcache, I'm at a loss at where I can store my data to make sure it is processed. Thank you! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Gae and visualization api
On Fri, Oct 14, 2011 at 1:04 AM, uover82 uove...@gmail.com wrote:
Hi Nick - thanks for the help!
I believe I'm using a relative url, off of localhost. Here's a
representative page:
Yes, you're embedding the hostname - 'localhost' - in your page. That's an
absolute URL, not a relative one, and will break when you're no longer
running off localhost.
-Nick Johnson
!DOCTYPE html
html
head
titleESE Pct./title
!--Load the AJAX API--
script type=text/javascript src=http://www.google.com/jsapi;/
script
script type=text/javascript
// Load the Visualization API and the ready-made Google table
visualization.
google.load(visualization, 1, {packages:
['corechart,table,annotatedtimeline']});
// Set a callback to run when the API is loaded.
google.setOnLoadCallback(init);
// Send the queries to the data sources.
function init() {
query1 = new google.visualization.Query('ese?url=http://localhost:
8080/gv/11032010_trends_daily_p.csvpivot=1dtx=0tr=0');
query1.send(handleCsvDsResponse1);
}
// Handle the csv data source query response1
function handleCsvDsResponse1(response) {
if (response.isError()) {
alert('Error in query: ' + response.getMessage() + ' ' +
response.getDetailedMessage());
return;
}
var data = response.getDataTable();
var formatter_short = new
google.visualization.DateFormat({formatType: 'short'});
formatter_short.format(data, 0);
var chart = new
google.visualization.LineChart(document.getElementById('chart_div'));
chart.draw(data,{pointSize: 0, title: 'Event/Subevent (ESE)
Analysis - Percentages', hAxis: {slantedTextAngle: 70}});
}
/script
/head
body
div id='chart_div' style='width: 1650px; height: 490px;'/div
div id=csv_div style='width: 1650px; height: 400px;'/div
/body
/html
The problem seems to be in the Query. Is there another url entry I
should be using
other than localhost? Also, I'm interested in GAE hosting the
datasource rather than
my host.
Let me know what you think - thanks
John
On Oct 12, 5:33 pm, Nick Johnson nickjohn...@google.com wrote:
Hi,
It sounds like you've embedded the URL of your development environment
into
your code somewhere. Naturally, when deployed, your browser (or the
Visualization API) can't access files stored on your local machine.
Instead of hard-coding URLs, you should determine the host header from
the
current request - exactly how to do so depends on your framework - and
use
that. Alternately, you can use relative URLs if the Visualization API
supports that.
-Nick Johnson
On Thu, Oct 13, 2011 at 5:39 AM, uover82 uove...@gmail.com wrote:
Hi All,
I'm developing an application using app engine and the google
visualization api using java. It currently uses a static file (csv)
located in war as a datasource. My app works fine in my local
development server but fails with errors like the following after
uploading:
java.io.IOException: Could not fetch URL:http://localhost:8080/...
csv path/filename
Any ideas/thoughts on this?
Let me know - thanks
John
--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] My site no longer working
Hi Kate, Your app appears to be returning a 500 server error (with no text in the body) when fetching the base URL (/) of your app. It's impossible to say why this is without seeing your code and logs. Have you checked the logs in the App Engine admin console to see what is being logged for these requests? -Nick Johnson On Thu, Oct 13, 2011 at 10:42 AM, Kate mss.ka...@gmail.com wrote: I haven't touched a thing, yet suddenly my site throws an error when the url does not contain the filename. e.g. http://www.coolabah.com gives a server error while http://www.coolabah.com/index.htm works fine. Can anyone help me! The error is The website cannot display the page HTTP 500 Most likely causes: •The website is under maintenance. •The website has a programming error. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Gae and visualization api
Hi, It sounds like you've embedded the URL of your development environment into your code somewhere. Naturally, when deployed, your browser (or the Visualization API) can't access files stored on your local machine. Instead of hard-coding URLs, you should determine the host header from the current request - exactly how to do so depends on your framework - and use that. Alternately, you can use relative URLs if the Visualization API supports that. -Nick Johnson On Thu, Oct 13, 2011 at 5:39 AM, uover82 uove...@gmail.com wrote: Hi All, I'm developing an application using app engine and the google visualization api using java. It currently uses a static file (csv) located in war as a datasource. My app works fine in my local development server but fails with errors like the following after uploading: java.io.IOException: Could not fetch URL: http://localhost:8080/ ... csv path/filename Any ideas/thoughts on this? Let me know - thanks John -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: My site no longer working
Hi Kate, You haven't included the complete stacktrace, including the all important last line that tells us what actually went wrong. Can you paste the whole thing, please? -Nick Johnson On Thu, Oct 13, 2011 at 11:35 AM, Kate mss.ka...@gmail.com wrote: This is what I am getting. Traceback (most recent call last): File C:\google_appengine\google\appengine\ext\webapp\_webapp25.py, line 701, in __call__ handler.get(*groups) File C:\wwwhome\aussieclouds\index.py, line 396, in get doRender(self,'index.htm') I read the blogs and found that a new version of the SDK was deployed yesterday. So I downloaded the new version to help me troubleshoot but it was of no help. I do not have any idea as to how to proceed with this! On Oct 12, 8:25 pm, Nick Johnson nickjohn...@google.com wrote: Hi Kate, Your app appears to be returning a 500 server error (with no text in the body) when fetching the base URL (/) of your app. It's impossible to say why this is without seeing your code and logs. Have you checked the logs in the App Engine admin console to see what is being logged for these requests? -Nick Johnson On Thu, Oct 13, 2011 at 10:42 AM, Kate mss.ka...@gmail.com wrote: I haven't touched a thing, yet suddenly my site throws an error when the url does not contain the filename. e.g. http://www.coolabah.comgives a server error while http://www.coolabah.com/index.htmworks fine. Can anyone help me! The error is The website cannot display the page HTTP 500 Most likely causes: •The website is under maintenance. •The website has a programming error. -- You received this message because you are subscribed to theGoogleGroups GoogleApp Engine group. To post to this group, send email togoogle-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: My site no longer working
Hi Kate, From the stacktrace, it looks like you're trying to parse an XML file you get from an external URL, and your XML parser is encountering a parse error, which you're not catching. You can see the logs for your live app by going to the admin console for your app and clicking on 'logs'. -Nick Johnson On Thu, Oct 13, 2011 at 11:46 AM, Kate mss.ka...@gmail.com wrote: Nick, thanks for the quick reply. Here is the log of when I run it locally. 2011-10-12 20:04:27 Running command: ['C:\\Python27\\pythonw.exe', 'C: \\google_appengine\\dev_appserver.py', '--admin_console_server=', '-- port=8080', u'C:\\wwwhome\\aussieclouds'] Warning: You are using a Python runtime (2.7) that is more recent than the production runtime environment (2.5). Your application may use features that are not available in the production environment and may not work correctly when deployed to production. INFO 2011-10-13 00:04:35,358 rdbms_sqlite.py:58] Connecting to SQLite database '' with file 'c:\\docume~1\\kjuliff\\locals~1\\temp\ \dev_appserver.rdbms' WARNING 2011-10-13 00:04:35,578 dev_appserver.py:4648] Could not initialize images API; you are likely missing the Python PIL module. ImportError: No module named _imaging INFO 2011-10-13 00:04:35,967 dev_appserver_multiprocess.py:637] Running application aussieclouds on port 8080: http://localhost:8080 WARNING 2011-10-13 00:05:21,046 __init__.py:97] You are using the default Django version (0.96). The default Django version will change in an App Engine release in the near future. Please call use_library() to explicitly select a Django version. For more information see http://code.google.com/appengine/docs/python/tools/libraries.html#Django ERROR2011-10-13 00:05:23,390 _webapp25.py:464] no element found: line 1, column 0 Traceback (most recent call last): File C:\google_appengine\google\appengine\ext\webapp\_webapp25.py, line 701, in __call__ handler.get(*groups) File C:\wwwhome\aussieclouds\index.py, line 396, in get doRender(self,'index.htm') File C:\wwwhome\aussieclouds\index.py, line 111, in doRender theevents=frontpage.getevents() File C:\wwwhome\aussieclouds\frontpage.py, line 240, in getevents dom = parse(http://www.gumnut.com/clouds/getevents.cfm;) File C:\wwwhome\aussieclouds\frontpage.py, line 229, in parse return minidom.parseString(r.content) File C:\Python27\lib\xml\dom\minidom.py, line 1924, in parseString return expatbuilder.parseString(string) File C:\Python27\lib\xml\dom\expatbuilder.py, line 940, in parseString return builder.parseString(string) File C:\Python27\lib\xml\dom\expatbuilder.py, line 223, in parseString parser.Parse(string, True) ExpatError: no element found: line 1, column 0 INFO 2011-10-13 00:05:23,405 dev_appserver.py:4143] GET / HTTP/ 1.1 500 - INFO 2011-10-13 00:05:23,421 dev_appserver_index.py:255] Updating C:\wwwhome\aussieclouds\index.yaml INFO 2011-10-13 00:07:26,358 dev_appserver.py:4143] GET /_ah/ admin HTTP/1.1 302 - INFO 2011-10-13 00:07:26,500 dev_appserver.py:4143] GET /_ah/ admin/datastore HTTP/1.1 200 - INFO 2011-10-13 00:07:26,515 dev_appserver_index.py:255] Updating C:\wwwhome\aussieclouds\index.yaml INFO 2011-10-13 00:07:26,703 dev_appserver.py:4143] GET /_ah/ admin/images/google.gif HTTP/1.1 200 - 2011-10-12 20:09:43 (Process exited with code -1) 2011-10-12 20:09:47 Running command: ['C:\\Python27\\pythonw.exe', 'C: \\google_appengine\\dev_appserver.py', '--admin_console_server=', '-- port=8080', u'C:\\wwwhome\\aussieclouds'] Warning: You are using a Python runtime (2.7) that is more recent than the production runtime environment (2.5). Your application may use features that are not available in the production environment and may not work correctly when deployed to production. INFO 2011-10-13 00:09:50,842 rdbms_sqlite.py:58] Connecting to SQLite database '' with file 'c:\\docume~1\\kjuliff\\locals~1\\temp\ \dev_appserver.rdbms' WARNING 2011-10-13 00:09:50,905 dev_appserver.py:4648] Could not initialize images API; you are likely missing the Python PIL module. ImportError: No module named _imaging INFO 2011-10-13 00:09:50,983 dev_appserver_multiprocess.py:637] Running application aussieclouds on port 8080: http://localhost:8080 WARNING 2011-10-13 00:10:32,078 __init__.py:97] You are using the default Django version (0.96). The default Django version will change in an App Engine release in the near future. Please call use_library() to explicitly select a Django version. For more information see http://code.google.com/appengine/docs/python/tools/libraries.html#Django INFO 2011-10-13 00:10:32,203 dev_appserver.py:4143] GET /_ah/ admin HTTP/1.1 302 - INFO 2011-10-13 00:10:32,203 dev_appserver_index.py:255] Updating C:\wwwhome\aussieclouds\index.yaml INFO 2011-10-13 00:10:32,250 dev_appserver.py:4143] GET /_ah/ admin HTTP/1.1 302 - INFO 2011-10-13 00:10
Re: [google-appengine] Re: My site no longer working
2011-10-13 00:10:32,078 __init__.py:97] You are using the default Django version (0.96). The default Django version will change in an App Engine release in the near future. Please call use_library() to explicitly select a Django version. For more information seehttp:// code.google.com/appengine/docs/python/tools/libraries.html#Django INFO 2011-10-13 00:10:32,203 dev_appserver.py:4143] GET /_ah/ admin HTTP/1.1 302 - INFO 2011-10-13 00:10:32,203 dev_appserver_index.py:255] Updating C:\wwwhome\aussieclouds\index.yaml INFO 2011-10-13 00:10:32,250 dev_appserver.py:4143] GET /_ah/ admin HTTP/1.1 302 - INFO 2011-10-13 00:10:32,328 dev_appserver.py:4143] GET /_ah/ admin/datastore HTTP/1.1 200 - INFO 2011-10-13 00:10:32,342 dev_appserver_index.py:255] Updating C:\wwwhome\aussieclouds\index.yaml INFO 2011-10-13 00:10:32,437 dev_appserver.py:4143] GET /_ah/ admin/datastore HTTP/1.1 200 - INFO 2011-10-13 00:10:32,562 dev_appserver.py:4143] GET /_ah/ admin/images/google.gif HTTP/1.1 200 - BUT this is using the newest SDK for Python. Last time I deployed my site was about 5 days ago. All was fine then. How can I see the live log? On Oct 12, 8:36 pm, Nick Johnson nickjohn...@google.com wrote: Hi Kate, You haven't included the complete stacktrace, including the all important last line that tells us what actually went wrong. Can you paste the whole thing, please? -Nick Johnson On Thu, Oct 13, 2011 at 11:35 AM, Kate mss.ka...@gmail.com wrote: This is what I am getting. Traceback (most recent call last): File C:\google_appengine\google\appengine\ext\webapp\_webapp25.py, line 701, in __call__ handler.get(*groups) File C:\wwwhome\aussieclouds\index.py, line 396, in get doRender(self,'index.htm') I read the blogs and found that a new version of the SDK was deployed yesterday. So I downloaded the new version to help me troubleshoot but it was of no help. I do not have any idea as to how to proceed with this! On Oct 12, 8:25 pm, Nick Johnson nickjohn...@google.com wrote: Hi Kate, Your app appears to be returning a 500 server error (with no text in the body) when fetching the base URL (/) of your app. It's impossible to say why this is without seeing your code and logs. Have you checked the logs in the App Engine admin console to see what is being logged for these requests? -Nick Johnson On Thu, Oct 13, 2011 at 10:42 AM, Kate mss.ka...@gmail.com wrote: I haven't touched a thing, yet suddenly my site throws an error when the url does not contain the filename. e.g. http://www.coolabah.comgivesaserver error while http://www.coolabah.com/index.htmworksfine. Can anyone help me! The error is The website cannot display the page HTTP 500 Most likely causes: •The website is under maintenance. •The website has a programming error. -- You received this message because you are subscribed to theGoogleGroups GoogleApp Engine group. To post to this group, send email togoogle-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to theGoogleGroups GoogleApp Engine group. To post to this group, send email togoogle-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Change Class of Polymodel Instance
On Wed, Oct 5, 2011 at 11:25 PM, PatrickCD patrick.do...@gmail.com wrote: Nick, It's never 'safe' to mess with the internals of Python like this. As a rule, if you're trying to change the class of an existing object, that's a code smell, and you should almost certainly be implementing it differently, such Hi. As a rule, when someone uses the term code smell, that's a cliche smell. Feel free to reword it any way you wish. Trying to change the class of an object usually means you're trying to do something that OO wasn't designed to do. Just because Python makes it possible doesn't mean you should do it. Google bundles django with AppEngine, and django messes with the internals of Python like crazy, so Google hopefully does think it's safe in some cases. Bundling doesn't signify endorsement. Also, I'm pretty sure Django doesn't do anything like modifying a class instance's parents after it's instantiated. I want to put some data into the datastore. Depending on the value of that data, I want it to map to a different class when I get it back. Changing the __class__ attribute currently works. I posted this question for two reasons: - to find out whether or not this approach was likely to break in some cases or with future versions of DataStore API. Absolutely no guarantees on this front. You're breaking any number of invariants by doing this, so things may break without notice at any time. - because changing __class__ feels hackish and I was hoping someone might suggest a better approach It _is_ a hack. A better approach would be to use a single class with appropriate properties, or create a new instance of a new class and discard the old one at such time as you need to make the change. -Nick Johnson -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Change Class of Polymodel Instance
On Thu, Oct 6, 2011 at 1:34 AM, PatrickCD patrick.do...@gmail.com wrote: Nick, It _is_ a hack. A better approach would be to use a single class with appropriate properties, or create a new instance of a new class and discard the old one at such time as you need to make the change. Thanks for your reply. I'm sure you're bored of this thread so I'll try to summarise. To take another example, SQLAlchemy supports what it calls Single Table Inheritance - http://www.sqlalchemy.org/docs/orm/inheritance.html#single-table-inheritance . In this approach, there is a field type that SQLAlchemy uses to determine what class the row should be mapped to. So if you change the 'type' you change the class. This is true, but I think if you ask the SQLAlchemy authors, they would also say that they don't expect this to change after a row is created. You seem to be saying that each record in the datastore should map to one and only one Python class, so a similar approach doesn't work. No, you can map datastore entities to objects any way you like. I'm just pointing out that the thing you're trying to do is a hack, and violates some basic expectations OO systems make, and the way you're trying to do it is unsupported and may break. Polymodel Objects in the GAE Datastore have a property class which stores a list of super classes. This property appears to be used by the DataStore API in a similar way to how type is used by SQLAlchemy. It's currently possible to change the value of this field by setting the __class__ attribute of an instance and saving it back to the DataStore. However, you state: You're breaking any number of invariants by doing this, so things may break without notice at any time. Somewhat cryptic, but I'll take your word for it. Invariants like the set of superclasses of an instance will remain constant after an object is created. It's possible that not just us, but also other Python libraries expect that invariant, so once you start messing with it, you may get unexpected problems elsewhere. So perhaps the question boils down to whether or not there _is_ a reliable way to change the value of the class property for polymodel records? I'm guessing the answer is no, but I'd like to check. You could use the low level datastore API to modify the class property in the datastore. If I understand correctly, you want the behavior of a class's methods to depend on the 'type' of entity being handled, and that 'type' may change over the lifetime of an entity. Let me suggest an alternative way to do what you're trying to do: - Create a DeveloperBehavior base class, which encapsulates the behaviors that change between types - Create subclasses for each type of behavior. - Create a 'behaviour' property on your model class, which fetches or instantiates the appropriate behavior for the current entity. - Either call the behavior methods directly, or wrap them with methods on your entity. This will allow you to change the behavior at runtime, without messing with Python and DB internals. -Nick Johnson Thanks Patrick -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] IRC office hours on now!
Our fortnightly IRC office hours are on now! Join us on irc.freenode.net, channel #appengine, to chat with the App Engine team. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Communication between Servlets and Tasks
On Tue, Oct 4, 2011 at 7:15 PM, Mark Silberbauer bowserm...@gmail.comwrote: I would like to write a method which handles the flow of communication on XMPP. The sequence of things I'd like to do is: 1. Send message. 2. Wait for response. 3. Process the response. Since we could be waiting longer than 30s for the response (step 2) I'll be teeing up a task to take care of this. This task will need to send the message and then wait for a response on the XMPP servlet handling the incoming message. My question is: How do I wait in the task servlet thread for the response to arrive in the XMPP Servlet? You can't. Instead, you should use an asynchronous pattern: Send the message, and register a handler for incoming XMPP messages. That handler should match up the response to the corresponding request (stored in the datastore if necessary) and perform appropriate processing on it. I'd normally use a listener pattern where the listener would store the message in a field in the Task object and then trigger a Semaphore to signal that a message has arrived. Like this: 1. Install listener in XMPP servlet in a static field. 2. Send message. 3. Wait for semaphore. Meanwhile, in the XMPP servlet thread, a response will arrive and it will call the listener's callback method which stores the message and releases the semaphore. 4. Get message from field and process. I tried this and it worked fine on the development server. However, when I uploaded to the cloud I found that I'd install the listener on the XMPP servlet (step 1) but then a new instance of the servlet would be instantiated when the message came in and there would no longer be a reference to the listener to call, event through the listener is a static field. My conclusion is XMPPServlet is run in a completely different VM meaning the static field is not shared between that servlet and the task one. Is this correct? An App Engine app can be run on any number of machines; synchronization primitives designed for communication between threads will not work. -Nick Johnson In general what is the best practice for communication between these servlets? How to I share data (normally I would've stored it in an object's field) and how do I signal from one to the other when events occur (normally I would've used a semaphore)? Sorry about the long winded question. Tell me if it's not clear and I'll refine it a bit. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/Ek_Xr5Etj20J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [google-appengine] Simple alternative to AppScale and TyphoonAE
Hi Ricardo, I'd recommend taking another look at TyphoonAE; it's pretty much exactly designed for what you're trying to do. TyphoonAE's support for app deployment is simply a convenience to make development easier. -Nick Johnson 2011/10/5 Ricardo Bánffy rban...@gmail.com Hi. Both TyphoonAE and AppScale seem very geared towards PaaS providers who want to let their clients safely deploy applications written for GAE. Has anyone started a project aimed at allowing individuals or small teams to deploy apps writter for GAE on their own servers? I understand the idea of GAE (and AppScale and TyphoonAE) is to make insanely scalable deployments, but, sincerely, most of us will never need to go beyond 10 rps served off an EC2 t1.micro instance. And, if it really needs to scale, redeploying to Google's infrastructure would be a no brainer. -- Ricardo Bánffy http://www.dieblinkenlights.com http://twitter.com/rbanffy -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] ImageService API Privacy
Hi Bruno, The only part of the Images API that provides an external URL is get_serving_url. If you want to control access to image blobs, don't use that functionality, and serve images yourself (using the blobstore serving support) after performing authentication checks. -Nick Johnson On Wed, Oct 5, 2011 at 7:13 AM, Bruno Sandivilli bruno.sandivi...@gmail.com wrote: Why, how can i save sensitive data(images) using the ImageService API, since it provides an world wide visible url ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Backend Deferred Tasks eating CPU Time?
On Wed, Oct 5, 2011 at 2:27 AM, Rishi Arora rishi.ar...@ship-rack.comwrote: I don't think backend CPU time counts against your front-end instance CPU hours quota. Backends are purely billed based on uptime - and I think this is true for both current and new pricing (starting November 1). But you mentioned something about datastore CPU? It is likely that the most of your billing is because of this. That's correct - datastore operations are currently billed by charging for CPU time. If you do any datastore operations on a backend, they'll show up as regular CPU hours. -Nick Johnson Few more questions: Is the reason for 20 backend instances that you want to execute all your 20 deferred tasks parallely? You mentioned you have ~4500 tasks. How long does each one take, and how often does each one need to execute in a day? Lets assume that neither of these tasks are sensitive to latency, and can be executed at any time during the entire day. If each task takes 10 seconds on average, and needs to execute, for example, 6 times a day... that's a total of 4500 * 10 * 6 / 3600 = 75 instance hours. You should try to schedule your backends yourself so that you only pay for 75 instance hours. If you allow 20 instances to get created, then at some point all these 20 instances will complete their work, and then idle for 15 minutes (or at least billed for 15 minutes of idle time after the last task completes processing), before they're shutdown. You'll be wasting 5 instance hours each time this happens. I think your focus should be to minimize your instance hours by minimizing the number of parallel instances you allow running. In my calculation above you only need a total of 75 instance hours, and so you should set instances to 3 or 4 in backends.yaml. A yet another way of doing this in a more controlled fashion is by using pull queues instead of enqueueing tasks on the regular push type task queues. You can enqueue all your 4500 tasks on a single pull queue, and all your backends will constantly run pulling tasks out of pull queues and executing them, until the pull queue is empty. Then they can be woken up again by a cron job to go check the pull queue again. Lastly, any cost you're incurring because of Datastore CPU hours in the current pricing model, or because of Datastore writes in the new pricing model - those can't be avoided. You will incur those costs regardless of the context of execution of your tasks - front-end or backend. Hope this helps. On Tue, Oct 4, 2011 at 9:33 AM, someone1 someo...@gmail.com wrote: Hello, Thank you for the replies. The tasks are indeed being run on my backend as I am being charged for backend usage (I don't use backends in any other way otherwise). I also see the _ah/deferred logs on my backend ID but not on my app. The CPU time being used seems to correspond to the Datastore CPU time, are the two currently linked? Even under the new pricing scheme, would I be billed for my main app up-time when executing deferred tasks on my backend? What part of my main app is considered under use when executing tasks on the backend? The max number of instances allowed on any backend is 20. My backend is setup as dynamic B1 with 20 instances. I let Google determine how many instances needed to be up and running when I queue up my tasks (usually all 20 run for 20-30 minutes each). Again, I'd just like to know what part of my main app is being used during the backend operation that is eating up my CPU. I really haven't coded anything else on my app except for this data mining portion which should all be run on the backend now. Thanks, Prateek On Oct 4, 8:38 am, Rishi Arora rishi.ar...@ship-rack.com wrote: I think deferred tasks is an excellent use case for backends. That's how I use my backend as well. Can you confirm from your logs that your tasks are indeed being processed on the backend? In the drop down for app versions, there's a special version which is named after your backend. Select that to check your logs specific to the backend. Also, I'm assuming the reason you're blowing through your budget is because you're spanning out multiple, possibly hundreds of instances. Can you find out how many instances get spawned for your deferred tasks? Can you find out how many backend instances are being spawned, if the backend is indeed being used for your tasks? Finally, when you configured your backend, what did you set as your instances parameter in backends.yaml? I don't know what the default is, but it is likely unlimited. In your case, a instance of 1 or 2 sounds sufficient, but you'll have to play around with that, based on how much queueing occurs for your tasks. On Tue, Oct 4, 2011 at 12:06 AM, Gerald Tan woefulwab...@gmail.com wrote: I believe CPU time will no longer be billable after the new pricing is out -- You received
Re: [google-appengine] Change Class of Polymodel Instance
Hi Patrick,
It's never 'safe' to mess with the internals of Python like this. As a rule,
if you're trying to change the class of an existing object, that's a code
smell, and you should almost certainly be implementing it differently, such
as by using a single class and changing a status variable.
-Nick Johnson
On Wed, Sep 28, 2011 at 12:42 AM, PatrickCD patrick.do...@gmail.com wrote:
Hello,
Is it safe to change the __class__ attribute of a PolyModel instance
and then save it back to the datastore?
e.g.
class DraftProject(PolyModel):
def someaction(self):
print 'Action permitted at this status'
class ApprovedProject(DraftProject):
def someaction(self):
raise Exception('not allowed at this status')
p = DraftProject.get(object_key)
p.__class__ = ApprovedProject
p.put()
I've a got a model class that corresponds to a project in a workflow
application. The project can be at different statuses. Depending on
the status, some actions are permissible, and some are not. I'd like
to use the Template design pattern to model this (http://
en.wikipedia.org/wiki/Template_method_pattern)
If setting the class attribute directly is unsafe, could anyone
suggest a better way to model this?
Thanks
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] remote_api_shell and PYTHONSTARTUP
Hi Owen, Why not just take a copy of remote_api_shell.py (it's very straightforward) and modify it as needed? -Nick Johnson On Tue, Oct 4, 2011 at 11:44 AM, Owen Strain o...@humblebundle.com wrote: Hi all, I'm trying to figure out a way to run a few statements automatically whenever I start the remote (or local) api shell. My goal is just to import some commonly-used stuff to save typing the import statements each time. I was hoping I could set the standard PYTHONSTARTUP environment variable to point to a file of startup commands, but that doesn't seem to work. Has anyone else figured out a way to do this? Thanks, -- Owen -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Keeping sensitive information in memory
Hi Alexander, We don't generally talk in detail about our datacenters or our security infrastructure externally, but some information is available. The link Phillip provided goes into some details, and this article about our security certifications, and the links from it regarding our data security statements should be relevant: http://googleenterprise.blogspot.com/2011/08/security-first-google-apps-and-google.html?utm_source=entblogutm_medium=blogutm_campaign=Feed%3A+OfficialGoogleEnterpriseBlog+%28Official+Google+Enterprise+Blog%29 -Nick Johnson On Tue, Oct 4, 2011 at 4:07 PM, Alexander Konovalenko alex...@gmail.comwrote: I am going to process sensitive information in my app and want to prevent it from ever getting written to disk in an unencrypted form. I hope that all disks in the Google data centers are properly sanitized before being disposed of (are they?). But defense in depth never hurts. The processing of sensitive data will occur both in front-end and back-end instances. The data has to be stored unencrypted in memory. Do the appservers running Python and Java instances use any form of unencrypted swap? SSD, hard disk, whatever. If so, I will file a feature request to provide a way to lock memory pages from being swapped out (using the mlock() system call). Note that even when there is plenty of free RAM, the OS can still swap some pages out. Alexander -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Change Class of Polymodel Instance
On Wed, Oct 5, 2011 at 2:09 PM, Murph paul.j.mu...@googlemail.com wrote: Nick, any comment on my approach of changing the PolyModel's class via db.to_dict and creating a new model with the same key? That seems to me like a relatively clean safe way to do it. The desire to be able to do this for my case, is to be able to move records from Person to User, then User to SuperUser, gaining extra fields needed as the person descends the hierarchy. I suspect that's a fairly common scenario, and seems almost exactly the type of thing that PolyModel was designed for. In that case you're not changing the class of an object, just creating a new object with existing data, which is fine. It's still a very strange thing to be doing, though, and you should just have the full set of fields on the basic 'User' record, not using them where they're superfluous. PolyModel is designed for the same sort of modelling situations that regular OO is designed for. All Cars have LicenseNumbers, but different types of car may have different properties - and a Porshe will never turn into an Audi. -Nick Johnson -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/O3DQBp7xtBAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Keeping sensitive information in memory
Hi Alexander, On Wed, Oct 5, 2011 at 4:53 PM, Alexander Konovalenko alex...@gmail.comwrote: On Wed, Oct 5, 2011, Nick Johnson nickjohn...@google.com wrote: Hi Alexander, We don't generally talk in detail about our datacenters or our security infrastructure externally, but some information is available. The link Phillip provided goes into some details, and this article about our security certifications, and the links from it regarding our data security statements should be relevant: http://googleenterprise.blogspot.com/2011/08/security-first-google-apps-and-google.html?utm_source=entblogutm_medium=blogutm_campaign=Feed%3A+OfficialGoogleEnterpriseBlog+%28Official+Google+Enterprise+Blog%29 Hello Nick, Thanks for the link. Because I did not receive an answer to my question, I will assume that you may use swap. So I went forward and filed a feature request to support mlock() in Python and Go runtimes. Is potential usage of swap really relevant here, given what we've made public about how we treat our drives? What is your threat model? -Nick Johnson http://code.google.com/p/googleappengine/issues/detail?id=6041 Alexander -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] More Apps For Domains getting in the way stuff
Hi Brandon, What you describe isn't compatible with the current configuration. Simply set up Google Apps on the domain solely to associate the domain name with the App Engine app, and configure the app to accept logins from any Google Account. There's no need to create accounts for users of the app on the domain. -Nick Johnson On Fri, Sep 30, 2011 at 3:57 PM, Brandon Wirtz drak...@digerat.com wrote: I ranted once before about the issues Apps For Domains created for my Adwords account with the unified login, but I have a new rant from a potential customer… ** ** They currently use exchange mail and Group policy objects. They don’t want to be responsible, for these student accounts Documents, Messenger, Google+ etc. They quite specifically want those to be personal accounts, so that they don’t have any potential liability for how students use those accounts. ** ** In my view this makes for a pretty good case of why not to require GAE to be tied to Apps For Domains. Not just in my app, but in any org that would decide the wanted an app on their domain but the email addresses they provide are not employees, or there are so many of them they wouldn’t want to administer them through AFD (which is not designed for administering 100k people. ** ** I know BestBuy and some other large Orgs are on GAE, are they using AppsForDomains? Are they running on their primary domain? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] More Apps For Domains getting in the way stuff
Hi Brandon, On Fri, Sep 30, 2011 at 4:52 PM, Brandon Wirtz drak...@digerat.com wrote: I may have explained poorly… ** ** It isn’t that the App needs the logins, it’s that all of the personal accounts become Managed accounts, Every one of the users would go through the “Migration” on their Google Voice, Google Plus, Picasa, etc. ** ** The worry of the school is that the pic of a 12 year old girl in her underwear posted to picasa goes from being “owned” by the personal account, to being owned by an organizational account. ** ** Also every time a user forgets their password, they go from doing support through Google, to support through the administrator. That’s a lot of overhead on a bunch of students. ** ** This is not a big deal for me, the client isn’t going to pay me enough that I care, but I though they raised some good points that I wanted to share.* *** ** There's no need to migrate any accounts at all. Your app can accept signins from any Google account, not just those on an Apps domain. -Nick Johnson ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson *Sent:* Thursday, September 29, 2011 11:14 PM *To:* google-appengine@googlegroups.com *Subject:* Re: [google-appengine] More Apps For Domains getting in the way stuff ** ** Hi Brandon, ** ** What you describe isn't compatible with the current configuration. Simply set up Google Apps on the domain solely to associate the domain name with the App Engine app, and configure the app to accept logins from any Google Account. There's no need to create accounts for users of the app on the domain. ** ** -Nick Johnson ** ** On Fri, Sep 30, 2011 at 3:57 PM, Brandon Wirtz drak...@digerat.com wrote: I ranted once before about the issues Apps For Domains created for my Adwords account with the unified login, but I have a new rant from a potential customer… They currently use exchange mail and Group policy objects. They don’t want to be responsible, for these student accounts Documents, Messenger, Google+ etc. They quite specifically want those to be personal accounts, so that they don’t have any potential liability for how students use those accounts. In my view this makes for a pretty good case of why not to require GAE to be tied to Apps For Domains. Not just in my app, but in any org that would decide the wanted an app on their domain but the email addresses they provide are not employees, or there are so many of them they wouldn’t want to administer them through AFD (which is not designed for administering 100k people. I know BestBuy and some other large Orgs are on GAE, are they using AppsForDomains? Are they running on their primary domain? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. ** ** -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Support for HTTP Patch in URLFetch
On Mon, Sep 26, 2011 at 7:57 PM, Sean Murphy seanm5t45...@gmail.com wrote:
Hi Nick,
You should probably send this to google-appengine-python@ instead, then.
Thanks - given that I've start this here, I'll finish it here. (I did look
for the appengine
python list, but could not find it easily - could I suggest that a pointer
to the group be
put in the welcome and readme for this group?)
I'm doing some work using the Salesforce REST API. This API uses the HTTP
Patch
command to update an object in SF. I note that URLFetch currently does
not support
Patch - any plans to support in future?
PATCH is actually a part of WebDav, not HTTP. If you'd like support for it
added to urlfetch, you should file a feature request on the issue tracker.
I'm a little confused - Salesforce does refer to it as a HTTP command and
this RFC
http://tools.ietf.org/html/rfc5789
does give the impression that it is part of HTTP rather than Webdav. Any
chance you'd
be able to shed further light on this?
You're right - it's not part of webdav specifically. That RFC extends HTTP,
though, so it's not a part of HTTP/1.1.
I think the solution is to use httplib2 - it's a bit clunky, however.
Unfortunately, I don't believe this will work - httplib is just a wrapper
for URLFetch, so any restrictions that apply to URLFetch apply equally to
httplib(2).
This can work using the following (which I got from some SF examples):
queryConn = httplib2.Http()
resp,content = queryConn.request( obj_url+'?_HttpMethod=PATCH',
POST,
headers= {'Authorization': 'OAuth
'+access_token, 'Content-Type':' application/json'},
body=simplejson.dumps(update_fields)
)
I have not looked into what happens under the hood, however...
All this is doing is making a POST request, specifying a query string
parameter '_HttpMethod'. You can do this using urlfetch, urllib, or the
mechanism of your choice.
-Nick Johnson
BR,
Seán.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/XSp99pCaBh4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Another perspective on the upcoming datastore prices
Hi Alex, On Sat, Sep 24, 2011 at 3:58 AM, Alex Epshteyn alexander.epsht...@gmail.com wrote: Consider an average dynamic page in any web application. It might do one entity write and fetch a small result set. Under the new GAE billing model, this might cost 25 Write Ops, and 21 Read Ops (a pretty conservative estimate). How did you compute these figures? You say they're 'conservative', but these are really quite high. 25 write ops for one entity write implies quite a lot of custom indexes, and 21 read ops implies you're fetching 20 datastore entities - which isn't likely to be the case on anything other than a listing type page. On top of that, a scalable app should be using memcache and caching headers, meaning most requests don't encounter this cost at all. -Nick Johnson Those amount to ~ $40 per million page views, which is more than the average revenue from Google AdSense for the same million pageviews! There's something really wrong with this picture. The reason AdSense does so well is that web hosting in this day and age costs a lot less than what AdSense earns. But now GAE hosting is about to cost much more than what AdSense earns (and so far I only counted just the datastore ops, so the true costs might be 2-3x higher than $40 per million pageviews). The bottom line: developers will be losing money by hosting an ad- supported application on GAE! What's going on here? Google App Engine was supposed to be more cost- effective than the alternatives, but these new prices seem to be totally out of whack with the reality of current web economics. To the management at Google who came up with these prices: please consider consulting the AdSense team before these prices go into effect! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Support for HTTP Patch in URLFetch
On Fri, Sep 23, 2011 at 8:46 PM, Sean Murphy seanm5t45...@gmail.com wrote: Hi, Issue for GAE-python. You should probably send this to google-appengine-python@ instead, then. I'm doing some work using the Salesforce REST API. This API uses the HTTP Patch command to update an object in SF. I note that URLFetch currently does not support Patch - any plans to support in future? PATCH is actually a part of WebDav, not HTTP. If you'd like support for it added to urlfetch, you should file a feature request on the issue tracker. I think the solution is to use httplib2 - it's a bit clunky, however. Unfortunately, I don't believe this will work - httplib is just a wrapper for URLFetch, so any restrictions that apply to URLFetch apply equally to httplib(2). -Nick Johnson Thoughts/comments appreciated. BR, Seán. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bnDsbTzsfYgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: About the mail api again, the surrounding services gae is proud of
Hi Tapir, There's no way for the mail API to determine if an email address is valid synchronously (before the call returns). If the domain exists and has an MX record, the only way to determine if the address is valid is to send the message to it - and sending email is something that happens asynchronously, and is subject to queueing and other delays standard for a mail system. The only way to determine this at runtime would be to set a reply-to address that your app controls, and look for bounce messages (which have a specified format). This is a feature of email itself, not a limitation of the Mail API. -Nick Johnson On Sun, Sep 18, 2011 at 6:57 PM, Tapir tapir@gmail.com wrote: I tried sending emails to some fake addresses, but there is no errors thrown and mails bounced. So I can't know if an email is sent successfully or not. If an email is not sent successfully, gae should tell developers what happened, right? On Sep 13, 1:05 am, nischalshetty nischalshett...@gmail.com wrote: I had once read in one of the threads here that email from gae gets marked as spam on yahoo. Apart from that, it was also mentioned by one of the GAE devs not to rely on this for mission critical emails. There are other services that would suffice your need. Having said that, have you tried registering on your site and checking if the email is being sent? How do you know that the reason for them not confirming is because they did not receive the confirmation email? - Users might be just lazy to confirm the subscription - Your email might have landed in their spam folder - They might have provided you with a fake email - They might have misspelled their email - They might think your email is a welcome email and hence just ignore it - see this http://bvckup.tumblr.com/post/9101329123/please-confirm-your-email-ad... Having said that, I'm not implying GAE is indeed sending all the email, just giving you more options to explore :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Status of edge cache?
Hi Joshua, The edge cache is just that - a cache. It doesn't guarantee that no 304 requests will reach your app, only that they will be satisfied by the cache if possible - and if the headers you set permit it. The more popular your content is, the more effect you're likely to see from the cache. -Nick Johnson On Fri, Sep 16, 2011 at 1:26 PM, Joshua Bronson jabron...@gmail.com wrote: Hey GAE devs, We just enabled billing for our app, but contrary to past reportshttps://groups.google.com/d/topic/google-appengine/8QgEUBOiNFw/discussion, 304 responses are still being served by our Python code rather than by GAE's edge cache, whether requested via our appspot domain or our Google Apps domain. Was the edge cache disabled for non-static resources since last report? Any info you could provide on the status of the edge cache would be super helpful. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2kmTyfznZmEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Does RPC object block response until async operations finish?
Hi Keakon, Yes, the instance will wait for all pending RPCs to complete before returning the response. Asynchronous RPCs are useful in that they allow you to do other work while the RPC runs. -Nick Johnson On Tue, Sep 13, 2011 at 6:06 PM, keakon lolicon kea...@gmail.com wrote: Hi Google guys, I just did a test of async db and memcache operations. I put or deleted 100 entities and immediately returned. The async operation call took only 0.01s, the sync one took 0.2s, but both of their total response time in the backend log was over 200ms, and I could also feel the latency was longer than a no-op request. So I think before the server sending respond to browser, the RPC object will wait its async operations to finish or get error. Does it mean if I don't care whether the operation is successful (like updating a counter), I can do an async call without waiting it by myself? -- keakon My blog(Chinese): www.keakon.net Blog source code: https://bitbucket.org/keakon/doodle/ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: how to reindex a property that once set to indexed = false?
On Mon, Sep 5, 2011 at 4:42 PM, saintthor saintt...@gmail.com wrote: the password was hashed. i think to query name and password together may take less db ops if password is wrong. isn't it? The number of operations is the same; fewer entities would be returned. In return, though, you're incurring an extra index entry for every record. You _should_ be salting your passwords (and preferably using RCrypt, SCrypt or PBKDF2), though, in which case you can't do an index lookup for the password anyway. -Nick Johnson On 9月5日, 下午1时43分, Nick Johnson nickjohn...@google.com wrote: Also, you don't need to index the password field - just fetch the user, then check the password. I sincerely hope you're not storing the password in the clear, though! -Nick On Mon, Sep 5, 2011 at 3:02 PM, Simon Knott knott.si...@gmail.com wrote: Hi, You need to retrieve all objects that were written in this period out of the datastore and re-put them - single-property indexes are only written on putting the entity. If you can't work out which entities were written in this period, then you will need to retrieve all entities and re-put them. It should be noted that query indexes, which use these single-property indexes in the background, are re-generated on deployment. Cheers, Simon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/VYJDjjca21MJ. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Max request rate for a single Java instance, limited to 1000?
Hi, The reason you're seeing the results you are is because of the CPU limit. The number of threads that will run in parallel is affected by how much CPU each thread uses - there's no point starting more threads if the CPU is already fully utilized, it would just slow the existing requests down. The good news is, you're fully utilizing your appserver. If you want to increase the concurrency further, you'd need to reduce the amount of CPU time you're consuming. -Nick Johnson On Mon, Sep 5, 2011 at 8:31 AM, de Witte jcreator.xi...@gmail.com wrote: I'm trying optimize our application to the max, so far so good, but... It appears that the threadpool of the Java instance is limited to 18, for which ~10 are used to handle requests. Our average latency is 100ms. 1 minute / 100ms = 600 requests, x10 threads, so a single instance should be able to handle *6000 requests a minute. * According to the logs the limit is 1000 request a minute. A higher rate will trigger a new instance regardless the average latency time. regardless any settings. Is this correct or can we get a higher request rate per minute? Otherwise there is no *cost *involved reason to optimize if your latency is below *600ms*. (Hence, frontend is most of the time waiting (idle) for the backend) Another side effect of this limitation is that the the Min Pending Latency setting has no effect. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/o3uDifrLVaEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: CPU limits per request
Greg's the one to listen to. :) -Nick On Tue, Sep 6, 2011 at 5:30 AM, Gregory D'alesandre gr...@google.comwrote: That limit will be dropped once you are paying for all the instances that you use (meaning as soon as the new pricing goes into effect). Greg On Mon, Sep 5, 2011 at 11:24 AM, stevep prosse...@gmail.com wrote: Nick said: Although the latter [1000 ms limit] may still be in place when we switch over to the new billing, I would expect we will increase or remove it over time Request: Can you keep us an updated about the 1000 ms limit: a) That you do commit to removing it, and b) Time Frame for its removal. This limit was, at least for me, a very important design factor. I'd assumed it would be dropped when you instituted instance resource allocation/billing. Thanks, stevep -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: how to reindex a property that once set to indexed = false?
2011/9/6 saintthor saintt...@gmail.com hash is enough for me. my site is not an e-bank. This should not matter. If your password database is compromised, the risk is not yours, it's your users'. Many users reuse passwords between sites, and if your site provides an easy avenue to determining what those passwords are, your users are vulnerable elsewhere, on better secured sites. There is no good reason not to use a proper salting and password stretching scheme; my site is not important enough is not an excuse. -Nick Johnson On 9月5日, 下午3时58分, Nick Johnson nickjohn...@google.com wrote: On Mon, Sep 5, 2011 at 4:42 PM, saintthor saintt...@gmail.com wrote: the password was hashed. i think to query name and password together may take less db ops if password is wrong. isn't it? The number of operations is the same; fewer entities would be returned. In return, though, you're incurring an extra index entry for every record. You _should_ be salting your passwords (and preferably using RCrypt, SCrypt or PBKDF2), though, in which case you can't do an index lookup for the password anyway. -Nick Johnson On 9月5日, 下午1时43分, Nick Johnson nickjohn...@google.com wrote: Also, you don't need to index the password field - just fetch the user, then check the password. I sincerely hope you're not storing the password in the clear, though! -Nick On Mon, Sep 5, 2011 at 3:02 PM, Simon Knott knott.si...@gmail.com wrote: Hi, You need to retrieve all objects that were written in this period out of the datastore and re-put them - single-property indexes are only written on putting the entity. If you can't work out which entities were written in this period, then you will need to retrieve all entities and re-put them. It should be noted that query indexes, which use these single-property indexes in the background, are re-generated on deployment. Cheers, Simon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/VYJDjjca21MJ. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: how to reindex a property that once set to indexed = false?
On Tue, Sep 6, 2011 at 11:20 AM, Pol i...@pol-online.net wrote: Talking abou this, what do you think of using bcrypt.hashpw(password, bcrypt.gensalt())? I've read in a few places it was supposed to be a good solution, but I discovered this morning that the AppEngine version, having to be pure Python, changes the default log_round for salt generation from 1024 to 1, otherwise it takes way too long. In practice, what does this mean for security? bcrypt and scrypt are both good options. scrypt or PBKDF2 are probably better choices on App Engine since the underlying hash functions are implemented in C. BTW: Since this is so important, you guys should be a Google approved password hashing function as part of GAE :) There's a feature request for it in the issue tracker. :) On Sep 5, 5:13 pm, Nick Johnson nickjohn...@google.com wrote: 2011/9/6 saintthor saintt...@gmail.com hash is enough for me. my site is not an e-bank. This should not matter. If your password database is compromised, the risk is not yours, it's your users'. Many users reuse passwords between sites, and if your site provides an easy avenue to determining what those passwords are, your users are vulnerable elsewhere, on better secured sites. There is no good reason not to use a proper salting and password stretching scheme; my site is not important enough is not an excuse. -Nick Johnson On 9月5日, 下午3时58分, Nick Johnson nickjohn...@google.com wrote: On Mon, Sep 5, 2011 at 4:42 PM, saintthor saintt...@gmail.com wrote: the password was hashed. i think to query name and password together may take less db ops if password is wrong. isn't it? The number of operations is the same; fewer entities would be returned. In return, though, you're incurring an extra index entry for every record. You _should_ be salting your passwords (and preferably using RCrypt, SCrypt or PBKDF2), though, in which case you can't do an index lookup for the password anyway. -Nick Johnson On 9月5日, 下午1时43分, Nick Johnson nickjohn...@google.com wrote: Also, you don't need to index the password field - just fetch the user, then check the password. I sincerely hope you're not storing the password in the clear, though! -Nick On Mon, Sep 5, 2011 at 3:02 PM, Simon Knott knott.si...@gmail.com wrote: Hi, You need to retrieve all objects that were written in this period out of the datastore and re-put them - single-property indexes are only written on putting the entity. If you can't work out which entities were written in this period, then you will need to retrieve all entities and re-put them. It should be noted that query indexes, which use these single-property indexes in the background, are re-generated on deployment. Cheers, Simon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/VYJDjjca21MJ . To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You
Re: [google-appengine] Another HR Refactoring Issue - When to clear a cache?
Hi Joshua, Can you not do a consistent get for the data when you have a cache miss, to ensure you're fetching the latest copy, and cache that? Looking at how NDB does its caching may be instructive here. -Nick Johnson On Mon, Sep 5, 2011 at 1:16 AM, Joshua Smith joshuaesm...@charter.netwrote: My monkeypatching solution (see my recent post in the -python group), which Guido says I shouldn't use, but which is just so darned pretty I can't help it, has gotten me through the first challenge of switching to HR, which is dealing with google search results containing keys into my old app's data store. So now I'm looking at the big Kahuna problem of consistency. Here's my first messy challenge there: My app puts a list of boards on the home page for a town, along with the list of meetings. Generating that list of boards was taking a lot of CPU, but they hardly ever change, so I put in a memcache system that built the HTML when it wasn't in the cache, and then cached it before serving. I clear the cache whenever the list of boards changes in some way. Well that ain't gonna work in HR. It's quite possible that I update a board, clear the cache, and someone comes and hits that page before eventually consistent comes to pass. So now I've got a cached copy of the stale data. (Note that I cannot use entity groups to solve this because some boards are municipal agencies, and therefore cannot be parented to the town that is building its list. I could parent all boards to some global parent, but, well, yuck.) I have some different ideas about how to fix this, but I'm wondering if anyone else who's done the port to HR has come up with a solution they find particularly elegant? I assume this is a pretty common problem, so there must be a design pattern out there… somewhere. Here are my ideas: - Clear the cache with a periodic task that re-clears it several times. I'm thinking a recurring geometric retry would be prudent (1, 2, 4, 8, 16, 32, 64, 128, 256, 512 seconds, and then pray that we have consistency) - Checksum the modified or new board, and put that sum into memcache. When generating the new board, confirm that any checksums are good. This seems more deterministic, except I don't trust memache not to squelch the checksum record. So perhaps I should do something in the datastore. This feels like it's be about 10x as much code as the stupid geometric flush. Any suggestions? -Joshua -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: how to reindex a property that once set to indexed = false?
Also, you don't need to index the password field - just fetch the user, then check the password. I sincerely hope you're not storing the password in the clear, though! -Nick On Mon, Sep 5, 2011 at 3:02 PM, Simon Knott knott.si...@gmail.com wrote: Hi, You need to retrieve all objects that were written in this period out of the datastore and re-put them - single-property indexes are only written on putting the entity. If you can't work out which entities were written in this period, then you will need to retrieve all entities and re-put them. It should be noted that query indexes, which use these single-property indexes in the background, are re-generated on deployment. Cheers, Simon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/VYJDjjca21MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: CPU limits per request
Hi Sergey, There aren't any limits on CPU time consumed by a request, but there are limits on the wallclock time used - 30 seconds hard limit, and 1000 milliseconds for requests to be considered for scaling. Although the latter may still be in place when we switch over to the new billing, I would expect we will increase or remove it over time - it existed because billing by CPU time did not cater for requests that are long-running but use very little CPU time. -Nick Johnson On Mon, Sep 5, 2011 at 2:48 PM, Sergey Schetinin mal...@gmail.com wrote: (Another attempt to send this, because apparently the first two were marked as spam. Trying from another email. Sorry if this is duplicate, I waited 18hours before sending this again.) After new pricing kicks in, will there still be limits on cpu usage per request? I suspect the answer is yes, so I would also like to know why other than that just how we wrote the thing. To me personally a lot of GAE limitations made much more sense in the old pricing model because all of them (including pricing) contributed to the impression that GAE is so efficient in stuffing machines chock-full of instances that the hosts run busy most of the time. A model like that requires instances to be spinned up shut down all the time and the limits serve to make this efficient and the resource usage more predictable. With the new pricing model we are told that spinning up instances is inefficient (in fact 15min worth of resources in a couple seconds inefficient) and the machines are supposedly sit idle most of the time (hence the removal of CPU charges. Having all of that in mind it only makes sense to remove limits on per-request CPU usage, time limits on request handling, urlfetch timeouts etc etc. Thanks. On 5 September 2011 07:36, Sergey Schetinin ser...@maluke.com wrote: (Second attempt to send this, because apparently the first one was marked as spam. Sorry if this is duplicate, I waited 18hours before sending this again.) After new pricing kicks in, will there still be limits on cpu usage per request? I suspect the answer is yes, so I would also like to know why other than that just how we wrote the thing. To me personally a lot of GAE limitations made much more sense in the old pricing model because all of them (including pricing) contributed to the impression that GAE is so efficient in stuffing machines chock-full of instances that the hosts run busy most of the time. A model like that requires instances to be spinned up shut down all the time and the limits serve to make this efficient and the resource usage more predictable. With the new pricing model we are told that spinning up instances is inefficient (in fact 15min worth of resources in a couple seconds inefficient) and the machines are supposedly sit idle most of the time (hence the removal of CPU charges. Having all of that in mind it only makes sense to remove limits on per-request CPU usage, time limits on request handling, urlfetch timeouts etc etc. Thanks. -- http://self.maluke.com/ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: RE: RE: [google-appengine] Is an APP ENGINE Application Attacking websites? - DOTS user agent / browser
Hi Sami, Have you seen the latest report regarding a bot from our IP range with no user agent at all crawling sites? Do you have any idea who/what this might be? -Nick On Wed, Aug 10, 2011 at 4:49 AM, Sami Shalabi shal...@google.com wrote: Hi Folks Sorry about this. We are aware of the issue and have turning the service off until we have a fix out. Best, Sami -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/CTlpt-mFwF4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Receiving Server Errors
We're currently experiencing production issues. I'll post an update to the downtime notify list with details very shortly. Sincere apologies for the inconvenience. -Nick Johnson On Fri, Aug 19, 2011 at 11:32 AM, fishbone goog...@gmail.com wrote: I'm receiving Server Errors when trying to look view the datastore through GAE dashboard. This seems to be happening for all my apps. Below is an example URL that caused the error. https://appengine.google.com/datastore/explorer?app_id=s~outdoorschronicleversion_id=1.352526798856258922 -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/5435J67zp8oJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Receiving Server Errors
Please see this thread on the downtime notify group for updates: http://groups.google.com/group/google-appengine-downtime-notify/browse_thread/thread/ff24c9b1ed5e39a2 -Nick On Fri, Aug 19, 2011 at 11:53 AM, Nick Johnson nickjohn...@google.comwrote: We're currently experiencing production issues. I'll post an update to the downtime notify list with details very shortly. Sincere apologies for the inconvenience. -Nick Johnson On Fri, Aug 19, 2011 at 11:32 AM, fishbone goog...@gmail.com wrote: I'm receiving Server Errors when trying to look view the datastore through GAE dashboard. This seems to be happening for all my apps. Below is an example URL that caused the error. https://appengine.google.com/datastore/explorer?app_id=s~outdoorschronicleversion_id=1.352526798856258922 -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/5435J67zp8oJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [google-appengine] Re: erroring
We're currently experiencing production issues. I'll post an update to the downtime notify list with details very shortly. Sincere apologies for the inconvenience. -Nick Johnson On Fri, Aug 19, 2011 at 11:51 AM, Rishi Arora rishi.ar...@ship-rack.comwrote: This is the only thing I can access right now, and it does not look good: http://code.google.com/status/appengine On Thu, Aug 18, 2011 at 8:45 PM, blackpawn pharmapsycho...@gmail.comwrote: things seem back to normal for me now, thanks Google heroes!!! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bswxS9n8-s8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: erroring
Please see this thread on the downtime notify group for updates: http://groups.google.com/group/google-appengine-downtime-notify/browse_thread/thread/ff24c9b1ed5e39a2 On Fri, Aug 19, 2011 at 12:12 PM, ZeroCool zero...@gmail.com wrote: Master/Slave datastore apps are down. HR ones are still working. Portal is down atm. On Aug 19, 10:06 am, reco r...@nex9.com wrote: thanx nick! good luck! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] 500 error usually occurs
App Engine is currently experiencing production issues. Please see this thread on the downtime notify group for details: http://groups.google.com/group/google-appengine-downtime-notify/browse_thread/thread/ff24c9b1ed5e39a2?hl=en -Nick On Fri, Aug 19, 2011 at 12:23 PM, saintthor saintt...@gmail.com wrote: can not open my site http://djdqbbs.appspot.com, nor update, nor console. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: erroring
On Fri, Aug 19, 2011 at 12:46 PM, fedex1 fed...@gmail.com wrote: A couple of questions. What strategy do you use to move from Master Slave to High Replication? I assume it's not as easy as changing a configuration parameter. In fact, you cannot change Datastore Replication Options after you've created an application it seems. Do you simply create a new application using High Replication as the Datastore Replication Option and then migrate your data and migrate your source? Yes. If you'd like to discuss this, though, you should probably start a new thread. Also, right after this outage, I tried to create a new application for example event-registration-hr and the admin console said it was available. I created it and it seemed to fail, but the name event- registration-hr was now taken, but not allocated to my applications. I tried again with another name and it happened again. Maybe an after effect of the outage. The outage is still ongoing. Please see this thread in the groups for details: http://groups.google.com/group/google-appengine-downtime-notify/browse_thread/thread/ff24c9b1ed5e39a2?hl=en -Nick Thanks, Ralph On Aug 18, 9:57 pm, Stephen Mayer stephen.ma...@gmail.com wrote: I'm seeing problems with the control panel, but my site is on HR datastore and the site is fine. Perhaps its only impacting master/slave datastore users? Stephen -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Docs List API change?
Hi Brian, Unless you need indexing, you shouldn't be storing values in a StringProperty. A TextProperty is a better choice. -Nick Johnson On Tue, Aug 16, 2011 at 10:17 AM, Brian Olson br...@cloudlock.com wrote: Anyone else notice that suddenly the Docs List API returns next-URI links greater than 500 characters and now they can't be stored in a StringProperty? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rdvDXIejtIkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Task Queue on backend runs much slower than it's configured to
Your task queue can only send tasks to your backend as fast as the backend can execute them, and unlike a task queue, it can't dynamically spin up more instances to handle the load. Setting your rate to 360/m will only work if your backend can handle 6 requests a second (eg, latency less than 166ms). Why aren't you just using a task queue for this? -Nick Johnson On Tue, Aug 16, 2011 at 9:22 AM, pdknsk pdk...@googlemail.com wrote: I've got a dynamic backend just for sending mail (billing enabled). Mails are enqueued elsewhere, not on this backend, practically instantly. The queue is configured like this. - name: mail rate: 360/m bucket_size: 40 retry_parameters: task_age_limit: 1d min_backoff_seconds: 20 And the dashboard figures it out like this. mail 360/m 0.75/s 40.0 Slower than configured at 0.75/s enforced rate, but still acceptable, because mails are send much slower. 2011-08-15 16:04:28.329 /mail/ 200 94ms 0cpu_ms 0kb instance=0 2011-08-15 16:04:08.254 /mail/ 200 26ms 23cpu_ms 0kb instance=0 2011-08-15 16:03:48.250 /mail/ 200 30ms 23cpu_ms 0kb instance=0 2011-08-15 16:03:28.258 /mail/ 200 44ms 0cpu_ms 0kb instance=0 2011-08-15 16:02:30.802 /mail/ 200 317ms 0cpu_ms 0kb instance=0 2011-08-15 16:02:30.770 /_ah/start 404 89ms 70cpu_ms 0kb instance=0 The previous run worked better, but you'll still notice the huge delay between the first and second mail. 2011-08-15 13:02:27.016 /mail/ 200 53ms 0cpu_ms 0kb instance=0 2011-08-15 13:02:26.676 /mail/ 200 45ms 0cpu_ms 0kb instance=0 2011-08-15 13:02:26.545 /mail/ 200 248ms 23cpu_ms 0kb instance=0 2011-08-15 13:02:25.833 /mail/ 200 217ms 0cpu_ms 0kb instance=0 2011-08-15 13:01:50.969 /mail/ 200 347ms 0cpu_ms 0kb instance=0 2011-08-15 13:01:50.931 /_ah/start 404 200ms 70cpu_ms 0kb instance=0 What could be the reason? There are no errors on the backend in the past 7 days. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: How to use Datastore Statistics.
Hi Kwame, You can get the size of an entity by calling len(db.model_to_protobuf(your_model)). This doesn't include index overhead (just like the datastore stats), so you'll need to estimate that. -Nick Johnson On Mon, Aug 1, 2011 at 8:33 PM, Kwame iweg...@gmail.com wrote: Thanks Nick. Any tips on how to get data size for entities? I looked through the Appengine Python API, and I don't see a way to get stats like size, etc. Thanks. On Aug 1, 1:57 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi Kwame, We don't calculate statistics for arbitrary subsets of your data - this is impractical, since we can't know what way you'll want to slice-and-dice your data ahead of time. If you want these statistics, you'll need to maintain them yourself. -Nick Johnson On Sat, Jul 30, 2011 at 9:41 AM, Kwame iweg...@gmail.com wrote: I'm trying to use Appengine Datastore statistics in Python. The API docs (http://code.google.com/appengine/docs/python/datastore/ stats.html) show how to get stats for all entities or entities by name and property. Does anyone know of examples of how to get stats for particular entities in an application? For example how to get the amount of disk space used by all entities belonging to one user in an application? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Google App Engine and Dedicated IP
Hi Alok, The IP addresses for both incoming and outgoing requests are shared amongst all apps. We have no plans to offer static IPs except insofar as will be necessary for SSL on incoming requests. You should not rely on IP addresses for authentication. -Nick Johnson On Sat, Jul 30, 2011 at 4:58 AM, alok bhardwaj yaletut...@gmail.com wrote: Hi Nick, Is the set or bundle of IP addresses assigned to an App stable at all? Any progress in having static IP to make calls from a GAE app? Thanks, Alok -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/kJImg3Xw0ZYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Stackoverflow
In the meantime, it's worth noting that there's a thriving community of App Engine askers and answerers on Stack Overflow, including myself. -Nick Johnson On Tue, Aug 2, 2011 at 10:56 AM, Ikai Lan (Google) ika...@google.comwrote: We considered doing this a year and a half ago. I'll talk to the team and see what the enthusiasm level is on this. The Android team does this and, at least last I heard, they were pretty happy with it. I'll see what they think of it now. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Mon, Aug 1, 2011 at 12:49 PM, Kaan Soral kaanso...@gmail.com wrote: +1 On Aug 1, 5:03 pm, Vivek Puri v...@vivekpuri.com wrote: Is it possible to migrate AppEngine group to Stackoverflow or a white label solution -http://stackexchange.com/. It is becoming impossible to search anything meaningful and identify top threads in the current form of the group. Any good solutions provided are buried quickly below getting stated kind of questions or daily issue related questions. Also, any important announcements made by AppEngine team are hard to locate and best answers to questions are hard to validate. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Pre-allocating datastore IDs - plumbing this with db.Model
On Tue, Aug 2, 2011 at 2:14 AM, Tim meer...@gmail.com wrote: On Monday, 1 August 2011 11:16:40 UTC+1, Stephen wrote: Even though you're allocating from two ranges rather than one, that's still less than the datastore would do things automatically, so you still have to be careful. For example, suppose you later add an admin web interface that can also create new cars and add them to garages. If you let the datastore allocate IDs, the first car you add to any garage through this new code path will get ID 1 or thereabouts, because children get an ID from the range associated with their parent. Eventually the admin-generated car IDs may clash with those pre-allocated from the global car sequence given out to remote clients, and old data will be over written. So you might want to enforce that a Car model can't be created without passing in an ID. I think I'm only now getting the subtlety of my faulty assumption. db.allocate_ids for a given kind does not give unique IDs for that kind, it gives IDs that are only unique for that kind AND for a specific parent as specified by the key used to make the call to db.allocate_ids(). That's correct. So I can't pre-allocate a bunch of IDs that can be used to make keys with whatever parent may be required without running the risk of duplicate IDs because the ID allocation mechanism, like the Keys themselves, is actually parent specific. You could allocate all your IDs out of a single global pool (by specifying a fixed key to db.allocate_ids). As long as you never create an entity without specifying an ID, that will work fine - but if you do create an entity with no ID specified, there's a high probability the generated one will already exist. This pretty much blows pre-allocated IDs out of the water for me, at least for what I had in mind, unless I either drop entity groups altogether, or do something like like put all items into an an entity group per user (so the parent is something that represents the user, meaning that I know the parent to use). Of course I could just manage my own ID allocation, but I'm never really keen on that when a data persistance layer has its own ID/Key mechanism. Entity groups bear close consideration. If you don't need them for transactional integrity, don't use them. -Nick Johnson Oh well, back to the drawing board, thanks for the patient explanations -- Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/fZyPanewo-AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Pre-allocating datastore IDs - plumbing this with db.Model
Hi Tim, An ID is part of the key. When you call db.allocate_ids, you pass in a datastore key which specifies the kind name and parent (if any) for which to allocate the ID range. The IDs returned can then be used to create keys as you normally would from an ID: start_range, end_range = db.allocate_ids(my_key, 10); # my_key is any datastore key for i in range(start_range, end_range): a_new_key = db.Key.from_path(my_key.kind(), i, parent=my_key.parent()) # Do something with a_new_key. -Nick Johnson On Thu, Jul 28, 2011 at 12:49 AM, Tim meer...@gmail.com wrote: This is a bit of an advanced question for the experts with regards to Keys, IDs, Models and the db.allocate_ids() functionality - those who can answer it won't need much more explanation, so apologies if it doesn't make much sense to everyone else (but you may want to file it away in case you ever need to do the same). My one-page-webapp uses a datastore framework that gets a bit distraught when it has create new items and allocate temporary IDs which will later get replaced with true IDs from the datastore (I've been using the actual Key strings as ids), so I thought I'd have a look at pre-allocating IDs. In the absence of cookbook examples, I've concluded - I can pre-allocate a range of IDs for a given class using db.allocate_ids(), but as individual keys encode the id and the parent instance (if any) then I can't convert these pre-allocated IDs into pre-allocated Keys if I may be using parent objects. False - see above. - So I should change the client to use IDs (or maybe type+id tuples), and have queries to the GAE server return items with 'obj.key().id()' rather than 'str(obj.key())' Switching to passing IDs in generated pages is a good idea in general, but has absolutely no impact on the functionality. - The client can be given a range of pre-allocated IDs that it can safely assign to new records as it creates them, knowing they won't change when the item is created in the datastore Correct. - For updating/deleting items on GAE, I'll replace calls to db.get(key) (where the key came from the client, and yes I then check the object is valid etc before I proceed) with MyModel.get_by_id(ids) (I know the types of objects by this stage so it's not like I'm doing a heterogeneous fetch) This is what you should do if you're passing IDs instead of complete keys, but as noted above, has no impact on functionality. - I'll still store references to other objects by db.ReferenceProperty type (ie key) rather than simple id as it makes migration of data easier (ie use of IDs rather than Keys is a client layer convenience only) Correct. - I'll implement a new model base class for all db.Model derived classes, that defines a new optional id parameter and constructs the correct key (my python knowledge of the constructs is bit green, so I think this does what I expect - kind() is a reserved but undocumented instance method that looks safer than using __name__ by looking at the SDK source) You can do this if you want, but it's purely convenience. Don't override the constructor, create a class method instead. The constructor is used to reconstitute existing entities from the datastore in addition to creating new ones, so overriding it correctly is difficult. The kind() method is a class method, not an instance method, and is documented: http://code.google.com/appengine/docs/python/datastore/modelclass.html#Model_kind class BaseModel(db.Model): def __init__(self, id=None, parent=None, **args): if id != None: args[key] = db.Key.from_path(self.kind(), id, parent=parent) db.Model.__init__(self, parent=parent, **args) The db.Model ctor docs say that key can't be used with parent or key_name, looking at the SDK source implies the last line of the above should be fine, but would I do better to omit the parent=parent parameter and rely on the fact that, if needed, the parent object is looked up form the key? They can't be used together because a fully qualified key includes the parent and key name, and thus there's no need to specify them separately. - I can then make instances as before 'obj = MyModel(someproperty=97, another=Hello world)' but I now have a special optional id property for construction. I haven't changed parent or property semantics or the like. Does the above look reasonable enough? I take it the efficiency is pretty much the same (ie there's no great added overhead such as extra database calls introduced by any of the above) but as I'm no great python expert, and the calls above are documented but not always explained, does the above look reasonable enough or am I laying myself open to a world of pain to come? Cheers -- Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view
Re: [google-appengine] How to use Datastore Statistics.
Hi Kwame, We don't calculate statistics for arbitrary subsets of your data - this is impractical, since we can't know what way you'll want to slice-and-dice your data ahead of time. If you want these statistics, you'll need to maintain them yourself. -Nick Johnson On Sat, Jul 30, 2011 at 9:41 AM, Kwame iweg...@gmail.com wrote: I'm trying to use Appengine Datastore statistics in Python. The API docs (http://code.google.com/appengine/docs/python/datastore/ stats.html) show how to get stats for all entities or entities by name and property. Does anyone know of examples of how to get stats for particular entities in an application? For example how to get the amount of disk space used by all entities belonging to one user in an application? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Problem using Google Apps Viewer to view a blob
On Thu, Jul 21, 2011 at 2:08 AM, Joshua Smith joshuaesm...@charter.netwrote:
Excellent question. No, I hadn't.
I just added this:
self.response.headers['Content-Type'] =
str(mimetypes.guess_type(document.title)[0]) or 'application/octet-stream'
and the headers now say:
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Type:
application/vnd.openxmlformats-officedocument.wordprocessingml.document
Date: Wed, 20 Jul 2011 16:06:14 GMT
Server: Google Frontend
Content-Length: 10762
But there is no change. .doc files still work, and .docx files do not.
Determine the actual mimetype that docx files should be served as, and make
sure you're serving as that. Don't expect the mimetype module to know.
-Nick
Any other ideas?
-Joshua
On Jul 19, 2011, at 11:59 PM, Nick Johnson (Google) wrote:
Have you checked what Mime-Type you're serving the documents as, and if
it's valid for docx documents?
-Nick Johnson
On Wed, Jul 20, 2011 at 9:55 AM, Joshua Smith joshuaesm...@charter.netwrote:
If you look at this URL:
http://www.mytowngovernment.org/meeting/1234040
and click the link for one of the .doc files, they work in google's
viewer. But if you click on one of the .docx files, they don't.
When you click on the link to view, it does this:
dlurl=re.sub('viewer','download',self.request.uri)
self.redirect(http://docs.google.com/viewer?url=%s; %
urllib.quote(dlurl))
and the download handler is simply:
documentKey = self.request.get(document)
document = DocumentModel.get(documentKey)
self.send_blob(document.blob.key())
Everything I've read says that google's docs viewer should be able to view
.docx files, so I'm guessing the problem is in the way the blob is served
from GAE to Google Apps.
Any ideas?
-Joshua
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: HR database, ancestor queries and data consistency: docs not 100% clear
On Mon, Jul 25, 2011 at 4:30 PM, Pol i...@pol-online.net wrote: On Jul 25, 7:30 am, Nick Johnson (Google) nick.john...@google.com wrote: The first part specifically refers to ancestor queries. Ancestor queries always operate on the latest copy of the data, as they're restricted to an entity group; regular queries don't have that guarantee. I understand that, but my problem is with the indexes: even in the scope of HR + transactions, the doc is confusing regarding wether or not there can be a race-condition on indexes update. For instance: - transaction A modifies entity X and commits - then transaction B happens almost concurrently: - at this point, using db.get() X will return the latest X no matter what - however, using db.GqlQuery() with a predicate that matches X *may not* return X because the indexes may not be done updating yet (the A - B thing in the doc) So is this race-condition possible or not? No matter what, you might wanna rewrite a couple sentences in this article :) Again, this depends on if you're using an ancestor query. Queries that include ancestor filters are strongly consistent, queries that don't aren't. My understanding of this article is therefore the following (assuming an HR datastore and that we are inside a transaction): 1) Using db.get() / db.put() on an entity always executes on its latest version: no race conditions possible, period. - Observations in my app appears to confirm that behavior. Correct, with the usual caveat about transactionality (if you're not doing this inside a transaction, you have a potential race condition). Yes, I was assuming HR datastore + transaction (see above). 2) Using db.GqlQuery() / db.put() on entities in the same entity group might still be subject to race-conditions: for instance, a previously *committed* transaction modified the entities that would be returned by the query in the later transactions, but the datastore indexes are not up-to-date yet (or something like that), so the query doesn't see the updated entities - even if calling db.get() directly on them would return the versions that match the query. - I've reviewed my code carefully and I think I'm seeing such race conditions in my app. Correct - but again, any update outside a transaction always has possible race conditions. Same here: I was also assuming HR datastore + transaction + ancestor queries (by definition since inside a transaction). So is my code wrong or am I observing some very rare but expected race conditions? Thanks - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] HR database, ancestor queries and data consistency: docs not 100% clear
On Mon, Jul 25, 2011 at 7:56 AM, Pol i...@pol-online.net wrote: Hi, I'm having some trouble with this article in the scope of the HR datastore: http://code.google.com/appengine/articles/transaction_isolation.html It says: In a High Replication datastore the transaction is typically completely applied within a few hundred milliseconds after the commit() returns. However, even if it is not completely applied, subsequent reads, writes, and ancestor queries will always reflect the results of the commit() because these operations apply any outstanding modifications before executing. But then just afterwards, it also says: However, if a concurrent request executes a query whose predicate (the 'where clause' for you SQL/GQL fans out there) is not satisfied by the pre-update entity—but is satisfied by the post-update entity —the entity will only be part of the result set if the query executes after the apply operation has reached Milestone B. ancestor queries will always reflect the results of the commit() because these operations apply any outstanding modifications before executing != the entity will only be part of the result set if the query executes after the apply operation has reached Milestone B The first part specifically refers to ancestor queries. Ancestor queries always operate on the latest copy of the data, as they're restricted to an entity group; regular queries don't have that guarantee. My understanding of this article is therefore the following (assuming an HR datastore and that we are inside a transaction): 1) Using db.get() / db.put() on an entity always executes on its latest version: no race conditions possible, period. - Observations in my app appears to confirm that behavior. Correct, with the usual caveat about transactionality (if you're not doing this inside a transaction, you have a potential race condition). 2) Using db.GqlQuery() / db.put() on entities in the same entity group might still be subject to race-conditions: for instance, a previously *committed* transaction modified the entities that would be returned by the query in the later transactions, but the datastore indexes are not up-to-date yet (or something like that), so the query doesn't see the updated entities - even if calling db.get() directly on them would return the versions that match the query. - I've reviewed my code carefully and I think I'm seeing such race conditions in my app. Correct - but again, any update outside a transaction always has possible race conditions. Can Google folks confirm #2? And if yes, how do you effectively work around this limitation? Is there such a thing as a flush() command? Some extra param to say commit and wait for real completion at the end of the transaction? I'm OK with extra latency per-entity group, but I really need to avoid race conditions in this specific part of my app. Do the update in a transaction, which fetches the entity, updates it, and stores it. -Nick Johnson Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to select version of Django when using 'import template'
http://code.google.com/appengine/docs/python/tools/libraries.html#Django -Nick Johnson On Thu, Jul 21, 2011 at 12:55 PM, lezizi w...@lezizi.org wrote: How to select version of Django when using 'import template' from google.appengine.ext.webapp import template -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jEdHGKGypssJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1.5.2 is out!
Hi Waleed, The min pending latency knob controls the minimum latency at which extra instances will be spawned. If your latency is too high, you can turn the setting down. -Nick Johnson On Fri, Jul 22, 2011 at 10:17 AM, Waleed Abdulla wal...@ninua.com wrote: Will there be a max pending latency option later? My problem is usually that the pending latency is too high. Waleed On Thu, Jul 21, 2011 at 3:37 PM, Brandon Donnelson branflake2...@gmail.com wrote: Why is the eclipse repository always delayed about a day? Brandon Donnelson http://gwt-examples.googlecode.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/7FLaHDyVLdgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Problem using Google Apps Viewer to view a blob
Have you checked what Mime-Type you're serving the documents as, and if it's
valid for docx documents?
-Nick Johnson
On Wed, Jul 20, 2011 at 9:55 AM, Joshua Smith joshuaesm...@charter.netwrote:
If you look at this URL:
http://www.mytowngovernment.org/meeting/1234040
and click the link for one of the .doc files, they work in google's viewer.
But if you click on one of the .docx files, they don't.
When you click on the link to view, it does this:
dlurl=re.sub('viewer','download',self.request.uri)
self.redirect(http://docs.google.com/viewer?url=%s; %
urllib.quote(dlurl))
and the download handler is simply:
documentKey = self.request.get(document)
document = DocumentModel.get(documentKey)
self.send_blob(document.blob.key())
Everything I've read says that google's docs viewer should be able to view
.docx files, so I'm guessing the problem is in the way the blob is served
from GAE to Google Apps.
Any ideas?
-Joshua
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Worried about performance of an HTTP based game.
Hi, The easiest way to do this is to not do it, if you'll forgive the misdirection. What is the nature of the update you need to do? Is there a way you can avoid having to update every record, or to update only modified records? What is the upper-limit on players per game, and do the updates for one player depend on what the other player is doing? It's effectively impossible to offer useful suggestions without more concrete details about what you're actually doing. -Nick Johnson On Fri, Jul 15, 2011 at 9:15 PM, Zippoxer zippo...@gmail.com wrote: I'm working on a game and that game will have an update that will run every 10 seconds. In this update every player's data is fetched from the datastore, processed by my code (changed) and saved to the datastore. My worry is how long this update would take if there are 2,000 players in the datastore. I'm almost sure this would take above 30 seconds, especially because I use Python. But I may be wrong in my expectation or in the method of the update. Thanks for reading. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XuVreG-Uld8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Any mechanisms to block/wait without polling?
Hi Jim, Blocking in App Engine is generally strongly discouraged. Can you provide an example use-case in which you need to do this? -Nick Johnson On Tue, Jul 19, 2011 at 4:43 AM, Jim Mittler chicago...@gmail.com wrote: I have some patterns that I would very much like to try to implement in GAE, but I can't seem to get past a requirement to coordinate a signal between two tasks running in GAE. There seems to be no GAE mechanism to wait for an event without going into some sort of polling loop. Is there something maybe that I'm missing or is this just wishful thinking? Thanks!! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] appengine seems to giving zero byte response for small responses
Hi,
You are missing this stanza from the end of your handler script:
if __name__ == '__main__':
main()
This instructs Python to execute your main function on initial import.
-Nick Johnson
On Thu, Jul 14, 2011 at 11:01 AM, piased pranavade...@gmail.com wrote:
Hello All,
I have a simple app which requires login. But instead of redirect the
user directly to google login I have a link on my main page for Sign
In. This page is generated as follows:
class Login(webapp.RequestHandler):
# GET /login
def get(self):
user = users.get_current_user()
if not user:
self.response.out.write(Not signed in )
self.response.out.write('a href=%sSign In/a' %
(users.create_login_url('/')))
else:
self.response.out.write('Hello, %s ' % (user.nickname()))
self.response.out.write('a href=%sSign Out/a' %
(users.create_logout_url('/')))
So the response is pretty small, a few bytes. But if I do a wget to
the above link it give zero bytes in response the first time I fetch
it, on subsequent tries is works fine.
From the logs I see the following message in the first request
This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application.
Whereas there is no such message in the second request which works.
Is there something I am missing or is appengine trying to do something
for small responses and screwing up especially for new instances ?
I would appreciate any help.
Thanks
-- Pranav
P.S.
Here are the headers for the zero byte response:
HTTP request sent, awaiting response...
HTTP/1.0 200 OK
Date: Thu, 14 Jul 2011 00:25:29 GMT
Content-Type: text/html
Server: Google Frontend
Content-Length: 0
Connection: Keep-Alive
Length: 0 [text/html]
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Google should want to provide messaging in App Engine
We also recently released support for the Prospective Search API, which provides support for the criteria-matching and notification use-cases of Ray's original post. -Nick Johnson On Wed, May 11, 2011 at 12:55 PM, Brandon Wirtz drak...@digerat.com wrote: I lied. ** ** Using the Java Task Queuing Service API and Camel I was able to set up JMS via ActiveMQ while on a conference call. It’s about 20 minutes worth of work, and the hardest part is that several of the links to the project bits you need to set it up are broken so you have to google some of the code snips from the documentation, that and I suck at G-Auth. ** ** ** ** ** ** ** ** ** ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *Brandon Wirtz *Sent:* Tuesday, May 10, 2011 7:12 PM *To:* google-appengine@googlegroups.com *Subject:* RE: [google-appengine] Google should want to provide messaging in App Engine ** ** I’d guess you failed to Google me or visit my page on Wikipedia before you responded. ** ** I’m going to cease commenting on this thread before it gets too far off topic. ** ** -Brandon Analyst Standard Poor’s SMPTE Voting Member IBM S/36 Certified Solution Expert SCDJWS MCSE CCSP CCNP Microsoft Alumni First Person in Ohio to be charged with hacking under the DMCA ** ** ** ** ** ** *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *rpmfl72 *Sent:* Tuesday, May 10, 2011 6:45 PM *To:* google-appengine@googlegroups.com *Subject:* Re: [google-appengine] Google should want to provide messaging in App Engine ** ** Brandon, I believe you may be the inspiration for Mark Twain's quote that it is better to keep your mouth shut and appear stupid than to open it and remove all doubt. You are showing your ignorance of both the AppEngine APIs and of efficient enterprise system design practices. True, messaging, and JMS in particular, although my focus was not to attempt to push JMS on Google as you allege, can be tunneled through HTTP for a single request / response, but I'm talking about true publish / subscribe capabilities; and no, it cannot be implemented or I would have done it. The App Engine does not allow hosted applications to listen on a port and with good reason. Maybe when you get to that class on security you might understand why. And, yes, its true that I was writing software decades before you were born on systems you have never heard of, such as the Burroughs B27 Unisys line, but my diverse experience has actually made me better. I know you're just reacting to the fact that most of the concepts in my posting are above you head, but stay in school and you'll be ok. You might also learn why the Ferrari cannot have a continuous variable transmission at this time, although my F350 does have a remote trunk unlock because it is an Excursion which was built on the F350 frame! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Thursday 7/14 Downtime
Hi Ryan, On Wed, Jul 13, 2011 at 12:33 AM, Ryan Goldstein r...@moberg.com wrote: Regarding https://groups.google.com/d/msg/google-appengine-downtime-notify/C7d8rXE-7sA/CUdvP1Z8vwAJ ... 1. I assume the subject stating July 15 is incorrect, and the real date is Thursday, July 14, as in the body. That's correct. 2. Will this downtime affect apps that don't use the datastore (i.e. only use the Google Spreadsheets API to read/write data)? Only insofar as it will also result in Memcache being cleared. Other than that, your app will continue to serve as normal. -Nick Johnson Thanks, -Ryan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Sdy_q1wx20cJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Multiple Instances of the Same App
Hi Mark, That's not the case - you can map multiple domains to the same app, either by setting them up as separate Google Apps domains, or by adding them as aliases for an existing domain. Your app can then use the Host header to distingush between domains and serve different content to them. -Nick Johnson On Sat, Jul 9, 2011 at 10:51 PM, Mark marklawso...@gmail.com wrote: I just wanted to add a +1 here. The issue for me is that 1 app = 1 domain. I would love to be able to map a domain (www.theclient.com) to a specific part of my app www.myapp.com/theclient. But it looks like I will have to duplicate my app and perhaps use multiple google accounts to do so. It seems dodgy to me as I doubt any instance of the app would exceed the free threshold but the sum of all apps would. Can I get some extra details on how people are handling this and/or how google will handle this. On Jul 9, 4:35 pm, Gregory D'alesandre gr...@google.com wrote: Hey Erik, I think you are referring to this section: 4.4. You may not develop multiple Applications to simulate or act as a single Application or otherwise access the Service in a manner intended to avoid incurring fees. What you are doing is not being done to avoid incurring fees so it does not violate our terms. We also need to build better support for this sort of thing in the future as a number of people have asked for it. Hope that helps! Greg D'Alesandre Senior Product Manager, Google App Engine On Mon, Jul 4, 2011 at 10:00 AM, Brandon Wirtz drak...@digerat.com wrote: I'm in the same boat. Google has let me get away with running the same app customized for the user. All of my apps are paid apps running on different domains. I'm all for lobbying to get app reseller accounts where we can markup our services on the billing page. If you come up with a good way to get the billing information by API let me know because I'd like to have a better way to bill clients and generate usage reports. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Eric Kolotyluk Sent: Monday, July 04, 2011 9:56 AM To: Google App Engine Subject: [google-appengine] Multiple Instances of the Same App I remember reading some policy that Google prohibits people people from basically running the same app under different registration. I gather one reason for this is so that people don't exploit the free nature of apps, or so that Google is not replicating essentially the same app everywhere. What ever the reason I don't want to violate Google's policies. We have an app we want to develop for our customers, but we essentially want each customer to have their own instance of the app for quotas and billing purposes. Basically, if our customers want the service, they would pay Google directly, rather than us figuring out who uses what and billing our customers. It would also make it easy for our software to automatically create the customer's app on app engine and keep it up-to-date. An alternative design would be to have some way to invoke a central app, but for service operations and quota have some way to bill things to a specific account. Does Google have any way to do this that does not violate the policies? The alternative for us is setting up a separate account for each customer on either Amazon, Microsoft, or some other cloud, and essentially giving each customer their own VM instance. There are pros and cons to this, as there are with using the Google PAAS, and I am trying to figure out what our best options are. Cheers, Eric -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received
Re: [google-appengine] Re: IP addresses for traffic from Google App
Hi Ruoshui, App Engine will make outgoing HTTP requests from a subset of the addresses returned by the TXT record _netblocks.google.com. Note that these can change at any time, so you can't simply fetch the list as it stands now and assume that will continue to be the case. IP-based authentication is a poor choice here, as you will open yourself up to access by any App Engine application. Instead, you should implement a proper authentication system and use that instead of relying on IP ranges for your security. -Nick Johnson On Wed, Jul 6, 2011 at 10:49 PM, liu ruos...@aquamw.com wrote: Hi Barry, Thank you so much for your response. I have looked through the SDC, and realised that it only supports linux installation, which is not possible on our windows machine. We are doing a big commercial project in which a large Wireless Sensor Network is deployed at a UK based global semiconductor company site to monitor the energy usage, but the problem is we have to give them valid IP address for Google App Engine apps in order for them to whitelist them at the firewall. So we can send inbound and outbound data through firewall between WSN and GAE apps. Otherwise, the firewall will block any traffic from the apps residing at Google App Engine. I was not able to find such information from GAE doc, so I would like to have any helps from you guys. Since this is a big commercial project using GAE service, so it is really urgent and important for us to nail down this issue asap. Many thanks, Best regards, Ruoshui On Jun 30, 7:25 am, Barry Hunter barrybhun...@gmail.com wrote: This sounds like an application for the Secure data connector http://code.google.com/securedataconnector/ If you where to whitelist appengines many IPs - you would be opening up your firewall to anyone with a app on appengine - or even access to a comprismised or low security app. (think shell.appspot.com ) On Thu, Jun 30, 2011 at 9:50 AM, liu ruos...@aquamw.com wrote: Dear all, We are setting up a network locally which is protected by a firewall, and we also have an application running on Google app engine to pull the data out of the network remotely. The problem is how I can whitelist the IP addresses for the traffic flow between the local network and the application running on google app engine please? I can't find any IP address range for the google app engine, so that I can put them in the allowable list at the firewall. Any helps and suggestions are highly appreciated. Best regards, Liu -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp:// groups.google.com/group/google-appengine?hl=en.- Hide quoted text - - Show quoted text - -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Biweekly developer chat starting now in #appengine on irc.freenode.net
Every first and third Wednesday of the month, the App Engine team hosts IRC Chat Time, an opportunity for you to get answers to your App Engine-related questions in real-time. Here's the schedule: - Every FIRST Wednesday of the month from 7:00 to 8:00 p.m. PDT - Every THIRD Wednesday of the month from 9:00 to 10:00 a.m. PDT These chat sessions take place on the #appengine channel on irc.freenode.net. For a list of IRC clients, see the following URL: http://en.wikipedia.org/wiki/List_of_IRC_clients We welcome all App Engine-related questions and we will try to answer as many as we can in the hour session. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Thread-safe Python Tips
On Tue, Jul 5, 2011 at 8:47 AM, Joshua Smith joshuaesm...@charter.netwrote:
Thanks for that clarification. I'm sure there's a reason for this
asymmetry (must declare globals to write them, but not to read them), but
it's really weird.
Since Python doesn't require variable declaration at all, there's no way for
it to know if foo = 3 is intended to create a local or a global without an
explicit declaration. If it assumed a local unless a global already existed,
creating a global in your module could unexpectedly change the behavior of a
function that happened to use the same name for a local variable.
-Nick Johnson
On Jul 1, 2011, at 6:12 PM, Geoffrey Spear wrote:
On Jun 29, 1:40 pm, Joshua Smith joshuaesm...@charter.net wrote:
I have this code in one of my apps:
townCache = {}
def getTown(id):
if not id in townCache:
townCache[id] = TownModel.get_by_id(id)
return townCache[id]
Is this thread safe? I think it is, because the worst that happens is
the assignment happens redundantly with the same data.
Random other question: Why don't I have to say global townCache at the
top of that function?
You can't *assign* to a global variable in another scope without the
global keyword; however, townCache is the global name here, not
townCache[id].
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Channel API question about channel.open([token]) JS call
Hi, On Mon, Jul 4, 2011 at 3:49 PM, ksafez216 mell...@gmail.com wrote: When I call channel.open([token]) it opens a new channel and the Channel Presence feature detects this. Then, the client's browser is shut down for a few minutes and is restarted. My app checks to see if they still have a valid token. If so, the client calls channel.open([token]), using the same token. Again, the Channel Presence listener detects that a channel was connected. Now, am I charged for two channels being open, even though it's using the same client ID and token? Every time you call channel.create on the server, a new channel is created, and a new token is returned. Billing only happens for server-side actions. -Nick Johnson The thing is I don't get a channel disconnected notification, I only get the two channel connected notifications. I'm just trying to figure out exactly what happens when I call channel.open([token]) in the client TWICE using the same token. How am I charged? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Any way to manually construct response and exit?
Hi Michael, You can use CGI - simply output the headers, a blank line, then the body to the HTTP response. This is what run_wsgi_app does. You can't do this from within a WSGI app, though, as the app will then do the same thing when your code returns. You shouldn't call sys.exit - it won't do anything, and handlers persist for multiple requests anyway. None of this will help, however, as the bug is outside the appserver and isn't WSGI related. -Nick Johnson On Fri, Jul 1, 2011 at 10:38 PM, Michael mich...@atastypixel.com wrote: Hello! I've been having trouble with an App Engine production bug (http:// code.google.com/p/googleappengine/issues/detail?id=5272), which is preventing my use of application-specific headers. I was wondering, is there a way to manually construct a response, bypassing WSGI entirely, and simply exiting the application? This way, I could simply write the HTTP response, headers, and content to stdout myself, bypassing the bug. Attempting to do so, then calling sys.exit does not achieve the desired result: the server still appears to be creating its own headers, and everything written to standard out (including the HTTP headers) are output as content. Many thanks, Michael -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Creating a login url without requiring email permissions
Hi Jay, No, there's currently no way to do this. I presume you're using OpenID logins, since Google logins don't explicitly prompt for this? If that's the case, you could choose to use a third-party OpenID library instead of the built-in support. -Nick Johnson On Tue, Jun 28, 2011 at 2:22 PM, jay kyburz@gmail.com wrote: Hello all, I use standard Google authentication to manage my users. I don't actually need my users email address, is it possible to have somebody login with out sharing their email with me. In other words, the users would not need to give my site permission to see their email. (An extra step in the log in process) At the moment if a user doesn't want to share their real email with me the login simply fails and they have to create a fake Google account to log into my service. Jay. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] is that possible to implement socket for tcp/ip data streaming in python app
Hi Liu, You're correct - you can't open sockets on App Engine. You need to have the remote service send the data to your app over HTTP. If this isn't possible, you'll need a proxy running on another machine that can receive the data and proxy it to your app over HTTP. -Nick Johnson On Thu, Jun 23, 2011 at 11:36 PM, liu ruos...@aquamw.com wrote: hi all, I am currently facing a bottleneck problem in which I am trying to implement socket in python application for tcp/ip data streaming. Specifically, I need to open a socket connection between a remote server (sending data stream) and python app residing on google app engine (receiving data stream in xml). The question is Google App Engine doesn't support the socket programming, am I right? If this is the case, are there any possible ways to work around this please? Any helps or hints are really appreciated! Regards, Liu -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] write to file system in development mode?
Hi Gerad, While it's possible to do this, as other posters point out, it involves reaching into the internals of the dev_appserver in ways that may break in future. You haven't specified why you need to do this, but it seems likely that writing an external script that does what you need and running it outside the dev_appserver is a better option. External scripts can even access your local or production datastore - see the remote_api_shell.py for an example. -Nick Johnson On Fri, Jun 24, 2011 at 10:36 AM, Gerad Suyderhoud ger...@gmail.com wrote: Long story, but... I need to be able to write to the filesystem when running the development appserver. Anybody know of a way to get around Google's monkey patches preventing this? (Again, I only need to do this in development). Notes: 1. Yes, I truly do need to write to the file system (file gets picked up by another service). 2. No, it doesn't need file system write access in production. Thanks in advance for any help -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
