[google-appengine] Practical Functional Distributed Programming, using Python on App Engine
Hi all, I don't know if it's appropriate to post this here, but this might be interesting to a lot of people who are using Python on App Engine. https://medium.com/the-infinite-machine/practical-functional-distributed-programming-using-python-on-app-engine-e19490580508 -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPACjrjVm7TvxoaZrrVXRh%2B6UGQChicYQwt8Jcc54kqPMg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Search limit
I do this is a slightly more costly way Nick; I generate my new document, read any existing document from the index, compare, only put on a change. I like your memcache idea, I could possibly cache the last written version. On 22 June 2017 at 11:38, Nickolas Daskalou <n...@daskalou.com> wrote: > That's a nice way of working around the rate limits Emlyn. > > In your _post_put function you can also compare a hash of the last > document you put in the Search API index for the Datastore entity, and only > send to the Search API if the current document's hash is different. > > To make it fast and cost-effective, you can use Memcache to store the hash > of the most recent document sent to the Search API for a particular > Datastore entity. > > This method is most useful if your app does a lot of updates to Datastore > entities which would not actually change the Search API document at all > (e.g. when updating Datastore entity properties which are not indexed in > the Search API). > > Nick > > > On 22 June 2017 at 11:42, Emlyn <emlynore...@gmail.com> wrote: > >> Thanks George, I read that, that's great. I may have to go for a support >> level in the near future. >> >> For anyone who's interested, I was able to fix my problems with the >> Search API as follows: >> >> My code enqueues a huge amount of tasks very rapidly, which modify >> datastore objects. In the _post_put handlers, these also update search >> indices. >> >> I was having problems that the datastore updates were working, but the >> rate was too high for the search indices; I was getting timeouts, quota >> limitations, bad things :-) >> >> I've changed my _post_put handlers to do the updates to search indices in >> separate tasks, on a queue I've configured just for this. It's rate limited >> to below the maximum rate for the search api (ie: maximum search throughput >> is 15000/minute, or 250/s, so I've set it lower than this). >> >> That's completely fixed the problem. And because I'm not hitting quota >> limits, the real throughput is much higher than it was; I don't see this >> queue hitting the limit and lagging behind the rest of the job. >> >> Now I get to sleep comfortably at night :-) Thanks for the help. >> >> >> On 22 June 2017 at 03:53, 'George (Cloud Platform Support)' via Google >> App Engine <google-appengine@googlegroups.com> wrote: >> >>> Hi Emlyn, >>> >>> News is not entirely bad: "Note: Although these limits are enforced by >>> the minute, the Cloud Platform Console displays the daily totals for each. >>> Customers with Silver, Gold, or Platinum support can request higher >>> throughput limits by contacting their support representative." This is to >>> be read just below the quotas paragraph we speak about. In short, you may >>> consider asking for a quota increase. >>> >>> Information on how to reduce your app's needs for paid resources, and so >>> keep within quotas, you may consult the "Managing App Resources" >>> documentation page >>> <https://cloud.google.com/appengine/docs/standard/python/console/managing-resources> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to google-appengine+unsubscr...@googlegroups.com. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> Visit this group at https://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40 >>> googlegroups.com >>> <https://groups.google.com/d/msgid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40googlegroups.com?utm_medium=email_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Emlyn >> >> https://medium.com/the-infinite-machine - A publication about Google App >> Engine >> sutllang.com - My language sUTL >> https://plus.google.com/u/0/100281903174934656260 - Google+ >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengine+unsubscr...@googlegroups.com. &g
Re: [google-appengine] Re: Search limit
Thanks George, I read that, that's great. I may have to go for a support level in the near future. For anyone who's interested, I was able to fix my problems with the Search API as follows: My code enqueues a huge amount of tasks very rapidly, which modify datastore objects. In the _post_put handlers, these also update search indices. I was having problems that the datastore updates were working, but the rate was too high for the search indices; I was getting timeouts, quota limitations, bad things :-) I've changed my _post_put handlers to do the updates to search indices in separate tasks, on a queue I've configured just for this. It's rate limited to below the maximum rate for the search api (ie: maximum search throughput is 15000/minute, or 250/s, so I've set it lower than this). That's completely fixed the problem. And because I'm not hitting quota limits, the real throughput is much higher than it was; I don't see this queue hitting the limit and lagging behind the rest of the job. Now I get to sleep comfortably at night :-) Thanks for the help. On 22 June 2017 at 03:53, 'George (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote: > Hi Emlyn, > > News is not entirely bad: "Note: Although these limits are enforced by the > minute, the Cloud Platform Console displays the daily totals for each. > Customers with Silver, Gold, or Platinum support can request higher > throughput limits by contacting their support representative." This is to > be read just below the quotas paragraph we speak about. In short, you may > consider asking for a quota increase. > > Information on how to reduce your app's needs for paid resources, and so > keep within quotas, you may consult the "Managing App Resources" > documentation page > <https://cloud.google.com/appengine/docs/standard/python/console/managing-resources> > . > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/623ea57b-1c0c-4454-a854- > 5cb5fa84bd8b%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40googlegroups.com?utm_medium=email_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP%2BqEf%2Byz%2BxwKCn2GwT1EfGsuA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Search limit
Thanks George. Not quite the news I wanted, but better to know. On 21 June 2017 at 01:13, 'George (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote: > Hello Emlyn, > > The 15000 documents per minute quota is meant for the whole app. You can > see in the document you link to that the action is named "Adding documents > to Indexes", so generally, not one index or per index. > > In the Java "Quotas" document > <https://cloud.google.com/appengine/quotas#search> the corresponding > entry is "Maximum documents added or deleted 15,000 per minute". No > mentioning of index(es) is made, value being defined per app. Similar > statements can be read in the corresponding Python and Go documents. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/bec25533-ce04-4ae6-8f3c- > d622a546d9bc%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/bec25533-ce04-4ae6-8f3c-d622a546d9bc%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPCWTQJYrqdkp34J3wGz%2BpXv6ryR3AtykWd0%2B%2Bbj5vbdmA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Search limit
There's a documented rate limitation on the search api (which I am hitting :-( ), of - 15,000 Documents added/deleted per minute https://cloud.google.com/appengine/quotas#search Is this per index, or for my entire app? -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPCd1cvSDoZ-vZPEzxjSn2P5euuiLnNRDq95Qi01m%2B2zBg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Task rate limits?
Thanks for that feedback Jordan, that's potentially really useful. So maybe you are proposing that something like this is happening: - I enqueue a massive amount of tasks very quickly (and those tasks enqueue more tasks etc). - Inevitably, there is less capacity available (instances) than necessary. So some tasks get *individually* delayed (they fail to be handled, and go into exponential backoff). - AppEngine keeps spinning up instances until the contention goes away - But, some tasks may have failed to be scheduled enough times in the interim that they now appear "stuck" for a little while. What's actually going on is that they've been "backed off", and have a timestamp in the future when they will try to run. So, it'll look like some tasks are getting stuck. Is that right? Further, and crucially to me, is this invisible via the console? ie: would these tasks be sitting there and not running, but I can't see any evidence of why? I'm asking this because this isn't, for instance, affecting the task's ETA (it might say something like "8 mins ago"). On 16 June 2017 at 06:21, 'Jordan (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote: > The limits on Task Queue calls can be found on the Quota page > <https://cloud.google.com/appengine/quotas#Task_Queue>, specifically the > 'Queue execution rate' being of importance. > > Too many tasks being executed on a single queue will inevitably result in > underlying contention. This leads to the slow down of task execution (aka > exponential-backoff retry) while tasks wait for resources to handle them. > It is recommended to instead shard your tasks across multiple queues to get > around this limit. You can also tweak your queue.yaml > <https://cloud.google.com/appengine/docs/standard/python/config/queueref> > *'max_concurrent_requests'* setting to prevent any single queue from > hitting this limit. > > Additionally, it is always good to check your logs to ensure that you > didn't see any errors during the gap, as you could be hitting other quotas > limits (which would require time to refill). > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/308b61be-732f-4b05-8fe2- > 925ac46c0874%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/308b61be-732f-4b05-8fe2-925ac46c0874%40googlegroups.com?utm_medium=email_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBojegLPJqbXSKrCPcf9Ma5u1rtAxPko%3DC_17_Se2f_vQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Task rate limits?
I don't see any yellow tooltips. A bit more info: I'm not sure, but I think it applies to certain tasks, not the entire queue. So maybe under load some rare tasks get into a temporary error/stuck state? On 14 June 2017 at 21:25, Attila-Mihaly Balazs <dify@gmail.com> wrote: > There seems to be some kind of "safety throttling" with the queues. > Whenever I hit it, there is a yellow warning sign next to the queue in the > cloud console: https://console.cloud.google.com/appengine/taskqueues and > the tooltip says something about the queue being throttled to avoid > impacting other customers. > > So perhaps you could check your cloud console and see if there are any > warnings whenever the queue seems to be throttled? > > Attila > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/a0cff223-8a7d-4de3-bb4e- > 28db0d0ad17a%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/a0cff223-8a7d-4de3-bb4e-28db0d0ad17a%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPC1T3vzEO2qwk%3DBpnWb41keW2JrV%3D-YdJgxmegXd86iUQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Anyone experiencing the new console log view being a bit busted?
> > By the way, I've noticed each log item now has a link to the line in the > source code which generated the log item (for Python at least). This is a > great idea and very helpful. Nice work. > I want to second this. These source code links seem to have come and gone a bit; they're great. > Hello - I'm an engineer from the Logging UI team. >> >> Thanks for this feedback! This was part of an experiment, yes, and we >> have rolled it back for now. >> >> The aim was to help clarify and "group" these long-running requests split >> across multiple log entries. From this and other feedback, we are pursuing >> a mode where the "first of _", "part of _" annotations will still be there, >> but are rethinking grouping the entire "long running request" data >> underneath each of those entries. >> >> For more input - would you prefer the data shown the way it was (with >> different data split across the different entries), or (if possible, >> assuming the browser wouldn't hang) the data grouped all together beneath >> each of the entries (like was attempted here)? >> >> I don't mind if the data is split across entries. Alternatively, if you're going to show the whole list under one (or each) item, I would need an easy way to get to the end. Very often these hugely long lists have an error right at the end, which is why I'm examining them in the first place. So just showing the first few lines then having a "show more" button isn't useful. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPCHvC-zK9zSHVbVoyYiMoWxbxZn92aBFPBWZ9jNttUkAA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Task rate limits?
I'm doing a lot with code that enqueues huge amounts of tasks (hundreds of thousands) in short periods of time (over a couple of minutes). Some tasks enqueue other tasks, in a fan-out. I'm 99% sure that I'm hitting some kind of hidden rate limitation from time to time. The behaviour I'm seeing is that my queue stops processing any events suddenly, for a few minutes, might be for ten minutes. Then it picks up and continues like nothing happened. Is there some kind of rate limit under the covers that I'm hitting? -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBYJFLEB7sbC3N9Kk814MEEs4HEbSZUytbZb0yRp_29zw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Anyone experiencing the new console log view being a bit busted?
They're logs from a Python (standard) AppEngine application. But today the view has reverted back to how it used to be, everything's working again. Maybe I was B in an A/B test? On 14 June 2017 at 00:01, 'George (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote: > Hello Emlyn, > > This issue seems to manifest itself in particular circumstances, as I can > not reproduce it based on the received information. More detail is needed, > preferably a step-by-step description of the procedure. What types of logs > are these, for which operations? > > All other information you deem relevant will be highly appreciated. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/22c28b34-d34e-4a9c-a080- > 63582cb8db91%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/22c28b34-d34e-4a9c-a080-63582cb8db91%40googlegroups.com?utm_medium=email_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Emlyn https://medium.com/the-infinite-machine - A publication about Google App Engine sutllang.com - My language sUTL https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBrbDV_7bxcQiAx597s_wviQSmhskut2ckxbe9abbigUQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] App Version - testing
All versions share the same data. Be careful there :-) On 13 June 2017 at 18:51, Richard Cheesmar <chezand...@gmail.com> wrote: > Hi, > > Will adding data to a non default app version (testing) add it to the > default ndb store? > > I haven't deployed another version yet just need to confirm this in > advance as the new version uses modified model entities and I don't want to > change the live version model entities whilst testing the new version. > > Cheers > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/d0cbcce4-af89-48ae-833f- > f1af01fb1fad%40googlegroups.com > <https://groups.google.com/d/msgid/google-appengine/d0cbcce4-af89-48ae-833f-f1af01fb1fad%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPDJ-wvFLAEqG4HKwUTwO-rMYrFzKZ4ZasN7yrJfVKvm3g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Best Practices for Search API?
"The search API is pretty cool. I hope Google makes an official service out of it someday." I second this! I live in fear of deprecation. How could we possibly replace it? On 10 June 2017 at 16:04, Jeff Schnitzer <j...@infohazard.org> wrote: > No not at all; quite the opposite, really. I was storing word fragments > for typeahead, eg “foobar” became [“f”, “fo”, “foo”, “foob”, “fooba”, > “foobar”]. It was very expensive to index in the datastore and very cheap > to index in the search api. > > The search API is pretty cool. I hope Google makes an official service out > of it someday. > > Jeff > > On Fri, Jun 9, 2017 at 3:45 PM, Nickolas Daskalou <n...@daskalou.com> > wrote: > >> Jeff, were you storing docs in the Search API somewhat intelligently, >> e.g. using short field names, "0" instead of "false" for boolean values >> etc., or you just sent documents to the Search API without much thought and >> it was still efficiently storing them? >> >> Nick >> On 10/06/2017 8:40 AM, "Jeff Schnitzer" <j...@infohazard.org> wrote: >> >>> The search index is incredibly efficient. I had some data I was indexing >>> in the datastore, and the index was consuming 100GB+. When I moved it to >>> the Search API, the index consumed a few GB. Afterwards I felt silly for >>> asking for the quota raise in advance. >>> >>> YMMV, of course. >>> >>> Not a direct answer to your question but something to keep in mind. I >>> was surprised by how small the index is. >>> >>> Jeff >>> >>> On Thu, Jun 8, 2017 at 11:34 PM, Emlyn <emlynore...@gmail.com> wrote: >>> >>>> I've just noticed that the old 10GB limit on search indices can be >>>> raised on request to 200GB max, on an index by index basis. >>>> >>>> I've got a use case where I'd ideally like search indices to be >>>> unbounded, but 200GB will go a long way. Is there any penalty associated >>>> with this, though? Do the indices perform worse as they get larger? Does >>>> anyone know why there is a limit at all? >>>> >>>> -- >>>> Emlyn >>>> >>>> http://point7.wordpress.com - My blog >>>> https://plus.google.com/u/0/100281903174934656260 - Google+ >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to google-appengine+unsubscr...@googlegroups.com. >>>> To post to this group, send email to google-appengine@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/google-appengine. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxP >>>> g_kvX4-aUm3w%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxPg_kvX4-aUm3w%40mail.gmail.com?utm_medium=email_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to google-appengine+unsubscr...@googlegroups.com. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> Visit this group at https://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVX >>> TSjQt1qYGNMCJQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com?utm_medium=email_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengine+unsubscr...@googlegroups.com. >> To post to this group, send email to google-appengine@googlegroups.com. >> Visit this group at https://groups.google.com/group/google-appengine. >>
Re: [google-appengine] Best Practices for Search API?
I've noticed the same thing Jeff, it's really brilliant. But what I'd like to put into it is a pretty dense time series of real time events; it's really difficult to put a sensible upper bound on how big that might get. On 10 Jun. 2017 8:10 am, "Jeff Schnitzer" <j...@infohazard.org> wrote: The search index is incredibly efficient. I had some data I was indexing in the datastore, and the index was consuming 100GB+. When I moved it to the Search API, the index consumed a few GB. Afterwards I felt silly for asking for the quota raise in advance. YMMV, of course. Not a direct answer to your question but something to keep in mind. I was surprised by how small the index is. Jeff On Thu, Jun 8, 2017 at 11:34 PM, Emlyn <emlynore...@gmail.com> wrote: > I've just noticed that the old 10GB limit on search indices can be raised > on request to 200GB max, on an index by index basis. > > I've got a use case where I'd ideally like search indices to be unbounded, > but 200GB will go a long way. Is there any penalty associated with this, > though? Do the indices perform worse as they get larger? Does anyone know > why there is a limit at all? > > -- > Emlyn > > http://point7.wordpress.com - My blog > https://plus.google.com/u/0/100281903174934656260 - Google+ > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxP > g_kvX4-aUm3w%40mail.gmail.com > <https://groups.google.com/d/msgid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxPg_kvX4-aUm3w%40mail.gmail.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/ msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_ fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com <https://groups.google.com/d/msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com?utm_medium=email_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPAyf9BkjzR_f80ReRW6NAWw6oadSDsCOw1i5%2BMPNaGzFQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Best Practices for Search API?
I've just noticed that the old 10GB limit on search indices can be raised on request to 200GB max, on an index by index basis. I've got a use case where I'd ideally like search indices to be unbounded, but 200GB will go a long way. Is there any penalty associated with this, though? Do the indices perform worse as they get larger? Does anyone know why there is a limit at all? -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxPg_kvX4-aUm3w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Unable to change billing limits
No I didn't, and there it is! Fixed. You're a life saver. On 22 April 2016 at 16:48, George-Cristian Bîrzan <gcbir...@cloudpipes.com> wrote: > Did you try the new console, on console.cloud.google.com, it's under > appengine settings? > > On 22 Apr 2016 10:12 am, "Emlyn" <emlynore...@gmail.com> wrote: >> >> When I try to set my Max Daily Budget for my appengine app, I get this >> error: >> >> --- >> Billing Command Failed >> >> There was an unexpected error and your budget settings were not saved. >> Please try again. If this problem persists, please contact support. >> --- >> >> See screenshot attached. >> >> This page seems to have changed in the last couple of days, and also >> stopped working. >> >> For me this was unfortunate, as my production system went over quota >> and I couldn't fix it. This didn't lead to happy things. >> >> Does anyone know a way to get around this? Is there somewhere else I >> can change the max daily budget? >> >> -- >> Emlyn >> >> http://point7.wordpress.com - My blog >> https://plus.google.com/u/0/100281903174934656260 - Google+ >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengine+unsubscr...@googlegroups.com. >> To post to this group, send email to google-appengine@googlegroups.com. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/CAMp1VPBh%2BhrH7fn4931eivZDGCxGPe2oEsL1T%2B7JsNrsMr_-Qg%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/CA%2B3w0iVap1fDNY-q%2BTyP3R7Sw%2BEoB3o%3DKYym47KNQqt15C%3DSeA%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPAk5130juL2cgXSpSHF-pWv7RTKVAeuiRq9X7XTeNyiNA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Unable to change billing limits
When I try to set my Max Daily Budget for my appengine app, I get this error: --- Billing Command Failed There was an unexpected error and your budget settings were not saved. Please try again. If this problem persists, please contact support. --- See screenshot attached. This page seems to have changed in the last couple of days, and also stopped working. For me this was unfortunate, as my production system went over quota and I couldn't fix it. This didn't lead to happy things. Does anyone know a way to get around this? Is there somewhere else I can change the max daily budget? -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBh%2BhrH7fn4931eivZDGCxGPe2oEsL1T%2B7JsNrsMr_-Qg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Some of my applications appear to have stale caches in Australia
Thanks Greg, that's really useful. We've had that setting (24 hour expiry) for a long time, but I've never seen it actually honoured like this (and weirdly only for Australia). But I'll file this under "not a bug but a feature" and go fix our cache settings. On 18 November 2015 at 07:28, Greg Jones <greg.jo...@gmail.com> wrote: > Your page is setting cache-control with a max age of 24hrs, so if it's been > less than that then things are working as they're supposed to - if you need > new content to be reflected more quickly, you should change the > cache-control. If you check the headers on the response and see a non-zero > 'Age', then it means the response has come from an intermediate cache. > Google has caches on it's "edge", but it's possible there are others > depending on how you connect. > > (https://redbot.org/?uri=http%3A%2F%2Fnickdemo.tes-testing.appspot.com is a > useful tool for testing these kinds of things) > > Greg > > > On Tuesday, 17 November 2015 08:52:59 UTC, Emlyn wrote: >> >> Hi, >> >> If you go to this appengine app link: >> >> http://nickdemo.tes-testing.appspot.com/#/login?returl=%2F >> >> you'll see one of two things. >> >> If you are in Australia, it'll say "Please Login" (old code) >> If you are not in Australia, it'll say "Please Loggin" (new code) >> >> Does anyone understand why this is happening? So far it's been like >> this all day. Is there a problem in Google's infrastructure? >> >> -- >> Emlyn >> >> http://point7.wordpress.com - My blog >> https://plus.google.com/u/0/100281903174934656260 - Google+ > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/9bcc4f8e-d304-4438-8f23-0301db5a5a04%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPD5SfuDxcqGW9dmTR%2BuRGwfzuZket716%3D81CEAxDP7yGA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Some of my applications appear to have stale caches in Australia
Hi, If you go to this appengine app link: http://nickdemo.tes-testing.appspot.com/#/login?returl=%2F you'll see one of two things. If you are in Australia, it'll say "Please Login" (old code) If you are not in Australia, it'll say "Please Loggin" (new code) Does anyone understand why this is happening? So far it's been like this all day. Is there a problem in Google's infrastructure? -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPCeZ-OGFOydCcuUhCPEAKeiOKMRpWGMSdGOQMs3eW%3DHXQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Where is the Datastore Admin in the new Developers Console?
Thanks for this! Why doesn't the greyed out link in the datastore console point here?? On 7 October 2015 at 21:40, Mark Cummins <mcumm...@gmail.com> wrote: > Actually, I just discovered it is available in the new Developers Console, > thought it's in a very non-obvious place. It's in > > Cloud Datastore > Settings > Open Datastore Admin > > Similarly, I thought Cron Jobs were gone, but they're just weirdly hidden in > a tab in the Task Queues view. > > > > > On Wednesday, 7 October 2015 11:35:48 UTC+1, Oliver Urs Lenz wrote: >> >> Thanks very much for your solution! I am quite exasperated as to how >> Google doesn't seem to care about usability. >> >> On Wednesday, October 7, 2015 at 11:50:41 AM UTC+2, Mark Cummins wrote: >>> >>> Quick workaround I found. If you look at the greyed-out link you'll see >>> the URL has an '' parameter. If you remove that you can still get >>> to all of the greyed out links, at least for now. >>> >>> I hope Google isn't just going to dump this functionality. We rely on >>> Datastore Admin to manage our backups. We also use the bulk entity delete >>> quite often. As far as I can see neither of these are possible in the new >>> Developers Console. >>> In general, to be honest I also just find the old Developers Console much >>> more pleasant to use. >>> >>> >>> On Wednesday, 7 October 2015 10:36:57 UTC+1, Oliver Urs Lenz wrote: >>>> >>>> As of today, the Datastore Admin (along with a lot of other menu >>>> options) in the old App Engine console is greyed out and struck strough, >>>> with a tool tip saying "This feature has moved to the new Developers >>>> Console". But clicking on it leads to me to the Datastore Dashboard, no >>>> Datastore Admin appears to be available in the Developers Console. >>>> >>>> Am I overlooking the Datastore Admin in the Developers Console, and if >>>> so, where is it located? (I need the Datastore Admin to see whether backups >>>> completed succesfully and what string of characters was assigned to them.) >>>> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/ab313576-3925-4c74-bb23-1f9a727385d1%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPAvjFQtLG0G7ZETfru2Y5kLH1K5YM9DHKoBE1xkAaWR9Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Simple Search API billing estimate bug
I'm really hoping there's a bug, or else I'm getting charged $107,374.14 per app for today. That's a little outside what I've budgeted :-) Anyone else seeing that? Here's a screenshot: https://goo.gl/photos/249wBB1HBYwGtuVq9 -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPAMJGikZKp84x7rhvc-7sXVhE50CACVtnpZxcb-kSfQCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Simple Search API billing estimate bug
Seems to be fixed. That was quick! On 14 July 2015 at 17:53, Emlyn emlynore...@gmail.com wrote: I'm really hoping there's a bug, or else I'm getting charged $107,374.14 per app for today. That's a little outside what I've budgeted :-) Anyone else seeing that? Here's a screenshot: https://goo.gl/photos/249wBB1HBYwGtuVq9 -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPD5p%3DUtrrDybohbs8H_CXGXycTgx9jUw9U-6MKYTBBXHg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?
Thanks very much Ryan, I'll give it a shot. If I hit any walls or notice any issues, I'll let you know. On 18 June 2015 at 23:21, Ryan (Cloud Platform Support) rbruy...@google.com wrote: Salutations Emlun, There is work on LineInput Readers here. It is still Beta and has not been accepted by the master thread yet so be careful using it. On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote: This is a python appengine question, mapreduce 1.9.21 I have code writing lines to a blob in the local blobstore, then processing that using mapreduce BlobstoreLineInputReader. Given that the files api is going away, I thought I'd retarget all my processing to cloud storage. I would expect to find a class called GoogleCloudStorageLineInputReader, but there isn't anything like that. Is there something way I can use GoogleCloudStorageInputReader to read lines? Another possibility is using GoogleCloudStorageRecordInputReader, but for that my input file needs to be in LevelDB format and I don't know how to create that except with a GoogleCloudStorageConsistentRecordOutputWriter, which I don't know how to use outside a mapreduce context. How might I do that? Or am I doing this all wrong, is there some other possibility I've missed? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBsUMJqCwUtCb8k_a8peniGeOBPbCiCDPaxngQHhi06qQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?
I'm trying this library here as pointed to by Ryan: https://github.com/rbruyere/appengine-mapreduce But I'm getting an odd mismatch between it and the Cloud Storage client library; it's looking for file_name on GCSFileStat objects, but in the library these only have filename I'm using version 1.9.21.0 of the cloud storage client library. Is there some other version I should be using? On 19 June 2015 at 10:57, Emlyn emlynore...@gmail.com wrote: Thanks very much Ryan, I'll give it a shot. If I hit any walls or notice any issues, I'll let you know. On 18 June 2015 at 23:21, Ryan (Cloud Platform Support) rbruy...@google.com wrote: Salutations Emlun, There is work on LineInput Readers here. It is still Beta and has not been accepted by the master thread yet so be careful using it. On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote: This is a python appengine question, mapreduce 1.9.21 I have code writing lines to a blob in the local blobstore, then processing that using mapreduce BlobstoreLineInputReader. Given that the files api is going away, I thought I'd retarget all my processing to cloud storage. I would expect to find a class called GoogleCloudStorageLineInputReader, but there isn't anything like that. Is there something way I can use GoogleCloudStorageInputReader to read lines? Another possibility is using GoogleCloudStorageRecordInputReader, but for that my input file needs to be in LevelDB format and I don't know how to create that except with a GoogleCloudStorageConsistentRecordOutputWriter, which I don't know how to use outside a mapreduce context. How might I do that? Or am I doing this all wrong, is there some other possibility I've missed? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBJXKEMUsGM2xxrOg7vRkNk6-ONCiP0t%2BN%2BE6cRi7C1vg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?
oh, update: In the cloudstorage library, in common.py, I added this: @property def file_name(self): return self.filename to the class GCSFileStat, and now GoogleCloudStorageLineInputReader appears to be working. Woohoo! On 19 June 2015 at 13:52, Emlyn emlynore...@gmail.com wrote: I'm trying this library here as pointed to by Ryan: https://github.com/rbruyere/appengine-mapreduce But I'm getting an odd mismatch between it and the Cloud Storage client library; it's looking for file_name on GCSFileStat objects, but in the library these only have filename I'm using version 1.9.21.0 of the cloud storage client library. Is there some other version I should be using? On 19 June 2015 at 10:57, Emlyn emlynore...@gmail.com wrote: Thanks very much Ryan, I'll give it a shot. If I hit any walls or notice any issues, I'll let you know. On 18 June 2015 at 23:21, Ryan (Cloud Platform Support) rbruy...@google.com wrote: Salutations Emlun, There is work on LineInput Readers here. It is still Beta and has not been accepted by the master thread yet so be careful using it. On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote: This is a python appengine question, mapreduce 1.9.21 I have code writing lines to a blob in the local blobstore, then processing that using mapreduce BlobstoreLineInputReader. Given that the files api is going away, I thought I'd retarget all my processing to cloud storage. I would expect to find a class called GoogleCloudStorageLineInputReader, but there isn't anything like that. Is there something way I can use GoogleCloudStorageInputReader to read lines? Another possibility is using GoogleCloudStorageRecordInputReader, but for that my input file needs to be in LevelDB format and I don't know how to create that except with a GoogleCloudStorageConsistentRecordOutputWriter, which I don't know how to use outside a mapreduce context. How might I do that? Or am I doing this all wrong, is there some other possibility I've missed? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPCnbwDQpqdiimFMVsD_h%3Dh6NJZfr31fgs7%2BF2ep6DFATg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?
This is a python appengine question, mapreduce 1.9.21 I have code writing lines to a blob in the local blobstore, then processing that using mapreduce BlobstoreLineInputReader. Given that the files api is going away, I thought I'd retarget all my processing to cloud storage. I would expect to find a class called GoogleCloudStorageLineInputReader, but there isn't anything like that. Is there something way I can use GoogleCloudStorageInputReader to read lines? Another possibility is using GoogleCloudStorageRecordInputReader, but for that my input file needs to be in LevelDB format and I don't know how to create that except with a GoogleCloudStorageConsistentRecordOutputWriter, which I don't know how to use outside a mapreduce context. How might I do that? Or am I doing this all wrong, is there some other possibility I've missed? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/062df9e9-c3ff-4cec-b454-6ceafd5c62a8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Python 2 End of Life 2020
course of action is to file usability bugs against managed VMs until they work as well or better than what you're used to. More specific is better. Jeff On Thu, Apr 16, 2015 at 9:19 PM, Jay Kyburz j...@jaykyburz.comwrote: Hello Googlers, There is an interesting discussion today on Hacker News about how Debian has started moving from P2 to P3 because of the looming EOL deadline in 2020. https://news.ycombinator.com/item?id=9388502 Does anybody know if there are plans in the works to provide a Python 3 implementation of App Engine I can port my apps to sometime in the next 5 years. I've been searching around but can find nothing. 5 years may sound like a long time to many of you, but the last 5 years of using App Engine have seemed to fly by for me. I have a lot of code to port and it might be a biggish job. If Google is looking for more staff to handle the port I would be happy to contribute. Jay. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com. To post to this group, send email to google-a...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a336be6a-f0e3-487f-aa4c-36dd8c1117c5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-a...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8c465186-d499-4df2-abf4-2551e16060eb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com. To post to this group, send email to google-a...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/BCB5BEF3-E37F-4196-A979-9AB066CE1029%40rakkoon.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com. To post to this group, send email to google-a...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uiPia5W_H_veOCpUvw9N9heuOeT4BX0hjn%2B9XY3iFsP2g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- Andrew Jessup | Product Manager, Google Cloud Platform | jes...@google.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bd5d91f3-ea9d-45cc-9c73-a544993ca9ab%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPAxPAEe4P9%2B%3DtB3MRRDS2W_7n790mMg_DPLvfz77R%3DKmA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Distributed Promises library for python on appengine
Hi AppEngine Pythonistas, I've written a useful library for distributed processing in AppEngine, implementing distributed promises. They're described in detail in this post: https://point7.wordpress.com/2014/11/21/distributed-promises-for-appengine/ I want to open source the library, can anyone help me with best practices for creating an open source python package, including how to make that most easily accessible to appengine developers? -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Defered function that can take lambdas, closures
Hi all, This might be interesting to anyone who uses deferred.defer in python on appengine. I've just written this code that expands what you can do with defer. You can defer a lambda, and you can defer (functions with) closures. It's implemented using the original defer, but marshals a function's code across the defer boundary (using the marshal library and a wrapper function) and keeps the function's closure information intact. Note that this wont take an arbitrary callable, you need to be passing a function. I'd love any feedback; am I doing anything disastrous? Could anything be done a better way? import marshal, types, sys from google.appengine.ext import deferred # defer any function object to be run in a background task def Defer(aFunction, *args, **kwargs): if not aFunction or not hasattr(aFunction, func_code): raise Exception (First argument required, must be a function) # recursively turn a function's closure info into something marshallable def MarshalClosureValues(aClosure): logging.debug(aClosure) lmarshalledClosureValues = [] if aClosure: lclosureValues = [lcell.cell_contents for lcell in aClosure] logging.debug(lclosureValues: %s % lclosureValues) lmarshalledClosureValues = [ [marshal.dumps(litem.func_code), MarshalClosureValues(litem.func_closure)] if hasattr(litem, func_code) else [marshal.dumps(litem)] for litem in lclosureValues ] return lmarshalledClosureValues # marshall the function's code lmarshalledFunc = marshal.dumps(aFunction.func_code) # marshall the function's closure info lmarshalledClosureValues = MarshalClosureValues(aFunction.func_closure) # also grab the function's module (for restoring appropriate globals) lmoduleName = aFunction.__module__ # call defer on our wrapper function, with all this info; # it will reverse the above and call the origin function deferred.defer(WrapDeferred, lmarshalledFunc, lmarshalledClosureValues, lmoduleName, *args, **kwargs) # Reverse the marshalling process and call reconstituted function def WrapDeferred(aMarshalledFunc, aMarshalledClosureValues, aModuleName, *args, **kwargs): # get appropriate globals. These are being used for functions inside closure info # as well as top level function; is this dangerous? lglobals = sys.modules[aModuleName].__dict__ # Creates a cell for a value by wrapping it in a function and then pulling it out # of the closure info def make_cell(value): return (lambda x: lambda: x)(value).func_closure[0] # Reverse marshalling process on closure info def UnmarshalClosureValues(aMarshalledClosureValues): lclosure = None if aMarshalledClosureValues: lclosureValues = [ marshal.loads(item[0]) if len(item) == 1 else types.FunctionType(marshal.loads(item[0]), lglobals, closure=UnmarshalClosureValues(item[1])) for item in aMarshalledClosureValues if len(item) = 1 and len(item) = 2 ] lclosure = tuple([make_cell(lvalue) for lvalue in lclosureValues]) return lclosure # unmarshal the function code lfunctionCode = marshal.loads(aMarshalledFunc) # unmarshal the closure info lclosure = UnmarshalClosureValues(aMarshalledClosureValues) # create a new function using the unmarshalled code, closure info, and globals lfunction = types.FunctionType(lfunctionCode, lglobals, closure=lclosure) # call the function! lfunction(*args, **kwargs) -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Mapreduce Input Reader for Search API?
Has anyone got a Search API input reader for mapreduce (python)? I'd find it useful. Would anyone else? -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Appengine ecosystem?
I've been lurking on this mailing list for quite a while, and suspected the move to SO was a bad idea; answers are excellent, but you can't have a community on SO. You need forums. The best thing I've found elsewhere is the official cloud platform community on G+: https://plus.google.com/communities/105461889719084929003 Pretty low volume, but looks ok. On 27 June 2014 20:11, Diego Duclos diego.duc...@palmstonegames.com wrote: I find visiting the #appengine IRC channel answers this need for me. Though it's quiet at times, more people joining it would help tremendously there. We also have this very mailing list of course. On Fri, Jun 27, 2014 at 12:12 PM, Philip Kilner phil.kil...@gmail.com wrote: Hi, On 27/06/14 09:15, timh wrote: Unfortunately it seems all the good appengine discussions went away with the SO introduction however SO is no good for discussions about approach, idea's etc... it also seems like a lot of input etc disappeared when people like Nick Johnson and Ikai went elsewhere. Just my 2c worth. +1 I find that listening to the chit-chat of form discussions is a great way to discover how other people use tools and helps me learn, and the lack of such a forum since the advent of SO as the official channel has made App Engine that much less attractive as a result. Ultimately, Google's changes in this area have convinced me that I am not the intended audience for App Engine. As a solo developer, my needs have been much better served since I switched to platforms where the vendors see the value in this sort of community. -- Regards, PhilK 'a bell is a cup...until it is struck' -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Anybody interested in a Datastore Manager for GAE?
Great idea, I'm interested too. There's always the issue that datastore objects have code associated with them, that you can't tend to touch in a console. But that's ok I think. ndb is my datastore of choice. I'd like to see one that you could upload as an alternate version of your codebase, so it can get right in there and do anything your code can do. On 9 August 2012 07:11, Backpack georgen...@gmail.com wrote: Something like phpMyAdmin for MySQL but adapted to the datastore. If so, answer these questions: - What datastore: db, ndb or SQL? - How much would you pay for that? - Pay once use forever with yearly versions? ~$100/ver - or pay monthly with free updates? ~$10/mo - web based or desktop app? I am thinking on doing one with the latest HTML,CSS,JS stuff (sorry IE) and just wanted to see if there was any interest in a tool like that. The data viewer and data admin stuff offered by gae are limited in functionality. Thanks for your time in answering the questions. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Appengine ecosystem?
What are the good communities for appengine / google cloud? Where are the good online resources for appengine / google cloud? What are the good third party tools for appengine / google cloud? I've been cranking away on commercial appengine work, really busy, so I've not been paying much attention to the outside world. Reflecting for a minute, I think I'm probably doing a lot of things the hard way (and probably crappy way). However, I've never really identified much in the way of good communities, resources, tools for appengine. I currently subscribe to the google group http://groups.google.com/group/google-appengine And of course I use the docs StackOverflow extensively. What cool things are you using that you think everyone else should also be using? btw, I'm asking this question elsewhere, and will try to compile and repost a useful list of feedback. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Should I use google.appengine.ext.mapreduce ?
Ok, that's what I'm doing. Thanks. It's really very cool, btw. Nice work Google. Of course you've gotta love code as its own documentation, but I'm ok with that ;-) On 12 February 2014 15:50, Vinny P vinny...@gmail.com wrote: On Tue, Feb 11, 2014 at 9:34 PM, Emlyn emlynore...@gmail.com wrote: I've just started working with mapreduce. I followed the doc online, downloaded the mapreduce library as instructed, and things are going well. However, I just noticed that it's already in the sdk as google.appengine.ext.mapreduce Should I be using that? Or should I be using the latest code from svn? Use the latest code from svn. Yes there's already a copy in the SDK, but MapReduce is still experimental and is subject to rapid breaking changes. It's a lot easier to track down possible bugs and issues if you're consistently using the latest versions. - -Vinny P Technology Media Advisor Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Should I use google.appengine.ext.mapreduce ?
I've just started working with mapreduce. I followed the doc online, downloaded the mapreduce library as instructed, and things are going well. However, I just noticed that it's already in the sdk as google.appengine.ext.mapreduce Should I be using that? Or should I be using the latest code from svn? -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Consistent 503 errors while updating Go app
There definitely seems to be an intermittent issue around precompilation. Also, is anyone seeing wildly varying speed of writes and reads of the datastore? On 28 January 2014 15:26, Vinny P vinny...@gmail.com wrote: On Mon, Jan 27, 2014 at 2:08 PM, Todd Olson t...@pendo.io wrote: I simply cannot imagine that we are the only ones encountering these issues, yet I cannot find any information/announcement. What are we doing wrong? Well, that's a bit of an open-ended question. If you can narrow it down (i.e. instance startup issues, optimization, etc) I can help with specific questions. If your issue is more of a general architecture design problem, it would probably be better idea to pay for a support package and contact Google directly. Google can access your application, data, and much more profiling information than I have access to. - -Vinny P Technology Media Advisor Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] channel.send_message() fails when called from a backend
I have an app with one backend running. Call it fred. My web client hits my app on the default version (as is normal). The handler sets up a channel subscription, yada yada. Other bits of the code happily send information up the channel using channel.send_message() . My backend, however, is in a different code version (called fred, same as the channel, this is how backends work as far as I can tell). So I have two versions: default and fred. When the backend fred tries to call channel.send_message for a channel established outside the backend (ie: in code running the default version), it fails silently. I believe this is because the subscription id is invalid, given that we're in the wrong codebase (fred, rather than default). I've been able to get around this by making a call to a handler on the default codebase to do the channel.send_message() on fred's behalf (ie: using appid.appspot.com rather than fred.appid.appspot.com). This then works. Possibly all of this would go away if I was using modules. At some point in the future I'll convert over. But I guess I'm just putting this here so someone might notice this (it's a bit of a trap, a bit annoying), and maybe might tell me if I'm doing something obviously wrong, need to do it differently. Thanks -- Emlyn http://point7.wordpress.com - My blog https://plus.google.com/u/0/100281903174934656260 - Google+ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Can't restore backups of objects that contain blobfields with compress=True
I agree, that's totally a problem. I can cope with that though, if only I could restore my backups! On 16 May 2013 22:05, troberti tij...@firigames.com wrote: A similar problem (or the same problem) is when you edit (ie. save) a entity with a compressed property in the Datastore viewer. The compressed property cannot be loaded afterwards. I have filed an issue on the ndb issue tracker: http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=202but it does not seem to be high priority. On Thursday, May 16, 2013 1:38:25 PM UTC+2, Emlyn wrote: Hi, Python 2.7, ndb. This probably requires someone from the appengine team. I'm getting inconsistent failures when restoring instances of this object from backup: class Event(ndb.Model): dtStored = ndb.DateTimeProperty(auto_now_**add = True) strType = ndb.StringProperty() keyClient = ndb.KeyProperty() keyAssertingPrincipal = ndb.KeyProperty() txtData = ndb.TextProperty(compressed = True) keysRelated = ndb.StringProperty(repeated = True) version = ndb.IntegerProperty() It's the only type that causes problems. When I restore it, the mapreduce tasks fail (see below). I do get a handful of objects (less than 20, when there are a couple of thousand). Of the objects that do load, there's another issue; the blobs don't know they're compressed, and load incorrectly. The issue is detailed here: https://code.google.com/**p/googleappengine/issues/**detail?id=8599https://code.google.com/p/googleappengine/issues/detail?id=8599 I've actually got a workaround for that, so the only real problem is that all the objects don't load. datastore_backup_restore_**datastore_backup_2013_05_16Job #15811165735318781AA47 Processed items per shard Overview - Failed - Elapsed time: 00:03:25 - Start time: 16/5/2013 20:54:09 - files: [/blobstore/**AMIfv97SZ99dXOQM_** mnRvAM0zq46DVrNbtTIgHNldiWKhvZ**m_Gya_**gaAaaH8FmUO44MyCgd7uWIlE4zjC3N **nLuw0w3v8aU2GZxOellbwnMjPoy4Ld**yKhI_TAL9xFFi60UGLfYXIILT-** g0PHyfs6HvbqOslDqxNf7eKRGyg-**KZTJZnbLrZ6AV46Y,/blobstore/** AMIfv95f1-**rDfkt9mU8tpjDSZFItFygt0T8wudsK** bQzZGyZJZXOGLQTQylONwc9hpQJioJ**X3OIV8OJ2XmPLrP4I23BXefgTVyt6F** tc7Hnk2kmnFIVXcDaFwBsYwvjGKhwv**Y2aYicSM6W7MuED1XuXZFEIEUq378d** 7I50Nqa7wyADoS7ZTvxJvvA,/**blobstore/AMIfv95J9-** gr38xASdG532e6PRzVbRzgCpcofSM9**bkHBUjJp4LlpfdbeUavUxBcd8gNxNj** a6is5H5QNNWpD3NgU_AYIP1-0zC4-**fLcGq2_mBS-fKg_** owihzibI8p35RW5yQK8s6XHS4MGGyx**Yw9IKZcLvknqJ6ZTLcW2ZqqhaXL_** 5dH8l1nrrvI,/blobstore/**AMIfv94yitrEvcVwe0qUNegoTxu3_**IRqknWnvy_** iEieoaMVgjA1jbMoU31hMS4zcN8Rfb**6-**nehAhrL1A02d9HIz2v6O76Az9w98-** iXW2tIc4DU8HX5Yw-59T2-**IVF3kdK60UQM7nebsboVZMKzFZ5Bub**NGXEATlN0EV_** jYc6Mx3fXiN3MOcaXrg,/**blobstore/**AMIfv95igj5MqPIMPYte9O_** yyRvlWFApsDquW-**XKXYKtbB36PCcnoJdG1fkefapX6ijn**SMoSTaF_** NnEVqcZF7dNuTQNoUnbAOOHzDwDuZp**X1pNIrPzPXP3IGBChjEvknB6SfTQXA** G8MhU5CSpj4qztyYeGQRpAAaF0UyGN**OGj0QO3IYh0hXWgQ8,/**blobstore/** AMIfv96E9ULQbP0LocTy9gcPyRcS3R**6kSlcmeWsCbFSLd_Gmyqq-1tX-** IyBZL_TKYYsqNSe9_**T9usaVsGAHVj9BpCd76SLiDG3vrebq** mfiFQt91sizGRQKUoIqjDvB6jBX5cn**2wMwAzgaj19hOyhU9mKW7ooLIZkbNb** Op2Udf4hueV8mbvBCwJY,/**blobstore/**AMIfv96ruBqhkI3QHpegsd-** SM2kM1RM9xOpyVggJRJHNJe5guB7F9**Yzz9DkfADvX4JqrbJl8TwXuRtZZP2O**_** 27oHSSVu2zdErd7D2pUBuZod5ivm7G**l7jg7FdiSdgaaKQw6PZSTj_** CGVZ55PMdCgy7hrtjaqQLBbqmoVCcs**5u1mo-diZ3QDYyZY,/blobstore/** AMIfv9432ZpwBD6v_**0LqKKtDmoShp0XndRJSl-R_** DUHUleVp12EbIR0cxPGUsnZcnEGRFD**sDd0PsfvDiWjUsYKZUY_LPko_** gcKdIvEEKDOHbLm8jp3ESTdSUAosFI**ja-3-JJWJnqgaumeGbw9_** Kmv6wCB27os1i203v61d1PCNL5jYVP**iqYTcno]Collapsehttps://ah-builtin-python-bundle-dot-tes-test-eca.appspot.com/_ah/mapreduce/detail?mapreduce_id=15811165735318781AA47 - kind_filter: [Event] - namespace: null - original_app: null Counters - io-read-bytes: 28696 (139.98/**sec avg.) - io-read-msec: 410 (2/sec avg.) - mapper-calls: 29 (0.14/sec avg.) Mapper status ShardStatusDescriptionLast work itemTime elapsed 0 failed[u'/blobstore/ **AMIfv97SZ99dXOQM_**mnRvAM0zq46DVrNbtTIgHNldiWKhvZ**m_Gya_** gaAaaH8FmUO44MyCgd7uWIlE4zjC3N**nLuw0w3v8aU2GZxOellbwnMjPoy4Ld** yKhI_TAL9xFFi60UGLfYXIILT-**g0PHyfs6HvbqOslDqxNf7eKRGyg-** KZTJZnbLrZ6AV46Y']:0 'jXj\x0es~tes-test-ecarF\x0b\** x12\teventtype\x08assessor\**x0c\x0b\x12\x05Event$** 870ad402-6e77-4f49-80 00:03:00 1 failed [u'/blobstore/AMIfv95f1-** rDfkt9mU8tpjDSZFItFygt0T8wudsK**bQzZGyZJZXOGLQTQylONwc9hpQJioJ** X3OIV8OJ2XmPLrP4I23BXefgTVyt6F**tc7Hnk2kmnFIVXcDaFwBsYwvjGKhwv** Y2aYicSM6W7MuED1XuXZFEIEUq378d**7I50Nqa7wyADoS7ZTvxJvvA']:0'jXj\x0es~tes-test-ecarF\x0b\ **x12\teventtype\x08assessor\**x0c\x0b\x12\x05Event$** 0492ea38-82d4-4554-86 00:03:00 2 failed [u'/blobstore/AMIfv95J9-** gr38xASdG532e6PRzVbRzgCpcofSM9**bkHBUjJp4LlpfdbeUavUxBcd8gNxNj** a6is5H5QNNWpD3NgU_AYIP1-0zC4-**fLcGq2_mBS-fKg_** owihzibI8p35RW5yQK8s6XHS4MGGyx
Re: [google-appengine] Re: gaeutilities session not working in ie9
Yup, sorry, I did diagnose it, gaesessions is fine. My app lives in an iframe inside another page. That turns out to cause the issue. Apparently IE9 requires something called a Privacy Policy for iframed-in pages before it'll save cookies at Medium security level (which is the default). Excellent explanation at this URL: http://stackoverflow.com/questions/389456/cookie-blocked-not-saved-in-iframe-in-internet-explorer Incidentally, I tried Beaker instead, and it works fine straight out of the box, good to know, It didn't fix the problem of course, but it was good to get it going and get a feel for that. On 17 November 2011 02:38, bowman.jos...@gmail.com bowman.jos...@gmail.com wrote: The state is saved via a cookie that is a token the library keys off of. If you haven't changed the default settings then you should have a cookie in your browser named something like gaeutilities-session. If IE is configured to not accept cookies from the domain you are testing, then that would likely be the problem. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/E0wymcFXQecJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] AttributeError: 'ChannelPresenceConnection' object has no attribute 'shutdown'
Hi,
I'm playing with the Channel API at the moment. It's good! And by
playing, I mean implementing in a commercial product (of course).
It's all working happily, I just have a minor niggle that it's killing
my dev appserver. Basically, as soon as I call channel.open(), blamo!
The appserver goes down like a ton of bricks. So for an app that's
relying fairly heavily on the channel API, the dev appserver unusable
like this.
Here's a test page:
html
head/head
body
script type=text/javascript src=/_ah/channel/jsapi/script
script
channel = new goog.appengine.Channel('xyz');
socket = channel.open();
/script
Hello
/body
Here's the result:
===
Traceback (most recent call last):
File
/usr/lib/python2.7/SocketServer.py, line 284, in
_handle_request_noblock
self.process_request(request, client_address)
File /usr/lib/python2.7/SocketServer.py, line 311, in process_request
self.shutdown_request(request)
File /usr/lib/python2.7/SocketServer.py, line 459, in shutdown_request
request.shutdown(socket.SHUT_WR)
AttributeError: 'ChannelPresenceConnection' object has no attribute 'shutdown'
Exception happened during processing of request from ('0.1.0.10', 80)
ERROR2011-11-16 00:55:41,127 dev_appserver_main.py:664] Error encountered:
Traceback (most recent call last):
File
/opt/google/google_appengine/google/appengine/tools/dev_appserver_main.py,
line 657, in main
http_server.serve_forever()
File /opt/google/google_appengine/google/appengine/tools/dev_appserver.py,
line 3527, in serve_forever
self.handle_request()
File /opt/google/google_appengine/google/appengine/tools/dev_appserver.py,
line 3490, in handle_request
self._handle_request_noblock()
File /usr/lib/python2.7/SocketServer.py, line 287, in
_handle_request_noblock
self.shutdown_request(request)
File /usr/lib/python2.7/SocketServer.py, line 459, in shutdown_request
request.shutdown(socket.SHUT_WR)
AttributeError: 'ChannelPresenceConnection' object has no attribute 'shutdown'
Now terminating.
===
I'm using python 2.7 and sdk 1.6.0
I've just checked that if I switch back to python 2.5 (ie: runtime:
python instead of runtime: python27) then it seems to work. However,
that's not really an option, as I'm becoming fairly dependent on
python 2.7 now throughout my app.
Any workarounds, hacks, fixes even? Or just ideas?
--
Emlyn
http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google
Buzz posts,
comments and all.
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] gaeutilities session not working in ie9
Hi all, I'm having an issue with gaeutilities sessions; they're just not retaining session info between calls in IE9 (fine in firefox and chrome). I haven't diagnosed this in depth yet, just wondering whether anyone has seen this. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Feedback on the new pricing
Would it be cheaper to run it as a single multitenancy app? On 9 November 2011 20:35, Daniel Florey daniel.flo...@gmail.com wrote: Hi, we are running a tool on App Engine that allows users to sync GMail contacts between different Google / Google Apps accounts. https://www.google.com/enterprise/marketplace/viewListing?productListingId=2083+12830481423215493060 We make a dedicated deploy to app engine for each customer (company) so that each client has full control over the datastore permissions etc. Right now we have ~2000 instances running on GAE. A lot of these companies switched to Google Apps after evaluating our tool that gives them missing core functionality. Each client pays between 50€ (small enterprises) and 200€ (large enterprises) for our tool and is also in charge for handling the app engine costs. Most of the small businesses (non-profits or enterprises with very few people) have been able to run the app within the free quota. When exceeding the quota they have been charged a few cents per day on top for the resources consumed. After the price change I received hundreds of support requests / complaints in the last two days because people where running out of quota as the apps have been shut down. Our clients may be willing to pay the additional resources required, but most of them will not be able to pay $9 / month which by far exceeds the costs of our tool. This would increase the total cost from 50€/year to at least 150€/year for even the smallest companies. So what killing our business is not the new quota (we have spent a lot of effort to optimize the resources consumption with good results) it is the minimum fee of $9 / month. Any ideas how to handle this? Thanks, Daniel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/thg1JWk6R7kJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Feedback on the new pricing
You know what'd be cool (and forgive me if this already exists)? An app framework that could load custom plugin modules, so one AppEngine app could run multiple back end style tasks at the same time, from different developers. Then a customer could pay their $9/month for the app, and plug in as many modules as required. Lots of commercial apps have a model similar to the OP (sell a lot of times, very cheaply, to many many customers) and are being affected by the $9/month minimum, so something like this would let them be pooled together. On 9 November 2011 21:57, Emlyn emlynore...@gmail.com wrote: Would it be cheaper to run it as a single multitenancy app? On 9 November 2011 20:35, Daniel Florey daniel.flo...@gmail.com wrote: Hi, we are running a tool on App Engine that allows users to sync GMail contacts between different Google / Google Apps accounts. https://www.google.com/enterprise/marketplace/viewListing?productListingId=2083+12830481423215493060 We make a dedicated deploy to app engine for each customer (company) so that each client has full control over the datastore permissions etc. Right now we have ~2000 instances running on GAE. A lot of these companies switched to Google Apps after evaluating our tool that gives them missing core functionality. Each client pays between 50€ (small enterprises) and 200€ (large enterprises) for our tool and is also in charge for handling the app engine costs. Most of the small businesses (non-profits or enterprises with very few people) have been able to run the app within the free quota. When exceeding the quota they have been charged a few cents per day on top for the resources consumed. After the price change I received hundreds of support requests / complaints in the last two days because people where running out of quota as the apps have been shut down. Our clients may be willing to pay the additional resources required, but most of them will not be able to pay $9 / month which by far exceeds the costs of our tool. This would increase the total cost from 50€/year to at least 150€/year for even the smallest companies. So what killing our business is not the new quota (we have spent a lot of effort to optimize the resources consumption with good results) it is the minimum fee of $9 / month. Any ideas how to handle this? Thanks, Daniel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/thg1JWk6R7kJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Dining Philosophers
Thanks, I had no idea that Memcache was so capable (I haven't looked closely enough recently). You could absolutely build a semaphore using those methods (cas would do the job nicely). Except of course that it could disappear at any moment. I can't help thinking that there must be some way to cope with that, however. Worth a ponder! On 27 October 2011 18:53, Murph paul.j.mu...@googlemail.com wrote: I think it's basically possible to do locking, mutexes, semaphores, etc in memcache, by taking advantage of some of the semantics, plus the new-ish compare store stuff. There's the obvious caveat that it could be flushed without warning, or disabled for a while, so it's not a high reliability thing. def add(self, key, value, time=0, min_compress_len=0, namespace=None): Sets a key's value, iff item is not already in memcache. def replace(self, key, value, time=0, min_compress_len=0, namespace=None): Replaces a key's value, failing if item isn't already in memcache. def cas(self, key, value, time=0, min_compress_len=0, namespace=None): Compare-And-Set update. def incr(self, key, delta=1, namespace=None, initial_value=None): Atomically increments a key's value. def decr(self, key, delta=1, namespace=None, initial_value=None): Atomically decrements a key's value. It seems to me the you could carefully use the above in various creative ways to implement this, if you require frequent use and the datastore costs would be prohibitive for it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cAlYR6jxlogJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Dining Philosophers
Germ of an idea: Assume we can implement semaphore entirely in memcache, ignoring the fact that it can be flushed. Then, to cope with being flushed, we could have an idea of a Semaphore as being in one of three states; Ok, Gone and Restarting. Ok is just normal operation Gone is when you know it's been flushed You detect Gone by noticing the problem when you try to touch the Semaphore (in Wait() and Signal() calls). You restart the Semaphore. Restarting is a period where you are trying to get back to Ok state. Basically, the idea will be that if the Semaphore disappears, we go into a state (Restarting) where we are going to wait for a while until we can know that the semaphore should be unused, then reset it to it's initial state (ie: unused) and we're good to go again. The way we'll do this is to wait a bit for anyone who might be using the semaphore to finish, and then once we've waiting long enough, we just assume everyone is done and proceed. It's a timeout, and it means imposing a maximum time length on holding the Semaphore that callers need to obey. Semaphores have two bits of internal state which needs to be maintained: - Counter - Wait List The Wait list really needs to be in the datastore, because if it gets flushed there's just no way to reconstitute it, Waiters will be lost forever. But the counter has more promise. Try an algorithm like this: Semaphores only have two methods which need to accomodate these states: Wait() Touch the semaphore. What state are we in? OK: Proceed as normal (if the counter is above zero then decrement it and proceed, else suspend on the wait list) GONE: Reconstruct the semaphore in Restarting state. Suspend on the wait list. RESTARTING: Suspend on the wait list. Signal() Touch the semaphore. What state are we in? OK: Proceed as normal (if there is anyone on the wait list, awaken one. Otherwise, just increment the counter) GONE: Reconstruct the semaphore in RESTARTING state. Increment the counter. If it's equal to TokenMax, then Start the semaphore. RESTARTING: Increment the counter. If it's equal to TokenMax, then Start the semaphore. Semaphores are extended as follows: A reconstructed Semaphore in RESTARTING state is initialised with Counter = 0. We are going to restart the semaphore by - defining a maximum length of time that a Semaphore token can be held (if you're using Front End tasks, just set it longer than the maximum task length). - when we restart, we schedule a Start task for a time in the future equal to the maximum task length. - When the Start task runs, it checks the Semaphore. If it's still in RESTARTING state, it Starts the Semaphore. - There is an ABA problem here. A Start task is scheduled, the Semaphore restarts before the task runs, then goes back into GONE and RESTARTING again (and another Start task is kicked off), then the original Start task runs and prematurely wakes the Semaphore. Fix this by adding a StartTaskID (uuid) to the Semaphore, generate a new one each time the Start Task runs, and pass it to the Start Task. An old StartTask will have the wrong uuid and can terminate itself based on that. Start the Semaphore: Either in the Start Task, or as the result of a Signal, we've determined that it's ok to Start the semaphore. So set the Counter to MaxCounter and change the state to OK. If there is anyone on the Wait List, wake as many of them as the counter allows, decrementing the counter for each one. === I think this approach can work, and the cool thing is that the counter and the state should be implementable in Memcache. If the resource you're using isn't normally under contention (very commonly this is true, contention is unusual), then your most common operations will be in Memcache and wont touch the datastore. This also assumes that cache flushing is uncommon (if it flushed often, then it should still be correct, but the performance will be poor, because the restarting process is pretty slow). If the cache flushes really, really often (sub second?) then you'll starve, but I don't think memcache does that! Any thoughts? On 27 October 2011 23:24, Emlyn emlynore...@gmail.com wrote: Thanks, I had no idea that Memcache was so capable (I haven't looked closely enough recently). You could absolutely build a semaphore using those methods (cas would do the job nicely). Except of course that it could disappear at any moment. I can't help thinking that there must be some way to cope with that, however. Worth a ponder! On 27 October 2011 18:53, Murph paul.j.mu...@googlemail.com wrote: I think it's basically possible to do locking, mutexes, semaphores, etc in memcache, by taking advantage of some of the semantics, plus the new-ish compare store stuff. There's the obvious caveat that it could be flushed without warning, or disabled for a while, so it's not a high reliability thing. def add(self, key, value, time=0, min_compress_len=0, namespace=None): Sets a key's value
Re: [google-appengine] Re: The Dining Philosophers
On 28 October 2011 12:19, Brandon Wirtz drak...@digerat.com wrote: I would think that it would make more sense to store the last read Memcache state to instance memory (basically free), if Memcache is flushed detect and upload last state to memcache. There's no way to guarantee it'd be there, is there? The task that runs and find out that there's no memcache might be in an entirely new instance. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Emlyn Sent: Thursday, October 27, 2011 5:04 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: The Dining Philosophers Germ of an idea: Assume we can implement semaphore entirely in memcache, ignoring the fact that it can be flushed. Then, to cope with being flushed, we could have an idea of a Semaphore as being in one of three states; Ok, Gone and Restarting. Ok is just normal operation Gone is when you know it's been flushed You detect Gone by noticing the problem when you try to touch the Semaphore (in Wait() and Signal() calls). You restart the Semaphore. Restarting is a period where you are trying to get back to Ok state. Basically, the idea will be that if the Semaphore disappears, we go into a state (Restarting) where we are going to wait for a while until we can know that the semaphore should be unused, then reset it to it's initial state (ie: unused) and we're good to go again. The way we'll do this is to wait a bit for anyone who might be using the semaphore to finish, and then once we've waiting long enough, we just assume everyone is done and proceed. It's a timeout, and it means imposing a maximum time length on holding the Semaphore that callers need to obey. Semaphores have two bits of internal state which needs to be maintained: - Counter - Wait List The Wait list really needs to be in the datastore, because if it gets flushed there's just no way to reconstitute it, Waiters will be lost forever. But the counter has more promise. Try an algorithm like this: Semaphores only have two methods which need to accomodate these states: Wait() Touch the semaphore. What state are we in? OK: Proceed as normal (if the counter is above zero then decrement it and proceed, else suspend on the wait list) GONE: Reconstruct the semaphore in Restarting state. Suspend on the wait list. RESTARTING: Suspend on the wait list. Signal() Touch the semaphore. What state are we in? OK: Proceed as normal (if there is anyone on the wait list, awaken one. Otherwise, just increment the counter) GONE: Reconstruct the semaphore in RESTARTING state. Increment the counter. If it's equal to TokenMax, then Start the semaphore. RESTARTING: Increment the counter. If it's equal to TokenMax, then Start the semaphore. Semaphores are extended as follows: A reconstructed Semaphore in RESTARTING state is initialised with Counter = 0. We are going to restart the semaphore by - defining a maximum length of time that a Semaphore token can be held (if you're using Front End tasks, just set it longer than the maximum task length). - when we restart, we schedule a Start task for a time in the future equal to the maximum task length. - When the Start task runs, it checks the Semaphore. If it's still in RESTARTING state, it Starts the Semaphore. - There is an ABA problem here. A Start task is scheduled, the Semaphore restarts before the task runs, then goes back into GONE and RESTARTING again (and another Start task is kicked off), then the original Start task runs and prematurely wakes the Semaphore. Fix this by adding a StartTaskID (uuid) to the Semaphore, generate a new one each time the Start Task runs, and pass it to the Start Task. An old StartTask will have the wrong uuid and can terminate itself based on that. Start the Semaphore: Either in the Start Task, or as the result of a Signal, we've determined that it's ok to Start the semaphore. So set the Counter to MaxCounter and change the state to OK. If there is anyone on the Wait List, wake as many of them as the counter allows, decrementing the counter for each one. === I think this approach can work, and the cool thing is that the counter and the state should be implementable in Memcache. If the resource you're using isn't normally under contention (very commonly this is true, contention is unusual), then your most common operations will be in Memcache and wont touch the datastore. This also assumes that cache flushing is uncommon (if it flushed often, then it should still be correct, but the performance will be poor, because the restarting process is pretty slow). If the cache flushes really, really often (sub second?) then you'll starve, but I don't think memcache does that! Any thoughts? On 27 October 2011 23:24, Emlyn emlynore...@gmail.com wrote: Thanks, I had no idea that Memcache was so capable (I haven't looked
Re: [google-appengine] Re: The Dining Philosophers
On 25 October 2011 02:18, Kyle kyle.rich...@gmail.com wrote: Is there a reason that you didn't use the memcached service to implement semaphores? -Kyle I'm assuming you mean Memcache? Well, two things. Firstly, as far as I know there are no transactions or locks or anything like that for memcache, so where you need a critical section, how will you implement it? (This is not actually a rhetorical question; is there a way?) Secondly, you can't trust Memcache to stick around, as far as I know. If one task constructs a semaphore in memcache, then will that still be there when another tasks tries to wait() on it? Anything that is implemented in terms of Semaphores really needs to be able to rely on them to behave. It might be good to come up with an alternative construct that can tolerate memcache's behaviour, I'm totally open to suggestions. On Oct 23, 6:23 am, Emlyn emlynore...@gmail.com wrote: Here's a new AppEngine article from me, The Dining Philosophers. It includes a full working implementation of Semaphores using the Datastore, and implementation of flawed and successful solutions to the classic Dining Philosophers problem using Semaphores. http://appenginedevelopment.blogspot.com/2011/10/dining-philosophers Fun for hard core comp sci types ;-) -- Emlyn http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all.http://point7.wordpress.com- My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] The Dining Philosophers
Here's a new AppEngine article from me, The Dining Philosophers. It includes a full working implementation of Semaphores using the Datastore, and implementation of flawed and successful solutions to the classic Dining Philosophers problem using Semaphores. http://appenginedevelopment.blogspot.com/2011/10/dining-philosophers.html Fun for hard core comp sci types ;-) -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Python client for the channels API ?
This might sound really weird, but it seems to me that the Channels would make a great interprocess communications mechanism for AppEngine, especially if you're running multiple long lived tasks in backends. Is there any way to consume channels (ie: a channels client) in AppEngine Python? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Python 2.7, threadsafe: true, and deferred.defer
What SDK version are you using? Cheers, Brian I was using 1.5.2 . Using 1.5.5 fixed it. I'm an idiot. Thanks ;-) -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Threaded Python AppEngine for Dummies
is ready for python 2.7, largely around newer versions of libraries being used (eg: webob has changed). Check this page: http://code.google.com/appengine/docs/python/python27/newin27.html -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Threaded Python AppEngine for Dummies
On 15 October 2011 09:12, Ikai Lan (Google) ika...@google.com wrote: Yep, your thinking here is correct! Be careful when using global memory as a cache, though. Instances are capped at 128mb of memory, and if you exceed that, your instance will be killed. This could lead to instance thrashing. [On another note: congrats, you got me to read a long email ;).] -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai Thanks for reading the long email. Sorry, I should keep them shorter, but I'm a natural blatherer. I want to run some tests on the efficacy of using threadsafe:true. Actually hitting real resources in those tests is a bit rude (datastore might be ok, but urlfetch is a bit tough on the target/victim). If I use time.sleep() (eg: use frontend tasks that basically go time.sleep(10)), is that going to block in a similar way to urlfetch or db gets/puts, ie: in a way that'll let the instance process more work? On Fri, Oct 14, 2011 at 2:34 AM, Emlyn emlynore...@gmail.com wrote: These are my first thoughts about approaching threaded python 2.7 apps. Please critique this, I could be totally wrong here! And I don't want to be wrong. Thanks in advance. Hello, dummy here. I'm just beginning my first experiments with python 2.7 apps, using threadsafe: true. But I'm a clueless n00b as far as python goes. Well, not a n00b, but still a beginner. And then this multi-threading thing turns up, and I find myself groaning oh man, really, does it have to get this complex? I think I hear a lot of similar groans out there ;-) I'm betting that the whole multithreaded thing in python appengine apps is scaring plenty of people. I've done a lot of concurrent programming, but the prospect of dealing with threading in python has daunted me a bit because I'm a beginner with python and appengine as it is - this just makes life harder. But hey, it's being added for a reason; I'd best quit complaining and start figuring it out! Thinking about threads and python, I realised that I didn't know how I needed to actually use multi-threading to make my apps leaner and meaner. I mean, why would I use them? They're for doing inherently concurrent things. Serving up pages isn't inherently concurrent stuff, at the app development level. What exactly is expected here? Shouldn't the framework be doing that kind of thing for me? And of course that was the aha moment. The framework *is* doing the work for me. The situation with python appengine development up until now has been that instances process serially. They take a request, see it through to its end. They take another request. And so on. That's cool, but instances spend a lot of time sitting around waiting when they could be doing more work. But with the new python 2.7 support, you can tell appengine that it would be ok to give instances more work when they are blocked waiting for something. eg: if they are doing a big url fetch, or a long query from datastore, something like that, then it's cool to give them another request to begin working on, and come back to the waiting request later when its ready. You do that by setting threadsafe: true in your app.yaml . Being threadsafe sounds scary! But actually it shouldn't be a huge deal. Pretty much it's about what you shouldn't do. Multi-threading means having multiple points of execution on the one codebase in the one address space. Anything you do to touch things external to that (like datastore, memcache, url fetches) shouldn't care about that (assuming the client libraries are threadsafe). And normal code touching local variables will be fine. Probably the only real thing you've got to worry about is using instance memory (global variables more or less). That's because multiple requests, ie: multiple threads, can come in and fiddle with that global memory at the same time. You can fix that with some concurrency primitives, but if that sounds scary you can just avoid touching global memory in the first place. So if you're using instance memory as part of a caching strategy, for instance (caching like instance-memory - memcache - datastore), then you either need to make the instance memory caching threadsafe, or just stop using instance memory for that purpose. The other big gotcha, implied by this issue with global memory, is libraries. Which libraries are threadsafe? Plenty probably aren't, especially some of those shady 3rd party python libs you found lying around on code.google.com . Why not? Because they use global memory. But the built in libs should be ok, unless we've been specifically told they're not, and I don't recall any information like that. Oh, and your app needs to use WSGI script handlers, presumably because the cgi method we were recommended to use in py 2.5 apps is not threadsafe. So to sum up, if you aren't too sure about multi threading and want to keep it simple, it seems like you
Re: [google-appengine] Threaded Python AppEngine for Dummies
I've posted a modified version of my original post here as a blog post. Multi-threaded Python 2.7 WTFAQ? http://appenginedevelopment.blogspot.com/2011/10/multi-threaded-python-27-wtfaq.html -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz On 15 October 2011 13:00, Emlyn emlynore...@gmail.com wrote: On 15 October 2011 09:12, Ikai Lan (Google) ika...@google.com wrote: Yep, your thinking here is correct! Be careful when using global memory as a cache, though. Instances are capped at 128mb of memory, and if you exceed that, your instance will be killed. This could lead to instance thrashing. [On another note: congrats, you got me to read a long email ;).] -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Oct 14, 2011 at 2:34 AM, Emlyn emlynore...@gmail.com wrote: These are my first thoughts about approaching threaded python 2.7 apps. Please critique this, I could be totally wrong here! And I don't want to be wrong. Thanks in advance. Hello, dummy here. I'm just beginning my first experiments with python 2.7 apps, using threadsafe: true. But I'm a clueless n00b as far as python goes. Well, not a n00b, but still a beginner. And then this multi-threading thing turns up, and I find myself groaning oh man, really, does it have to get this complex? I think I hear a lot of similar groans out there ;-) I'm betting that the whole multithreaded thing in python appengine apps is scaring plenty of people. I've done a lot of concurrent programming, but the prospect of dealing with threading in python has daunted me a bit because I'm a beginner with python and appengine as it is - this just makes life harder. But hey, it's being added for a reason; I'd best quit complaining and start figuring it out! Thinking about threads and python, I realised that I didn't know how I needed to actually use multi-threading to make my apps leaner and meaner. I mean, why would I use them? They're for doing inherently concurrent things. Serving up pages isn't inherently concurrent stuff, at the app development level. What exactly is expected here? Shouldn't the framework be doing that kind of thing for me? And of course that was the aha moment. The framework *is* doing the work for me. The situation with python appengine development up until now has been that instances process serially. They take a request, see it through to its end. They take another request. And so on. That's cool, but instances spend a lot of time sitting around waiting when they could be doing more work. But with the new python 2.7 support, you can tell appengine that it would be ok to give instances more work when they are blocked waiting for something. eg: if they are doing a big url fetch, or a long query from datastore, something like that, then it's cool to give them another request to begin working on, and come back to the waiting request later when its ready. You do that by setting threadsafe: true in your app.yaml . Being threadsafe sounds scary! But actually it shouldn't be a huge deal. Pretty much it's about what you shouldn't do. Multi-threading means having multiple points of execution on the one codebase in the one address space. Anything you do to touch things external to that (like datastore, memcache, url fetches) shouldn't care about that (assuming the client libraries are threadsafe). And normal code touching local variables will be fine. Probably the only real thing you've got to worry about is using instance memory (global variables more or less). That's because multiple requests, ie: multiple threads, can come in and fiddle with that global memory at the same time. You can fix that with some concurrency primitives, but if that sounds scary you can just avoid touching global memory in the first place. So if you're using instance memory as part of a caching strategy, for instance (caching like instance-memory - memcache - datastore), then you either need to make the instance memory caching threadsafe, or just stop using instance memory for that purpose. The other big gotcha, implied by this issue with global memory, is libraries. Which libraries are threadsafe? Plenty probably aren't, especially some of those shady 3rd party python libs you found lying around on code.google.com . Why not? Because they use global memory. But the built in libs should be ok, unless we've been specifically told they're not, and I don't recall any information like that. Oh, and your app needs to use WSGI script handlers, presumably because the cgi method we were recommended to use in py 2.5 apps is not threadsafe. So to sum up, if you aren't too sure about multi threading and want to keep it simple, it seems like you can get your existing app processing parallel requests by doing
Re: [google-appengine] New Billing: Absolutely make sure you set Max Idle Instances to a fixed value
np Johan. I was confused earlier on by posts such as http://blorn.com/post/10013293300/the-unofficial-google-app-engine-price-change-faq which focuses on multithreading to get pricing down, which I think is just wrong. There are lots of good reasons to write multithreaded code, but AppEngine pricing isn't really one of them. On 13 October 2011 18:22, Johan Euphrosine pro...@google.com wrote: Hi Emlyn, Thanks for sharing those articles, it is very nice that you were able to backup the billing formula with hard facts. As it was discussed in the groups during the pricing model change the billing formula under the new model will be: billable_instances_rate = min(active_instances_rate + max_idle_instances, total_instances_rate) where in the dashboard: - active_instances_rates is the yellow line - total_instances_rate is the blue line - max_idle_instances is the upper bound of Idle Instances performance settings If you set max_idle_instances to automatic, it's equivalent to setting it to a very large number making the formula essentially become: billable_instances_rate = total_instances_rate See the following threads where Jon McAlister commented about the billing formula: https://groups.google.com/d/msg/google-appengine/zuRXAphGnPk/UiTgTIIesL0J https://groups.google.com/d/msg/google-appengine/W-17IhgwrLI/05Wti7I39EUJ https://groups.google.com/d/msg/google-appengine/T-dJtXmOO8U/npM69XZAJFcJ On Wed, Oct 12, 2011 at 3:10 PM, Emlyn emlynore...@gmail.com wrote: I've been testing this hypothesis: Hypothesis: Ignoring the 15 minute cost for spinning up new instances, the price we pay is the moment by moment minimum of (total instances) and (active instances + Max Idle Instances). If Max Idle Instances is set to Automatic, then we pay for the moment by moment total instances. It holds. That is, if you leave the default Max Idle Instances setting (Automatic), you'll be billed for every bit of instance time the scheduler chooses to run. If you set it to a fixed number, you'll have a fixed cost cap based on the actual work you are doing (a lot more like cpu time), in most cases a lot lower. Here are a couple of posts in detail, with graphs, billing numbers, pictures! The Spiny Norman Test http://appenginedevelopment.blogspot.com/2011/10/spiny-norman-test.html Go Spiny Norman, Go http://appenginedevelopment.blogspot.com/2011/10/go-spiny-norman-go.html -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] New Billing: Absolutely make sure you set Max Idle Instances to a fixed value
does not, in practice, at the moment, change the actual performance of actual apps significantly, only the guaranteed performance. People's needs vary, but I'd hazard a guess that actual price is higher on most dev's radars (particularly smaller devs) than performance guarantees, especially while actual performance is good. For those of us who are price sensitive (and we are talking about price), lowering max idle instances (with the current scheduler behaviour) is a no brainer. Meanwhile, for python users, multi threading is only just becoming an option, and may still not be ready for production use (I don't know, I haven't tried it yet). Also, modifying existing apps to work correctly in a multi-threaded environment is non-trivial (read: tricky and risky) development work. It's something we should all do eventually, but do the benefits as of right now actually warrant the costs and the risks? I'm not convinced that they do. So I'll stand by this on pricing: Theoretically, multi-threaded apps are superior on AppEngine and you should go that way. But in practice, as of right now, and especially for Python, they are not, particularly given that a good pragmatic solution to the issue of instance pricing is available at the touch of a slider. This pragmatic situation may (likely will) change, but right now it is absolutely the case. Note that I'm not saying that multi-threaded code is a bad idea. It's a good idea for all kinds of other reasons. But currently, pricing is not one of those reasons. Reading back over this, if I were on the AppEngine team I would be thinking he's saying we should aggressively clean up idle instances above Max if we want people to move to Python 2.7 and write more efficient multi-threaded code. I may have shot myself in the foot ;-) On Thu, Oct 13, 2011 at 1:21 AM, Emlyn emlynore...@gmail.com wrote: np Johan. I was confused earlier on by posts such as http://blorn.com/post/10013293300/the-unofficial-google-app-engine-price-change-faq which focuses on multithreading to get pricing down, which I think is just wrong. There are lots of good reasons to write multithreaded code, but AppEngine pricing isn't really one of them. On 13 October 2011 18:22, Johan Euphrosine pro...@google.com wrote: Hi Emlyn, Thanks for sharing those articles, it is very nice that you were able to backup the billing formula with hard facts. As it was discussed in the groups during the pricing model change the billing formula under the new model will be: billable_instances_rate = min(active_instances_rate + max_idle_instances, total_instances_rate) where in the dashboard: - active_instances_rates is the yellow line - total_instances_rate is the blue line - max_idle_instances is the upper bound of Idle Instances performance settings If you set max_idle_instances to automatic, it's equivalent to setting it to a very large number making the formula essentially become: billable_instances_rate = total_instances_rate See the following threads where Jon McAlister commented about the billing formula: https://groups.google.com/d/msg/google-appengine/zuRXAphGnPk/UiTgTIIesL0J https://groups.google.com/d/msg/google-appengine/W-17IhgwrLI/05Wti7I39EUJ https://groups.google.com/d/msg/google-appengine/T-dJtXmOO8U/npM69XZAJFcJ On Wed, Oct 12, 2011 at 3:10 PM, Emlyn emlynore...@gmail.com wrote: I've been testing this hypothesis: Hypothesis: Ignoring the 15 minute cost for spinning up new instances, the price we pay is the moment by moment minimum of (total instances) and (active instances + Max Idle Instances). If Max Idle Instances is set to Automatic, then we pay for the moment by moment total instances. It holds. That is, if you leave the default Max Idle Instances setting (Automatic), you'll be billed for every bit of instance time the scheduler chooses to run. If you set it to a fixed number, you'll have a fixed cost cap based on the actual work you are doing (a lot more like cpu time), in most cases a lot lower. Here are a couple of posts in detail, with graphs, billing numbers, pictures! The Spiny Norman Test http://appenginedevelopment.blogspot.com/2011/10/spiny-norman-test.html Go Spiny Norman, Go http://appenginedevelopment.blogspot.com/2011/10/go-spiny-norman-go.html -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you
Re: [google-appengine] New Billing: Absolutely make sure you set Max Idle Instances to a fixed value
On 13 October 2011 21:42, Jeff Schnitzer j...@infohazard.org wrote: Let me summarize: You ran an experiment and discovered that GAE left idle instances running, above and beyond your max-idle-instances setting, for your application at the time that you ran the test. Ok. Well, that test was in fact designed to force the scheduler's hand. I made the burstiest load that I could (I guess I could have changed the default queue settings to make it even worse). I was testing a conjecture about how the billing works, not the normal performance, and it's wouldn't be right to draw conclusions about normal scheduler behaviour based on that. Unfortunately there's little reason to believe that this behavior (lots of free instances) will continue for other applications at other times; it may just be that you ran at an off-peak period. There's little reason to believe this behavior will continue as Google optimizes the scheduler, and a lot of reason to believe that it won't (free instances are bad for the suddenly-relevant bottom line). (Well, I ran the tests for 20 hours at a time, so not offpeak I think.) In my main real app, changing the scheduler down to min idle instances = 1 more or less halves the idle instances (they're still up at 6 or 7 or so while the active count is below 1). But I don't care too much about that, because it's mostly background processing. A little latency here and there means nothing in that context. Should you set the max-idle-instances to a fixed number instead of auto? Depends on your app. If you know how to balance your traffic, startup latency, and avg request lantecy, then sure. If you you're more concerned about your bill than your user experience, then sure. Does this mean you don't need to move to a multi-threaded system? No, because whatever number of idle instances you think you need, you can divide that by N if you go multithreaded. N is a hard number to predict because it depends on how much of your app is i/o bound. Normal Java appservers serving typical webapps achieve concurrency in the hundreds with ease. Ikai once mentioned that the initial concurrency of Java was hardcoded at 10, but that has probably changed by now. I don't know what it will be like with Python but really, it's hard to imagine anything else you can possibly do that will have an order of magnitude effect on your bill. It can't have an order of magnitude effect on your bill unless you are running with Max Idle Instances on Auto or have set it far too high. Jeff Changing Max Idle Instances to 1 for http://my.syyn.cc dropped my projected new billing charges by a factor of 6, with no perceivable performance change (see http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ and http://point7.wordpress.com/2011/09/07/appengine-tuning-an-instance-of-success/ For me, to get another order of magnitude would be nice, but is no longer an emergency. And that's something important, because a lot of people rage-quit AppEngine when the new billing appeared, in the usually false belief that they were hosed under the new system. In fact, while I was in the process of introducing AppEngine in my commercial work for our new development, I got some push back specifically around this issue, because people had read that you must have multi-threaded code or else the billing would be sky high (probably reading your article in fact). And the perception was that this would make the whole thing too hard, and why don't we look at some other platform? So I'm *not* saying there's no point to a multi-threaded code base. I'm saying that it is not the only way to fix instance pricing woes, or even the best way (especially from a developer effort POV). And the idea that multi-threading is the best and only way is turning small time developers off the platform. But I'll tell you what; the effect of multi-threading on appengine python app instance pricing warrants a good hard look, with numbers and tables and graphs and whatnot. Reading your stuff in detail has convinced me to put it on my shortlist, particularly with an eye to developing some simple techniques for web apps, for people who would otherwise be intimidated by heavy concurrency work. So I apologise for raising your hackles, and thank you for the interaction, and the motivation to take a look at multi-threaded Python. On Thu, Oct 13, 2011 at 3:32 AM, Emlyn emlynore...@gmail.com wrote: On 13 October 2011 19:46, Jeff Schnitzer j...@infohazard.org wrote: I'm afraid you are still confused. Possibly not. You have ignored the entire point of the max idle instances slider in the first place. I might be accused of ignoring its intended function and focusing on its practical function. GAE keeps idle instances around so that sudden bursts of traffic don't cause users to sit around waiting while your django/spring app spends 5+ seconds loading. Sure, so that's going
[google-appengine] Python 2.7, threadsafe: true, and deferred.defer
I'm trying to give python 2.7 a go, with an app that uses deferred.defer. I'm getting this error when I try to upload it: Error 400: --- begin server output --- Error when loading application configuration: Invalid object: Threadsafe cannot be enabled with CGI handler: $PYTHON_LIB/google/appengine/ext/deferred/handler.py Is there a workaround? Or should I even be using deferred.defer in this context? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Green line, Billed, on Instances graph.
There's a green line, Billed, on the instances graph. Now you can ignore my blog posts, and just play around with what moves the green line. Brilliant, good stuff AppEngine people. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Python 2.7, threadsafe: true, and deferred.defer
app.yaml: --- application: emlynoregan version: 1 runtime: python27 api_version: 1 threadsafe: true builtins: - deferred: on handlers: - url: /.* script: main.py --- Upload via appcfg.py: --- Application: emlynoregan; version: 1 Host: appengine.google.com Starting update of app: emlynoregan, version: 1 Scanning files on local disk. Error 400: --- begin server output --- Error when loading application configuration: Invalid object: Threadsafe cannot be enabled with CGI handler: $PYTHON_LIB/google/appengine/ext/deferred/handler.py --- On 14 October 2011 00:50, Greg Darke (Google) darke+goo...@google.com wrote: Have you specified 'runtime: python' or 'runtime: python27'? If you are using 'runtime: python', then setting 'threadsafe: true' is not supported. On 14 October 2011 00:59, Emlyn emlynore...@gmail.com wrote: I am using the builtin handler in app.yaml On Oct 14, 2011 12:25 AM, Greg Darke darke+goo...@google.com wrote: On Fri, Oct 14, 2011 at 12:10:09AM +1030, Emlyn wrote: I'm trying to give python 2.7 a go, with an app that uses deferred.defer. I'm getting this error when I try to upload it: Error 400: --- begin server output --- Error when loading application configuration: Invalid object: Threadsafe cannot be enabled with CGI handler: $PYTHON_LIB/google/appengine/ext/deferred/handler.py Is there a workaround? Or should I even be using deferred.defer in this context? The easiest way to solve this problem is to remove the explicit handler you are defining, and use the builtin directive as described in http://code.google.com/appengine/docs/python/config/appconfig.html#Builtin_Handlers If you are running the deferred handler at non-default url, you will need to specify the WSGI application in your app.yaml. This is google.appengine.ext.deferred.application -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Python 2.7, threadsafe: true, and deferred.defer
You are specifying a CGI script here. If you want to use CGI then set
threadsafe to no.
Cheers,
Brian
Ah, gotcha. A fog of deep misunderstanding is slowly lifting from my
brain, possibly. I could still have some Dunning-Kruger going, so
here's my next attempt:
app.yaml:
---
application: emlynoregan
version: 1
runtime: python27
api_version: 1
threadsafe: true
builtins:
- deferred: on
handlers:
- url: /.*
script: main.app
---
and my main.py now says:
---
import webapp2
from starter import Starter
app = webapp2.WSGIApplication([('/spinystarter', Starter)], debug=True)
---
But I'm still getting the same error. It's related to the built-in
defer, because when I remove
builtins:
- deferred: on
from my app.yaml, it's all good. I suspect I need to not use the
builtin directive, and instead do
app = webapp2.WSGIApplication([('/spinystarter', Starter), (something,
something else)], debug=True)
but I don't know what something and something else are supposed to
be. Does anyone know?
---
Upload via appcfg.py:
---
Application: emlynoregan; version: 1
Host: appengine.google.com
Starting update of app: emlynoregan, version: 1
Scanning files on local disk.
Error 400: --- begin server output ---
Error when loading application configuration:
Invalid object:
Threadsafe cannot be enabled with CGI handler:
$PYTHON_LIB/google/appengine/ext/deferred/handler.py
---
On 14 October 2011 00:50, Greg Darke (Google) darke+goo...@google.com
wrote:
Have you specified 'runtime: python' or 'runtime: python27'?
If you are using 'runtime: python', then setting 'threadsafe: true' is
not supported.
On 14 October 2011 00:59, Emlyn emlynore...@gmail.com wrote:
I am using the builtin handler in app.yaml
On Oct 14, 2011 12:25 AM, Greg Darke darke+goo...@google.com wrote:
On Fri, Oct 14, 2011 at 12:10:09AM +1030, Emlyn wrote:
I'm trying to give python 2.7 a go, with an app that uses
deferred.defer. I'm getting this error when I try to upload it:
Error 400: --- begin server output ---
Error when loading application configuration:
Invalid object:
Threadsafe cannot be enabled with CGI handler:
$PYTHON_LIB/google/appengine/ext/deferred/handler.py
Is there a workaround? Or should I even be using deferred.defer in this
context?
The easiest way to solve this problem is to remove the explicit handler
you are defining, and use the builtin directive as described in
http://code.google.com/appengine/docs/python/config/appconfig.html#Builtin_Handlers
If you are running the deferred handler at non-default url, you will
need to specify the WSGI application in your app.yaml. This is
google.appengine.ext.deferred.application
--
Emlyn
http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google
Buzz posts,
comments and all.
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Emlyn
http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google
Buzz posts,
comments and all.
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] New Billing: Absolutely make sure you set Max Idle Instances to a fixed value
I've been testing this hypothesis: Hypothesis: Ignoring the 15 minute cost for spinning up new instances, the price we pay is the moment by moment minimum of (total instances) and (active instances + Max Idle Instances). If Max Idle Instances is set to Automatic, then we pay for the moment by moment total instances. It holds. That is, if you leave the default Max Idle Instances setting (Automatic), you'll be billed for every bit of instance time the scheduler chooses to run. If you set it to a fixed number, you'll have a fixed cost cap based on the actual work you are doing (a lot more like cpu time), in most cases a lot lower. Here are a couple of posts in detail, with graphs, billing numbers, pictures! The Spiny Norman Test http://appenginedevelopment.blogspot.com/2011/10/spiny-norman-test.html Go Spiny Norman, Go http://appenginedevelopment.blogspot.com/2011/10/go-spiny-norman-go.html -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Controlling amazon ec2 instances from appengine
I'd love to hear people's experiences with controlling amazon ec2 instances from appengine. Is anyone doing it? Any hints for best practices? I've come across the python library Boto (http://code.google.com/p/boto/) for talking to amazon; is that the right tool for the job? Also, and this comes up again and again with AppEngine, what do you do when you need to store any kind of private key in AppEngine? Is there any better practise than plain text in the datastore, which isn't just security by obscurity? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] The Spiny Norman Test
A new test about AppEngine billing, following up from The Amazing Story of AppEngine and the Two Orders of Magnitude. This one's called The Spiny Norman Test. http://appenginedevelopment.blogspot.com/2011/10/spiny-norman-test.html -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Simple task processing class - Worker
Hi all,
I've just started a blog on professional level AppEngine coding, and
the first substantive post is on a python class I built for using
tasks on Push Queues for doing scheduled job processing.
It's here: http://appenginedevelopment.blogspot.com/2011/10/worker.html
I'd love some feedback on this. I'm still a bit of a n00b to
AppEngine, so I might be doing it wrong. Are there better ways to do
job processing on AppEngine that I'm missing?
I've posted it inline assuming there are people as lazy as me who
can't be bothered clicking. You're missing links and such, but you can
get the gist.
The Worker
Emlyn O'Regan 1 Oct 2011
One of the first serious Google AppEngine subjects I've approached
recently is the problem of doing work in the background. In my
particular case I needed to do some intensive and error prone tasks,
then send an email with the results (which is also error prone), on a
schedule.
I was going to write some standard job-processing-in-a-loop kind of
code, with the loop being processed as a cron job (set up in
cron.yaml). That's what Syyncc does. But some bit of my brain kept
grumbling about the inelegance of that approach. You're on a platform
that wants to do it a different way, says my brain (and who am I to
disagree?).
And the cron thing is kind of bad, because it doesn't scale. Let's say
I schedule a job every two minutes. It can get through some fixed
amount of work (maybe 10 jobs?) before it hits its time limit. It can
never do more than that. That's nasty.
People often recommend backends for this kind of work. With them, you
stick jobs on a pull queue, and pull them off with the backend. Each
backend can process a limited amount of jobs, but you can set them to
be automatically created in response to workload, which is cool.
But I'm partial to push queues, what were previously just called Task
Queues. At any point in code you can schedule a task to run, which
simply comes through as a post to a url in your app:
taskqueue.add(url='/dosomething', params={'key': key})
It's a bit clunky, because you need to set up a handler for the url,
and implement the Post method.
Oh wait, no you don't. Nick Johnson wrote the excellent deferred.defer
library, which takes care of the public url and thunking the call from
there into a method of your choice. So instead your call can look like
this:
from google.appengine.ext import deferred
def do_something_expensive(a, b, c=None):
logging.info(Doing something expensive!)
# Do your work here
# Somewhere else
deferred.defer(do_something_expensive, Hello, world!, 42, c=True)
That's cool, isn't it!
What's also cool about tasks is that you can delay them, either by
specifying a countdown or an eta. Using a countdown (number of seconds
before execution) is interesting, because you can delay tasks, ie:
spread the work out a bit. But using an eta is really fascinating,
because it lets you schedule work for specific times. So if you need
to schedule an email to go out at midnight, a task with an eta will do
that for you, with no real plumbing required on your part. (Can you do
this with a pull queue? You may be able to use eta to stop tasks
showing up through the lease system before a specified time, I'm not
sure about this.)
This is all great for performing scheduled background tasks. Except,
what if they fail? Or take a long time to complete? In fact, how can
you report on the status of these tasks? Well, you can't. There's no
way to go in and find out much about the task through any APIs. Even
if there was, you'd probably need custom information suited to the job
at hand anyway.
What I need is an object in the datastore that maps to the task. I
personally prefer an object oriented approach (ok, I'm an old man set
in my ways, yes I know). So, what I'd like is a base object which lets
me set up a task, kick it off, record its progress, and lets me see
afterwards how it went.
So I created the Worker. The worker is a base class polymodel object,
that you can use to do background jobs. You need to override it, and
provide it with a job to do (doExecute()) and a method for calculating
the next time to run if you want a repeating job
(doCalculateNextRun()). You can also provide a specific queue name
(override GetQueue()) and you can specify whether or not it should run
immediately (override ExecuteImmediately()). If ExecuteImmediately()
returns false, then on the first, immediate run it wont call
doExecute(), but instead will call doCalculateNextRun() and reschedule
itself.
So for instance, if you want to run a background job immediately (say
send an email), you make this class:
class SendAnEmailImmediately(Worker)
def doExecute(self):
logging.info(Sending emails to %s % lemailStr)
lmessage = mail.EmailMessage(
sender=a...@example.com,
to=be...@example.com,
subject= Hi Betty,
body=I
Re: [google-appengine] Re: The Max Idle Instances setting is really lame.
On 29 September 2011 09:20, Tapir tapir@gmail.com wrote: ok, then it is really not very helpful for performance, it is just a way to limit the cost. right? Yup On Sep 29, 7:14 am, Barry Hunter barrybhun...@gmail.com wrote: Sounds like you have misunderstood Max Idle Instances. Its the maximum number you will be charged for. It is not a guarantee. If you have spiky traffic, it might mean you will have spare capacity ready to handle more requests. Wont help with the first spike. Think you looking for Min Idle Instances. That is only available right now if you have purchased Always On I bleive. Once the new pricing goes live, it will be open to all. On Thu, Sep 29, 2011 at 12:07 AM, Tapir tapir@gmail.com wrote: It is not very useful. When I set it 3, gae will still only keep one idle instances, so my app will still encounter slow startup when 2 visits coming at the same time. Setting it higher is helpless to avoid the java app slow startup problem. I just found the only difference between set Max Idle Instances as 1 and 3 is the total used CPU hours, about 25:29, which means except the free 24 hours, the total used CPU hours is 1:5. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: gaeutilities sessions, plz to help a n00b
On 21 September 2011 01:41, bowman.jos...@gmail.com bowman.jos...@gmail.com wrote: The purpose of restricting logins to one session is to avoid session hijacking. gaeutilities has features that help your site avoid session hijacking which have been made even easier with tools like Firesheep - http://codebutler.com/firesheep Since (as of last I checked) you can't use ssl when using your own domains cookie sniffing is simple for appengine apps. I don't know if I'm understanding this; why would that help? Wouldn't a sidejacked session look exactly like the currently logged in user anyway? How does restricting logged in users to always use the same session help here? What it would do, I guess, is allow you to keep stuff like profile info in the session, and have it immediately available on login. Sure, other libraries are faster, and if all you care about is performance, then I'd suggest using them. The only reason to choose gaeutilities is it was written with security prioritized over performance, therefore is more secure than the other libraries. Not to say it's secure, without ssl it's not truly secure, but it's much more difficult to spoof a gaeutilities session if configured correctly. I'm sticking with gaeutilities for now, because the security looks pretty solid. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XWaPWJ54gt8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: gaeutilities sessions, plz to help a n00b
On 22 September 2011 11:26, bowman.jos...@gmail.com bowman.jos...@gmail.com wrote: The security is the rotating session token. It works basically like this. A session token is valid for x seconds, and then a backlog of x tokens are considered valid. The reason that multiple tokens are valid is to support sites using ajax requests which may have request/response out of sequence with each other. Avoiding taking multiple requests per page view into account, and just doing a single page/response scenario here's how it happens. This model also assumes that each page view is generating a new token, that's not required as you can make the token expiration as long as you want for your application. You request a page, which generates a session. Session token is set in a cookie. Next page request, the token is valid, but expired. A new cookie is set with a new session token. Next page request, same thing. Now, if you lock your user profile to only accepting one session, then hijacking will create a scenario where either the hijacker or user loses their session. So.. User creates session, get's token. Hijacker sniffs token, connects using it, and get's another token. User makes a request, generating a new token. Hijacker connects still using the token they had, which generates a new one. ... Eventually either the user or hijacker has a token that's expired so a new session needs to be created. If it's the user, when they log in they invalidate the session the hijacker is using and reclaim their access. Now, it's not fool proof. If the hijacker is using a complicated enough system they can keep sniffing and resetting their cookies with the victims tokens. They can at least have some access time on the users account. They can also just sniff again to jump back on the session when they get kicked off. There's no way to make it truely secure, just more difficult. Ok, that makes sense. The biggest problem with gaeutilities though is it's currently pretty much unsupported. I've stopped using appengine and with having 2 kids now I don't have time to dedicate to a project I'm not using. I learned python writing gaeutilities, and have since figured out ways to improve the performance - https://github.com/joerussbowman/gaeutilities/issues/2 Oh, totally understand. I'm open to pull requests, or even to someone forking the project and continuing it. I'd be happy to act as an advisor or anything required to assist as long as the contributors can deal with my limited availability. If anyone just wants to fork the entire project and carry it on as long as I'm comfortable with the approaches taken I'd even point everyone to it. The only qualification I have is that security remain a primary motivator of design. Of course if it's a fork of my code and ideas I'd also like to continue to receive credit. Well, if I begin to run into issues with it that I need to patch myself, I'll yell out, maybe get involved. Thanks for the awesome library though, it really is smooth to use. Great stuff. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Interrogate a push queue from code?
Is there a way to interrogate a push queue from code? ie: list the tasks in the queue, and maybe get the status of each task? Or, is there a way, given a task object, to get the current status of the task? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] gaeutilities sessions, plz to help a n00b
I'm trying the gaeutilities session library for some new apps I'm building. http://gaeutilities.appspot.com/session I'm using it for two things: - basic, old school session stuff (ie: a dictionary persistent across user visits) - securing ajax callbacks (the session contains the indication of whether the user is logged in; session library's cookie goes from server to client, back through ajax call through rest interface to server, session is reconsituted based on it, if it's not the same one then there's no login indication, call fails) The library is pretty cool, a bit magical actually. I'm reading the code to try to understand the magic, and think I'm getting a handle on it. However, in the doc, there is this: In order to take advantage of the token system for an authentication system, you will want to tie sessions to accounts, and make sure only one session is valid for an account. You can do this by setting a db.ReferenceProperty(_AppEngineUtilities_Session) attribute on your user Model, and use the get_ds_entity() method on a valid session to populate it on login. Why would I want to do this? I'm happy for two separate logins by the same person to have different sessions, and I'm happy for subsequent visits to begin with an empty session dictionary each time. Am I missing something here? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] gaeutilities sessions, plz to help a n00b
Thanks Matt, that looks great. I'll give it a shot. On 20 September 2011 12:13, Matt Jibson matt.jib...@gmail.com wrote: gaeutilities sounds nice, but in practice it is real slow. Use gaesessions or webapp2 sessions instead: https://github.com/dound/gae-sessions/wiki/comparison-with-alternative-libraries On Mon, Sep 19, 2011 at 8:21 PM, Emlyn emlynore...@gmail.com wrote: I'm trying the gaeutilities session library for some new apps I'm building. http://gaeutilities.appspot.com/session I'm using it for two things: - basic, old school session stuff (ie: a dictionary persistent across user visits) - securing ajax callbacks (the session contains the indication of whether the user is logged in; session library's cookie goes from server to client, back through ajax call through rest interface to server, session is reconsituted based on it, if it's not the same one then there's no login indication, call fails) The library is pretty cool, a bit magical actually. I'm reading the code to try to understand the magic, and think I'm getting a handle on it. However, in the doc, there is this: In order to take advantage of the token system for an authentication system, you will want to tie sessions to accounts, and make sure only one session is valid for an account. You can do this by setting a db.ReferenceProperty(_AppEngineUtilities_Session) attribute on your user Model, and use the get_ds_entity() method on a valid session to populate it on login. Why would I want to do this? I'm happy for two separate logins by the same person to have different sessions, and I'm happy for subsequent visits to begin with an empty session dictionary each time. Am I missing something here? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Python library for uploading to AppEngine
Yep. Or, can you consume appcfg.py from inside your app? If you can, then that's the answer to my question. yes, in advance, I am a hopeless n00b, sob On 11 September 2011 15:47, Jan Zawadzki / Hapara jan.zawad...@hapara.com wrote: At the risk of asking the obvious, you mean other than appcfg.py? J On Sep 11, 4:48 pm, Emlyn emlynore...@gmail.com wrote: Is there a python library for uploading code to AppEngine? I'd like to build an App that can upload code to other apps, has anyone done this already? -- Emlyn http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all.http://point7.wordpress.com- My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Python library for uploading to AppEngine
On 11 September 2011 16:17, Robert Kluin robert.kl...@gmail.com wrote: It is just a Python script. Open it up and have a look at what it does. It's not just a python script, it's a fair sized system. I had a quick look a while back and thought, hmm, this will take work. So before I get into it again, I thought I'd be an idiot for not asking first if someone else has done the work. If not, I'll be happy to do it and post the results! On Sun, Sep 11, 2011 at 01:37, Emlyn emlynore...@gmail.com wrote: Yep. Or, can you consume appcfg.py from inside your app? If you can, then that's the answer to my question. yes, in advance, I am a hopeless n00b, sob On 11 September 2011 15:47, Jan Zawadzki / Hapara jan.zawad...@hapara.com wrote: At the risk of asking the obvious, you mean other than appcfg.py? J On Sep 11, 4:48 pm, Emlyn emlynore...@gmail.com wrote: Is there a python library for uploading code to AppEngine? I'd like to build an App that can upload code to other apps, has anyone done this already? -- Emlyn http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all.http://point7.wordpress.com- My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Python library for uploading to AppEngine
My google-fu is failing. Do you have any specific links to more info on uploading code using remote_api? On 11 September 2011 16:58, Tim Hoffman zutes...@gmail.com wrote: HI As robert says it is just python. Also its using the remote_api. So all definately do-able. I run remote_api stuff inside zope to transparently access the datastore from within zope. Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TKWNqEhYNuIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
On 11 September 2011 18:17, Gerald Tan woefulwab...@gmail.com wrote: I'm ok with that. cool Your next concern would be the Datastore Reads. I think you should be able to bring that down a lot by using memcache and/or combining your monitors into a few entities. I'm not familiar with python, but I believe you can use pickle to serialize an array of monitors into a blob which you can store as one entity. You may need to split your list of monitors into a few groups to stay under the max entity size of 1MB, but this will drastically cut down on the number of reads that you need, especially if you combine with a memcache. Oh, I don't need to visit all the monitors, far from it. It's just a bit of crap code doing that which needs to be refactored out entirely. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/3QXRW2UJcB8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
I've just posted the last of what became 4 posts in this series. http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ http://point7.wordpress.com/2011/09/04/appengine-tuning-1/ http://point7.wordpress.com/2011/09/07/appengine-tuning-an-instance-of-success/ http://point7.wordpress.com/2011/09/10/appengine-tuning-schlemiel-youre-fired/ tl;dr is, that my pricing's back down really low, things have worked out. btw I've had great feedback, tips and techniques from this community. Thanks! I think that, regarding longevity of a tech, the culture that builds around a it is just as important as the tech itself. All signs are that AppEngine is going to be a long term viable platform. On 3 September 2011 19:46, Emlyn emlynore...@gmail.com wrote: Hi all, I don't think I've posted here before, but I've been an appengine user for a while now (closing on 2 years? Is that even possible?). And like many, I had a rude shock with the new pricing (going from $0.50/day to $50/day). However, I dug into what I'm actually being charged for, and I think it's all actually in my control to sort out, and that in itself is sort of fascinating. I wrote a long blog post on this, which people might find interesting. The Amazing Story Of Appengine And The Two Orders Of Magnitude http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ I'd be really grateful for feedback, especially if I've gotten anything wildly wrong. I haven't actually made any of the changes that I've foreshadowed in the post, that's for the next day or two, and I'll write a followup article on how it goes. Thanks in advance for having a look! -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Python library for uploading to AppEngine
Is there a python library for uploading code to AppEngine? I'd like to build an App that can upload code to other apps, has anyone done this already? -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Oh wow, you're absolutely right. Going back to the billing on the 4th of September (after I changed Max Idle Instances and before I made any code changes), I was already seeing the full price drop. So, I didn't need to make the changes to spread out my tasks; that dropped the blue line down, but I don't pay for the blue line. So optimisation is even easier than I thought. Gerald, I'll quote you in an update to my post, if you're ok with that. On 11 September 2011 02:04, Gerald Tan woefulwab...@gmail.com wrote: Nice blog Emlyn. The reason why your Frontend Instance hours are lower than you expected is because you assumed that you will be billed for the area under the BLUE line in the Instance graph. It's not. You are being billed for the area under the YELLOW line (Active Instance) PLUS your Max Idle Instance setting. So your Active Instances is hovering at around ~0.72, and I assume you have set your application's Max Idle Instance to 1. Therefore ~1.72 * 24 = ~41.28 Instance Hours -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dWoTZKzCy7kJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Here's the next post, showing the results of just changing the performance sliders. Again, hard data. graphs, all the good stuff. http://point7.wordpress.com/2011/09/04/appengine-tuning-1/ On 3 September 2011 19:46, Emlyn emlynore...@gmail.com wrote: Hi all, I don't think I've posted here before, but I've been an appengine user for a while now (closing on 2 years? Is that even possible?). And like many, I had a rude shock with the new pricing (going from $0.50/day to $50/day). However, I dug into what I'm actually being charged for, and I think it's all actually in my control to sort out, and that in itself is sort of fascinating. I wrote a long blog post on this, which people might find interesting. The Amazing Story Of Appengine And The Two Orders Of Magnitude http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ I'd be really grateful for feedback, especially if I've gotten anything wildly wrong. I haven't actually made any of the changes that I've foreshadowed in the post, that's for the next day or two, and I'll write a followup article on how it goes. Thanks in advance for having a look! -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] The Amazing Story Of Appengine And The Two Orders Of Magnitude
Hi all, I don't think I've posted here before, but I've been an appengine user for a while now (closing on 2 years? Is that even possible?). And like many, I had a rude shock with the new pricing (going from $0.50/day to $50/day). However, I dug into what I'm actually being charged for, and I think it's all actually in my control to sort out, and that in itself is sort of fascinating. I wrote a long blog post on this, which people might find interesting. The Amazing Story Of Appengine And The Two Orders Of Magnitude http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ I'd be really grateful for feedback, especially if I've gotten anything wildly wrong. I haven't actually made any of the changes that I've foreshadowed in the post, that's for the next day or two, and I'll write a followup article on how it goes. Thanks in advance for having a look! -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] The Amazing Story Of Appengine And The Two Orders Of Magnitude
On 4 September 2011 01:35, Joshua Smith joshuaesm...@charter.net wrote: I also identified task queues as the source of my excessive instances. I suspect this is a quite common issue, and together with datastore access bugs (sorry, but you never should have written it like that) I totally agree with you! I don't clearly remember writing that bit of code, but it looks like a kludgy workaround for a bug. This app is an evenings weekends thing, so what can I do in 10 mins to fix critical issue X is often the driving factor. That doesn't lead to good code ;-) like the one you found, are conspiring to make a lot of these crazy new billing numbers. I think that's going to end up being true. What Google are doing with the new billing is surfacing the true costs of the platform, which were hidden before. It's pain we've got to go through, and it'll be better for all of us in the long run. That said, it would have been nice to have had a little longer between getting access to the new billing tools and actually wearing the new costs. But it does focus the mind! I think a lot of people just need to take a deep breath and look at their apps the way you did. It's certainly a lot less work than migrating to EC2. Absolutely. We need to not lose sight of just how great a PAAS platform this is, and how much we gain by using it. On Sep 3, 2011, at 6:16 AM, Emlyn wrote: Hi all, I don't think I've posted here before, but I've been an appengine user for a while now (closing on 2 years? Is that even possible?). And like many, I had a rude shock with the new pricing (going from $0.50/day to $50/day). However, I dug into what I'm actually being charged for, and I think it's all actually in my control to sort out, and that in itself is sort of fascinating. I wrote a long blog post on this, which people might find interesting. The Amazing Story Of Appengine And The Two Orders Of Magnitude http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ I'd be really grateful for feedback, especially if I've gotten anything wildly wrong. I haven't actually made any of the changes that I've foreshadowed in the post, that's for the next day or two, and I'll write a followup article on how it goes. Thanks in advance for having a look! -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] The Amazing Story Of Appengine And The Two Orders Of Magnitude
+1 On 4 September 2011 03:51, Joshua Smith joshuaesm...@charter.net wrote: Yes, that certainly seems possible. But it's a lot more trouble than just having a way to say to the scheduler, run these when you have an idle instance with nothing better to do, or DO NOT spin up an instance just to handle this task On Sep 3, 2011, at 12:14 PM, peterk wrote: You can pull from task queues instead of having them push, right? That could help a lot where task queues are the source of instance spin up. I'd even be happy to dedicated a back-end to task queue pull-work if it was necessary - at least that is totally under your control. On Sep 3, 5:05 pm, Joshua Smith joshuaesm...@charter.net wrote: I also identified task queues as the source of my excessive instances. I suspect this is a quite common issue, and together with datastore access bugs (sorry, but you never should have written it like that) like the one you found, are conspiring to make a lot of these crazy new billing numbers. I think a lot of people just need to take a deep breath and look at their apps the way you did. It's certainly a lot less work than migrating to EC2. On Sep 3, 2011, at 6:16 AM, Emlyn wrote: Hi all, I don't think I've posted here before, but I've been an appengine user for a while now (closing on 2 years? Is that even possible?). And like many, I had a rude shock with the new pricing (going from $0.50/day to $50/day). However, I dug into what I'm actually being charged for, and I think it's all actually in my control to sort out, and that in itself is sort of fascinating. I wrote a long blog post on this, which people might find interesting. The Amazing Story Of Appengine And The Two Orders Of Magnitude http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... I'd be really grateful for feedback, especially if I've gotten anything wildly wrong. I haven't actually made any of the changes that I've foreshadowed in the post, that's for the next day or two, and I'll write a followup article on how it goes. Thanks in advance for having a look! -- Emlyn http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com- My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Application Error: 5 when calling WordPress via xmlrpclib
It's already in a task. Also, I'm not directly using urlfetch; I'm using
pyblog.py which uses xmlrpclib which I guess ultimately uses urlfetch, so I
can't set the deadline directly (although there do seem to be timeout
mechanisms, they just don't work past 5 seconds as GAE cuts the call off).
One thing I'm hoping someone can answer is, will using the new backend
functionality result in getting a longer time limit here? Does anyone know?
On 25 May 2011 13:44, Robert Kluin robert.kl...@gmail.com wrote:
Hi Emlyn,
You could move the request to a task, then increase the deadline.
That would probably be the easiest solution.
http://code.google.com/appengine/docs/python/urlfetch/fetchfunction.html
Robert
On Tue, May 17, 2011 at 09:13, Emlyn emlynore...@gmail.com wrote:
Hi,
I'm struggling with talking to WordPress from my gae app.
I've got code that works some of the time, but sometime throws
Application Error: 5, which I believe means it is being timed out by
GAE. Application Error: 5 is thrown if the communication with
WordPress takes longer than 5 seconds.
Here's the code (which runs in the default instance, not a backend):
from pyblog import WordPress
...
try:
logging.debug(About to post to wp)
wp = WordPress(lserverapi, lusername, lpassword)
lwppost = {}
lwppost['description'] = lpostcontent
lwppost['title'] = lpostcontent[:140]
result = wp.new_post(lwppost)
lpostresult = Posted to wp, result: %s % ( str(result) )
logging.info(lpostresult)
except Exception, ex:
logging.error(ex)
lpostresult = '** Exception: %s **' % (str(ex))
pyblog is here: http://code.google.com/p/python-blogger/
pyblog uses xmlrpclib (it's a fairly trivial wrapper over xmlrpclib).
It's driving me to distraction, because the wp.new_post() call times
out, but also succeeds (ie: the post is created). But for what I'm
doing, I *must* get the id back from the new_post() call so I can
store it away for later (and so recognise later that I actually
created the post).
Is there a way to lengthen the timeout past 5 seconds for xmlrpclib?
Or
Will using a backend make the timeout longer / let me set a longer
timeout?
Or
Is there something different/better I should be doing to talk to
WordPress?
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz
posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio that I built
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio that I built
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Application Error: 5 when calling WordPress via xmlrpclib
Gah, really, is that the only way? I'd really, really like to not have to
have a proxy sitting somewhere else doing the heavy lifting (well, the heavy
waiting really).
On 25 May 2011 14:02, Brandon Wirtz drak...@digerat.com wrote:
URL Fetch is good for 10 seconds. WP often is not that fast. Likely you
will need to write a PHP Proxy that excepts Post requests and use it to
forward to the XMLRPC receiver on Wordpress, that way you can send the data
and have PHP do the waiting. (that’s assuming you are adding not querying).
*From:* google-appengine@googlegroups.com [mailto:
google-appengine@googlegroups.com] *On Behalf Of *Emlyn
*Sent:* Tuesday, May 24, 2011 9:29 PM
*To:* google-appengine@googlegroups.com
*Subject:* Re: [google-appengine] Application Error: 5 when calling
WordPress via xmlrpclib
It's already in a task. Also, I'm not directly using urlfetch; I'm using
pyblog.py which uses xmlrpclib which I guess ultimately uses urlfetch, so I
can't set the deadline directly (although there do seem to be timeout
mechanisms, they just don't work past 5 seconds as GAE cuts the call off).
One thing I'm hoping someone can answer is, will using the new backend
functionality result in getting a longer time limit here? Does anyone know?
On 25 May 2011 13:44, Robert Kluin robert.kl...@gmail.com wrote:
Hi Emlyn,
You could move the request to a task, then increase the deadline.
That would probably be the easiest solution.
http://code.google.com/appengine/docs/python/urlfetch/fetchfunction.html
Robert
On Tue, May 17, 2011 at 09:13, Emlyn emlynore...@gmail.com wrote:
Hi,
I'm struggling with talking to WordPress from my gae app.
I've got code that works some of the time, but sometime throws
Application Error: 5, which I believe means it is being timed out by
GAE. Application Error: 5 is thrown if the communication with
WordPress takes longer than 5 seconds.
Here's the code (which runs in the default instance, not a backend):
from pyblog import WordPress
...
try:
logging.debug(About to post to wp)
wp = WordPress(lserverapi, lusername, lpassword)
lwppost = {}
lwppost['description'] = lpostcontent
lwppost['title'] = lpostcontent[:140]
result = wp.new_post(lwppost)
lpostresult = Posted to wp, result: %s % ( str(result) )
logging.info(lpostresult)
except Exception, ex:
logging.error(ex)
lpostresult = '** Exception: %s **' % (str(ex))
pyblog is here: http://code.google.com/p/python-blogger/
pyblog uses xmlrpclib (it's a fairly trivial wrapper over xmlrpclib).
It's driving me to distraction, because the wp.new_post() call times
out, but also succeeds (ie: the post is created). But for what I'm
doing, I *must* get the id back from the new_post() call so I can
store it away for later (and so recognise later that I actually
created the post).
Is there a way to lengthen the timeout past 5 seconds for xmlrpclib?
Or
Will using a backend make the timeout longer / let me set a longer
timeout?
Or
Is there something different/better I should be doing to talk to
WordPress?
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz
posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio that I built
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio that I built
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine
Re: [google-appengine] Application Error: 5 when calling WordPress via xmlrpclib
What I am experiencing via xmlrpclib is 5 seconds, in a task.
On 25 May 2011 14:15, Robert Kluin robert.kl...@gmail.com wrote:
I think the 10 second deadline is only for 'online' (aka user)
requests, in tasks it is 10 *minutes* isn't it?
http://code.google.com/appengine/docs/python/urlfetch/overview.html#Requests
Robert
On Wed, May 25, 2011 at 00:41, Emlyn emlynore...@gmail.com wrote:
Gah, really, is that the only way? I'd really, really like to not have to
have a proxy sitting somewhere else doing the heavy lifting (well, the
heavy
waiting really).
On 25 May 2011 14:02, Brandon Wirtz drak...@digerat.com wrote:
URL Fetch is good for 10 seconds. WP often is not that fast. Likely
you
will need to write a PHP Proxy that excepts Post requests and use it to
forward to the XMLRPC receiver on Wordpress, that way you can send the
data
and have PHP do the waiting. (that’s assuming you are adding not
querying).
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Emlyn
Sent: Tuesday, May 24, 2011 9:29 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Application Error: 5 when calling
WordPress via xmlrpclib
It's already in a task. Also, I'm not directly using urlfetch; I'm using
pyblog.py which uses xmlrpclib which I guess ultimately uses urlfetch,
so I
can't set the deadline directly (although there do seem to be timeout
mechanisms, they just don't work past 5 seconds as GAE cuts the call
off).
One thing I'm hoping someone can answer is, will using the new backend
functionality result in getting a longer time limit here? Does anyone
know?
On 25 May 2011 13:44, Robert Kluin robert.kl...@gmail.com wrote:
Hi Emlyn,
You could move the request to a task, then increase the deadline.
That would probably be the easiest solution.
http://code.google.com/appengine/docs/python/urlfetch/fetchfunction.html
Robert
On Tue, May 17, 2011 at 09:13, Emlyn emlynore...@gmail.com wrote:
Hi,
I'm struggling with talking to WordPress from my gae app.
I've got code that works some of the time, but sometime throws
Application Error: 5, which I believe means it is being timed out by
GAE. Application Error: 5 is thrown if the communication with
WordPress takes longer than 5 seconds.
Here's the code (which runs in the default instance, not a backend):
from pyblog import WordPress
...
try:
logging.debug(About to post to wp)
wp = WordPress(lserverapi, lusername, lpassword)
lwppost = {}
lwppost['description'] = lpostcontent
lwppost['title'] = lpostcontent[:140]
result = wp.new_post(lwppost)
lpostresult = Posted to wp, result: %s % ( str(result) )
logging.info(lpostresult)
except Exception, ex:
logging.error(ex)
lpostresult = '** Exception: %s **' % (str(ex))
pyblog is here: http://code.google.com/p/python-blogger/
pyblog uses xmlrpclib (it's a fairly trivial wrapper over xmlrpclib).
It's driving me to distraction, because the wp.new_post() call times
out, but also succeeds (ie: the post is created). But for what I'm
doing, I *must* get the id back from the new_post() call so I can
store it away for later (and so recognise later that I actually
created the post).
Is there a way to lengthen the timeout past 5 seconds for xmlrpclib?
Or
Will using a backend make the timeout longer / let me set a longer
timeout?
Or
Is there something different/better I should be doing to talk to
WordPress?
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz
posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio that I built
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google
Groups Google App Engine group.
To post to this group, send email to
google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz
posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio
[google-appengine] Application Error: 5 when calling WordPress via xmlrpclib
Hi,
I'm struggling with talking to WordPress from my gae app.
I've got code that works some of the time, but sometime throws
Application Error: 5, which I believe means it is being timed out by
GAE. Application Error: 5 is thrown if the communication with
WordPress takes longer than 5 seconds.
Here's the code (which runs in the default instance, not a backend):
from pyblog import WordPress
...
try:
logging.debug(About to post to wp)
wp = WordPress(lserverapi, lusername, lpassword)
lwppost = {}
lwppost['description'] = lpostcontent
lwppost['title'] = lpostcontent[:140]
result = wp.new_post(lwppost)
lpostresult = Posted to wp, result: %s % ( str(result) )
logging.info(lpostresult)
except Exception, ex:
logging.error(ex)
lpostresult = '** Exception: %s **' % (str(ex))
pyblog is here: http://code.google.com/p/python-blogger/
pyblog uses xmlrpclib (it's a fairly trivial wrapper over xmlrpclib).
It's driving me to distraction, because the wp.new_post() call times
out, but also succeeds (ie: the post is created). But for what I'm
doing, I *must* get the id back from the new_post() call so I can
store it away for later (and so recognise later that I actually
created the post).
Is there a way to lengthen the timeout past 5 seconds for xmlrpclib?
Or
Will using a backend make the timeout longer / let me set a longer timeout?
Or
Is there something different/better I should be doing to talk to WordPress?
--
Emlyn
http://my.syyn.cc - Synchonise Facebook, WordPress and Google Buzz posts,
comments and all.
http://www.blahblahbleh.com - A simple youtube radio that I built
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
