[google-appengine] Re: Using GAE for a REST API
FYI, here's another RESTful framework on GAE called "jsonengine" which allows you to develop a simple web app without server side coding. http://code.google.com/p/jsonengine/ Thanks, Kaz On 11月25日, 午前6:41, supercobra wrote: > Hello there, > > What is your best practice to develop REST on GAE? > > Daniel > -- > superco...@gmail.com > > > > > > > > On Wed, Nov 24, 2010 at 2:36 PM, Jay wrote: > > Yahel, GAE is a great platform for building and delivering RESTful > > services. Of course there are quotas. But you are in control of how > > much quota you want to a large extent. > > > On Nov 23, 9:43 pm, Yahel Carmon wrote: > >> Hi all, > > >> I'm new to playing with GAE, and I'm thinking of using it to build a very > >> simple API for one of my projects. I like the idea of the > >> scaling (mostly) handling itself for an API. > > >> However, I've read things about things like rate limiting triggering > >> captchas for end users, a la Google services. Does that type of rate > >> limiting present problems with using GAE as a backend for a REST API? > > >> I apologize if I'm asking a dumb or obvious question, but its surprisingly > >> difficult to Google for if GAE can be used to build APIs without finding > >> pages* *about GAE's API. > > >> Thanks in advance! > > >> -- > >> Yahel Carmon > >> (917) 445-3498 > >> Twitter:http://twitter.com/yahelc > >> Facebook:http://facebook.com/yahel > >> LinkedIn:http://www.linkedin.com/in/yahelc > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Do Task Queues dream of MapReduce?
Thanks for checking the slide. Although they were taking video for the sessions at DevFest, my session was in Japanese and they may not publish the video. Thanks, Kaz On Apr 6, 1:52 am, "Ikai L (Google)" wrote: > Very cool! Is there a video of this somewhere (the slides are pretty good > themselves, but it'd be nice to match it up with audio). > > > > On Fri, Apr 2, 2010 at 6:18 PM, kazunori_279 wrote: > > Hi all, > > > FYI, here's my presentation slides for my session in DevFest in Japan > > last month, where I discussed about how to utilize the parallelism of > > the App Engine via Task Queue: > > >http://prezi.com/rewcweixasyj/do-task-queues-dream-of-mapreduce/ > > > Contents: > > > Tips and tricks about Google App Engine's Task Queue service and > > parallel processing with it. > > > - 1. What is Task Queue > > - 2. Parallel Query Demo > > - 3. The App Engine Parallelism > > - 4. Concurrency Control on TQ > > > Thanks, > > > Kaz > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Do Task Queues dream of MapReduce?
Hi all, FYI, here's my presentation slides for my session in DevFest in Japan last month, where I discussed about how to utilize the parallelism of the App Engine via Task Queue: http://prezi.com/rewcweixasyj/do-task-queues-dream-of-mapreduce/ Contents: Tips and tricks about Google App Engine's Task Queue service and parallel processing with it. - 1. What is Task Queue - 2. Parallel Query Demo - 3. The App Engine Parallelism - 4. Concurrency Control on TQ Thanks, Kaz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Any guarantees on time sync among servers?
Hi Ikai, Oops, I pasted a wrong link. Here's my code: http://code.google.com/p/sth-samples/source/browse/trunk/sth-samples/src/jp/co/sth/samples/service/CounterService.java By "Log" I do not mean the log messages, but actual entities which works as journals of the counter updates (or whatever you write on memcache). Instead of updating existing entities (which can be a bottleneck), it would be scalable to add new entities to gain the durability. Thanks, Kaz On Apr 1, 2:24 am, "Ikai L (Google)" wrote: > That link goes to a groups post. > > I wouldn't use logging for this. Right now we are saving logs for what looks > like 90 days, but this may change such that logging is only until you run > out of space. Also - you can't programmatically filter on values in logs at > the application level easily. > > > > On Tue, Mar 30, 2010 at 8:12 PM, kazunori_279 wrote: > > FYI, here's my another proposal as an alternative to the sharded > > counter: LogCounter. > > >http://groups.google.com/group/google-appengine/browse_thread/thread/... > > > The basic idea is to use MemcacheService#increment to get unique and > > sequential integer values, while writing logs for them to obtain a > > durability. While it should be as durable as the option 4, while it > > should be more scalable and faster because the log writing can be in > > parallel and there's no point of race condition. > > > Thanks, > > > Kaz > > > On Mar 27, 11:10 pm, jbdhl wrote: > > > Nice points. I actually think approach 2) will suffice. Thanks alot! > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Any guarantees on time sync among servers?
FYI, here's my another proposal as an alternative to the sharded counter: LogCounter. http://groups.google.com/group/google-appengine/browse_thread/thread/da02ebd3001223f7/622bc30435097715?show_docid=622bc30435097715 The basic idea is to use MemcacheService#increment to get unique and sequential integer values, while writing logs for them to obtain a durability. While it should be as durable as the option 4, while it should be more scalable and faster because the log writing can be in parallel and there's no point of race condition. Thanks, Kaz On Mar 27, 11:10 pm, jbdhl wrote: > Nice points. I actually think approach 2) will suffice. Thanks alot! -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Parallel access to appengine
Hi All
> Actually, the default limit is 30 active requests.
> I can only reach 10 active requests without error.
Same here. I've been checking the instance count on my Task Queue
servlet and I've never seen it exceeds the 10 instance limit. There
are two other App Engine developers on my twitter TL who are also
experiencing the 10 limit. What's this? What's the relation between
"30 active requests" and the number of instances? Since an App Engine
Java instance can only run one request at a time, there should be 30
instances to handle 30 active requests.
Actually, it was 16 last year - I and another developer was observing
a 16 instance limit when you make your servlet busy with TQ or many
client requests. I've been thinking it must be a some kind of internal
safe valve. Is it possible to extend that by submitting a request to
Google? So that we can use TQ as a mean of large distributed batch
processing just like MapReduce.
Thanks,
Kaz
On Mar 3, 12:45 am, Gary Orser wrote:
> But that's the point. I can not reach 30 active requests.
> I can only reach 10 active requests without error.
>
> Any ideas on how I can debug this?
>
> Cheers, Gary.
>
> On Mar 2, 7:05 am, "Nick Johnson (Google)"
> wrote:
>
> > Hi,
>
> > On Tue, Mar 2, 2010 at 1:54 PM, Wooble wrote:
> > > The 500 requests per second number relies on the probably-unreasonable
> > > assumption that each request can complete in ~75ms. Deliberately
> > > making your requests take a whole 3 seconds each is, obviously, not
> > > going to work. You can only have 10 instances active at a time by
> > > default; if the pages you're serving actually take 3 seconds to
> > > complete you'll need to optimize things a whole lot or be stuck with a
> > > 3.33 request/sec maximum.
>
> > Actually, the default limit is 30 active requests.
>
> > -Nick Johnson
>
> > > On Mar 1, 11:33 pm, Gary Orser wrote:
> > > > Hi Nick,
>
> > > > Hmm, I was running tests on a billing enabled appspot today. 100
> > > > requests/test.
>
> > > > 10 threads getting a URL with a 3 second sleep (to emulate
> > > > computation) on appspot, was the most I could get without getting 500
> > > > errors.
> > > > If I raised the thread pool beyond 10, I started getting errors??
>
> > > > That doesn't reconcile very well with this statement from the
> > > > appengine website.
> > > > "Requests
> > > > The total number of requests to the app. The per-minute quotas for
> > > > application with billing enabled allow for up to 500 requests per
> > > > second--more than one billion requests per month. If your application
> > > > requires even higher quotas than the "billing-enabled" values listed
> > > > below, you can request an increase in these limits here.
> > > > "
>
> > > > Is there some billing setting that affects this?
>
> > > > Cheers, Gary
>
> > > > PS. dead simple request handler.
>
> > > > import time
> > > > from django import http
> > > > def sit(req):
> > > > time.sleep(3)
> > > > return http.HttpResponse('foo')
>
> > > > errors are:
>
> > > > 03-01 04:15PM 48.177 /sit/91 500 10019ms 0cpu_ms 0kb gzip(gfe)
> > > > 153.90.236.210 - - [01/Mar/2010:16:15:58 -0800] "GET /sit/91 HTTP/1.1"
> > > > 500 0 - "gzip(gfe)" ".appspot.com"
> > > > W 03-01 04:15PM 58.197
> > > > Request was aborted after waiting too long to attempt to service your
> > > > request. Most likely, this indicates that you have reached your
> > > > simultaneous dynamic request limit. This is almost always due to
> > > > excessively high latency in your app. Please seehttp://
> > > code.google.com/appengine/docs/quotas.htmlfor more details.
>
> > > > On Mar 1, 2:36 pm, Michael Wesner wrote:
>
> > > > > Correction/addition to my last email.
>
> > > > > It turns out that our requests for this EC2 pull thing are actually
> > > much faster now. Gary and our other devs have reworked it. I need
> > > updated
> > > numbers, but they don't take 10s, probably more like 2s. We still have
> > > some
> > > heavy ~5s services though, so the same issue exists with the simul-req
> > > stuff, just to less extent. We don't actually hit this limit much now
> > > with
> > > the current beta that is in production, but it is low traffic at the
> > > moment.
> > > We are just getting ready to ramp up heavily.
>
> > > > > I asked Nick what we should do, well just today after my last email, I
> > > have made contact with a Developer Advocate and whatnot, which is
> > > fantastic.
> > > It looks like we, as a business, will be able to have better contact
> > > with
> > > the GAE team. We would very much like to continue working with you to
> > > figure
> > > out what actions we can take and what provisioning we can do to make our
> > > product successful and scale it as we grow in the near future. Gary Orser
> > > will be replying to this thread soon with more findings from both our real
> > > app code and a little test app we are using and which he will share with
> > > you.
>
> > > > > We plan on having a pr
