[google-appengine] Re: Simultaneous requests
If your app is resistant to malicious use, ie the gathered data is presented to the user only from which it was taken... than I would suggest to shift your api calls from Server side to client side. Do all the api calls on client side and POST them to your server for processing using FBJS. Even limited processing can easily be done on client as well. I do similar kind of stuff for google-spreadsheet. Download the row data and process using Javascript rather than sending query parameter to be processed by google-doc servers. My 2 cents ... :) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Simultaneous Requests
>From the quota details page, the current default limit for simultaneous active dynamic requests is around 30 per app. http://code.google.com/appengine/docs/quotas.html#Request_Limits (However, it may be possible to raise these limits on a case by case basis http://code.google.com/appengine/kb/billing.html#cpu ) Your approach sounds reasonable, but I was curious about your mention of threading. Would that be client side threading? Ajax triggers would be another good solution which I've seen apps use. Thank you, Jeff On Mar 30, 2:38 pm, MajorProgamming wrote: > I am currently working on a way to mass email [in a short period of > time] using Google App Engine. I figured that as of now the best way > to do this would be to run many requests in parallel. I was wondering > if my app would accept many requests at once, and if so what would the > limit be [in the paid version]? > > Would this work? Is this the best way to do this? > > As for implementing the parallel requests I figured the simplest way > would be to use the python threading. I was wondering if using AJAX > would be any better, or if it would even work for parallel requests?? > > Thanks, --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
Well the only way I could think of would be client side threading (so that I could process multiple requests). Or simultaneous ajax (not so well supported). Either way it would be on the client side, because as far as I know it can't be done on GAE side... On Mar 31, 2:52 pm, Jeff S wrote: > From the quota details page, the current default limit for > simultaneous active dynamic requests is around 30 per app. > > http://code.google.com/appengine/docs/quotas.html#Request_Limits > > (However, it may be possible to raise these limits on a case by case > basishttp://code.google.com/appengine/kb/billing.html#cpu) > > Your approach sounds reasonable, but I was curious about your mention > of threading. Would that be client side threading? Ajax triggers would > be another good solution which I've seen apps use. > > Thank you, > > Jeff > > On Mar 30, 2:38 pm, MajorProgamming wrote: > > > I am currently working on a way to mass email [in a short period of > > time] using Google App Engine. I figured that as of now the best way > > to do this would be to run many requests in parallel. I was wondering > > if my app would accept many requests at once, and if so what would the > > limit be [in the paid version]? > > > Would this work? Is this the best way to do this? > > > As for implementing the parallel requests I figured the simplest way > > would be to use the python threading. I was wondering if using AJAX > > would be any better, or if it would even work for parallel requests?? > > > Thanks, --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
Wow>> Doesn't seem like GAE is happy with this arrangement:
I set up a simple system like so (in python)
[code]
import threading
import urllib2
class MyThread ( threading.Thread ):
def __init__ (self,j):
threading.Thread.__init__(self)
self.j=j
self.html=''
def run ( self ):
for x in xrange(500):
try:
response = urllib2.urlopen('http://the test
url')
html = response.read()
self.html=html
except urllib2.HTTPError, e:
print e.code
except urllib2.URLError, e:
print e.reason
# at end of thread life
print self.html
#init threads (10)
for x in xrange ( 10 ):
z=MyThread(x)
z.start()
[/code]
I ran this code on the client side. It ran fine but at approximately
100 requests it began to return Operation Timed Out Errors (code
10060). This is probably due to some Denial of Service protection by
GAE firewalls (?)
Is there any way for me to tell GAE that this is my app, and it's
legit, or is there any way to work around it?
[note: on the server side, nothing turned up on these errors which
leads me to believe it was a very low level protection]
On Mar 31, 2:52 pm, Jeff S wrote:
> From the quota details page, the current default limit for
> simultaneous active dynamic requests is around 30 per app.
>
> http://code.google.com/appengine/docs/quotas.html#Request_Limits
>
> (However, it may be possible to raise these limits on a case by case
> basishttp://code.google.com/appengine/kb/billing.html#cpu)
>
> Your approach sounds reasonable, but I was curious about your mention
> of threading. Would that be client side threading? Ajax triggers would
> be another good solution which I've seen apps use.
>
> Thank you,
>
> Jeff
>
> On Mar 30, 2:38 pm, MajorProgamming wrote:
>
> > I am currently working on a way to mass email [in a short period of
> > time] using Google App Engine. I figured that as of now the best way
> > to do this would be to run many requests in parallel. I was wondering
> > if my app would accept many requests at once, and if so what would the
> > limit be [in the paid version]?
>
> > Would this work? Is this the best way to do this?
>
> > As for implementing the parallel requests I figured the simplest way
> > would be to use the python threading. I was wondering if using AJAX
> > would be any better, or if it would even work for parallel requests??
>
> > Thanks,
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
At this point I am suspicious that the timeout error may coming from
the client side. Which version of Python are you using to run the
client? Also, did you set any timeout options elsewhere (looks like
no)?
It may also be the case that the app responds more slowly after the
initial burst due to an issue like datastore contention. If later
requests are more CPU intensive, responses may come back more slowly
than the early requests due to the behavior described in the Active
Requests section of the docs I linked to earlier:
"""
Applications that are heavily CPU-bound, on the other hand, may incur
some additional latency in long-running requests in order to make room
for other apps sharing the same servers.
"""
Could you tell us a bit more about what is being executed in these
requests?
Thank you,
Jeff
On Mar 31, 7:44 pm, MajorProgamming wrote:
> Wow>> Doesn't seem like GAE is happy with this arrangement:
>
> I set up a simple system like so (in python)
>
> [code]
> import threading
> import urllib2
>
> class MyThread ( threading.Thread ):
> def __init__ (self,j):
> threading.Thread.__init__(self)
> self.j=j
> self.html=''
>
> def run ( self ):
> for x in xrange(500):
> try:
> response = urllib2.urlopen('http://thetest
> url')
> html = response.read()
> self.html=html
> except urllib2.HTTPError, e:
> print e.code
> except urllib2.URLError, e:
> print e.reason
> # at end of thread life
> print self.html
>
> #init threads (10)
> for x in xrange ( 10 ):
> z=MyThread(x)
> z.start()
> [/code]
>
> I ran this code on the client side. It ran fine but at approximately
> 100 requests it began to return Operation Timed Out Errors (code
> 10060). This is probably due to some Denial of Service protection by
> GAE firewalls (?)
>
> Is there any way for me to tell GAE that this is my app, and it's
> legit, or is there any way to work around it?
>
> [note: on the server side, nothing turned up on these errors which
> leads me to believe it was a very low level protection]
>
> On Mar 31, 2:52 pm, Jeff S wrote:
>
> > From the quota details page, the current default limit for
> > simultaneous active dynamic requests is around 30 per app.
>
> >http://code.google.com/appengine/docs/quotas.html#Request_Limits
>
> > (However, it may be possible to raise these limits on a case by case
> > basishttp://code.google.com/appengine/kb/billing.html#cpu)
>
> > Your approach sounds reasonable, but I was curious about your mention
> > of threading. Would that be client side threading? Ajax triggers would
> > be another good solution which I've seen apps use.
>
> > Thank you,
>
> > Jeff
>
> > On Mar 30, 2:38 pm, MajorProgamming wrote:
>
> > > I am currently working on a way to mass email [in a short period of
> > > time] using Google App Engine. I figured that as of now the best way
> > > to do this would be to run many requests in parallel. I was wondering
> > > if my app would accept many requests at once, and if so what would the
> > > limit be [in the paid version]?
>
> > > Would this work? Is this the best way to do this?
>
> > > As for implementing the parallel requests I figured the simplest way
> > > would be to use the python threading. I was wondering if using AJAX
> > > would be any better, or if it would even work for parallel requests??
>
> > > Thanks,
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
I believe I'm using python 2.5. The script I posted before is the
entire script, so no timeouts.
The server side is simply outputting 'a':
self.response.out.write('a')
On Apr 1, 6:34 pm, Jeff S wrote:
> At this point I am suspicious that the timeout error may coming from
> the client side. Which version of Python are you using to run the
> client? Also, did you set any timeout options elsewhere (looks like
> no)?
>
> It may also be the case that the app responds more slowly after the
> initial burst due to an issue like datastore contention. If later
> requests are more CPU intensive, responses may come back more slowly
> than the early requests due to the behavior described in the Active
> Requests section of the docs I linked to earlier:
>
> """
> Applications that are heavily CPU-bound, on the other hand, may incur
> some additional latency in long-running requests in order to make room
> for other apps sharing the same servers.
> """
>
> Could you tell us a bit more about what is being executed in these
> requests?
>
> Thank you,
>
> Jeff
>
> On Mar 31, 7:44 pm, MajorProgamming wrote:
>
> > Wow>> Doesn't seem like GAE is happy with this arrangement:
>
> > I set up a simple system like so (in python)
>
> > [code]
> > import threading
> > import urllib2
>
> > class MyThread ( threading.Thread ):
> > def __init__ (self,j):
> > threading.Thread.__init__(self)
> > self.j=j
> > self.html=''
>
> > def run ( self ):
> > for x in xrange(500):
> > try:
> > response =
> > urllib2.urlopen('http://thetesturl')
> > html = response.read()
> > self.html=html
> > except urllib2.HTTPError, e:
> > print e.code
> > except urllib2.URLError, e:
> > print e.reason
> > # at end of thread life
> > print self.html
>
> > #init threads (10)
> > for x in xrange ( 10 ):
> > z=MyThread(x)
> > z.start()
> > [/code]
>
> > I ran this code on the client side. It ran fine but at approximately
> > 100 requests it began to return Operation Timed Out Errors (code
> > 10060). This is probably due to some Denial of Service protection by
> > GAE firewalls (?)
>
> > Is there any way for me to tell GAE that this is my app, and it's
> > legit, or is there any way to work around it?
>
> > [note: on the server side, nothing turned up on these errors which
> > leads me to believe it was a very low level protection]
>
> > On Mar 31, 2:52 pm, Jeff S wrote:
>
> > > From the quota details page, the current default limit for
> > >simultaneousactive dynamic requests is around 30 per app.
>
> > >http://code.google.com/appengine/docs/quotas.html#Request_Limits
>
> > > (However, it may be possible to raise these limits on a case by case
> > > basishttp://code.google.com/appengine/kb/billing.html#cpu)
>
> > > Your approach sounds reasonable, but I was curious about your mention
> > > of threading. Would that be client side threading? Ajax triggers would
> > > be another good solution which I've seen apps use.
>
> > > Thank you,
>
> > > Jeff
>
> > > On Mar 30, 2:38 pm, MajorProgamming wrote:
>
> > > > I am currently working on a way to mass email [in a short period of
> > > > time] using Google App Engine. I figured that as of now the best way
> > > > to do this would be to run many requests in parallel. I was wondering
> > > > if my app would accept many requests at once, and if so what would the
> > > > limit be [in the paid version]?
>
> > > > Would this work? Is this the best way to do this?
>
> > > > As for implementing the parallel requests I figured the simplest way
> > > > would be to use the python threading. I was wondering if using AJAX
> > > > would be any better, or if it would even work for parallel requests??
>
> > > > Thanks,
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
I believe I'm using python 2.5. The script I posted before is the
entire script, so no timeouts.
The server side is simply outputting 'a':
self.response.out.write('a')
On Apr 1, 6:34 pm, Jeff S wrote:
> At this point I am suspicious that the timeout error may coming from
> the client side. Which version of Python are you using to run the
> client? Also, did you set any timeout options elsewhere (looks like
> no)?
>
> It may also be the case that the app responds more slowly after the
> initial burst due to an issue like datastore contention. If later
> requests are more CPU intensive, responses may come back more slowly
> than the early requests due to the behavior described in the Active
> Requests section of the docs I linked to earlier:
>
> """
> Applications that are heavily CPU-bound, on the other hand, may incur
> some additional latency in long-running requests in order to make room
> for other apps sharing the same servers.
> """
>
> Could you tell us a bit more about what is being executed in these
> requests?
>
> Thank you,
>
> Jeff
>
> On Mar 31, 7:44 pm, MajorProgamming wrote:
>
> > Wow>> Doesn't seem like GAE is happy with this arrangement:
>
> > I set up a simple system like so (in python)
>
> > [code]
> > import threading
> > import urllib2
>
> > class MyThread ( threading.Thread ):
> > def __init__ (self,j):
> > threading.Thread.__init__(self)
> > self.j=j
> > self.html=''
>
> > def run ( self ):
> > for x in xrange(500):
> > try:
> > response =
> > urllib2.urlopen('http://thetesturl')
> > html = response.read()
> > self.html=html
> > except urllib2.HTTPError, e:
> > print e.code
> > except urllib2.URLError, e:
> > print e.reason
> > # at end of thread life
> > print self.html
>
> > #init threads (10)
> > for x in xrange ( 10 ):
> > z=MyThread(x)
> > z.start()
> > [/code]
>
> > I ran this code on the client side. It ran fine but at approximately
> > 100 requests it began to return Operation Timed Out Errors (code
> > 10060). This is probably due to some Denial of Service protection by
> > GAE firewalls (?)
>
> > Is there any way for me to tell GAE that this is my app, and it's
> > legit, or is there any way to work around it?
>
> > [note: on the server side, nothing turned up on these errors which
> > leads me to believe it was a very low level protection]
>
> > On Mar 31, 2:52 pm, Jeff S wrote:
>
> > > From the quota details page, the current default limit for
> > > simultaneous active dynamic requests is around 30 per app.
>
> > >http://code.google.com/appengine/docs/quotas.html#Request_Limits
>
> > > (However, it may be possible to raise these limits on a case by case
> > > basishttp://code.google.com/appengine/kb/billing.html#cpu)
>
> > > Your approach sounds reasonable, but I was curious about your mention
> > > of threading. Would that be client side threading? Ajax triggers would
> > > be another good solution which I've seen apps use.
>
> > > Thank you,
>
> > > Jeff
>
> > > On Mar 30, 2:38 pm, MajorProgamming wrote:
>
> > > > I am currently working on a way to mass email [in a short period of
> > > > time] using Google App Engine. I figured that as of now the best way
> > > > to do this would be to run many requests in parallel. I was wondering
> > > > if my app would accept many requests at once, and if so what would the
> > > > limit be [in the paid version]?
>
> > > > Would this work? Is this the best way to do this?
>
> > > > As for implementing the parallel requests I figured the simplest way
> > > > would be to use the python threading. I was wondering if using AJAX
> > > > would be any better, or if it would even work for parallel requests??
>
> > > > Thanks,
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
You might want to have a look to this App Engine video by Ken
Ashcraft:
http://www.youtube.com/watch?v=dP99fLhGwAU
App Engine scales, but not instantaneously, so if you want to do a
large number of requests in parallel, you need to increase the volume
gradually.
Julian
On Apr 3, 4:26 am, MajorProgamming wrote:
> I believe I'm using python 2.5. The script I posted before is the
> entire script, so no timeouts.
>
> The server side is simply outputting 'a':
> self.response.out.write('a')
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
Julian brings up a good point. I was also wondering what kind of QPS
you were seeing on your app during one of these tests. Since the
response from the server likely comes back pretty quickly, ten threads
could generate quite a few requests in a second.
Thank you,
Jeff
On Apr 2, 8:01 pm, Julian wrote:
> You might want to have a look to this App Engine video by Ken
> Ashcraft:http://www.youtube.com/watch?v=dP99fLhGwAU
>
> App Engine scales, but not instantaneously, so if you want to do a
> large number of requests in parallel, you need to increase the volume
> gradually.
>
> Julian
>
> On Apr 3, 4:26 am, MajorProgamming wrote:
>
> > I believe I'm using python 2.5. The script I posted before is the
> > entire script, so no timeouts.
>
> > The server side is simply outputting 'a':
> > self.response.out.write('a')
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Simultaneous Requests
Actually In terms of requests per second I was seeing was around 8
[didn't pay constant refresh attention]... Strangely though, the total
number of requests didn't accurately reflect the actual requests I
performed (in the dashboard -- not in the quota details page). This
may be due to some bug in the system. It doesn't really matter though
because the dashboard view seems to only be an overview, not for
accuracy...
Another thing: no matter how many threads I have there will always be
timeouts. I tried using only one thread (i.e. just running a python
for loop), there were a total of two timeouts. As the number of
threads increase, the number of timeouts increase... Additionally,
when a timeout occurs all the threads seem to fail simultaneously.
They continue to fail for a short period of time, and then they all
resume together.
Just a bit of information that I gathered from quite a bit of
debugging...
On Apr 3, 9:07 pm, Jeff S wrote:
> Julian brings up a good point. I was also wondering what kind of QPS
> you were seeing on your app during one of these tests. Since the
> response from the server likely comes back pretty quickly, ten threads
> could generate quite a few requests in a second.
>
> Thank you,
>
> Jeff
>
> On Apr 2, 8:01 pm, Julian wrote:
>
> > You might want to have a look to this App Engine video by Ken
> > Ashcraft:http://www.youtube.com/watch?v=dP99fLhGwAU
>
> > App Engine scales, but not instantaneously, so if you want to do a
> > large number of requests in parallel, you need to increase the volume
> > gradually.
>
> > Julian
>
> > On Apr 3, 4:26 am, MajorProgamming wrote:
>
> > > I believe I'm using python 2.5. The script I posted before is the
> > > entire script, so no timeouts.
>
> > > The server side is simply outputting 'a':
> > > self.response.out.write('a')
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
