[google-appengine] Re: Accessing raw post data

[Hans]

Hi Nick,

thanks for the reply. I already tried self.request.body, but it is
just a subset of self.request.
It also seems that the value is already processed somehow by the
serverengine because it only returns (according to my example):
comment=foobar&image_file=FieldStorage%28%27image_file%27%2C+
%27Blume.jpg%27%29

What I want is the body as send by the client, e.g.:
[...]
ThIs_Is_tHe_bouNdaRY_$
Content-Disposition: form-data; name="comment"

foobar
[...]

On 27 Jan., 06:57, "Nick Johnson (Google)" 
wrote:
> Hi Hans,
>
> You can access the request body via self.request.body.
>
> -Nick Johnson
>
> On Thu, Jan 27, 2011 at 1:28 AM, Hans wrote:
> > Hello!
>
> > I'd like to have access to the raw post data an app receives.
>
> > The output of self.request in a webapp.RequestHandler is not complete
> > or it is already modified by the server-engine (dev_appserver or the
> > real GAE).
>
> > If I send multipart/form-data to the app the body of the original
> > request is something like this:
> > [...]
> > ThIs_Is_tHe_bouNdaRY_$
> > Content-Disposition: form-data; name="comment"
>
> > foobar
>
> > ThIs_Is_tHe_bouNdaRY_$
> > Content-Disposition: form-data; name="image_file";
> > filename="flower.jpg"
> > Content-Type: image/jpeg
>
> > ...binary data...
> > [...]
>
> > The example body conent is processed correctly in the app engine, but
> > self.request returns this:
> > POST /upload
> > Content-Length: -1
> > Content-Type: multipart/form-data; boundary=--
> > ThIs_Is_tHe_bouNdaRY_$
> > Host: dev.example.local:8080
>
> > comment=foobar&image_file=FieldStorage%28%27image_file%27%2C+
> > %27Blume.jpg%27%29
>
> > The body is different here. How can I access the correct body or
> > complete raw post data?
>
> > Sincerly,
> > Hans
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> Nick Johnson, Developer Programs Engineer, App Engine
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> 368047

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Large scale database

[Wim den Ouden]

Hi Michael,
You have to do some test with your data to figure it out.
I have small entities but use transactions (no groups), to save i
havely use taskqueues and tune how often they run per minute to keep
inside the quota.
With http://code.google.com/intl/nl/appengine/docs/python/tools/appstats.html
you can see how it is going
gr
wim

2011/1/26 Michael McClain :
> So I can only insert 4 items per second in the database?
>
> I need to insert 1000 items simultaneously for each new document I'm
> inserting in the system...
> Is there a way to do this?
>
> Thank you,
> Michael
>
> On Jan 26, 5:36 pm, Wim den Ouden  wrote:
>> Hi Michael,
>> You need to spread the load as much as possible (parallel taskqueues),
>> keep processor load without extreme spikes.
>> Datastore writes max to about 4 per second (master/slave), 1 per
>> second high replication datastore.
>> When writing the app (triall and error) divide every load as much as
>> possible in smaller chunks, don't load to much libraries etc. (load
>> time)
>> gr
>> wim
>>
>> 2011/1/26 Michael McClain :
>>
>>
>>
>> > Hi everybody,
>>
>> >  I am new to Google Apps Engine.
>> >  I'm interested in using a cloud storage service for a large scale database
>> > system.
>>
>> >  I have to store billions of keys. Where each key has a 12 bytes value.
>> >  During query time I need to perfome one thousand simultaneous queries to
>> > retrieve 5000 thousand keys for each query. So I need to retrieve
>> > 5000*1000*12 bytes = 60 megabytes each time my application is run.
>>
>> >  I would like to understand if this is feasible using Google Datastore.
>> >  What would be the latency in performing one thousand simultaneous queries?
>>
>> >  Is there a Google representative that could assist me in understanding if
>> > my app is feasible for deployment in Google Datastore?
>> >  I'm interested in Google Datastore because it can scale usage growth and
>> > peak time traffic.
>>
>> > Thank you,
>> > Michael
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> gr
>> Wim den Ouden
>> Custom applications,https://e-comm.appspot.com/
>> Free open source E-commerce framework (web) 
>> apps,http://code.google.com/p/relat/
>> Gae developer tips,http://code.google.com/p/relat/wiki/gaetips
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>
>



-- 
gr
Wim den Ouden
Custom applications, https://e-comm.appspot.com/
Free open source E-commerce/E-bookkeeping/E-business framework (web)
apps, http://code.google.com/p/relat/
Gae developer tips, http://code.google.com/p/relat/wiki/gaetips

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] I can't access my app from my computer. Google "unusual traffic" error message.

[Albert]

I try to access my app from my computer a few minutes ago. This error
message appears. I'm assuming it's fine when accessed from somewhere
else.


Our systems have detected unusual traffic from your computer network.
Please try your request again later. Why did this happen?

This page appears when Google automatically detects requests coming
from your computer network which appear to be in violation of the
Terms of Service. The block will expire shortly after those requests
stop.

This traffic may have been sent by malicious software, a browser plug-
in, or a script that sends automated requests. If you share your
network connection, ask your administrator for help — a different
computer using the same IP address may be responsible. Learn more

Sometimes you may see this page if you are using advanced terms that
robots are known to use, or sending requests very quickly.



I was busy testing the API I'm developing (on my app), when this
suddenly came up.

I've been constantly developing and testing my app for a few months
already, and this only happened today. So I'm not sure what happened.
Should I be worried?

And another thing, does this mean that google implements some sort of
automatic DOS protection on AppEngine apps? I always assumed that we
had to manually set this up ourselves.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session

[Waleed Abdulla]

I just started getting the same exact issue about 15 minutes ago and it
brought down my app.

I make a lot of API calls from my main server to my app engine app, and now
the app engine is rejecting all these requests. Issue can be replicated from
the command line as follows:

*> wget api.networkedblogs.com/api/posts.getBlogPosts?bid=343457&limit=15*

Which returns:

*[1] 22560*
*[root@web1 temp]# --2011-01-27 03:20:53--
http://api.networkedblogs.com/api/posts.getBlogPosts?bid=343457*
*Resolving api.networkedblogs.com... 74.125.53.121*
*Connecting to api.networkedblogs.com|74.125.53.121|:80... connected.*
*HTTP request sent, awaiting response... 302 Found*
*Location:
http://sorry.google.com/sorry/?continue=http://api.networkedblogs.com/api/posts.getBlogPosts%3Fbid%3D343457[following]
*
*--2011-01-27 03:20:57--
http://sorry.google.com/sorry/?continue=http://api.networkedblogs.com/api/posts.getBlogPosts%3Fbid%3D343457
*
*Resolving sorry.google.com... 74.125.53.105, 74.125.53.106, 74.125.53.147,
...*
*Connecting to sorry.google.com|74.125.53.105|:80... connected.*
*HTTP request sent, awaiting response... 503 Service Unavailable*
*2011-01-27 03:20:57 ERROR 503: Service Unavailable.*

The same command above succeeds if I run it from my local machine, but it
fails if I run it from my Web server. The app engine seems to be falsely
considering my other server as a DoS attempt and blocking all requests from
it.

This is a blocker issue. Any help from the App Engine team is greatly
appreciated.

Regards,
Waleed





On Tue, Jan 25, 2011 at 11:06 AM, Denis Volokhovskiy <
altitudebre...@gmail.com> wrote:

> After some time, have retried the same session - without this issue.
>
> Looks like it is rare case.
>
>
>
>
>
>
> On Jan 25, 6:15 pm, Denis Volokhovskiy 
> wrote:
> > Hi Google Team,
> >
> > I have performed datastore maintenance from the remote api console for
> > python app (likeourselvesapp),
> > much records were processed in cycle by single requests (I think about
> > 100),
> > and suddenly have encountered automatic redirect (HTTP 302) to Turing
> > test page (with captcha entering)
> > right from the remote shell.
> > Then I exited and tried to deploy, and here traces for the same
> > denial:
> >
> > Application: likeourselvesapp; version: v132-2.
> > Server: appengine.google.com.
> > Scanning files on local disk.
> > Scanned 500 files.
> > Initiating update.
> > Error 302: --- begin server output ---
> > 
> > 302 Moved
> > 302 Moved
> > The document has moved
> > http://sorry.google.com/sorry/?continue=http://
> > appengine.google.com/api/appversion/create%3Fversion%3Dv132-2%26app_id
> > %3Dlikeourselvesapp">here.
> > 
> > --- end server output ---
> >
> > I tried to open specified URL with Turing test and entering captcha -
> > but it still abandoned my further requests.
> >
> > After several minutes problem gone,
> > but such delays is harmful for normal maintenance from remote console.
> >
> > We are billing-enabled app, and I'm registered developer for a long
> > time.
> > Why such problem happens, and is any solution to avoid it?
> >
> > Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] I can't access my app from my computer. Google "unusual traffic" error message.

[Wim den Ouden]

Had the same, next day it was gone
gr
wim

2011/1/27 Albert :
> I try to access my app from my computer a few minutes ago. This error
> message appears. I'm assuming it's fine when accessed from somewhere
> else.
>
> 
> Our systems have detected unusual traffic from your computer network.
> Please try your request again later. Why did this happen?
>
> This page appears when Google automatically detects requests coming
> from your computer network which appear to be in violation of the
> Terms of Service. The block will expire shortly after those requests
> stop.
>
> This traffic may have been sent by malicious software, a browser plug-
> in, or a script that sends automated requests. If you share your
> network connection, ask your administrator for help — a different
> computer using the same IP address may be responsible. Learn more
>
> Sometimes you may see this page if you are using advanced terms that
> robots are known to use, or sending requests very quickly.
>
> 
>
> I was busy testing the API I'm developing (on my app), when this
> suddenly came up.
>
> I've been constantly developing and testing my app for a few months
> already, and this only happened today. So I'm not sure what happened.
> Should I be worried?
>
> And another thing, does this mean that google implements some sort of
> automatic DOS protection on AppEngine apps? I always assumed that we
> had to manually set this up ourselves.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>
>



-- 
gr
Wim den Ouden
Custom applications, https://e-comm.appspot.com/
Free open source E-commerce/E-bookkeeping/E-business framework (web)
apps, http://code.google.com/p/relat/
Gae developer tips, http://code.google.com/p/relat/wiki/gaetips

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Large scale database

[Wim den Ouden]

Hi Michael,
With small entities and without transactions, groups, high replication
etc. you can save much more entities p/s
from Nick Johnson (google)
batch put, also delete

updated = []
for entity in MyModel.all().filter("color =",
old_favorite).fetch(100):
  entity.color = new_favorite
  updated.append(entity)
db.put(updated)

gr
wim

2011/1/26 Michael McClain :
> Hi everybody,
>
>  I am new to Google Apps Engine.
>  I'm interested in using a cloud storage service for a large scale database
> system.
>
>  I have to store billions of keys. Where each key has a 12 bytes value.
>  During query time I need to perfome one thousand simultaneous queries to
> retrieve 5000 thousand keys for each query. So I need to retrieve
> 5000*1000*12 bytes = 60 megabytes each time my application is run.
>
>  I would like to understand if this is feasible using Google Datastore.
>  What would be the latency in performing one thousand simultaneous queries?
>
>  Is there a Google representative that could assist me in understanding if
> my app is feasible for deployment in Google Datastore?
>  I'm interested in Google Datastore because it can scale usage growth and
> peak time traffic.
>
> Thank you,
> Michael
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
gr
Wim den Ouden
Custom applications, https://e-comm.appspot.com/
Free open source E-commerce/E-bookkeeping/E-business framework (web)
apps, http://code.google.com/p/relat/
Gae developer tips, http://code.google.com/p/relat/wiki/gaetips

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Albert]

The weirdness is that I can access all of Google's other services,
like gmail, analytics, groups, docs, etc from my computer just fine.
But not my site. How can I continue testing if this is the case? I
don't think I can wait until the next day.

Any ideas on how to resolve this, other than waiting it out?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Amir]

Same here, I was testing my app, and bamm! redirected to 
http://sorry.google.com/sorry/?continue=http://www.jaavuu.com/

It's never happened to me before.  It seems to only block me on my custom 
domains.  But now, other google app products are blocking me.  Not good.

Maybe it a safety feature to make sure you don't do any development after 
4am? I guess it's time to sleep.


http://rookery9.aviary.com.s3.amazonaws.com/6426500/6426678_1d37_625x625.jpg

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] I can't access my app from my computer. Google "unusual traffic" error message.

[Sahid Orentino Ferdjaoui]

Hi

Same problem for me.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[gg tv]

yeah seems screwed up use your appname.appspot.com url (does not help 
your users) and file a production ticket the link is above somewhere

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Amir]

Try testing on your *.appspot.com domain. It might not be blocked.
Also see if you can access other services mapped to custom domains like 
blogs and mail, i.e. blog.mydomain.com
I have a feeling that those are the ones being blocked.
  -Amir

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Amir]

I tested my app from a different ip address, and it's up and running ( 
http://www.jaavuu.com ), so it seems like they are just blocking specific ip 
addresses... that's good. (kind of)
   -Amir

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Koen Bok]

And same here. .appspot domain works, custom domain is borked. No Captcha's 
too. Incredibly annoying.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Unusual traffic?

[Livid]

I have a medium size community site http://www.v2ex.com/ running on
App Engine, it's running well but today all my users just keep
receiving this:

>> Our systems have detected unusual traffic from your computer network. Please 
>> try your request again later.

Because ghs.google.com is blocked in China and most of my users come
from China, so I have setup nginx reverse proxies on Rackspace, Linode
and other VPS providers, this method was working well but not today.

Is Google changing anything?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Livid]

Same here: http://www.v2ex.com/

On Jan 27, 5:21 pm, Albert  wrote:
> I try to access my app from my computer a few minutes ago. This error
> message appears. I'm assuming it's fine when accessed from somewhere
> else.
>
> 
> Our systems have detected unusual traffic from your computer network.
> Please try your request again later. Why did this happen?
>
> This page appears when Google automatically detects requests coming
> from your computer network which appear to be in violation of the
> Terms of Service. The block will expire shortly after those requests
> stop.
>
> This traffic may have been sent by malicious software, a browser plug-
> in, or a script that sends automated requests. If you share your
> network connection, ask your administrator for help — a different
> computer using the same IP address may be responsible. Learn more
>
> Sometimes you may see this page if you are using advanced terms that
> robots are known to use, or sending requests very quickly.
>
> 
>
> I was busy testing the API I'm developing (on my app), when this
> suddenly came up.
>
> I've been constantly developing and testing my app for a few months
> already, and this only happened today. So I'm not sure what happened.
> Should I be worried?
>
> And another thing, does this mean that google implements some sort of
> automatic DOS protection on AppEngine apps? I always assumed that we
> had to manually set this up ourselves.
>
> Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[gg tv]

make sure I tried from a different network and my app loaded once then was 
blocked

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Adam Sah]

happened to me too, now it's working again.  source IP: 67.169.72.13
example app affected: bbf-prod

adam

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Ip filtering and redirect to sorry.google.com

[Mikhail Kashkin]

Hello,

just typed one of our site url in address bar in browser and got redirect to
sorry service. Then checked all other our GAE-based sites and again got
unfriendly page with this content:

Our systems have detected unusual traffic from your computer network. Please
> try your request again later. Why did this happen?

IP address: 195.138.76.153

Time: 2011-01-27T09:48:56Z

URL: http://www.experika.com/


We are developers and most using Linux or OSX (viruses-free OSes). The same
time http://experika.appspot.com/ works.

Is this is temporary problem? And how we can prevent such blocking in
future? I'm especially worry about our visitors. Blacklist tab shows only
246 requests from our IPs. Do other people have this problem?

-- 
Mikhail Kashkin
http://www.xen.ru/
http://app-engine.tumblr.com/ - блог об App Engine

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[chrisfarms]

Also affecting me and several other legitimate IPs.

Access still ok via *.appspot.com domain, but blocked via custom
domain.


On Jan 27, 9:48 am, Livid  wrote:
> Same here:http://www.v2ex.com/
>
> On Jan 27, 5:21 pm, Albert  wrote:
>
>
>
>
>
>
>
> > I try to access my app from my computer a few minutes ago. This error
> > message appears. I'm assuming it's fine when accessed from somewhere
> > else.
>
> > 
> > Our systems have detected unusual traffic from your computer network.
> > Please try your request again later. Why did this happen?
>
> > This page appears when Google automatically detects requests coming
> > from your computer network which appear to be in violation of the
> > Terms of Service. The block will expire shortly after those requests
> > stop.
>
> > This traffic may have been sent by malicious software, a browser plug-
> > in, or a script that sends automated requests. If you share your
> > network connection, ask your administrator for help — a different
> > computer using the same IP address may be responsible. Learn more
>
> > Sometimes you may see this page if you are using advanced terms that
> > robots are known to use, or sending requests very quickly.
>
> > 
>
> > I was busy testing the API I'm developing (on my app), when this
> > suddenly came up.
>
> > I've been constantly developing and testing my app for a few months
> > already, and this only happened today. So I'm not sure what happened.
> > Should I be worried?
>
> > And another thing, does this mean that google implements some sort of
> > automatic DOS protection on AppEngine apps? I always assumed that we
> > had to manually set this up ourselves.
>
> > Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Waleed Abdulla]

Looks like GAE defense system is incorrectly blocking a lot of IPs. I posted
more details in a previous thread.




On Thu, Jan 27, 2011 at 2:05 AM, chrisfarms  wrote:

>
> Also affecting me and several other legitimate IPs.
>
> Access still ok via *.appspot.com domain, but blocked via custom
> domain.
>
>
> On Jan 27, 9:48 am, Livid  wrote:
> > Same here:http://www.v2ex.com/
> >
> > On Jan 27, 5:21 pm, Albert  wrote:
> >
> >
> >
> >
> >
> >
> >
> > > I try to access my app from my computer a few minutes ago. This error
> > > message appears. I'm assuming it's fine when accessed from somewhere
> > > else.
> >
> > > 
> > > Our systems have detected unusual traffic from your computer network.
> > > Please try your request again later. Why did this happen?
> >
> > > This page appears when Google automatically detects requests coming
> > > from your computer network which appear to be in violation of the
> > > Terms of Service. The block will expire shortly after those requests
> > > stop.
> >
> > > This traffic may have been sent by malicious software, a browser plug-
> > > in, or a script that sends automated requests. If you share your
> > > network connection, ask your administrator for help — a different
> > > computer using the same IP address may be responsible. Learn more
> >
> > > Sometimes you may see this page if you are using advanced terms that
> > > robots are known to use, or sending requests very quickly.
> >
> > > 
> >
> > > I was busy testing the API I'm developing (on my app), when this
> > > suddenly came up.
> >
> > > I've been constantly developing and testing my app for a few months
> > > already, and this only happened today. So I'm not sure what happened.
> > > Should I be worried?
> >
> > > And another thing, does this mean that google implements some sort of
> > > automatic DOS protection on AppEngine apps? I always assumed that we
> > > had to manually set this up ourselves.
> >
> > > Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session

[Waleed Abdulla]

Update: I put my php app in maintenance mode so it's not accessible and
doesn't make any calls to my GAE app, and left it for 10 minutes, then
enabled it again, and now GAE is accepting requests from the PHP server.
Odd, but worked for me.




On Thu, Jan 27, 2011 at 1:26 AM, Waleed Abdulla  wrote:

> I just started getting the same exact issue about 15 minutes ago and it
> brought down my app.
>
> I make a lot of API calls from my main server to my app engine app, and now
> the app engine is rejecting all these requests. Issue can be replicated from
> the command line as follows:
>
> *> wget api.networkedblogs.com/api/posts.getBlogPosts?bid=343457&limit=15*
>
> Which returns:
>
> *[1] 22560*
> *[root@web1 temp]# --2011-01-27 03:20:53--
> http://api.networkedblogs.com/api/posts.getBlogPosts?bid=343457*
> *Resolving api.networkedblogs.com... 74.125.53.121*
> *Connecting to api.networkedblogs.com|74.125.53.121|:80... connected.*
> *HTTP request sent, awaiting response... 302 Found*
> *Location:
> http://sorry.google.com/sorry/?continue=http://api.networkedblogs.com/api/posts.getBlogPosts%3Fbid%3D343457[following]
> *
> *--2011-01-27 03:20:57--
> http://sorry.google.com/sorry/?continue=http://api.networkedblogs.com/api/posts.getBlogPosts%3Fbid%3D343457
> *
> *Resolving sorry.google.com... 74.125.53.105, 74.125.53.106,
> 74.125.53.147, ...*
> *Connecting to sorry.google.com|74.125.53.105|:80... connected.*
> *HTTP request sent, awaiting response... 503 Service Unavailable*
> *2011-01-27 03:20:57 ERROR 503: Service Unavailable.*
>
> The same command above succeeds if I run it from my local machine, but it
> fails if I run it from my Web server. The app engine seems to be falsely
> considering my other server as a DoS attempt and blocking all requests from
> it.
>
> This is a blocker issue. Any help from the App Engine team is greatly
> appreciated.
>
> Regards,
> Waleed
>
>
>
>
>
> On Tue, Jan 25, 2011 at 11:06 AM, Denis Volokhovskiy <
> altitudebre...@gmail.com> wrote:
>
>> After some time, have retried the same session - without this issue.
>>
>> Looks like it is rare case.
>>
>>
>>
>>
>>
>>
>> On Jan 25, 6:15 pm, Denis Volokhovskiy 
>> wrote:
>> > Hi Google Team,
>> >
>> > I have performed datastore maintenance from the remote api console for
>> > python app (likeourselvesapp),
>> > much records were processed in cycle by single requests (I think about
>> > 100),
>> > and suddenly have encountered automatic redirect (HTTP 302) to Turing
>> > test page (with captcha entering)
>> > right from the remote shell.
>> > Then I exited and tried to deploy, and here traces for the same
>> > denial:
>> >
>> > Application: likeourselvesapp; version: v132-2.
>> > Server: appengine.google.com.
>> > Scanning files on local disk.
>> > Scanned 500 files.
>> > Initiating update.
>> > Error 302: --- begin server output ---
>> > 
>> > 302 Moved
>> > 302 Moved
>> > The document has moved
>> > http://sorry.google.com/sorry/?continue=http://
>> > appengine.google.com/api/appversion/create%3Fversion%3Dv132-2%26app_id
>> > %3Dlikeourselvesapp">here.
>> > 
>> > --- end server output ---
>> >
>> > I tried to open specified URL with Turing test and entering captcha -
>> > but it still abandoned my further requests.
>> >
>> > After several minutes problem gone,
>> > but such delays is harmful for normal maintenance from remote console.
>> >
>> > We are billing-enabled app, and I'm registered developer for a long
>> > time.
>> > Why such problem happens, and is any solution to avoid it?
>> >
>> > Thank you
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Albert]

I took a walk for a while. I just checked my app again. It's back now.
I don't know if it's because I didn't access it for a while, or if
Google fixed the problem. I hope everyone else's is doing alright now.

I hope this doesn't happen again. And it would probably be some sort
of relief if someone from Google could give us a hint on what
happened.

Thanks!



Albert

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Strom]

I also encountered this, as well as my clients.

On Jan 27, 12:31 pm, Albert  wrote:
> I took a walk for a while. I just checked my app again. It's back now.
> I don't know if it's because I didn't access it for a while, or if
> Google fixed the problem. I hope everyone else's is doing alright now.
>
> I hope this doesn't happen again. And it would probably be some sort
> of relief if someone from Google could give us a hint on what
> happened.
>
> Thanks!
>
> Albert

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Is it safe to extend WSGIApplication?

[Devraj Mukherjee]

Hi all,

Our application uses Python on the server side and Closure/JS on the
client side. Most of our server side code is to provide a RESTful
interface our models.

Since the application is quite complex and have a fair number of REST
services we were think of "writing our code better". The idea was to
write a wrapper around the WSGIApplication lifecycle to achieve the
following

- Assist with JSON serialization by moving common tasks to a parent class
- Validation framework for GET / POST / PUT / DELETE requests
- And of course execute the the right method in the Request handler

To me the best way of doing this is to intervene with the
WSGIApplication lifecycle.

Although whenever it comes to extending the core framework for
anything, I always become weary and can't help but think its a bad
idea.

Any suggestions or warnings? :)

Thanks for your time.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Large scale database

[djidjadji]

You can save yourself appending to the updated list.

updated = MyModel.all().filter("color =", old_favorite).fetch(100)
for entity in updated:
 entity.color = new_favorite
db.put(updated)


2011/1/27 Wim den Ouden :
> Hi Michael,
> With small entities and without transactions, groups, high replication
> etc. you can save much more entities p/s
> from Nick Johnson (google)
> batch put, also delete
>
> updated = []
> for entity in MyModel.all().filter("color =",
>    old_favorite).fetch(100):
>  entity.color = new_favorite
>  updated.append(entity)
> db.put(updated)
>
> gr
> wim
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Cron job Failure

[djidjadji]

And you have the advantage of a retry in case of a failure, like no
writes during maintenance, task are suspended until the maintenance is
complete. You can add new tasks but they are put on hold for a while.

2011/1/26 Kaan Soral :
> Add the job to a queue instead of running it directly in cron
> that way it can run for 10 minutes

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] taskqueue Deprecation Warning

[devGS]

I'm getting the following warning from dev_appserver.py:
"\Google\google_appengine\google\appengine\tools\dev_appserver.py:
1798: DeprecationWarning: google.appengine.api.labs.taskqueue is
deprecated, please use google.appengine.api.taskqueue  description)"

Does that mean that the taskqueue won't work? This is the most recent
SDK.
Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Sudden redirect to Turing anti-bot test during remote api session

[Tim Hoffman]

Hi

You might want to have a look at this thread 
h
*https://groups.google.com/d/topic/google-appengine/YrkJqnpCoL0/discussion *
I ran into this a long time ago and I found it occurred if I wasn't reusing 
the remote_api handle.

I had been recreating the remote_api handle on each transaction, which is 
inefficient and after 200 transactions I would be
redirected to a recaptcha service so I could prove I wasn't a bot ;-)

Rgds

T

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Domain for each user

[Geoffrey Spear]

On Tuesday, January 25, 2011 7:16:03 PM UTC-5, Jean-Francois wrote:
>
> Hi, 
>
> I have a website on App Engince with my own domain, for example 
> www.mywebsite.com 
>
> User can create a space which becomes myuser.mywebsite.com 
>
> Now, if my user wants to buy a custom domain name to have www.myuser.com 
> which forwards (but not visible) to myuser.mywebsite.com 
>
> How can I do this ? 
>
>
The user will need to register myuser.com as a Google Apps domain, add the 
application to the domain, and setup a CNAME from www.myuser.com to 
ghs.google.com.  

Then in your application you do the same sort of dispatching you need to get 
myuser.mywebsite.com to work, except using the whole hostname instead of 
just the subdomain to do the mapping. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Namespace vulnerabilities?

[Geoffrey Spear]

On Thursday, January 27, 2011 2:52:48 AM UTC-5, tobik wrote:
>
> Thank you! That's what I wanted to know. 
>
> So basically passing Keys in urls is not a good idea, but using pure 
> IDs and Model.get_by_id() should be secure enough. Or am I wrong? I 
> tried to get access to data across namespaces but it ended up with 
> request error. So it worked as expected. 
>
>
Yes; IDs are only unique within a namespace, entity group, and kind, so 
passing an ID from one namespace to an instance using a different namespace 
will either get you a different entity from the calling instance's namespace 
or nothing, never the entity in the original namespace.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Is it safe to extend WSGIApplication?

[Geoffrey Spear]

WSGIApplication is a part of the webapp framework; it's hardly "core".  You 
can ignore webapp altogether and use another framework or write your own; 
basing your new custom framework on webapp with a few modifications to one 
class is rather mild :)

Based on your description of what you want to do, though, have you 
considered using middlewear?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: taskqueue Deprecation Warning

[Geoffrey Spear]

No, the taskqueue will still work; that's just a warning that it's no longer 
in Labs and that you're using the old name to import it. I'm not aware of 
any deprecated features disappearing from App Engine yet, but it's still 
definitely good practice to change your imports to use the new location.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Domain for each user

[Sahid Orentino Ferdjaoui]

Hello Geoffrey,

In jaikuengine you have a good example to do this.

http://code.google.com/p/jaikuengine/source/browse/trunk/middleware/domain.py

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: taskqueue Deprecation Warning

[devGS]

Great! Thanks, the import is being used by web2py framework, hopefully
they will update it in the next release.


On Jan 27, 5:15 pm, Geoffrey Spear  wrote:
> No, the taskqueue will still work; that's just a warning that it's no longer
> in Labs and that you're using the old name to import it. I'm not aware of
> any deprecated features disappearing from App Engine yet, but it's still
> definitely good practice to change your imports to use the new location.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Globally DOS blocked?

[Andrin von Rechenberg]

okay someone else started a thread with this subject.

 I can't access my app from my computer. Google "unusual traffic" error
message.

Closing this one. reply on the other one

On Thu, Jan 27, 2011 at 10:42 AM, Andrin von Rechenberg
wrote:

> Okay, so it seems that after you requested about 20 pages on miumeet you
> hit the DOS limit!
>
> Like a max of 20 requests per user per hour or so! Oh oh
>
> -Andrin
>
> On Thu, Jan 27, 2011 at 10:30 AM, Andrin von Rechenberg <
> andri...@gmail.com> wrote:
>
>> Hey there
>>
>> We are currently serving about 800 qps peak. But now it suddenly dropped.
>>
>> When I go on my app at www.miumeet.com the only thing I see is:
>>
>> Our systems have detected unusual traffic from your computer network.
>> Please try your request again later. Why did this 
>> happen?
>>
>> This page appears when Google automatically detects requests coming from
>> your computer network which appear to be in violation of the Terms of
>> Service . The block will expire
>> shortly after those requests stop.
>>
>> This traffic may have been sent by malicious software, a browser plug-in,
>> or a script that sends automated requests. If you share your network
>> connection, ask your administrator for help — a different computer using the
>> same IP address may be responsible. Learn 
>> more
>>
>> Sometimes you may see this page if you are using advanced terms that
>> robots are known to use, or sending requests very quickly.
>>
>> IP address: 217.162.207.89
>> Time: 2011-01-27T09:26:38Z
>> URL: http://www.miumeet.com/
>>
>>
>>
>> Any idea? What is going on?
>>
>> -Andrin
>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Globally DOS blocked?

[Andrin von Rechenberg]

Hey there

We are currently serving about 800 qps peak. But now it suddenly dropped.

When I go on my app at www.miumeet.com the only thing I see is:

Our systems have detected unusual traffic from your computer network. Please
try your request again later. Why did this
happen?

This page appears when Google automatically detects requests coming from
your computer network which appear to be in violation of the Terms of
Service . The block will expire shortly
after those requests stop.

This traffic may have been sent by malicious software, a browser plug-in, or
a script that sends automated requests. If you share your network
connection, ask your administrator for help — a different computer using the
same IP address may be responsible. Learn
more

Sometimes you may see this page if you are using advanced terms that robots
are known to use, or sending requests very quickly.

IP address: 217.162.207.89
Time: 2011-01-27T09:26:38Z
URL: http://www.miumeet.com/



Any idea? What is going on?

-Andrin

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Andrin von Rechenberg]

same is happening for me. We used to serve 800 qps. now almost nothing.

An IP is blocked after requesting like 20 pages.

I told someone at Google. I hope they resolve it quickly.

Cheers,
-Andrin

On Thu, Jan 27, 2011 at 10:49 AM, gg tv  wrote:

> make sure I tried from a different network and my app loaded once then was
> blocked
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] need a clarity about maximum amount of time allowed for datastore operations

[Phani Kumar]

exception Timeout()
Raised when the datastore operation exceeds the maximum amount of
time allowed for datastore operations.

create_rpc(deadline=None, callback=None,
read_policy=STRONG_CONSISTENCY)

Creates a remote procedure call (RPC) configuration object for
setting the read policy and datastore call deadline for some API
calls.
The datastore call deadline specifies an amount of time the
application runtime environment will wait for the datastore to return
a result before aborting with an error. By default, the runtime
environment waits until the request handler deadline has elapsed.

from above two definitions i need a clarity about maximum amount of
time allowed for datastore operations by default.
i am confused about it

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Globally DOS blocked?

[Andrin von Rechenberg]

Okay, so it seems that after you requested about 20 pages on miumeet you hit
the DOS limit!

Like a max of 20 requests per user per hour or so! Oh oh

-Andrin

On Thu, Jan 27, 2011 at 10:30 AM, Andrin von Rechenberg
wrote:

> Hey there
>
> We are currently serving about 800 qps peak. But now it suddenly dropped.
>
> When I go on my app at www.miumeet.com the only thing I see is:
>
> Our systems have detected unusual traffic from your computer network.
> Please try your request again later. Why did this 
> happen?
>
> This page appears when Google automatically detects requests coming from
> your computer network which appear to be in violation of the Terms of
> Service . The block will expire
> shortly after those requests stop.
>
> This traffic may have been sent by malicious software, a browser plug-in,
> or a script that sends automated requests. If you share your network
> connection, ask your administrator for help — a different computer using the
> same IP address may be responsible. Learn 
> more
>
> Sometimes you may see this page if you are using advanced terms that robots
> are known to use, or sending requests very quickly.
>
> IP address: 217.162.207.89
> Time: 2011-01-27T09:26:38Z
> URL: http://www.miumeet.com/
>
>
>
> Any idea? What is going on?
>
> -Andrin
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Accessing raw post data

[Robert Kluin]

Hi Hans,
  The request is a webob Request object, check out the webob docs.
  http://pythonpaste.org/webob/reference.html#request-body



Robert





On Thu, Jan 27, 2011 at 03:09, Hans  wrote:
> Hi Nick,
>
> thanks for the reply. I already tried self.request.body, but it is
> just a subset of self.request.
> It also seems that the value is already processed somehow by the
> serverengine because it only returns (according to my example):
> comment=foobar&image_file=FieldStorage%28%27image_file%27%2C+
> %27Blume.jpg%27%29
>
> What I want is the body as send by the client, e.g.:
> [...]
> ThIs_Is_tHe_bouNdaRY_$
> Content-Disposition: form-data; name="comment"
>
> foobar
> [...]
>
> On 27 Jan., 06:57, "Nick Johnson (Google)" 
> wrote:
>> Hi Hans,
>>
>> You can access the request body via self.request.body.
>>
>> -Nick Johnson
>>
>> On Thu, Jan 27, 2011 at 1:28 AM, Hans 
>> wrote:
>> > Hello!
>>
>> > I'd like to have access to the raw post data an app receives.
>>
>> > The output of self.request in a webapp.RequestHandler is not complete
>> > or it is already modified by the server-engine (dev_appserver or the
>> > real GAE).
>>
>> > If I send multipart/form-data to the app the body of the original
>> > request is something like this:
>> > [...]
>> > ThIs_Is_tHe_bouNdaRY_$
>> > Content-Disposition: form-data; name="comment"
>>
>> > foobar
>>
>> > ThIs_Is_tHe_bouNdaRY_$
>> > Content-Disposition: form-data; name="image_file";
>> > filename="flower.jpg"
>> > Content-Type: image/jpeg
>>
>> > ...binary data...
>> > [...]
>>
>> > The example body conent is processed correctly in the app engine, but
>> > self.request returns this:
>> > POST /upload
>> > Content-Length: -1
>> > Content-Type: multipart/form-data; boundary=--
>> > ThIs_Is_tHe_bouNdaRY_$
>> > Host: dev.example.local:8080
>>
>> > comment=foobar&image_file=FieldStorage%28%27image_file%27%2C+
>> > %27Blume.jpg%27%29
>>
>> > The body is different here. How can I access the correct body or
>> > complete raw post data?
>>
>> > Sincerly,
>> > Hans
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com
>> > .
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> Nick Johnson, Developer Programs Engineer, App Engine
>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
>> 368047
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Stephen Johnson]

Has this affected anyone's task queues? About six hours ago mine started
going nuts. I'm wondering if it was lots of retries due to being blocked
which would be weird that google blocks itself.

On Thu, Jan 27, 2011 at 2:51 AM, Andrin von Rechenberg
wrote:

> same is happening for me. We used to serve 800 qps. now almost nothing.
>
> An IP is blocked after requesting like 20 pages.
>
> I told someone at Google. I hope they resolve it quickly.
>
> Cheers,
> -Andrin
>
>
> On Thu, Jan 27, 2011 at 10:49 AM, gg tv  wrote:
>
>> make sure I tried from a different network and my app loaded once then was
>> blocked
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Namespace vulnerabilities?

[tobik]

Great, thanks.

On 27 led, 16:06, Geoffrey Spear  wrote:
> On Thursday, January 27, 2011 2:52:48 AM UTC-5, tobik wrote:
>
> > Thank you! That's what I wanted to know.
>
> > So basically passing Keys in urls is not a good idea, but using pure
> > IDs and Model.get_by_id() should be secure enough. Or am I wrong? I
> > tried to get access to data across namespaces but it ended up with
> > request error. So it worked as expected.
>
> Yes; IDs are only unique within a namespace, entity group, and kind, so
> passing an ID from one namespace to an instance using a different namespace
> will either get you a different entity from the calling instance's namespace
> or nothing, never the entity in the original namespace.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Brandon Wirtz]

Double check that you aren't getting loop backs from your app, Also check
your log to see if you have a throttle code in the error logs.

 

I was having this issue when using Fetch and because of 301/302's was
creating infinite loops with in my app.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Stephen Johnson
Sent: Thursday, January 27, 2011 9:55 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: I can't access my app from my computer.
Google "unusual traffic" error message.

 

Has this affected anyone's task queues? About six hours ago mine started
going nuts. I'm wondering if it was lots of retries due to being blocked
which would be weird that google blocks itself. 

On Thu, Jan 27, 2011 at 2:51 AM, Andrin von Rechenberg 
wrote:

same is happening for me. We used to serve 800 qps. now almost nothing.

 

An IP is blocked after requesting like 20 pages.

 

I told someone at Google. I hope they resolve it quickly.

 

Cheers,

-Andrin

 

On Thu, Jan 27, 2011 at 10:49 AM, gg tv  wrote:

make sure I tried from a different network and my app loaded once then was
blocked

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
 .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
 .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Namespace vulnerabilities?

[Jeff Schwartz]

IMHO passing clear ids as text in urls is not a good idea because anything a
hacker can gleam about data is opening up a can of worms that they could use
to try to exploit your site and even corrupt your data. I'd look to encrypt
the id or at least apply a hash such as base64 at a a minimum to obfuscate
the actual id. While base64 wont deter a committed hacker it will deter many
malcontents who will move on to easier pickings. Best if you can, generate
an encrypted id on the server and pass that back via the url from the
client.

On Thu, Jan 27, 2011 at 1:06 PM, tobik  wrote:

> Great, thanks.
>
> On 27 led, 16:06, Geoffrey Spear  wrote:
> > On Thursday, January 27, 2011 2:52:48 AM UTC-5, tobik wrote:
> >
> > > Thank you! That's what I wanted to know.
> >
> > > So basically passing Keys in urls is not a good idea, but using pure
> > > IDs and Model.get_by_id() should be secure enough. Or am I wrong? I
> > > tried to get access to data across namespaces but it ended up with
> > > request error. So it worked as expected.
> >
> > Yes; IDs are only unique within a namespace, entity group, and kind, so
> > passing an ID from one namespace to an instance using a different
> namespace
> > will either get you a different entity from the calling instance's
> namespace
> > or nothing, never the entity in the original namespace.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>


-- 
*Jeff Schwartz*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: I can't access my app from my computer. Google "unusual traffic" error message.

[Stephen Johnson]

No fetches. No errors in the logs. It looks like the map reduce job kept
executing the same three data input splits over and over again even though
they all said there was no quota and they were aborting. Never had that
happen before. Was just curious since it seemed to happen when people began
complaining of the blocking issues. Thanks.

On Thu, Jan 27, 2011 at 11:28 AM, Brandon Wirtz  wrote:

> Double check that you aren’t getting loop backs from your app, Also check
> your log to see if you have a throttle code in the error logs.
>
>
>
> I was having this issue when using Fetch and because of 301/302’s was
> creating infinite loops with in my app.
>
>
>
>
>
> *From:* google-appengine@googlegroups.com [mailto:
> google-appengine@googlegroups.com] *On Behalf Of *Stephen Johnson
> *Sent:* Thursday, January 27, 2011 9:55 AM
> *To:* google-appengine@googlegroups.com
> *Subject:* Re: [google-appengine] Re: I can't access my app from my
> computer. Google "unusual traffic" error message.
>
>
>
> Has this affected anyone's task queues? About six hours ago mine started
> going nuts. I'm wondering if it was lots of retries due to being blocked
> which would be weird that google blocks itself.
>
> On Thu, Jan 27, 2011 at 2:51 AM, Andrin von Rechenberg 
> wrote:
>
> same is happening for me. We used to serve 800 qps. now almost nothing.
>
>
>
> An IP is blocked after requesting like 20 pages.
>
>
>
> I told someone at Google. I hope they resolve it quickly.
>
>
>
> Cheers,
>
> -Andrin
>
>
>
> On Thu, Jan 27, 2011 at 10:49 AM, gg tv  wrote:
>
> make sure I tried from a different network and my app loaded once then was
> blocked
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: RE: [google-appengine] Where can I find commercial successfully websites created using GAE?

[vlad]

@Brandon

Brandon, kudos. I enjoyed reading your site a lot. It has a nice mix of 
topics and data-to-noise ratio is pretty good. I was getting confused on how 
site makes money since I have not seen a single ad...then I remembered than 
I am running Firefox with AdBlock+ extension installed :)

Regards
Vlad

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: RE: [google-appengine] Where can I find commercial successfully websites created using GAE?

[Brandon Wirtz]

Ah, so you are one of those people who read websites, but don't want the
people who make the content to make any money.  I bet you listen to NPR and
play CD's when they go to pledge breaks..

 

www.xyhd.tv is not the primary source of income, but it does pretty well,
and CDN In A Box has done a lot to increase the profitability.
LockerGnome.com and it's network of sites will be upgraded to this shortly,
along with several other sites which have recently become clients.  We have
been really happy with the scalability and elasticity of GAE for our need.
And CDN In A Box has kicked the ass of every CDN solution we have been able
to find, and was a much better solution for us than Squid On AWS.  Serving
content from MemCache on GAE is BLAZING Fast.  And Serving things from
/static blows every CDN we've tested against out of the water.  

 

What is even better, is that the more people you get hitting a page the
faster it gets. so when a million people show up we are ready, but if only 1
person an hour shows up we aren't going broke.

  

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of vlad
Sent: Thursday, January 27, 2011 11:19 AM
To: google-appengine@googlegroups.com
Subject: Re: RE: [google-appengine] Where can I find commercial successfully
websites created using GAE?

 

@Brandon

Brandon, kudos. I enjoyed reading your site a lot. It has a nice mix of
topics and data-to-noise ratio is pretty good. I was getting confused on how
site makes money since I have not seen a single ad...then I remembered than
I am running Firefox with AdBlock+ extension installed :)

Regards
Vlad

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Namespace vulnerabilities?

[tobik]

That's funny because passing clear ids in url is quite common in
classic webs written in PHP. Practically all PHP frameworks/CMS
(including the big ones like Joomla or MODx) do that so I never
thought of it as a possible vulnerability. Or is there something more
I should worry about in this particular situation (Python,
AppEngine...)?

On 27 led, 19:34, Jeff Schwartz  wrote:
> IMHO passing clear ids as text in urls is not a good idea because anything a
> hacker can gleam about data is opening up a can of worms that they could use
> to try to exploit your site and even corrupt your data. I'd look to encrypt
> the id or at least apply a hash such as base64 at a a minimum to obfuscate
> the actual id. While base64 wont deter a committed hacker it will deter many
> malcontents who will move on to easier pickings. Best if you can, generate
> an encrypted id on the server and pass that back via the url from the
> client.
>
>
>
> On Thu, Jan 27, 2011 at 1:06 PM, tobik  wrote:
> > Great, thanks.
>
> > On 27 led, 16:06, Geoffrey Spear  wrote:
> > > On Thursday, January 27, 2011 2:52:48 AM UTC-5, tobik wrote:
>
> > > > Thank you! That's what I wanted to know.
>
> > > > So basically passing Keys in urls is not a good idea, but using pure
> > > > IDs and Model.get_by_id() should be secure enough. Or am I wrong? I
> > > > tried to get access to data across namespaces but it ended up with
> > > > request error. So it worked as expected.
>
> > > Yes; IDs are only unique within a namespace, entity group, and kind, so
> > > passing an ID from one namespace to an instance using a different
> > namespace
> > > will either get you a different entity from the calling instance's
> > namespace
> > > or nothing, never the entity in the original namespace.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> *Jeff Schwartz*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Namespace vulnerabilities?

[Ikai Lan (Google)]

I think in most cases it's okay. For instance, if you have a blog and
someone figures out that the IDs of your posts are sequential, they can
potentially ... view an older or newer post. Generally speaking, as long as
URLs are not your only mechanism for enforcing an ACL, you should be okay.

The reason you might want a non-sequential ID is to make it more difficult
to do automated content scraping.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
Blogger: http://googleappengine.blogspot.com
Reddit: http://www.reddit.com/r/appengine
Twitter: http://twitter.com/app_engine



On Thu, Jan 27, 2011 at 1:50 PM, tobik  wrote:

> That's funny because passing clear ids in url is quite common in
> classic webs written in PHP. Practically all PHP frameworks/CMS
> (including the big ones like Joomla or MODx) do that so I never
> thought of it as a possible vulnerability. Or is there something more
> I should worry about in this particular situation (Python,
> AppEngine...)?
>
> On 27 led, 19:34, Jeff Schwartz  wrote:
> > IMHO passing clear ids as text in urls is not a good idea because
> anything a
> > hacker can gleam about data is opening up a can of worms that they could
> use
> > to try to exploit your site and even corrupt your data. I'd look to
> encrypt
> > the id or at least apply a hash such as base64 at a a minimum to
> obfuscate
> > the actual id. While base64 wont deter a committed hacker it will deter
> many
> > malcontents who will move on to easier pickings. Best if you can,
> generate
> > an encrypted id on the server and pass that back via the url from the
> > client.
> >
> >
> >
> > On Thu, Jan 27, 2011 at 1:06 PM, tobik  wrote:
> > > Great, thanks.
> >
> > > On 27 led, 16:06, Geoffrey Spear  wrote:
> > > > On Thursday, January 27, 2011 2:52:48 AM UTC-5, tobik wrote:
> >
> > > > > Thank you! That's what I wanted to know.
> >
> > > > > So basically passing Keys in urls is not a good idea, but using
> pure
> > > > > IDs and Model.get_by_id() should be secure enough. Or am I wrong? I
> > > > > tried to get access to data across namespaces but it ended up with
> > > > > request error. So it worked as expected.
> >
> > > > Yes; IDs are only unique within a namespace, entity group, and kind,
> so
> > > > passing an ID from one namespace to an instance using a different
> > > namespace
> > > > will either get you a different entity from the calling instance's
> > > namespace
> > > > or nothing, never the entity in the original namespace.
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to google-appengine@googlegroups.com
> .
> > > To unsubscribe from this group, send email to
> > > google-appengine+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
> >
> > --
> > *Jeff Schwartz*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Experiencing Datastore Latency

[Michael Weinberg]

App Engine Team,

My app currently experiences unusually high latency on datastore
operations. The app engine status page shows that a latency issue is
under currently investigation - do you have an estimate on when it
would be resolved?

These log lines might give a clue, notice the 7 seconds delay between
the first and second lines:

01-27 01:59PM 12.692
org.datanucleus.ConnectionManagerImpl allocateConnection: Connection
added to the pool :
org.datanucleus.store.appengine.DatastoreConnectionFactoryImpl
$DatastoreManagedConnection@4f14da
D 01-27 01:59PM 19.335
org.datanucleus.ConnectionManagerImpl$1 managedConnectionPostClose:
Connection removed from the pool :
org.datanucleus.store.appengine.DatastoreConnectionFactoryImpl
$DatastoreManagedConnection@4f14da
D 01-27 01:59PM 19.336
org.datanucleus.ConnectionManagerImpl allocateConnection: Connection
added to the pool :
org.datanucleus.store.appengine.DatastoreConnectionFactoryImpl
$DatastoreManagedConnection@380e75
D 01-27 01:59PM 19.336
org.datanucleus.ConnectionManagerImpl allocateConnection: Connection
found in the pool :
org.datanucleus.store.appengine.DatastoreConnectionFactoryImpl
$DatastoreManagedConnection@380e75

Thanks,
Michael Weinberg

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is it safe to extend WSGIApplication?

[Devraj Mukherjee]

Hi Geoffrey,

On Fri, Jan 28, 2011 at 2:13 AM, Geoffrey Spear  wrote:
> WSGIApplication is a part of the webapp framework; it's hardly "core".  You
> can ignore webapp altogether and use another framework or write your own;
> basing your new custom framework on webapp with a few modifications to one
> class is rather mild :)

Thanks.

> Based on your description of what you want to do, though, have you
> considered using middlewear?

I have, but apart from,
http://code.google.com/p/appengine-rest-server/ I haven't seen any
others that suit the requirements.

Any that I might have missed and you can point out?

Cheers.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Red/Yellow warnings for offline requests

[Ernesto Karim Oltra]

Hi,

I've read crons and tasks have a deadline of 10 minutes, so I'm using
more than one second in the requests, but the logs page colurise them
as if they were a normal, online request.

952cpu_ms 672api_cpu_ms -> orange (it's a cron!! It has 10 minutes *
60 seconds * 1000 ms, isn't it?)
1185cpu_ms 672api_cpu_ms -> red

...and so on. I know I could simply ignore the colours, but they are
so strong and impressive! :)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Red/Yellow warnings for offline requests

[Ernesto Karim Oltra]

Hi,

I've read crons and tasks have a deadline of 10 minutes, so I'm using
more than one second in the requests, but the logs page colurise them
as if they were a normal, online request.

952cpu_ms 672api_cpu_ms -> orange (it's a cron!! It has 10 minutes *
60 seconds * 1000 ms, isn't it?)
1185cpu_ms 672api_cpu_ms -> red

...and so on. I know I could simply ignore the colours, but they are
so strong and impressive! :)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] QPS versus Req/sec

[smwatch]

Can some Google Expert please clarify this for us?

What is the difference between QPS versus Req/sec
When will google api limits start hitting, we read it is 400 Requests per 
Secs 

is the limit for 400 Average QPS or 400 Req/Sec as shown below for a given 
URL?

Below are some of our details
Number of Instances - Details Average QPS Average Latency Average Memory   30 
total 2.478 23.7 ms  10.1 MBytes 
Req/Sec current  Requests last 17 hrs  Avg CPU (API) last hr  % CPU last 17 
hrs
/myauoURL 
 78.6  4.56M /myurl2 1.3

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] public url access from browser app engine requesting google sign in

[pieceovcake]

when i try to access a url from my application 
http://myapp.com/public/OrderConfirm from browser app engine KEEPS asking me 
to log in.


*
  

Can anyone one help?


 
  OrderConfirm
  com.taxi.server.task.ProcessOrderConfirm
  
  
  OrderConfirm
  /public/OrderConfirm
  
 
 
  
  
  public urls
  /public/*
  
  
  *
  
  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] appcfg.py download_app exception when checking for latest SDK updates

[Wes]

Trying to download the app and I'm getting this exception:

$ appcfg.py download_app -A myappname -V 13 test
Server: appengine.google.com.
Traceback (most recent call last):
  File "/opt/local/var/google_appengine/appcfg.py", line 69, in 
run_file(__file__, globals())
  File "/opt/local/var/google_appengine/appcfg.py", line 65, in run_file
execfile(script_path, globals_)
  File "/opt/local/var/google_appengine/google/appengine/tools/appcfg.py", 
line 3100, in 
main(sys.argv)
  File "/opt/local/var/google_appengine/google/appengine/tools/appcfg.py", 
line 3091, in main
result = AppCfgApp(argv).Run()
  File "/opt/local/var/google_appengine/google/appengine/tools/appcfg.py", 
line 2013, in Run
self.action(self)
  File "/opt/local/var/google_appengine/google/appengine/tools/appcfg.py", 
line 2943, in __call__
return method()
  File "/opt/local/var/google_appengine/google/appengine/tools/appcfg.py", 
line 2334, in DownloadApp
updatecheck.CheckForUpdates()
  File "/opt/local/var/google_appengine/google/appengine/tools/appcfg.py", 
line 448, in CheckForUpdates
if self.config.api_version not in api_versions:
AttributeError: 'NoneType' object has no attribute 'api_version'


I was able to hack in a return statement in CheckForUpdates to get around 
this, but I'm curious as to why this is happening. It seems to work fine 
after skipping CheckForUpdates. 

Mac OSX, app engine SDK 1.4.0

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] QPS versus Req/sec

[Nick Johnson (Google)]

Hi smwatch,

On Fri, Jan 28, 2011 at 12:30 PM, smwatch  wrote:

> Can some Google Expert please clarify this for us?
>
> What is the difference between QPS versus Req/sec
>

These are the same thing - though QPS (Queries Per Second) is a slightly
broader term.


> When will google api limits start hitting, we read it is 400 Requests per
> Secs
>
> is the limit for 400 Average QPS or 400 Req/Sec as shown below for a given
> URL?
>

It used to be the case that if you were going to exceed 400QPS, you would
need to contact us to get an exemption so you could exceed that amount
without issues. That's no longer the case - your (billing enabled) app can
now handle as many QPS as you like without intervention from us.

-Nick Johnson


>
> Below are some of our details
> Number of Instances - Details Average QPS Average Latency Average Memory   30
> total 2.478 23.7 ms  10.1 MBytes
> Req/Sec current  Requests last 17 hrs  Avg CPU (API) last hr  % CPU last
> 17 hrs
> /myauoURL
>  78.6  4.56M /myurl2 1.3
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Is it safe to extend WSGIApplication?

[Nick Johnson (Google)]

Hi Devraj,

It's certainly fine to extend WSGIApplication if you wish, as long as you
don't depend on internal implementation details. Most of what you want to
do, though, seems like it would be better served by creating a subclass of
RequestHandler that all your handlers extend.

-Nick Johnson

On Thu, Jan 27, 2011 at 10:41 PM, Devraj Mukherjee  wrote:

> Hi all,
>
> Our application uses Python on the server side and Closure/JS on the
> client side. Most of our server side code is to provide a RESTful
> interface our models.
>
> Since the application is quite complex and have a fair number of REST
> services we were think of "writing our code better". The idea was to
> write a wrapper around the WSGIApplication lifecycle to achieve the
> following
>
> - Assist with JSON serialization by moving common tasks to a parent class
> - Validation framework for GET / POST / PUT / DELETE requests
> - And of course execute the the right method in the Request handler
>
> To me the best way of doing this is to intervene with the
> WSGIApplication lifecycle.
>
> Although whenever it comes to extending the core framework for
> anything, I always become weary and can't help but think its a bad
> idea.
>
> Any suggestions or warnings? :)
>
> Thanks for your time.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>


-- 
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Bunch of DeadlineExceededExceptions starting around 8pm PST

[coltsith]

I've had about 60 in the past 45 minutes, compared to 2 or 3 every
hour. Anyone else picking up chop like this?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Accessing raw post data

[Nick Johnson (Google)]

Hi Hans,

self.request is an instance of webob's Request object. If you're just
stringifying it (with str(), repr(), or implicitly), it will attempt to show
its contents in a way that seems sensible, but that's not necessarily how it
was sent over the wire. self.request.body will contain the raw body of the
request, exactly as it's sent over the wire. If that's URLEncoded instead of
formencoded, that's because your request was sent URLEncoded.

-Nick Johnson

On Thu, Jan 27, 2011 at 7:09 PM, Hans wrote:

> Hi Nick,
>
> thanks for the reply. I already tried self.request.body, but it is
> just a subset of self.request.
> It also seems that the value is already processed somehow by the
> serverengine because it only returns (according to my example):
> comment=foobar&image_file=FieldStorage%28%27image_file%27%2C+
> %27Blume.jpg%27%29
>
> What I want is the body as send by the client, e.g.:
> [...]
> ThIs_Is_tHe_bouNdaRY_$
> Content-Disposition: form-data; name="comment"
>
> foobar
> [...]
>
> On 27 Jan., 06:57, "Nick Johnson (Google)" 
> wrote:
> > Hi Hans,
> >
> > You can access the request body via self.request.body.
> >
> > -Nick Johnson
> >
> > On Thu, Jan 27, 2011 at 1:28 AM, Hans  >wrote:
> > > Hello!
> >
> > > I'd like to have access to the raw post data an app receives.
> >
> > > The output of self.request in a webapp.RequestHandler is not complete
> > > or it is already modified by the server-engine (dev_appserver or the
> > > real GAE).
> >
> > > If I send multipart/form-data to the app the body of the original
> > > request is something like this:
> > > [...]
> > > ThIs_Is_tHe_bouNdaRY_$
> > > Content-Disposition: form-data; name="comment"
> >
> > > foobar
> >
> > > ThIs_Is_tHe_bouNdaRY_$
> > > Content-Disposition: form-data; name="image_file";
> > > filename="flower.jpg"
> > > Content-Type: image/jpeg
> >
> > > ...binary data...
> > > [...]
> >
> > > The example body conent is processed correctly in the app engine, but
> > > self.request returns this:
> > > POST /upload
> > > Content-Length: -1
> > > Content-Type: multipart/form-data; boundary=--
> > > ThIs_Is_tHe_bouNdaRY_$
> > > Host: dev.example.local:8080
> >
> > > comment=foobar&image_file=FieldStorage%28%27image_file%27%2C+
> > > %27Blume.jpg%27%29
> >
> > > The body is different here. How can I access the correct body or
> > > complete raw post data?
> >
> > > Sincerly,
> > > Hans
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to google-appengine@googlegroups.com
> .
> > > To unsubscribe from this group, send email to
> > > google-appengine+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
> >
> > --
> > Nick Johnson, Developer Programs Engineer, App Engine
> > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
> Number:
> > 368047
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>


-- 
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.