from:"Dave"

(what I was getting it with the previous post is: if any of you need this
for your day jobs, you are welcome to try the same yourselves in the
meantime ;) )

On Wednesday, April 29, 2015 at 11:56:52 AM UTC-5, Dave Loomer wrote:

All: I won't have time to do this in the next several hours (my app engine
app is not my day job), but I think tonight I'll try changing the
api.twitter.com endpoint to point to a URL I own, and then examine the
URLs / query strings and headers received from my prod vs. dev code to see
if the encoding is any different.

I figure that could help point to whether it's a urlfetch issue. Only
other thing I can think of, if it's not urlfetch, is that the oauth
signatures are being generated differently in prod vs. dev, due to a
difference in some built-in python library on the prod server, and that
will be much more difficult to debug since the timestamp and nonce prevent
you from comparing dev vs prod requests character-by-character.

On Wednesday, April 29, 2015 at 9:36:17 AM UTC-5, Ryan Barrett wrote:

hi all! a number of us have started seeing twitter API errors in prod GAE
in the last week or so. write calls and some search calls are returning
HTTP 401 {errors:[{code:32,message:Could not authenticate you.}]}
for all users with no code changes on our end. the particularly odd part is
that the same calls with the same code, twitter app key/secret, and even
*the
exact same user access token key/secret* work fine in dev_appserver.

we're on python and mostly using tweepy. our current theory is that
twitter has blacklisted or graylisted app engine's external facing IPs. can
you think of any other ideas?

more details in this twitter dev forum post.
https://twittercommunity.com/t/post-to-statuses-update-json-started-hitting-error-32-could-not-authenticate-you-with-no-code-changes/36495

thanks in advance!

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/2ce6529e-dfea-41cd-9891-ce6c9b37ed64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: twitter API suddenly returning error 32 “Could not authenticate you.” only in prod GAE

All: I won't have time to do this in the next several hours (my app engine
app is not my day job), but I think tonight I'll try changing the
api.twitter.com endpoint to point to a URL I own, and then examine the URLs
/ query strings and headers received from my prod vs. dev code to see if
the encoding is any different.

I figure that could help point to whether it's a urlfetch issue. Only other
thing I can think of, if it's not urlfetch, is that the oauth signatures
are being generated differently in prod vs. dev, due to a difference in
some built-in python library on the prod server, and that will be much more
difficult to debug since the timestamp and nonce prevent you from comparing
dev vs prod requests character-by-character.

On Wednesday, April 29, 2015 at 9:36:17 AM UTC-5, Ryan Barrett wrote:

hi all! a number of us have started seeing twitter API errors in prod GAE
in the last week or so. write calls and some search calls are returning
HTTP 401 {errors:[{code:32,message:Could not authenticate you.}]}
for all users with no code changes on our end. the particularly odd part is
that the same calls with the same code, twitter app key/secret, and even *the
exact same user access token key/secret* work fine in dev_appserver.

we're on python and mostly using tweepy. our current theory is that
twitter has blacklisted or graylisted app engine's external facing IPs. can
you think of any other ideas?

more details in this twitter dev forum post.
https://twittercommunity.com/t/post-to-statuses-update-json-started-hitting-error-32-could-not-authenticate-you-with-no-code-changes/36495

thanks in advance!

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/3c59b082-5b10-45fc-91bf-c5ea8ef6efa2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: twitter API suddenly returning error 32 “Could not authenticate you.” only in prod GAE

I am reaching similar conclusions as David (I am @kidneybingos on that
Twitter dev thread). I have tried lots of punctuation characters, and most
succeed, but the following always cause errors:

' (apostrophe)
!
*
(
)

My conclusion is similarly that urlfetch is encoding something different as
of a week ago. I, like others in the thread, had been running the exact
same code with 100% success prior to then, and had been issuing these API
calls hundreds of times per day for a few years.

On Wednesday, April 29, 2015 at 10:44:01 AM UTC-5, David Fischer wrote:

I'm seeing the same problem on Appengine production (and not with
dev_appserver) and I'm using the same Twitter library. However, I was
seeing some tweets succeeding and some failures. After another user in the
Twitter Dev forum posted, I looked at the content of the tweets. The
failing tweets all contained the ! character. After removing that
character from tweets, I'm seeing a 100% success rate.

My working theory is that urlfetch is somehow changing how parameters are
escaped and that's causing the HMAC in Twitter's Oauth to be rejected.

On Wednesday, April 29, 2015 at 7:36:17 AM UTC-7, Ryan Barrett wrote:

we're on python and mostly using tweepy. our current theory is that
twitter has blacklisted or graylisted app engine's external facing IPs. can
you think of any other ideas?

more details in this twitter dev forum post.
https://twittercommunity.com/t/post-to-statuses-update-json-started-hitting-error-32-could-not-authenticate-you-with-no-code-changes/36495

thanks in advance!

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/d6d0df7b-a334-48cb-aa85-b0bf1506ffc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: twitter API suddenly returning error 32 “Could not authenticate you.” only in prod GAE

And just to reiterate - the characters seem to encode correctly (e.g.
apostrophe encodes to %27), and at any rate, when I log the URLs generated
by Tweepy the URLs generated on my dev instance are identical to prod. Yet
it works in dev, not in prod. My guess is something is happening in the
urlfetch code which we can't debug in prod.

On Wednesday, April 29, 2015 at 11:46:38 AM UTC-5, Dave Loomer wrote:

I am reaching similar conclusions as David (I am @kidneybingos on that
Twitter dev thread). I have tried lots of punctuation characters, and most
succeed, but the following always cause errors:

' (apostrophe)
!
*
(
)

My conclusion is similarly that urlfetch is encoding something different
as of a week ago. I, like others in the thread, had been running the exact
same code with 100% success prior to then, and had been issuing these API
calls hundreds of times per day for a few years.

On Wednesday, April 29, 2015 at 10:44:01 AM UTC-5, David Fischer wrote:

My working theory is that urlfetch is somehow changing how parameters are
escaped and that's causing the HMAC in Twitter's Oauth to be rejected.

On Wednesday, April 29, 2015 at 7:36:17 AM UTC-7, Ryan Barrett wrote:

hi all! a number of us have started seeing twitter API errors in prod
GAE in the last week or so. write calls and some search calls are returning
HTTP 401 {errors:[{code:32,message:Could not authenticate
you.}]} for all users with no code changes on our end. the
particularly odd part is that the same calls with the same code, twitter
app key/secret, and even *the exact same user access token key/secret*
work fine in dev_appserver.

we're on python and mostly using tweepy. our current theory is that
twitter has blacklisted or graylisted app engine's external facing IPs. can
you think of any other ideas?

more details in this twitter dev forum post.
https://twittercommunity.com/t/post-to-statuses-update-json-started-hitting-error-32-could-not-authenticate-you-with-no-code-changes/36495

thanks in advance!

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/8ab8ab7d-9741-4c79-8191-a75a45d90c77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: twitter API suddenly returning error 32 “Could not authenticate you.” only in prod GAE

Interesting. Perhaps the difference is that I tested only GET and not POST
(still, I get behavioral differences between dev and test with GET, even if the
request looks the same in my tests).

Or, maybe your method uncovers something that mine does not. Could you try a
GET (maybe using the search API) to see if there's a difference?

On Apr 29, 2015, at 11:35 PM, Waleed wal...@ninua.com wrote:

I simplified the test to one line:

urlfetch.fetch(http://173.203.209.168/home/wizards?status=Test%27;,
method=POST)

On production, I receive:
POST /home/wizards?status=Test' HTTP/1.1

And in local dev server, I receive:
POST /home/wizards?status=Test%27 HTTP/1.1

--
You received this message because you are subscribed to a topic in the Google
Groups Google App Engine group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/google-appengine/6ILDt39anbs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/69cc83ab-30ce-48d3-a27b-fff113f44623%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/40C54D41-40BE-42EC-9096-46674CBEE6DD%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: twitter API suddenly returning error 32 “Could not authenticate you.” only in prod GAE

Well bad news, the URL and headers received from my hosted app from a
Tweepy GET request were identical in prod vs. dev. We might need to go back
to attacking from the Twitter side to see if they can explain why they're
rejecting our API requests.

I realized when requesting from a dummy URL I didn't need to worry about
changing nonce and timestamp; I just hard-coded them (to realistic values)
to see if the oauth signatures were the same. And they were. So I have
failed to identify what these punctuation characters have to do with the
authentication error.

Caveat: the signatures are the same, but the 'Authorization' request header
is slightly different; its elements are ordered differently in prod vs dev
for whatever reason. But I'm not really concerned because the ordering is
the same regardless of whether I use the punctuation; i.e. dev and prod
Authorization headers differ in the same way even for requests that would
return a successful response from the Twitter API.

To see what I'm talking about, here is the Authorization request header in
production:
'OAuth realm=, oauth_nonce=12345678, oauth_timestamp=1430361562,
oauth_consumer_key=, oauth_signature_method=HMAC-SHA1,
oauth_version=1.0, oauth_token=y,
oauth_signature=AiqwN51QaZnwsBkikcVh9rENc84%3D'

And here it is in dev:
'OAuth realm=, oauth_token=y, oauth_nonce=12345678,
oauth_timestamp=1430361562,
oauth_signature=AiqwN51QaZnwsBkikcVh9rENc84%3D, oauth_version=1.0,
oauth_signature_method=HMAC-SHA1, oauth_consumer_key='

These are for the query dogs cat)s which I confirmed returns the
authentication error from prod (but not dev). In each case the URL
requested of my hosted app is:

/1.1/users/search.json?q=dogs%20%26%20cat)s

Note that the ) is not encoded. However, that remains the case when calling
from dev, so I don't see why this could be the cause of the problem.

On Wednesday, April 29, 2015 at 1:29:34 PM UTC-5, Ryan Barrett wrote:

thanks for the comprehensive debugging, Dave and David! it definitely
sounds like new header munging on app engine's part may be the problem.
Dave, your idea of sending the API calls to a different endpoint and
examining the headers sounds like a great lead. i'm looking forward to the
results!
as another data point, there may be more to it than just punctuation
characters. i see the error with pretty much any tweet content and
username. i reproduced it just now by attempting to tweet foo (just those
three characters) as @schnarfed. sigh.

On Wednesday, April 29, 2015 at 9:58:38 AM UTC-7, Dave Loomer wrote:

(what I was getting it with the previous post is: if any of you need this
for your day jobs, you are welcome to try the same yourselves in the
meantime ;) )

On Wednesday, April 29, 2015 at 11:56:52 AM UTC-5, Dave Loomer wrote:

All: I won't have time to do this in the next several hours (my app
engine app is not my day job), but I think tonight I'll try changing the
api.twitter.com endpoint to point to a URL I own, and then examine the
URLs / query strings and headers received from my prod vs. dev code to see
if the encoding is any different.

On Wednesday, April 29, 2015 at 9:36:17 AM UTC-5, Ryan Barrett wrote:

hi all! a number of us have started seeing twitter API errors in prod
GAE in the last week or so. write calls and some search calls are
returning
HTTP 401 {errors:[{code:32,message:Could not authenticate
you.}]} for all users with no code changes on our end. the
particularly odd part is that the same calls with the same code, twitter
app key/secret, and even *the exact same user access token key/secret*
work fine in dev_appserver.

we're on python and mostly using tweepy. our current theory is that
twitter has blacklisted or graylisted app engine's external facing IPs.
can
you think of any other ideas?

more details in this twitter dev forum post.
https://twittercommunity.com/t/post-to-statuses-update-json-started-hitting-error-32-could-not-authenticate-you-with-no-code-changes/36495

thanks in advance!

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google

[google-appengine] Re: Returning custom HTTP status codes from endpoints

2014-06-20 Thread Dave Goldberg

I return custom wrapper object with a field for my own internal status 
codes. The wrapper object is a generic class so the playload/data can be 
whatever I need to return.

Essentially unless something goes wrong my Endpoints always return 200. The 
caller has to inspect the object returned to get the true status code. It 
works fine but I loose a bit of type safety from my endpoints as it's not 
always clear what type of data the endpoint returns. 




On Tuesday, May 27, 2014 4:06:44 PM UTC-4, Matthew Flint wrote:

 Hello team,

 We all love REST-full-ness, so I'd like to return more REST-ful status 
 codes from my endpoints. (Example: 201 CREATED from a POST endpoint which 
 creates a resource).

 But I can't find any way to set the HTTP status code. Can anyone help 
 please?

 Some other (sad-path) status codes seem to be driven by throwing an 
 exception - eg. NotFoundException causes a 404. But I want to return a 
 different status code *and* XML/JSON in the response.

 Thanks all!
   Matthew


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Download logs from another project?

2014-06-20 Thread Dave Goldberg

I'm using:

LogService ls = LogServiceFactory.getLogService();

LogQuery lq = new LogQuery();

This works fine and I can download the logs from the GAE project I make the 
call from. I'm wondering if I can get logs from a different project? I 
don't see any way to use OAuth and provide credentials either in the 
LogService or the LogQuery to get logs from anywhere but the project which 
is executing. 

Is this possible? 

 

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] URL not found (404 error)

2013-09-03 Thread Dave Roscoe

I had been using app engine for a while with no problems.  In the last 
couple of days, my app stopped working.  The URL for my app is not being 
found.  If I redeploy the app, it works for a short time, and then stops 
working again.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Problems with deploying

2013-02-15 Thread Dave Loomer

Same here.

Why is there no system status dashboard for admin infrastructure?
The dashboard seems to ignore things like deployment errors, problems
loading task queue pages in the admin console, etc. that seem to plague a
lot of us regularly. I know there's an issue for it someplace, just too
lazy to look it up right now ...


On Fri, Feb 15, 2013 at 8:02 AM, Reiot reio...@gmail.com wrote:

 It's happening again.

 GAE dashboard extremely slow with 503 errors and couldn't deploy new
 version for an hour...


 On Saturday, February 2, 2013 8:27:33 PM UTC+9, Jesús Espejo wrote:

 Hello,

 Is anyone experimenting issues uploading applications to GAE? I cannot do
 it, it remains in the exponential back-off algorithm in 60 seconds
 forever... Also i have problems launching backends:

 Error: Server ErrorThe service you requested is not available yet.

 Please try again in 30 seconds.

 My app is for Python.

 Thanks in advance,

 Jesús.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] App's admin console broken since last night's maintenance

Issues occurring over the past several hours:

   - Non-console-related: one of my backends wouldn't start. Tasks would 
   just error out in the queue, and nothing (error or otherwise) showed in the 
   logs for that backend. Instances page would show no instance running. This 
   started last night but has since resolved itself.
   - Dashboard page shows Application statistics are currently 
   unavailable. with no graphs, quota info, etc. Started this morning, still 
   occurs.
   - Can't get into task queues. Persistent Server Error / A server error 
   has occurred. (This had been an intermittent issue over the past few weeks 
   but is now consistent). Started this morning, still occurs.

This is M/S app mn-live. I have other M/S apps which do not exhibit this 
behavior, although the mn-live one is by far the heaviest in terms of 
data, queues, and general resource usage.  All are paid apps (billing 
enabled).

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: App's admin console broken since last night's maintenance

Correction on item #1 regarding backend not starting. This is still 
occurring. The log shows that some tasks did run on the backend overnight, 
but right now I can't start any tasks on it. Still shows no instances 
running.


On Thursday, February 7, 2013 7:14:52 AM UTC-6, Dave Loomer wrote:

 Issues occurring over the past several hours:

- Non-console-related: one of my backends wouldn't start. Tasks would 
just error out in the queue, and nothing (error or otherwise) showed in 
 the 
logs for that backend. Instances page would show no instance running. This 
started last night but has since resolved itself.
- Dashboard page shows Application statistics are currently 
unavailable. with no graphs, quota info, etc. Started this morning, still 
occurs.
- Can't get into task queues. Persistent Server Error / A server 
error has occurred. (This had been an intermittent issue over the past 
 few 
weeks but is now consistent). Started this morning, still occurs.

 This is M/S app mn-live. I have other M/S apps which do not exhibit this 
 behavior, although the mn-live one is by far the heaviest in terms of 
 data, queues, and general resource usage.  All are paid apps (billing 
 enabled).


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] App's admin console broken since last night's maintenance

All seems well now, except that my frequent-tasks backend still won't
start.


On Thu, Feb 7, 2013 at 7:42 AM, Sameer Lodha sams...@gmail.com wrote:

 Same issue with our HR Python25 Apps. Not able to View the Task Queue
 screen for any of our Apps. And the Dashboard is intermittently unavailable.


 On Thu, Feb 7, 2013 at 7:08 PM, Akitoshi Abe akitoshi...@gmail.comwrote:

 Hello,

 Same issue occurring in some of my apps. I checked more than 20 High
 Replication Python27 apps and about half of them have problem on Dashboard
 page Application statistics are currently unavailable...

 Akitoshi Abe


 2013/2/7 Dave Loomer dloo...@gmail.com

 Issues occurring over the past several hours:

- Non-console-related: one of my backends wouldn't start. Tasks
would just error out in the queue, and nothing (error or otherwise) 
 showed
in the logs for that backend. Instances page would show no instance
running. This started last night but has since resolved itself.
- Dashboard page shows Application statistics are currently
unavailable. with no graphs, quota info, etc. Started this morning, 
 still
occurs.
- Can't get into task queues. Persistent Server Error / A server
error has occurred. (This had been an intermittent issue over the past 
 few
weeks but is now consistent). Started this morning, still occurs.

 This is M/S app mn-live. I have other M/S apps which do not exhibit
 this behavior, although the mn-live one is by far the heaviest in terms
 of data, queues, and general resource usage.  All are paid apps (billing
 enabled).

 --
 You received this message because you are subscribed to the Google
 Groups Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine?hl=en
 .
 For more options, visit https://groups.google.com/groups/opt_out.




  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Problems with deploying

I'm getting the 503 error on backend frequent-tasks for app mn-live. 

There's an existing issue for this; not sure if everyone who experiences 
this needs to open a new issue or if we should just keep the issue open 
since it hasn't been resolved.

http://code.google.com/p/googleappengine/issues/detail?id=6081


On Wednesday, February 6, 2013 3:51:34 PM UTC-6, Saurabh Gupta wrote:

 Thanks, Jason, that is good to know!


 On Saturday, February 2, 2013 2:30:39 PM UTC-5, Jason Brummett wrote:

 My account is premiere and has the same problem right now.  I do have the 
 option to contact the Enterprise Support center, but haven't done so yet, 
 but I can confirm that the 500 bux didn't avoid the issue.

 On Saturday, February 2, 2013 1:26:01 PM UTC-5, Saurabh Gupta wrote:

 Do you know if premiere accounts get better uptime/service? 

 I'm a paid app, but not a premiere app (not paying the $500/month) , so 
 just wondering if there is a difference in the underlying uptime/ 
 availability etc. in paid versus premiere. 

 Thanks



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Google cloud sql instance

2012-08-03 Thread Dave


Hello, 
 
  I am new to google cloud sql instance, and I try to develop an app engine 
application in JUno eclipse  using google cloud sql.
For my configuration under the properties, it is aking me the instance of 
the database which i know but i forget the database username and password.
 
I don't know where should I go to llok for my database user name and 
password.
 
Any help would be appreciated.
 
 
Thanks,
 
Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/m-FbYCw94k4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] download_data performance?

I want to download the full database for one of my apps. Entities
consume a relatively modest 2GB. (Indexes all told consume 19GB, but
my impression is that download_data won't download these?)

appcfg.py download_data has been running now for 12+ hours on a very
fast downstream connection.

Is there a faster way to download all data from a GAE app? Some
settings I can tweak that might help move things along?

Thanks,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: download_data performance?

I'm not sure how I missed that! Thanks.

-Dave

On Jun 5, 10:53 am, c h how...@umich.edu wrote:
 by default download_data is throttled.  read the docs for appcfg.py to see
 the settings and change them.







 On Tuesday, June 5, 2012 10:40:55 AM UTC-7, Dave Peck wrote:

  I want to download the full database for one of my apps. Entities
  consume a relatively modest 2GB. (Indexes all told consume 19GB, but
  my impression is that download_data won't download these?)

  appcfg.py download_data has been running now for 12+ hours on a very
  fast downstream connection.

  Is there a faster way to download all data from a GAE app? Some
  settings I can tweak that might help move things along?

  Thanks,
  Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: download_data performance?

Wait, hang on.

2GB of entities. Default bandwidth limit of 250,000 bytes/sec. So,
assuming we exactly saturate that limit, it should take ~2 hours 43
minutes. Now, we'll never saturate, so maybe estimate a 2x or even 3x
multiple of that time? That's still far less than the 14+ hours my
download job has been running.

-Dave

On Jun 5, 11:03 am, Dave Peck davep...@gmail.com wrote:
 I'm not sure how I missed that! Thanks.

 -Dave

 On Jun 5, 10:53 am, c h how...@umich.edu wrote:







  by default download_data is throttled.  read the docs for appcfg.py to see
  the settings and change them.

  On Tuesday, June 5, 2012 10:40:55 AM UTC-7, Dave Peck wrote:

   I want to download the full database for one of my apps. Entities
   consume a relatively modest 2GB. (Indexes all told consume 19GB, but
   my impression is that download_data won't download these?)

   appcfg.py download_data has been running now for 12+ hours on a very
   fast downstream connection.

   Is there a faster way to download all data from a GAE app? Some
   settings I can tweak that might help move things along?

   Thanks,
   Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: download_data performance?

~970MB right now.

On Jun 5, 11:33 am, Barry Hunter barrybhun...@gmail.com wrote:
 How big is the data you have already downloaded?

 You should be able to see the size of the file being written to.

 (or find the temporally file its been written to)

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] App Engine Framework Usage Statistics

2012-05-14 Thread Dave

I am trying to understand the most widely used java framework for
developing server side web applications on Appengine. My goal is to
pick the most heavily used one for an upcoming class.  I have seen no
usage statistics. Are there any available? Are java frameworks more or
less popular than Python frameworks?

The JEE 6 spec  has Servlets 3.0, JSP 2.2 and JSF 2.0. Appengine is
running none of these specifications.
Appengine is currently running an old version of servlets 2.5.  Is
there any announced plan for Appengine to support servlets 3.0.
Appengine does not directly support the full JSF 2.0 release. Is there
any plan to?

Since the latest versions of servlets, jsp and jsf are not currently
supported by Appengine then is there one or two leading frameworks
that java developers are using on Appengine. This information would be
helpful for new users to best focus their initial learning to proven
systems used by experienced developers.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

2012-03-07 Thread Dave Peck

  Should be possible:http://code.google.com/appengine/docs/java/
config/appconfig.html#Cust...

Ah, nice feature. Thanks; curious when this was added.

Cheers,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

2012-03-07 Thread Dave Peck

Hi Chris,

Thanks!

 Sorry that you had issues with Checkout. We continue to work with them on
 improvements that they should make to the service (and I've forward this
 thread on to them).

Good to hear. If I can provide further details, just let me know. In
the final analysis, it seemed to us that the billing update issued
from my iPhone worked... but it didn't. More accurate feedback would
have been helpful.

 Did you figure out why the DS reads increased? Is this still an issue?

The operating theory at the moment is that a change to how our back-
end services report back to App Engine triggered a latent (and really
nasty) performance bug. We're still tracking down the details now;
Alfred has been immensely helpful offline!

 Setting your budget higher is probably not a bad idea, but totally up to
 you. I think the pros and cons have been outlined (in one way or another)
 here :-)

Indeed they were. ;-)

Cheers,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Huge number of datastore reads?

My production application went over its daily budget today. Let's just
say that my daily budget is roughly 3x what I've ever actually needed
on my busiest day. Today was not my busiest day by a long shot.

It appears that I serviced a small number of requests today. However,
the dashboard claims that I performed roughly 10,000x the number of
datastore read ops as requests, at a cost of
$LOTS_OF_MONEY_FOR_ONE_DAY

This seems like nonsense. My code hasn't changed in a while. I've
never reached a quota quite like this. And there's no way my average
request requires 10,000 read ops. Just no.

Google team -- is there someone I can speak with? I'd like to
understand in detail what happened and how to prevent it going
forward.

-Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

The ID is get-cloak-live.

No remote queries; everything would have to have been generated by
handling requests (some of which were our cron jobs.)

Thanks,
Dave

On Mar 6, 4:53 pm, Alfred Fuller arfuller+appeng...@google.com
wrote:
 What is your app id?

 Did you perform a lot of queries using remote api?







 On Tue, Mar 6, 2012 at 4:40 PM, Dave Peck davep...@gmail.com wrote:
  My production application went over its daily budget today. Let's just
  say that my daily budget is roughly 3x what I've ever actually needed
  on my busiest day. Today was not my busiest day by a long shot.

  It appears that I serviced a small number of requests today. However,
  the dashboard claims that I performed roughly 10,000x the number of
  datastore read ops as requests, at a cost of
  $LOTS_OF_MONEY_FOR_ONE_DAY

  This seems like nonsense. My code hasn't changed in a while. I've
  never reached a quota quite like this. And there's no way my average
  request requires 10,000 read ops. Just no.

  Google team -- is there someone I can speak with? I'd like to
  understand in detail what happened and how to prevent it going
  forward.

  -Dave

  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

To be clear: I lost real customers and real money today.

The problem was compounded by apparent issues with Google Checkout on
the iPhone. I got a downtime notification and quickly determined the
root cause. I was on the road, so I pulled over and immediately logged
in with my iPhone to update the billing information. I more-than-
doubled our daily budget and submitted it. It looked like everything
worked. (That is to say: the form let me type in a new budget and
submit it.) Billing was frozen for 30 minutes. I kept refreshing,
only to discover that for whatever reason _it hadn't worked_. 30
minutes later, I tried again. This doomed us to another 30 minutes of
downtime. It wasn't until I raced home to my laptop that I was able to
successfully update the billing. This needs to be fixed.

The worst of it: we looked like rank amateurs in several ways, but
perhaps none greater than our 500 page. App Engine decided to display
a generic Google-logo'd Over Quota 500 page instead of our custom
500 error page. What customer in their right mind, after seeing such
an embarrassment, would think that we're serious about our business?

-Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

Although I was hoping for useful, non-judgmental replies too.

-Dave

On Mar 6, 10:08 pm, Brandon Wirtz drak...@digerat.com wrote:
  The worst of it: we looked like rank amateurs in several ways, but perhaps

 I'd say no offense but I'd be lying.

 More than doubled

 My Daily budget sits at 50x my biggest recorded day on almost every app I
 have.  If your model is to make money, and you are pretty sure you don't
 have a bug that is going to cost you millions of dollars. There is no reason
 to not set your budget sky high. If you have only 30 minutes to get your app
 back with in quota, you are rank amateurs. Your Scale is only limited by
 your budget. If you aren't going to set you budget to as large as you
 imagine you would want to scale you are doing it wrong.

 If you are going to run with the training wheels on, don't be surprised when
 you hit a pot hole and your back tire just spins with you going nowhere.

 -Brandon

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

2012-02-05 Thread Dave Loomer

In my case, since I was getting the 20-second delay almost 100% of the
time, setting countdown=1 was the answer.  If you only see it happen
every 20 or more request then of course it won't help.

In my case I also run all tasks on the backend. They're slightly more
expensive per hour than frontends (due merely to the lower number of
free hours) but in my case I more than make up for it with the fact
that I have full control on the number of requests that will spin up,
and I need to be able to control that number separately for tasks vs.
users hitting my site.

On Feb 5, 11:24 am, Carter jcmas...@gmail.com wrote:
 We regularly but erratically see 10-20 minute delays in running push
 queue tasks.
 The tasks sit in the queue with ETA as high as 20 minutes *ago*
 without any errors or retries.

 (the problem seems unrelated to queue settings since our Maximum Rate,
 Enorced Rate and Maximum Concurrent all far exceed the queue's
 throughput at the time of the delays)

 Any tips or clues on how to prevent this while still using push queues
 without backends?

 On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote:







  Hey Dave,
    Hopefully Nick will be able to offer some insight into the cause of
  your issues.  I'd guess it is something related to having very few
  tasks (one) in thequeue, and it not getting scheduled rapidly.

    In your case, you could use pull queues to immediately fetch the
  nexttaskwhen finished with atask.  Or even to fetch multiple tasks
  and do the work in parallel.  Basically you'd have a backend that ran
  a loop (possibly initiated via a pushtask) that would lease atask,
  or tasks, from the pullqueue, do the work, delete those tasks, then
  repeat from the lease stage.  The cool thing is that if you're, for
  example, using URL Fetch to pull data  this might let you do it in
  parallel without increasing your costs much (if any).

  Robert

  On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote:
   Here are logs from three consecutivetaskexecutions over the past weekend,
   with only identifying information removed. You'll see that eachtask
   completes in a few milliseconds, but are 20 seconds apart (remember: I've
   already checked myqueueconfigurations, nothing else is running on this
   backend, and I later solved the problem by setting countdown=1 when adding
   thetask).  I don't see any pending latency mentioned.

   0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
   api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
   task_name=15804554889304913211 instance=0
   0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
   api_cpu_ms=0
   cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461
   instance=0
   0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
   api_cpu_ms=0
   cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
   instance=0

   The 20 seconds seems to happen regardless of length oftask. Even though my
   tasks mostly complete in a couple minutes, I do have cases where they take
   several minutes, and I don't see a difference. Of course, when atasktakes
   5-10 minutes to complete, I'm going to notice and care about a 20-second
  delaymuch less than when I'm trying to spin through a few tasks in a minute
   (which is a real-world need for me as well).

   When reading up on pull queues a while back, I was a little confused about
   where I would use them with my own backends. I definitely could see an
   application for offloading work to an AWS Linux instance. But in either
   case, could you explain why it might help?

   I saw you mention in a separate thread how M/S can perform differently 
   from
   HRD even in cases where one wouldn't expect to see a difference. When I 
   get
   around to it I'm going to create a tiny HRD app and run the same tests
   through that.

   I also wonder if M/S could be responsible for frequent latencies in my 
   admin
   console. Those have gotten more frequent and annoying the past couple of
   months ...

   --
   You received this message because you are subscribed to the Google Groups
   Google App Engine group.
   To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.

   To post to this group, send email to google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

2012-02-05 Thread Dave Loomer

 that I have full control on the number of requests that will spin up,
err, number of instances that will spin up, rather ...

On Feb 5, 11:30 am, Dave Loomer dloo...@gmail.com wrote:
 In my case, since I was getting the 20-second delay almost 100% of the
 time, setting countdown=1 was the answer.  If you only see it happen
 every 20 or more request then of course it won't help.

 In my case I also run all tasks on the backend. They're slightly more
 expensive per hour than frontends (due merely to the lower number of
 free hours) but in my case I more than make up for it with the fact
 that I have full control on the number of requests that will spin up,
 and I need to be able to control that number separately for tasks vs.
 users hitting my site.

 On Feb 5, 11:24 am, Carter jcmas...@gmail.com wrote:







  We regularly but erratically see 10-20 minute delays in running push
  queue tasks.
  The tasks sit in the queue with ETA as high as 20 minutes *ago*
  without any errors or retries.

  (the problem seems unrelated to queue settings since our Maximum Rate,
  Enorced Rate and Maximum Concurrent all far exceed the queue's
  throughput at the time of the delays)

  Any tips or clues on how to prevent this while still using push queues
  without backends?

  On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote:

   Hey Dave,
     Hopefully Nick will be able to offer some insight into the cause of
   your issues.  I'd guess it is something related to having very few
   tasks (one) in thequeue, and it not getting scheduled rapidly.

     In your case, you could use pull queues to immediately fetch the
   nexttaskwhen finished with atask.  Or even to fetch multiple tasks
   and do the work in parallel.  Basically you'd have a backend that ran
   a loop (possibly initiated via a pushtask) that would lease atask,
   or tasks, from the pullqueue, do the work, delete those tasks, then
   repeat from the lease stage.  The cool thing is that if you're, for
   example, using URL Fetch to pull data  this might let you do it in
   parallel without increasing your costs much (if any).

   Robert

   On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote:
Here are logs from three consecutivetaskexecutions over the past 
weekend,
with only identifying information removed. You'll see that eachtask
completes in a few milliseconds, but are 20 seconds apart (remember: 
I've
already checked myqueueconfigurations, nothing else is running on this
backend, and I later solved the problem by setting countdown=1 when 
adding
thetask).  I don't see any pending latency mentioned.

0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
task_name=15804554889304913211 instance=0
0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
api_cpu_ms=0
cpm_usd=0.60 queue_name=overnight-tasks 
task_name=15804554889304912461
instance=0
0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
api_cpu_ms=0
cpm_usd=0.60 queue_name=overnight-tasks 
task_name=4499136807998063691
instance=0

The 20 seconds seems to happen regardless of length oftask. Even though 
my
tasks mostly complete in a couple minutes, I do have cases where they 
take
several minutes, and I don't see a difference. Of course, when 
atasktakes
5-10 minutes to complete, I'm going to notice and care about a 20-second
   delaymuch less than when I'm trying to spin through a few tasks in a 
   minute
(which is a real-world need for me as well).

When reading up on pull queues a while back, I was a little confused 
about
where I would use them with my own backends. I definitely could see an
application for offloading work to an AWS Linux instance. But in either
case, could you explain why it might help?

I saw you mention in a separate thread how M/S can perform differently 
from
HRD even in cases where one wouldn't expect to see a difference. When I 
get
around to it I'm going to create a tiny HRD app and run the same tests
through that.

I also wonder if M/S could be responsible for frequent latencies in my 
admin
console. Those have gotten more frequent and annoying the past couple of
months ...

--
You received this message because you are subscribed to the Google 
Groups
Google App Engine group.
To view this discussion on the web visit
   https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
   http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

2012-02-05 Thread Dave Loomer

As the OP you may be interested in my app ID as well: mn-live.  I
provided some logs a few posts back and some exhaustive details at the
beginning.

However, you won't see this issue popping up anymore on my app since I
solved it by setting countdown=1 a week ago. Since then, tasks start
very reliably after a 1.5 second delay.  If I remove the countdown
parameter, then it returns to 20 seconds (+/- .01) pretty reliably.

On Feb 5, 5:59 pm, Nicholas Verne nve...@google.com wrote:
 We would have no need to shoot anyone.

 However, the explanations quickly become obsolete. They enter the
 folklore in the form that was current at the time and become
 entrenched as incorrect information when the implementations have
 changed.

 Task Queues use best effort scheduling. They're not real time all the
 time, although when our best efforts are running smoothly they can
 appear real time. For scheduling, the task eta marks the earliest time
 at which the task can run. We can't guarantee that a task WILL run at
 that time.

 Steve, we're interested to know about the 10-20 minute delays you've
 seen. Can you tell us the app id, queue, and whether the tasks were
 added transactionally? An example from your logs would be very
 helpful.

 Nick Verne







 On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote:
  Carter wrote: We regularly but erratically see 10-20 minute delays in
  running push queue tasks.

  Been a burr under the saddle forever. What I really don't understand
  -- assuming GAE engineers never see the benefit of providing at least
  one priority/reliability queue -- is why the heck there is never any
  explanation about how tasks get scheduled, and why these weird delays
  happen. It is either: 1) If we told you we would have to shoot you, or
  2) We can't see the benefit of you understanding this.

  -stevep

  On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote:
  We regularly but erratically see 10-20 minute delays in running push
  queue tasks.
  The tasks sit in the queue with ETA as high as 20 minutes *ago*
  without any errors or retries.

  (the problem seems unrelated to queue settings since our Maximum Rate,
  Enorced Rate and Maximum Concurrent all far exceed the queue's
  throughput at the time of the delays)

  Any tips or clues on how to prevent this while still using push queues
  without backends?

  On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote:

   Hey Dave,
     Hopefully Nick will be able to offer some insight into the cause of
   your issues.  I'd guess it is something related to having very few
   tasks (one) in thequeue, and it not getting scheduled rapidly.

     In your case, you could use pull queues to immediately fetch the
   nexttaskwhen finished with atask.  Or even to fetch multiple tasks
   and do the work in parallel.  Basically you'd have a backend that ran
   a loop (possibly initiated via a pushtask) that would lease atask,
   or tasks, from the pullqueue, do the work, delete those tasks, then
   repeat from the lease stage.  The cool thing is that if you're, for
   example, using URL Fetch to pull data  this might let you do it in
   parallel without increasing your costs much (if any).

   Robert

   On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote:
Here are logs from three consecutivetaskexecutions over the past 
weekend,
with only identifying information removed. You'll see that eachtask
completes in a few milliseconds, but are 20 seconds apart (remember: 
I've
already checked myqueueconfigurations, nothing else is running on this
backend, and I later solved the problem by setting countdown=1 when 
adding
thetask).  I don't see any pending latency mentioned.

0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
task_name=15804554889304913211 instance=0
0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
api_cpu_ms=0
cpm_usd=0.60 queue_name=overnight-tasks 
task_name=15804554889304912461
instance=0
0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
api_cpu_ms=0
cpm_usd=0.60 queue_name=overnight-tasks 
task_name=4499136807998063691
instance=0

The 20 seconds seems to happen regardless of length oftask. Even 
though my
tasks mostly complete in a couple minutes, I do have cases where they 
take
several minutes, and I don't see a difference. Of course, when 
atasktakes
5-10 minutes to complete, I'm going to notice and care about a 
20-second
   delaymuch less than when I'm trying to spin through a few tasks in a 
   minute
(which is a real-world need for me as well).

When reading up on pull queues a while back, I was a little confused 
about
where I would use them with my own backends. I definitely could see an
application for offloading work to an AWS Linux instance. But in either

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

Here are logs from three consecutive task executions over the past weekend, 
with only identifying information removed. You'll see that each task 
completes in a few milliseconds, but are 20 seconds apart (remember: I've 
already checked my queue configurations, nothing else is running on this 
backend, and I later solved the problem by setting countdown=1 when adding 
the task).  I don't see any pending latency mentioned.

0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks 
task_name=15804554889304913211 instance=0
0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks 
task_name=15804554889304912461 instance=0
0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks 
task_name=4499136807998063691 instance=0


The 20 seconds seems to happen regardless of length of task. Even though my 
tasks mostly complete in a couple minutes, I do have cases where they take 
several minutes, and I don't see a difference. Of course, when a task takes 
5-10 minutes to complete, I'm going to notice and care about a 20-second 
delay much less than when I'm trying to spin through a few tasks in a 
minute (which is a real-world need for me as well).

When reading up on pull queues a while back, I was a little confused about 
where I would use them with my own backends. I definitely could see an 
application for offloading work to an AWS Linux instance. But in either 
case, could you explain why it might help?

I saw you mention in a separate thread how M/S can perform differently from 
HRD even in cases where one wouldn't expect to see a difference. When I get 
around to it I'm going to create a tiny HRD app and run the same tests 
through that.

I also wonder if M/S could be responsible for frequent latencies in my 
admin console. Those have gotten more frequent and annoying the past couple 
of months ...

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

The tasks are not run transactionally, and in my testing the task is the 
only one in queue. In fact, I also ran the tests *somewhat* successfully on 
a separate app where this was the only code running. I say somewhat 
because, as I stated in my original post, the 20-second delays didn't 
happen every time; more like once every four requests. But when they did 
occur, it was almost precisely 20 seconds.

Below is the entirety of my handler:

class AdhocHandler2(webapp.RequestHandler):
def get(self):
task_create_time = datetime.datetime.now()
import string
task_create_time_s = task_create_time.strftime(%Y-%m-%d %H:%M:%S) + . + 
string.zfill(task_create_time.microsecond, 6)
t = taskqueue.Task(url='/batch/adhoc', params={'task_create_time': 
task_create_time_s}, target='overnight-external-data',
method='GET')
t.add(queue_name = 'overnight-tasks')

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/orTdYjTm7SkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

To do one better, here is the entirety of the Python code:

#!/usr/bin/env python
#
# Copyright 2007 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the License);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an AS IS BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import wsgiref.handlers
import datetime, time

from google.appengine.ext import webapp, db
from google.appengine.ext.webapp.util import run_wsgi_app 
from google.appengine.api import taskqueue

class AdhocHandler2(webapp.RequestHandler):
def get(self):
task_create_time = datetime.datetime.now()
import string
task_create_time_s = task_create_time.strftime(%Y-%m-%d %H:%M:%S) + . + 
string.zfill(task_create_time.microsecond, 6)
t = taskqueue.Task(url='/batch/adhoc', params={'task_create_time': 
task_create_time_s}, target='overnight-external-data',
method='GET')
t.add(queue_name = 'overnight-tasks')

def main():
application = webapp.WSGIApplication([ \
('/batch/adhoc2', AdhocHandler2), \
],
 debug=True)

run_wsgi_app(application)

if __name__ == '__main__':
main()

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ELiNYLg4B9cJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

And here is backends.yaml:

backends:
- name: overnight-external-data
  class: B1
  options: dynamic
  instances: 1

and queue.yaml:

queue:
- name: overnight-tasks
  rate: 50/s
  bucket_size: 50
  retry_parameters:
max_backoff_seconds: 1800

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/nIP2SeQDMUsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

Finally, it's probably an important clue that when I explicitly set 
countdown=1 when creating the task, the delay in executing the task is 
always almost exactly 1.5 seconds (not sure why it's not 1.0). If I don't 
set a countdown value, it's almost as if I had set countdown=20. Except 
that the ETA in the Task Queue Details page in the admin console reflects 
the approximate time the task was created, rather than adding 20 seconds to 
that.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/sbd5Clqozg8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Weird latency

2012-01-31 Thread Dave Loomer

My ignorant question: Why are we discussing M/S vs. HRD when the OP said he 
isn't accessing any data in serving his page?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/T8WGpRtowF4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

2012-01-31 Thread Dave Loomer

Hi Robert,

When targeting backend thebackend, this was done merely by specifying 
target=thebackend when creating the Task object. And then in backends.yaml 
I modify the instances parameter for the backend in question for each of my 
various tests.

I'm not really too concerned about failfast, so I'll skip that one for the 
moment ...

I haven't played with max concurrent requests, but in the test where I used 
a frontend, I did not see any additional instances spin up (and my site is 
very low-use, so there was probably a max of 1-2 instances to start with).

I see the same length of delay (~20 seconds) whether I'm testing with my 
rapid-fire test handler, or running in production with tasks that spend 1+ 
minutes doing actual work (lots of RPCs- datastore, URLfetch, memcache).

Hope this helps.
Dave

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xF79RpgjpAYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

2012-01-29 Thread Dave Loomer

Interesting. I saw your thread but wasn't entirely sure if it was the
issue. I think the thing that threw me off was that your delays were
being reflected in the request ms in the logs, while in my case they
mostly aren't.

Does setting the task countdown work for you? Or is ~1 second delay
still too much in your case?

On Jan 29, 10:24 am, pdknsk pdk...@googlemail.com wrote:
 I had the same problem, with often quite exact 20 seconds delays. More
 details in this thread.

 http://groups.google.com/group/google-appengine/browse_thread/thread/...

 In another configuration, the delay increased to several minutes.
 Please star this bug.

 http://code.google.com/p/googleappengine/issues/detail?id=6022

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] (mostly) Consistent 20-second delay in starting backend tasks

The abstract is that I have a hobby app (granted, I put a lot of time and 
energy into it) that does tons of mapreduce-esque backend processing 
through tasks that execute, then create a new task for the next step, etc. 
My site will never generate revenue so I aim to someday get my daily costs 
down to about a dollar or not much more; thus, I'm trying to do everything 
by limiting each of my backends (3 total backends each doing specialized 
tasks) to one instance each, while at the same time it's obviously 
important that tasks complete as quickly as possible. I'm looking for the 
happy medium between too many and too few instances; while I may find that 
adding instances or breaking up tasks so that I have fewer tasks running 
longer can to some degree address the concerns listed below, I've still 
discovered some things that concern and confuse me and I'd love to know the 
answers regardless.

*My concerns in summary*: I have noticed that even though the logs showed 
about 1 minute request handling time for each executed task, I was only 
executing about two tasks every three minutes. After looking into it, the 
problem seems to lie in the fact that I am configuring each backend with 
just a single instance. Running some metrics on a mapreduce batch sequence 
run on a single instance, a batch run that takes 255 minutes to complete 
through the executing of 183 tasks spends 82 of those minutes waiting for 
tasks to start after adding them to the queue. I've done lots and lots of 
analysis running controlled tests to assure myself that the issue is 
unrelated to my backend or task queue configuration, nor is my code doing 
anything that should cause any of this. There are no errors shown in the 
logs, and the logs (and my knowledge of my own app) also show me that none 
of my code is doing anything at all in the task queue or on the backend in 
question during these 82 minutes of waiting. It all seems to come down to 
how the GAE scheduler handles scheduling of requests or tasks on a 
single-instance backend.

*Details* -
Backend is standard B1 Dynamic. To create a more controlled and 
reproducible test I wrote a handler that does nothing but generate a new 
task which in turn runs that same handler, rinse and repeat. These 
requests, which the logs show to complete in a few milliseconds (with some 
exceptions; see below), exhibit pretty much the exact same delays which my 
production code, which takes an average of a minute to complete a task 
because it does actual work, shows. A variation of the simplified 
bare-bones handler I created adds a wrinkle of passing an argument to the 
launched task which records the time at which the task was added, so that 
the top of the handler can compare the current time with the time that the 
task was created in order to estimate the delay in starting the task. The 
delay is generally very, very close to exactly 20 seconds when running on a 
single-instance backend, which is more or less the same behavior as my 
production code and the bare-bones example described above.

*Tests*
Note that each of the scenarios below has been run many, many times, 
switching frequently from one test to the other to more or less eliminate 
the effects of periodic conditions in the infrastructure. Results have been 
extremely consistent so I'm confident in the numbers.
*
*
1. *Run the task on a frontend rather than on my backend*.
*Result: 100 tasks complete in a second or two. Curiously, the logs show 
that all requests are served by the same frontend instance.*

2. *Increase the number of instances on my backend to 5*.
*Result: Essentially the same performance as when running the tasks on the 
frontend. **Delay in starting a task ranges 0.01-.05 seconds. **By the time 
all tasks complete, two instances are spun up. Instances page shows 
requests distributed evenly amongst the two instances. Logs show that 
requests complete in 7-15 ms.*
*
*
3. *4-instance backend*.
*Result: By the time all tasks complete, two instances are spun up. 
Instances page shows requests distributed evenly amongst the two instances. 
Delay in starting a task ranges 0.28-.1.2 seconds. Logs show that requests 
themselves complete in around 500 ms.*
*Observations/Concerns: If the same number of instances are running as in 
the 5-instance configuration, with an ample number of idle instances 
waiting and ready, why are the results different? In the real world these 
results are just as acceptable to me as in the 5-instance configuration, 
because in the end it doesn't change cost much, but I'm still curious. And 
really, 500 ms is a long time to serve a request whose handler does nothing 
but add another task.*

4. *3-instance backend*.
*Result: By the time all tasks complete, 3 instances (yes, three, as 
opposed to two when configuring my backend with 4-5 instances) are spun up. 
Instances page shows the vast majority (75-85%) of requests go to one of 
the instances. Delay in starting a task ranges 0.01-.1.2

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

FWIW should note that my app is master/slave.  The pressure to move to 
HRD/Python 2.7 is very real, but right now I have too many concerns with 
replication delays and reading others' migration headaches with data 
volumes similar to mine, so I have no short-term plans to migrate.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/FddU5krbHIsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

I've been able to nearly solve the delay problem by setting countdown=1 in 
the Task constructor. This reduces the delay from 20 to about 1.5. Not sure 
why it's not closer to 1.0 but this will be fine. The time to serve the 
simple request is unaffected.

Still a strange bug (?).

Also, some other things lead me to believe the GAE just wants to rest for 
20 seconds. I started to wonder if in my previous tests I was just running 
things through too fast, even though that seemed unlikely since I was 
seeing the same delays in production when tasks take 1+ minutes to 
complete. But I put a time.sleep(10) in my handler just to make sure. 
Strangely, what this did was reduce the delay from 20 seconds down to 10. 
So the sleep gives 10 seconds of rest and then GAE still wants 10 more? 
Something like that. When I replaced time.sleep(10) with a tight loop 
lasting 10 seconds, my higher delays returned although they were more 
erratic (i.e. not consistently near 20 seconds).

Also, I tried creating a new app containing only the code I had written for 
my test, and got somewhat different results. I got the 
almost-exactly-20-seconds delay, but only about once every 4-5 requests. 
The other times it was near zero. Maybe this is because nothing else is 
going on in the app, so GAE has less need for rest? Almost as if it were 
a quota thing -- except the fact that I demonstrated earlier that I can 
eliminate the delay entirely by running tasks on a frontend, or using a 
5-instance backend, or (as now demonstrated) setting countdown=1 seems to 
eliminate that possibility.

All very strange.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/wYRmOXSR5-wJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Finding the logs for failed cron jobs and failed tasks