from:"Waleed Abdulla"

Re: [google-appengine] Re: GAE / Datastore issues right now?

2018-02-15 Thread Waleed Abdulla

Same here. Affecting all our apps. Seems like a global issue.

On Thu, Feb 15, 2018 at 12:16 PM, Patrick Tessier <
patrick.tess...@bookedin.net> wrote:

> Yes, I am seeing the same thing. Just in the last 10 minutes... Datastore
> timeouts etc.
>
>
> On Thursday, February 15, 2018 at 2:07:48 PM UTC-6, Thanasis Delenikas
> wrote:
>>
>> Is anybody else experiencing issues with GAE (Java) in the last 10
>> minutes or so?
>>
>> I see a ridiculous number of ConcurrentModificationException  - my app
>> is almost dead, nothing moves. All datastore ops seem to throw exceptions.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/62a6cace-202b-4c21-b8fa-
> 03b84bbb99fe%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAK0G7O-M7AQ9SeD--TPTR%3DXROKeXZjKdPZu64myY4rgBrgAxDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: The API call taskqueue.BulkAdd() took too long to respond and was cancelled

2014-07-15 Thread Waleed Abdulla

+1
Seeing a lot of those errors recently.




On Mon, Jul 14, 2014 at 9:55 AM, DB do...@hubmanity.com wrote:

 Same here. Mainly happens with taskqueue.BulkAdd(), but we've had a few
 for urlfetch.Fetch() and mail.Send(). We also didn't have any problem like
 this before, it suddenly started happening from a few days ago, and it's
 still happening frequently.

 google.appengine.api.apiproxy_rpc in CheckSuccess
  
 https://app.getsentry.com/hubmanity/hubmanity/group/25760336/DeadlineExceededError:
 The API call taskqueue.BulkAdd() took too long to respond and was cancelled.



 On Wednesday, July 9, 2014 3:47:22 PM UTC-7, Stuart Mitchell wrote:

 Hi all for the past week or so I have been seeing ALOT of these errors in
 a previously well functioning app.

 I haven't updated the app for 54 days so it isn't new code, has anyone
 else seen this problem.

 Stu

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Email feedback loop notifications when emails are marked as spam

2014-04-23 Thread Waleed Abdulla

The top email APIs (like Sendgrid and Amazon SES) notify applications when
an email bounces or is marked as spam. They know when an email is marked as
spam through email feedback loop agreements with major email client
providers.

App Engine notifies
applicationshttps://developers.google.com/appengine/docs/python/mail/bouncewhen
an email bounces back. But is there a way to know if an email was
marked as spam? Do those bounce notifications cover spam reports as well?
The documentation does not include a lot of details about the type of
notifications we get (soft bounces, hard bounces, spam reports, ...etc.).

Can anyone shed some light?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Problems with ImageUrls served from Blob Store of image files uploaded to Google Cloud Storage

2013-09-26 Thread Waleed Abdulla

I started noticing something that might be related today. I store images in
the Blob Store as well, and the generated serving URL works when I open it
in a browser. But that same URL returns 404 when I fetch it from inside the
app with urlfetch. It wasn't working for a few hours, but now it is.
Something is not right.




On Wed, Sep 25, 2013 at 11:50 PM, Aritz Uriarte de Alayo
ar...@egolike.comwrote:

 Hi Vinny!

 Well, the thing about this issue is that all urls I stored in my Model are
 correct, I mean, they point out to the correct pictures, but Google Cloud
 Storage or what have you is having some performance problems.

 I say that because it seems to be fixed from time to time only to get the
 same problem back for 1-2 hours and then wokr fine again. The Image files
 are image/jpeg, but as I said, those urls I use work wonders usually, it
 has only been recently that they are being slow showing the pictures.

 The aspect ratio is maintained if you use the =sXXX param at the end of
 the url as far as I know, and the maximum parameter value I send is 640. To
 be fair, when I set that parameter to 100 or 200 it works blazing fast, but
 when there are performance issues the server randomly sends me to 500 Error
 pages when using a url with =s640. Another fun fact is that if you reload
 that url in the browser once or twice it ends up loading the image without
 issues, and further accesses to said url (with the parameter set to 640)
 are instantaneous.

 To make it clear, I use the (
 ImagesServiceFactory.getImagesService().getServingUrl(suo); ) method to
 generate the url once and store it in my Model. Such url is created using a
 BlobKey generated from a Google Cloud Storage image/jpeg file and then I
 serve it with different resolutions attaching =sXXX at the end of such
 url.

 The problem seems to come and go these days while it worked perfectly for
 over 2 weeks prior to this, which leads me to think it has something to do
 with Google Cloud Storage maintenance or BlobStore acting weird.

 Anyway, thanks for the reply!

 On Wednesday, September 25, 2013 3:47:30 PM UTC+2, Vinny P wrote:

 On Mon, Sep 23, 2013 at 8:32 AM, Aritz Uriarte de Alayo 
 ar...@egolike.com **wrote:


 We have an application that stores image files in Google Cloud Storage
 and uses Blob Store to serve the ImageUrls. To prevent the server lag when
 generating said urls we decided to store them in some Model classes. Then,
 if we needed to serve the images at different pixel resolutions we could
 attach =sXXX to the end of the url as mentioned in the documentation.

 Now this has been working perfectly up until this past friday
 (09/20/2013), ever since then, some urls would take like 6-8 seconds to
 load the image or just straight up send an error message like this



 ImagesService occasionally has difficulty processing certain types of
 images. For the images you're encountering problems with, do they share
 anything in common? For example, are they extremely large, small, a certain
 filetype, etc?

 Try resizing the image by maintaining the aspect ratio; i.e. if you
 reduce the height by 50%, also reduce the width by 50%. If you must resize
 by differing percentages, try instead to use the crop tool.



 -
 -Vinny P
 Technology  Media Advisor
 Chicago, IL

 App Engine Code Samples: 
 http://www.**learntogoogleit.comhttp://www.learntogoogleit.com/

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] blobstoreservice serve with google storage - unreliable behavior

2013-09-25 Thread Waleed Abdulla

I noticed something similar today in one of my apps which I uploaded today.
The other app (an exact copy, one is dev and the other is production) works
okay.

I store images in the Blob Store, and at some point I fetch them
with urlfetch.fetch(). These calls are now returning a 404 error.

For example, this link:

http://lh6.ggpht.com/o-VZKbpDso3CaDrHRuH7DAfzj8SFEEjXVloTzbXhJTtJMhy5wpf0Poy6Gsvab92hCx8NeAgARnAg1n4b_T05

It opens okay when I open it in my browser, but returns 404 when I opened
from inside the app with urlfetch.

app id: crowdchaos

On Wed, Sep 25, 2013 at 1:31 PM, IT supp...@i-telerad.net wrote:

Hi Google team,

Are there any performance issues with the google cloud storage?

My application initiates a download of two files based on user preferences
using blobstoreservice.serve. Both files located in the same bucket. One
file is served without an issue and another is not served.

Until today that functionality worked.

Any help would be appreciated.

Thanks

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an
email to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] blobstoreservice serve with google storage - unreliable behavior

2013-09-25 Thread Waleed Abdulla

Oh, and the same urlfetch call returns 500 error when done in the local dev
server. Which used to work before I downloaded the recent update.

On Wed, Sep 25, 2013 at 7:14 PM, Waleed Abdulla wal...@ninua.com wrote:

I noticed something similar today in one of my apps which I uploaded
today. The other app (an exact copy, one is dev and the other is
production) works okay.

I store images in the Blob Store, and at some point I fetch them
with urlfetch.fetch(). These calls are now returning a 404 error.

For example, this link:

http://lh6.ggpht.com/o-VZKbpDso3CaDrHRuH7DAfzj8SFEEjXVloTzbXhJTtJMhy5wpf0Poy6Gsvab92hCx8NeAgARnAg1n4b_T05

It opens okay when I open it in my browser, but returns 404 when I opened
from inside the app with urlfetch.

app id: crowdchaos

On Wed, Sep 25, 2013 at 1:31 PM, IT supp...@i-telerad.net wrote:

Hi Google team,

Are there any performance issues with the google cloud storage?

My application initiates a download of two files based on user
preferences using blobstoreservice.serve. Both files located in the same
bucket. One file is served without an issue and another is not served.

Until today that functionality worked.

Any help would be appreciated.

Thanks

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Has anyone received a warning email like this?

2013-07-17 Thread Waleed Abdulla

John,
I received an email from Google a day later (which was a pleasant
surprise) and they explained that they noticed a spike in rejected emails
from this app. As I mentioned earlier, we send emails through a third party
service. But then I realized that there is just one email that I send
through GAE, and this email is to myself whenever an unexpected error
happens in the app. It allows me to catch critical errors quickly. And as
it happened, we had a production issue a couple of days earlier that caused
a flood of errors, and therefore a flood of emails to me, and GAE
interpreted that as spam. It seems that they go by the number of emails,
rather than the number of recipients.

The part that I don't understand is why they mentioned rejected
emails. I received all the emails, so I don't understand what they meant
by rejected. But, anyway, the misunderstanding seems to have been
resolved and everything is back to normal.

Waleed

On Tue, Jul 16, 2013 at 8:25 PM, John Wheeler j...@highvolumeseller.comwrote:

Hi -

Can you post an update on your situation? It troubles me to hear what
Google sent you and I'd like to know if it was resolved properly.

Thanks,
John

On Friday, July 12, 2013 12:22:29 AM UTC-7, Waleed wrote:

Julie,
I replied to the email, and I hope to get an answer. Although, the
way they've worded it, we'll do our best to answer..., doesn't give me
much confidence that they will in time. The email seems automated, and it's
possible that GAE might block the app automatically in 5 days if whatever
triggered this isn't fixed. I don't want to have a downtime, so I'm trying
to solve the problem. I just need to know what to solve :)

And, yes, I agree that the reference to bulk email is interesting.
We've switched emails of all our apps to a different provide a long time
ago. And this particular app only sends a few emails, and none of them
through App Engine. I thought maybe someone hacked into the app and used it
to send emails, but GAE's dashboard confirms that we're not sending any.

On Thu, Jul 11, 2013 at 10:50 PM, Google Tasks Backup Moderator
gtb_mo...@logic-a-to-b.**com wrote:

Hi Waleed,

Have you received a response from Google? If you would like us to help
you understand that unusual activity, please reply directly to this email.

It is indeed puzzling that they send a link to Bulk Email Sender
guidelines if your app does not send emails.

Regards,

Julie

On 12 July 2013 15:30, Waleed Abdulla wal...@ninua.com wrote:

On Thu, Jul 11, 2013 at 8:04 PM, Lawrence Mok lawr...@a4support.comwrote:

so what have you done to get this email?

That's exactly what I'm trying to figure out. This is the site, in case
someone can point out something wrong about it.

http://www.symphonytools.com

Visit this group at
http://groups.google.com/**group/google-appenginehttp://groups.google.com/group/google-appengine
.
For more options, visit
https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Has anyone received a warning email like this?

2013-07-12 Thread Waleed Abdulla

Julie,
 I replied to the email, and I hope to get an answer. Although, the way
they've worded it, we'll do our best to answer..., doesn't give me much
confidence that they will in time. The email seems automated, and it's
possible that GAE might block the app automatically in 5 days if whatever
triggered this isn't fixed. I don't want to have a downtime, so I'm trying
to solve the problem. I just need to know what to solve :)

And, yes, I agree that the reference to bulk email is interesting.
We've switched emails of all our apps to a different provide a long time
ago. And this particular app only sends a few emails, and none of them
through App Engine. I thought maybe someone hacked into the app and used it
to send emails, but GAE's dashboard confirms that we're not sending any.





On Thu, Jul 11, 2013 at 10:50 PM, Google Tasks Backup Moderator 
gtb_modera...@logic-a-to-b.com wrote:

 Hi Waleed,

 Have you received a response from Google? If you would like us to help
 you understand that unusual activity, please reply directly to this email.
 

 It is indeed puzzling that they send a link to Bulk Email Sender
 guidelines if your app does not send emails.

 Regards,

 Julie



 On 12 July 2013 15:30, Waleed Abdulla wal...@ninua.com wrote:

 On Thu, Jul 11, 2013 at 8:04 PM, Lawrence Mok lawre...@a4support.comwrote:

 so what have you done to get this email?


 That's exactly what I'm trying to figure out. This is the site, in case
 someone can point out something wrong about it.

 http://www.symphonytools.com


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/groups/opt_out.




  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Has anyone received a warning email like this?

2013-07-11 Thread Waleed Abdulla

I received this email stating that one of my apps potentially violates
GAE's terms. It's a new app, has only a few users, doesn't send emails, and
doesn't do anything out of the ordinary. The email is very vague but
threatens to block my account if I don't fix the problem in 5 days. I have
no idea what to fix. Has anyone received such email? How did you handle it?

Regards,
Waleed




-- Forwarded message --
From: google-cloud-complia...@google.com
Date: Thu, Jul 11, 2013 at 3:14 PM
Subject: [5-749501307] Your Google App Engine Application
To: nore...@google.com


Hello,

We recently detected activity associated with your Google App Engine
application s~symphonytools that potentially violates the App Engine Terms
of Service. To see your application, please go to
http://appengine.google.com.

Rest assured that your account and its associated applications are still
active. If you would like us to help you understand that unusual activity,
please reply directly to this email. We will do our best to reply to your
submission as soon as possible.

If we continue to detect unusual activity on your App Engine applications
over the next 5 business days, we may take action on your account to
protect our users. You will be notified of any changes to your account.

We recommend that you read through the App Engine Terms of Service,
http://code.google.com/appengine/terms.html, to learn more about prohibited
activity, policy violations and other activities that may adversely affect
your account standing. We are committed to your business and look forward
to helping you solve this issue.

Please also ensure you read through the App Engine Bulk Email Sender
guidelines and verify that your application complies with the stated
guidelines: https://developers.google.com/appengine/docs/python/mail/#bulk

Thank you,

Google App Engine team

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Has anyone received a warning email like this?

2013-07-11 Thread Waleed Abdulla

On Thu, Jul 11, 2013 at 8:04 PM, Lawrence Mok lawre...@a4support.comwrote:

 so what have you done to get this email?


That's exactly what I'm trying to figure out. This is the site, in case
someone can point out something wrong about it.

http://www.symphonytools.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: DKIM - not clear why not working

2012-10-02 Thread Waleed Abdulla

You seem to be doing better than I am. I'm not even able to get Google Apps
dashboard to verify my DKIM DNS records for over a week now and I've tried
every possible thing I can think of.



On Tue, Oct 2, 2012 at 11:28 AM, johnP j...@thinkwave.com wrote:

 Wanted to 'bump' this issue - almost a month has gone by. I have posted
 this question here on the board, on StackOverflow, and have filed a
 production issue.  I got an email that the production issue was
 'Acknowledged'.  Yay!

 Anyone have any thoughts as to next steps to resolve this issue?  Is the
 $500 paid support the only viable alternative?  Do they respond to
 questions to people who purchase the paid support?

  johnP




 On Wednesday, September 5, 2012 11:20:52 AM UTC-7, johnP wrote:


 Just a bit confused why emails sent by my app are not being signed with
 DKIM.

 1.  On my Google Apps dashboard, it says that my domain is
 Authenticating email
 2.  I seem to have my DNS TXT record setup using the specified values
 (also, am assuming if I have the green-light for authenticating email, this
 value has been validated by Google Apps).
 3.  Email send is being triggered by user browsing an application on
 custom url: www.hi___e .com
 4.  From address is similar to Salt Point via Hi___e
 announce-noreply@hi___e .com
 5.  announce-noreply@hi___e .com is an owner of the account.

 I'm assuming if all the above parameters are met, the emails should be
 signed.  Dunno which of my assumptions above is erroneous.

 Thanks -


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/nYm91qkd6vEJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DKIM - not clear why not working

2012-10-02 Thread Waleed Abdulla

Confirmed there is a problem on Google side:

https://productforums.google.com/forum/#!category-topic/apps/verification-and-mx-records/FOsrXULt66Y

On Tue, Oct 2, 2012 at 3:19 PM, Waleed Abdulla wal...@ninua.com wrote:

You seem to be doing better than I am. I'm not even able to get Google
Apps dashboard to verify my DKIM DNS records for over a week now and I've
tried every possible thing I can think of.

On Tue, Oct 2, 2012 at 11:28 AM, johnP j...@thinkwave.com wrote:

Wanted to 'bump' this issue - almost a month has gone by. I have posted
this question here on the board, on StackOverflow, and have filed a
production issue. I got an email that the production issue was
'Acknowledged'. Yay!

Anyone have any thoughts as to next steps to resolve this issue? Is the
$500 paid support the only viable alternative? Do they respond to
questions to people who purchase the paid support?

johnP

On Wednesday, September 5, 2012 11:20:52 AM UTC-7, johnP wrote:

Just a bit confused why emails sent by my app are not being signed with
DKIM.

1. On my Google Apps dashboard, it says that my domain is
Authenticating email
2. I seem to have my DNS TXT record setup using the specified values
(also, am assuming if I have the green-light for authenticating email, this
value has been validated by Google Apps).
3. Email send is being triggered by user browsing an application on
custom url: www.hi___e .com
4. From address is similar to Salt Point via Hi___e
announce-noreply@hi___e .com
5. announce-noreply@hi___e .com is an owner of the account.

I'm assuming if all the above parameters are met, the emails should be
signed. Dunno which of my assumptions above is erroneous.

Thanks -

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: NDB ImportError!

2012-09-25 Thread Waleed Abdulla

The first request is the warmup request, and in my case it only has 3 lines
to import ndb, db, and one other module. So it shouldn't take a long time
at all. I can't verify right now since I already killed the instance, but I
think a DeadlineExceededError is very unlikely in this case.



On Mon, Sep 24, 2012 at 10:41 PM, Anand Mistry amis...@google.com wrote:

 Is a DeadlineExceededError exception raised on the first request to the
 instance?


 On Tuesday, 25 September 2012 13:00:00 UTC+10, Waleed wrote:

 Hi all,
 I'm getting random import errors from time to time. They start on one
 of the instances, and then that instance keeps generating those errors
 until I shut it down.

 I tried putting a _ah/warmup request, and in that request I do the
 import:

 * from google.appengine.ext import ndb*

But that doesn't seem to solve the problem. What am I missing? I'm
 including the stack trace below.

 Thanks
 Waleed


 

 2012-09-24 13:28:32.968 /panorama/blogger?stream_**
 suffix=__bloggerlast_visit_**date=1348495705view=manage**
 offset=0limit=5parent_page_**name=panorama 500 291ms 0kb Mozilla/5.0
 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
 AlexaToolbar/alxf-2.15
 80.87.92.2 - - [24/Sep/2012:13:28:32 -0700] GET
 /panorama/blogger?stream_**suffix=__bloggerlast_visit_**
 date=1348495705view=manage**offset=0limit=5parent_page_**name=panorama
 HTTP/1.1 500 0 
 http://www.networkedblogs.**com/panoramahttp://www.networkedblogs.com/panorama
 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
 AlexaToolbar/alxf-2.15 www.networkedblogs.com ms=291 cpu_ms=238
 instance=**00c61b117c7c127a4e5e1415265fc6**70470c
 E 2012-09-24 13:28:32.788
 Traceback (most recent call last):
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/runtime/wsgi.**py, line 195, in Handle
 handler = _config_handle.add_wsgi_**middleware(self._LoadHandler()**)
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/runtime/wsgi.**py, line 250, in _LoadHandler
 __import__(cumulative_path)
   File /base/data/home/apps/s~**networkedblogshr/live.**
 361951794551741701/website/**panorama_page.py, line 9, in module
 from google.appengine.ext import ndb
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/ext/ndb/__**init__.py, line 7, in module
 from tasklets import *
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/ext/ndb/**tasklets.py, line 68, in module
 from .google_imports import apiproxy_stub_map
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/ext/ndb/**google_imports.py, line 23, in module
 from google.appengine.api import prospective_search
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/api/**prospective_search/__init__.**py, line 24, in
 module
 from prospective_search import *
   File /base/python27_runtime/**python27_lib/versions/1/**
 google/appengine/api/**prospective_search/**prospective_search.py, line
 75, in module
 from google.appengine.api.**prospective_search import error_pb
 ImportError: cannot import name error_pb

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/0rcUU4U_gOIJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Naked https (SSL) domain?

2012-09-24 Thread Waleed Abdulla

Anyone know if VIP solves the problem and allows naked domains?

Based on my quick reading of the
documentationhttps://developers.google.com/appengine/docs/ssl,
it says that virtual IPs might change, but it doesn't say if that happens
only when changing VIP settings or if it could happen at any time!

Waleed

On Sun, Jul 22, 2012 at 10:22 AM, Drake drak...@digerat.com wrote:

Talk to the Apps For Domains guys. If you get a good answer share it with
the list. Currently I think this is the biggest limitation of GAE/GAFD.
You pretty much have to have a redirect server out in the world that 301s
all your naked requests to the www domain.

** **

That is a sucky solution. (it is also sub optimal if you want short domain
names)

** **

*From:* google-appengine@googlegroups.com [mailto:
google-appengine@googlegroups.com] *On Behalf Of *Thomas Wiradikusuma
*Sent:* Sunday, July 22, 2012 10:13 AM
*To:* google-appengine@googlegroups.com
*Subject:* [google-appengine] Naked https (SSL) domain?

** **

Hi guys,

** **

I'm using GAE's official SSL SNI support in my app. Opening
https://www.temanmudik.com is fine. But opening https://temanmudik.com(naked
domain) gives warning (in Chrome) that essentially says, Hey, the
cert is not for temanmudik.com, it's Google's. Non-SSL (
http://temanmudik.com) properly redirects to www. My appid is neytap.***
*

** **

So I got creative, I logged in to enom (from Google Apps) then removed all
A mappings (@ - google IPs) and replaced them with one URL Redirect (@ -
http://www.temanmudik.com). Non-SSL naked domain is still working, but
now Chrome gives me This webpage is not available. Google Chrome's
connection attempt to temanmudik.com was rejected. The website may be
down, or your network may not be properly configured. when I try to access
https://temanmudik.com.

** **

Anyone know how to properly redirect naked https domain in GAE? I notice
there is similar question in this forum (
https://groups.google.com/forum/?fromgroups#!newtopic/google-appengine/google-appengine/A-8I1X9vouk),
but it's 2009 and uses unofficial SSL support.

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/BtUQJeGPVCcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] NDB ImportError!

2012-09-24 Thread Waleed Abdulla

Hi all,
I'm getting random import errors from time to time. They start on one
of the instances, and then that instance keeps generating those errors
until I shut it down.

I tried putting a _ah/warmup request, and in that request I do the
import:

* from google.appengine.ext import ndb*

   But that doesn't seem to solve the problem. What am I missing? I'm
including the stack trace below.

Thanks
Waleed




2012-09-24 13:28:32.968
/panorama/blogger?stream_suffix=__bloggerlast_visit_date=1348495705view=manageoffset=0limit=5parent_page_name=panorama
500 291ms 0kb Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101
Firefox/15.0.1 AlexaToolbar/alxf-2.15
80.87.92.2 - - [24/Sep/2012:13:28:32 -0700] GET
/panorama/blogger?stream_suffix=__bloggerlast_visit_date=1348495705view=manageoffset=0limit=5parent_page_name=panorama
HTTP/1.1 500 0 http://www.networkedblogs.com/panorama; Mozilla/5.0
(Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
AlexaToolbar/alxf-2.15 www.networkedblogs.com ms=291 cpu_ms=238
instance=00c61b117c7c127a4e5e1415265fc670470c
E 2012-09-24 13:28:32.788
Traceback (most recent call last):
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py,
line 195, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py,
line 250, in _LoadHandler
__import__(cumulative_path)
  File
/base/data/home/apps/s~networkedblogshr/live.361951794551741701/website/panorama_page.py,
line 9, in module
from google.appengine.ext import ndb
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/__init__.py,
line 7, in module
from tasklets import *
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py,
line 68, in module
from .google_imports import apiproxy_stub_map
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/google_imports.py,
line 23, in module
from google.appengine.api import prospective_search
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/api/prospective_search/__init__.py,
line 24, in module
from prospective_search import *
  File
/base/python27_runtime/python27_lib/versions/1/google/appengine/api/prospective_search/prospective_search.py,
line 75, in module
from google.appengine.api.prospective_search import error_pb
ImportError: cannot import name error_pb

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Anyone else having difficulty deploying to GAE in the last two days?

2012-08-22 Thread Waleed Abdulla

Usually when I deploy to GAE (using appcfg.py) it takes less than a minute.
Occasionally, it might takes a couple of minutes. But in the last two days,
it's been consistently taking 5 to 10 minutes each time! I'm deploying to
our dev app, which has very little traffic. Sometimes it even fails after
10 minutes of waiting. Anyone else getting this?


. initial uploading steps go fast ...
03:26 PM Compilation completed.
03:26 PM Starting deployment.
03:26 PM Checking if deployment succeeded.
03:26 PM Will check again in 1 seconds.
03:26 PM Checking if deployment succeeded.
03:26 PM Will check again in 2 seconds.
03:26 PM Checking if deployment succeeded.
03:26 PM Will check again in 4 seconds.
03:26 PM Checking if deployment succeeded.
03:26 PM Will check again in 8 seconds.
03:26 PM Checking if deployment succeeded.
03:26 PM Will check again in 16 seconds.
03:26 PM Checking if deployment succeeded.
03:26 PM Will check again in 32 seconds.
03:27 PM Checking if deployment succeeded.
03:27 PM Will check again in 60 seconds.
03:28 PM Checking if deployment succeeded.
03:28 PM Will check again in 60 seconds.
03:29 PM Checking if deployment succeeded.
03:29 PM Will check again in 60 seconds.
 and it keeps repeating the last two lines for 10 minutes or more

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Backend performance, compared

2012-08-08 Thread Waleed Abdulla

Thanks, Jeff. Is it possible to repeat the test with qps  10 to rule out
the limit that Johan pointed out? In other words, how big is the
performance difference if you had less requests that do more work?




On Wed, Aug 8, 2012 at 10:48 AM, Jeff Schnitzer j...@infohazard.org wrote:

 On Wed, Aug 8, 2012 at 10:42 AM, Drake drak...@digerat.com wrote:
  Jeff your Low volume is effecting your numbers. The more you do the more
  consistent the numbers become.  I don't know what black magic Google
 does,
  but the more people I have the more my instances get faster, the faster
 my
  datastore reads (and writes) become, and the better the hit ratio on my
  memcache.

 I do not believe you without quantitative measurements.  I see no
 reason to believe that something that sucks at low volume will
 suddenly stop sucking at high volume, and I have a lot of reason to
 distrust your wild assertions.

 Also:  The system requiring 10 backends is a real-world app with 2k+
 simultaneous users.  How much more volume do you think it needs?

 Jeff

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: ArgumentError: An error occurred parsing (locally or remotely) the arguments to datastore_v3.Put().

2012-08-01 Thread Waleed Abdulla

I'm getting this error as well, a lot. Did you find a work-around?

I opened an issue, but a quick search shows that the app engine team thinks
this is fixed already!

http://code.google.com/p/googleappengine/issues/detail?id=7917





On Mon, May 21, 2012 at 1:40 PM, jay kyburz@gmail.com wrote:

 Oh and the logs are chockablock full of

 DeadlineExceededErrors just importing modules like random or sys.

 Looks like something is going a little haywire.



 On Tuesday, 22 May 2012 06:18:45 UTC+10, jay wrote:

 Hey All,

 My App started throwing this error over the last 24 hours or so and users
 have been complaining the site is slow.

 After doing a search I see that there was an error in the Bug DB from
 earlier in the year but it is marked fixed.

 AppId = jupitersfolly

 Any help tracking down what is causing the problem would be greatly
 appreciated.

 Jay.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/ghnGv8UC7ecJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] HRD migration tool hangs for hours - how to find a root cause?

2012-07-04 Thread Waleed Abdulla

On my initial copy I remember it staying at 99% for a long time, but it
eventually finished. I was copying a lot of data so the full copy took 3.5
days. In the sync phase, I remember it staying at 99% for almost half the
time of every sync and sometimes more. My first sync took 12 hours, and the
second 6 hours. On the last sync it stayed on 99% too long so I stopped it
and jumped to the last stage (read-only mode) and that made it much faster.

Waleed


On Tue, Jul 3, 2012 at 12:58 AM, Alexander Trakhimenok 
alexander.trakhime...@gmail.com wrote:

 We are migrating our app to an HRD version and the migration tool process 
 hangs on 2nd step Copy at 99% for 10+ hours.

 We've restarted migration but it did not help - still hangs at 99% and the 
 approximately remaining time is steadily and slowly growing.

 I've been googling around and as I understand it can be because of indexes 
 issue (5K index entry limit per entity) or some other reason that prevent an 
 entity to be stored to the new HRD database. Actually removing some indexes 
 helped us to move from 98.50% to 99% and  not  any further.

 But we can not find any error/status logs that would point us on what exactly 
 entity kind/id causing the issue.

 On the Migration Tool screen it says:
 =
 If there is a problem, you can pause the migration. While paused, all 
 migration mapreduces will be paused and but no other actions will be taken 
 until an administrator either resumes the migration or reverts the process. 
 To pause the migration, please click the button below.
 =

 But how do we know if there is a problem and what exactly the problem is? It 
 would be cool to have a last or list if error messages  entity keys for the 
 migration process.

 Any ideas how to identify and solve the issue? We have major release pending 
 due to limitations of M/S database and would greatly appreciate any help.


 I've raised a production issue for this as well 
 http://code.google.com/p/googleappengine/issues/detail?id=7787


 but may be someone from community can help us to resolve it before Google 
 Engineers notice the ticket.


 Alexander Trakhimenok


 Lead developer on http://www.myclasses.org/ project powered by GAE (AppEngine 
 + Python + Django).

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/AYfYbuyKGsYJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Is GAE down for anyone else?

2012-06-27 Thread Waleed Abdulla

Our app went down and started showing the Server Error page for all URLs,
but the dashboard doesn't show any new requests hitting the app as of 10
minutes ago. We're on HRD (migrated two weeks ago). Anyone else noticing a
similar problem?

Waleed
App id: networkedblogshr.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Is GAE down for anyone else?

2012-06-27 Thread Waleed Abdulla

I managed to fix the problem by uploading the app again. This is the first
time I encountered an issue like this where the dashboard works but
requests never reach my instances (or maybe they do but the instances are
not capable of logging them).



On Wed, Jun 27, 2012 at 1:55 AM, Waleed Abdulla wal...@ninua.com wrote:

 Our app went down and started showing the Server Error page for all URLs,
 but the dashboard doesn't show any new requests hitting the app as of 10
 minutes ago. We're on HRD (migrated two weeks ago). Anyone else noticing a
 similar problem?

 Waleed
 App id: networkedblogshr.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Sort Orders Are Ignored on Multi-ValuedProperties?

2012-06-22 Thread Waleed Abdulla

I have a model with a multi-valued property, events, and I want to filter
by a specific range of events and sort by them at the same time:

class MyModel(db.Model):
  events = db.StringListProperty(indexed=True)

print MyModel.all().filter(events =, x).filter(events =,
y).order(events).fetch(100)


I expect the results to be sorted by whichever value in the list
property that matched the range condition (and that's how I understand the
indexs to be organized). This seems to be working as expected in the SDK,
and if I reverse the order (-events) it returns the results in reverse as
well, which is great.

However, I'm concerned about the section below from the
documentationhttps://developers.google.com/appengine/docs/python/datastore/queries
which
states that sort orders are ignored. It's not clear to me, though, if the
sort order is ignored completely or ignored only for the other values in
the property list that did not match the range condition. And if it is
ignored, then how come reversing the sort oder in the code above also
reversed the order of the results I get?

Sort Orders Are Ignored on Properties with Equality Filters

One important caveat is queries with both an equality filter and a sort
order on a multi-valued property. In those queries, the sort order is
disregarded. For single-valued properties, this is a simple optimization.
Every result would have the same value for the property, so the results do
not need to be sorted further.

However, multi-valued properties may have additional values. Since the sort
order is disregarded, the query results may be returned in a different
order than if the sort order were applied. (Restoring the dropped sort
order would be expensive and require extra indices, and this use case is
rare, so the query planner leaves it off.)

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Task Queue API: Pricing Risk?

2012-06-22 Thread Waleed Abdulla

I wonder if the task queue is using the datastore behind the scenes and the
cost of those calls are included in the datastore quota?  Have you checked
to see if using the task queue increases your datastore calls? If so, then
it's not exactly free.



On Fri, Jun 22, 2012 at 3:33 PM, Jason Collins jason.a.coll...@gmail.comwrote:

 I think everyone on this thread should just shush. ;)

 On Jun 22, 11:22 am, vlad vlad.troyan...@gmail.com wrote:
  Why did you single out TaskQueue? There are other secondary services
 which
  are kind of free on GAE currently once you take the primary costs out.
 For
  example memcache comes to mind. Can they jack up the price? Of course
 they
  can. Probably with little notice and/or justification. Truth is
  marketing/business side was never a strong suite of GAE team. They are
  excellent engineers. If you are concerned about pricing risk you may want
  to look at AWS. Biz model there seems more straightforward/sustainable.
  Thus pricing risk seems lower but I would not bet my house on that.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] App engine app spinning up instances out of control

2012-06-08 Thread Waleed Abdulla

Johan,
Thanks for letting us know. I just filed a production issue for this
before reading your response:

http://code.google.com/p/googleappengine/issues/detail?id=7664





On Fri, Jun 8, 2012 at 7:47 AM, Johan Euphrosine pro...@google.com wrote:

 The Master Slave infrastructure is currently suffering increased
 latency, the reliability is aware of the problem and working on a
 solution.

 On Fri, Jun 8, 2012 at 4:41 PM, John Wheeler j...@highvolumeseller.com
 wrote:
  Hi, I have three apps. Only the one on the Master/Slave datastore is
 taking
  a hit. It spins up a new instance for each request and has 20 instances
  right now. The app is highvolumeseller.
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/vxuDn8qbcUMJ.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.



 --
 Johan Euphrosine (proppy)
 Developer Programs Engineer
 Google Developer Relations

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Appengine down (M/S Datastore)

2012-06-08 Thread Waleed Abdulla

Please add your app details to this issue:

http://code.google.com/p/googleappengine/issues/detail?id=7664



On Fri, Jun 8, 2012 at 8:26 AM, Nikhil Jain nikhil.jaint...@gmail.comwrote:

 Dear Doug,
 Here in India also we'r promising to our customer about 99.9% up-time
 servers.
 but this is truly unprofessional-ism from GAE team.

 Dear GAE team,
 please take it this on serious basis, we loosing our customer and accuracy
 in data as well.
 Seeking your help on urgent basis


 On Fri, Jun 8, 2012 at 8:51 PM, doright doug.stodd...@gmail.com wrote:

 yes, I'm experiencing massive latencies and timeouts  M/S datastore too.


 This is at least the third time its happened in the last few months, AT
 THE SAME TIME ON A FRIDAY.

 Googlers, WHATEVER you're doing, or NOT DOING, at this time, please stop!

 Its end of day, end of week for us here in Europe, so very bad time to be
 down.

 www.bikeshopmanager.com



 On Friday, June 8, 2012 5:15:51 PM UTC+2, Nikhil Jain wrote:

 At your end is it working, please update??

 Appengine team, Please resolved it asap.

 On Fri, Jun 8, 2012 at 8:31 PM, Nikhil Jain 
 nikhil.jaint...@gmail.comwrote:

 Appengine seems down,
 Any idea??


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/0TPZHiOc9UEJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Problem with mail quota

2012-06-08 Thread Waleed Abdulla

I'm hit by this problem as well. Trying to migrate an app from M/S to HRD,
but can't because the new app can't handle the email load. We sends
thousands of emails a day right now from the M/S app, and if we make the
switch to HRD we'll not be able to send emails for a while. Even worse, the
new app will probably not be billed at all because we can't direct traffic
to it yet, because it can't send emails.

I think this feature should only trigger for new customers, and not for
the already-paying customers.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unusually high memcache latencies

2012-05-22 Thread Waleed Abdulla

I'm noticing excessive datastore delays today (M/S), and generally a lot of
API calls timing out. Might be related. Errors like:

*DeadlineExceededError: The API call logservice.Flush() took too long to
respond and was cancelled.*

Also:

*class 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError':
The API call urlfetch.Fetch() took too long to respond and was cancelled.*




On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann gay...@gmail.com wrote:

 I think I'm having the same issue.

 How did you get that look at memcache latency specifically?

 On May 22, 9:41 am, Rishi Arora rishi.ar...@ship-rack.com wrote:
  Ah, I was mistaken.  The restriction is the other way round.  Py2.7
  requires HRD.
 
  On Tue, May 22, 2012 at 11:24 AM, Joshua Smith joshuaesm...@charter.net
 wrote:
 
 
 
 
 
 
 
   HRD does not require Py2.7
 
   On May 22, 2012, at 12:01 PM, Rishi Arora wrote:
 
   Also, I imagine going to HRD would indirectly help my situation because
   I'll be forced to upgrade to python2.7.  So, while my instance is
 waiting
   forever for memcache to return some data, it would be available to
 service
   other requests, and keep the number of active instances low.  Am I
 correct
   in this reasoning?
 
   On Tue, May 22, 2012 at 10:59 AM, Rishi Arora 
 rishi.ar...@ship-rack.comwrote:
 
   My app usually consumes ~30 instance hours every day, and occasionally
   spikes to ~35 when Master-Slave datastore latencies go up.  We will be
   transitioning to HRD soon, but today large latency spikes have caused
 my
   instance hours to reach ~50 already, in the first 9 hours of the day.
  I
   peeked at the appstats output to see where this spike is coming from,
 and
   it appears to be memcache (see attached pic).  This seems absurd and
 not
   something app operators should be expected to pay for.  Anybody else
 seen
   this?
 
   Thanks in advance
   - Rishi
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
--
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unusually high memcache latencies

2012-05-22 Thread Waleed Abdulla

I meant that the extra delays might not be just memcache, but an API delay
in general because I'm noticing it with the logging service and the
urlfetch service as well, as shown by the errors I listed. It's really hard
to pinpoint these things because different apps have different use patterns
so they might notice the problem in different ways.




On Tue, May 22, 2012 at 1:59 PM, Rishi Arora rishi.ar...@ship-rack.comwrote:

 Perhaps they're related, but Google generally doesn't do anything about
 datastore deadline exceeded errors.  They claim that you should either be
 using HRD or make your app tolerant of M/S data-store spikes by making
 asynchronous calls.  In my case, I studied appstat output thoroughly over a
 large sample, and the problem is specifically for memcache API calls.

 FYI, here's a production issue I just logged, which you should consider
 starring, so that it gets attention:

 code.google.com/p/googleappengine/issues/detail?id=7554



 On Tue, May 22, 2012 at 3:27 PM, Waleed Abdulla wal...@ninua.com wrote:

 I'm noticing excessive datastore delays today (M/S), and generally a lot
 of API calls timing out. Might be related. Errors like:

 *DeadlineExceededError: The API call logservice.Flush() took too long to
 respond and was cancelled.*

 Also:

 *class
 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError': The API
 call urlfetch.Fetch() took too long to respond and was cancelled.*




 On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann gay...@gmail.comwrote:

 I think I'm having the same issue.

 How did you get that look at memcache latency specifically?

 On May 22, 9:41 am, Rishi Arora rishi.ar...@ship-rack.com wrote:
  Ah, I was mistaken.  The restriction is the other way round.  Py2.7
  requires HRD.
 
  On Tue, May 22, 2012 at 11:24 AM, Joshua Smith 
 joshuaesm...@charter.netwrote:
 
 
 
 
 
 
 
   HRD does not require Py2.7
 
   On May 22, 2012, at 12:01 PM, Rishi Arora wrote:
 
   Also, I imagine going to HRD would indirectly help my situation
 because
   I'll be forced to upgrade to python2.7.  So, while my instance is
 waiting
   forever for memcache to return some data, it would be available to
 service
   other requests, and keep the number of active instances low.  Am I
 correct
   in this reasoning?
 
   On Tue, May 22, 2012 at 10:59 AM, Rishi Arora 
 rishi.ar...@ship-rack.comwrote:
 
   My app usually consumes ~30 instance hours every day, and
 occasionally
   spikes to ~35 when Master-Slave datastore latencies go up.  We will
 be
   transitioning to HRD soon, but today large latency spikes have
 caused my
   instance hours to reach ~50 already, in the first 9 hours of the
 day.  I
   peeked at the appstats output to see where this spike is coming
 from, and
   it appears to be memcache (see attached pic).  This seems absurd
 and not
   something app operators should be expected to pay for.  Anybody
 else seen
   this?
 
   Thanks in advance
   - Rishi
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
--
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google
 Groups Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine

Re: [google-appengine] Re: Seeing huge delays on task queue execution - 13sec+ per task

2012-03-28 Thread Waleed Abdulla

Agreed. More control over priority of tasks would be nice. I'd like to be
able to designate a queue as high-priority so it's tasks execute at the
same priority as user requests, or low-priority so they only execute when
there are free instances and no other pending requests.



On Tue, Mar 27, 2012 at 6:48 PM, Gopal Patel patelgo...@gmail.com wrote:

 i think this happen only when they release a new version of app engine. so
 it linked to app wise upgrade.


 On Wed, Mar 28, 2012 at 3:30 AM, vlad vlad.troyan...@gmail.com wrote:

 I have been screaming about this for a looong time. The offcial position
 of GAE team is - we don't guarantee on time execution. Even though this
 issue is not about on time execution but rather about bugs/inefficiency of
 their scheduler. They just closed my ticket about that. Feel free to star

 http://code.google.com/p/googleappengine/issues/detail?id=7185
 http://code.google.com/p/googleappengine/issues/detail?id=4901


 On Tuesday, March 27, 2012 5:25:51 AM UTC-7, Jan Zawadzki / Hapara wrote:

 Is anyone else seeing these delays?

 Jan

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/AeanzAICxSAJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] In your opinion what are the most complex web applications built on top of google app engine ?

2012-03-27 Thread Waleed Abdulla

 Waleed is the sleeping giant on this list.  Every so often he steps out of
 the shadow says something like “I made this change and it cut my instances
 by about 450” or ...

Haha. Having a huge monthly bill is *not* something to celebrate :)




 Waleed, we'd love to hear some war stories if you feel like writing
 about your lessons learned :-)


- About two years ago, the App Engine team organized a meetup for a few
early adapters to meet with the App Engine team. In one conversation with
one of the App Engine developers, he casually said: So you're using App
Engine in production?! You're very brave. I don't think I would do that.

 In hindsight, he might've been right. The App Engine was so unstable then.
I think the early-adapters had more confidence in it than it's own
developers.

- One thing I learned the hard way was how to write code with errors in
mind. I'm sure everyone who's been on GAE for a while has learned that as
well. GAE is a distributed system and failures happen a lot. It's not a
fault in GAE, but rather the nature of distributed systems. On GAE, though,
you're running on a distributed system right from the start, not only after
your app grows. So the only responsible approach is to assume that any code
you write will sometimes crash midway. Writing idempotent code comes
naturally to me now. Learning how to order which entities to save first and
which to save last is another useful skill to reduce data inconsistencies
when the datastore times out and the rest of your function doesn't execute.

- Recently, I've been storing structured data in single entities more
often. It's not for every case, but there are so many situations in which
the best way to store your data is to simply convert everything to a json
string and store it in a text property. And not only to cut cost, but it
actually makes a lot of sense to be able to load one entity and immediately
have a rich data structure with everything you need for that operation. I
believe the new NDB supports structured properties out of the box now.

- It still surprises me sometimes how much can be done with so little
code. Back in 2009, Bill Katz was one of the early developers excited about
GAE. He's one of the sharpest people I've met. Among many things, he built
a full-text search
libraryhttp://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-App-Engine
that
was simple and elegant. It supported full-text search with stemming and
multi-word searches, and it was all done in about 500 lines of code (and a
stemming library). I've been using it since and I'm very happy with it.
It's the reason I'm not too desperate for the App Engine team to launch
their long-over-due search feature.

I'd love to hear your stories as well if anyone is interested in
sharing.

Waleed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: equivalent for is_saved() with ndb

2012-03-27 Thread Waleed Abdulla

I've been looking for a good way to do something similar as well. I load
(or create) an entity, do some operations on it (that may or may not change
the entity), and then at the end I need to save it or skip saving it if
nothing has changed. It would be nice to have something like is_dirty()
which is True if the entity has never been saved or has changed since it
was loaded.



On Tue, Mar 27, 2012 at 3:19 PM, Guido van Rossum gu...@google.com wrote:

 Sorry to hear that. I guess you could manually set a flag on the
 parent entity that indicates that you haven't written it yet, and
 clear it in a pre_put_hook.

 BTW is there any information in the parent entities? Did you know you
 can create child entities for which no parent actually exists? All you
 need is the parent *key*, it doesn't need to have an entity.

 --Guido

 On Tue, Mar 27, 2012 at 14:47, Andreas a.schmi...@gmail.com wrote:
  not a good option in my case.
 
  On Mar 27, 2012, at 5:23 PM, Guido van Rossum wrote:
 
  You might be able to use get_or_insert() and benefit from the in-memory
 cache.
 
  On Tue, Mar 27, 2012 at 13:49, Andreas a.schmi...@gmail.com wrote:
  exactly what im doing. i create the keys myself.
 
  rootkey = ndb.Key('root', 'key')
  a = Asset(key=ndb.Key('Asset', 'keyname', parent=rootkey))
 
  a._has_complete_key() # returns true
 
  in my app i work in with entity batches and sometimes it happens that
 i need
  to create ancestors for that entity which could already exist so till
 now i
  used is_saved() to filter out the entities that are already saved to
 the
  datastore to avoid putting them again (or at least trying to put them
 again)
 
 
  On Mar 27, 2012, at 3:38 PM, Guido van Rossum wrote:
 
  On Tuesday, March 27, 2012 6:41:25 AM UTC-7, aschmid wrote:
 
  is there an equivalent of the db.Model function is_saved() with
 ndb.Model?
 
 
  No; what are you trying to do? You might be able to check whether the
 entity
  has a key and if so, whether that key isn't incomplete:
 
  if ent.key and ent.key.id():
# It has a complete key.
  else:
# Hasn't been written, ever.
 
  However this can be fooled if you explicitly set the key or the id
 when you
  create an entity, e.g.
 
  ent = Employee(id='joe')
 
  or
 
  ent = Employee(key=ndb.Key(Employee, 'joe'))
 
  (These two are equivalent.)
 
  --Guido van Rossum
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/MeORtJLdaqEJ.
  To post to this group, send email to google-appengine@googlegroups.com
 .
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com
 .
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
 
  --
  --Guido van Rossum (python.org/~guido)
 
  --
  You received this message because you are subscribed to the Google
 Groups Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google
 Groups Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 



 --
 --Guido van Rossum (python.org/~guido)

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] In your opinion what are the most complex web applications built on top of google app engine ?

2012-03-26 Thread Waleed Abdulla

I built an RPG Facebook game on app engine a couple of years ago. The
text-based kind, like Mafia Wars, in which you do jobs, earn gold, buy
weapons, level up, and fight other players. The technology behind it was
fairly complex. The game didn't take off, though, and I stopped working on
it.

My current app, NetworkedBlogs, is a big app as well: 1TB of data, 60~
front-end instances + 28 backend instances, 37 entity types (many of which
contain a lot of data in JSON format, as Jeff said, they would expand to
several tables each on a relational database).

Waleed



On Mon, Mar 26, 2012 at 1:17 PM, Kaan Soral kaanso...@gmail.com wrote:

 I've been reading about your project, on your posts

 I am assuming it's not called Stremor :)
 With .co .net and possible typo-names still unbought and with a Plesk
 domain, looks very unprofessional to me :)

 You should buy those - and possible typos, if Stremor is the actual name,
 but I bet its not, it doesn't sound cute


 On Monday, March 26, 2012 11:07:10 PM UTC+3, Brandon Wirtz wrote:

 Assuming we don’t fall flat on our face…

 Likely we will be one of the most complex web applications on GAE.

 Stremor.com



 Sorry I can’t give you lots of details. But we are building a large
 content platform, and if you look at some of my questions you can also
 probably figure out that we are doing Natural Language processing on the
 content.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/TTT9YkSMRSMJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Sharp increase in instance count starting two days ago!

2012-03-17 Thread Waleed Abdulla

Haha @Brandon Jeff. Thanks for the chuckle :) Believe it or not, I do
check for read-only mode and I have a lot of code dedicated to recovering
from GAE errors. True story.

I plan to switch to HRD at some point, but I fear it might be a major
hassle because we have a lot of data and entity types, and we use the
blobstore as well. Also, it worries me that if something goes wrong after I
flip the switch, I have no way to contact anyone to get help.

Good point about the reserve of goodwill, Jeff. I've definitely been
tapping into my goodwill savings recently.

Waleed

On Fri, Mar 16, 2012 at 10:07 PM, Ugorji ugo...@gmail.com wrote:

Hear, hear.

This is a clear case of moral hazard. The incentives to optimize and
reduce latency are backwards, since more efficiency leads directly to
reduced revenue for Google. We just have to trust that Google wouldn't
screw us?

Like Jeff says, there's a large amount of goodwill, but there has to be
some give from Google's part.

On Saturday, March 17, 2012 12:26:50 AM UTC-4, Jeff Schnitzer wrote:

To repeat that important point:

The problem with the current (new) billing model is that when the
service provider screws up, the customer pays. In the long run this
can only alienate the customer. How many times do I want to call my
cellphone provider and tell them that they screwed up my bill, even if
they apologize and give me a credit? Having actually gone through
exactly this scenario, the answer is twice before I change
providers.

Google has a significant reserve of goodwill with me, so it would take
a lot more than a few billing issues to make me choose another
platform. But I can't imagine that too many other people feel the
same way.

The solution to this *seems* pretty straightforward - Google should
stop the clock when executing internal RPC calls. We're already
paying for datastore operations. If you need to change what a
datastore operation costs to make it revenue-neutral, so be it. But
we've gone a step backwards - the whole point of moving to
bill-by-datastore-ops was to make pricing more transparent, yet what
we've actually produced is bill by datastore ops plus a random
additional amount of instance hours depending on how sick the
datastore happens to be right now. We were better off with
api_cpu_ms, at least that was consistent.

Actually, when you think about it, charging instance hours only makes
sense for single-threaded apps. In multithreaded apps, concurrency is
dependent on CPU usage, so charging by the megacycle really does make
sense. Really, single-threaded GAE needs a totally different billing
model than multi-threaded GAE.

Jeff

On Fri, Mar 16, 2012 at 10:34 PM, Brandon Wirtz drak...@digerat.com
wrote:
More seriously…

Waleed has one of those apps that I believe the concistency model makes
MS
the “right” choice for. Because eventual is not as instant as you might
want.

That said, I think MS seems to be a lot more temperamental in terms of
how
fast it performs and how the scheduler responds to conditions.

1200 is a crap ton, and while I realize the SLA doesn’t cover MS. This
seems
like a “Billing” error kind of thing that Google should take some
responsibility for.

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.**
com google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscribe@**googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en
.

Re: [google-appengine] Python2.7 Port: My 10x one line of code

2012-03-17 Thread Waleed Abdulla

Try with threadsafe:False and see if it affects the delay. It shouldn't,
but it's worth a try. When I switched one of my apps to 2.7 a few months
ago, enabling threadsafe caused 10x increase in response time so I disabled
it right away.



On Sat, Mar 17, 2012 at 11:59 AM, Brandon Wirtz drak...@digerat.com wrote:

 There are a number of string manipulation functions that seem slower on 2.7
 I don't think I had encountered any at 10x, but that one may call several
 that are half as fast nested.



 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Switched to HRD - So far not impressed

2012-03-17 Thread Waleed Abdulla

I thought HRD was supposed to solve the bad neighbor problem. As I read
somewhere (don't have the reference handy right now), when your app issues
a read request, it's sent to multiple datastore replicas in parallel and
the app takes the data from whichever datastore replies first.





On Fri, Mar 16, 2012 at 8:26 PM, Daniel Kramer danielkra...@gmail.comwrote:

  Interesting, I hadn't though about it that way.  Thanks for the info

 On Mar 16, 2012, at 5:54 PM, vlad wrote:

 Daniel keep in mind that you data is still co-hosted with dozens or
 hundreds on other apps on the same physical node. That is what makes
 performance volatile not M/S vs HRD. HRD did solve the problem for writes
 somewhat but not for reads. Fundamentally if an app which runs on the same
 node is really active *your* app's performance will suffer. So it is dump
 luck. Since you got a new appid when migrating you got new neighbors so
 to speak - apps which share bigtable node with your app.


 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/7NEAU3WOrG8J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Sharp increase in instance count starting two days ago!

2012-03-16 Thread Waleed Abdulla

Two days ago, the number of instances assigned to my app started going up
considerably and surpassed 1200 instance at some point (see graph). The
number of billed instances went up as well, but not a significant hike like
the total non-billed instances. Anyone know why this is happening? I'm on
Py2.5  master/slave datastore.

Waleed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

attachment: high_appengine_instances.png

Re: [google-appengine] Need more backend instances!

2012-01-31 Thread Waleed Abdulla

Robert,
I don't have a link to the request now, but it was a form to request
quota increases. And, no, I don't have premier support.  Someone from the
App Engine team was kind enough to reach out to help, so I'm hoping I'll
get the extra instances I need soon.

Waleed




On Mon, Jan 30, 2012 at 10:11 PM, Robert Kluin robert.kl...@gmail.comwrote:

 Hey Waleed,
   How did you file the request?  Do you have premier support?



 Robert




 On Mon, Jan 30, 2012 at 19:56, Waleed Abdulla wal...@ninua.com wrote:

 We're hitting the limit of how many backend instances we can create
 (currently at 20x B4 instances). The limiting factor is that we're not
 allowed to have more than a total of 10GB of allocated memory. I submitted
 a request for a limit increase more than a month ago and haven't heard
 back.

 I like that GAE allows me to scale, but we're not able to scale now due
 to artificial limitations. Can anyone from Google help with this? app id is
 networkedblogs.

 Regards,
 Waleed



  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Need more backend instances!

2012-01-30 Thread Waleed Abdulla

We're hitting the limit of how many backend instances we can create
(currently at 20x B4 instances). The limiting factor is that we're not
allowed to have more than a total of 10GB of allocated memory. I submitted
a request for a limit increase more than a month ago and haven't heard
back.

I like that GAE allows me to scale, but we're not able to scale now due to
artificial limitations. Can anyone from Google help with this? app id is
networkedblogs.

Regards,
Waleed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: 1.6.2 Pre-Release SDKs Out

2012-01-26 Thread Waleed Abdulla

It would be super cool if the backup feature allows me to backup my data to
Google's Cloud SQL so it's a backup and, as the same time, a good place to
run reports that require the complex queries that SQL is good at.



On Wed, Jan 25, 2012 at 9:39 AM, someone1 someo...@gmail.com wrote:

 Strike that, I was looking in the wrong place. On a side note, will
 google.appengine.ext.ndb.utils.DEBUG be set to True or False in
 production?

 On Jan 25, 12:35 pm, someone1 someo...@gmail.com wrote:
  Not updating to the latest NDB? I know its experimental but it got
  some pertinent bug fixes and optimization since the last SDK release.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] speed up fetching of larger data set

2012-01-26 Thread Waleed Abdulla

You didn't mention if you're fetching the entities using their keys or with
a query. If you can fetch with keys, it's faster. And, also, increase your
batch size:

db.get([list of 1000 keys])

Another thing you might want to try is using async operations so you
trigger the fetches one after another without waiting, and then collect the
data as it comes.

Are you on master/slave or high-replication datastore? I haven't tested it
myself, but I hear that HRD helps in this case because it reduces the
occasional  extended delays when reading the datastore, and since you're
fetching 6000 entities, the odds of one of them getting stuck and taking
several seconds to load goes higher.





On Wed, Jan 18, 2012 at 10:26 AM, Andreas a.schmi...@gmail.com wrote:

 45 seconds to fetch 6k entities sounds a little weird to me.
 what do you do with those entities after fetching them?
 did you profile this operation with appstats?

 On Jan 18, 2012, at 1:17 PM, GordonHo wrote:

 hi,

 i have occasionally the problem that I've to fetch several thousands
 entities from the datastore.
 so far i tried to solve the speed issue by splitting the fetch into
 several chunks beeing fetched at the same time.

 however so far i am quite unsatisfied with the speed, fetching ~6000
 entities takes about 45seconds. that is using 200 a batch size for the
 fetch.

 has anyone some experience how to speed this up?

 cheers,

 gordon

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/7SkembCEJ9AJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Budget changes not taking effect!!

2012-01-21 Thread Waleed Abdulla

So I avoided a downtime, but the problem is still there. It's been about 6
hours now since I updated my budget, and I tried 2 more times after that,
but it's still not updating. Anyone else getting this?






On Fri, Jan 20, 2012 at 9:54 PM, Brandon Wirtz drak...@digerat.com wrote:

 Despite the implication that you can change at a moments notice, don’t
 ever count on it.  We set out daily budget to about 200x what we actually
 run at.  Why would you ever want to hit the limit and have your site go
 down?

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Waleed Abdulla
 *Sent:* Friday, January 20, 2012 9:51 PM
 *To:* google-appengine
 *Subject:* [google-appengine] Budget changes not taking effect!!

 ** **

 I ran a big mapreduce job today and now my app is close to reaching it's
 max daily budget, so I'm trying to increase the budget to avoid a downtime.
 I went to the billing settings page, changed the budget, went through the
 Google Checkout successfully, and it took me back to the billing settings
 page as usual. But the max daily limit didn't change, even after waiting
 for two hours.

 ** **

 Then, I tried again, and as usual went through all the steps and received
 a confirmation email from Google Checkout, and yet, the max daily limit
 doesn't change. I've been trying to raise my budget for the last 3 hours
 with no luck. What's going on?

 ** **

 At this moment, in an effort to keep the app running as long as possible
 before midnight, I stopped task queues, reduced max idle instances,
 increased pending latency, ...etc. But that's likely not going to be enough
 and I'll probably suffer a downtime. If that happens, does it typically get
 covered by the SLA? 

 ** **

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Budget changes not taking effect!!

2012-01-20 Thread Waleed Abdulla

I ran a big mapreduce job today and now my app is close to reaching it's
max daily budget, so I'm trying to increase the budget to avoid a downtime.
I went to the billing settings page, changed the budget, went through the
Google Checkout successfully, and it took me back to the billing settings
page as usual. But the max daily limit didn't change, even after waiting
for two hours.

Then, I tried again, and as usual went through all the steps and received a
confirmation email from Google Checkout, and yet, the max daily limit
doesn't change. I've been trying to raise my budget for the last 3 hours
with no luck. What's going on?

At this moment, in an effort to keep the app running as long as possible
before midnight, I stopped task queues, reduced max idle instances,
increased pending latency, ...etc. But that's likely not going to be enough
and I'll probably suffer a downtime. If that happens, does it typically get
covered by the SLA?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: SSL on Custom domains before 2014 ?

2012-01-17 Thread Waleed Abdulla

I filled the form a couple of months ago and haven't heard back!



On Tue, Jan 17, 2012 at 6:48 AM, objectuser kevin.k.le...@gmail.com wrote:

 Cayden,

 I signed up for the Trusted Tester Program over a week ago and have not
 heard back.  Is there something else I need to do?

 Thanks!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/62Hy8dLnvy4J.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: App Engine 1.6.1 Released!

2011-12-13 Thread Waleed Abdulla

A couple of questions:

1. Since not all my requests are the same, I wouldn't want to use bigger
front-end instances across the board. Are there plans to allow setting
instance type based on app.yaml routes?

2. Can you clarify this point? Will mapreduce not be supported on Python
2.7?

WARNING: Starting with the 1.6.2 release of the experimental Python
2.7 runtime, the mapreduce and
 datastore_admin builtins will not be supported.


Regards,
Waleed



On Tue, Dec 13, 2011 at 11:46 AM, Vivek Puri v...@vivekpuri.com wrote:

 Ikai,

 Looking at my Application setting, i see that the Frontend Instance
 Class is set to F2 by default. I didnt change it to F2. However the
 docs(http://code.google.com/appengine/docs/adminconsole/
 performancesettings.html#Setting_the_Frontend_Instance_Class) say that
 F1 is default . Let me know which statement is correct.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] How to catch Out of Memory errors?

2011-12-05 Thread Waleed Abdulla

I'm getting a lot of Out of Memory errors, and when that happens GAE just
kills the instance right away and all I see in the logs is something like
this:

Exceeded soft private memory limit with 277.09 MB after servicing 1
requests total


To find where the problem is, I put logging.info() statements in the
code, but the problem is that nothing is logged when the instance is killed
this way. I also tried using logservice.flush(), but that doesn't seem to
help. I cannot get any data out of that instance to help me debug the
problem.

Any tips?

Waleed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Why am I not allowed to reserve backend instance hours?

2011-12-05 Thread Waleed Abdulla

In billing, I can pre-commit to X instance hours for the week and I get a
discounted price, $0.05 per hour as opposed to $0.08. So far so good.

Recently, I've been moving my tasks and cron jobs to backend instances so
that there is a clean separation between what needs to execute right away
and what can wait. But then I realized that backend instances don't use the
pre-committed hours. So I end up paying $0.08 per hour for B1 backend
instances rather than $0.05.

It seems that backend instances are a bad deal, and I should move
things back to front-end instances to benefit from the discounted price! Am
I missing something?

Waleed

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to catch Out of Memory errors?

2011-12-05 Thread Waleed Abdulla

That's the problem, nothing I log gets recorded. It's like when the
instance is killed, everything is deleted, including any events that were
logged before that point.




On Mon, Dec 5, 2011 at 7:38 PM, Brandon Wirtz drak...@digerat.com wrote:

 Add your own events to put in the log, and they will be…

 ** **

 Instance started,

 Defined some variables

 Started a loop

 Looped

 Looped again count 1

 Looped again count 2-infinity

 Finished looping

 ** **

 Wrote something to datastore

 Output the HTML Life is happy

 ** **

 That Kind of stuff.

 Along the way you will likely find you have  a loop until the end of time
 statement somewhere

 ** **

 ** **

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Waleed Abdulla
 *Sent:* Monday, December 05, 2011 7:17 PM
 *To:* google-appengine
 *Subject:* [google-appengine] How to catch Out of Memory errors?

 ** **

 I'm getting a lot of Out of Memory errors, and when that happens GAE
 just kills the instance right away and all I see in the logs is something
 like this:

 ** **

 Exceeded soft private memory limit with 277.09 MB after servicing 1
 requests total

 ** **

 ** **

 To find where the problem is, I put logging.info() statements in the
 code, but the problem is that nothing is logged when the instance is killed
 this way. I also tried using logservice.flush(), but that doesn't seem to
 help. I cannot get any data out of that instance to help me debug the
 problem. 

 ** **

 Any tips?

 ** **

 Waleed

 ** **

 ** **

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Why am I not allowed to reserve backend instance hours?

2011-12-05 Thread Waleed Abdulla

I think you misunderstood the question. I'm not asking about the free
hours. I'm asking about the hours I pre-commit to pay for.





On Mon, Dec 5, 2011 at 7:39 PM, Brandon Wirtz drak...@digerat.com wrote:

 You get 28 free hours unless you are only getting a bill for backend hours
 does it matter if the free hours are for front or back?

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Waleed Abdulla
 *Sent:* Monday, December 05, 2011 7:28 PM
 *To:* google-appengine
 *Subject:* [google-appengine] Why am I not allowed to reserve backend
 instance hours?

 ** **

 In billing, I can pre-commit to X instance hours for the week and I get a
 discounted price, $0.05 per hour as opposed to $0.08. So far so good. 

 ** **

 Recently, I've been moving my tasks and cron jobs to backend instances so
 that there is a clean separation between what needs to execute right away
 and what can wait. But then I realized that backend instances don't use the
 pre-committed hours. So I end up paying $0.08 per hour for B1 backend
 instances rather than $0.05. 

 ** **

 It seems that backend instances are a bad deal, and I should move
 things back to front-end instances to benefit from the discounted price! Am
 I missing something?

 ** **

 Waleed

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime

2011-11-15 Thread Waleed Abdulla

I just switched one of my apps to Python 2.7 and the average latency went
up from 125ms to about 1.3 seconds (about 10x). My app is 'not' CPU-bound.
It has a mix of different use patterns (some requests use urlfetch, others
load data from datastore or memcache). That variety should make it a good
candidate to benefit from multi-threading, but that doesn't seem to be
working as expected. While the number of instances went down from 9
instances to 6, the 10X increase in latency is not acceptable.

I spent a lot of time investigating it, and finally changed threadsafe to
false and that fixed the problem (and, of course, raised my instance count
to it's original level). Like the original poster, I was hoping that
multithreading will reduce my instance cost, but that plan failed.

Am I doing something wrong? Or is multi-threading not ready for live
traffic yet?

Waleed
app id: bitpixelshr





On Tue, Nov 15, 2011 at 3:10 AM, Jeff Schnitzer j...@infohazard.org wrote:

 Dunno about the Python library, but the standard Java jBCrypt library uses
 a random salt per-user by default.

 I'm also very suspicious of this idea that the attacker doesn't have the
 source code.  Python is trivially easy to decompile.  Also, where do you
 keep your source code?  In Github?  Opens up a whole new set of attack
 vectors, including disgruntled Github employees.

 Jeff

 On Mon, Nov 14, 2011 at 10:52 PM, Brandon Wirtz drak...@digerat.comwrote:

 Nick,

 ** **

 I agree, that my threat model assumes they didn’t get my source code.
 That “Somebody else’s problem” works under the assumption people are going
 to get my data, not my source code because I don’t ever write my own DB
 server code I am stuck using someone else’s which means the vulnerability
 that I am most likely to face is that somebody else’s screw up will be
 where my problem lies.

 ** **

 Granted this is a better strategy if you are running compiled code, since
 my code lives on the Google Server I’m at the mercy of Google’s Security,
 where as if I were running compiled code it would be less likely someone
 would get the code.

 ** **

 I would say that unique salt per user, is a good thing.  The most common
 way to attack a large password database is to look at the most common
 entries and compare against the most common passwords from other sources.
 If you know the 15 most used passwords and the 15 most often occurring
 database results you are a long ways towards knowing what those 15 values
 are and calculating the salt.  You aren’t crunching millions of
 combinations you are crunching 1000’s and once you have the salt, you take
 your already deciphered list of the most common passwords and you calculate
 the top 5k using bcrypt and you now have about 50% of the data in fewer
 than 10k operations.

 ** **

 Compare that with my scenario.  You have data. You don’t have the source
 code. The UserID or other “spoiler” is in every salt so the reoccurrence of
 a hash doesn’t correspond to a duplicate password, and now the computation
 is nearly impossible even if you have the source code, because you have to
 calculate every value for every user anyway.

 ** **

 Would Brcypt(Pass+UserID+Salt) be the best?  Yes.  But
 MD5(Pass+UserID+Salt) is going to still going to be orders of magnitude
 more difficult than Bcrypt(Pass+salt), because I can’t use knowledge of
 frequency tables to predict likely outcomes or detect duplicate passwords.
 

 ** **

 -Brandon

 ** **

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson
 *Sent:* Monday, November 14, 2011 6:21 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Help resolve massive performance
 regression in 2.7 vs 2.5 runtime

 ** **

 Hi Brandon,

 ** **

 What you say is fine if your threat model only includes script kiddies
 who don't have your source code. If either of those is not true - you have
 an adversary with some level of independent skill, or your source code is
 compromised - any method that relies on obscurity for its security will
 fare very poorly.

 ** **

 One thing to bear in mind is that if your app is ever compromised, your
 password database and/or source may be posted publicly; at that point, you
 no longer have to worry about just the initial attacker, but anyone with
 sufficient motivation.

 ** **

 Of course, using federated login like OpenID or the Users API obviates
 the need to store passwords at all, making it Someone Else's Problem. :)*
 ***

 ** **

 -Nick

 On Tue, Nov 15, 2011 at 1:07 PM, Brandon Wirtz drak...@digerat.com
 wrote:

 If I know your salt, I can “De-Hash” bcrypts faster than I can any of the
 “weird” combinations. Because there are libraries for doing so on ATI cards.
 

  

 If you do something weird a script kiddie can’t just pull code off the
 web and attack it. 

  

 You want to see who can

Re: [google-appengine] new billing policy and fan-out

2011-11-09 Thread Waleed Abdulla

The math looks right, but it might be missing a few additional cost items.
I think you're assuming 2 writes for index updates (ascending and
descending). Most likely you'll also need a composite index that includes
the date so you can show the items in a reverse chronological order
(assuming your feed is like twitter). So I'd say the cost is $0.06 in index
write ops per update rather than $0.04.

Depending on your specific requirements, you might be able to find
optimizations over the general case that could save you a lot. A few ideas:

- Create your own indexes. Native indexes require 2 ops for each property +
1 op for each composite index. What if you create your own UserStream
entity that contains the latest 500 updates for that user? With every new
update, you fetch and update each UserStream entity. That's 1 get + 1 write
per update per user, as opposed to 3 writes per update per user. Not a huge
saving, but it also gives you the option to choose to not update all users
streams (i.e. ignore those who haven't logged in for a while).

- If you have a limit on how many sources a user can follow (say, 400 max),
then you can create an index entity for each source, SourceFeed, that
contains the latest, say, 100 updates from that source. Then, to generate
the stream of one user, you load the 400 entities with db.get([list]),
which loads them in parallel, and then generate the stream in memory. This
will work only if you have a small limit on how many sources a user can
follow. The cost here will be when the user views the stream rather than
when an update is made. The cost will be lower only if your users don't
login too often. Otherwise, the previous solutions would be better. Here,
you can also choose to store the SourceFeed of popular sources in memcache
to further reduce read ops.

- You can use a backend instance with a lot of memory to store big parts of
recent updates so that for most users you can generate the stream in
memory.

Waleed




On Tue, Nov 8, 2011 at 9:52 PM, Herbert herber...@gmail.com wrote:

 I'm building a system with news feed style broadcasting. there won't
 be millions of users, probably just 100k and interesting feed would go
 to inbox of ~10k users.

 everyone's recommend the method to solve fan-out proposed by Brett,
 and i really like it, works something like this:

 class Feed(db.Model)
   content = db.TextProperty()

 class Receivers(db.Model)
  IDs = db.ListProperty(int)

 say an interesting Feed has five children of Receivers each with 2000
 IDs in the list, i like how it solves my problem, but I did the math
 on billing:

 that's 2000 * 5 * 2 writes = 20k low-level writes right there on
 creation, and cleaning up cost another 20k low-level writes. 40k low-
 level writes = $0.04 for a news feed. Am I doing the math wrong,
 missing something, missing the best way to handle this problem, or
 this seems a reasonable cost?

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Early experimental release of Yaac! Which allows you execute query with aggregation functions / multiple inequality filters etc.

2011-10-31 Thread Waleed Abdulla

I really like this project and have been hoping someone would do it for a
long time. The need to run arbitrary reports on the datastore is clear, and
the current tools are a pain. So this is a great start. My feedback:

- It's still buggy. I ran a few queries and only some of them worked, and
the others just get stuck in processing.

- Definitely change the name. Yet another... makes it feel cheap.

- I don't like the concept of running it as a separate version. I prefer to
drop the code in a separate folder in my project and access it with it's
own path (e.g. /yaac), just like appstats and mapreduce.

- I would suggest focusing on EGQL and making it really solid. Remove the
other features (blobstore, memcache, taskqueues...etc) to avoid
distractions until EGQL is robust. Expanding the scope at this early stage
will only spread your efforts too thin and you end up with many half-baked
features that no one wants to use.

Good luck,
Waleed



On Mon, Oct 31, 2011 at 9:24 AM, Max thebb...@gmail.com wrote:

 As we all know we can not use join, aggregation functions, group by or
 multiple inequalities in datastore query. Google's argument is these
 relational feature are not scalable. But there has to be some use cases
 like ad hoc request to generate some report. We then have to prepare / test
 our own mapreduce / pipeline scripts and deploy to production environment,
 which is usually error prone and not trivial.

 Project Yaac(Yet Another Admin Console) is targeted to solve this common
 use case. Yaac is not a standalone java or python library, instead it's a
 complete application which is designed to be deployed to your own domain
 (non-default version of course). Yaac provides Extended GQL(EGQL), which
 supports aggregation function / group by / complex nested expressions /
 multiple inequality filters / *not*, *or* clause in where condition. For
 instance, in a football datastore, you can execute following EGQL query:

  select HOME_TEAM, AWAY_TEAM, COUNT(1), sum(FTHG  FTAG), sum(FTHG =
 FTAG), sum(FTHG  FTAG)
  from MATCH
  where HOME_TEAM = Arsenal
  group by HOME_TEAM, AWAY_TEAM
  having sum(FTHG  FTAG)  sum(FTHG = FTAG) + sum(FTHG  FTAG)

 More details visit this page:
 http://code.google.com/p/yaac/wiki/EGQLReference Because it's an early
 experimental verion, some part of the language may subject to change in the
 future.

 There is also a sandbox environment
 http://sandbox.yetanotheradminconsole.appspot.com/. Anyone with a Google
 account can login and play with Sandbox. Simply go to Datastore --
 Extended GQL then execute above query. I have uploaded 86000 football
 matches (all matches played in 10 leagues for last 18 seasons). For
 example, to retrieve all matches played in Arsenal's home against Man
 United after year 2000 and at least one teams scored in the game. We can
 execute following query:

 select HOME_TEAM, AWAY_TEAM, format(MATCH_DATE, MMM dd, ), FTHG,
 FTAG
 from MATCH
 where HOME_TEAM = Arsenal and AWAY_TEAM = Man United and MATCH_DATE 
 datetime(2101) and FTHG + FTAG  0

 (FTHG means full time home goal :) )

 It only tooks several steps to deploy Yaac to your own domain, check out
 http://code.google.com/p/yaac/wiki/DeployGuideline

 Again, it's a very very early experimental release, you may notice that
 the performance is extremely slow. This is because all queries are executed
 by pipeline API in a niave way, sequential scanning over whole datastore. I
 will spend more time to optimize it these days.

 Please try to set batch size to a smaller value if you are not performing
 query with aggregation / group by, otherwise it's very easy for a result
 message to hit channel service single message size limit (32KB). This will
 also be fixed soon.

 I will also try to provide a better error / exception messages when there
 is an error in query syntax or during query execution

 Any comments? Feedbacks? Feature requests are all welcome!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/bwwGQ0MGjJ0J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Announcing SSL for Custom Domains Trusted Tester Program

2011-10-20 Thread Waleed Abdulla

Which of the options allows using naked domain? I'm guessing VIP, right?

Waleed



On Thu, Oct 20, 2011 at 9:00 AM, Anton Novopashin antonev...@gmail.comwrote:

 I have filled form allready

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/V333YDgn0rIJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] New pricing scheme and concerns about abuse

2011-09-11 Thread Waleed Abdulla

You're describing what's known as a denial of service attack. These attacks
could happen to any Website on any platform. On GAE you're more protected,
out of the box, than most other hosting solutions.

1. GAE automatically blocks IPs that it senses strange activity from. I've
encountered this on two occasions when I tried to copy data from one of my
other servers and hit my GAE app too fast. The block is removed
automatically when activity goes back to normal.

2. You can block IPs from the dashboard. Look for the blacklist menu. It
also shows the top IPs from which traffic comes.

3. If you want to write your own code, you might find Brett's dos.py module
useful. Brett wrote this for the PubSubHubbub project, but you can take it
and use it in your project.

http://code.google.com/p/pubsubhubbub/source/browse/trunk/hub/dos.py


Waleed



On Sun, Sep 11, 2011 at 2:27 AM, Bay troels...@gmail.com wrote:

 Say someone does not like a person behind one of the small/medium sized
 apps on app store (lets say 2-3 instances continously each day). For the
 sake of argument, say that it is me, but it could just as easily apply to
 any of you guys out there.

 What will stop a person of ill intend to make his own app engine app (or
 use any other similar service available on the web) and use this app *to
 hit my small/medium sized app engine app with hundreds of asyncrone requests
 each second - forcing the number of instance of my application to skyhigh
 levels and thus - depending on budget settings - either 1) bill me hundreds
 of USD for a few hours, or 2) hitting the instance hour quota very, very
 quickly...*

 Such an attack could theoretically be made from another app engine app,
 possibly even under the free quota...

 Under the old pricing scheme the latency would just go up (it happened a
 few times to me).

 Possible solutions:
 1) app engine team should provide an API for blacklist, so that I can
 programatically add IP's there
 2) app engine team could make a setting to control Max Number of
 Instances [not just max idle instances - because they will not be idle].
 A Max Number of Instances would of course make high latencies for all users
 on my app during an attack, but at least I wouldnt have to deal with hitting
 the quota levels = readjusting pricing = waiting for new ressources to be
 allocated = spending lots of time on damagecontrol + making the attacker
 happy...
 3) any other ideas?

 I am concerned. Please advise.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/U_bUW2BUVj0J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Python library for uploading to AppEngine

2011-09-11 Thread Waleed Abdulla

I've done something close in the past where I have a GAE app that pulls
Python script from another location and executes it. Not sure if that helps
you, but here it is. Make sure the script is safe, though, because it'll be
running against your own datastore.


try:
path = ROOT_URL + 'script.py'
result = urllib2.urlopen(path)
except:
logging.error('Error pulling: ' + path)
sys.exit(0)

try:
code = result.read()
exec(code.replace('\r\n', '\n'))
except:
logging.error('Error in script')
sys.exit(0)





On Sun, Sep 11, 2011 at 12:46 AM, Emlyn emlynore...@gmail.com wrote:

 My google-fu is failing. Do you have any specific links to more info
 on uploading code using remote_api?

 On 11 September 2011 16:58, Tim Hoffman zutes...@gmail.com wrote:
  HI
  As robert says it  is just python. Also its using the remote_api.
  So all definately do-able.  I run remote_api stuff inside zope to
  transparently access the datastore from within zope.
  Rgds
  Tim
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/TKWNqEhYNuIJ.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 



 --
 Emlyn

 http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google
 Buzz posts,
 comments and all.
 http://point7.wordpress.com - My blog
 Find me on Facebook and Buzz

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Instances + Memcache + Users

2011-09-09 Thread Waleed Abdulla

There is a function, get_stats(), that gives you some data about your
memcache usage, including the total size of stored data. You might want to
run a test by starting as much as you can in memcache while watching the
total stored data and when that hits a ceiling, you'll know how much is
available to you. My guess, though, is that the value probably changes
because it wouldn't be efficient to set aside some memcache space for each
app whether it's used or not, so the memcache space is probably shared among
apps.

http://code.google.com/appengine/docs/python/memcache/functions.html




On Thu, Sep 8, 2011 at 11:09 PM, Robert Kluin robert.kl...@gmail.comwrote:

 I don't think they publish any specific amount.  You might want to do
 some basic testing to see.  Also, they've lowered the RAM on
 front-ends to 128mb.  I would not be surprised if they've also reduced
 the available memcahce space, so I wouldn't rely on any old
 information.



 Robert





 On Fri, Sep 9, 2011 at 00:56, jay kyburz@gmail.com wrote:
  do we know how big the memcache is? It would be good to know roughly
  how much data we can jam in there.
 
  On Sep 9, 3:26 pm, Robert Kluin robert.kl...@gmail.com wrote:
  Hi Jay,
Correct, memcache is global to your app.  The same user's requests
  could go to any of your instances.  So caching user datain local
  memory is probably not worth it.
 
  Robert
 
 
 
 
 
 
 
  On Thu, Sep 8, 2011 at 21:51, jay kyburz@gmail.com wrote:
   Oh, I just read that memcache is constant across instances.
 
   On Sep 9, 12:43 pm, jay kyburz@gmail.com wrote:
   Quick question,
 
   In my app users do work on some data. Every action they do I have
 to
   pull the data out of the datastore, unpickle it, do the transform,
   pickle it again and put it back in the datastore.
 
   Do today I'm doing some optimization where I'll look in the memcache
   first for the unpicked data.
 
   I was wondering if the user is sitting at their desk doing something
   every 5-10 seconds or so, and I have 5 or 6 instances running, are
   they likely to be hitting the same instance over and over, based on
   geography (or something else), or will they be making requests of a
   completely random one of my instances.
 
   Just wondering.
 
   Thanks in advance.
 
   --
   You received this message because you are subscribed to the Google
 Groups Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group athttp://
 groups.google.com/group/google-appengine?hl=en.
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Changing max idle instances affects task queues?

2011-09-08 Thread Waleed Abdulla

I checked the tasks and I don't see excessive retries. The tasks do
encounter error sometimes, but most of them are caught and handled inside
the tasks without a retry. The errors that cause a retry (response 500) are
very few.

The tasks do usually take more than 1 second (that's why I pushed that work
into tasks). Robert, what's your point about  1s tasks? Is there a penalty
on long tasks? And if so, what's the logic behind it?

So I increased the rate and bucket size, and I reduced the pending latency
back to 'automatic' and increased the max idle instance to 40. That helped
yesterday, but today I have the same problem. And it's with two queues this
time.




Many of these tasks are time sensitive and they need to run within a few
seconds. An occasional delay of a minute or so is okay, but it's getting
into the hours. Am I missing something obvious? Are there other settings
that control the rate of queue execution?

Waleed






On Wed, Sep 7, 2011 at 10:35 PM, Robert Kluin robert.kl...@gmail.comwrote:

 When I adjusted those settings my performance was abismal.  I like Vivek's
 suggestion of moving one-at-a-time.

 Most likely your requests in that queue are taking over 1000ms, so they
 aren't getting ran as quickly.  Hopefully that is something that will change
 with the new pricing.


 Robert


 On Wed, Sep 7, 2011 at 15:52, Waleed Abdulla wal...@ninua.com wrote:

 I noticed today that the performance of my task queue has
 dropped considerably (see image). The highlighted queue is set to run 8
 tasks/second (i.e. ~ 480 per minute) but it's running at 25% of that rate,
 and a backlog is building up. The only thing I changed recently is reducing
 my max idle instances to 20 and min pending latency to 100ms.

 Anyone else noticing something similar? any tips to get the task queue to
 run faster?



 [image: Screen shot 2011-09-07 at 1.26.22 PM.png]

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Screen shot 2011-09-07 at 1.26.22 PM.png

Re: [google-appengine] Re: Built-in indexes are costing me $200/day

2011-09-07 Thread Waleed Abdulla

Agree. My biggest cost, even more than instances, are datastore Ops, and the
biggest part of those are index writes (70M index writes vs 15M puts). In
light of the new changes, it's more reasonable to have indexing off by
default, and have it enabled only as explicitly requested.

Also, controlling indexes should be done in one place only, index.yaml. It's
often confusing what the effect of the indexed=False parameter is,
especially if you apply it later. Do you have old entities indexed and new
ones not? And what if you set it to True again? You'll probably have a gap
of unindexed entities. How do you fix that? Run a map-reduce job to
read/write the whole table. It's not as clean and manageable as having
everything in one index.yaml file.

Waleed

P.S. same posted on issue page as well.



On Tue, Sep 6, 2011 at 11:53 PM, renderpaz kev...@gmail.com wrote:

 Thanks for pointing that out, If you have any sort of frequent input
 to your application, this is a whale of an issue, just didn't notice
 until the billing change.

 Others, please vote for this issue, it can't be that much of a
 challenge to fix.

 On Sep 6, 11:34 pm, Robert Kluin robert.kl...@gmail.com wrote:
  I'd love this too. Star 4231:
   http://code.google.com/p/googleappengine/issues/detail?id=4231
 
  Robert
 
 
 
  On Wed, Sep 7, 2011 at 01:31, renderpaz kev...@gmail.com wrote:
   Hello,
 
   Let me preface this by saying that I don't believe the planned pricing
   changes will have a dramatic impact on my costs - in-fact it has
   motivated me to trim the fat and I'm already seeing large savings.
 
   That said, my biggest cost is datastore writes.  I completely
   understand that updating a composite index should cost a data store
   write, but the problem is 90% of my data store writes are coming from
   the built-in indexes on each property.  (I specifically have 6
   properties).
 
   I would happily turn of the built in indexes on many of these
   properties, but using indexed=False also prevents me from including
   that property in a composite index as well!
 
   In retrospect I could easily have hashed the 6 properties into 1
   indexable property - but that seems ridiculous as all I want to do is
   say, Don't create built in indexes for these, but allow me to include
   them in composite indexes
 
   MAIN POINT:
   Is there a reason it is not possible to turn off built in indexes?  It
   seems unfair that I cannot disable these indexes but am charged full
   price for them (and its  my actual writes)
 
   Thanks,
   Kevin
 
   Datastore quota info provided below:
   Datastore API Calls  0%  127,540,291 of Unlimited
  Okay
   Datastore Queries0%  517,774 of UnlimitedOkay
   Blobstore API Calls  0%  0 of Unlimited  Okay
   Total Stored Data10% 157.95 of 1,601.00 GBytes
   Okay
   Blobstore Stored Data0%  0.00 of 1,601.00 GBytes
   Okay
   Data Sent to Datastore API   0%  17.19 of Unlimited
 GBytes   Okay
   Data Received from Datastore API 0%  12.01 of
 Unlimited GBytes   Okay
   Datastore CPU Time   0%  1,187.75 of Unlimited CPU
 hours Okay
   Datastore Entity Fetch Ops   0%  75,875,047 of
 Unlimited Okay
   -
   Datastore Entity Put Ops 0%  35,815,086 of
 Unlimited Okay
   -
   Datastore Entity Delete Ops  0%  31,931 of Unlimited
   Okay
   -
   Datastore Index Write Ops0%   288,168,806
 ** of Unlimited
   Okay
   ---
   Datastore Query Ops  0%  517,617 of UnlimitedOkay
   Datastore Key Fetch Ops  0%  14,858,227 of Unlimited
   Okay
   Datastore Id Allocation Ops  0%  0 of Unlimited  Okay
   Number of Indexes2%  3 of 200Okay
   Prospective Search Subscriptions 0%  0 of 10,000
 
   --
   You received this message because you are subscribed to the Google
 Groups Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group athttp://
 groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App

Re: [google-appengine] Best practice to save entity avoiding duplicate

2011-09-07 Thread Waleed Abdulla

Works, unless you want to allow users to edit their entries. If a user
changes the address after submitting then the new key is now different and
you'll end up saving a second entity. I wouldn't use any user editable data
in the key. Instead, I'd add a separate column to include your hash.



On Tue, Sep 6, 2011 at 11:32 PM, Robert Kluin robert.kl...@gmail.comwrote:

 Hi Thomas,
  If you want to avoid duplicates, the hash solution is great.
 Generate the hash, then In a transaction try to fetch the entity by
 id.  If you can't get it, insert it.


 Robert






 On Tue, Sep 6, 2011 at 21:16, Thomas Wiradikusuma
 wiradikus...@gmail.com wrote:
  Hi guys,
 
  I'm working on something like AirBnB but much simpler. I want to hint
  user whenever he/she tries to submit duplicate entry.
 
  Currently, my model looks like:
 
  House {
  Id
  Username
  Address
  State
  Country
  }
 
  The id is made of address+state+country MD5. Is this the correct way
  to do it? Or, should I go with auto-generated Id and add extra field
  named 'hash', or how?
 
 
 
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Changing max idle instances affects task queues?

2011-09-07 Thread Waleed Abdulla

I noticed today that the performance of my task queue has
dropped considerably (see image). The highlighted queue is set to run 8
tasks/second (i.e. ~ 480 per minute) but it's running at 25% of that rate,
and a backlog is building up. The only thing I changed recently is reducing
my max idle instances to 20 and min pending latency to 100ms.

Anyone else noticing something similar? any tips to get the task queue to
run faster?



[image: Screen shot 2011-09-07 at 1.26.22 PM.png]

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Screen shot 2011-09-07 at 1.26.22 PM.png

Re: [google-appengine] Re: Porting App Engine app to other provider - ideas and discussion

2011-09-03 Thread Waleed Abdulla

Although a lot of us are upset about how this was handled, I'd suggest
testing how much you can optimize first. If that doesn't help, I have a
couple of suggestions below. Here is my own plan:

1. Start with optimizing settings. Things like reducing max idle instances
and lowering the rate on low priority task queues to reduce spikes.

2. Set a minimum instance commitment to take advantage of the lower cost for
committed hours.

3. Optimize the datastore. My biggest cost is in datastore ops (mostly
writes), so I'm looking at adding (indexed=False) to model properties that I
don't query on, and using memcache more often.

4. If the above gets me back to a reasonable cost, then I'm staying. If not,
then I'm looking at AppScale http://code.google.com/p/appscale/ and
TyphoonAE http://code.google.com/p/typhoonae/. Both are open source
solutions that allow me to run GAE apps on other platforms. They're not
battle tested, so I expect to have to pitch in with a few bug fixes, but I
know that they've ran some apps successfully (TyphoonAE says they've ran the
PubSubHubbub server successfully, which happens to be one of the apps I run
as well). In terms of server hosting, I've used RackSpace and Softlayer, and
they're both really good.


I hope this helps. I have a huge amount of code which makes re-writing my
apps almost impossible. If your code base is still small, you might have
other options as well.

Waleed


On Sat, Sep 3, 2011 at 6:06 AM, Raymond C. windz...@gmail.com wrote:

 Regarding hosting options, I am looking at rackspace since there are many
 successful businesses outside are using it.  Now looking at if I should
 switch to node.js or keep running python on the new server.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/yOYBC1t3a6kJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Data Transfer to AWS

2011-09-01 Thread Waleed Abdulla

Another tool I used in the past to move my data into GAE (it works both
ways) is App Rocket. It uses timestamps to do real-time replication between
GAE and mysql. You might need to tweak it a little bit to fit your need
exactly, but it gets you most of the way there.

https://github.com/k7d/approcket/wiki/



On Thu, Sep 1, 2011 at 10:15 AM, Daniel danielshaneup...@gmail.com wrote:

 We will be facing the same problem as we leave app engine. I believe
 I'm going to attempt to use the bulk downloader to get the data to a
 local computer. I am slightly worried about the cost of transferring
 11TB. I believe that the process make take a couple of days, so I plan
 on moving inbound records to the new database before I start the d/l,
 that way I can have reduced functionality over the data transfer
 period but no data will be lost.

 -Daniel

 On Sep 1, 8:22 am, Millisecond millisec...@gmail.com wrote:
  Won't rehash the pricing discussions, but because of the pricing
  changes and the way it's been handled we're 90% sure we're going to
  move to AWS.
 
  Anyways... now I'm tasked with figuring out how to get our almost TB
  of data over there either into SimpleDB or into an RDS instance, not
  sure yet.
 
  Have other people done the move, how did it go?  Did you pull from
  AWS / push from GAE?  Take an intermediate backup and then load from
  some other mechanism (thinking S3) into SDB / RDS?
 
  Our app is adding over 1 GB / day with 20MM reads and 20MM writes, and
  we'd rather not take it offline for too long...  Thinking of some
  crazy scheme based on descending keys to move data over from key A
  downwards, shut off app, move over everything from key A upwards
  as we're mostly only writing new data.  Maybe special-case a few
  object classes and use timestamps to detect deltas.  Realize it's
  pretty app-specific, but wondering if other people have tackled that
  same problem and what their experiences were.
 
  Thanks for any and all help,
  -Casey

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] A comparison of your current bill against the new pricing model is now available in the Billing History page.

2011-08-31 Thread Waleed Abdulla

It's 300% for me (3x what I pay now). Most of the cost is datastore writes
and reads, followed by instance hours.



On Wed, Aug 31, 2011 at 1:26 PM, pdknsk pdk...@googlemail.com wrote:

 It's an about 50% increase for me, because of datastore writes.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Increaded DeadlineExceeded / Timeout exceptions

2011-08-09 Thread Waleed Abdulla

Happening again right now and it's much worse. Even the dashboard fails 50%
of the time. Most of the requests I tested are failing 90% of the time!!



On Mon, Aug 8, 2011 at 8:43 AM, Will vocalster@gmail.com wrote:

 Same here. Console shows ~ 3% failure rate, sharply increased volumn
 of end user complains.

 Will

 On Mon, Aug 8, 2011 at 1:35 AM, Waleed Abdulla wal...@ninua.com wrote:
  Yes, that's been going on for weeks. Some times the error rate is so high
  the app is basically non-functional.
 
 
  On Fri, Aug 5, 2011 at 10:50 AM, Christopher cspeckrun@gmail.com
  wrote:
 
  Within the past week or so we've noticed an increased amount of
  DeadlineExceeded / Timeout Exceptions on all our apps (including ones
  with billing setup). The appengine status page shows there's never
  been a problem however.
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is there any outage at appengine.. please respond if you feel the same

2011-08-09 Thread Waleed Abdulla

I also experienced an outage on two of my apps about an hour ago. It started
with a considerable slowing down of requests, and then it went completely
down, including the app engine dashboard. 15 minutes later it came back
online, and it's still okay so far. During that outage period, the app
engine status page did not show anything and continued to report that
nothing was wrong.





On Mon, Aug 8, 2011 at 11:59 PM, Mike Johnston m...@simperium.com wrote:

 We're also experiencing a complete outage right now.

 On Aug 8, 11:54 pm, Sandeep Koduri sandeep.kod...@gmail.com wrote:
  there are some kind of errors showing up sudden unexpected error :::
  but the system status is not showing up any changes in status. any others
  facing similar problems please post/reply so that this will taken into
  consideration and fixed soon

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is there any outage at appengine.. please respond if you feel the same

2011-08-09 Thread Waleed Abdulla

Johan,
It's back to normal now. My app ids are networkedblogs and networkedhub.
I'm attaching my errors/second graph during the outage period.

Waleed





On Tue, Aug 9, 2011 at 12:22 AM, Johan Euphrosine pro...@google.com wrote:

 Please open a production issue with your appid:

 http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

 Thanks in advance.

 On Tue, Aug 9, 2011 at 8:59 AM, Mike Johnston m...@simperium.com wrote:
  We're also experiencing a complete outage right now.
 
  On Aug 8, 11:54 pm, Sandeep Koduri sandeep.kod...@gmail.com wrote:
  there are some kind of errors showing up sudden unexpected error :::
  but the system status is not showing up any changes in status. any
 others
  facing similar problems please post/reply so that this will taken into
  consideration and fixed soon
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 



 --
 Johan Euphrosine (proppy)
 Developer Programs Engineer
 Google Developer Relations

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

attachment: errors per second.png

Re: [google-appengine] Re: Is there any outage at appengine.. please respond if you feel the same

2011-08-09 Thread Waleed Abdulla

Johan,
Yes, the logs confirm what you said, that only networkedblogs was
impacted. But, during the outage, I tried to open the dashboard of the other
app, networkedhub, and it failed as well and I couldn't access it so I
thought it's the same issue. Maybe it was just a coincidence that I couldn't
access the other app's dashboard at the time.

I stared the issue and I'll try to add more details if I can.

Thanks,
Waleed




On Tue, Aug 9, 2011 at 12:59 AM, Johan Euphrosine pro...@google.com wrote:

 After taking a closer look it seems that only networkedblogs has been
 impacted.

 Do you confirm this ?

 FYI, I'm tracking this on the following production issue:
 http://code.google.com/p/googleappengine/issues/detail?id=5533

 Feel free to start or comment it if you want to get notified about updates.

 Thanks in advance.

 On Tue, Aug 9, 2011 at 9:33 AM, Waleed Abdulla wal...@ninua.com wrote:
  Johan,
  It's back to normal now. My app ids are networkedblogs and
 networkedhub.
  I'm attaching my errors/second graph during the outage period.
  Waleed
 
 
 
 
  On Tue, Aug 9, 2011 at 12:22 AM, Johan Euphrosine pro...@google.com
 wrote:
 
  Please open a production issue with your appid:
 
 
 http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue
 
  Thanks in advance.
 
  On Tue, Aug 9, 2011 at 8:59 AM, Mike Johnston m...@simperium.com
 wrote:
   We're also experiencing a complete outage right now.
  
   On Aug 8, 11:54 pm, Sandeep Koduri sandeep.kod...@gmail.com wrote:
   there are some kind of errors showing up sudden unexpected error :::
   but the system status is not showing up any changes in status. any
   others
   facing similar problems please post/reply so that this will taken
 into
   consideration and fixed soon
  
   --
   You received this message because you are subscribed to the Google
   Groups Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
   http://groups.google.com/group/google-appengine?hl=en.
  
  
 
 
 
  --
  Johan Euphrosine (proppy)
  Developer Programs Engineer
  Google Developer Relations
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 



 --
 Johan Euphrosine (proppy)
 Developer Programs Engineer
 Google Developer Relations

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Increaded DeadlineExceeded / Timeout exceptions

2011-08-08 Thread Waleed Abdulla

Yes, that's been going on for weeks. Some times the error rate is so high
the app is basically non-functional.



On Fri, Aug 5, 2011 at 10:50 AM, Christopher cspeckrun@gmail.comwrote:

 Within the past week or so we've noticed an increased amount of
 DeadlineExceeded / Timeout Exceptions on all our apps (including ones
 with billing setup). The appengine status page shows there's never
 been a problem however.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: 1.5.2 is out!

2011-07-21 Thread Waleed Abdulla

Will there be a max pending latency option later? My problem is usually
that the pending latency is too high.

Waleed



On Thu, Jul 21, 2011 at 3:37 PM, Brandon Donnelson
branflake2...@gmail.comwrote:

 Why is the eclipse repository always delayed about a day?

 Brandon Donnelson
 http://gwt-examples.googlecode.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/7FLaHDyVLdgJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] HardDeadlineExceptions after the maintenance

2011-07-16 Thread Waleed Abdulla

Thanks, Robert. Do you know if they treat these issues in an urgent manner?
When your app is down and your hair is on fire, you tend to need immediate
attention. The fact that we cannot do anything to bring our apps back up
when GAE goes down puts more responsibility on the GAE team to be responsive
and fix serious problems quickly.




On Sat, Jul 16, 2011 at 7:05 AM, Robert Kluin robert.kl...@gmail.comwrote:

 Hey Waleed,
  Posting to the list is always good, but for this type of thing I'd
 also submit a prod issue (and ideally include a link to it in your
 groups post).

 http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

 Robert




 On Friday, July 15, 2011, Waleed Abdulla wal...@ninua.com wrote:
  Anyone know of a way to alert the GAE team? The status page doesn't show
 any problems so it's possible that they don't know that there is a problem.
 
 
  On Thu, Jul 14, 2011 at 11:04 PM, Waleed Abdulla wal...@ninua.com
 wrote:
  Confirming that the issue on the GAE's side. Some requests take a very
 long time and then timeout. App is completely down now for several hours.
 
 
 
 
 
 
 
  On Thu, Jul 14, 2011 at 10:44 PM, Black Snapper ja...@jecahill.com
 wrote:
 
 
 
  My users are extremely angry as well. We are getting negative reviews in
 our app thanks to this outage.
 
 
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 
  To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/1rDpQ3I6_EYJ.
 
 
 
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 
 
 
 
 
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 

 --
 --
 Robert Kluin
 Ezox Systems, LLC

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: HardDeadlineExceptions after the maintenance

2011-07-15 Thread Waleed Abdulla

Confirming that the issue on the GAE's side. Some requests take a very long
time and then timeout. App is completely down now for several hours.





On Thu, Jul 14, 2011 at 10:44 PM, Black Snapper ja...@jecahill.com wrote:

 My users are extremely angry as well. We are getting negative reviews in
 our app thanks to this outage.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/1rDpQ3I6_EYJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: HardDeadlineExceptions after the maintenance

2011-07-15 Thread Waleed Abdulla

Anyone know of a way to alert the GAE team? The status page doesn't show any
problems so it's possible that they don't know that there is a problem.



On Thu, Jul 14, 2011 at 11:04 PM, Waleed Abdulla wal...@ninua.com wrote:

 Confirming that the issue on the GAE's side. Some requests take a very long
 time and then timeout. App is completely down now for several hours.





 On Thu, Jul 14, 2011 at 10:44 PM, Black Snapper ja...@jecahill.comwrote:

 My users are extremely angry as well. We are getting negative reviews in
 our app thanks to this outage.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/1rDpQ3I6_EYJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: HardDeadlineExceptions after the maintenance

2011-07-15 Thread Waleed Abdulla

Update. Our apps are back to normal now, and the dashboard is accessible
again.



On Thu, Jul 14, 2011 at 11:32 PM, Waleed Abdulla wal...@ninua.com wrote:

 Anyone know of a way to alert the GAE team? The status page doesn't show
 any problems so it's possible that they don't know that there is a problem.



 On Thu, Jul 14, 2011 at 11:04 PM, Waleed Abdulla wal...@ninua.com wrote:

 Confirming that the issue on the GAE's side. Some requests take a very
 long time and then timeout. App is completely down now for several hours.





 On Thu, Jul 14, 2011 at 10:44 PM, Black Snapper ja...@jecahill.comwrote:

 My users are extremely angry as well. We are getting negative reviews in
 our app thanks to this outage.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/1rDpQ3I6_EYJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.





-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: HardDeadlineExceptions after the maintenance

2011-07-14 Thread Waleed Abdulla

I'm noticing strange behavior as well. Cannot pinpoint it yet, but my app
has been up and down since the maintenance. Will post more if I manage to
pinpoint the issue.

Waleed



On Thu, Jul 14, 2011 at 9:30 PM, nischalshetty nischalshett...@gmail.comwrote:

 Is anyone from the appengine team looking into this? Now I'm getting emails
 from my users! And they are angry. The maintenance did not go well, at least
 for our app!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/XD4RkNs-ezUJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: My master/slave to high replication datastore migration experience

2011-07-13 Thread Waleed Abdulla

I agree with Joshua. That's a very bad experience to force us to go through
to migrate. My migration is probably going to be much worse because I have
a lot of data, which means that it'll take a very long time (days?) to copy
the data across, and by that time the data would have changed on the
original app. I cannot bring the app down for extended periods of time while
I do this.

Not to mention how much of a hassle it is to change app IDs.

I posted an issue here, star it if you agree that this process should be
automated:

http://code.google.com/p/googleappengine/issues/detail?id=5250





On Wed, Jul 13, 2011 at 6:41 PM, Greg g.fawc...@gmail.com wrote:

 On Jul 13, 1:05 am, Joshua Smith joshuaesm...@charter.net wrote:
  Frankly, I've never completely groked what an entity group maps to in my
 (30+ years of) programming experience. It's certainly not a parent/child
 relationship in the usual sense.

 If you figure it out, let me know! I have a app that needs to update
 two distinct entity types at the same time. Because they are distinct,
 I can't do both in a transaction. Even if it was only one entity type,
 I don't want to lock all of them when writing to one, which I believe
 is a consequence of making them a group.

 A good article called entity groups for dummies seems warranted!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: high latency

2011-07-06 Thread Waleed Abdulla

I noticed a spike in memcache latency earlier today as well. It's also now
showing on the GAE status page:

http://code.google.com/status/appengine/detail/serving/2011/07/06#ae-trust-detail-helloworld-get-latency





On Wed, Jul 6, 2011 at 1:17 PM, Felippe Bueno felippe.bu...@gmail.comwrote:

 Still happening,

 Now latency is about 1033 ms.

 I only do 'memcache.get/set/increment'.

 And I'm getting Error: Server Error when I try to see appstats.

 I'm getting 500 erros even when getting my *static scripts.

 That time the problem started with 800 req/sec.

 What could be happening ?

 *With static scripts I mean that there are no rpc calls.


 Thanks you.
 Felippe Bueno


 On Mon, Jul 4, 2011 at 8:07 PM, Felippe Bueno felippe.bu...@gmail.comwrote:

 Hi,

 Is this normal ? (attached)

 I can see that when app gets 1000+ req/sec the latency stays too high
 (goes from 90ms to 600ms).

 I don't know if the req/sec is the cause or the symptom of increase in
 latency.

 currently I have 1500 req/sec and 674.8 ms of latency.

 Thanks


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Do I really need to switch to HR?

2011-07-06 Thread Waleed Abdulla

Please star this issue if you agree that Google should make the migration
process easier rather than putting the burdon on the developers. After all,
most developers signed up to GAE to avoid having to deal with issues like
that.

http://code.google.com/p/googleappengine/issues/detail?id=5250





On Wed, Jul 6, 2011 at 8:39 AM, Robert Kluin robert.kl...@gmail.com wrote:

 Keys do contain the appid.

 One solution would be to adjust your code to catch the exception that
 gets thrown when accessing the old apps data, the recreate the key
 for the new appid.

 Robert



 On Wednesday, July 6, 2011, Joshua Smith joshuaesm...@charter.net wrote:
  Another thing that occurs to me is that anyplace I've used a key external
 to the application would have to be dealt with.  For example, one of our
 apps has an RSS feed that our managers use to keep track of whether
 customers have uploaded new information.  This looks like:
 
  http://myapp.com/rss?cust=DAFHP983RPFDSALFHDSKLFJHLSDKAFH
 
  (where the gobledegook is the datastore key)
 
  My managers would need to update all their bookmarks, since I believe
 that key has the name of the app pickled in it someplace, right?
 
  Is there any way to avoid having the keys change when I switch to HR?
 
  If not, is there a way that I could decorate database queries so that
 they quietly turn old keys into new ones?
 
  -Joshua
 
  On Jul 5, 2011, at 3:35 PM, Joshua Smith wrote:
 
  I have several apps and I've got this pesky to-do list item for all of
 them, to switch them over to the HR datastore.
 
  In preparation, I've read blog entries by people who have done this, and
 I'm still on the fence about whether it is worth the trouble.
 
  I get very few datastore timeouts in my apps, and periodic maintenance
 has never really been an issue.
 
  I've read about gotchas with eventual consistency, and it seems unlikely
 that I could convince myself that I'm safe from this for any reasonably
 complex app.  (I'm not concerned that I cannot reprogram the apps to fix the
 problem, but rather I'm concerned that I could not ensure there weren't
 going to be problems.  If it ain't broke...)
 
  I'm not clear on how I would go about getting google to grandfather my
 old 2000 emails quotas if I create a new -hr version of my app.
 
  I've also read that migrating uses TONS of CPU, and it is therefore
 going to cost me money to do.
 
  Really, the only reason TO do the switch is that I've read repeated
 admonitions from google that I *should* be using HR.
 
  What's the consensus here?  Is it worth the time, risk, and cost?
 
  -Joshua
 
  --
  You received this message because you are subscribed to the Google
 Groups Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 

 --
 --
 Robert Kluin
 Ezox Systems, LLC

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The price of Scalability

2011-07-05 Thread Waleed Abdulla

I'm running 3 apps on GAE with a total of ~150 instances.

I had my app in PHP/Mysql on dedicated servers at first, and as I needed to
scale, I moved to Python/GAE. I cannot do a fair comparison between the cost
on dedicated servers and the cost on GAE because I also added new features.
With that in mind, at the time of migration my GAE cost was about 4X what I
paid on dedicated servers. But it's felt it was worth it because I get
scaling, fault tolerance, and server admin for free. Overall, I felt that
GAE was not cheap, but a good deal and I recommended it to everyone and got
several other startups to use it based on my recommendation.

I do tend to agree with the post on many points, though. The datastore gets
slow and fails often (I'm on Master/Slave). The taskqueue also gets slow
periodically creating long backlogs. And even when it's not slow, it
generally runs tasks several seconds after their scheduled time. The price
is more expensive than regular hosting if your app grows. And if you
encounter a bug, it takes weeks of complaining on the mailing list before
someone from Google lends a helping hand. That's on the dark side, but it's
not all dark though.

On the bright side, GAE is a lot of fun to work with. It hides all the
boring details of managing servers and let's you focus on your product. And
although it fails more often than dedicated servers, it's usually for
shorter periods and it gets back online without requiring you to do
anything. And, obviously, you don't need to worry about scaling too much.

Overall, even with the mentioned issues, I still like GAE more than other
options. But the new pricing makes it more expensive than I can afford. I'm
looking for ways to optimize my code to reduce cost, and also looking at
other options such as TyphoonAE and AppScale. Even if I end up moving some
of my apps somewhere else, I'll probably continue to use GAE for smaller
projects simply because of the easy setup and no hassle hosting.

Waleed



On Tue, Jul 5, 2011 at 3:49 PM, Brandon Wirtz drak...@digerat.com wrote:

 Support for a system with 100 users and 100k users is about the same.  No
 one bills for support separately (Fine GAE bills $500) so the small guys
 pay
 more to cover support and administration.


 -Original Message-
 From: google-appengine@googlegroups.com
 [mailto:google-appengine@googlegroups.com] On Behalf Of zdravko
 Sent: Tuesday, July 05, 2011 3:32 PM
 To: Google App Engine
 Subject: [google-appengine] Re: The price of Scalability

 This price of scalability is an interesting animal.  Looking at AWS (for an
 example) transfer out bandwidth charges alone, the lowest and most
 expensive
 usage bracket is full 6x (SIX TIMES) more expensive then the highest and
 least expensive bracket.  Considering that it's just bandwidth and that it
 considers the least amount of effort to dissect and repackage into
 smaller
 doses, it is quite clear how much us little guys are subsidization the big
 boys.

 Was the whole idea of volume buying not based on a  premise of economies
 of
 scale ?  Using delivery of physical goods as an example, was it all not
 supposed to be based on the premise that delivering a truck load of
 something is lot more economical then delivering a single pallet or a
 single
 book of matches?  If so, then what is it about bandwidth that makes it SIX
 TIMES more expensive to deliver to us smaller guys?

 Are these sorts of pricing discounts in fact not the world's biggest price
 collusions and price fixing?  How can the little guy ever manage to compete
 when it has to forever keep on subsidizing the big boys?
 Where would the big conglomates ever be and what would really their bottom
 lines look like if they had to pay their fare share?

 When it comes to bandwidth, I can not image a price differential that is
 more than TWICE - between the smallest and the biggest bandwidth users?

 ARE WE THE ONES WHO ARE MAKING FINANCIALLY VIABLE USAGE MONSTERS SUCH AS
 WWW.YOUTUBE.COM ?!?



 On Jul 3, 8:27 am, Tony lpth...@gmail.com wrote:
  I just read this
  posthttp://www-cs-students.stanford.edu/~silver/gae.html
 
  The author seems unhappy with App Engine's offerings and have switched
  to EC2, saying when the app scales big, the cost is very high (he used
  the old Master/Slave datastore with high latency counted against CPU
 time).
 
  I myself like app engine a lot because from what I heard it offers
  good scalability without complex setup and maintenance. But I don't
  have much experience with it to say if the claims in the above post are
 true.
  Especially now there are changes in pricing, I'm afraid the costs may
  be driven a lot higher.
 
  Can anyone who is having a popular app on app engine give me your
  thoughts on the post? Do you see the new pricing scheme make scaling
  app a lot more expensive?

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to

Re: [google-appengine] Python VS Java

2011-07-02 Thread Waleed Abdulla

I'm curious why your Java version costs more currently! Can you share more
details? I would've expected the Java version to cost less since Java has
JIT compilation so the code should run faster.

As for your questions, I'd say that the price difference will likely be
small compared to the cost of your own time (unless the app gets a lot of
load and costs a lot). So I'd say choose the language in which you feel more
comfortable. Python tends to get new GAE features first, which is a slight
vote for Python.




On Sat, Jul 2, 2011 at 12:41 PM, Brandon Wirtz drak...@digerat.com wrote:

 I have kind of asked this before… 

 ** **

 I am trying to decide if I take the python of my app and expand it, or the
 version I ported to Java.  The Java version currently costs more to run,
 than the python version, but the new pricing would put the Java version as
 cheaper, at least until instances support multiple threads on python.  ***
 *

 ** **

 So do I build out Python? Or Build out Java? 

 ** **

 How long from new pricing to Mult-threaded Python?

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] What's the difference between static and cached requests in the dashboard?

2011-07-01 Thread Waleed Abdulla

Thanks, Ross. That explains it. Didn't know about the front-end cache.



On Wed, Jun 29, 2011 at 7:12 PM, Ross M Karchner r...@eventgrinder.comwrote:

 I'm _pretty_ sure cached means served by the front end cache:

 http://www.kyle-jensen.com/proxy-caching-on-google-appengine

 On Wed, Jun 29, 2011 at 6:35 PM, Waleed Abdulla wal...@ninua.com wrote:

  I noticed a big increase in requests today, and it looks like it's due to
 a sharp increase in cached requests (see image). I know that dynamic
 requests refer to those that reach a request handler, and static are those
 that are delivered through the CDN. Does anyone know what cached requests
 are? I couldn't find any documentation about it. Are they simply static
 requests that return 304 (not modified)?


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Google DNS hosting service

2011-06-30 Thread Waleed Abdulla

I starred your feature request. But since naked domains are on the roadmap
and since the only way to make them work for GAE is for Google to host the
DNS server, then I'm guessing they're already working on it.



On Thu, Jun 30, 2011 at 11:34 AM, Alexander Konovalenko
alex...@gmail.comwrote:

 Wouldn't it be nice to host your DNS zones on Google servers?

 Currently you basically have three options:

 1) Use whatever DNS servers your domain registrar or a hosting provider
 offer.
 2) Get a specialized DNS hosting service (such as DNS Made Easy or
 DynDNS).
 3) Run your own DNS servers.

 Personally I prefer to use a specialized DNS hosting because dealing
 with performance and reliability issues and DDoS attacks can be a pain,
 and I wouldn't trust my registrar or a random hosting provider to do
 that either.

 Now, if Google provided its own DNS hosting, it would become trivial to
 use naked domains with App Engine. We'd just say point this naked
 domain to that App Engine app and the DNS service would automatically
 figure out when to change the IP addresses.

 I filed a feature request detailing what makes a good DNS hosting service:
 http://code.google.com/p/googleappengine/issues/detail?id=5269
 If you have something to add, please comment there.

  -- Alexander

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: What's the difference between static and cached requests in the dashboard?

2011-06-30 Thread Waleed Abdulla

Along with the sharp increase in cached requests I'm also noticing a sharp
increase in Outgoing Bandwidth (see image). It contradicts my original guess
that cached requests are those that return code 304 because those don't
send much data, while I'm noticing a 300% increase in Bytes Sent per
second.

So the question remains, what are cached requests?

2011/6/30 Ronoaldo José de Lana Pereira rpere...@beneficiofacil.com.br

https://lh4.googleusercontent.com/-MW9liqU53YU/TgyJk0g8lEI/AA0/HWn3PiZYcL0/increased-request-per-second.png
I'm experiencing the same here. I'm having lots of instances and most of
the requests are cached requests too. Does it means that they wasn't showing
up before, or that we are actually having more traffic. Also, if they are
most cached requests, why do we need more instances to serve them?

attachment: Screen shot 2011-06-30 at 6.09.40 PM.png

[google-appengine] What's the difference between static and cached requests in the dashboard?

2011-06-29 Thread Waleed Abdulla

I noticed a big increase in requests today, and it looks like it's due to a
sharp increase in cached requests (see image). I know that dynamic
requests refer to those that reach a request handler, and static are those
that are delivered through the CDN. Does anyone know what cached requests
are? I couldn't find any documentation about it. Are they simply static
requests that return 304 (not modified)?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

attachment: Screen shot 2011-06-29 at 2.16.37 PM.png

Re: [google-appengine] Re: Updated App Engine Pricing FAQ!

2011-06-28 Thread Waleed Abdulla

Another point is support. Most hosting companies give you phone support if
you pay $100/month (or less). How come GAE charges $500 per month for it?
It's understandable not to offer it for free users, or even below a certain
pay limit. But how about customers who already pay thousands of dollars a
month in hosting fees? It's totally unreasonable not to include support with
that. It doesn't make business sense and it alienates your paying customers.






On Tue, Jun 28, 2011 at 12:37 PM, Branko Vukelic bg.bra...@gmail.comwrote:

 On Tue, Jun 28, 2011 at 9:00 PM, Will vocalster@gmail.com wrote:
  Speaking for myself, I'm paying more than $75 per month right now, and
 yes,
  I will bite the bullet and pay up. But the feeling is totally different.
  Before, I stay because I want to; now, I stay because I have to. Think
 about
  Microsoft, they charge ridiculous high amount of dollar for their
 operating
  systems while others offer alternatives for free. Why? Because they think
  they can, people will bite the bullet and stay. But when a viable choice
  comes, people flee. And now is the beginning of the end of their empire.
  Will Google follow the suit?

 Well, that's the thing. It's the What did you expect? factor.

 Did we seriously expect that this (otherwise great) platform will be
 cheap forever? Look, Google is giving away their own tech, so we can
 build sites at ridiculously low price! Not a chance. If you think
 rationally about it, there was no chance it could have been that way
 except maybe initially to attract developers and lock them into the
 platform. Nobody lied to us here. They just never claimed it'd be
 almost free forever.

 You are aware that there was a preview of the business edition[1]? It
 lists some of the features that people here were screaming for (SSL on
 custom domains anyone?). Well, it will come to the 'normal' GAE, but
 I'm guessing not before the normal GAE _becomes_ the business edition.

 [1] http://code.google.com/appengine/business/

 --
 Branko Vukelić
 bra...@herdhound.com

 Lead Developer
 Herd Hound (tm) - Travel that doesn't bite
 www.herdhound.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is AppEngine Down ??

2011-06-27 Thread Waleed Abdulla

Same here. A lot of writes are timing out for several hours now (m/s
datastore), even though the app engine status page doesn't show anything out
of the ordinary.





On Sun, Jun 26, 2011 at 10:43 PM, Robert Kluin robert.kl...@gmail.comwrote:

 Yeah also seeing huge spike in deadline exceeded errors on Master
 Slave apps right now.







 On Mon, Jun 27, 2011 at 01:39, Mike mickn...@gmail.com wrote:
  I am seeing this also, and getting user complaints... Anyone oncall
  reading this?
 
  On Jun 27, 3:32 pm, Atif Gulzar atif.gul...@gmail.com wrote:
  Hi,
 
  Many of our applications at GAE are facing deadline exceed errors?
 
  --
  Best Regards,
  Atif Gulzar
 
  I  Unicode, ɹɐzlnƃ ɟıʇɐ
 
  --
  You received this message because you are subscribed to the Google Groups
 Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Switching to HR datastore without migrating to new app id

2011-06-27 Thread Waleed Abdulla

I posted another feature request asking for the ability to switch to HR
without losing the data. It seem reasonable that the GAE team should provide
such option. Please star it:

http://code.google.com/p/googleappengine/issues/detail?id=5250




On Mon, Jun 27, 2011 at 7:33 AM, Heiko Roth r...@egotec.com wrote:

 ++1

 same here

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Where is the extra latency coming from?

2011-06-27 Thread Waleed Abdulla

Check to see if there is a pending_ms value in the logs. If so, then that
indicates that your request was put in a queue to wait for it's turn to run.
I had a long thread about a similar question a week or so ago (just search
for pending_ms in the archives to find it).

Waleed


On Mon, Jun 27, 2011 at 7:20 AM, Spines kwste...@gmail.com wrote:

 I'm using GAE/Java.  I record the time it takes for my code to execute
 inside the code and output it to the logs as a debug message.   The time it
 takes for my code to execute is usually just a fraction of the time that is
 shown on the logs as the total time to respond to a request.

 For example, in my code, my debug message will say that my code took 58ms
 to run.  But the log message will show: / 421ms 130cpu_ms 20api_cpu_ms,
 indicating that it took 421ms to return a response.  This is typical of
 every request, this is not due to loading requests.  Also, I am the only
 person accessing my app right now, so it's not because it's waiting for
 another request to finish processing first.

 What causes this extra latency?  Is there any way I can fix it?


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/y4Lka5msRooJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Updated App Engine Pricing FAQ!

2011-06-25 Thread Waleed Abdulla

On Sat, Jun 25, 2011 at 12:17 AM, Gregory D'alesandre gr...@google.comwrote:

 Interesting idea.  Could you use backends for non-urgent tasks?  Then the
 scheduler wouldn't need to be involved at all.


I could use resident backends, but then I have to adjust the number of
instances up and down as load fluctuates throughout the day. And if I use
dynamic backends, then the scheduler will get involved, right?  Also, it's
not only for backend work; there are cases where you need to adjust priority
per API. For example, I have user facing APIs that should be really fast.
And I have other APIs that I use to sync data between GAE and my other
servers. Those don't have to be fast, so I'd rather keep the cost lower.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Updated App Engine Pricing FAQ!

2011-06-25 Thread Waleed Abdulla

Thanks for the clarification, Greg. It sounds that the main motivation for
these changes is the realization that idle instances floating around cost
more than originally anticipated. The new pricing
model incentivizes developers to optimize their apps to reduce idle time. I
understand why this makes sense.

The issue, for me at least, is that I assumed (or wished) that GAE will
optimize everything in such a way that I don't need to optimize my own code.
If I do a urlfetch that takes 5 seconds, GAE will somehow utilize that idle
waiting time efficiently for other things. And I also don't need to use
threads because running work serially or in parallel costs the same because
GAE works at such a big scale that it can optimize things better than I can.
That's the vision I wished for, and it's what I feel I'm losing. But I
understand if the technology is not there to make this a reality yet. I can
work with that and adjust my wishful expectations accordingly.

I did back-of-the-envelope calculations based on the numbers you posted, and
it seems that I'm looking at an increase of $10-15K to my monthly bill.
That's a significant enough number to get me worried. The big chunk of the
cost for me is in datastore ops, so optimizing to reduce idle time won't
help me that much. I'm looking for things to optimize and indexes to
delete. It's not clear yet whether the new prices are too expensive or the
current prices are too cheap. It would great if you guys provide a
well-balanced price comparison to help us understand the value we're
getting.

Waleed





On Sat, Jun 25, 2011 at 12:24 AM, Gregory D'alesandre gr...@google.comwrote:

 On Sat, Jun 25, 2011 at 12:24 AM, Waleed Abdulla wal...@ninua.com wrote:

  A: Many customers have optimized for low CPU usage to keep bills low,
 but in turn are often using a large amount of memory (by having high latency
 applications).


 How does that work? Can you share an example?

 Also, instead of pushing the burden of supporting threading on to the
 developers, shouldn't GAE optimize resources behind the scenes? You're
 running many instances on every physical computer, so when one instance is
 idel, others should be using the CPU and other resources. If keeping an idle
 instance in memory costs too much, then adjust the pricing by adding a cost
 per idle second, or something along these lines.


 That is in essence what we've done, added a charge having an instance
 residing in memory, the difference is that we will not be billing the CPU
 separately anymore.



 My point is: GAE has a lot of limitations compared to other solutions, but
 it has one HUGE advantage: it's simple and scales automatically. That
 advantage is so big that it trumps all the other limitations (at least for
 me). Now that I have to manage my instances and rewrite my code to handle
 threading, I can't help but feel that I'm losing that one big advantage.


 Sorry that hear you feel that way Waleed.  But maybe there is a
 misunderstanding, you don't have to manage instances at all (in fact the
 only options we give you allow you to tune how we manage your instances).
  It still scales automatically.  In terms of rewriting code to handle
 threading, we suggest it because it means you can do more with less memory
 and it will save you money but if you'd rather not and are willing to pay
 for it, you don't have to.

 Hope that helps!

 Greg




 Waleed



 On Fri, Jun 24, 2011 at 4:55 PM, Waleed Abdulla wal...@ninua.com wrote:

 Gregory, Thanks for the update.

 I think the scheduler min  max pending latency should be per url (i.e.
 in app.yaml) rather than being global at the app level. Most apps have
 requests that require a quick response (UI and APIs), and others that don't
 (cron, some tasks, backend work, ..etc). I'd probably want to set my max
 pending latency to 50ms for UI requests and 5 seconds for some non-urgent
 tasks.

 Waleed


 On Fri, Jun 24, 2011 at 4:04 PM, Jeff Schnitzer j...@infohazard.orgwrote:

 Thanks, this clarifies much!  Questions below:

 On Thu, Jun 23, 2011 at 11:49 PM, Gregory D'alesandre gr...@google.com
  wrote:

  Datastore APIs Q: Which operations are being charged for?
 A: There are 3 categories of Datastore operations:
 - Write operations (Entity Put, Entity Delete, Index Write), each of
 these operations will cost $0.10 per 100k operations
 - Read operations (Query, Entity Fetch), each of these operations will
 cost $0.07 per 100k operations
 - Small operations (Key Fetch, Id Allocation), each of these operations
 will cost $0.01 per 100k operations

  Q: Under the new scheme, is it more economical to do a keys-only
 query that fetches 1000 keys, and then do a get on the 500 of them that I
 need, or just do a regular (non keys-only) query for all 1000 directly?
 A: The first is more economical.  Fetching 1000 keys + fetching 500
 entities = $0.0001 + 0.00035 = $0.00045; fetching 1000 entities = $0.0007.


 This makes sense, and encourages more use of memcache. to hold

Re: [google-appengine] What's the syntax to schedule a cron every 100 minutes?

2011-06-24 Thread Waleed Abdulla

Thanks Robert. I filed an issue here:

http://code.google.com/p/googleappengine/issues/detail?id=5243



On Fri, Jun 24, 2011 at 12:58 PM, Robert Kluin robert.kl...@gmail.comwrote:

 Looks like this might be a bug.  99 minutes works fine, but it doesn't
 seem to like three digit numbers.

 You might want to search the issue tracker for a similar issue, or
 file a bug report if you can't find one.


 Robert





 On Thu, Jun 23, 2011 at 00:39, Waleed Abdulla wal...@ninua.com wrote:
  I tried this:
 
  - description: My Cron Job
url: /work/cronjob
schedule: every 100 minutes
  And this:
  - description: My Cron Job
url: /work/cronjob
schedule: every 1.6 hours
 
  And both error out when I try to deploy, with the message:
  Error parsing yaml file:
  Unable to assign value 'every 100 minutes' to attribute 'schedule':
  schedule 'every 100 minutes' failed to parse: line 1:8 mismatched input
 u'0'
  expecting set None
in ./cron.yaml, line 43, column 13
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Updated App Engine Pricing FAQ!

2011-06-24 Thread Waleed Abdulla

Gregory, Thanks for the update.

I think the scheduler min  max pending latency should be per url (i.e. in
app.yaml) rather than being global at the app level. Most apps have requests
that require a quick response (UI and APIs), and others that don't (cron,
some tasks, backend work, ..etc). I'd probably want to set my max pending
latency to 50ms for UI requests and 5 seconds for some non-urgent tasks.

Waleed


On Fri, Jun 24, 2011 at 4:04 PM, Jeff Schnitzer j...@infohazard.org wrote:

 Thanks, this clarifies much!  Questions below:

 On Thu, Jun 23, 2011 at 11:49 PM, Gregory D'alesandre gr...@google.comwrote:

  Datastore APIs Q: Which operations are being charged for?
 A: There are 3 categories of Datastore operations:
 - Write operations (Entity Put, Entity Delete, Index Write), each of these
 operations will cost $0.10 per 100k operations
 - Read operations (Query, Entity Fetch), each of these operations will
 cost $0.07 per 100k operations
 - Small operations (Key Fetch, Id Allocation), each of these operations
 will cost $0.01 per 100k operations

  Q: Under the new scheme, is it more economical to do a keys-only query
 that fetches 1000 keys, and then do a get on the 500 of them that I need, or
 just do a regular (non keys-only) query for all 1000 directly?
 A: The first is more economical.  Fetching 1000 keys + fetching 500
 entities = $0.0001 + 0.00035 = $0.00045; fetching 1000 entities = $0.0007.


 This makes sense, and encourages more use of memcache. to hold entities.
  One question that I've been wondering a while - presuming no caching, does
 this query-keys+batch-get approach produce higher latency than a simple
 query, and if so, by how much?

 Also, is there any way we can get the transaction timestamp out on
 datastore writes?  This would *dramatically* improve the robustness of code
 that tries to keep memcache in sync with the datastore during contention.
  I've spoken with Alfred and Max about this, but I don't know if it's a
 priority.  This could potentially reduce datastore bills by orders of
 magnitude.

 Thanks,
 Jeff

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Updated App Engine Pricing FAQ!

2011-06-24 Thread Waleed Abdulla


 A: Many customers have optimized for low CPU usage to keep bills low, but
 in turn are often using a large amount of memory (by having high latency
 applications).


How does that work? Can you share an example?

Also, instead of pushing the burden of supporting threading on to the
developers, shouldn't GAE optimize resources behind the scenes? You're
running many instances on every physical computer, so when one instance is
idel, others should be using the CPU and other resources. If keeping an idle
instance in memory costs too much, then adjust the pricing by adding a cost
per idle second, or something along these lines.

My point is: GAE has a lot of limitations compared to other solutions, but
it has one HUGE advantage: it's simple and scales automatically. That
advantage is so big that it trumps all the other limitations (at least for
me). Now that I have to manage my instances and rewrite my code to handle
threading, I can't help but feel that I'm losing that one big advantage.


Waleed



On Fri, Jun 24, 2011 at 4:55 PM, Waleed Abdulla wal...@ninua.com wrote:

 Gregory, Thanks for the update.

 I think the scheduler min  max pending latency should be per url (i.e. in
 app.yaml) rather than being global at the app level. Most apps have requests
 that require a quick response (UI and APIs), and others that don't (cron,
 some tasks, backend work, ..etc). I'd probably want to set my max pending
 latency to 50ms for UI requests and 5 seconds for some non-urgent tasks.

 Waleed


 On Fri, Jun 24, 2011 at 4:04 PM, Jeff Schnitzer j...@infohazard.orgwrote:

 Thanks, this clarifies much!  Questions below:

 On Thu, Jun 23, 2011 at 11:49 PM, Gregory D'alesandre 
 gr...@google.comwrote:

  Datastore APIs Q: Which operations are being charged for?
 A: There are 3 categories of Datastore operations:
 - Write operations (Entity Put, Entity Delete, Index Write), each of
 these operations will cost $0.10 per 100k operations
 - Read operations (Query, Entity Fetch), each of these operations will
 cost $0.07 per 100k operations
 - Small operations (Key Fetch, Id Allocation), each of these operations
 will cost $0.01 per 100k operations

  Q: Under the new scheme, is it more economical to do a keys-only query
 that fetches 1000 keys, and then do a get on the 500 of them that I need, or
 just do a regular (non keys-only) query for all 1000 directly?
 A: The first is more economical.  Fetching 1000 keys + fetching 500
 entities = $0.0001 + 0.00035 = $0.00045; fetching 1000 entities = $0.0007.


 This makes sense, and encourages more use of memcache. to hold entities.
  One question that I've been wondering a while - presuming no caching, does
 this query-keys+batch-get approach produce higher latency than a simple
 query, and if so, by how much?

 Also, is there any way we can get the transaction timestamp out on
 datastore writes?  This would *dramatically* improve the robustness of code
 that tries to keep memcache in sync with the datastore during contention.
  I've spoken with Alfred and Max about this, but I don't know if it's a
 priority.  This could potentially reduce datastore bills by orders of
 magnitude.

 Thanks,
 Jeff

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Why a big difference between execution time and response time?

2011-06-23 Thread Waleed Abdulla

Thanks Tom,
I'm looking for other solutions as well. I tried a couple of ideas with
backends, but I don't like that I have to manage instances. I loved the
vision of GAE from the start on the promise that I only need to write my
code the right way and then never worry about scaling.

   Now I find myself spending a lot of time trying to trick GAE into
launching more instances to handle my load. No problem with scaling the
code, it's just that GAE doesn't want to give me enough power to get the
work done fast enough. I'm guessing they changed the scaling algorithm last
week.

I changed my code to run several chains of tasks in parallel. That seems
to help. The hard part is not knowing how the scaling algorithm works, so I
end up re-writing my code many times to figure out which arrangement of
tasks gets GAE to give me enough resources.

Waleed




On Tue, Jun 21, 2011 at 10:34 PM, Tom Phillips tphill0...@gmail.com wrote:

 Hi Waleed,

 Because I've found tasks to be sometimes unreliable and unpredictable
 performance wise, I've implemented a strategy of engaging backend
 instance(s) as alternative non-task-based job processors.

 Basically, I now have two job processing engines that work differently
 and together offer redundancy and better performance. One (the
 traditional one) uses tasks, and the other doesn't - and I run them
 simultaneously. If one isn't working well for some reason, then at
 least the other is getting work done. In my case reliability is more
 of an issue than achieving very high throughput, but this could also
 work to increase overall throughput.

 It may not work quite as well for python as it does for java where I
 can have multiple worker threads running on each backend. And it isn't
 necessarily cheap. But it can pick up some slack for the task
 infrastructure, and if the cost allows you could always add more
 backend instances. With threadsafe in java land, it is even better
 since the backend is more fully utilized with multiple worker threads
 working on jobs simultaneously.

 Basically the implementation is to have one or more backend worker
 threads/instances that loop eternally looking look for jobs in the DB,
 grabbing batches of them and running them directly (with a short
 configurable pause if none are found in a cycle). Meanwhile, the other
 task-based processor is being driven by a cron job, which every minute
 essentially runs the same find me jobs code the backends use.
 Instead of running the jobs directly in the frontend, this one
 dispatches them out as tasks using DB cursors and task chaining to
 handle larger numbers of waiting tasks.

 It can create some contention on job records in the DB, though there
 should be ways you could minimize that if it becomes excessive. And as
 long as you mark newly found jobs as assigned within the same
 transaction that you query them out, you can avoid accidentally
 running the same job twice. Careful if you're on high-replication -
 you may need to account for consistency limitations while querying and
 assigning jobs (I'm on M/S).

 /Tom



 On Jun 21, 8:36 pm, Waleed Abdulla wal...@ninua.com wrote:
  Thank you, Nick.
  I did check your post before asking and pending_ms was missing. Also,
  I'm noticing throttle_code=2 in the log of some tasks! It would be
 great
  to cover that one as well.
 
  I've spent the last 3 days trying everything I can think of to make
 my
  app scale up, but GAE is working against me. I make the tasks longer, and
  GAE starts running them less often. I make them shorter, and my queues
 start
  to have a backlog. GAE simply doesn't want to scale up for me (started 3
  days ago). I'm running out of ideas to try. Would appreciate your help.
 
  More details about what I tried here:
 
  http://groups.google.com/group/google-appengine/browse_thread/thread/...
 
  Regards,
  Waleed
 
  On Tue, Jun 21, 2011 at 5:04 PM, Nick Johnson (Google) 
 
  nick.john...@google.com wrote:
   Hi Waleed,
 
   You may find this blog post useful for explaining the various fields of
 log
   entries:
  http://blog.notdot.net/2011/06/Demystifying-the-App-Engine-request-logs
 
   pending_ms is the amount of time the request spent waiting on an app
 server
   to become available to execute the task.
 
   -Nick Johnson
 
   On Wed, Jun 22, 2011 at 8:45 AM, Waleed Abdulla wal...@ninua.com
 wrote:
 
   Good catch, thanks. I'm guessing that pending_ms is the time a request
   stays on hold before execution? I couldn't find it documented
 anywhere. If
   so, then the question is: why would a request be in a pending state?
 Is it
   waiting for an instance to be free?
 
   This might also be related to my other thread about the task queue not
   scaling up [1].
 
   Thanks,
   Waleed
 
   [1]
  http://groups.google.com/group/google-appengine/browse_thread/thread/.
 ..
 
   On Tue, Jun 21, 2011 at 3:33 PM, Nicholas Verne nve...@google.com
 wrote:
 
   One other note:
 
   In a large number of cases, if you

Re: [google-appengine] Running a task on a backend?

2011-06-22 Thread Waleed Abdulla

Thank you, Greg. That was the issue.

Waleed

On Tue, Jun 21, 2011 at 9:54 PM, Greg Darke (Google)
darke+goo...@google.com wrote:

Are you using the taskqueue labs api?

The taskqueue labs api (google.appengine.api.labs.taskqueue) was
deprecated in 1.4.0, and thus new features will no longer be added to
the old api.

To use new features such as pull queues and the target parameter you
will need to change your import statement to use the new api:

from google.appengine.api import taskqueue

On 22 June 2011 14:34, Waleed Abdulla wal...@ninua.com wrote:
I'm following the steps explained in [1] trying to have my tasks run on
backend instances. I created 3 backend instances and I can see them in
the
dashboard. However, when I run this code to insert a task, I'm getting an
error:

taskqueue.Task(
url='/work/smart_poll_bootstrap',
params=dict(sequence=name, poll_type=poll_type, end_time=end_time),
target='bootstrap',
).add(POLLING_QUEUE, transactional=True)

The error I'm getting is:
TypeError: Invalid arguments: target

When I remove the target parameter, it works (but obviously doesn't
go
to the backend instances). I also tried values of 1.bootstrap and got
the
same error. This is my backends.yaml file:
backends:
- name: bootstrap
class: B1
instances: 3
options: dynamic

Anyone see a problem in the code above? This is failing in both, dev
and
production.
Thanks,
Waleed
[1]
http://code.google.com/appengine/docs/python/taskqueue/overview-push.html#Push_Queues_and_Backends

Re: [google-appengine] Simple query times out repeatedly for hours!

2011-06-22 Thread Waleed Abdulla

I'm afraid this issue is back again. This is how I reproduce it (app id:
networkedhub):

1. I go to the datastore viewer in the dashboard and enter this query:

*SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0) order
by eta, polling_period desc*

2. I hit run query and I get 500 error.

3. The query also fails in my code with a timeout error. Repeatedly. I had
the task run 60 times and they all failed. This query is used in a task
chain in my app. When it fails, the app stops completely.


This is how I've been solving it:

1. Originally the query was like this (notice there is no mention of
polling_period):

*SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0) order
by eta*

2. When it started failing with timeout errors a month ago and I couldn't
figure out why, I created a new index. I added polling_period to the index,
which I don't need in my query, to make sure GAE creates a new index:

*# A hack to get around datastore timeouts*
*- kind: KnownFeed*
*  properties:*
*  - name: eta*
*  - name: polling_period*

Then I changed my query to this (adding order by polling_period forces GAE
to use the new index):

*SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0) order
by eta, polling_period*
*
*

3. Two weeks later, timeouts started happening again with the new index. So
I had to create another one (added 'desc' order):

*- kind: KnownFeed*
*  properties:*
*  - name: eta*
*  - name: polling_period*
*direction: desc*

and changed my query to this:

*SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0) order
by eta, polling_period desc*
*
*
*
*
4. Two weeks later (now), the timeouts are happening again!


It looks like an index is good for two weeks and then it starts failing. Any
ideas?


Waleed




On Sun, May 29, 2011 at 8:02 PM, Waleed Abdulla wal...@ninua.com wrote:

 Yes, I don't think the writes are the problem either. The error happens on
 the SELECT, and it happens when there is no load. Also, it's not a code
 error because:

 1. It's been working for a year without problems and only started breaking
 two weeks ago with no change on my side
 2. Running the simple SELECT query from the Datastore Viewer times out as
 well.

 The hack that I came up with to create a second index seems to be helping
 (I described it in this thread a few emails ago). With the new index, the
 query works without problems some days and only times out about 30 to 40
 times on other days (as opposed to timing out hundreds of times before the
 new hack). I don't know how long this will keep working, though.

 But here is the scary thought: if I didn't find that hack, then my app
 would be broken for two weeks now and no one from Google has commented or
 looked into it.





 On Fri, May 27, 2011 at 10:43 AM, Robert Kluin robert.kl...@gmail.comwrote:

 Hey Waleed,
  I doubt 45writes / second  would cause any issues. I've sustained
 higher rates without problems, even with indexed datetimes.  I'm sure
 you've checked it carefully, but have you investigated the logic
 surrounding the cursor?  Maybe there is some subtle bug that is
 resulting in a bug on the first run?

  Hopefully Ikai might be able to chime in a give some further
 guidance.  This seems to be a strange issue, I'd very much like to
 know the resolution to this.


 Robert





 On Fri, May 27, 2011 at 03:00, Waleed Abdulla wal...@ninua.com wrote:
  Entities are added at a slow pace: around 500 or so new entities a day,
 and
  evenly distributed though out the day. So I don't think that's the
 issue.
  Updates, on the other hand are much more often: around 45/second. I
 update
  the eta property to the time at which this entity needs to be
 processed
  again (each entity represents a blog feed that my system pulls often).
 And I
  have tasks that pull the entities that have an eta  now and process
 them.
  The task pulls 50 entities at a time and then inserts a new task to
 continue
  the work (a chain of tasks). The first task runs the query I mentioned
  earlier, and then it passes the cursor to the next. Only the query of
 the
  first time is timing out, but once that works, the following tasks that
 use
  the cursor work without problems.
  Entities use a key_name that is a hash of the feed URL, so they're
 evenly
  distributed on disk. The index on the eta column, on the other hand,
 is
  probably not evenly distributed on disk. However, if the problem was due
 to
  a hot tablet, then I'd expect the issue to happen while updating the
 eta
  value while processing each entity. But that doesn't happen. All updates
  work without problems.
  When the first task in the chain runs the query mentioned earlier and it
  times out, then it doesn't insert the next task. And that means once
 that
  first task fails, the whole system stops. The task gets retried until it
  succeeds, which might take 20+ attempts. And due to the exponential
 back-off
  of the task queue, that usually takes hours

Re: [google-appengine] Re: Google blocks traffic to appengine application

2011-06-22 Thread Waleed Abdulla

Martino,
I've had the same issue you had a few months ago, on two occasions. I
have an external php server that calls my GAE server a lot to get data. One
day, without notice, GAE started blocking all requests coming from my php
server. The app works fine when I hit GAE from a different ip address. It
was only the IP address of my php server that was blocked.

I had to bring my website down for a little while (5 minutes or so) so
that my php server stops calling GAE. After that I tried again and it
worked. It's not clear what GAE considers a DoS attack, and there is no way
that I know of to add certain IP addresses to a while list (there is a black
list, though).

One that I had to do was to prevent my php server from re-trying if
requests to GAE fail. I suspect what might've happened was that GAE slowed
down (happens from time to time), and then requests started timing out, so
that caused my php server to keep retrying, and then GAE decided that my php
server was doing a DoS attack.

Someone on this list suggested that if I make all requests to GAE with
an authenticated admin user then that should solve the issue. Maybe. I
didn't try it. But it seems that the issue happens before the request hits
GAE, and that's why you don't see anything in your logs.

Waleed




On Wed, Jun 22, 2011 at 10:03 AM, Brandon Wirtz drak...@digerat.com wrote:

 As an SEO, not as an app developer.   I have billed more money to clients
 fixing issues caused by cloudflare than the client ever hoped to gain by
 using it.  When it screws up and serves a search bot with a captcha page
 your world ends.

 ** **

 As an App Developer, Google Edge Cache built in to Appengine does 70% of
 what CloudFlare does for scaling.  And just blacklisting bad bot IP’s and
 UserAgents will get you 25% of the way.  

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Francois MASUREL
 *Sent:* Wednesday, June 22, 2011 5:32 AM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Re: Google blocks traffic to appengine
 application

 ** **

 I'm actually using CloudFlare in front of AppEngine for some low traffic
 websites (ex: www.filhot.com) and, crossing fingers, everything seems to
 be going fine.  But still very interested in any information about this
 potential problem.

 ** **

 Thanx in advance.

 ** **

 Francois

 ** **

 On Wed, Jun 22, 2011 at 14:04, Alexander Konovalenko alex...@gmail.com
 wrote:

 On Fri, Jun 10, 2011, Martino A. Sabia ezu...@gmail.com wrote:
 [...]
  So i think that what happened was what your friend says on Google, they
 have
  some sort of filter in front of GAE. Do you have some suggestions on how
 i
  can submit my issue to google somehow - with some kind of probability to
 be
  read -?

 Martino, you could collect as much debugging information as possible
 when this happens again and submit a Production issue to the App
 Engine issue tracker. I guess the following information will help:

 1) the request URL
 2) the client's IP address
 3) the exact time when the error occurred
 4) the full error message or a screenshot
 5) mention that the request came through CloudFlare
 6) the outgoing CloudFlare IP address for that specific request
 (extracted from CloudFlare logs, perhaps with the help of their
 customer support)

 Please post to the group if you find out anything new about what can
 cause the error on the Google side and whether it can be related to
 CloudFlare. I'm interested in the details, too, because I'm planning
 to use a reverse proxy in front of App Engine.

  -- Alexander

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from

Re: [google-appengine] Simple query times out repeatedly for hours!

2011-06-22 Thread Waleed Abdulla

Thank you, Robert and Alfred.
That explains it. I'm already using taskqueue and cursors, and I use
key_names that are evenly distributed to avoid write contentions. But I
didn't account for the effect of soft deletes on the index. Now that I
know about this property of indexes, I'll design around it.

Much appreciated. I can finally sleep at night.

Waleed




On Wed, Jun 22, 2011 at 1:54 PM, Alfred Fuller 
arfuller+appeng...@google.com wrote:

 Ya, this sounds exactly like a 'churn' issue. See:
 http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/

 You don't have to delete an entity to cause this. Changing the ETA for all
 the values a the start of your query will delete the index values at the
 start of the query and add new values else where. For bigtable (which the
 datastore is built on) this is a 'soft' delete and must be skipped over for
 every scan until a 'compaction' happens. There are several options to solve
 this:

1. Use the task queue. This is the problem it was built to solve.
2. Use cursors. The problem comes from trying to find the first value.
Using cursors will cause the datastore to jump directly to that position. 
 If
you are worried about orphaning writes that insert values before the cursor
have another process running the raw query every so often to catch them (or
keep a minimum eta in memcache, or a maximum eta to eliminate clock skew
issues, etc)
3. Shard the ETA index so you can have several processers running not
reading/removing the same index values or locations

  - Alfred

 On Wed, Jun 22, 2011 at 1:28 PM, Robert Kluin robert.kl...@gmail.comwrote:

 Hey Waleed,
  Your issue sounds very similar to what happens when you do lots of
 deletes: old entities get scanned over in the index so as you delete
 more you have to do more work to find good entities.  Your not using
 entity groups or transactions when doing the update, right?  As I
 recall, indexes are not updated synchronously.  It sounds like maybe
 the indexes are lagging, resulting in the query timing out.

  It sounds very strange, maybe Alfred has some other ideas about what
 might be causing this.


 Robert






 On Wed, Jun 22, 2011 at 13:48, Waleed Abdulla wal...@ninua.com wrote:
  I'm afraid this issue is back again. This is how I reproduce it (app id:
  networkedhub):
  1. I go to the datastore viewer in the dashboard and enter this query:
  SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0)
 order
  by eta, polling_period desc
  2. I hit run query and I get 500 error.
  3. The query also fails in my code with a timeout error. Repeatedly. I
 had
  the task run 60 times and they all failed. This query is used in a task
  chain in my app. When it fails, the app stops completely.
 
  This is how I've been solving it:
  1. Originally the query was like this (notice there is no mention of
  polling_period):
  SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0)
 order
  by eta
  2. When it started failing with timeout errors a month ago and I
 couldn't
  figure out why, I created a new index. I added polling_period to the
 index,
  which I don't need in my query, to make sure GAE creates a new index:
  # A hack to get around datastore timeouts
  - kind: KnownFeed
properties:
- name: eta
- name: polling_period
  Then I changed my query to this (adding order by polling_period forces
 GAE
  to use the new index):
  SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0)
 order
  by eta, polling_period
 
  3. Two weeks later, timeouts started happening again with the new index.
 So
  I had to create another one (added 'desc' order):
  - kind: KnownFeed
properties:
- name: eta
- name: polling_period
  direction: desc
  and changed my query to this:
  SELECT * FROM KnownFeed where eta  datetime(2011, 06, 22, 17, 20, 0)
 order
  by eta, polling_period desc
 
  4. Two weeks later (now), the timeouts are happening again!
 
  It looks like an index is good for two weeks and then it starts failing.
 Any
  ideas?
 
  Waleed
 
 
 
  On Sun, May 29, 2011 at 8:02 PM, Waleed Abdulla wal...@ninua.com
 wrote:
 
  Yes, I don't think the writes are the problem either. The error happens
 on
  the SELECT, and it happens when there is no load. Also, it's not a code
  error because:
  1. It's been working for a year without problems and only started
 breaking
  two weeks ago with no change on my side
  2. Running the simple SELECT query from the Datastore Viewer times out
 as
  well.
  The hack that I came up with to create a second index seems to be
 helping
  (I described it in this thread a few emails ago). With the new index,
 the
  query works without problems some days and only times out about 30 to
 40
  times on other days (as opposed to timing out hundreds of times before
 the
  new hack). I don't know how long this will keep working, though.
  But here is the scary

[google-appengine] What's the syntax to schedule a cron every 100 minutes?

2011-06-22 Thread Waleed Abdulla

I tried this:

*- description: My Cron Job*
*  url: /work/cronjob*
*  schedule: every 100 minutes*

And this:

*- description: My Cron Job*
*  url: /work/cronjob*
*  schedule: every 1.6 hours*


And both error out when I try to deploy, with the message:

*Error parsing yaml file:*
*Unable to assign value 'every 100 minutes' to attribute 'schedule':*
*schedule 'every 100 minutes' failed to parse: line 1:8 mismatched input
u'0' expecting set None*
*  in ./cron.yaml, line 43, column 13*

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Requests hanging when using Chrome and development server (python)

2011-06-21 Thread Waleed Abdulla

Same here. I noticed it a few months ago with Chrome on Mac, and since then
switched to Firefox for my local development. It's odd.



On Tue, Jun 21, 2011 at 10:38 AM, JH ja...@mhztech.com wrote:

 I've noticed this as well with Chrome.  No idea why it happens though.

 On Jun 21, 12:28 pm, kwellman kenji.well...@gmail.com wrote:
  Hello,
  I'm using Google App Engine with the Flask web framework on Ubuntu and
  I've had this annoying problem with Chrome (Windows) for the past
  month. When I'm testing my program by running the GAE development
  server, my requests often hang and the page that I'm requesting will
  never load. In order to get it working again I have to switch to the
  development server process and hit Ctrl+C once or twice and then the
  page will finish loading fine.
 
  I've never had this problem with IE8 or Firefox 4, and it was working
  fine with Chrome until about a month or so ago. I prefer Chrome,
  especially for development, and would like to use it but this bug
  has driven to Firefox for now.
 
  Has anyone else experienced this problem?
 
  Thanks.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Task queue not scaling up!

2011-06-21 Thread Waleed Abdulla

So far, doing less work per task has been helping in getting the app to do
more work faster. But it's still not fast enough. I still see tasks execute
at a lower rate that I like them to. It seems to fluctuate through out the
day. Sometimes I'm running 140 tasks per minute (which is good), and other
times it runs 10 tasks per minute (very bad).

Any other tips to help get GAE run my tasks faster? This all started a few
days ago, and before that all was good and work was done really fast. Anyone
know if something changed in GAE a few days ago?





On Mon, Jun 20, 2011 at 11:33 AM, stevep prosse...@gmail.com wrote:

 It is vitally important that developers get a better sense of what
 affects task queue delays**. TQs are extremely important IMHO for
 maintaining low latency of the on-line handler. Yet there *must* be
 some predictability regarding task delays for at least one queue. If
 this already exists, point me to the docs Please!

 cheers,
 stevep

 ** - I've given up any hope of seeing a separate, high-reliability,
 low-volume queue. Now just hoping to gain some specifics from Google
 about how to predict TQ performance. Until then, we have Robert who I
 can't thank enough for his on-going help in these forums.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Why a big difference between execution time and response time?

2011-06-21 Thread Waleed Abdulla

Good catch, thanks. I'm guessing that pending_ms is the time a request stays
on hold before execution? I couldn't find it documented anywhere. If so,
then the question is: why would a request be in a pending state? Is it
waiting for an instance to be free?

This might also be related to my other thread about the task queue not
scaling up [1].

Thanks,
Waleed

[1]
http://groups.google.com/group/google-appengine/browse_thread/thread/f8d1c6f39ad2af27/c989b852d1011cd6

On Tue, Jun 21, 2011 at 3:33 PM, Nicholas Verne nve...@google.com wrote:

One other note:

In a large number of cases, if you the pending_ms (when it exists) to
your measurement, you come close to the ms value you circled at the
top.

Nick Verne

On Wed, Jun 22, 2011 at 8:18 AM, Nicholas Verne nve...@google.com wrote:
Waleed,

I'm going to have to ask one of the other infrastructure engineers
about the log lines. The value you have circled (18386ms) is not
always greater than the cpu_ms. I'm not sure what this value
indicates.

Nick Verne

On Wed, Jun 22, 2011 at 6:27 AM, Waleed Abdulla wal...@ninua.com
wrote:
Nicholas,
I implemented the change you suggested (making task_t0) a global
variable, but that didn't change the calculations. I'm attaching another
screen shot. There is a big difference between the actual processing
time
and the time reported in the logs. It doesn't happen in all tasks, but
it's
often enough that it's slowing down my application considerably and
causing
a lot of user complaints.
I also calculated the total time to load the module (including time
for
imports). It doesn't seem to take much time (sub 1 second), and it
happens
only once right after I deploy a new version. So that is not an issue.
I've set the processing rate, bucket size, and max concurrency on
the
task queue to high numbers (much higher than I need).
Any explanation for why this is happening?
Waleed

On Mon, Jun 20, 2011 at 6:54 PM, Brandon Wirtz drak...@digerat.com
wrote:

Use a fetch from one thread to another.

Echo Time
Fetch
Echo Time
Do something
Echo Time
Echo Fetch

Now you have a timeline of everything except the latency from User to
Appengine.

-Original Message-
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Nicholas Verne
Sent: Monday, June 20, 2011 6:03 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Why a big difference between execution
time
and response time?

Yes, that's true. If you try this:

task_t0 = None

...

def main(argv):
global task_t0
task_t0 = time.time()

You'll get a measurement per request as early as we can make it. For
warming
requests, it might still be worth measuring the import time

import time
import_start = time.time()
... other imports
import_duration = time.time() - import_start

The import_duration global should persist and tell you how long this
instance's imports took to process. Subsequent requests on that
instance
don't do the imports.

Nick Verne

On Tue, Jun 21, 2011 at 10:49 AM, Waleed Abdulla wal...@ninua.com
wrote:
A more useful measurement might be to set task_t0 as a global
variable at the top of your file, with only the import of time
preceding it

Wouldn't that cause the 2nd, 3rd, ...nth requests to the same handler
to show wrong values? I see why setting task_t0 at the top of the
file
would apply if it was the first request that causes the handler to be
loaded. But this is happening on an ongoing basis for all requests,
not only the first for that handler.

--
You received this message because you are subscribed to the Google
Groups Google App Engine group.
To post to this group, send email to
google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group

1 2 >

1 - 100 of 172 matches

Mail list logo