Re: [google-appengine] Re: too many appengine frontend instances

[Eric Ka Ka Ng]

we are with 200 frontend instances too.. the overall performance is
very slow, and the cost becomes huge


On 9 May 2014 17:24, Andrei Cosmin Fifiiţă andrei.fifi...@gmail.com wrote:

 The issue begun with v. 1.9.5 but now the new instaces are created with
 1.9.4 so i have about 220 instances on 1.9.5 and 110 on 1.9.4.
 On 9 May 2014 12:21, timh zutes...@gmail.com wrote:

 I am still on 1.9.4 and not seeing this.

 Is everyone experiencing this rolled over to 1.9.5 ?

 T

 On Friday, May 9, 2014 11:40:19 AM UTC+8, blackdew wrote:

 my server make too many Frontend instances

 may be it make instance per request

 from 2 hours ago

 and my last deploy was yesterday may be 24 hours ago

 so, help and give me any advice

 thanks

  --
 You received this message because you are subscribed to a topic in the
 Google Groups Google App Engine group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/topic/google-appengine/tlQPHCvP-14/unsubscribe
 .
 To unsubscribe from this group and all its topics, send an email to
 google-appengine+unsubscr...@googlegroups.com.

 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/d/optout.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: retrieve old logs from GAE

[Eric Ka Ka Ng]

Yes, suppose normally I can't get back the rotated logs. just wonder would
there be any special ways, coz i believe the rotated logs is not really
deleted permanently 



On 17 August 2013 11:19, Vinny P vinny...@gmail.com wrote:

 On Fri, Aug 16, 2013 at 7:58 PM, Eric Ka Ka Ng ngk...@gmail.com wrote:

 Yes. But does it mean I can immediately get the logs 2 months ago?



 If they're not currently in the App Engine logs page, no you cannot get
 past deleted logs.


 -
 -Vinny P
 Technology  Media Advisor
 Chicago, IL

 App Engine Code Samples: http://www.learntogoogleit.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] retrieve old logs from GAE

[Eric Ka Ka Ng]

Hi,

my app has default log setting with 1GB storage, which spanning like around
23 days.

for some special cases, i want to retrieve logs older than 23 days, like
around 2 months ago.

do anyone know, is there any methods I can do this? any channels to reach
google if would like to have this special request? coz that's quite
important to us. appreciate for any ideas or experiences. thx!

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: retrieve old logs from GAE

[Eric Ka Ka Ng]

Hi Jason,

Yes. But does it mean I can immediately get the logs 2 months ago?

- eric


On 17 August 2013 03:41, Jason Collins jason.a.coll...@gmail.com wrote:

 Can't you just increase your log storage allocation on Application
 Settings (see: Logs Retention)?
 j


 On Friday, 16 August 2013 10:33:08 UTC-6, Eric Ka Ka Ng wrote:

 Hi,

 my app has default log setting with 1GB storage, which spanning like
 around 23 days.

 for some special cases, i want to retrieve logs older than 23 days, like
 around 2 months ago.

 do anyone know, is there any methods I can do this? any channels to reach
 google if would like to have this special request? coz that's quite
 important to us. appreciate for any ideas or experiences. thx!

 - eric

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Task queue Problem

[Eric Ka Ka Ng]

Since around 2 hours ago, the task queues in our apps seems stop
functioning. The rate of task execution is very very slow (sometimes 0,
sometimes 10 / min, sometimes normal). is there any wrong with the task
queue and could someone from Google help? our apps relies heavily on task
queue

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Cant update Billing Setting

[Eric Ka Ka Ng]

One of our google app is reaching daily quota. I try to update the daily
quota, but was prompted with this error


Your application is being migrated to a new billing system. Please check
back later to view or change your billing settings. Or contact
supporthttps://support.google.com/code/go/cloud_billing if
you need immediate assistance.



Could anyone advise what I can do? I need to increase the budget and resume
the service immediately!

here is the app id
butterflybsix

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] why this happen? application migrated to a new billing system and WE CANNOT update quota to resume service

[Eric Ka Ka Ng]

One of our google app is reaching daily quota. I try to update the daily
quota, but was prompted with this error


Your application is being migrated to a new billing system. Please check
back later to view or change your billing settings. Or contact
supporthttps://support.google.com/code/go/cloud_billing if
you need immediate assistance.



Could anyone advise what I can do? we need to resume the service to our
customers asap!

here is the app id
butterflybsix

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Problems with deploying

[Eric Ka Ka Ng]

the problem lasts for half a day already, would google do sth to help and
fix? i think google should announce what's wrong with it and the service
level provided by GAE is really not satisfactory..

- eric


On 2 February 2013 20:49, Aurelian aurelian.vacar...@gmail.com wrote:

 Same problem for my app. (java).

 appid: riparautonline-hrd


 On Saturday, February 2, 2013 12:59:04 PM UTC+1, Roberto Saccon wrote:

 Yes me experiencing the same,  use the go backend (which uses the python
 uploader) and I see this:

 12:51 PM Compilation completed.
 12:51 PM Starting deployment.
 12:51 PM Checking if deployment succeeded.
 12:51 PM Will check again in 1 seconds.
 12:51 PM Checking if deployment succeeded.
 12:51 PM Will check again in 2 seconds.
 12:51 PM Checking if deployment succeeded.
 12:51 PM Will check again in 4 seconds.
 12:51 PM Checking if deployment succeeded.
 12:51 PM Will check again in 8 seconds.
 12:51 PM Checking if deployment succeeded.
 12:51 PM Will check again in 16 seconds.
 12:51 PM Checking if deployment succeeded.
 12:51 PM Will check again in 32 seconds.
 12:52 PM Checking if deployment succeeded.
 12:52 PM Will check again in 60 seconds.
 12:53 PM Checking if deployment succeeded.
 12:53 PM Will check again in 60 seconds.
 12:54 PM Checking if deployment succeeded.
 12:54 PM Will check again in 60 seconds.
 12:55 PM Checking if deployment succeeded.
 12:55 PM Will check again in 60 seconds.
 12:56 PM Checking if deployment succeeded.
 12:56 PM Will check again in 60 seconds.

 and so on, until it hungs up (Exception: Version not ready.)

 any idea wha to do ???

 On Saturday, February 2, 2013 12:27:33 PM UTC+1, Jesús Espejo wrote:

 Hello,

 Is anyone experimenting issues uploading applications to GAE? I cannot
 do it, it remains in the exponential back-off algorithm in 60 seconds
 forever... Also i have problems launching backends:

 Error: Server ErrorThe service you requested is not available yet.

 Please try again in 30 seconds.

 My app is for Python.

 Thanks in advance,

 Jesús.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Problems with deploying

[Eric Ka Ka Ng]

such service level is totally unacceptable.. google... it is a paid
service you know...

On 3 February 2013 02:12, Jesús Espejo jesp...@tuspistas.es wrote:

 It still continues with problems, i cannot deploy applications or even
 launch a backend... :-/

 El sábado, 2 de febrero de 2013 17:45:19 UTC, Saurabh Gupta escribió:

 Google has not acknowledged this as an issue. In fact, the system
 dashboard  
 (http://code.google.com/**status/appenginehttp://code.google.com/status/appengine)
 shows that everything is 'normal'

 This is really frustrating to have no feedback from Google on such an
 important issue.



 On Saturday, February 2, 2013 12:42:43 PM UTC-5, Michael Hermus wrote:

 Same here. Has this issue not yet been acknowledged by Google? Ack.

 On Saturday, February 2, 2013 6:27:33 AM UTC-5, Jesús Espejo wrote:

 Hello,

 Is anyone experimenting issues uploading applications to GAE? I cannot
 do it, it remains in the exponential back-off algorithm in 60 seconds
 forever... Also i have problems launching backends:

 Error: Server ErrorThe service you requested is not available yet.

 Please try again in 30 seconds.

 My app is for Python.

 Thanks in advance,

 Jesús.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] GAE access from mainland china and cloudfare

[Eric Ka Ka Ng]

before heard that there were access problem to GAE (ghs.google.com) from
mainland china.
would like to know latest status about this.

now we hv domain mapped to GAE instance and seems some users from mainland
china can access it without problems. but not sure if all users from
different part of china can access, at anytime? also, any help if going
through cloudfare?

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] GAE access from mainland china and cloudfare

[Eric Ka Ka Ng]

Thx Will for your input. so cloudfare is not going to help?

- eric




On 23 May 2012 01:16, Will vocalster@gmail.com wrote:

 GAE is still blocked in mainland China.

 The regime is constantly monitoring the network traffic and servers and
 taking actions whenever it sees fit. So I don't believe any simple method
 can claim access for 'all users' from 'all parts' at 'anytime'. If one
 server is deemed a 'threat', the whole DNS may be blocked, as is the case
 of GAE.

 Good luck,

 Will

 On Tue, May 22, 2012 at 8:02 AM, Eric Ka Ka Ng ngk...@gmail.com wrote:

 before heard that there were access problem to GAE (ghs.google.com) from
 mainland china.
 would like to know latest status about this.

 now we hv domain mapped to GAE instance and seems some users from
 mainland china can access it without problems. but not sure if all users
 from different part of china can access, at anytime? also, any help if
 going through cloudfare?

 - eric

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE access from mainland china and cloudfare

[Eric Ka Ka Ng]

Thx you guys for all the suggestions (and debating ;))

Will, our site is for general public, not mission critical, so infrequent
downtime or a few minutes glitches could be acceptable. Guess from your
description it's not ideal solution but acceptable, and should apply to us
too.

Jeff, full-time SSL, according to you, should further minimize chances for
getting blocked. i think we can consider that too, thx

Brando and Damon, thx a lots for the debating and i learn a lots from it.
Damon, we want to launch a site in China and I prefer to run on GAE (just
becoz i'm more familiar with dev on GAE than like EC2 or others) If you
have any advises /settings for using CF to help us to minimize downtime /
enhance performance / increase rank on SE (or have actual figures for
downtime with GAE apps on CF in China) , please drop me an email. thx!

cheers,
eric

On 23 May 2012 07:49, Damon Billian da...@cloudflare.com wrote:

 1. I have never viewed myself as a Cloud Flare Competitor. We built a
 Product because Cloud Flare F***ED so many bloggers and local
 businesses
 that we needed something that they could use as a stop gap until they
 could
 move to infrastructure that worked. 

 Umm...your product isn't competitive?

  2. BULL CRAP
 Prove something valid? Or something that wasn't resolved?

  3. Infrequently yeah... When your site get delisted it makes you
 feel
  better that it only happens once every 100 days.
 There are tons of reasons that a site could get delisted from Google,
 using CloudFlare or not. To assume that we're the cause because they
 are on CloudFlare is flawed logic. I also explained the Google
 crawling from new IPs that could cause a temporary issue.

  I should encourage people to use Cloud Flare because it typically
 was very
  good for getting me SEO business.

 Yes, you should. Please do.

  I wouldn't Piss on a Cloud Flare
  employee if they were on fire. 
 We wouldn't even accept it.


  2. BULL CRAP
 
  3. Infrequently yeah... When your site get delisted it makes you feel
  better that it only happens once every 100 days.
 
  I may be an ass at a time. I may have taken clients that are scum of the
  earth for doing reputation management.  I wouldn't Piss on a Cloud Flare
  employee if they were on fire.   That's where CF ranks in relationship to
  Cash For Gold, Politicians, and Pet Food companies that have had major
  recalls.
 
  I should encourage people to use Cloud Flare because it typically was
 very
  good for getting me SEO business.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] exceptionally high latency, causing bad experiences and high hosting cost

[Eric Ka Ka Ng]

from around last 12 hours, we see an exceptionally high latency on some of
our gae apps. before requests on average takes 200ms-1000ms. now it can
takes 2000ms - 5000ms. I try to measure the time as local client. like make
10 requests, 1-2 times the latency is normal 1s, half of the time is like
2s-5s, sometimes it even takes 10-20s!!

as a result, it also doubles the number of instances started by GAE, and so
as the frontend cost.

does anyone observe similar problem recently, or it's only me? is it
google's infrastructure prb, or anything i can do? if it's google's
infrastructure prb, i dont think it makes sense to transform such bad user
experiences (high latency) into doubling our hosting cost

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] App down due to Quotas exceeded

[Eric Ka Ka Ng]

we got the same prb last week. after 3-4 days we get replies from google
helping to fix the issue. we lost business, customers, and repetition
during the down time.

yes, i'm sure that's an internal bug for them and many others face the
similar problem. this problem affects many apps, and many real users. I
wont expect these problems happen on a google platform which has passed the
'beta' stage, and we are paying real money to deploy serious applications
on it. Please, google, be serious to the business.

- eric


On 7 April 2012 04:03, doright doug.stodd...@gmail.com wrote:

 My app has been down for over 3 hours now.  Its a production app, with
 paying customers. At least it did have paying customers 3 hours ago...

 Okay, so I've exceeded my quotas, just increase my budgets right?  Wrong -
 it won't let me, saying:
 A charge has been issued to the billing administrator. *(You will be
 able to make changes to your budget settings again once the outstanding
 payment is processed.)* 

 So I'm stuck, it won't let me make any changes even though I have a valid
 VISA card registered, ready and waiting.  There has never been a problem
 with the card, and its still valid.

 So, I send a mail to Please contact us at
 appengine_updated_pric...@google.comappengine_updated_pric...@google.com?subject=Billing%20question%20regarding%20mybikeshopmanager
  if
 you have questions or issues regarding billing..  No answer for 3 hours
 and counting.  I raised an issue about it, still no answer.  Finally I'm
 posting here, having reached a final dead end.

 There are so many ironies here its almost funny.  I exceeded my quotas
 because I was experiencing lots of problems with high latencies and warmup
 requests taking too long.  Some of my customers had expressed concern that
 the system couldn't handle the load.
 Second irony is due to the high latencies experienced, some customers had
 started to question if Google was a noddy platform and couldn't scale.  I
 of course went to great lengths to discuss that I'd in fact chosen this
 platform due to its robustness and scalability.  And then its down for over
 3 hours due to a billing bug.

  I agree I should have anticipated the quotas actually being exceeded, but
 given that I missed that moment I would have hoped the situation would have
 been recoverable in less than 3 hours!  Beware ALL.

 If there is someone out there who can help, please tell me how I get the
 app running again ASAP.

 many thanks.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/q3sc-jbv7PgJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Datastore Admin is not working?

[Eric Ka Ka Ng]

I have added
builins:
- datastore_admin: on

 and enabled Datastore Admin in the Application Setting Page. But when I
click the link in the admin console, it just show a blank page. Is there
any problems with the Datastore Admin? or there are some other things I
have to configure?

appid: beyond6ixdev2

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] [Urgent] cannot change billing settings for our app foodeasyb6

[Eric Ka Ka Ng]

I know this is mainly for development issue but I could find no way to
reach google. Hope anyone know how to fix it could provide answer, or can
help forwarding my problem to suitable personnels. Appreciate that.



We have a number of apps on GAE. One of them foodeasyb6 has exceeded
daily quota and I have tried to change the budget. However, after last time
updating, it's always stay in the status

Billing Status: Activating Billing

Billing will be enabled as soon as the new billing administrator's credit
card has been authorized and quotas have been adjusted. *(You will be able
to make changes to your budget settings again once the outstanding payment
is processed.)*

*
*

*Most my other gae app doesn't have this problem and the billing, with the
same credit card*

*
*

*I have no way to change it or activate it. Hundreds of our users are
currently affected while the system is down. Please help to fix*

*
*

*- eric*

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] indexes stuck in Error state

[Eric Ka Ka Ng]

Hi Ikai,

Could you please help to remove 2 indexes in our app, which is being stuck
in the Error state? I have tried to vacuum the indexes, but always failed.
And this also prevents me to deploy new indexes too.

app id: gaewedev.appspot.com


I believe there were probably some problems when i'm trying to build that 2
indexes, as they involves entries having a high number of indexed terms
(index with list with long entries). So maybe after you help to remove the
indexes and then i try to build, the same problem could happen again. (from
my experience, if adding index to an empty data set, then populate the data
with the already built index should be ok; or when adding new item, always
add one by one) do you have tips / hints / guildeline to prevent running
into the same problem again? or could help us to re-build that 2 indexes?

Thanks in advanced.

Best Regards,
Eric Ng

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: email sent by GAE would be regarded as spam for some email services

[Eric Ka Ka Ng]

me 2. Have integrated the SES service within our GAE apps.  work good and
solved our problems

sure it would be more nice if GAE can provide email service at the quality
level similar to the Amazon SES one (at least would not be classified as
SPAM), so we dont need to manage both . could we make this suggestion to GAE
team?

- eric


On 22 March 2011 05:30, Jamie H ja...@mhztech.com wrote:

 Wow, great idea!  I have been having some delivery issues lately with
 GAE and looking for an alternative.

 I just downloaded a pythong Amazon SES library, signed up for SES, and
 put it all together and it works like a charm with GAE!

 On Mar 20, 10:31 pm, Nickolas Daskalou n...@daskalou.com wrote:
  Hi Eric,
 
  Have you looked at Amazon's Simple Email Service (SES)?
 
  http://aws.amazon.com/ses/
 
  Pretty much the same cost as sending email using GAE, except with SES you
  only get 2,000 free emails if you're sending email from an EC2 instance.
 
  I haven't used it myself but apparently they scan outgoing mail to make
 sure
  it meets ISP standards. This leads me to believe emails sent via SES
 would
  be less likely to be automatically marked as spam than on GAE, since
 SES's
  spam-sending-reputation would be lower than GAE's, due to their
  pre-filtering and rate-limiting.
 
  Nick
 
  On 21 March 2011 13:51, Eric Ka Ka Ng ngk...@gmail.com wrote:
 
   Hi Ernesto,
 
   thx for your suggestion! yes, i think it shall work.
 
   but in general if we can't ask all our users to add our email to their
   contact list (or they are not willing to do so), how can emails sent by
 us
   prevent to be regarded as spam? all our subject, body etc. are nothing
   spam-liked, and seems it's the problem of the email server that
 actually
   sends the email. anyone share similar case, or has other solution?
 
   regards,
   eric
 
   On 19 March 2011 00:12, Ernesto Karim Oltra ernestoka...@gmail.com
 wrote:
 
   Ask the user to add the e-mail from address of your e-mails to their
   contacts list, so your e-mails would never been sent to spam again
   (for that users, at least).
 
   It's a bit hacky, but can do a great work meanwhile you find another
   solution.
 
   On 18 mar, 07:52, Eric Ka Ka Ng ngk...@gmail.com wrote:
we use mail.send_mail() to send some important messages to our users
   through
an app hosted on GAE, in which the 'from' has been set to one of a
registered admin for the app.
 
it works quite well for most users, except for some email service
   provider
(e.g. '...@yahoo.com.hk'), they would always automatically treat
 these
   emails
as spam and put them into spam box of the user's email account (if
 the
   user
has enabled the spam filtering feature, which is by default ON) .
 in
   this
case, many of our users do not aware for these important messages
 (they
would seldom look into mails in their spambox)
 
we have tried using different from, subject, body and the
 results
   are
the same. we suspect that it is the email server does matter, and
 maybe
before there were some other apps hosted on GAE sending spams to
 like '@
yahoo.com.hk', so '...@yahoo.com.hk' would regard all emails sent
 from
   this
email server in GAE as spam.
 
do anyone share similar experiences? or there should be other
 causes?
   any
ideas we can solve this problem? (successfully deliver the email to
 our
users using those email service without being regarded as spam) thx
 in
advanced!
 
- eric
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
--
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com

Re: [google-appengine] Re: email sent by GAE would be regarded as spam for some email services

[Eric Ka Ka Ng]

Hi Ikai,

thx for your note. with this, we could better decide on which cases we would
like to send emails through GAE and which we would like to use other
external services.

and would it be more formally stated in any documents, or any guideline for
these considerations be documented? if i'm a new GAE user, i would just use
the email service straight forwardly without these considerations and
background information.

- eric



On 23 March 2011 05:29, Ikai Lan (Google) ika...@google.com wrote:

 I actually think we are moving away from this direction. For customers
 sending large volumes of email or who require delivery guarantees, we'll be
 encouraging the use of services that are specifically designed for this
 functionality.

 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog: http://googleappengine.blogspot.com
 Twitter: http://twitter.com/app_engine
 Reddit: http://www.reddit.com/r/appengine



 On Tue, Mar 22, 2011 at 1:35 AM, Eric Ka Ka Ng ngk...@gmail.com wrote:

 me 2. Have integrated the SES service within our GAE apps.  work good and
 solved our problems

 sure it would be more nice if GAE can provide email service at the quality
 level similar to the Amazon SES one (at least would not be classified as
 SPAM), so we dont need to manage both . could we make this suggestion to GAE
 team?

 - eric


 On 22 March 2011 05:30, Jamie H ja...@mhztech.com wrote:

 Wow, great idea!  I have been having some delivery issues lately with
 GAE and looking for an alternative.

 I just downloaded a pythong Amazon SES library, signed up for SES, and
 put it all together and it works like a charm with GAE!

 On Mar 20, 10:31 pm, Nickolas Daskalou n...@daskalou.com wrote:
  Hi Eric,
 
  Have you looked at Amazon's Simple Email Service (SES)?
 
  http://aws.amazon.com/ses/
 
  Pretty much the same cost as sending email using GAE, except with SES
 you
  only get 2,000 free emails if you're sending email from an EC2
 instance.
 
  I haven't used it myself but apparently they scan outgoing mail to make
 sure
  it meets ISP standards. This leads me to believe emails sent via SES
 would
  be less likely to be automatically marked as spam than on GAE, since
 SES's
  spam-sending-reputation would be lower than GAE's, due to their
  pre-filtering and rate-limiting.
 
  Nick
 
  On 21 March 2011 13:51, Eric Ka Ka Ng ngk...@gmail.com wrote:
 
   Hi Ernesto,
 
   thx for your suggestion! yes, i think it shall work.
 
   but in general if we can't ask all our users to add our email to
 their
   contact list (or they are not willing to do so), how can emails sent
 by us
   prevent to be regarded as spam? all our subject, body etc. are
 nothing
   spam-liked, and seems it's the problem of the email server that
 actually
   sends the email. anyone share similar case, or has other solution?
 
   regards,
   eric
 
   On 19 March 2011 00:12, Ernesto Karim Oltra ernestoka...@gmail.com
 wrote:
 
   Ask the user to add the e-mail from address of your e-mails to their
   contacts list, so your e-mails would never been sent to spam again
   (for that users, at least).
 
   It's a bit hacky, but can do a great work meanwhile you find another
   solution.
 
   On 18 mar, 07:52, Eric Ka Ka Ng ngk...@gmail.com wrote:
we use mail.send_mail() to send some important messages to our
 users
   through
an app hosted on GAE, in which the 'from' has been set to one of a
registered admin for the app.
 
it works quite well for most users, except for some email service
   provider
(e.g. '...@yahoo.com.hk'), they would always automatically treat
 these
   emails
as spam and put them into spam box of the user's email account (if
 the
   user
has enabled the spam filtering feature, which is by default ON)
 . in
   this
case, many of our users do not aware for these important messages
 (they
would seldom look into mails in their spambox)
 
we have tried using different from, subject, body and the
 results
   are
the same. we suspect that it is the email server does matter, and
 maybe
before there were some other apps hosted on GAE sending spams to
 like '@
yahoo.com.hk', so '...@yahoo.com.hk' would regard all emails sent
 from
   this
email server in GAE as spam.
 
do anyone share similar experiences? or there should be other
 causes?
   any
ideas we can solve this problem? (successfully deliver the email
 to our
users using those email service without being regarded as spam)
 thx in
advanced!
 
- eric
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
--
   You received this message because you are subscribed to the Google
 Groups

Re: [google-appengine] Re: email sent by GAE would be regarded as spam for some email services

[Eric Ka Ka Ng]

Hi Ernesto,

thx for your suggestion! yes, i think it shall work.

but in general if we can't ask all our users to add our email to their
contact list (or they are not willing to do so), how can emails sent by us
prevent to be regarded as spam? all our subject, body etc. are nothing
spam-liked, and seems it's the problem of the email server that actually
sends the email. anyone share similar case, or has other solution?

regards,
eric




On 19 March 2011 00:12, Ernesto Karim Oltra ernestoka...@gmail.com wrote:

 Ask the user to add the e-mail from address of your e-mails to their
 contacts list, so your e-mails would never been sent to spam again
 (for that users, at least).

 It's a bit hacky, but can do a great work meanwhile you find another
 solution.

 On 18 mar, 07:52, Eric Ka Ka Ng ngk...@gmail.com wrote:
  we use mail.send_mail() to send some important messages to our users
 through
  an app hosted on GAE, in which the 'from' has been set to one of a
  registered admin for the app.
 
  it works quite well for most users, except for some email service
 provider
  (e.g. '...@yahoo.com.hk'), they would always automatically treat these
 emails
  as spam and put them into spam box of the user's email account (if the
 user
  has enabled the spam filtering feature, which is by default ON) . in
 this
  case, many of our users do not aware for these important messages (they
  would seldom look into mails in their spambox)
 
  we have tried using different from, subject, body and the results
 are
  the same. we suspect that it is the email server does matter, and maybe
  before there were some other apps hosted on GAE sending spams to like '@
  yahoo.com.hk', so '...@yahoo.com.hk' would regard all emails sent from
 this
  email server in GAE as spam.
 
  do anyone share similar experiences? or there should be other causes? any
  ideas we can solve this problem? (successfully deliver the email to our
  users using those email service without being regarded as spam) thx in
  advanced!
 
  - eric

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] email sent by GAE would be regarded as spam for some email services

[Eric Ka Ka Ng]

we use mail.send_mail() to send some important messages to our users through
an app hosted on GAE, in which the 'from' has been set to one of a
registered admin for the app.

it works quite well for most users, except for some email service provider
(e.g. '@yahoo.com.hk'), they would always automatically treat these emails
as spam and put them into spam box of the user's email account (if the user
has enabled the spam filtering feature, which is by default ON) . in this
case, many of our users do not aware for these important messages (they
would seldom look into mails in their spambox)


we have tried using different from, subject, body and the results are
the same. we suspect that it is the email server does matter, and maybe
before there were some other apps hosted on GAE sending spams to like '@
yahoo.com.hk', so '@yahoo.com.hk' would regard all emails sent from this
email server in GAE as spam.

do anyone share similar experiences? or there should be other causes? any
ideas we can solve this problem? (successfully deliver the email to our
users using those email service without being regarded as spam) thx in
advanced!

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] can we assume time is sync across all servers?

[Eric Ka Ka Ng]

recently we encounter some problems which we found some of the datetime
value persisted in the DS does not match with what we expect. usually we
record these time by calling datetime.utcnow() (we are using python) to get
the 'current' datetime values


in some situations (though i'm not yet 100% sure if there are other bugs or
is the time sync problem), we found that these persisted datetime values are
not quite in-sync. and i'm not talking about ms or s, the time is off for 30
mins!

e.g. two requests A and B were being made. B is made about a minute after A
and they follow exactly the same flow and would call datetime.utcnow() and
persist the values to DS. i expect persisted value made by request B is a
minute larger than one made by request A. but sometimes it turns out that
value made by request A is 30 mins larger than value made by request B. (and
sure under GAE, server handling request A and B can be different and we wont
know which server handles it)


before any further investigation, is it true that we can strongly assume all
server are time-sync? or if not there be some system bugs? or the way i'm
getting current time (datetime.utcnow() ) is not appropriate?

thanks in advance for any comments / feedbacks

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: can we assume time is sync across all servers?

[Eric Ka Ka Ng]

Hi Gopal, Calvin, and Ikai,

thx all for your sharing. Calvin, your handy script implements Gopal idea
could be very useful to many of us ;)

Ikai, yes, later we could do more testing and sampling, and with Calvin's
script, to log the time (and delta with google time) however, this is not a
problem frequently happen and we are currently busying with some other
application dev tasks, guess it may take some time to log and analyze this
problem, and would keep you posted. just thinking what kinds of 'examples'
we could provide? (you want log? data in DS? script to produce the log /
data? )


btw, how could we explain Brett's data pipelines' talk at 35:50 or i
interpret wrongly? and i believe Gopal, Calvin faced time sync issues and
thus have the idea and implementation to solve this (just me recently aware
about this problem). how much time is off did you experience?

thx a lot!

- eric

On 4 March 2011 03:12, Ikai Lan (Google) ika...@google.com wrote:

 Can you provide examples where the time is off by 30 minutes? This seems
 incorrect.

 There's definitely clock skew, but we're talking milliseconds, seconds or
 at worst, worst, worst maybe a minute or two.

 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog: http://googleappengine.blogspot.com
 Twitter: http://twitter.com/app_engine
 Reddit: http://www.reddit.com/r/appengine



 On Thu, Mar 3, 2011 at 5:54 AM, Ng Ka Ka Eric ngk...@gmail.com wrote:

 Hi Raymond,

 Just watched the video. Thx for pointing this reference.


 I really didnt expect that the machines are not time synced (can be off
 for 40mins?!) are there any technical difficulties to sync them? And in this
 case, for whatever reason if we want to record the time when the request is
 made into DS, how can we do that?

 - eric

 Sent from my iPhone

 On 2011年3月3日, at 下午6:20, Raymond C. windz...@gmail.com wrote:

 According to Brett's data pipelines' talk (
 http://www.youtube.com/watch?v=zSDC_TU7rtc, around 35:50), its not

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] about datastore admin copy function

[Eric Ka Ka Ng]

thx David! This is what i'm looking for! good reference!

there were some mapreduce jobs (copy), which i believe its callback task has
been deleted from the taskqueues when i cleaned the queue, being stuck in
the system and could not be aborted from http://{version}{host}/_ah/mapreduce
(nothing changed after i click abort in the page)

i know probably the jobs should no longer exist and it's just some task
entries being left in the datastore. know any ways i could clean up those
entries too?


- eric

On 22 February 2011 13:14, David Mora dla.m...@gmail.com wrote:

 is a mapreduce (just like the delete)

 you can access it thru: http://{version}{host}/_ah/mapreduce

 i think it has a callback that cleans up the job after it is done/aborted
 tho.

 On 21 February 2011 23:08, Tom Wu service.g2...@gmail.com wrote:
  +1
 
  2011/2/11 Eric Ka Ka Ng ngk...@gmail.com
 
  The new function to copy data from a GAE instance to another GAE
 instance
  makes some of our operations much more convenient! Thx for making this
  feature available!
  I know this function is still experimental, I have a question in using
  this feature.
 
  After i have started a copy / delete task, there would be a link to show
  the current status of the task. however, if i leave the page and later
 open
  the GAE admin console page, i couldn't find that link again and have no
 way
  to go back to the screen which i can check current status of the tasks.
 (i
  can see the operations' status from the datastore admin page, but i
 can't
  click to go to see the details)
 
  What's more, if there are any copy tasks fail for whatever reasons, I
 have
  no way to abort the task. I could manually purge the task queue to stop
 it,
  but the operations entries would still be under the Operation's status
 which
  cannot be removed.
  so would there be any accessible UI that we could monitor status / abort
  these copy / delete tasks in datastore admin?
  - eric
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 



 --
 http://about.me/david.mora

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] about datastore admin copy function

[Eric Ka Ka Ng]

yup, normally wouldn't manually touch those operations in the task queue.
just there were cases that there are some other problems in the task queues
so we would like to manually purge the task queues (which purge those
operations too), and that's why there are those zombie job left in this
case.

just to see if there are any ways to 'properly clean' these zombie job
entries (sure can again manually delete these entries on those
map-reduce-create entities through datastore viewer, just afraid again
without knowing all its flow, such manual intervention may create some other
problems), although it's not big deal to have those zombie entries being
shown for now

- eric

On 22 February 2011 14:16, David Mora dla.m...@gmail.com wrote:

 i would not touch the task queues for such operations, tho.

 They are used by the map reduce when it calculates the shards for each
 segment of the map (maps using the index, assigning segments to shards
 - and the shards execute via task queue doing the reduce).

 Probably you have a zombie job that is waiting for a shard to write
 into the datastore (these is how the shards communicate for some
 information)

 Anyways, glad it worked for you :)

 On 21 February 2011 23:51, Eric Ka Ka Ng ngk...@gmail.com wrote:
  thx David! This is what i'm looking for! good reference!
  there were some mapreduce jobs (copy), which i believe its callback task
 has
  been deleted from the taskqueues when i cleaned the queue, being stuck in
  the system and could not be aborted
  from http://{version}{host}/_ah/mapreduce (nothing changed after i click
  abort in the page)
  i know probably the jobs should no longer exist and it's just some task
  entries being left in the datastore. know any ways i could clean up those
  entries too?
 
  - eric
 
  On 22 February 2011 13:14, David Mora dla.m...@gmail.com wrote:
 
  is a mapreduce (just like the delete)
 
  you can access it thru: http://{version}{host}/_ah/mapreduce
 
  i think it has a callback that cleans up the job after it is
 done/aborted
  tho.
 
  On 21 February 2011 23:08, Tom Wu service.g2...@gmail.com wrote:
   +1
  
   2011/2/11 Eric Ka Ka Ng ngk...@gmail.com
  
   The new function to copy data from a GAE instance to another GAE
   instance
   makes some of our operations much more convenient! Thx for making
 this
   feature available!
   I know this function is still experimental, I have a question in
 using
   this feature.
  
   After i have started a copy / delete task, there would be a link to
   show
   the current status of the task. however, if i leave the page and
 later
   open
   the GAE admin console page, i couldn't find that link again and have
 no
   way
   to go back to the screen which i can check current status of the
 tasks.
   (i
   can see the operations' status from the datastore admin page, but i
   can't
   click to go to see the details)
  
   What's more, if there are any copy tasks fail for whatever reasons, I
   have
   no way to abort the task. I could manually purge the task queue to
 stop
   it,
   but the operations entries would still be under the Operation's
 status
   which
   cannot be removed.
   so would there be any accessible UI that we could monitor status /
   abort
   these copy / delete tasks in datastore admin?
   - eric
  
   --
   You received this message because you are subscribed to the Google
   Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
   http://groups.google.com/group/google-appengine?hl=en.
  
   --
   You received this message because you are subscribed to the Google
   Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
   http://groups.google.com/group/google-appengine?hl=en.
  
 
 
 
  --
  http://about.me/david.mora
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
 



 --
 http://about.me/david.mora


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] A serious problem was encountered with the process that handled this request, ..you should contact the App Engine team. (Error code 203)

[Eric Ka Ka Ng]

I have two tasks stuck in the task queues, keep on creating errors and
cannot completed successfully. Here are the error dumps


1st error


   1.

   AppEngine-Google; (+http://code.google.com/appengine)
gaewsdev.appspot.com ms=204109 cpu_ms=473266 api_cpu_ms=473266
cpm_usd=13.146328 queue_name=default task_name=12116044570446551933
exit_code=203

   2. W02-10 07:15AM 30.026

   A serious problem was encountered with the process that handled
this request, causing it to exit. This is likely to cause a new
process to be used for the next request to your application. If you
see this message frequently, you should contact the App Engine team.
(Error code 203)






2nd error


   1. 02-10 08:02AM 01.014

   Exceeded soft memory limit with 299.996 MB after servicing 14 requests total

   2. W02-10 08:02AM 01.051

   After handling this request, the process that handled this request
was found to be using too much memory and was terminated. This is
likely to cause a new process to be used for the next request to your
application. If you see this message frequently, you may have a memory
leak in your application.




Could anyone or the App Engine team advise how i could solve these problems?

my app id is gaewsdev


- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: A serious problem was encountered with the process that handled this request, ..you should contact the App Engine team. (Error code 203)

[Eric Ka Ka Ng]

Hi Tim,

thx for your reply. forgot to mention that i'm using python. i didn't use
apptrace but i did profiling and optimization using appstats

but one of the memory issue seems to be thrown from wsgi middleware
(appstats?) when it's doing the recording. correct me if i'm wrong, here is
the completed traceback.



   1. E2011-02-10 18:30:12.361

   type 'exceptions.MemoryError':
   Traceback (most recent call last):
 File 
/base/data/home/apps/gaewsdev/6.348249895332861938/admin/removeUser.py,
line 219, in module
   main()
 File 
/base/data/home/apps/gaewsdev/6.348249895332861938/admin/removeUser.py,
line 216, in main
   run_wsgi_app(application)
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py,
line 97, in run_wsgi_app
   run_bare_wsgi_app(add_wsgi_middleware(application))
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py,
line 117, in run_bare_wsgi_app
   for data in result:
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/appstats/recording.py,
line 859, in appstats_wsgi_wrapper
   end_recording(500, firepython_set_extension_data)
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/ext/appstats/recording.py,
line 933, in end_recording
   memcache.delete(lock_key(), namespace=config.KEY_NAMESPACE)
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py,
line 513, in delete
   self._make_sync_call('memcache', 'Delete', request, response)
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py,
line 86, in MakeSyncCall
   return stubmap.MakeSyncCall(service, call, request, response)
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py,
line 281, in MakeSyncCall
   rpc = stub.CreateRPC()
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/runtime/apiproxy.py,
line 193, in CreateRPC
   return RPC()
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/runtime/apiproxy.py,
line 103, in __init__
   super(RPC, self).__init__(*args, **kargs)
 File 
/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py,
line 62, in __init__
   self.request = request

   2. C2011-02-10 18:30:12.376

   Exceeded soft process size limit with 299.793 MB after servicing 1
requests total

   3. I2011-02-10 18:30:12.411

   This request caused a new process to be started for your
application, and thus caused your application code to be loaded for
the first time. This request may thus take longer and use more CPU
than a typical request for your application.

   4. W2011-02-10 18:30:12.411

   After handling this request, the process that handled this request
was found to be using too much memory and was terminated. This is
likely to cause a new process to be used for the next request to your
application. If you see this message frequently, you may have a memory
leak in your application.





On 11 February 2011 00:15, Tim Hoffman zutes...@gmail.com wrote:

 Hi

 This is a problem in your application code.  As the error says, you are
 using too much memory and the instance is being killed.

 You haven't said if you use python or java.  If you are using python then
 have a look at apptrace (The apptrace package provides a WSGI middleware for
 tracking memory usage in Google App Engine Python applications.)


 http://docs.pylonsproject.org/projects/pyramid_tutorials/dev/wiki2_auth/index.html

 Bottom line is you will need to do some profiling and debugging to work out
 where you are either consuming too much memory or leaking (keeping around
 globals that grow for instance).

 Rgds

 T

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] about datastore admin copy function

[Eric Ka Ka Ng]

The new function to copy data from a GAE instance to another GAE instance
makes some of our operations much more convenient! Thx for making this
feature available!

I know this function is still experimental, I have a question in using this
feature.


After i have started a copy / delete task, there would be a link to show the
current status of the task. however, if i leave the page and later open the
GAE admin console page, i couldn't find that link again and have no way to
go back to the screen which i can check current status of the tasks. (i can
see the operations' status from the datastore admin page, but i can't click
to go to see the details)


What's more, if there are any copy tasks fail for whatever reasons, I have
no way to abort the task. I could manually purge the task queue to stop it,
but the operations entries would still be under the Operation's status which
cannot be removed.

so would there be any accessible UI that we could monitor status / abort
these copy / delete tasks in datastore admin?

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: a serious CPU eating datastore_v3.Put() Re: errors in updating entry in the datastore, through our own python script or through gae admin console

[Eric Ka Ka Ng]

i'm the default one (master/slave)

yes, that spikes is suspicious, hope can have someone explaining this, or
advise how to avoid this

- eric

On 9 February 2011 17:42, Simon Knott knott.si...@gmail.com wrote:

 Are you on the Master/Slave datastore or the High Replication datastore?

 If those puts are putting in the same kinds of entity, then those spikes
 look a little suspicious to me!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: a serious CPU eating datastore_v3.Put() Re: errors in updating entry in the datastore, through our own python script or through gae admin console

[Eric Ka Ka Ng]

Hi Robert,

if i didn't trace wrongly, that suspiciously big CPU eating PUT is just
putting a single entity, not putting a batch of entities

but i'm not sure what these values mean

entity_write_bytes = 1668453L (which is 1.xMB? )
entity_writes = 1L (1 entity to write? )

- eric

On 10 February 2011 04:07, Robert Kluin robert.kl...@gmail.com wrote:

 Are you putting large batches of entities there?

 Compare the entity_write_bytes to some of the other entries.



 Robert






 On Wed, Feb 9, 2011 at 04:58, Eric Ka Ka Ng ngk...@gmail.com wrote:
  i'm the default one (master/slave)
  yes, that spikes is suspicious, hope can have someone explaining this, or
  advise how to avoid this
  - eric
 
  On 9 February 2011 17:42, Simon Knott knott.si...@gmail.com wrote:
 
  Are you on the Master/Slave datastore or the High Replication datastore?
  If those puts are putting in the same kinds of entity, then those spikes
  look a little suspicious to me!
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] errors in updating entry in the datastore, through our own python script or through gae admin console

[Eric Ka Ka Ng]

Hi,

recently we encounter this error. i have a gae app with a number of (50+)
entities kinds and data and around 100+ indexes. the data would be removed /
updated / added by the application normally. however, there are about 5
particular entries (out of thousands other entries )of one of the entity
kind, when there are some updates operated on those entries, we would
encounter server error (transaction too large) . those particular entries
shouldn't be specially large or different than other data.


we then try to directly update one of its string field (just a string field)
of these particular entries through gae admin console data viewer, and, we
would always get


Server Error

A server error has occurred.

Return to Applications screen » http://appengine.google.com/


since there are quite a number of indexes that built with that string field,
it could be expected that updating that string field could involve more
index building operations. however, even if it's more involving, we dont
expect it shall always returns failure. and other entries in the same entity
can be updated normally. now, we have no way to update those entries.


can anyone advise

- is there any way we could update those data through other means?

- any possible causes for such strange behavior? since in terms of storage
and indexing gae should be a black box distributed system to us, we dont
(and dont need to) understand the underlying distributed persistent /
indexing implementations / limitations. or this could potentially be a bug?


If anyone from google can help solving this prb, we could provide further
information (app id, the key of those problematic entries etc.) . thx in
advanced!


- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] about Tasks Storage Quota 'Task Queue Stored Task Bytes' - how to count and how to configure

[Eric Ka Ka Ng]

Hi Nick,

Thank for your reply! guess i have more understanding on how the Task Queue
Stored Task Bytes be counted

as Task Queue Stored Task Bytes increases in 'near real time' when task is
being added, but decreases 'every couple of hours', so within this 'couple
of hours', this counter is sort of accumulative?

for example, would the following scenario be possible?

assume T4-T3=T3-T2=T2-T1=T1-T0=30 mins, and the counter decreases 2 hours
after T0 (at T4)

before T0, there were no tasks, Task Queue Stored Task Byte = 0
from T0-T1, many tasks are added, and Task Queue Stored Task Byte = 100MB
from T1-T2, all previous added tasks has been finished processing, leaving
the queue empty. Task Queue Stored Task Byte is still 100MB
from T2-T4, some tasks are added, but since Task Queue Stored Task Byte =
100MB (over the quota), those tasks would be accumulated in the queue
 but cannot be executed, making Task Queue Stored Task Byte continue to be
increasing to let say 200MB
at T4, Task Queue Stored Task Byte started to decrease, by the amount of
task being executed from T1-T2. Task Queue Stored Task Byte = 100MB (over
quota). newly added tasks since T2 still cannot be executed


anyway, seems being able to increase the quota for Task Queue Stored Task
Byte would be a MUST-HAVE for us, in order to avoid the tasks being stuck
and hanging the system when there is a short burst of tasks . could you
advise how we could configure this quota on GAE?

Many thanks,
Eric Ng

On 8 December 2010 10:15, Nicholas Verne nve...@google.com wrote:

 Eric,

 Task Queue Stored Task Count is updated in near real time. Each task
 that is added to a queue is counted at that time, and each task that
 is deleted after sucessful execution is subtracted.
 Task Queue Stored Task Bytes increases as each task is added, but
 decreases only every couple of hours when a process runs which takes
 an accurate measurement of real usage.

 To estimate the average number of bytes/task, you could pause the
 relevant queue, then watch the Task Queue Stored Task Bytes change as
 you add tasks.

 Nick Verne

 On Tue, Dec 7, 2010 at 8:46 PM, Eric Ka Ka Ng ngk...@gmail.com wrote:
  Once upon a time we experienced problem when we reach the Task Queue
 Stored
  Task Bytes Quota Limit (i believe is 100MB for paid application) . It
  happened when 'producer' produce too many tasks into the queue before the
  'consumer' can consume them, leaving many many tasks being in the queue.
  Although we fixed the problem for 'producer' faster than 'consumer'
 problem,
  this trigger us to have a better understanding on how the quota 'Task
 Queue
  Stored Task Bytes' be counted and configured
 
  Specially i have 3 questions
  1. how to count the 'bytes' of a task queue stored task?
  let say each task consumes 50K, as we should have 100MB of such quota, it
  should have the capacity to store 2 tasks. but how could we calculate
 /
  estimate the task stored size?
  2. is this a counter showing the 'real-time current' total stored task
 size,
  or a counter being updated regularly, or an accumulative counter being
 reset
  in daily basis?
  i was thinking it should not be an accumulative counter, but sometimes
 when
  i check the system we dont have any tasks in the queue, but the Tasek
 Queue
  Stored Task Bytes is not 0
 
  3. how to configure the quota for Task Queue Stored Task Bytes'
  in GAE
  documentation
 http://code.google.com/appengine/docs/quotas.html#Task_Queue,
  it is stated that we could configure it should be configurable up to
 Stored
  Data (billable) . however from our current GAE admin console and billing
  setting, we found no way to configure it.
 
  appreciateif anyone can share (partial) answers for these or your
 thoughts
 
  Regards,
  Eric ng
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appeng...@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] about Tasks Storage Quota 'Task Queue Stored Task Bytes' - how to count and how to configure

[Eric Ka Ka Ng]

Thx again Nick. Yes, I need to tune total_storage_limit and I just find
actually the document about queue.yaml does talk about that, but i missed
that part before. This is what exactly i'm looking for.

cheers,
eric

On 9 December 2010 11:24, Nicholas Verne nve...@google.com wrote:

 from T2-T4, the attempts to add tasks to the queue will fail because
 the Task Queue Stored Task Bytes is at or over quota.

 The solution is to give yourself head room. You do this by setting
 your limit to, say, 500 MB with the follwing in queue.yaml

 total_storage_limit: 500M

 Alternatively, in queue.xml

 total-storage-limit 500M /total-storage-limit

 Note that your Task Queue Stored Task Bytes also counts toward your
 Total Stored Data.  More Total Stored Data can be obtained by enabling
 billing.

 Nick Verne

 On Thu, Dec 9, 2010 at 2:14 PM, Eric Ka Ka Ng ngk...@gmail.com wrote:
  Hi Nick,
  Thank for your reply! guess i have more understanding on how the Task
 Queue
  Stored Task Bytes be counted
  as Task Queue Stored Task Bytes increases in 'near real time' when task
 is
  being added, but decreases 'every couple of hours', so within this
 'couple
  of hours', this counter is sort of accumulative?
  for example, would the following scenario be possible?
  assume T4-T3=T3-T2=T2-T1=T1-T0=30 mins, and the counter decreases 2 hours
  after T0 (at T4)
  before T0, there were no tasks, Task Queue Stored Task Byte = 0
  from T0-T1, many tasks are added, and Task Queue Stored Task Byte = 100MB
  from T1-T2, all previous added tasks has been finished processing,
 leaving
  the queue empty. Task Queue Stored Task Byte is still 100MB
  from T2-T4, some tasks are added, but since Task Queue Stored Task Byte =
  100MB (over the quota), those tasks would be accumulated in the queue
   but cannot be executed, making Task Queue Stored Task Byte continue to
 be
  increasing to let say 200MB
  at T4, Task Queue Stored Task Byte started to decrease, by the amount of
  task being executed from T1-T2. Task Queue Stored Task Byte = 100MB (over
  quota). newly added tasks since T2 still cannot be executed
 
 
  anyway, seems being able to increase the quota for Task Queue Stored Task
  Byte would be a MUST-HAVE for us, in order to avoid the tasks being stuck
  and hanging the system when there is a short burst of tasks . could you
  advise how we could configure this quota on GAE?
  Many thanks,
  Eric Ng
  On 8 December 2010 10:15, Nicholas Verne nve...@google.com wrote:
 
  Eric,
 
  Task Queue Stored Task Count is updated in near real time. Each task
  that is added to a queue is counted at that time, and each task that
  is deleted after sucessful execution is subtracted.
  Task Queue Stored Task Bytes increases as each task is added, but
  decreases only every couple of hours when a process runs which takes
  an accurate measurement of real usage.
 
  To estimate the average number of bytes/task, you could pause the
  relevant queue, then watch the Task Queue Stored Task Bytes change as
  you add tasks.
 
  Nick Verne
 
  On Tue, Dec 7, 2010 at 8:46 PM, Eric Ka Ka Ng ngk...@gmail.com wrote:
   Once upon a time we experienced problem when we reach the Task Queue
   Stored
   Task Bytes Quota Limit (i believe is 100MB for paid application) . It
   happened when 'producer' produce too many tasks into the queue before
   the
   'consumer' can consume them, leaving many many tasks being in the
 queue.
   Although we fixed the problem for 'producer' faster than 'consumer'
   problem,
   this trigger us to have a better understanding on how the quota 'Task
   Queue
   Stored Task Bytes' be counted and configured
  
   Specially i have 3 questions
   1. how to count the 'bytes' of a task queue stored task?
   let say each task consumes 50K, as we should have 100MB of such quota,
   it
   should have the capacity to store 2 tasks. but how could we
   calculate /
   estimate the task stored size?
   2. is this a counter showing the 'real-time current' total stored task
   size,
   or a counter being updated regularly, or an accumulative counter being
   reset
   in daily basis?
   i was thinking it should not be an accumulative counter, but sometimes
   when
   i check the system we dont have any tasks in the queue, but the Tasek
   Queue
   Stored Task Bytes is not 0
  
   3. how to configure the quota for Task Queue Stored Task Bytes'
   in GAE
  
   documentation
 http://code.google.com/appengine/docs/quotas.html#Task_Queue,
   it is stated that we could configure it should be configurable up to
   Stored
   Data (billable) . however from our current GAE admin console and
   billing
   setting, we found no way to configure it.
  
   appreciateif anyone can share (partial) answers for these or your
   thoughts
  
   Regards,
   Eric ng
  
   --
   You received this message because you are subscribed to the Google
   Groups
   Google App Engine group.
   To post to this group, send email to
 google-appeng...@googlegroups.com

[google-appengine] about Tasks Storage Quota 'Task Queue Stored Task Bytes' - how to count and how to configure

[Eric Ka Ka Ng]

Once upon a time we experienced problem when we reach the Task Queue Stored
Task Bytes Quota Limit (i believe is 100MB for paid application) . It
happened when 'producer' produce too many tasks into the queue before the
'consumer' can consume them, leaving many many tasks being in the queue.
Although we fixed the problem for 'producer' faster than 'consumer' problem,
this trigger us to have a better understanding on how the quota 'Task Queue
Stored Task Bytes' be counted and configured


Specially i have 3 questions

1. how to count the 'bytes' of a task queue stored task?
let say each task consumes 50K, as we should have 100MB of such quota, it
should have the capacity to store 2 tasks. but how could we calculate /
estimate the task stored size?

2. is this a counter showing the 'real-time current' total stored task size,
or a counter being updated regularly, or an accumulative counter being reset
in daily basis?
i was thinking it should not be an accumulative counter, but sometimes when
i check the system we dont have any tasks in the queue, but the Tasek Queue
Stored Task Bytes is not 0


3. how to configure the quota for Task Queue Stored Task Bytes'
in GAE documentation
http://code.google.com/appengine/docs/quotas.html#Task_Queue, it is stated
that we could configure it should be configurable up to Stored Data
(billable) . however from our current GAE admin console and billing setting,
we found no way to configure it.


appreciateif anyone can share (partial) answers for these or your thoughts


Regards,
Eric ng

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: 500 Server Error on https://appengine.google.com

[Eric Ka Ka Ng]

+2

On 15 November 2010 12:58, Tom Wu service.g2...@gmail.com wrote:

 +1

 2010/11/15 Nick nick.beag...@arclight.com.au

 I am also getting this error

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unable to Access Dashboard or Logs

[Eric Ka Ka Ng]

same problem here. although access to my application seems to be normal, but
i can't access the Dashboard at this moment. any update from Google?

Regards,
Eric Ng

On 3 November 2010 09:46, Eli Jones eli.jo...@gmail.com wrote:

 Hit send before clarifying.

 This has been going on for about 20 to 30 minutes.

 I get the usual The server encountered an error and could not complete
 your request. error.


 On Tue, Nov 2, 2010 at 9:45 PM, Eli Jones eli.jo...@gmail.com wrote:

 I am unable to access the Dashboard, Logs or anything to do with my App.

 Anyone else having these problems?


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Uploading Indexes - Server Error

[Eric Ka Ka Ng]

we got the same problems. can't update index, can't vaccum index, even in
the admin page, click Datastore indexes, server returns error.


seems there are some problems related to indexes on GAE? any other
experience similar problems or people from Google can explain what's the
prb?

Best Regards,
Eric

On 26 October 2010 10:49, Eli Jones eli.jo...@gmail.com wrote:

 Oh, the error message it shows when I run appcfg.py update is:

 Your app was updated, but there was an error updating your indexes. Please
 retry later with appcfg.py update_indexes.


 On Mon, Oct 25, 2010 at 10:48 PM, Eli Jones eli.jo...@gmail.com wrote:

 I am able to update my app code with appcfg.py update.. but it always
 throws an error when it gets to the upload indexes part.

 If I try to only update_indexes, I keep getting the error as well:

 Server: appengine.google.com.
 Uploading index definitions.
 Error 500: --- begin server output ---

 Server Error (500)
 A server error has occurred.
 --- end server output ---


 I presume this is a transient issue and will be cleared up soon.. but I
 have nothing else to do while I wait.. so I am posting a message here in
 case some bored soul is watching the list.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] NeedIndexError: The built-in indices are not efficient enough (related to large result set for self merge-join?)

[Eric Ka Ka Ng]

Thx Robert. Your reference is always good materials for me to study. Would
study it and see if can come up with any solution to solve my problem
following Alfred's optimization! And if still cannot, would bring up the
qeustion again ;)

Best Regards,
Eric

On 12 October 2010 00:48, Robert Kluin robert.kl...@gmail.com wrote:

 Hey Eric,
  Perhaps you can come up with a method to identify the 'best' search
 parameters when deciding how to filter your items.

  You should also check out Alfred Fuller's IO talk, next gen queries
 on app engine.

 http://code.google.com/events/io/2010/sessions/next-gen-queries-appengine.html

  They are now starting to support some of those items.  From the
 1.3.8 (pre)release notes:
 - Removed limits on zigzag merge-join queries. Therefore the
 error The built-in
indices are not efficient enough for this query and your data.
 Please add a
composite index for this query. will no longer be thrown in many
 cases,
   enabling more types of exploratory queries without indexes.


 Maybe some of Alfred's optimizations can help you.


 Robert





 On Mon, Oct 11, 2010 at 05:35, Eric Ka Ka Ng ngk...@gmail.com wrote:
  From the slide Building scalable, complex apps on App Engine by Brett
  Slatkin
 
 http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf
 
  we learn techniques for building scalable services by using list and self
  merge-join. self merge-join is very useful to me as it does not require
  building additional custom indexes (the hard limit of 200 custom indexes
 is
  always a headache for me with my application, and always have to use many
  different techniques to reduce the custom indexes usage)
 
  however, recently i hit a problem
  NeedIndexError: The built-in indices are not efficient enough for this
 query
  and your data. Please add a composite index for this query.
 
  My model is very simple, just like this
  class Searchable(db.Model):
  phrases = db.StringListProperty()
 
  the query is
  query = searchable.all().filter('phrases =', word1).filter('phrases =',
  word2).fetch(1000)
 
  It works fine before. But now fail with some cases.
  I re-read slides, and find that about merge-join performance
  Scales with number of filters and size of result set Best for queries
 with
  fewer results (less than 100)
 
  i guess very likely, now when my application scales with more data (let
 say
  Searchable now stores 1 million records), when doing the query and if the
  result set is large (how large is large?  100?), i would then hit this
  'built-in indices are not efficient enough' problem. the query works fine
 if
  the returned result set is much smaller
 
  this is my guessing. so my questions are
  1. is my guess correct / making sense? what other cases would
  cause  'built-in indices are not efficient enough' problem? are there any
  others share similar experiences?
  2. is there any hard number, or formula, we can use to estimate by how
 large
  the result set is, would hit the 'built-in indices are not efficient
 enough'
  problem? other than size of result set, would there be other factor
 matter?
  (number of self merge-join, number of properties, number of indexed
  properties etc.)
  3. the most important question, any solution / workaround for that ?
 (other
  than building extra custom indexes which is not quite feasible for me as
 i'm
  already having 200 indexes)
 
 
  appreciate for any comments / suggestions.
  - eric ng
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appeng...@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] NeedIndexError: The built-in indices are not efficient enough (related to large result set for self merge-join?)

[Eric Ka Ka Ng]

From the slide Building scalable, complex apps on App Engine by Brett
Slatkin
http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf


we learn techniques for building scalable services by using list and self
merge-join. self merge-join is very useful to me as it does not require
building additional custom indexes (the hard limit of 200 custom indexes is
always a headache for me with my application, and always have to use many
different techniques to reduce the custom indexes usage)


however, recently i hit a problem

*NeedIndexError: The built-in indices are not efficient enough for this
query and your data. Please add a composite index for this query.*


My model is very simple, just like this

class Searchable(db.Model):
phrases = db.StringListProperty()


the query is
query = searchable.all().filter('phrases =', word1).filter('phrases =',
word2).fetch(1000)


It works fine before. But now fail with some cases.

I re-read slides, and find that about merge-join performance

***Scales with number of filters and size of result set Best for queries
with fewer results (less than 100)*


i guess very likely, now when my application scales with more data (let say
Searchable now stores 1 million records), when doing the query and if the
result set is large (how large is large?  100?), i would then hit
this 'built-in
indices are not efficient enough' problem. the query works fine if the
returned result set is much smaller


this is my guessing. so my questions are

1. is my guess correct / making sense? what other cases would cause  'built-in
indices are not efficient enough' problem? are there any others share
similar experiences?

2. is there any hard number, or formula, we can use to estimate by how large
the result set is, would hit the 'built-in indices are not efficient enough'
problem? other than size of result set, would there be other factor matter?
(number of self merge-join, number of properties, number of indexed
properties etc.)

3. the most important question, any solution / workaround for that ? (other
than building extra custom indexes which is not quite feasible for me as i'm
already having 200 indexes)



appreciate for any comments / suggestions.

- eric ng

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Super High Latency!! many 500 response! What happen to GAE?

[Eric Ka Ka Ng]

i'm NOT using Django so i guess it's not related to importing Django modules

the problem for me, was even for some very simple request which usually
takes 50ms, would still be facing deadlineExceedError.

but one common properties for all these long latency calls are, yes, they
are retrieving something from DS, so i think it's sth related to DS get

the problem happened yesterday, and now seems the latency is getting normal.
not sure if G has done anything about it...


On 23 September 2010 13:45, Raymond C. windz...@gmail.com wrote:

 Really disappointed that after 24 hours and still no one from Google
 is responding, or at least tell us someone is looking at it...

 On Sep 23, 10:46 am, ego008 ego...@gmail.com wrote:
  Same here.http://gaefons.appspot.com/
 
  2010/9/22 James Saull james.sa...@gmail.com
 
 
 
 
 
   I am using Django as is bundled by default by GAE and I am getting
   failures again today. It happened for most of yesterday - was fine
   this morning and now failing again. Even the simple pages which
   normally take a few cpu cycles. I can access the dashboard and view my
   data from there no problem.
 
   On Sep 22, 3:18 pm, Blixt andreasbl...@gmail.com wrote:
It's probably part due to using Django 1.1, since you are the third
person reporting this problem using Django 1.1 (assuming you are
 using
it :)
 
On Sep 22, 3:52 pm, Fredrik Bonander 
 carl.fredrik.bonan...@gmail.com
wrote:
 
 Same here. Could this be related to using django 1.1?
 
 ..fredrik
 
 On Sep 22, 2010, at 11:10 AM, Blixt wrote:
 
  Yup, same here. Maybe related to:
  -
  http://groups.google.com/group/google-appengine/browse_thread/thread/.
 ..
  and
  -
  http://groups.google.com/group/google-appengine/browse_thread/thread/.
 ..
 
  On Sep 22, 3:52 am, Eric Ka Ka Ng ngk...@gmail.com wrote:
  Have great troubles in serving requests today, even with the
 very
   simple one
  which usually takes 50ms!! What's wrong with GAE? Any system
   maintenance or
  incident happening?
 
  here is one of the log
 
   500 10530ms 0cpu_ms 0kb
  09-21 06:48PM 05.643
 
  Request was aborted after waiting too long to attempt to service
   your
  request. This may happen sporadically when the App Engine
 serving
  cluster is under unexpectedly high or uneven load. If you see
 this
  message frequently, please contact the App Engine team.
 
  - eric
 
  --
  You received this message because you are subscribed to the
 Google
   Groups Google App Engine group.
  To post to this group, send email to
   google-appeng...@googlegroups.com.
  To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib
 e...@googlegroups.com
   .
  For more options, visit this group athttp://
   groups.google.com/group/google-appengine?hl=en.
 
 --
 Fredrik Bonander
 carl.fredrik.bonan...@gmail.com
 +46 70 943 5441
 
 - the infinite power of the creative mind -
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%2Bunsubscrib
 e...@googlegroups.com
   .
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
  --
  GAE粉丝团http://gaefons.appspot.com/

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Super High Latency!! many 500 response! What happen to GAE?

[Eric Ka Ka Ng]

Have great troubles in serving requests today, even with the very simple one
which usually takes 50ms!! What's wrong with GAE? Any system maintenance or
incident happening?

here is one of the log


 500 10530ms 0cpu_ms 0kb
09-21 06:48PM 05.643

Request was aborted after waiting too long to attempt to service your
request. This may happen sporadically when the App Engine serving
cluster is under unexpectedly high or uneven load. If you see this
message frequently, please contact the App Engine team.




- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Stuck indexes

[Eric Ka Ka Ng]

Hi, my app id is 'dvmmyws'. there are some indexes being stuck in the state
' building' for quite a long time. Can help to move to error state so that i
can vaccum? Thx!

- eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] number of indexes for billing enabled app is 200 or 300?

[Eric Ka Ka Ng]

a quick question.

Sometimes before the number of indexes for billing enabled our app is 300.
Now it goes back to 200. Could anyone advise if the cut back is temp.? or is
there any ways we could request to extend to 300 or beyond 200?

Regards,
Eric

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: quota of number of indexes for billing-enabled app. still 200?

[Eric Ka Ka Ng]

Sorry to re-iterate this email. Please kindly to see if someone from Google
can answer. Thanks in advanced

Best Regards,
Eric Ng

On 13 August 2010 14:33, Eric Ka Ka Ng ngk...@gmail.com wrote:

 Hi,

 It was known that the original quota of number of indexes, 100 for
 billing-disabled app, and 200 for billing-enabled app, are quite
 limiting for some types of applications, and such restriction is not
 extendable even we as users want to pay for it. This really posts
 difficulties, technical challenges, and limitations when we were
 designing and implementing applications on GAE.

 Starting from about last month, we notice that the quota for our
 billing-enabled app raised to 300! This is REALLY a great news for us,
 and expanded lots of possibilities to us (although ideally we hope #
 of indexes can be 'pay as you need'  attributes, at least extending
 100 more to us is good) Therefore, we started to plan and expand our
 features to utilize the extra 100 indexes.


 However, suddenly we find that the no. of indexes suddenly drop back
 to 200! This would be a nightmare for us as some of our new features
 would already need the 200+th indexes

 Could someone from Google or other users having similar experiences
 explains this? Although I understand Google may never officially
 announce that number of indexes for billing-enabled app would be
 increased to 300. we did find this change in our app admin console
 quota page before and we didn't expect it would suddenly drop back.
 Please kindly help to see if it could be possible to keep them to 300,
 even we need to pay extra for such option.

 Best Regards,
 Eric Ng

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] quota of number of indexes for billing-enabled app. still 200?

[Eric Ka Ka Ng]

Hi,

It was known that the original quota of number of indexes, 100 for
billing-disabled app, and 200 for billing-enabled app, are quite
limiting for some types of applications, and such restriction is not
extendable even we as users want to pay for it. This really posts
difficulties, technical challenges, and limitations when we were
designing and implementing applications on GAE.

Starting from about last month, we notice that the quota for our
billing-enabled app raised to 300! This is REALLY a great news for us,
and expanded lots of possibilities to us (although ideally we hope #
of indexes can be 'pay as you need'  attributes, at least extending
100 more to us is good) Therefore, we started to plan and expand our
features to utilize the extra 100 indexes.


However, suddenly we find that the no. of indexes suddenly drop back
to 200! This would be a nightmare for us as some of our new features
would already need the 200+th indexes

Could someone from Google or other users having similar experiences
explains this? Although I understand Google may never officially
announce that number of indexes for billing-enabled app would be
increased to 300. we did find this change in our app admin console
quota page before and we didn't expect it would suddenly drop back.
Please kindly help to see if it could be possible to keep them to 300,
even we need to pay extra for such option.

Best Regards,
Eric Ng

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Extremely long deployment wait

[Eric Ka Ka Ng]

too bad it even starts raising the following error:

Error 403: --- begin server output ---

Too Many Versions (403)
The application already has the maximum number of versions.
--- end server output ---

could someone from google help?


On 23 June 2010 15:45, Siegfried Hirsch siegfried.hir...@gmail.com wrote:
 +1
 the same here

 On Wed, Jun 23, 2010 at 6:44 AM, Eric Ka Ka Ng ngk...@gmail.com wrote:
 me 2. I experienced these problems in recent few days. Each time, the
 problem lasts for a hour or two. This annoys us very much as we can't
 do any deployment and developmenti'm not sure if this problem
 happens just for some applications, or all applications.  could anyone
 from google team tell us whats going on and when these problems would
 be fixed?


 Regards,
 Eric Ng Ka Ka


 On 23 June 2010 12:27, tav t...@espians.com wrote:
 Sadly, yes.

 On Jun 23, 5:24 am, renderpaz kev...@gmail.com wrote:
 Anyone else seeing 5+ min wait deploying python apps?

 Kevin

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.





 --
 Siegfried Hirsch
 Neue Anschrift: hhS - Lipowskystr. 16 - 81373 München - (089) 5484
 3564 - skype:shirsch
 Fax +49 - (0)89 - 943 992 698 - http://www.rss-blogger.de
 Bei Google Buzz? Hier gibts die http://www.buzz-charts.com

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Maximum number of versions

[Eric Ka Ka Ng]

i got the same problem and i hv 6 versions only.

recently the deployment time is very slow, and can often fails. when
these problems happens, the Maximum number of versions problem may
occur too. i guess there are something wrong with the deployment
system now...


On 23 June 2010 17:02, iceanfire iceanf...@gmail.com wrote:
 You didn't properly read his question. He said he had less than 10
 versions so the limitation shouldn't apply here. Anyways, I had
 the same problem, I currently have 7 versions (after deleting two) and
 I still get that error.

 On May 27, 3:26 pm, Aurelian aurelian.vacar...@googlemail.com wrote:
 it's quite normal, because the maximum number of applications is 10..

 i keep only 8 versions, and always delete the oldest.

 ciao..

 On May 27, 6:42 pm, Jairo Vasquez jairo.vasq...@gmail.com wrote:



  Hi,

  This error is very common.

  Error 403: --- begin server output ---

  Too Many Versions (403)
  The application already has the maximum number of versions.
  --- end server output ---

  And I only have 9 versions right now. Usually I have less versions than 
  that
  and sometimes the error appear.

  [image: icon.gif] Jairo Vasquez
  *        Paymentez Lead Developer*
        O +1.786.975.2339 x 213
        M +57.311.303.8417
        Medellín, Colombia

   icon.gif
   1KViewDownload

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Extremely long deployment wait

[Eric Ka Ka Ng]

me 2. I experienced these problems in recent few days. Each time, the
problem lasts for a hour or two. This annoys us very much as we can't
do any deployment and developmenti'm not sure if this problem
happens just for some applications, or all applications.  could anyone
from google team tell us whats going on and when these problems would
be fixed?


Regards,
Eric Ng Ka Ka


On 23 June 2010 12:27, tav t...@espians.com wrote:
 Sadly, yes.

 On Jun 23, 5:24 am, renderpaz kev...@gmail.com wrote:
 Anyone else seeing 5+ min wait deploying python apps?

 Kevin

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Number of indexes

[Eric Ka Ka Ng]

Hi Shinichi,

we can vacuum_indexes for those indexes in 'Serving' state. however,
there are cases that the index being stuck in Building state. In
such case, no luck and have to ask people from Google to help to
change the state of those indexes to Error and we could then run
vacuum_indexes to remove them

- eric

On 15 March 2010 15:06, Shinichi Nakanishi stouton...@gmail.com wrote:
 I didn't even notice there were zombie indexes.

 I ran appcfg with vacuum_indexes option.  I deleted some zombie
 indexes, but still a lot of indexes which status is Building.

 I have run appcfg vacuum_indexes for several times, but no luck so far.

 Can we delete these indexes with status Building?

 Shinichi

 On Sun, Mar 14, 2010 at 20:39, Eric Ka Ka Ng ngk...@gmail.com wrote:
 Hi,

 I think you can use the 'vacuum index' command to remove indexes no
 longer required, then you wont have 'zombie' ;)

 - eric

 On 15 March 2010 11:31, Iap iap...@gmail.com wrote:
 Hi,
 I will eventually have the same problem because I have many zombie
 indexes.
 These indexes dose not been removed even it was omitted from the index.yaml.
 (The schema of Model has been changing during the developing ,
 that is the reason why some indexes becomes zombie)

 2010/3/15 Eric Ka Ka Ng ngk...@gmail.com

 Hi Shinichi,

 I had exactly the same problem as yours. As far as I know, currently
 billing disabled app has limitation on max. 100 indexes. Billing
 enabled app has limitation on max. 200 indexes. These numbers are not
 billable and there are no ways we could further increase this number
 ,even if the app needs it and we are willing to pay more for the
 required resources. Please correct me if that's not the case anymore
 now.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Number of indexes

[Eric Ka Ka Ng]

Hi Shinichi,

I had exactly the same problem as yours. As far as I know, currently
billing disabled app has limitation on max. 100 indexes. Billing
enabled app has limitation on max. 200 indexes. These numbers are not
billable and there are no ways we could further increase this number
,even if the app needs it and we are willing to pay more for the
required resources. Please correct me if that's not the case anymore
now.

- eric

On 14 March 2010 10:47, Shinichi Nakanishi stouton...@gmail.com wrote:
 Hi.

 The number of indexes of our application has reached 100.

 Is it possible to increase this number if we enable billing?

 Shinichi

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Number of indexes

[Eric Ka Ka Ng]

Hi,

I think you can use the 'vacuum index' command to remove indexes no
longer required, then you wont have 'zombie' ;)

- eric

On 15 March 2010 11:31, Iap iap...@gmail.com wrote:
 Hi,
 I will eventually have the same problem because I have many zombie
 indexes.
 These indexes dose not been removed even it was omitted from the index.yaml.
 (The schema of Model has been changing during the developing ,
 that is the reason why some indexes becomes zombie)

 2010/3/15 Eric Ka Ka Ng ngk...@gmail.com

 Hi Shinichi,

 I had exactly the same problem as yours. As far as I know, currently
 billing disabled app has limitation on max. 100 indexes. Billing
 enabled app has limitation on max. 200 indexes. These numbers are not
 billable and there are no ways we could further increase this number
 ,even if the app needs it and we are willing to pay more for the
 required resources. Please correct me if that's not the case anymore
 now.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: how large data can i use in App Caching and how long will it be cached?

[Eric Ka Ka Ng]

Hi dburns,

according to 
http://code.google.com/intl/en/appengine/docs/python/runtime.html#App_Caching

handler script with main() would cache itself and the global env as an
imported module

however, in my case, I have a variable defined inside a module that
would be imported (and cached) , but i would like to have that
variable be re-initialized in every requests. In particular, with this
example


### mymodule.py
counter = 0
def increment():
global counter
counter += 1
return counter


### myhandler.py
import mymodule

print Content-Type: text/plain
print 
print My number:  + str(mymodule.increment())
print My number:  + str(mymodule.increment())


the variable counter is cached  and subsequent requests use the
value from the previous request
but for my case, I would like to disable the caching and to have
counter being re-initialized to 0 in every request. The 1st
str(mymodule.increment()) always gives 1, and the 2nd always gives 2

any way to achieve this?

- eric


On 14 February 2010 01:52, dburns drrnb...@gmail.com wrote:
 In Python at least, GAE looks for a function called main() to enable
 app caching.  Simply rename main() to something else.


 On Feb 13, 6:41 am, Eric Ka Ka Ng ngk...@gmail.com wrote:
 is it possible to 'disable' the app caching behavior?

 - eric

 On 12 February 2010 17:48, saintthor saintt...@gmail.com wrote:

  see the demo in this page:http://code.google.com/intl/en/appengine/
  docs/python/runtime.html#App_Caching

  ### mymodule.py
  counter = 0
  def increment():
     global counter
     counter += 1
     return counter

  ### myhandler.py
  import mymodule

  print Content-Type: text/plain
  print 
  print My number:  + str(mymodule.increment())

  do you mean if the site has not accessed for some minutes, counter
  will be reset to 0?

  On 2月12日, 下午4时24分, Tim Hoffman zutes...@gmail.com wrote:
  App caching could last as little as a few minutes if your site is not
  used.
  In addition if multiple instances are run then only one instance will
  have the counter with the correct value.

  You should store your obj in the datastore and cache it in memcache.

  module level caching is really only useful for cacheable things for
  each instance,
  for example compiled templates.

  T

  On Feb 12, 3:34 pm, saintthor saintt...@gmail.com wrote:

   ### mymodule.py
   counter = LargeObj()

   ### myhandler.py
   import mymodule

   print Content-Type: text/plain
   print 
   print My number:  + str(mymodule.counter)

   if sizeof counter is greater than 1M, can it work?

   if there is no request for days, will counter still be cached?

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appeng...@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.



 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: how large data can i use in App Caching and how long will it be cached?

[Eric Ka Ka Ng]

great thx to Darren and Stephen! Stephen's solution is very simple and
easy, and it works for me. I believe Darren's one works too but
finally i choose Stephen's one as this gives finer control and I can
adjust when to retain the counter value, when to reset, without
scarifying the performance while the module can still be cached. thx
again to all of you

- eric


On 15 February 2010 00:49, Stephen sdea...@gmail.com wrote:


 On Feb 14, 11:30 am, Eric Ka Ka Ng ngk...@gmail.com wrote:

 however, in my case, I have a variable defined inside a module that
 would be imported (and cached) , but i would like to have that
 variable be re-initialized in every requests. In particular, with this
 example

 ### mymodule.py
 counter = 0
 def increment():
     global counter
     counter += 1
     return counter

 ### myhandler.py
 import mymodule

 print Content-Type: text/plain
 print 
 print My number:  + str(mymodule.increment())
 print My number:  + str(mymodule.increment())

 I would like to disable the caching and to have
 counter being re-initialized to 0 in every request.

 Any way to achieve this?



 ### mymodule.py

 counter = 0

 def increment():
    global counter
    counter += 1
    return counter

 de reset():
    global counter
    counter = 0


 ### myhandler.py

 import mymodule

 def main():
    mymodule.reset()
    respond()

 def respond():
    print Content-Type: text/plain
    print 
    print My number:  + str(mymodule.increment())
    print My number:  + str(mymodule.increment())

 if __name__ == '__main__':
    main()

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: how large data can i use in App Caching and how long will it be cached?

[Eric Ka Ka Ng]

Sorry for a side-tracked question and if it's not appropriate i would
send another email for the question.

for exactly the same global counter as in the following example, what
I would like to do is to AVOID the app caching behavior. That is to
say, for different requests handled by myhandler.py, i would like each
request having their local value of 'counter', where 'counter' is a
variable defined in an imported module (as in mymodule.py)

is it possible to 'disable' the app caching behavior?

- eric



On 12 February 2010 17:48, saintthor saintt...@gmail.com wrote:
 see the demo in this page:http://code.google.com/intl/en/appengine/
 docs/python/runtime.html#App_Caching

 ### mymodule.py
 counter = 0
 def increment():
    global counter
    counter += 1
    return counter


 ### myhandler.py
 import mymodule

 print Content-Type: text/plain
 print 
 print My number:  + str(mymodule.increment())


 do you mean if the site has not accessed for some minutes, counter
 will be reset to 0?


 On 2月12日, 下午4时24分, Tim Hoffman zutes...@gmail.com wrote:
 App caching could last as little as a few minutes if your site is not
 used.
 In addition if multiple instances are run then only one instance will
 have the counter with the correct value.

 You should store your obj in the datastore and cache it in memcache.

 module level caching is really only useful for cacheable things for
 each instance,
 for example compiled templates.

 T

 On Feb 12, 3:34 pm, saintthor saintt...@gmail.com wrote:

  ### mymodule.py
  counter = LargeObj()

  ### myhandler.py
  import mymodule

  print Content-Type: text/plain
  print 
  print My number:  + str(mymodule.counter)

  if sizeof counter is greater than 1M, can it work?

  if there is no request for days, will counter still be cached?

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Scalability problem in GAE

[Eric Ka Ka Ng]

I have also installed this lib and am having fun with it. personally i
'feel' the overhead is really insignificant

refer to your diagram,

i'm not sure whether the blue bars are actually showing time without
the profiler, i think it may mean something else (i also wanna know
what the blue bar and red rectangle refer to, and will dig into the
doc to find out what it actually means. anyone know?). the overhead
should be the time labeled with 'overhead' (in your case, which is
0ms)

my personal experience is, for most calls, the 'overhead' is always
1% of the 'real'

- eric

2010/1/20 Danny Tuppeny da...@tuppeny.com:
 I haven't used this in production yet (I only read that you can), though I'd
 avoid leaving it on for long periods if you don't need it.
 When running in production, it will show two bars - the one showing how much
 time was taken without the profiler, and one showing how long was actually
 taking (including the profiler overhead, since that's what you'll be charged
 with). There's a screenshot here:
 http://2.bp.blogspot.com/_DkVkyz39PB0/SyLYRJPlsgI/Deg/5FiuJi3SpjQ/s1600-h/スクリーンショット（2009-12-12+8.39.08）.png
 The overhead isn't insignificant, so you probably don't want to leave the
 profiler turned on, but the blue bars show the time without the profiler, so
 the results are still accurate :-)
 Danny

 2010/1/20 Eric Ka Ka Ng ngk...@gmail.com

 wow!! this is THE TOOL that I'm looking for long time!! previously i
 have to write codes to do similar profiling and analysis with much
 limited scales and functions. Now, THIS provides almost everything I
 want already without any efforts from developers! I could say it is
 one of the best tool on GAE. Great Work!!

 how about the overhead? from the doc I know we already can read from
 the INFO log about its overhead for recording the RPC events and
 storing it. for normal applications, what's your experiences or
 expectations on the added overhead? (like 1% CPU time increase?) if
 the overhead is relatively small, I think we can include it even in
 the production system without noticeable performance impact.


 many thanks,
 eric

 2010/1/19 mani doraisamy mdorais...@orangescape.com:
  I guess the reason why this is so damn good is because it gives an
  integrated picture across datastore, memcache and web request, without
  much overhead! Given that the variations are fixed (GAE's limitation
  of not allowing all libraries/api/rpc calls), this gives the full
  picture of the application.
 
  On Jan 19, 4:18 pm, mani doraisamy mdorais...@orangescape.com wrote:
  Thanks for the link, Danny. This is the best profiling tool i have
  used. Well done, Guido!
 
  Will get back with the performance data for off-peak Vs peak hours.
 
  thanks,
  mani
 
  On Jan 17, 9:37 pm, Danny Tuppeny da...@tuppeny.com wrote:
 
 
 
   Have you identified what part of your page is taking additional time?
 
   Guido van Rossum wrote a library called Appstats to help profile time
   spent on API calls. I'd recommend setting it up (it works in
   production) to see if you can identify where this time is being
   spent.
 
   I wrote a short article about Appstats on my blog with some
   screenshots showing the sort of data you can get out of it:
 
 
   http://blog.dantup.com/2010/01/profiling-google-app-engine-with-appstats
 
   Hope this helps.
 
   Danny
 
   On Jan 16, 8:24 pm, mani doraisamy mdorais...@orangescape.com
   wrote:
 
Recently, we have been running into frequent performance problems,
especially between 7-10pm IST. Requests that used to take 600ms are
taking almost 4 secs.
 
- Has anyone faced similar problems recently?
- What is the limit for memcache per account? How do we find the
cache
expiry pattern? (actual expiry Vs specified expiry)
 
thanks,
mani
 
  --
  You received this message because you are subscribed to the Google
  Groups Google App Engine group.
  To post to this group, send email to google-appeng...@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
 
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed

Re: [google-appengine] Re: Scalability problem in GAE

[Eric Ka Ka Ng]

yes exactly. I think this is the correct interpretation of the blue
line and red line.

- eric

2010/1/21 Robert Kluin robert.kl...@gmail.com:
 The blue line is wall-clock time.  The red line is CPU or API time.
 As Danny mentioned, the blue line indicates blocking time of each
 call.  The red line indicates what you are charged for each.

 Robert








 On Wed, Jan 20, 2010 at 12:43 PM, Danny Tuppeny da...@tuppeny.com wrote:
 2010/1/20 Danny Tuppeny da...@tuppeny.com

 2010/1/20 Eric Ka Ka Ng ngk...@gmail.com

 i'm not sure whether the blue bars are actually showing time without
 the profiler, i think it may mean something else (i also wanna know
 what the blue bar and red rectangle refer to, and will dig into the
 doc to find out what it actually means. anyone know?). the overhead
 should be the time labeled with 'overhead' (in your case, which is
 0ms)

 The screenshot I saw said blue was time taken and red was time charged,
 which I assumed meant the *actual* time taken with the overhead. It's
 possible it meant that the timings were rounded for the costing, I don't
 know. I fixed what I was doing, so I didn't investigate any further.

 I just saw the screenshots posted above, which makes me think I'm wrong. It
 looks more like the red bar is the time taken (and charged), but the blue
 bar is the time the app spent blocked waiting for the return. Eg. your
 script waited for 5 seconds for the response, but for some reason
 (transmission time? blocking?) it was only 1 second of CPU in the datastore.
 Hopefully someone knowledgeable might know for sure :-)
 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.





-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: question is regarding billable resources:

[Eric Ka Ka Ng]

Hi Wooble,

how about the number of custom indexes? as far as i know, it's not a
billable option yet, and an billing-enabled app should have a higher
quota for this (but yet we can't pay to increase such quota)

when our apps is getting more complicated and covering more aspects, I
can see the number of custom indexes would be increasing too, and very
soon we will be hitting the limit which we can't pay to overcome.

(or can we get more quota if the app is close to this limit by
contacting Google?)

best regards,
eric


2010/1/19 Wooble geoffsp...@gmail.com:
 These other quotas are not billable and can not be increased by paying
 for more.  I believe you may be able to get more quota if you app is
 close to these limits by contacting Google.

 On Jan 13, 3:39 am, Dmitri srey...@gmail.com wrote:
 HI All,
 My question is regarding billable resources:

 As per the faq - When billing is enabled, you can split your budget
 between five billable resources:

     * CPU Time
     * Bandwidth in
     * Bandwidth out
     * Storage
     * Email

 How about the  Requests,Datastore API Calls - what is my app exceeds
 ( 4320 Requests , Datastore API Calls 141241791 ) . I have chat
 feature in my app and i intend to do it by polling the database.

 Regards

 Dmitri

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.





-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Scalability problem in GAE

[Eric Ka Ka Ng]

wow!! this is THE TOOL that I'm looking for long time!! previously i
have to write codes to do similar profiling and analysis with much
limited scales and functions. Now, THIS provides almost everything I
want already without any efforts from developers! I could say it is
one of the best tool on GAE. Great Work!!

how about the overhead? from the doc I know we already can read from
the INFO log about its overhead for recording the RPC events and
storing it. for normal applications, what's your experiences or
expectations on the added overhead? (like 1% CPU time increase?) if
the overhead is relatively small, I think we can include it even in
the production system without noticeable performance impact.


many thanks,
eric

2010/1/19 mani doraisamy mdorais...@orangescape.com:
 I guess the reason why this is so damn good is because it gives an
 integrated picture across datastore, memcache and web request, without
 much overhead! Given that the variations are fixed (GAE's limitation
 of not allowing all libraries/api/rpc calls), this gives the full
 picture of the application.

 On Jan 19, 4:18 pm, mani doraisamy mdorais...@orangescape.com wrote:
 Thanks for the link, Danny. This is the best profiling tool i have
 used. Well done, Guido!

 Will get back with the performance data for off-peak Vs peak hours.

 thanks,
 mani

 On Jan 17, 9:37 pm, Danny Tuppeny da...@tuppeny.com wrote:



  Have you identified what part of your page is taking additional time?

  Guido van Rossum wrote a library called Appstats to help profile time
  spent on API calls. I'd recommend setting it up (it works in
  production) to see if you can identify where this time is being spent.

  I wrote a short article about Appstats on my blog with some
  screenshots showing the sort of data you can get out of it:

 http://blog.dantup.com/2010/01/profiling-google-app-engine-with-appstats

  Hope this helps.

  Danny

  On Jan 16, 8:24 pm, mani doraisamy mdorais...@orangescape.com wrote:

   Recently, we have been running into frequent performance problems,
   especially between 7-10pm IST. Requests that used to take 600ms are
   taking almost 4 secs.

   - Has anyone faced similar problems recently?
   - What is the limit for memcache per account? How do we find the cache
   expiry pattern? (actual expiry Vs specified expiry)

   thanks,
   mani

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.





-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Running some code immediately before a Datastore put()?

[Eric Ka Ka Ng]

would a PreCallHook works for your case?

http://code.google.com/appengine/articles/hooks.html

- eric

2010/1/14 Nickolas Daskalou n...@daskalou.com:
 I have a property of a Model that is a hash of another property of the same
 Model, eg:

 class MyModel(db.Model):
   something = db.StringProperty()
   something_hash = db.StringProperty()
   def generate_hash(self):
     self.something_hash = sha1(self.something)

 I want generate_hash() to be automatically called just before the entity is
 put() into the Datastore, without the developer having to explicitly call
 entity.generate_hash().

 I know I can create a put() method that makes such a call and then calls
 put() on the superclass, but (a) is that the best way to do it, and (b) if
 the put() is part of a batch put (eg. via db.put(entities)), will the put()
 method of each model instance still be called?

 Nick

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Bulk Updates

[Eric Ka Ka Ng]

how about batch update using db.put()?

following is extracted from
http://googleappengine.blogspot.com/2009/06/10-things-you-probably-didnt-know-about.html


For example, take a look at this common pattern:

for entity in MyModel.all().filter(color =,
old_favorite).fetch(100):
  entity.color = new_favorite
  entity.put()


Doing the update this way requires one datastore round trip for the
query, plus one additional round trip for each updated entity - for a
total of up to 101 round trips! In comparison, take a look at this
example:

updated = []
for entity in MyModel.all().filter(color =,
old_favorite).fetch(100):
  entity.color = new_favorite
  updated.append(entity)
db.put(updated)

By adding two lines, we've reduced the number of round trips required
from 101 to just 2!



- eric


2010/1/14 Patrick Twohig patr...@namazustudios.com:
 So I'm looking at trying to reset large amounts of data en masse.  Say I
 want to reset a counter on every account each week, how would I go about
 implementing something like that?  Would I have to go through each object,
 update it, and store it?  If that's the case, how do I go about doing so
 without killing my quota or running up my usage extremely high?

 Thanks,
 Patrick.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: This App Engine is a JOKE

[Eric Ka Ka Ng]

well, for me, i don think i could judge how much programming
experience others are having. what i can contribute is my answers /
experiences to some objective questions

question no.1 - process running for more than 30 seconds
sure, whatever kinds of batch processing, huge updates, urlfetch for
thousands of URLs etc. can take long long time. but this is just NOT
what GAE built for. it doesn't allow (at least currently) any task
executing for more than 10s. the intent is very clear, it is suitable
for hosting web services / frontend that user directly interacts with,
and will not expect to get the response 10s, but not for batch
processing

question no. 2 - can appengine scale?
this question should be refined to can appengine scale for the tasks
what it is designed and supposed to do well (for types of tasks (e.g.
batch processing) not suitable for GAE, then we dont bother to know
whether it can scale)
from what google claims, the 'scalability' lies on the fact that the
underlying BigTable performance is about constant, regardless how
large the datasize is. so the response time is not increasing with
increasing number of data, but it does increase for increasing number
of returned data. from this perspective, gae doesn't scale if the
returned data keep increasing, or doesn't [work well] with badly
written queries (model designed) (with bad implementation, no matter
there are 100 or 100 millions records, query time can take the same 10
seconds)

question no.3 - is it cloud computing, or over rate-limiting customers
well, cloud computing is not the god, and each platform is designed
for some specific use and no single platform can serve and do well for
all purposes. if the platform is suitable for what i want to do, scale
well in it, and bill me according to the usage, i could consider it as
a good cloud computing platform.

- eric

2010/1/14 Locke locke2...@gmail.com:
 Nick, you must have very little experience programming if you can't
 think of a reason for a process to run for more than thirty seconds.
 Ever hear of background tasks? Cron? TaskQueue? Ever use URLFetch?
 That eats your execution time no matter how efficient your app is.
 Right now, appengine just does not scale. There are way too many hard
 limits and rate limits.

 It makes sense to rate-limit free users. It does NOT make sense to
 rate-limit paying customers. Cloud computing is supposed to scale.



 On Jan 13, 6:19 pm, nickmilon nickmi...@gmail.com wrote:
 Thanasis

 take it easy - it was just a joke (may be of a bad taste but, as you
 mentioned it was not my day) , and nothing to do with your English the
 intention was just to make you think that sometimes we write bad code
 in a harry and then blame the infrastructure or whatever.

 Getting to GAE now, this late start symptom has been covered in other
 threads here and there are explanations - solutions which I will not
 repeat.
 I have huge pages with more than 60 KB of js and some more of CSS and
 3rd party js etc etc, and which I have not even optimized yet, still
 hot start or cold start never exceeds 3'' including DNS lookups for
 complete load  you can try it herehttp://www.geognos.com/geo/en/cc/af.html
 or try  http://milon.appspot.com/stravon/db/cc/83.212.217.149latest
 performs a fetch from 100K+ records that include the complete IP
 allocations database and since nobody is using it  you can be sure you
 will get a cold start.

 Besides, I have seen other people here complain about their appls been
 slow and I verified it my self, the miracle is same applications
 became very fast in a couple of weeks time - I do not think the
 infrastructure improved that much meanwhile, just they have rewritten
 their code in a better way.

 Now this is my experience and that of many others with whom I talk
 about GAE, I can't exclude the possibility that you people complaining
 have a kind of bad lack and your Apps ride on a bad server or anything
 else.
 Still I think you got to know better GAE before you start
 complaining,  study the docs read the previous posts and if you have
 something special to solve come back here and describe your problem
 without any remarks, I am sure you will find people willing to help,
 or...  you are free to switch to a conventional hosting platform if
 you feel better there, staying here just complaining that GAE is joke
 it is not helpful neither too you nor to the community.

 Happy coding

 Nick

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe 

Re: [google-appengine] Re: This App Engine is a JOKE

[Eric Ka Ka Ng]

 1) There are no data-intensive applications that do not at some point
 require batch processing.  No matter how clever you are in designing
 your application, at some point you will need to run a big query to
 calculate statistics, change the model, clean the data, etc.  If you
 haven't had to do this yet you haven't run your app long enough

yes, i do agree data-intensive applications very likely needs some
data cleaning tasks, and from times to times the model needs to be
updated. to do those 'data cleaning' and model updating tasks, similar
to your approaches, i use the techniques to split the large task into
multiple small tasks




 2) Appengine *does* support batch processing.  It just does so in a
 very clunky and akward way:  the task queue.  You write a task that
 crunches some data, watches for when it gets near the 30s deadline,
 and then respawns itself to pick up where it left off.

ya, and task queue for me, is a queue, which enable periodic /
scheduled tasks to be executed, and can make my system to be more
asynchronized  (response to the users quickly and put some tasks into
the queue which can be executed later). i'm not sure how the task
queue makes batch processing becoming easy on GAE. (your approach is
also some kind of breaking a large task into small tasks, and those
smaller tasks can be executed directly without putting into a queue)
so for me, GAE is still not a platform suitable for 'batch processing'



 The question is not whether GAE supports batch processing - it does -
 it's just a question of how clumsy the API is.  It would be a lot less
 clumsy if the 30s limitation were removed.

as long as GAE post such limitations (30s for task in queue or 10s for
normal task) , i dont think GAE is intent for real batch processing
(all those techniques used by you and me are just some walk-around for
housekeeping tasks, but not the real service that can be provided to
general users)

- eric
-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Measures taken to speed index building

[Eric Ka Ka Ng]

Totally agree what Sudhir is suggesting. I think there are at least 2
things the index building can be improved

1. currently the index building is still a little bit buggy, sometimes
I (and I believe some others) would have indexes being stuck in
building state, when I believe there were errors there but for some
unknown reasons they can't be transited to Error state. And we, as
programmer, can do nothing to prevent or fix this. What we can do is
to send an email to google and ask them help changing the indexes to
Error state so that we can vacuum it.

2. putting index building tasks into the global queue is totally
understandable, but could we have a little smart feature that if the
system knows that the indexes to be built are on 'empty' model (or the
required CPU is very minimal if there are only a few entries), this
could be executed immediately without putting on the queue for
waiting. (or there can be multiple queues for light-weight or
heavy-duty tasks)

- eric

2010/1/3 Sudhir sudhi...@gmail.com:
 Absolutely... which is the problem. Indexes on *empty* datastores took
 7 hours last night. I understand fully that 7 hours might be expected
 for millions of objects and gigabytes of data, but not for a 76kb
 datastore with three objects, 80% of which is the datastore statistics
 and metadata. My app is still in development and there's nothing in
 there expect test1, test2 and test3.

 I've seen on the docs that index building might take 12h to 24h just
 waiting on the queue, irrespective of how much data actually needs to
 be indexed. While I appreciate the fact that the GAE team is being
 very open and honest about this, this will be a deal breaker for lots
 of people.

 I love the GAE idea, and I'm basing all my hopes of writing the next
 killer app and making a billion dollars on this platform, but stuff
 like this is a big turn off for anybody, I think. It completely takes
 the 'instant' and 'quick' out of the process, and I think it needs to
 be fixed before it gives GAE a bad rep.

 Sudhir

 On Jan 3, 4:53 am, Barry Hunter barrybhun...@googlemail.com wrote:
 A index built against a empty datastore (or at least a Model without
 any Entities) should always be available immediately. Or at least that
 is my experience. This makes sense because such an index doesnt
 actully need to go though the 'bulk build' pipeline (which can be a
 cause of delay if the pineline is busy - the index builder is a
 background process)

 of course adding Entities to this empty index will happen in realtime,
 so just add the data after deploying the index.

 2010/1/2 Sudhir sudhi...@gmail.com:



  Does anyone know what efforts are being taken to speed up index
  building and what their status is?

  I'm finding it very difficult to be a champion for GAE in my company -
  one of the (implicitly) advertised features of GAE is the whole click
  to deploy and start running feature. Things don't bode very well if
  I'm doing a demo for the clients / bosses and we all need sit and wait
  12 to 24 hours for GAE to index two strings that say Hello World and
  test.

  Its especially frustrating when I need to deploy and test during
  development (stuff like custom search doesn't work on localhost).

  I've been hitting refresh the last six hours.

  The production issues pilot is a great step in the right direction,
  and I filed an issue and all, but is the GAE team actually thinking of
  responding to 100,000 (and counting) developers?

  Just wondering, how on earth do you guys do demos at I/O? I don't
  think any of the presenters hit deploy and said okay, let's all come
  back tomorrow, or maybe the day after, and we'll see if this
  works :-/

  Sudhir

  --

  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appeng...@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

 --

 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] stuck index on app id 267526273

[Eric Ka Ka Ng]

I also have 7 composite indexes stuck in building in my app for more than
a week, please remove them

kind: OfferHistory

sellercategories, userid, offertime
sellercategories, userid, price
sellercategories, userid, price desc
sellercategories, sellerid, smileys, offertime
sellercategories, userid, sellerid, offertime
userid, itemkey, offertime
userid, sellerid, itemkey, offertime

Thanks,
Eric


2009/12/10 navku...@gmail.com navku...@gmail.com

 I have the following index stuck in Building in my app for several
 days, and I would like this index removed:

 - kind: UpdateSearchIndex
  ancestor: no
  properties:
   - name: organization
   - name: terms
   - name: terms
   - name: stamp
 direction: desc

 Thanks for the quick response

 Navin

 --

 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] index creation quota

[Eric Ka Ka Ng]

Hi,

I have an billing enabled application that hits the index creation
quota (which I think is about 100 from the threads I read). I
understand exceeding 100 composite indexes maybe a sign of bad schema
design. The main reason for high number of composite indexes is, for
some 'model' in the application, they contains about let say 7 fields
(which is not a high number), and these 7 fields can freely form
different filters with every combinations to answer users queries.

Please increase the index creation quota, or it is ok if we can pay as
a billable option for the overuse of indexes (i can't find any option
for this in the billing setting). Or please advise if there are any
alternative ways to implement filtering based on combinations of
fields without building the required composite indexes.

Our app id is 'dvmmyws'

Thanks in advanced and we are really urgently need such relaxation or
billable option

Regards,
Eric Ng

--

You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] index creation quota

[Eric Ka Ka Ng]

Hi Ikai,

May we know what would be that higher index quota? coz we are already
hitting the ceiling and not able to create anymore we need.

- eric

2009/12/5 Ikai L (Google) ika...@google.com

 Eric,

 Billing enabled apps have a higher index quota. If you enabled billing, you
 should have a higher ceiling for the number of indexes you can create.

 On Fri, Dec 4, 2009 at 1:22 AM, Eric Ka Ka Ng ngk...@gmail.com wrote:

 Hi,

 I have an billing enabled application that hits the index creation
 quota (which I think is about 100 from the threads I read). I
 understand exceeding 100 composite indexes maybe a sign of bad schema
 design. The main reason for high number of composite indexes is, for
 some 'model' in the application, they contains about let say 7 fields
 (which is not a high number), and these 7 fields can freely form
 different filters with every combinations to answer users queries.

 Please increase the index creation quota, or it is ok if we can pay as
 a billable option for the overuse of indexes (i can't find any option
 for this in the billing setting). Or please advise if there are any
 alternative ways to implement filtering based on combinations of
 fields without building the required composite indexes.

 Our app id is 'dvmmyws'

 Thanks in advanced and we are really urgently need such relaxation or
 billable option

 Regards,
 Eric Ng

 --

 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.





 --
 Ikai Lan
 Developer Programs Engineer, Google App Engine

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appeng...@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


--

You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.