[google-appengine] Re: Anyone else seeing a high percentage of 520 errors from Cloud Flare?
Did you get any joy? Find anything out from CF? On Sunday, October 13, 2013 7:12:50 PM UTC+2, Kaan Soral wrote: I've enabled warmup requests to see whether it would change anything After deploying changes, all instances were killed, as expected I've waited some instanced to spin up, they didn't Then I visited the app from the cloudflare url, and instances still didn't spin up, I got a 520 CF error again Then I visited the app through the *.appspot.com url, and an instance spinned up to serve the request So I think the issue might be with appengine, not responding to CF requests -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Anyone else seeing a high percentage of 520 errors from Cloud Flare?
The issue stopped right after I disabled Website Preloader on CF, might be coincidental with a low probability CF didn't reply to the ticket interesting turn of events On Tuesday, October 15, 2013 10:02:35 AM UTC+3, Richard Watson wrote: Did you get any joy? Find anything out from CF? On Sunday, October 13, 2013 7:12:50 PM UTC+2, Kaan Soral wrote: I've enabled warmup requests to see whether it would change anything After deploying changes, all instances were killed, as expected I've waited some instanced to spin up, they didn't Then I visited the app from the cloudflare url, and instances still didn't spin up, I got a 520 CF error again Then I visited the app through the *.appspot.com url, and an instance spinned up to serve the request So I think the issue might be with appengine, not responding to CF requests -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] how reduce memory usage on mapreduce controller callback
Thanks Jason, Me too, I starred this one long time ago. Saludos. Moisés Belchín. 2013/10/15 Jason Galea ja...@lecstor.com thanks Jason, yeh I starred that one a while back. On Mon, Oct 14, 2013 at 11:58 PM, Jason Collins jason.a.coll...@gmail.com wrote: You guys should star https://code.google.com/p/googleappengine/issues/detail?id=9610 In the past, I've found surprising amount of memory use when working with NDB, especially when using repeated properties. In my experiments, it was very easy to blow up instances with a seemingly small number of entities. I think it's wrapped up in protobuf deserialization which is low enough down the stack that no one seems to have much appetite to touch. j On Monday, 14 October 2013 04:14:05 UTC-6, Jason Galea wrote: Hi Moises, you can find all the details here.. https://developers.** google.com/appengine/docs/**python/ndb/cachehttps://developers.google.com/appengine/docs/python/ndb/cache The In-Context Cache The in-context cache persists only for the duration of a single incoming HTTP request and is visible only to the code that handles that request. It's fast; this cache lives in memory. When an NDB function writes to the Datastore, it also writes to the in-context cache. When an NDB function reads an entity, it checks the in-context cache first. If the entity is found there, no Datastore interaction takes place. My take-away is that the in-context cache is handy when different parts of your code are calling get() on the same entities and would certainly make things faster, but comes with the trade-off that all entities you touch are staying in memory until the request completes, even if you don't need them any more. With queries you're going to be loading all entities regardless so disabling the in-context cache alone likely won't help much. If you do a keys_only query, though, and get() each entity in turn, then disabling the in-context cache should reduce memory usage (assuming that some/all of the memory used by previous entities is able to be re-used..). Once again, though, you'll likely be sacrificing speed for less memory usage. This is mostly based on how I believe the different parts would/should work, I have no hard evidence.. Jason On Mon, Oct 14, 2013 at 5:36 PM, Moises Belchin moises...@gmail.comwrote: Hi Jason, Thanks for the detailed answer. I'm very surprised that no one else is talking about these issues. I'm using ndb and my appstats are off. I could see a incredible great improvement in my app when I turned off the stats. So, I recommend people only use stats for testing or debug. As you mentioned I'll try projection queries. Could you explain something more about NDB in-memory cache? Thanks again. Saludos. Moisés Belchín. 2013/10/12 Jason Galea ja...@lecstor.com Hi Moises, we're currently trying to deal with this issue too. Not in mapreduce, just regular handlers. same here - Python 2.7 and F2 instances with 256MB Early on I found that fetching 1000 entities and looping through them to update a property would blow the instance. Reducing this to say 100 fixed the issue. (than when create another task do do the next 100) After handling this request - as far as I understand this isn't so bad unless you're blowing it on every request and starting a new instance is detrimental. While handling this request - this concerns us most as the request does not complete and breaks stuff.. and we see far too many of them. I've spent more than a little time trying to work out what is causing the blowouts but as far as I've been able to work out, memory usage and what causes it is near impossible (or just very, very hard). Are you using NDB? if so.. - you could try disabling the in-memory cache. As I see it, even though you only access one entity at a time, NDB's in memory cache will store them all until the request is completed. - you could try projection queries if you don't need the complete object (or possibly even if you do). Projection queries get their data from an index and the entities returned cannot be put() so I assume they are not cached at all. We're trialling some fixes with these atm. ** If anyone knows any of this is incorrect, please let me know.. I'm actually surprised there is not more discussion of these issues from what we have experienced so maybe we're doing something fundamentally wrong, but I don't believe so. oh, is appstats turned on? I believe the most noticeable improvement we've seen was when we turned it off.. regards, Jason On Fri, Oct 11, 2013 at 5:51 PM, Moises Belchin moises...@gmail.comwrote: Hi Vinny, Thanks for the tips, but actually I'm not loading a file. I'm only using mapreduce lib for read all the entities for one of my kinds, work with them (I only read some properties to compose the csv line format) and then I write to CSV file on cloud storage using mapreduce
[google-appengine] DATASTORE VIEWER
On appengine i make requests on the databases, but i am limited to 20 results by Page how can i make this 20 go up to 100 ? thx for your help ! Martin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] DATASTORE VIEWER
Hello, You have 2 options: 1 - In the old console ( appengine.google.com ) in the datastore viewer you can play with this two params in the URL: limit=20offset=20 2 - You can access the new console ( cloud.google.com/console ) click in your project and then you can use the *Cloud Datastore - Query* section to Query the Datastore, and in this case google loads more results as you keep scrolling down the page 2013/10/15 Martin Descours descours.mar...@gmail.com On appengine i make requests on the databases, but i am limited to 20 results by Page how can i make this 20 go up to 100 ? thx for your help ! Martin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- Alejandro González alejandro.gonza...@intelygenz.com +34 666 57 79 13 http://es.linkedin.com/pub/alejandro-gonzález-rodrigo/19/684/466/ http://www.intelygenz.com/ If you have a dream http://www.intelygenz.com/en/cases* we can write the * code http://www.thegameofcode.com/ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: DATASTORE VIEWER
You can go up to 200 by modifying the limit parameter in the url : limit=200 François On Tuesday, October 15, 2013 11:50:34 AM UTC+2, Martin Descours wrote: On appengine i make requests on the databases, but i am limited to 20 results by Page how can i make this 20 go up to 100 ? thx for your help ! Martin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: DATASTORE VIEWER
Of course... sorry i suck, i didnt think about the url thx françois M. 2013/10/15 Francois Masurel f.masu...@gmail.com You can go up to 200 by modifying the limit parameter in the url : limit=200 François On Tuesday, October 15, 2013 11:50:34 AM UTC+2, Martin Descours wrote: On appengine i make requests on the databases, but i am limited to 20 results by Page how can i make this 20 go up to 100 ? thx for your help ! Martin -- You received this message because you are subscribed to a topic in the Google Groups Google App Engine group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/59xMBZNQXgQ/unsubscribe . To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] DATASTORE VIEWER
Thx Alejandro, it works fine with the URL trick ! Martin 2013/10/15 Alejandro González Rodrigo alejandro.gonza...@intelygenz.com Hello, You have 2 options: 1 - In the old console ( appengine.google.com ) in the datastore viewer you can play with this two params in the URL: limit=20offset=20 2 - You can access the new console ( cloud.google.com/console ) click in your project and then you can use the *Cloud Datastore - Query* section to Query the Datastore, and in this case google loads more results as you keep scrolling down the page 2013/10/15 Martin Descours descours.mar...@gmail.com On appengine i make requests on the databases, but i am limited to 20 results by Page how can i make this 20 go up to 100 ? thx for your help ! Martin -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- Alejandro González alejandro.gonza...@intelygenz.com +34 666 57 79 13 http://es.linkedin.com/pub/alejandro-gonzález-rodrigo/19/684/466/ http://www.intelygenz.com/ If you have a dream http://www.intelygenz.com/en/cases* we can write the *code http://www.thegameofcode.com/ -- You received this message because you are subscribed to a topic in the Google Groups Google App Engine group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/59xMBZNQXgQ/unsubscribe . To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Anyone else seeing a high percentage of 520 errors from Cloud Flare?
' This would be possible that the preloader is making more requests at rate that appengine either can't keep up with or triggers a rate limiting security rule. ' This is the reply from CloudFlare Not that I don't like CloudFlare, but is there anyone using CloudFlare only for the DNS/ naked domain to www. redirect feature? I'm thinking of going the manual SSL route with appengine and using cloudflare only for the redirect, which is extremely useful (Another plus is that, appengine provides an approximate user lat,lon value in the header, which is extremely useful as a default value :) On Tuesday, October 15, 2013 10:31:59 AM UTC+3, Kaan Soral wrote: The issue stopped right after I disabled Website Preloader on CF, might be coincidental with a low probability CF didn't reply to the ticket interesting turn of events On Tuesday, October 15, 2013 10:02:35 AM UTC+3, Richard Watson wrote: Did you get any joy? Find anything out from CF? On Sunday, October 13, 2013 7:12:50 PM UTC+2, Kaan Soral wrote: I've enabled warmup requests to see whether it would change anything After deploying changes, all instances were killed, as expected I've waited some instanced to spin up, they didn't Then I visited the app from the cloudflare url, and instances still didn't spin up, I got a 520 CF error again Then I visited the app through the *.appspot.com url, and an instance spinned up to serve the request So I think the issue might be with appengine, not responding to CF requests -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Looks like 1.8.6 has rolled out, seeing a few small issues.
1.8.6 is now being shown in the dashboard, and I think we are going to have a few days of instabililty ;-( - dashboard failures, memcache rapid purging. Also the new Memcache Total Cache Size doesn't appear show any data. I wonder if running instances (one reserved instance has been running for 2 days) will need to be restarted before that data can be collected ? T -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] BlobInfo object from a BlobKey created using blobstore.create_gs_key
On Mon, Oct 14, 2013 at 8:16 PM, PK p...@gae123.com wrote: the permissions on the bucket look good. What about the permissions on the files created by the app? How are these supposed to look? Are they inherited from the bucket, or should they be explicitly specified? If the app is creating and writing the file, it should have the correct permissions to subsequently read/write from it - GCS automatically grants * read* privileges to users with *full_control* and *write* privileges. If you want to, you can try explicitly setting the ACL of an object using the Python GCS library: https://developers.google.com/storage/docs/gspythonlibrary#changing-acls To verify that your application can access the GCS bucket, you can also try running Google's example application: https://github.com/GoogleCloudPlatform/storage-acl-setter-python Alternatively, consider skipping the blobstore API altogether - use the GCS Python library to handle all processing. This might be a slower option though. - -Vinny P Technology Media Advisor Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
