Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-15 Thread Moises Belchin
Thanks Jason, Me too, I starred this one long time ago. Saludos. Moisés Belchín. 2013/10/15 Jason Galea > thanks Jason, yeh I starred that one a while back. > > > > On Mon, Oct 14, 2013 at 11:58 PM, Jason Collins > wrote: > >> You guys should star >> https://code.google.com/p/googleappengin

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-14 Thread Jason Galea
thanks Jason, yeh I starred that one a while back. On Mon, Oct 14, 2013 at 11:58 PM, Jason Collins wrote: > You guys should star > https://code.google.com/p/googleappengine/issues/detail?id=9610 > > In the past, I've found surprising amount of memory use when working with > NDB, especially when

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-14 Thread Jason Collins
I dug up some old research I had done which found the entities that should be smaller than 10kB were lugging around almost 100kB in-memory due to the in-memory protobuf representation. On Monday, 14 October 2013 07:58:46 UTC-6, Jason Collins wrote: > > You guys should star > https://code.google

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-14 Thread Jason Collins
You guys should star https://code.google.com/p/googleappengine/issues/detail?id=9610 In the past, I've found surprising amount of memory use when working with NDB, especially when using repeated properties. In my experiments, it was very easy to blow up instances with a seemingly small number o

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-14 Thread Jason Galea
Hi Moises, you can find all the details here.. https://developers.google.com/appengine/docs/python/ndb/cache "The In-Context Cache The in-context cache persists only for the duration of a single incoming HTTP request and is "visible" only to the code that handles that request. It's fast; this ca

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-14 Thread Moises Belchin
Hi Jason, Thanks for the detailed answer. I'm very surprised that no one else is talking about these issues. I'm using ndb and my appstats are off. I could see a incredible great improvement in my app when I turned off the stats. So, I recommend people only use stats for testing or debug. As you

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-11 Thread Jason Galea
Hi Moises, we're currently trying to deal with this issue too. Not in mapreduce, just regular handlers. same here - "Python 2.7 and F2 instances with 256MB" Early on I found that fetching 1000 entities and looping through them to update a property would blow the instance. Reducing this to say 10

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-11 Thread Moises Belchin
Hi Vinny, Thanks for the tips, but actually I'm not loading a file. I'm only using mapreduce lib for read all the entities for one of my kinds, work with them (I only read some properties to compose the csv line format) and then I write to CSV file on cloud storage using mapreduce FileOutputWriter

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-10 Thread Vinny P
On Thu, Oct 10, 2013 at 5:47 AM, Moises Belchin wrote: > I'm getting a lot of memory limit critical errors in my app when I use > mapreduce library. > I'm working with Python 2.7, app engine 1.8.5 and F2 instances with 256MB. > > Has anyone got these critical errors? > > Hello Moises, How larg

[google-appengine] how reduce memory usage on mapreduce controller callback

2013-10-10 Thread Moises Belchin
Hi all, I'm getting a lot of memory limit critical errors in my app when I use mapreduce library. I'm working with Python 2.7, app engine 1.8.5 and F2 instances with 256MB. See attached image. [image: Imágenes integradas 1] Has anyone got these critical errors? Are there any best practices to av