Re: [google-appengine] how reduce memory usage on mapreduce controller callback

[Jason Galea]

Hi Moises,

we're currently trying to deal with this issue too. Not in mapreduce, just
regular handlers.

same here - "Python 2.7 and F2 instances with 256MB"

Early on I found that fetching 1000 entities and looping through them to
update a property would blow the instance. Reducing this to say 100 fixed
the issue. (than when create another task do do the next 100)

"After handling this request" - as far as I understand this isn't so bad
unless you're blowing it on every request and starting a new instance is
detrimental.

"While handling this request" - this concerns us most as the request does
not complete and breaks stuff.. and we see far too many of them.

I've spent more than a little time trying to work out what is causing the
blowouts but as far as I've been able to work out, memory usage and what
causes it is near impossible (or just very, very hard).

Are you using NDB? if so..
- you could try disabling the in-memory cache. As I see it, even though you
only access one entity at a time, NDB's in memory cache will store them all
until the request is completed.
- you could try projection queries if you don't need the complete object
(or possibly even if you do). Projection queries get their data from an
index and the entities returned cannot be put() so I assume they are not
cached at all. We're trialling some fixes with these atm.

** If anyone knows any of this is incorrect, please let me know..

I'm actually surprised there is not more discussion of these issues from
what we have experienced so maybe we're doing something fundamentally
wrong, but I don't believe so.

oh, is appstats turned on? I believe the most noticeable improvement we've
seen was when we turned it off..

regards,

Jason






On Fri, Oct 11, 2013 at 5:51 PM, Moises Belchin wrote:

> Hi Vinny,
>
> Thanks for the tips, but actually I'm not loading a file. I'm only using
> mapreduce lib for read all the entities for one of my kinds, work with them
> (I only read some properties to compose the csv line format) and then I
> write to CSV file on cloud storage using mapreduce FileOutputWriter.
>
> Any idea why I'm getting this Criticals memory errors?
>
> Thanks all again.
>
>
> Saludos.
> Moisés Belchín.
>
>
> 2013/10/10 Vinny P 
>
>> On Thu, Oct 10, 2013 at 5:47 AM, Moises Belchin 
>>  wrote:
>>
>>> I'm getting a lot of memory limit critical errors in my app when I use
>>> mapreduce library.
>>>  I'm working with Python 2.7, app engine 1.8.5 and F2 instances with
>>> 256MB.
>>>
>>> Has anyone got these critical errors?
>>>
>>>
>>
>> Hello Moises,
>>
>> How large is your input file? Are you loading the entire file into memory
>> at once? If so, try moving to a BlobstoreLineInputReader which reads in one
>> line at a time - it reduces the memory being used during file processing.
>>
>>
>>
>> -
>> -Vinny P
>> Technology & Media Advisor
>> Chicago, IL
>>
>> App Engine Code Samples: http://www.learntogoogleit.com
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
Jason Galea
lecstor.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Servlets 3.0, And In Query cursors

[Vinny P]

On Fri, Oct 11, 2013 at 3:18 AM, srk 
 wrote:

> Can you please give an update when app engine will start supporting
> servets 3.0. I am desperately looking for it. I would be grateful if
> someone could provide me an approximate date
>


Is there a particular component of the Servlet 3.0 spec that you need?
Depending on your needs, there may be an equivalent or an alternative
possibility on App Engine.


On Fri, Oct 11, 2013 at 3:18 AM, srk 
 wrote:

> when gae will start returning query cursors for IN operators. I have
> posted the limitation below that is mentioned in an app engine doc. Because
> the NOT_EQUAL and IN operators are implemented with multiple queries,
> queries that use them do not support cursors, nor do composite queries
> constructed with the CompositeFilterOperator.or method
>


You'll probably be waiting for some time on this feature request. As the
documentation notes, IN and NOT_EQUAL queries actually execute multiple
queries against the data store, which makes it difficult for cursors to
keep track of the current query position.  You may want to consider
alternative options, such as Cloud SQL.


-
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: major MEMCACHE changes started October 5th

[Vinny P]

On Fri, Oct 11, 2013 at 1:02 PM, James Gilliam  wrote:

> I try and/or evaluate all the suggestions.  The worse day for shared
> memcache was Saturday. Today and yesterday was much improved.  However,
> they still more aggressively purge shared memcache than they use to.  I
> know this because I track it everyday.  The premium memcache is great
> function since it allows more cache; however 1 gig is too much for me and
> too costly ... $1100 a year.  Not reporting the real time billing correctly
> is a pain but something I can live with.
>


Hello James,

Are you experiencing this behavior across multiple application IDs, or is
it one application having performance issues? What you can try is - if you
have a paid support package - is to call Google up and ask them to move
your app ID to a different place in the physical infrastructure. You might
get better neighbors sharing your server, and less contention for memcache
resources. If you don't have a paid support contract, you can try
redeploying your app as Barry noted above.

In addition (depending on your application's size and instance class) you
may be able to cram some data into an in-instance cache. Try profiling your
app and checking to see if there's some free RAM to use.

With that said, I +1 Tim and Barry's comments, this just doesn't look
intentional on Google's part.

-
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[timh]

I really think you confuse various localised and/or widespread problems 
with intent.  I just don't see "intent"

I have been using appengine (totally reliant on it) since 2008.  memcache 
has always been vague in it's rentention of records.  See me earlier 
response to this thread , you see I had oldest items sitting at 23 hours on 
the 7th yesterday when I looked it was 2 hours.  Right now it's sitting at 
12 mins.   It varies all the time.  Go back 3 years, rentention times where 
all over the shop.

On Saturday, October 12, 2013 2:02:03 AM UTC+8, James Gilliam wrote:
>
> I try and/or evaluate all the suggestions.  The worse day for shared 
> memcache was Saturday. Today and yesterday was much improved.  However, 
> they still more aggressively purge shared memcache than they use to.  I 
> know this because I track it everyday.  The premium memcache is great 
> function since it allows more cache; however 1 gig is too much for me and 
> too costly ... $1100 a year.  Not reporting the real time billing correctly 
> is a pain but something I can live with.  No idea if they depreciated the 
> shared memcache on purpose to force me into premium, but whatever they did 
> screwed me -- that is for sure.
>
> On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>>
>> GAE has drastically changed the way MEMCACHE works; as a result my 
>> application (ogeekcom) overall usage jumped by approximately 5 times with 
>> the same approximate bandwidth output.  Like a 400% increase in price.
>>
>> Specifically, they are purging shared memcache very aggressively -- 
>> possible in an effort to force people to signup for paid memcache.
>>
>> As a result of this change, my application is using many more datastore 
>> reads and many more instances to compensate for the poor memcache 
>> performance.
>>
>> Like always, this was done without any announcement at all.
>>
>> If they made this change to increase make applications cost more to run, 
>> it is illegal.  
>>
>> There is no problem with them offering a premium service for memcache, 
>> but it is illegal to degrade the previous service to force people into the 
>> paid model.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Can't create application "invalid name" no matter what I choose..

[Vinny P]

On Fri, Oct 11, 2013 at 5:27 AM, Ben Ritchie 
 wrote:

> No matter what I choose as the app name it says "invalid"
>


The application name "asdfafasdf" shown in your screenshot seems to be
legitimately invalid, as I'm getting the same error:
http://imgur.com/oilppL5

You just have to play around a little with names to get a valid application
ID. Remember that app IDs can't share names with an already-existing Gmail
account, or previously-deleted application IDs.


-
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] ImagesService getServingUrl fails often

[Theodore Book]

Are you still having this problem? I noticed today that *every* request 
that I make to get a serving URL is failing with an 
ImagesServiceFailureException.  I'm not sure how long this has been going 
on - a couple of days at most, probably just hours.  There was no change in 
my code; just the repeated failure.  Is anyone else experiencing this?



   1. 
   2. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Backend instance killed 3 seconds after being started.

[Francois Masurel]

Things seem to be back to normal, fingers crossed.


On Friday, October 11, 2013 9:11:19 PM UTC+2, Francois Masurel wrote:
>
> New record :
> *
> *
> *219 ms*
>
>
>
> 
> When will my tasks get executed on this instance ?
>
>
> On Friday, October 11, 2013 5:49:19 PM UTC+2, Francois Masurel wrote:
>>
>> Tasks are piling up waiting for my backend instance to start :
>>
>>
>> 
>>
>> It's a basic scaling instance with a 5 minutes idle timeout (App ID: 
>> bblcts1).
>> Hoping it's only a temporarily glitch on GAE side.
>>
>> François
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Backend instance killed 3 seconds after being started.

[Francois Masurel]

New record :
*
*
*219 ms*



When will my tasks get executed on this instance ?


On Friday, October 11, 2013 5:49:19 PM UTC+2, Francois Masurel wrote:
>
> Tasks are piling up waiting for my backend instance to start :
>
>
> 
>
> It's a basic scaling instance with a 5 minutes idle timeout (App ID: 
> bblcts1).
> Hoping it's only a temporarily glitch on GAE side.
>
> François
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[James Gilliam]

I try and/or evaluate all the suggestions.  The worse day for shared 
memcache was Saturday. Today and yesterday was much improved.  However, 
they still more aggressively purge shared memcache than they use to.  I 
know this because I track it everyday.  The premium memcache is great 
function since it allows more cache; however 1 gig is too much for me and 
too costly ... $1100 a year.  Not reporting the real time billing correctly 
is a pain but something I can live with.  No idea if they depreciated the 
shared memcache on purpose to force me into premium, but whatever they did 
screwed me -- that is for sure.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
> GAE has drastically changed the way MEMCACHE works; as a result my 
> application (ogeekcom) overall usage jumped by approximately 5 times with 
> the same approximate bandwidth output.  Like a 400% increase in price.
>
> Specifically, they are purging shared memcache very aggressively -- 
> possible in an effort to force people to signup for paid memcache.
>
> As a result of this change, my application is using many more datastore 
> reads and many more instances to compensate for the poor memcache 
> performance.
>
> Like always, this was done without any announcement at all.
>
> If they made this change to increase make applications cost more to run, 
> it is illegal.  
>
> There is no problem with them offering a premium service for memcache, but 
> it is illegal to degrade the previous service to force people into the paid 
> model.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: GAE loading new instances on each home page access

[Jorge Amat]

Yesterday happened to me that when I set the min idle instance to 1,  after 
some minutes it was again set as automatic.without any consent. Not sure if 
this has been fixed but this is a big problem in terms of expenses for 
users specially for low traffics apps.

El viernes, 11 de octubre de 2013 13:25:52 UTC+1, Francois Masurel escribió:
>
> Just switched back to automatic scaling and everything seems to be back to 
> normal, no more crazy instance loading.
>
> There were definitely something buggy on GAE side which seems to have been 
> fixed.
>
>
>
> On Friday, October 11, 2013 1:23:10 PM UTC+2, Francois Masurel wrote:
>>
>> Just missed a point : basic scaling instances are still billed like 
>> backend instance hours and not front end hours even for the default module.
>>
>> So we have only 8 hours offered instead of 28 hours for automatic scaling 
>> instances :-(
>>
>> Please Google fix this, quick.
>>
>> Created an issue for it : 
>> https://code.google.com/p/googleappengine/issues/detail?id=10105
>>
>> François
>>
>>
>>
>> On Wednesday, October 9, 2013 5:35:25 PM UTC+2, Francois Masurel wrote:
>>>
>>> Ok guys,
>>>
>>> Looks like I solved the problem by setting *basic scaling* instead of 
>>> *automatic 
>>> scaling *for my default module, it works also with *manual scaling*.
>>>
>>> No more crazy instance starts and stops, that's cool :-)
>>>
>>> There don't seem to be the 10 concurrent requests maxi limit any more.
>>>
>>> Still have one question : how does it scale ?  Here is what is said in 
>>> the documentation :
>>>
>>> *A basic scaling module version is configured with a maximum number of 
>>> instances using the max_instances basic scaling parameter.*
>>>
>>> *The number of live instances scales with the processing volume.*
>>>
>>>
>>> What is "processing volume" exactly?  How is it evaluated? There is no 
>>> pending latency or such parameters.
>>>
>>> I could start a few more instances by doing some stress tests.  Started 
>>> instance are kept alive as defined by the idle_timeout basic scaling 
>>> parameter.  But there still seem to have some user facing loading requests.
>>>
>>> I guess the automatic scaling mode still needs polish and is not 
>>> production ready yet, unless I missed something.
>>>
>>> François
>>>
>>>
>>> On Wednesday, October 9, 2013 2:38:50 PM UTC+2, Francois Masurel wrote:

 Please, STOP killing instances after 30s of inactivity (average time 
 measured, might be even less).

 Aren't we paying a minimum 15 minutes startup fee for those started and 
 killed instances ?  May be you could increase this delay to at least 2 
 minutes ?

 These {censored} loading requests make our app totally unusable.



 On Wednesday, October 9, 2013 12:33:52 PM UTC+2, Francois Masurel wrote:
>
> Same problem with F2 instances.  This f... scheduler is getting us 
> crazy.
>
> There is nothing wrong with starting a new instance but it should wait 
> for the instance being properly loaded before redirecting traffic to it.
>
>
>
> On Wednesday, October 9, 2013 12:09:42 PM UTC+2, Francois Masurel 
> wrote:
>>
>> It looks like GAE Java instances don't handle more than 10 concurrent 
>> requests.
>>
>> As a result, our GWT app home page which makes about 20 requests 
>> needs at least 2 instances loaded to get displayed.
>>
>> And as instances are shutdown very agressivelly (a few seconds), our 
>> users are getting hit by these loading requests pretty often.
>>
>> This is making our app totally useless (~10-20s to load a new 
>> instance).
>>
>> Is there a way to increase this max number of concurrent requests ?
>>
>> François
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Crazy increases of Frontend Instances in google app engine for java and unfair expenses for users.

[Jorge Amat]

My billing apps are getting *non sense increases of frontend instances* as 
already has been reported 
here
 or 
here.
 
I think it could be related to the also already known lantency issues when 
using frameworks like spring on gae.

Trying to avoid this situation (and unfair expenses), I tried to set the 
idle instances configuration, but sometimes it happens to me that *when I 
set the min idle instance to 1, after some minutes it's again set as 
automatic, without any consent*. Other thing I have tried is to pause the 
defauls task queue, since suddently starts to make a lot of requests of the 
type /*ah/queue/*_deferred_ , however doesn't look to be the solution for 
the non justified increases of instance hours and consequently money 
expenses.

Then, looks like the *the billing section is done in a way it's extremely 
easy to set more money than the necessary*: why the Maximum Daily Budget is 
not set dinamically with a value according to the Weekly Discounted 
Instance Hours the users put? I mean, why would set a user 0 as Maximum 
Daily Budget and some value greater than 0 as Weekly Discounted Instance 
Hours, if these last ones never would be applied due to the first 
configuration?

Any advise on these issues?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: major MEMCACHE changes started October 5th

[Barry Hunter]

On 11 October 2013 09:42, James Gilliam  wrote:

> Barry --
>
> I am not arguing a conspiracy at all.  I am saying that what GAE did,
> whether it be on purpose or not, resulted in a huge increase in the cost of
> hosting.
>

How do you know something was 'changed'? The evience presented so far does
still point being something like a noisey neighbour problem. As noted by
others, not everyone is experiencing the 'issue'


>  It has forced me to opt in for premium memcache.
>

Have you tried any of the ideas presented in the thread? Redeploying etc.

There's also CloudSQL, could use that as a cache. FOr your data sounds like
about $130 a year.


>  It remains to be seen if GAE responds to this at all but the history
> suggests they will not.
>

I also dount they will bother, with all this rhetoric


> Also, they changed this on Saturday without any warning either --
> something they should be held accountable for as well.
>
> Right now,  I have to opt in for the premium memcache at a cost of almost
> $1100 a year because 1 gig is the minimum size I can get.  They do not even
> prorate the cost based on what I am actually using -- something they
> routinely do in other billing areas, e.g. reads, bandwidth, emails, etc.
> So, why not prorate the premium hosting.  In my case the cost could drop by
> 90% since I will likely not use more than 100 megs.
>
> On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
>> GAE has drastically changed the way MEMCACHE works; as a result my
>> application (ogeekcom) overall usage jumped by approximately 5 times with
>> the same approximate bandwidth output.  Like a 400% increase in price.
>>
>> Specifically, they are purging shared memcache very aggressively --
>> possible in an effort to force people to signup for paid memcache.
>>
>> As a result of this change, my application is using many more datastore
>> reads and many more instances to compensate for the poor memcache
>> performance.
>>
>> Like always, this was done without any announcement at all.
>>
>> If they made this change to increase make applications cost more to run,
>> it is illegal.
>>
>> There is no problem with them offering a premium service for memcache,
>> but it is illegal to degrade the previous service to force people into the
>> paid model.
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[James Gilliam]

GAE just responded to the billing issue with the following

Updates:
Status: WorkAsIntended

Comment #1 on issue 10108 by jbelmo...@google.com: dedicated memcache 
billed all at once
http://code.google.com/p/**googleappengine/issues/detail?**id=10108

The billing table in the dashboard reflects the accounting done to manage 
your app's daily spend.  For resources billed in units of time like 
dedicated memcache, for the purposes of daily spend the assumption is made 
that you'll have the resource enabled for the remainder of the day.  Hence 
17.17 hours reflects the time from when you enabled dedicated memcache 
until midnight.

Your actual dedicated memcache usage will be reflected correctly in the 
daily billing usage report on the usage history page.


Once again, GAE missed the point.

By inaccurately showing MEMCACHE billing during the day and declaring works 
as intended, the app is at risk of being stopped when the billing is 
reached.

I had to up my daily billing because of their bug.

While we are on inaccurate real time billing ... they also do not add the 
bandwidth charges during the day ... another bug they will undoubtedly call 
works as intended.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
> GAE has drastically changed the way MEMCACHE works; as a result my 
> application (ogeekcom) overall usage jumped by approximately 5 times with 
> the same approximate bandwidth output.  Like a 400% increase in price.
>
> Specifically, they are purging shared memcache very aggressively -- 
> possible in an effort to force people to signup for paid memcache.
>
> As a result of this change, my application is using many more datastore 
> reads and many more instances to compensate for the poor memcache 
> performance.
>
> Like always, this was done without any announcement at all.
>
> If they made this change to increase make applications cost more to run, 
> it is illegal.  
>
> There is no problem with them offering a premium service for memcache, but 
> it is illegal to degrade the previous service to force people into the paid 
> model.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[James Gilliam]

Barry --

I am not arguing a conspiracy at all.  I am saying that what GAE did, 
whether it be on purpose or not, resulted in a huge increase in the cost of 
hosting.  It has forced me to opt in for premium memcache.  It remains to 
be seen if GAE responds to this at all but the history suggests they will 
not.  Also, they changed this on Saturday without any warning either -- 
something they should be held accountable for as well.

Right now,  I have to opt in for the premium memcache at a cost of almost 
$1100 a year because 1 gig is the minimum size I can get.  They do not even 
prorate the cost based on what I am actually using -- something they 
routinely do in other billing areas, e.g. reads, bandwidth, emails, etc.   
So, why not prorate the premium hosting.  In my case the cost could drop by 
90% since I will likely not use more than 100 megs.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
> GAE has drastically changed the way MEMCACHE works; as a result my 
> application (ogeekcom) overall usage jumped by approximately 5 times with 
> the same approximate bandwidth output.  Like a 400% increase in price.
>
> Specifically, they are purging shared memcache very aggressively -- 
> possible in an effort to force people to signup for paid memcache.
>
> As a result of this change, my application is using many more datastore 
> reads and many more instances to compensate for the poor memcache 
> performance.
>
> Like always, this was done without any announcement at all.
>
> If they made this change to increase make applications cost more to run, 
> it is illegal.  
>
> There is no problem with them offering a premium service for memcache, but 
> it is illegal to degrade the previous service to force people into the paid 
> model.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: major MEMCACHE changes started October 5th

[Barry Hunter]

I'm still intrigued how you know this is deliberate?

Granted you've had a bad experience, but it does not mean there is a
conspiracy against you.


Have you tried redeploying, will often end up in a different part of the
infrastructure, and so move out of a neighbourhood with a noisy neighbour.
Might have to repeat the process a few times to see the effect.


On 11 October 2013 09:25, James Gilliam  wrote:

> There should be a GAE Legal Defense Fund to represent people when GAE acts
> inappropriately.  I am totally dependent on GAE for my business and I
> suspect many other people are as well.  What GAE has done with MEMCACHE is
> wrong imo and we need somebody to call them out on this formally.  What
> they did was a wholesale price increase by reducing the function of shared
> memcache, literally forcing people like me to opt for the paid premium
> version.  The operative word here is "forcing" which is coercion in my view.
>
> On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
>> GAE has drastically changed the way MEMCACHE works; as a result my
>> application (ogeekcom) overall usage jumped by approximately 5 times with
>> the same approximate bandwidth output.  Like a 400% increase in price.
>>
>> Specifically, they are purging shared memcache very aggressively --
>> possible in an effort to force people to signup for paid memcache.
>>
>> As a result of this change, my application is using many more datastore
>> reads and many more instances to compensate for the poor memcache
>> performance.
>>
>> Like always, this was done without any announcement at all.
>>
>> If they made this change to increase make applications cost more to run,
>> it is illegal.
>>
>> There is no problem with them offering a premium service for memcache,
>> but it is illegal to degrade the previous service to force people into the
>> paid model.
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[James Gilliam]

There should be a GAE Legal Defense Fund to represent people when GAE acts 
inappropriately.  I am totally dependent on GAE for my business and I 
suspect many other people are as well.  What GAE has done with MEMCACHE is 
wrong imo and we need somebody to call them out on this formally.  What 
they did was a wholesale price increase by reducing the function of shared 
memcache, literally forcing people like me to opt for the paid premium 
version.  The operative word here is "forcing" which is coercion in my view.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
> GAE has drastically changed the way MEMCACHE works; as a result my 
> application (ogeekcom) overall usage jumped by approximately 5 times with 
> the same approximate bandwidth output.  Like a 400% increase in price.
>
> Specifically, they are purging shared memcache very aggressively -- 
> possible in an effort to force people to signup for paid memcache.
>
> As a result of this change, my application is using many more datastore 
> reads and many more instances to compensate for the poor memcache 
> performance.
>
> Like always, this was done without any announcement at all.
>
> If they made this change to increase make applications cost more to run, 
> it is illegal.  
>
> There is no problem with them offering a premium service for memcache, but 
> it is illegal to degrade the previous service to force people into the paid 
> model.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Backend instance killed 3 seconds after being started.

[Francois Masurel]

Tasks are piling up waiting for my backend instance to start :



It's a basic scaling instance with a 5 minutes idle timeout (App ID: 
bblcts1).
Hoping it's only a temporarily glitch on GAE side.

François

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[pdknsk]

> Saturday is the day the love pushing out bad code.
>

 I laughed when I read this. This should be the subtitle of the (largely 
useless) App Engine Status page, in bold.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[Alex Burgel]

I see this bug as well. Dedicated memcache is in preview release, so bugs 
like this shouldn't be surprising.

I filed an issue:

https://code.google.com/p/googleappengine/issues/detail?id=10108

On Friday, October 11, 2013 10:42:18 AM UTC-4, James Gilliam wrote:
>
> Now there is a billing issue with memcache premium.  It goes along with 
> 0.00 all day and then plunk all of it at once.  It doesn't add the cost 
> hour by hour -- clearly a bug. The next day, they plunk like 18 hours right 
> at the beginning of the day, which is rolled back later in the day.  But 
> they show the WRONG billing all through the day and I am afraid they will 
> deactivate my application with these errors.
>
> In my opinion, they don't do any real beta testing before plunking stuff 
> out.  My guess is they let developers push whenever they want, at least 
> that is what is suggested with the types of error they create.  Saturday is 
> the day the love pushing out bad code.
>
> They should make premium memcache free until they fix the problems and let 
> us opt in for less than 1 gig --- 1 gig of memcache is way TOO much for 
> many apps and they have made free shared memcache worthless -- their plan 
> all along imo.
>
> On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>>
>> GAE has drastically changed the way MEMCACHE works; as a result my 
>> application (ogeekcom) overall usage jumped by approximately 5 times with 
>> the same approximate bandwidth output.  Like a 400% increase in price.
>>
>> Specifically, they are purging shared memcache very aggressively -- 
>> possible in an effort to force people to signup for paid memcache.
>>
>> As a result of this change, my application is using many more datastore 
>> reads and many more instances to compensate for the poor memcache 
>> performance.
>>
>> Like always, this was done without any announcement at all.
>>
>> If they made this change to increase make applications cost more to run, 
>> it is illegal.  
>>
>> There is no problem with them offering a premium service for memcache, 
>> but it is illegal to degrade the previous service to force people into the 
>> paid model.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Need help: Pathological instancing behavior

[Jorge Amat]

Sometimes  happens to me that when I set the min idle instance to 1,  after 
some minutes it's again set as automatic.without any consent.
Other thing I have tried is to pause the defauls task queue, since 
suddently starts to make a lot of requests, however doesn't look to be the 
best solution for the non justified increase of instance hours and 
consequently money expenses for users.

El viernes, 11 de octubre de 2013 15:50:24 UTC+1, Jorge Amat escribió:
>
> Any news on this? I'm having similar issues nowadays and can't believe 
> this is not working properly yet.
>
> El miércoles, 1 de agosto de 2012 03:18:26 UTC+1, Kristopher Giesing 
> escribió:
>>
>> I've been working for some time on an app I plan to deploy.  Over the 
>> development period I have done testing on the QA instance of my app (which 
>> is a separate app ID from the one I'll use in production) but have not paid 
>> serious attention to performance since the functionality wasn't yet 
>> complete.
>>
>> This past weekend I did some light real-world type testing, and the 
>> performance was terrible.  With QPS in the 5-10 range I was seeing 
>> latencies in the range of 10-20s.  Obviously, the first place I looked was 
>> my own code, but after some easy optimizations I got my query handling down 
>> to the 200ms range, and the response times on the client were still usually 
>> very bad (1s minimum, up to 10-20s max).  (Pings from the client to the 
>> appspot.com domain are pretty stable at 30ms.)
>>
>> Looking further, I discovered that Google App Engine is spawning new 
>> instances of my app at a pathological rate.  For example, I left the app 
>> stats console open for a while, and found that just loading the appstats 
>> URI by itself would cause a new instance to spawn!  This is with zero QPS 
>> otherwise - my app was completely idle, and none of my own code was running 
>> at all.
>>
>> I tried adjusting the min and max instance settings, but GAE seems to 
>> ignore them, as far as I can tell; the max is not honored, and although it 
>> does keep instances running to accommodate the min, those running instances 
>> don't prevent new ones from spawning.  It's as if GAE doesn't think they 
>> really exist. I also have the min latency set to the max of 15.0s, which is 
>> far beyond the rise time I'm seeing for GAE spawning new instances. 
>>
>> Here's a set of logs taken from a very light load on the server, with a 
>> 2/2 instance max/min setting.  I include logs from a previous slice of time 
>> so that you can see the /appstats URI also spawning new instances, but I 
>> want to emphasize that this is a continuous set of logs - there are no 
>> requests in this time window that aren't listed below.  I also include a 
>> screenshot of the instance panel in the console so you can see that the 
>> instances are, in fact, being created.
>>
>> 2012-07-31 18:58:12.541 /_ah/channel/connected/ 200 101ms 0kb
>> 2012-07-31 18:58:10.894 /api/chat/G:1286432802 200 132ms 0kb Mozilla/5.0 
>> (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
>> Mobile/9B206
>> 2012-07-31 18:58:09.809 /_ah/channel/connected/ 200 1076ms 0kb
>> 2012-07-31 18:58:09.107 /api/game/72001/users 200 12312ms 0kb Mozilla/5.0 
>> (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
>> Mobile/9B206
>> 2012-07-31 18:58:07.725 /api/chat/G:658732847 200 7232ms 0kb Mozilla/5.0 
>> (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
>> Mobile/9B176
>> 2012-07-31 18:58:00.279 /api/game/72001/users 200 8820ms 0kb Mozilla/5.0 
>> (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
>> Mobile/9B176
>> I 2012-07-31 18:58:00.279 This request caused a new process to be started 
>> for your application, and thus caused your application code to be loaded 
>> for the first time. This requ
>> 2012-07-31 18:57:56.393 /api/game/72001 200 6044ms 3kb Mozilla/5.0 (iPad; 
>> CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
>> Mobile/9B206
>> 2012-07-31 18:57:55.920 /_ah/warmup 200 5105ms 0kb
>> I 2012-07-31 18:57:55.916 This request caused a new process to be started 
>> for your application, and thus caused your application code to be loaded 
>> for the first time. This requ
>> 2012-07-31 18:57:51.324 /api/game/72001 200 7151ms 3kb Mozilla/5.0 (iPad; 
>> CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
>> Mobile/9B176
>> 2012-07-31 18:57:49.779 /_ah/warmup 200 4629ms 0kb
>> I 2012-07-31 18:57:49.778 This request caused a new process to be started 
>> for your application, and thus caused your application code to be loaded 
>> for the first time. This requ
>> 2012-07-31 18:57:01.504 /_ah/warmup 200 5225ms 0kb
>> I 2012-07-31 18:57:01.503 This request caused a new process to be started 
>> for your application, and thus caused your application code to be loaded 
>> for the first time. This requ
>> 2012-07-31 18:13:56.599 /_ah/warmup 200 4784ms 0kb
>> I 2012-07-31 18:13:56.5

[google-appengine] Re: Instances/Java go crazy

[Jorge Amat]

Sometimes it has happened to me that when I set the min idle instance to 1, 
 after some minutes it's again set as automatic.without any consent.
Other thing I have tried is to pause the defauls task queue, since 
suddently starts to make a lot of requests, however doesn't look to be the 
solution for the non justified increase of instance hours and consequently 
expenses for users.

El viernes, 11 de octubre de 2013 15:45:29 UTC+1, Jorge Amat escribió:
>
> Still having these same issues nowadays,  this is outrageous.
>
>
> El lunes, 30 de julio de 2012 16:47:00 UTC+1, Mos escribió:
>>
>> Anyone else seeing issue with GAE instance management? 
>> In one minute - 10 requests - around 5 instances are started. Old ones 
>> do not response. New instances are created again and again... 
>>
>> http://code.google.com/p/googleappengine/issues/detail?id=7910 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Need help: Pathological instancing behavior

[Jorge Amat]

Any news on this? I'm having similar issues nowadays and can't believe this 
is not working properly yet.

El miércoles, 1 de agosto de 2012 03:18:26 UTC+1, Kristopher Giesing 
escribió:
>
> I've been working for some time on an app I plan to deploy.  Over the 
> development period I have done testing on the QA instance of my app (which 
> is a separate app ID from the one I'll use in production) but have not paid 
> serious attention to performance since the functionality wasn't yet 
> complete.
>
> This past weekend I did some light real-world type testing, and the 
> performance was terrible.  With QPS in the 5-10 range I was seeing 
> latencies in the range of 10-20s.  Obviously, the first place I looked was 
> my own code, but after some easy optimizations I got my query handling down 
> to the 200ms range, and the response times on the client were still usually 
> very bad (1s minimum, up to 10-20s max).  (Pings from the client to the 
> appspot.com domain are pretty stable at 30ms.)
>
> Looking further, I discovered that Google App Engine is spawning new 
> instances of my app at a pathological rate.  For example, I left the app 
> stats console open for a while, and found that just loading the appstats 
> URI by itself would cause a new instance to spawn!  This is with zero QPS 
> otherwise - my app was completely idle, and none of my own code was running 
> at all.
>
> I tried adjusting the min and max instance settings, but GAE seems to 
> ignore them, as far as I can tell; the max is not honored, and although it 
> does keep instances running to accommodate the min, those running instances 
> don't prevent new ones from spawning.  It's as if GAE doesn't think they 
> really exist. I also have the min latency set to the max of 15.0s, which is 
> far beyond the rise time I'm seeing for GAE spawning new instances. 
>
> Here's a set of logs taken from a very light load on the server, with a 
> 2/2 instance max/min setting.  I include logs from a previous slice of time 
> so that you can see the /appstats URI also spawning new instances, but I 
> want to emphasize that this is a continuous set of logs - there are no 
> requests in this time window that aren't listed below.  I also include a 
> screenshot of the instance panel in the console so you can see that the 
> instances are, in fact, being created.
>
> 2012-07-31 18:58:12.541 /_ah/channel/connected/ 200 101ms 0kb
> 2012-07-31 18:58:10.894 /api/chat/G:1286432802 200 132ms 0kb Mozilla/5.0 
> (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
> Mobile/9B206
> 2012-07-31 18:58:09.809 /_ah/channel/connected/ 200 1076ms 0kb
> 2012-07-31 18:58:09.107 /api/game/72001/users 200 12312ms 0kb Mozilla/5.0 
> (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
> Mobile/9B206
> 2012-07-31 18:58:07.725 /api/chat/G:658732847 200 7232ms 0kb Mozilla/5.0 
> (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
> Mobile/9B176
> 2012-07-31 18:58:00.279 /api/game/72001/users 200 8820ms 0kb Mozilla/5.0 
> (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
> Mobile/9B176
> I 2012-07-31 18:58:00.279 This request caused a new process to be started 
> for your application, and thus caused your application code to be loaded 
> for the first time. This requ
> 2012-07-31 18:57:56.393 /api/game/72001 200 6044ms 3kb Mozilla/5.0 (iPad; 
> CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
> Mobile/9B206
> 2012-07-31 18:57:55.920 /_ah/warmup 200 5105ms 0kb
> I 2012-07-31 18:57:55.916 This request caused a new process to be started 
> for your application, and thus caused your application code to be loaded 
> for the first time. This requ
> 2012-07-31 18:57:51.324 /api/game/72001 200 7151ms 3kb Mozilla/5.0 (iPad; 
> CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) 
> Mobile/9B176
> 2012-07-31 18:57:49.779 /_ah/warmup 200 4629ms 0kb
> I 2012-07-31 18:57:49.778 This request caused a new process to be started 
> for your application, and thus caused your application code to be loaded 
> for the first time. This requ
> 2012-07-31 18:57:01.504 /_ah/warmup 200 5225ms 0kb
> I 2012-07-31 18:57:01.503 This request caused a new process to be started 
> for your application, and thus caused your application code to be loaded 
> for the first time. This requ
> 2012-07-31 18:13:56.599 /_ah/warmup 200 4784ms 0kb
> I 2012-07-31 18:13:56.581 This request caused a new process to be started 
> for your application, and thus caused your application code to be loaded 
> for the first time. This requ
> 2012-07-31 18:06:53.651 /appstats/stats 200 898ms 285kb Mozilla/5.0 
> (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) 
> Version/6.0 Safari/536.25
> 2012-07-31 18:06:52.618 /appstats 302 33ms 0kb Mozilla/5.0 (Macintosh; 
> Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 
> Safari/536.25
> 2012-07-31 18:06:14.538 /appstats/static/p

[google-appengine] Re: Instances/Java go crazy

[Jorge Amat]

Still having these same issues nowadays,  this is outrageous.


El lunes, 30 de julio de 2012 16:47:00 UTC+1, Mos escribió:
>
> Anyone else seeing issue with GAE instance management? 
> In one minute - 10 requests - around 5 instances are started. Old ones 
> do not response. New instances are created again and again... 
>
> http://code.google.com/p/googleappengine/issues/detail?id=7910 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[James Gilliam]

Now there is a billing issue with memcache premium.  It goes along with 
0.00 all day and then plunk all of it at once.  It doesn't add the cost 
hour by hour -- clearly a bug. The next day, they plunk like 18 hours right 
at the beginning of the day, which is rolled back later in the day.  But 
they show the WRONG billing all through the day and I am afraid they will 
deactivate my application with these errors.

In my opinion, they don't do any real beta testing before plunking stuff 
out.  My guess is they let developers push whenever they want, at least 
that is what is suggested with the types of error they create.  Saturday is 
the day the love pushing out bad code.

They should make premium memcache free until they fix the problems and let 
us opt in for less than 1 gig --- 1 gig of memcache is way TOO much for 
many apps and they have made free shared memcache worthless -- their plan 
all along imo.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>
> GAE has drastically changed the way MEMCACHE works; as a result my 
> application (ogeekcom) overall usage jumped by approximately 5 times with 
> the same approximate bandwidth output.  Like a 400% increase in price.
>
> Specifically, they are purging shared memcache very aggressively -- 
> possible in an effort to force people to signup for paid memcache.
>
> As a result of this change, my application is using many more datastore 
> reads and many more instances to compensate for the poor memcache 
> performance.
>
> Like always, this was done without any announcement at all.
>
> If they made this change to increase make applications cost more to run, 
> it is illegal.  
>
> There is no problem with them offering a premium service for memcache, but 
> it is illegal to degrade the previous service to force people into the paid 
> model.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: major MEMCACHE changes started October 5th

[Alex Burgel]

I filed a production issue on this. I suggest everyone who is seeing this 
problem do the same.

I also switched the dedicated memcache service until this is fixed. so far 
so good.

On Thursday, October 10, 2013 2:58:37 PM UTC-4, James Gilliam wrote:
>
> Here is what I know so far ...
>
> I ran the premium memcache for 18 hours yesterday and saw no cache 
> purging.  My cache size grew to 110 megs with 6000 items and the oldest 
> cache item was 18 hours.Since the minimum cache is 1 gig, I assume 
> there would be no purging until the size was above that.
>
> When using shared cached, the purging is now very aggressive.  Since 
> Saturday, I have seen the oldest cache item at less than 15 mins, total 
> cache of 10 megs and items at 500 or less.  It does vary a lot but as the 
> purging increases the cost of the application increases, as does the 
> performance.  Right now it is performing better than these numbers but with 
> the shared cache you never know when they will purge the shit out of the 
> cache and the application starts consuming other resources to compensate 
> for poor memcache performance.
>
> Something clearly changed on Saturday when I saw my costs increase by 4 to 
> 5 times as a direct result of memory cache purging. 
>
> The only way you can guarantee some amount of memory cache is with the 
> premium memcache product and the minimum size is 1 gig (roughly 10 times 
> what I really need).  This costs 12 cents per hour per gig, 2.88 per day, 
> roughly 90 per month, over 1000 per year.
>
> It would be less painful if you could allocate smallest than 1 gig, e.g. 
> 100 megs.  So the costs would drop by a factor of 10.
>
> I will be forced to opt for the premium memcache if the shared cache 
> continues to perform so poorly, or rewrite lots of code so I am less 
> dependent on memcache.  Both of these are direct burdens from GAE changing 
> shared memcache.
>
> On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:
>>
>> GAE has drastically changed the way MEMCACHE works; as a result my 
>> application (ogeekcom) overall usage jumped by approximately 5 timesd. 
>>  Like a 400% increase in price.
>>
>> Specifically, they are purging shared memcache very aggressively -- 
>> possible in an effort to force people to signup for paid memcache.
>>
>> As a result of this change, my application is using many more datastore 
>> reads and many more instances to compensate for the poor memcache 
>> performance.
>>
>> Like always, this was done without any announcement at all.
>>
>> If they made this change to increase make applications cost more to run, 
>> it is illegal.  
>>
>> There is no problem with them offering a premium service for memcache, 
>> but it is illegal to degrade the previous service to force people into the 
>> paid model.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: GAE loading new instances on each home page access

[Francois Masurel]

Just switched back to automatic scaling and everything seems to be back to 
normal, no more crazy instance loading.

There were definitely something buggy on GAE side which seems to have been 
fixed.



On Friday, October 11, 2013 1:23:10 PM UTC+2, Francois Masurel wrote:
>
> Just missed a point : basic scaling instances are still billed like 
> backend instance hours and not front end hours even for the default module.
>
> So we have only 8 hours offered instead of 28 hours for automatic scaling 
> instances :-(
>
> Please Google fix this, quick.
>
> Created an issue for it : 
> https://code.google.com/p/googleappengine/issues/detail?id=10105
>
> François
>
>
>
> On Wednesday, October 9, 2013 5:35:25 PM UTC+2, Francois Masurel wrote:
>>
>> Ok guys,
>>
>> Looks like I solved the problem by setting *basic scaling* instead of 
>> *automatic 
>> scaling *for my default module, it works also with *manual scaling*.
>>
>> No more crazy instance starts and stops, that's cool :-)
>>
>> There don't seem to be the 10 concurrent requests maxi limit any more.
>>
>> Still have one question : how does it scale ?  Here is what is said in 
>> the documentation :
>>
>> *A basic scaling module version is configured with a maximum number of 
>> instances using the max_instances basic scaling parameter.*
>>
>> *The number of live instances scales with the processing volume.*
>>
>>
>> What is "processing volume" exactly?  How is it evaluated? There is no 
>> pending latency or such parameters.
>>
>> I could start a few more instances by doing some stress tests.  Started 
>> instance are kept alive as defined by the idle_timeout basic scaling 
>> parameter.  But there still seem to have some user facing loading requests.
>>
>> I guess the automatic scaling mode still needs polish and is not 
>> production ready yet, unless I missed something.
>>
>> François
>>
>>
>> On Wednesday, October 9, 2013 2:38:50 PM UTC+2, Francois Masurel wrote:
>>>
>>> Please, STOP killing instances after 30s of inactivity (average time 
>>> measured, might be even less).
>>>
>>> Aren't we paying a minimum 15 minutes startup fee for those started and 
>>> killed instances ?  May be you could increase this delay to at least 2 
>>> minutes ?
>>>
>>> These {censored} loading requests make our app totally unusable.
>>>
>>>
>>>
>>> On Wednesday, October 9, 2013 12:33:52 PM UTC+2, Francois Masurel wrote:

 Same problem with F2 instances.  This f... scheduler is getting us 
 crazy.

 There is nothing wrong with starting a new instance but it should wait 
 for the instance being properly loaded before redirecting traffic to it.



 On Wednesday, October 9, 2013 12:09:42 PM UTC+2, Francois Masurel wrote:
>
> It looks like GAE Java instances don't handle more than 10 concurrent 
> requests.
>
> As a result, our GWT app home page which makes about 20 requests needs 
> at least 2 instances loaded to get displayed.
>
> And as instances are shutdown very agressivelly (a few seconds), our 
> users are getting hit by these loading requests pretty often.
>
> This is making our app totally useless (~10-20s to load a new 
> instance).
>
> Is there a way to increase this max number of concurrent requests ?
>
> François
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: GAE loading new instances on each home page access

[Francois Masurel]

Just missed a point : basic scaling instances are still billed like backend 
instance hours and not front end hours even for the default module.

So we have only 8 hours offered instead of 28 hours for automatic scaling 
instances :-(

Please Google fix this, quick.

Created an issue for it : 
https://code.google.com/p/googleappengine/issues/detail?id=10105

François



On Wednesday, October 9, 2013 5:35:25 PM UTC+2, Francois Masurel wrote:
>
> Ok guys,
>
> Looks like I solved the problem by setting *basic scaling* instead of 
> *automatic 
> scaling *for my default module, it works also with *manual scaling*.
>
> No more crazy instance starts and stops, that's cool :-)
>
> There don't seem to be the 10 concurrent requests maxi limit any more.
>
> Still have one question : how does it scale ?  Here is what is said in the 
> documentation :
>
> *A basic scaling module version is configured with a maximum number of 
> instances using the max_instances basic scaling parameter.*
>
> *The number of live instances scales with the processing volume.*
>
>
> What is "processing volume" exactly?  How is it evaluated? There is no 
> pending latency or such parameters.
>
> I could start a few more instances by doing some stress tests.  Started 
> instance are kept alive as defined by the idle_timeout basic scaling 
> parameter.  But there still seem to have some user facing loading requests.
>
> I guess the automatic scaling mode still needs polish and is not 
> production ready yet, unless I missed something.
>
> François
>
>
> On Wednesday, October 9, 2013 2:38:50 PM UTC+2, Francois Masurel wrote:
>>
>> Please, STOP killing instances after 30s of inactivity (average time 
>> measured, might be even less).
>>
>> Aren't we paying a minimum 15 minutes startup fee for those started and 
>> killed instances ?  May be you could increase this delay to at least 2 
>> minutes ?
>>
>> These {censored} loading requests make our app totally unusable.
>>
>>
>>
>> On Wednesday, October 9, 2013 12:33:52 PM UTC+2, Francois Masurel wrote:
>>>
>>> Same problem with F2 instances.  This f... scheduler is getting us crazy.
>>>
>>> There is nothing wrong with starting a new instance but it should wait 
>>> for the instance being properly loaded before redirecting traffic to it.
>>>
>>>
>>>
>>> On Wednesday, October 9, 2013 12:09:42 PM UTC+2, Francois Masurel wrote:

 It looks like GAE Java instances don't handle more than 10 concurrent 
 requests.

 As a result, our GWT app home page which makes about 20 requests needs 
 at least 2 instances loaded to get displayed.

 And as instances are shutdown very agressivelly (a few seconds), our 
 users are getting hit by these loading requests pretty often.

 This is making our app totally useless (~10-20s to load a new instance).

 Is there a way to increase this max number of concurrent requests ?

 François






-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Can't create application "invalid name" no matter what I choose..

[Ben Ritchie]

No matter what I choose as the app name it says "invalid"


I guess your servers are broken




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

[Mitul Golakiya]

Thanks Alftred, It worked...

Thank you so much for you help...



On Friday, October 11, 2013 1:22:22 PM UTC+5:30, Alfred Fuller wrote:
>
> Thats because you are using 'db' instead of 'ndb'
>
>
> On Fri, Oct 11, 2013 at 12:44 AM, Mitul Golakiya 
> 
> > wrote:
>
>> I have tried this code, but still it is creating entity as class name (*
>> DynamicEntity*) not as given my custom entity name(*MyCustomEntity*):
>>
>> class DynamicEntity(db.Expando):
>> @classmethod
>> def _get_kind(cls):
>> return 'MyCustomEntity'
>>
>> newEntity = DynamicEntity()
>> newEntity.fname = "Test"
>> newEntity.put()
>>
>>
>>
>>
>> On Friday, October 11, 2013 11:39:43 AM UTC+5:30, Alfred Fuller wrote:
>>
>>> FYI, the following function should be sufficient to register a dynamic 
>>> kind in the kind_map:
>>>
>>> def addExpandoForKind(kind):
>>>   class Dummy(db.Expando):
>>> @classmethod
>>> def _get_kind(cls):
>>>   return kind
>>>
>>> though it is not a "good" solution :-) (there are all sorts of memory, 
>>> thread and request related issues)
>>>
>>>
>>>
>>> On Thu, Oct 10, 2013 at 10:54 PM, timh  wrote:
>>>
  You could use metaclasses to create dynamic models, however you would 
 always have to create a matching class (which would register the class to 
 kind map)
 before you could perform any query.

 I think you should elaborate more on what you are trying to do, we may 
 be able to suggest an approach which doesn't require you to go this far.

 Regards

 Tim


 On Wednesday, October 9, 2013 9:32:03 PM UTC+8, Mitul Golakiya wrote:
>
> Hello All,
>
> We are developing one app on GAE with python.
>
> We are using datastore.py for querying data from datastore, because we 
> have to define our entities at runtime. So we can not use db.Model to 
> define models. And retrieve data by models.
>
> We have to use "Not Equal" operator to retrieve some data from 
> datastore.
> Suppose, I want to retrieve all Persons, with name is not equal to 
> "Test".
>
> Here is my code :
>
> filerObj = {}
>> filterObj["name!="] = "Test"
>>
>> dbObj = datastore.Query("Person", filterObj)
>> dbObj.Run()
>> recordsList = dbObj.Get()
>>
>
>
> By this code, we are getting empty result.
> Any idea about what's wrong ??
>
> Thanks in Advance...
>
  -- 
 You received this message because you are subscribed to the Google 
 Groups "Google App Engine" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to google-appengi...@**googlegroups.com.
 To post to this group, send email to google-a...@googlegroups.**com.

 Visit this group at 
 http://groups.google.com/**group/google-appengine
 .
 For more options, visit 
 https://groups.google.com/**groups/opt_out
 .

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com .
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

[timh]

Ignore the namespace comment.

On Friday, October 11, 2013 5:43:43 PM UTC+8, timh wrote:
>
> As I said
>
> I would use a metaclass to create classes on demand.
> However you would need to use some name mangling scheme so that each users 
> class name was garunteed to be unique otherwise 
> you could end up in all sorts of mess.
>
> Alternately specify an entity name as a property of the class (or use it 
> to define the namespace), and use the same class for all users, along with 
> their own custom properties.
>
> this would be a much simpler approach.
>
> Tim
>
>
> On Friday, October 11, 2013 3:50:54 PM UTC+8, Mitul Golakiya wrote:
>>
>> Thanks Tim Hoffman,
>>
>> We are creating one app in which, on admin side User can create his own 
>> entities and can create fields in that entity.
>> So as the documentation I have read, to store any entity in datastore 
>> which have dynamic fields, we have to use ndb or Expando.
>>
>> But I am getting entity name at runtime, so I am not able to define 
>> static classes for my entities.
>> So how can I create dynamic entities with dynamic field support ??
>>
>>
>> On Friday, October 11, 2013 11:24:49 AM UTC+5:30, timh wrote:
>>>
>>> You could use metaclasses to create dynamic models, however you would 
>>> always have to create a matching class (which would register the class to 
>>> kind map)
>>> before you could perform any query.
>>>
>>> I think you should elaborate more on what you are trying to do, we may 
>>> be able to suggest an approach which doesn't require you to go this far.
>>>
>>> Regards
>>>
>>> Tim
>>>
>>>
>>> On Wednesday, October 9, 2013 9:32:03 PM UTC+8, Mitul Golakiya wrote:

 Hello All,

 We are developing one app on GAE with python.

 We are using datastore.py for querying data from datastore, because we 
 have to define our entities at runtime. So we can not use db.Model to 
 define models. And retrieve data by models.

 We have to use "Not Equal" operator to retrieve some data from 
 datastore.
 Suppose, I want to retrieve all Persons, with name is not equal to 
 "Test".

 Here is my code :

 filerObj = {}
> filterObj["name!="] = "Test"
>
> dbObj = datastore.Query("Person", filterObj)
> dbObj.Run()
> recordsList = dbObj.Get()
>


 By this code, we are getting empty result.
 Any idea about what's wrong ??

 Thanks in Advance...

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

[timh]

As I said

I would use a metaclass to create classes on demand.
However you would need to use some name mangling scheme so that each users 
class name was garunteed to be unique otherwise 
you could end up in all sorts of mess.

Alternately specify an entity name as a property of the class (or use it to 
define the namespace), and use the same class for all users, along with 
their own custom properties.

this would be a much simpler approach.

Tim


On Friday, October 11, 2013 3:50:54 PM UTC+8, Mitul Golakiya wrote:
>
> Thanks Tim Hoffman,
>
> We are creating one app in which, on admin side User can create his own 
> entities and can create fields in that entity.
> So as the documentation I have read, to store any entity in datastore 
> which have dynamic fields, we have to use ndb or Expando.
>
> But I am getting entity name at runtime, so I am not able to define static 
> classes for my entities.
> So how can I create dynamic entities with dynamic field support ??
>
>
> On Friday, October 11, 2013 11:24:49 AM UTC+5:30, timh wrote:
>>
>> You could use metaclasses to create dynamic models, however you would 
>> always have to create a matching class (which would register the class to 
>> kind map)
>> before you could perform any query.
>>
>> I think you should elaborate more on what you are trying to do, we may be 
>> able to suggest an approach which doesn't require you to go this far.
>>
>> Regards
>>
>> Tim
>>
>>
>> On Wednesday, October 9, 2013 9:32:03 PM UTC+8, Mitul Golakiya wrote:
>>>
>>> Hello All,
>>>
>>> We are developing one app on GAE with python.
>>>
>>> We are using datastore.py for querying data from datastore, because we 
>>> have to define our entities at runtime. So we can not use db.Model to 
>>> define models. And retrieve data by models.
>>>
>>> We have to use "Not Equal" operator to retrieve some data from datastore.
>>> Suppose, I want to retrieve all Persons, with name is not equal to 
>>> "Test".
>>>
>>> Here is my code :
>>>
>>> filerObj = {}
 filterObj["name!="] = "Test"

 dbObj = datastore.Query("Person", filterObj)
 dbObj.Run()
 recordsList = dbObj.Get()

>>>
>>>
>>> By this code, we are getting empty result.
>>> Any idea about what's wrong ??
>>>
>>> Thanks in Advance...
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Servlets 3.0, And In Query cursors

[srk]

Hi gae team,

Can you please give an update when app engine will start supporting servets 
3.0. I am desperately looking for it. I would be grateful if someone could 
provide me an approximate date. thanks a lot in advance.

Also can you kindly give us an update when gae will start returning query 
cursors for IN operators. This will be of great use for lot of apps. I have 
posted the limitation below that is mentioned in an app engine doc. 


   - Because the NOT_EQUAL and IN operators are implemented with multiple 
   queries, queries that use them do not support cursors, nor do composite 
   queries constructed with the CompositeFilterOperator.or method

thanks a lot once again.

Srikanth

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

[Alfred Fuller]

Thats because you are using 'db' instead of 'ndb'


On Fri, Oct 11, 2013 at 12:44 AM, Mitul Golakiya wrote:

> I have tried this code, but still it is creating entity as class name (*
> DynamicEntity*) not as given my custom entity name(*MyCustomEntity*):
>
> class DynamicEntity(db.Expando):
> @classmethod
> def _get_kind(cls):
> return 'MyCustomEntity'
>
> newEntity = DynamicEntity()
> newEntity.fname = "Test"
> newEntity.put()
>
>
>
>
> On Friday, October 11, 2013 11:39:43 AM UTC+5:30, Alfred Fuller wrote:
>
>> FYI, the following function should be sufficient to register a dynamic
>> kind in the kind_map:
>>
>> def addExpandoForKind(kind):
>>   class Dummy(db.Expando):
>> @classmethod
>> def _get_kind(cls):
>>   return kind
>>
>> though it is not a "good" solution :-) (there are all sorts of memory,
>> thread and request related issues)
>>
>>
>>
>> On Thu, Oct 10, 2013 at 10:54 PM, timh  wrote:
>>
>>>  You could use metaclasses to create dynamic models, however you would
>>> always have to create a matching class (which would register the class to
>>> kind map)
>>> before you could perform any query.
>>>
>>> I think you should elaborate more on what you are trying to do, we may
>>> be able to suggest an approach which doesn't require you to go this far.
>>>
>>> Regards
>>>
>>> Tim
>>>
>>>
>>> On Wednesday, October 9, 2013 9:32:03 PM UTC+8, Mitul Golakiya wrote:

 Hello All,

 We are developing one app on GAE with python.

 We are using datastore.py for querying data from datastore, because we
 have to define our entities at runtime. So we can not use db.Model to
 define models. And retrieve data by models.

 We have to use "Not Equal" operator to retrieve some data from
 datastore.
 Suppose, I want to retrieve all Persons, with name is not equal to
 "Test".

 Here is my code :

 filerObj = {}
> filterObj["name!="] = "Test"
>
> dbObj = datastore.Query("Person", filterObj)
> dbObj.Run()
> recordsList = dbObj.Get()
>


 By this code, we are getting empty result.
 Any idea about what's wrong ??

 Thanks in Advance...

>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengi...@**googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.**com.
>>>
>>> Visit this group at 
>>> http://groups.google.com/**group/google-appengine
>>> .
>>> For more options, visit 
>>> https://groups.google.com/**groups/opt_out
>>> .
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] how reduce memory usage on mapreduce controller callback

[Moises Belchin]

Hi Vinny,

Thanks for the tips, but actually I'm not loading a file. I'm only using
mapreduce lib for read all the entities for one of my kinds, work with them
(I only read some properties to compose the csv line format) and then I
write to CSV file on cloud storage using mapreduce FileOutputWriter.

Any idea why I'm getting this Criticals memory errors?

Thanks all again.


Saludos.
Moisés Belchín.


2013/10/10 Vinny P 

> On Thu, Oct 10, 2013 at 5:47 AM, Moises Belchin 
>  wrote:
>
>> I'm getting a lot of memory limit critical errors in my app when I use
>> mapreduce library.
>> I'm working with Python 2.7, app engine 1.8.5 and F2 instances with 256MB.
>>
>> Has anyone got these critical errors?
>>
>>
>
> Hello Moises,
>
> How large is your input file? Are you loading the entire file into memory
> at once? If so, try moving to a BlobstoreLineInputReader which reads in one
> line at a time - it reduces the memory being used during file processing.
>
>
> -
> -Vinny P
> Technology & Media Advisor
> Chicago, IL
>
> App Engine Code Samples: http://www.learntogoogleit.com
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

[Mitul Golakiya]

Thanks Tim Hoffman,

We are creating one app in which, on admin side User can create his own 
entities and can create fields in that entity.
So as the documentation I have read, to store any entity in datastore which 
have dynamic fields, we have to use ndb or Expando.

But I am getting entity name at runtime, so I am not able to define static 
classes for my entities.
So how can I create dynamic entities with dynamic field support ??


On Friday, October 11, 2013 11:24:49 AM UTC+5:30, timh wrote:
>
> You could use metaclasses to create dynamic models, however you would 
> always have to create a matching class (which would register the class to 
> kind map)
> before you could perform any query.
>
> I think you should elaborate more on what you are trying to do, we may be 
> able to suggest an approach which doesn't require you to go this far.
>
> Regards
>
> Tim
>
>
> On Wednesday, October 9, 2013 9:32:03 PM UTC+8, Mitul Golakiya wrote:
>>
>> Hello All,
>>
>> We are developing one app on GAE with python.
>>
>> We are using datastore.py for querying data from datastore, because we 
>> have to define our entities at runtime. So we can not use db.Model to 
>> define models. And retrieve data by models.
>>
>> We have to use "Not Equal" operator to retrieve some data from datastore.
>> Suppose, I want to retrieve all Persons, with name is not equal to "Test".
>>
>> Here is my code :
>>
>> filerObj = {}
>>> filterObj["name!="] = "Test"
>>>
>>> dbObj = datastore.Query("Person", filterObj)
>>> dbObj.Run()
>>> recordsList = dbObj.Get()
>>>
>>
>>
>> By this code, we are getting empty result.
>> Any idea about what's wrong ??
>>
>> Thanks in Advance...
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

[Mitul Golakiya]

I have tried this code, but still it is creating entity as class name (*
DynamicEntity*) not as given my custom entity name(*MyCustomEntity*):

class DynamicEntity(db.Expando):
@classmethod
def _get_kind(cls):
return 'MyCustomEntity'

newEntity = DynamicEntity()
newEntity.fname = "Test"
newEntity.put()



On Friday, October 11, 2013 11:39:43 AM UTC+5:30, Alfred Fuller wrote:
>
> FYI, the following function should be sufficient to register a dynamic 
> kind in the kind_map:
>
> def addExpandoForKind(kind):
>   class Dummy(db.Expando):
> @classmethod
> def _get_kind(cls):
>   return kind
>
> though it is not a "good" solution :-) (there are all sorts of memory, 
> thread and request related issues)
>
>
>
> On Thu, Oct 10, 2013 at 10:54 PM, timh >wrote:
>
>> You could use metaclasses to create dynamic models, however you would 
>> always have to create a matching class (which would register the class to 
>> kind map)
>> before you could perform any query.
>>
>> I think you should elaborate more on what you are trying to do, we may be 
>> able to suggest an approach which doesn't require you to go this far.
>>
>> Regards
>>
>> Tim
>>
>>
>> On Wednesday, October 9, 2013 9:32:03 PM UTC+8, Mitul Golakiya wrote:
>>>
>>> Hello All,
>>>
>>> We are developing one app on GAE with python.
>>>
>>> We are using datastore.py for querying data from datastore, because we 
>>> have to define our entities at runtime. So we can not use db.Model to 
>>> define models. And retrieve data by models.
>>>
>>> We have to use "Not Equal" operator to retrieve some data from datastore.
>>> Suppose, I want to retrieve all Persons, with name is not equal to 
>>> "Test".
>>>
>>> Here is my code :
>>>
>>> filerObj = {}
 filterObj["name!="] = "Test"

 dbObj = datastore.Query("Person", filterObj)
 dbObj.Run()
 recordsList = dbObj.Get()

>>>
>>>
>>> By this code, we are getting empty result.
>>> Any idea about what's wrong ??
>>>
>>> Thanks in Advance...
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com .
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.