date:20140505

Re: [google-appengine] Very low performance in appengine endpoints

2014-05-05 Thread timh

You images are too small to see detail. However just looking at the graph 
and number of steps it looks like when you fetch entities
you are doing it one entity at a time which means a lot of round trips 
(rpcs).  Which can slow things down enourmously.

You probably need to look at the code that retrieves entities.  Look at 
using larger batch sizes.  You haven't mentioned what language or 
datastore access mechanism (ie on python, db or ndb) etc.

T

On Monday, May 5, 2014 5:19:25 AM UTC+8, DiTieM wrote:
>
> Hello Vinni,
>
> Thank you for the quick reply. 
>
> I did install appstats and try to find out if there were the bottle necks. 
> The chart was basically reading from the datastore and then there was a 5 
> seconds without any "information". Now it behaves different because it 
> reads all from memcache (as you will see, there is no big improvement). I 
> am going to make an screenshot without the memcache, it will take some time.
>
>
> I also measured the time (using time.time( )) between different moments of 
> the answer. Here is the log:
>
>  22:31:54.677 GETTING KEYS 0.182710
>  22:31:58.785 GETTING TRIPS 4.290660
>  22:31:58.785 FILTERING 0.000270
>  22:31:58.794 GETtING TA PROFILE 0.008220
>  22:32:04.771 COMPOSING ANSWER 5.977440
>  22:32:04.854 GETTING RATES 0.083230
>
> If there is any other information you can consider useful just let me know.
>
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

2014-05-05 Thread DiTieM


>
> You images are too small to see detail
>


Apologies, "Google Groups" resized. I did not notice till you mentioned. 
The image is 1.7 mg and it was 1400 pixels wide. I cropped it, hope this 
time it will be more visible.
 

> You probably need to look at the code that retrieves entities.  Look at 
> using larger batch sizes.  You haven't mentioned what language or 
> datastore access mechanism (ie on python, db or ndb) etc.
>

the tag of the message: python27, I thought people used that for "saying 
the language", I guess not easy to read. I am using ndb, as db is 
deprecated.
 
Removing some if-then-else, the code is like this:

query = TATripDB.query( TATripDB.draft == 0 )

 if request.offset:
 cursor = Cursor( urlsafe = request.offset )
 else:
 cursor = None

# limit = request.limit
rows, next_cursor, more  = query.fetch_page( 200, start_cursor = 
cursor )

ret = [ ]
com_cache = { }
cur_cache = { }

if rows:
unique_temporal_user = self.get_ta_profile( rows[ 0 
].from_agency )

for r in rows:
ret.append( get_tatrip_answer( unique_temporal_user
# user_set[ r.key.parent( 
).id( ) ]
  , r
  , vlc_of_trip_key( 
userID_key, r.key )
  , cur_cache
  , com_cache ) )

 return ExploreTATripA( trips  = ret
 , next_token = next_token 
 , cur_rates  = cur_rates  )


The version of the memcache introduce some changes:

tatrips_keys = memcache.get( 'TATRIPS_KEYS' )

if tatrips_keys:
rows = map( lambda x: self.get_tatrip_from_key( x 
), tatrips_keys )

I thought of using this method to avoid putting big object in memcache. 
This is maybe what you refer as getting in larger batch size? Later I am 
going to do some more experiments. Thank you for the suggestion!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

2014-05-05 Thread timh

Hi

Completely missed the tag (can't say I ever look at them)

Looking at the graph you are doing an awful lot of memcache gets and sets 
one after another and each rpc costs time (round trip etc)
Just looking at the graph it tells me you code is behaving inefficiently. 
 You want as few steps in the graph as possible  as each one has a fairly 
fixed overhead in terms of roundtrip.
For instance if there is 30ms base overhead for each round trip excluding 
any work, and you do 100 rpcs then thats an immediate 3000ms added to you 
processing time.

These means start batching operations,  for instance you can do get_multi 
and set_multi when using memcache.  

I am having trouble working out what you code is trying to do (hard to read 
here and incomplete), but basically you need to try and batch all 
operations.
Its not clear why you have so many memcache.set's interleaved with all the 
gets.  Again try and do it all at the end in a few operations.

The evidence as to where all you time is going is in the graph.

T


On Monday, May 5, 2014 6:37:36 PM UTC+8, DiTieM wrote:
>
> You images are too small to see detail
>>
>
>
> 
> Apologies, "Google Groups" resized. I did not notice till you mentioned. 
> The image is 1.7 mg and it was 1400 pixels wide. I cropped it, hope this 
> time it will be more visible.
>  
>
>> You probably need to look at the code that retrieves entities.  Look at 
>> using larger batch sizes.  You haven't mentioned what language or 
>> datastore access mechanism (ie on python, db or ndb) etc.
>>
>
> the tag of the message: python27, I thought people used that for "saying 
> the language", I guess not easy to read. I am using ndb, as db is 
> deprecated.
>  
> Removing some if-then-else, the code is like this:
>
> query = TATripDB.query( TATripDB.draft == 0 )
>
>  if request.offset:
>  cursor = Cursor( urlsafe = request.offset )
>  else:
>  cursor = None
>
> # limit = request.limit
> rows, next_cursor, more  = query.fetch_page( 200, start_cursor = 
> cursor )
>
> ret = [ ]
> com_cache = { }
> cur_cache = { }
>
> if rows:
> unique_temporal_user = self.get_ta_profile( rows[ 0 
> ].from_agency )
>
> for r in rows:
> ret.append( get_tatrip_answer( unique_temporal_user
> # user_set[ r.key.parent( 
> ).id( ) ]
>   , r
>   , vlc_of_trip_key( 
> userID_key, r.key )
>   , cur_cache
>   , com_cache ) )
>
>  return ExploreTATripA( trips  = ret
>  , next_token = next_token 
>  , cur_rates  = cur_rates  )
>
>
> The version of the memcache introduce some changes:
>
> tatrips_keys = memcache.get( 'TATRIPS_KEYS' )
>
> if tatrips_keys:
> rows = map( lambda x: self.get_tatrip_from_key( x 
> ), tatrips_keys )
>
> I thought of using this method to avoid putting big object in memcache. 
> This is maybe what you refer as getting in larger batch size? Later I am 
> going to do some more experiments. Thank you for the suggestion!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Error deploying application: loop at 87% on sending batch files

2014-05-05 Thread Gilberto Torrezan Filho

Well, removing the symbolic links *helps* solving the issue, but sometimes 
the deployment still fails.

It's like 1 success in 3-4 attempts. Always with the same symptom: stuck at 
87% and sending files till the hard limit of 1 is reached.

Deploying an app has become a painful experience. Is anybody else with the 
same issue?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] ndb.add_flow_exception has no effect

2014-05-05 Thread Josh Whelchel (Loudr)

I am unable to remove certain exceptions from our logs via 
ndb.add_flow_exception.

class CompletedExpectedException(Exception):
> """This Exception is completely expected. 
> We don't need to see it as "errors" in the logs, or GAE dashboard 
> graphs."""
> pass
> # The following code, when on the 'global' app level, does not have an effect.
> ndb.add_flow_exception(CompletedExpectedException)
> # The following code does not work, eitherclass 
> WarmupHandler(webapp2.RequestHandler):
> """/_ah/warmup or /_ah/start"""
> def get(self):
> ndb.add_flow_exception(CompletedExpectedException)
>
> def post(self):
> self.get()
>
>
The code above does not prevent "CompletedExpectedException" from appearing 
in the logs.

This is primarily a problem because the only way that I know to 'fail' a 
deferred task is to throw an exception. (I know that I can use a direct 
taskqueue implementation with self.error(500) for this, but I'd still like 
to understand the functionality of ndb.add_flow_exception).

It is documented here: 
https://developers.google.com/appengine/docs/python/ndb/functions

As a side note - I would think this function, as documented, should belong 
in a different API than the datastore. *shrug*

Also asked on Stack 
Overflow,
 
if you would like that sweet, sweet reputation. :P

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: images api issues

2014-05-05 Thread Josh Whelchel (Loudr)

We are also seeing semi-frequent "TransformationError" exceptions being 
raised with no message attributed to them.

On Thursday, May 1, 2014 3:24:20 AM UTC-7, Rafael Sanches wrote:
>
> Anyone else having recurrent issues with the images API?
>
> I know Google doesn't guarantee any SLA for that API, but the error rates 
> are just terrible. 
>
> Can someone please let me know what's happening and when it's going to be 
> fixed?
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] App Engine 1.9.4 Released

2014-05-05 Thread Eric Han

Hi Everyone,
We are pleased to announce the latest release of Google App Engine 1.9.4. For
more about the focus of this and the last release, please see our blog post
(http://googlecloudplatform.blogspot.com/2014/04/announcing-release-of-google-app-engine-1-9-3.html).

Version 1.9.4 - April 30, 2014

*Python*

- Fixed an issue with the Search API where it does not correctly parse
expressions that use the subtraction operator without surrounding whitespaces.
- Fixed an issue with the devappserver Search API allowing searches
for a field with a negated value.
- Fixed an issue with the devappserver Search API not handling
searches for empty quotes the same way as production.
- Fixed an issue with the devappserver Search API not matching
documents with atom fields the same way as production.
- Fixed an issue with the devappserver Search API allowing expressions
that use the snippet operator over a numeric field.
- Fixed an issue with the devappserver Search API allowing sorting
expressions to use the snippet operator.
- Fixed an issue with the devappserver Search API not validating that
cursors are from a previous search.
- Fixed an issue with the devappserver Search API not parsing queries
that use the fuzzy search operator (~).
- Fixed an issue with the devappserver Search API not validating the
default value for sorting expressions on date fields.
- Fixed an issue with the devappserver memcache viewer not displaying
items with non-ASCII characters properly.
- https://code.google.com/p/googleappengine/issues/detail?id=9459
- Fixed an issue with the devappserver allowing arbitrarily large file
uploads using HTTP POST requests. It now matches production limits of 32MB.
- https://code.google.com/p/googleappengine/issues/detail?id=10384
- Fixed an issue with NDB deserialization looping infinitely when
nested value is None.
- https://code.google.com/p/googleappengine/issues/detail?id=10758

*PHP*

- Fixed an issue with libcurl giving a load error for Mac OS X Snow Leopard
clients.
- https://code.google.com/p/googleappengine/issues/detail?id=10405
- Fixed an issue with making simultaneous mysqli connections to the
same CloudSQL instance causing the request to time out.

thanks, Eric Han
Google Cloud Platform

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

62 matches

Mail list logo