Re: [google-appengine] Very low performance in appengine endpoints

[Vinny P]

On Thu, May 8, 2014 at 5:40 AM, DiTieM  wrote:

> Thank you Vinny. The log shows about 120ms to 240ms (so yes, I am still
> "suspecting" endpoints is slow). The browser shows about 1.5 seconds (and
> that is what I really perceive). I am investigating now where the time is
> gone. I am not sure it is due to the network speed as I have tested in
> high-speed networks too.
>


If you want to compare speed what you could do is create a simple Endpoints
application and then write the same in low level code (handle JSON encoding
yourself, map the correct keys, routing, etc). Make sure you issue the
correct CORS headers
as
well.


On Thu, May 8, 2014 at 5:40 AM, DiTieM  wrote:

> I am not sure it is due to the network speed as I have tested in
> high-speed networks too.
>


Try opening up a Compute Engine machine, SSHing in, and making a request to
your application. The connection between App Engine and the Compute Engine
VM should flow within Google's own internal network; your measurements
won't be affected by crossing networks.


-
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[DiTieM]

Thank you Vinny.

How large are the objects you're storing in Memcache? Memcache objects can 
> be as large as 1 MB: you can fit a lot of data into a single object.  Push 
> as much data into a single object or set of objects as you can, then 
> retrieve them in one call. 
>

Just for the record, in case someone reads this thread in the future. I did 
many changes, and reduced the time from 10 seconds to few ms (although this 
is the server time, the user perceived time is still too long). There are 
no brilliant ideas, just pure logic. Basically:

1- Using "multi" functions improves quite significantly.
2- Storing big object in the cache is not such a good idea, at least in my 
case it did not work (or I did not know how to implement it). What I had to 
do is to divide the item in 2 parts: a summary and detailed view. So when I 
am getting a list of items on the screen, I just get the minimal necessary 
information. Most of the fields removed where just numbers, but the fact of 
using JSON makes the payload to be much bigger.

On a second step I did something that can be "tricky". Instead of storing 
the Datastore object and convert it to endpoints message, I store the 
message itself. In this way I save the processing of generating the 
endpoints message. 

Accessing large items in memcache is "expensive" (80ms-100ms in my case). 
So I did a list of the datastore keys ID (notice, 
datastore_object.key.id()) of the summary objects. I use this list of keys 
id to multi_get all the datastore summary elements from the memcache in 1 
batch.  

The final result (41ms):


The real time is a bit more than what the chart shows, because the decoding 
and encoding  of endpoints is not taken into account. The log shows about 
120ms to 240ms (so yes, I am still "suspecting" endpoints is slow). The 
browser shows about 1.5 seconds (and that is what I really perceive). I am 
investigating now where the time is gone. I am not sure it is due to the 
network speed as I have tested in high-speed networks too. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[Vinny P]

 On Sun, May 4, 2014 at 4:19 PM, DiTieM  wrote:

> Hello Vinni, Thank you for the quick reply.
> I did install appstats and try to find out if there were the bottle necks.
>

So as you can see from the AppStats graph, the vast majority of the request
is spent waiting for datastore and memcache operations - not with Endpoints
processing.

The problem here is you're doing too many calls to the datastore and
memcache. Even worse, you're making these calls serially, rather than in
parallel. I see you posted some code; I'm not entirely sure what it's
supposed to do (perhaps you could expand further), but the short
explanation is that you need to rewrite the code to make these calls
asynchronously, to batch your get/set calls, and to make much fewer calls
in the first place. See the memcache documentation at
https://developers.google.com/appengine/docs/python/memcache/clientclassfor
async info.


On Mon, May 5, 2014 at 5:37 AM, DiTieM  wrote:
>
> I thought of using this method to avoid putting big object in memcache.
>


How large are the objects you're storing in Memcache? Memcache objects can
be as large as 1 MB: you can fit a lot of data into a single object.  Push
as much data into a single object or set of objects as you can, then
retrieve them in one call.

 Consider also purchasing dedicated memcache (
https://developers.google.com/appengine/docs/adminconsole/memcache ) and
see if that helps performance.


-
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[timh]

Hi

Completely missed the tag (can't say I ever look at them)

Looking at the graph you are doing an awful lot of memcache gets and sets 
one after another and each rpc costs time (round trip etc)
Just looking at the graph it tells me you code is behaving inefficiently. 
 You want as few steps in the graph as possible  as each one has a fairly 
fixed overhead in terms of roundtrip.
For instance if there is 30ms base overhead for each round trip excluding 
any work, and you do 100 rpcs then thats an immediate 3000ms added to you 
processing time.

These means start batching operations,  for instance you can do get_multi 
and set_multi when using memcache.  

I am having trouble working out what you code is trying to do (hard to read 
here and incomplete), but basically you need to try and batch all 
operations.
Its not clear why you have so many memcache.set's interleaved with all the 
gets.  Again try and do it all at the end in a few operations.

The evidence as to where all you time is going is in the graph.

T


On Monday, May 5, 2014 6:37:36 PM UTC+8, DiTieM wrote:
>
> You images are too small to see detail
>>
>
>
> 
> Apologies, "Google Groups" resized. I did not notice till you mentioned. 
> The image is 1.7 mg and it was 1400 pixels wide. I cropped it, hope this 
> time it will be more visible.
>  
>
>> You probably need to look at the code that retrieves entities.  Look at 
>> using larger batch sizes.  You haven't mentioned what language or 
>> datastore access mechanism (ie on python, db or ndb) etc.
>>
>
> the tag of the message: python27, I thought people used that for "saying 
> the language", I guess not easy to read. I am using ndb, as db is 
> deprecated.
>  
> Removing some if-then-else, the code is like this:
>
> query = TATripDB.query( TATripDB.draft == 0 )
>
>  if request.offset:
>  cursor = Cursor( urlsafe = request.offset )
>  else:
>  cursor = None
>
> # limit = request.limit
> rows, next_cursor, more  = query.fetch_page( 200, start_cursor = 
> cursor )
>
> ret = [ ]
> com_cache = { }
> cur_cache = { }
>
> if rows:
> unique_temporal_user = self.get_ta_profile( rows[ 0 
> ].from_agency )
>
> for r in rows:
> ret.append( get_tatrip_answer( unique_temporal_user
> # user_set[ r.key.parent( 
> ).id( ) ]
>   , r
>   , vlc_of_trip_key( 
> userID_key, r.key )
>   , cur_cache
>   , com_cache ) )
>
>  return ExploreTATripA( trips  = ret
>  , next_token = next_token 
>  , cur_rates  = cur_rates  )
>
>
> The version of the memcache introduce some changes:
>
> tatrips_keys = memcache.get( 'TATRIPS_KEYS' )
>
> if tatrips_keys:
> rows = map( lambda x: self.get_tatrip_from_key( x 
> ), tatrips_keys )
>
> I thought of using this method to avoid putting big object in memcache. 
> This is maybe what you refer as getting in larger batch size? Later I am 
> going to do some more experiments. Thank you for the suggestion!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[DiTieM]

>
> You images are too small to see detail
>


Apologies, "Google Groups" resized. I did not notice till you mentioned. 
The image is 1.7 mg and it was 1400 pixels wide. I cropped it, hope this 
time it will be more visible.
 

> You probably need to look at the code that retrieves entities.  Look at 
> using larger batch sizes.  You haven't mentioned what language or 
> datastore access mechanism (ie on python, db or ndb) etc.
>

the tag of the message: python27, I thought people used that for "saying 
the language", I guess not easy to read. I am using ndb, as db is 
deprecated.
 
Removing some if-then-else, the code is like this:

query = TATripDB.query( TATripDB.draft == 0 )

 if request.offset:
 cursor = Cursor( urlsafe = request.offset )
 else:
 cursor = None

# limit = request.limit
rows, next_cursor, more  = query.fetch_page( 200, start_cursor = 
cursor )

ret = [ ]
com_cache = { }
cur_cache = { }

if rows:
unique_temporal_user = self.get_ta_profile( rows[ 0 
].from_agency )

for r in rows:
ret.append( get_tatrip_answer( unique_temporal_user
# user_set[ r.key.parent( 
).id( ) ]
  , r
  , vlc_of_trip_key( 
userID_key, r.key )
  , cur_cache
  , com_cache ) )

 return ExploreTATripA( trips  = ret
 , next_token = next_token 
 , cur_rates  = cur_rates  )


The version of the memcache introduce some changes:

tatrips_keys = memcache.get( 'TATRIPS_KEYS' )

if tatrips_keys:
rows = map( lambda x: self.get_tatrip_from_key( x 
), tatrips_keys )

I thought of using this method to avoid putting big object in memcache. 
This is maybe what you refer as getting in larger batch size? Later I am 
going to do some more experiments. Thank you for the suggestion!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[timh]

You images are too small to see detail. However just looking at the graph 
and number of steps it looks like when you fetch entities
you are doing it one entity at a time which means a lot of round trips 
(rpcs).  Which can slow things down enourmously.

You probably need to look at the code that retrieves entities.  Look at 
using larger batch sizes.  You haven't mentioned what language or 
datastore access mechanism (ie on python, db or ndb) etc.

T

On Monday, May 5, 2014 5:19:25 AM UTC+8, DiTieM wrote:
>
> Hello Vinni,
>
> Thank you for the quick reply. 
>
> I did install appstats and try to find out if there were the bottle necks. 
> The chart was basically reading from the datastore and then there was a 5 
> seconds without any "information". Now it behaves different because it 
> reads all from memcache (as you will see, there is no big improvement). I 
> am going to make an screenshot without the memcache, it will take some time.
>
>
> I also measured the time (using time.time( )) between different moments of 
> the answer. Here is the log:
>
>  22:31:54.677 GETTING KEYS 0.182710
>  22:31:58.785 GETTING TRIPS 4.290660
>  22:31:58.785 FILTERING 0.000270
>  22:31:58.794 GETtING TA PROFILE 0.008220
>  22:32:04.771 COMPOSING ANSWER 5.977440
>  22:32:04.854 GETTING RATES 0.083230
>
> If there is any other information you can consider useful just let me know.
>
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[DiTieM]

Hello Vinni,

Thank you for the quick reply. 

I did install appstats and try to find out if there were the bottle necks. 
The chart was basically reading from the datastore and then there was a 5 
seconds without any "information". Now it behaves different because it 
reads all from memcache (as you will see, there is no big improvement). I 
am going to make an screenshot without the memcache, it will take some time.


I also measured the time (using time.time( )) between different moments of 
the answer. Here is the log:

 22:31:54.677 GETTING KEYS 0.182710
 22:31:58.785 GETTING TRIPS 4.290660
 22:31:58.785 FILTERING 0.000270
 22:31:58.794 GETtING TA PROFILE 0.008220
 22:32:04.771 COMPOSING ANSWER 5.977440
 22:32:04.854 GETTING RATES 0.083230

If there is any other information you can consider useful just let me know.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[Vinny P]

On Sun, May 4, 2014 at 2:28 PM, DiTieM  wrote:

> I would like to ask if someone has experienced _very_ low performance when
> using GAE endpoints.
> I am basically retrieving 20 elements from the datastore and fill-in a
> "messages" of 44 fields, most of them integers and strings (from which only
> 16 are repeated, but usually do not have more than 1 or 2 elements). It
> takes about 6 seconds.
> Then I decided to store the message in the memcache and simple retrieve
> it. Now it takes about 3 seconds.
> Does anyone knows about endpoints performance? Does it has to do with the
> size of the message, the number of elements, both combined?
>


It's difficult to answer performance-related queries without understanding
what exactly your app does and any potential bottlenecks.

Can you install AppStats on your application (
https://developers.google.com/appengine/docs/java/tools/appstats ) and post
some example request profiles?


-
-Vinny P
Technology & Media Advisor
Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Very low performance in appengine endpoints

[DiTieM]

I would like to ask if someone has experienced _very_ low performance when 
using GAE endpoints. 

I am basically retrieving 20 elements from the datastore and fill-in a 
"messages" of 44 fields, most of them integers and strings (from which only 
16 are repeated, but usually do not have more than 1 or 2 elements). It 
takes about 6 seconds.

Then I decided to store the message in the memcache and simple retrieve it. 
Now it takes about 3 seconds.

Does anyone knows about endpoints performance? Does it has to do with the 
size of the message, the number of elements, both combined?

Thanks in advance,
  David

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.