Re: [google-appengine] Very low performance in appengine endpoints

[DiTieM]

Thank you Vinny.

How large are the objects you're storing in Memcache? Memcache objects can 
> be as large as 1 MB: you can fit a lot of data into a single object.  Push 
> as much data into a single object or set of objects as you can, then 
> retrieve them in one call. 
>

Just for the record, in case someone reads this thread in the future. I did 
many changes, and reduced the time from 10 seconds to few ms (although this 
is the server time, the user perceived time is still too long). There are 
no brilliant ideas, just pure logic. Basically:

1- Using "multi" functions improves quite significantly.
2- Storing big object in the cache is not such a good idea, at least in my 
case it did not work (or I did not know how to implement it). What I had to 
do is to divide the item in 2 parts: a summary and detailed view. So when I 
am getting a list of items on the screen, I just get the minimal necessary 
information. Most of the fields removed where just numbers, but the fact of 
using JSON makes the payload to be much bigger.

On a second step I did something that can be "tricky". Instead of storing 
the Datastore object and convert it to endpoints message, I store the 
message itself. In this way I save the processing of generating the 
endpoints message. 

Accessing large items in memcache is "expensive" (80ms-100ms in my case). 
So I did a list of the datastore keys ID (notice, 
datastore_object.key.id()) of the summary objects. I use this list of keys 
id to multi_get all the datastore summary elements from the memcache in 1 
batch.  

The final result (41ms):


The real time is a bit more than what the chart shows, because the decoding 
and encoding  of endpoints is not taken into account. The log shows about 
120ms to 240ms (so yes, I am still "suspecting" endpoints is slow). The 
browser shows about 1.5 seconds (and that is what I really perceive). I am 
investigating now where the time is gone. I am not sure it is due to the 
network speed as I have tested in high-speed networks too. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[DiTieM]

>
> You images are too small to see detail
>


Apologies, "Google Groups" resized. I did not notice till you mentioned. 
The image is 1.7 mg and it was 1400 pixels wide. I cropped it, hope this 
time it will be more visible.
 

> You probably need to look at the code that retrieves entities.  Look at 
> using larger batch sizes.  You haven't mentioned what language or 
> datastore access mechanism (ie on python, db or ndb) etc.
>

the tag of the message: python27, I thought people used that for "saying 
the language", I guess not easy to read. I am using ndb, as db is 
deprecated.
 
Removing some if-then-else, the code is like this:

query = TATripDB.query( TATripDB.draft == 0 )

 if request.offset:
 cursor = Cursor( urlsafe = request.offset )
 else:
 cursor = None

# limit = request.limit
rows, next_cursor, more  = query.fetch_page( 200, start_cursor = 
cursor )

ret = [ ]
com_cache = { }
cur_cache = { }

if rows:
unique_temporal_user = self.get_ta_profile( rows[ 0 
].from_agency )

for r in rows:
ret.append( get_tatrip_answer( unique_temporal_user
# user_set[ r.key.parent( 
).id( ) ]
  , r
  , vlc_of_trip_key( 
userID_key, r.key )
  , cur_cache
  , com_cache ) )

 return ExploreTATripA( trips  = ret
 , next_token = next_token 
 , cur_rates  = cur_rates  )


The version of the memcache introduce some changes:

tatrips_keys = memcache.get( 'TATRIPS_KEYS' )

if tatrips_keys:
rows = map( lambda x: self.get_tatrip_from_key( x 
), tatrips_keys )

I thought of using this method to avoid putting big object in memcache. 
This is maybe what you refer as getting in larger batch size? Later I am 
going to do some more experiments. Thank you for the suggestion!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Very low performance in appengine endpoints

[DiTieM]

Hello Vinni,

Thank you for the quick reply. 

I did install appstats and try to find out if there were the bottle necks. 
The chart was basically reading from the datastore and then there was a 5 
seconds without any "information". Now it behaves different because it 
reads all from memcache (as you will see, there is no big improvement). I 
am going to make an screenshot without the memcache, it will take some time.


I also measured the time (using time.time( )) between different moments of 
the answer. Here is the log:

 22:31:54.677 GETTING KEYS 0.182710
 22:31:58.785 GETTING TRIPS 4.290660
 22:31:58.785 FILTERING 0.000270
 22:31:58.794 GETtING TA PROFILE 0.008220
 22:32:04.771 COMPOSING ANSWER 5.977440
 22:32:04.854 GETTING RATES 0.083230

If there is any other information you can consider useful just let me know.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Very low performance in appengine endpoints

[DiTieM]

I would like to ask if someone has experienced _very_ low performance when 
using GAE endpoints. 

I am basically retrieving 20 elements from the datastore and fill-in a 
"messages" of 44 fields, most of them integers and strings (from which only 
16 are repeated, but usually do not have more than 1 or 2 elements). It 
takes about 6 seconds.

Then I decided to store the message in the memcache and simple retrieve it. 
Now it takes about 3 seconds.

Does anyone knows about endpoints performance? Does it has to do with the 
size of the message, the number of elements, both combined?

Thanks in advance,
  David

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Running on previous app engine version

[DiTieM]

I guess the answer is no, but right now that endpoints are not working on 
1.8.5, is there any way I can force my app to run on 1.8.4?

Thanks in advance,
   David

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Error parsing ProtoRPC request. Unable to parse request content: Base64 decoding error...

[DiTieM]

 

Hi

I have an application running on iOS with GAE endpoints. The application 
takes a PNG file, applies Base64 encoding and send it to the server. This 
was working nicely, but suddenly it stopped working. I suspect it has got 
broken in GAE environment 1.8.2. From the xCode console:

(lldb) po error

$4 = 0x1730 Error Domain=com.google.GTLJSONRPCErrorDomain Code=400 "The 
operation couldn’t be completed. (Error parsing ProtoRPC request (Unable to 
parse request content: Base64 decoding error: character mapping must return 
integer, None or unicode))" UserInfo=0x173ccce0 {error=Error parsing 
ProtoRPC request (Unable to parse request content: Base64 decoding error: 
character mapping must return integer, None or unicode), 
GTLStructuredError=GTLErrorObject 0x173ccc70: {message:"Error parsing 
ProtoRPC request (Unable to parse request content: Base64 decoding error: 
character mapping must return integer, None or unicode)" code:400 
data:[1]}, NSLocalizedFailureReason=(Error parsing ProtoRPC request (Unable 
to parse request content: Base64 decoding error: character mapping must 
return integer, None or unicode))}

The EndPoint function in python is not even called (I put a logging message 
on the first line). The server log does not even show an error in the 
logged call.

Can someone throw any light?


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: ndb query bug? query works on appengine website but not on GAE server

[DiTieM]

Just in case this may be of help for someone.

When using get_or_insert, the argument is forced to be an string. But when 
using ndb.Key( ..., THE_ID ), THE_ID can be a number.

The issue was this:

PatientDB.get_or_insert( patient_id ) # string

SurveyDB(  parent = ndb.Key( PatientDB, int(patient_id) ) # read note

Hence, when looking for surveys: SurveyDB.query( ancestor = a_patient.key 
).fetch( ) returns no results.

Note: I put int(patient_id) to see the clear effect. In reality, the 
patient_id was transformed to an int by endpoints API.

Datastore Viewer prints the key ids without commas, so it is impossible to 
see the bug there.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] ndb query bug? query works on appengine website but not on GAE server

[DiTieM]

I have been struggling with this for a couple of days and I am touching 
down. 

I have this (simplified for the case):

class PatientDB ( ndb.Model ):
name= ndb.StringProperty( )
doctor  = ndb.KeyProperty( )

class SurveyDB ( ndb.Model ):
date = ndb.IntegerProperty( )
survey_spec  = ndb.KeyProperty( kind=SurveySpecDB )
answers  = ndb.StructuredProperty( SurveyAnswerDB, repeated=True )


A SurveyDB is created with PatiendDB Key as parent. Additionally, the 
SurveyDB has a survey specification (survey_spec) that is a list of 
questions and the possible replies.

They query to retrieve the surveys is like:

query = SurveyDB.query( SurveyDB.survey_spec == survey_key, ancestor = 
patient.key ).order( -SurveyDB.date )

Notice that it is a query that implies ANCESTOR and a KEY. Other ways of 
writing the same (all tested):

   query = SurveyDB.query( ancestor = patient.key ).filter( 
SurveyDB.survey_spec == survey_spec ).order( -SurveyDB.date )
   query = ndb.gql( "SELECT * FROM SurveyDB WHERE ANCESTOR IS :1 AND 
survey_spec = :2 ORDER BY date DESC", patient.key, survey_spec )

By running this query *on the server* I get *ZERO* results. *In the 
appengine.google.com webpage I get the correct results*.

Extra things:
- removing the "order" does not do anything

- The index exists and it serving:
  - kind: SurveyDB
ancestor: yes
properties:
- name: survey_spec
- name: date
  direction: desc

- Simplifying the query to just use ancestor OR just use the survey_spec 
returns data


am I missing something or is this a bug?

Thans for any light!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] endpointscfg generate wrong discovery file (gen_discovery_doc)

[DiTieM]

Simple command:

endpointscfg.py  gen_discovery_doc -o ~/projects/TestApp/generated -f rpc 
services.UserApi

Generates wrong discovery file. The piece of code that is failing is this 
(I removed all that was not relevant)

@endpoints.api( name='user', version='v1', description='User API'
  , allowed_client_ids=PERMS
  , hostname=g_hostname )
class UserApi(remote.Service):

@endpoints.method(UserGetProfileStatsQ, UserGetProfileStatsA
  , path='user/stats'
  , name='user.get_stats'
  , http_method='GET'
  )
def get_profile_stats(self, request):
...
return UserGetProfileStatsA( ... )


@endpoints.method(UserSetProfileQ, message_types.VoidMessage
  , path='user/profile'
  , name='user.save_profile'
  , http_method='POST'
  )
def set_profile(self, request):
...
return message_types.VoidMessage( )


@endpoints.method(message_types.VoidMessage, UserGetProfileA
  , path='user/profile'
  , name='user.get_profile'
  , http_method='GET'
  )
def get_profile(self, request):
...
return UserGetProfileA( ... )



The first method, get_profile_stats, in the discovery file, instead of 

  "user.user.getstats": {
   "id": "user.user.getstats",
   "allowGet": true,
   "parameters": {
"$ref": "UserApiMessagesUserGetProfileStatsQ"
   },
   "returns": {
"$ref": "UserApiMessagesUserGetProfileStatsA"
   },

generates something like:

  "user.user.getstats": {
   "id": "user.user.getstats",
   "allowGet": true,
   "parameters": {
"dummy": {
 "type": "string"
},
"user_id": {
 "type": "string"
}
   },
   "returns": {
"$ref": "UserApiMessagesUserGetProfileStatsA"
   },

Basically, it has "expanded" the UserApiMessagesUserGetProfileStatsQ to its 
fields. Therefore, the next step generates libraries for iOS that do not 
require any argument when doing the query for "get_user_profile_stats".

Some things I have found:

- If the name in endpoints.method match in 2 different functions but differ 
in http_method, the generation fails too. Example, I want to user 
user/profile (name="user.profile" for both: get_profile and set_profile) 
with GET HTTP method to read it, and with POST HTTP method to modify it. It 
cannot be done.

- If the method of get_user_stats has HTTP method = "PUT" (whatever that is 
not POST or GET), it generates "correctly" the discovery file.

I am wondering if I am misunderstanding the meaning of "path" and "name" 
fields in the endpoints.method. Or maybe I misunderstood the whole 
endpoints thing, and it is 1 class per each set of "PUT", "GET", "POST", 
... methods (but in the tictactoe example there are 2 POST and 1 GET in the 
same class).

Any help is highly appreciated. Thanks in advance!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.