One of example with datastore (same happens for memcache or even between two methods calls of my code) - in between those two lines in QuoteRepository in simple call to Objectify().save(....)
Could it be with some threadsafe locking? Or is it possible that logging is actually slowing the app down? [s~app-id-replaced/1.400432737224109618].<stdout>: 2017-04-10 04:49:38 DEBUG QuoteRepository:72 - Saving quote 123 into database [s~app-id-replaced/1.400432737224109618].<stdout>: 2017-04-10 04:49:49 DEBUG QuoteRepository:74 - Saved quote 123 into database On Monday, 10 April 2017 09:05:40 UTC+12, Tomas wrote: > > Hello there, > > in last couple of months I've rewritten my java app running on app engine > and re-deployed with maven. Unfortunately from that time I'm experiencing > following issues: > > 1) extremely slow memcache queries - 700ms+ for simple getting object with > list of simple beans inside - very easy to serialise/deserialise but still > I see very slow memcache gets (even when the object is not in cache at all) > 2) slow datastore gets (again I got around 400ms at best, but could go > over 1s very often - I'm using objectify batch fetch (list of ids) but I > see similar for simple one object .load.now() > 3) pending requests - these drives me really crazy because I can't trace > why they spend so much time in the queue - I didn't touch any default > setting just threadsafe=true. My micro service is very simple > > - get request for list of data (that queries cache and then datastore) > - that usually gets me list of 20-30 beans from datastore which I sometime > need to update - I create the task with list of ids and post them in one > query to task queue and return the non-updated list meanwhile > > - task receive the list and queries external service by rest with the > whoile list as a body > - the external service iterates list in parallel and for each item finds > fresh data and post one by one back to my app engine service > > - on receiving the update I just save/update the item in the database > > So I get spike of items updates sometime, but nothing serious - let's say > 50 simple post updates per second (the body contains like 5 short strings) > - but I can see that single updated can take 10ms or 2seconds (spending 90% > of time in the pending queue). > > So I wonder if I'm missing some setting somewhere as I thought that if > there would be peak of pending request, app engine would spin another > instance to catch up - but I rarely see more that two instance while having > pending requests on both of them. Also it looks like that these simple > updates (which are not user facing request) are slowing down the main user > requests significantly (I can get 5-20 seconds pending on the main user > request when trying to get the list). > > Just a not, my cold start is around 4-5s and I can see the app is not > starting from cold with these spikes so it must be something else. > > This was working better with old version of my code but the new version is > much more maintainable si I'm not going back. I wonder if this could be > caused by sitting beside some heavy app which is slowing the engine for me > or I'm just doing something really stupid (very possible!). > > Many thanks for any advice! > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/21b6e437-70b5-4d13-aded-fbded2be638f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
