Re: [google-appengine] Trying to figure out why my app has unexpected overhead
Thanks for the reply — that’s the lag I’m seeing, correct. However, I log around that pickling step to get time stamps and it looks like the step only takes ~20ms to do: logging.debug(To get to index search took %s seconds%(time.time()-start_time)) search_results_pickle = jsonpickle.encode(results, unpicklable=False) logging.debug(To get to pickling took %s seconds%(time.time()-start_time)) start_time is from the beginning of the post(self) function and all time stamps are calculated from that. On a sample run, these are my time stamps: D 2014-07-22 14:29:24.732 To get to index search took 0.620759963989 seconds D 2014-07-22 14:29:24.749 To get to pickling took 0.637880086899 seconds So subtracting those two numbers gets something like ~17ms. The data I’m pickling is ~10kb. On Jul 23, 2014, at 1:47 AM, Vinny P vinny...@gmail.com wrote: On Tue, Jul 22, 2014 at 7:17 PM, Adrian adrianj.rand...@gmail.com wrote: I have been playing with all of the settings I can think of (including going from an F1 to F4 instance), but to no avail I am consistently getting a ~1 second lag on this POST request. For some reason, the process seems to just stop for arbitrary amounts of time, and it is severely degrading my app's performance. Note that this is not a situation where cold instances are being spun up, nor where there are many instances trying to access a given entity (which shouldn't matter anyway). To clarify: the gap you're referring to is the approximately 750 - 1700 space in the first AppStats screenshot, correct? It looks like the pickling operation is taking a long time. If you reduce the index search limit to 20, then to 10, then to 5 records, what impact does this have on performance? How large is the data you're pickling? - -Vinny P Technology Media Consultant Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to a topic in the Google Groups Google App Engine group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/RAMo1k6o4vE/unsubscribe. To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Strange requests in my log
Does anybody know what are these strange requests in my log? What 'http://gungamesz.com/' and 'http://strigkaomsk.ru' means in them? Tnx 2014-07-23 03:20:50.181 / 200 74ms 2kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) module=default version=not_ver 46.118.159.53 - - [22/Jul/2014:18:20:50 -0700] GET / HTTP/1.1 200 2941 http://gungamesz.com/; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) www.my_gae_app.com ms=74 cpu_ms=40 cpm_usd=0.000329 app_engine_release=1.9.7 trace_id=c4f6c3cf952affbcf665db101bdde952 instance=00c61b117c4af04d988479170a16be36f45bfc53 2014-07-23 03:20:50.179 / 200 71ms 2kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) module=default version=not_ver 46.118.159.53 - - [22/Jul/2014:18:20:50 -0700] GET / HTTP/1.1 200 2941 http://strigkaomsk.ru/; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) www.my_gae_app.com ms=71 cpu_ms=22 cpm_usd=0.000329 app_engine_release=1.9.7 trace_id=1711c891a04f3d17b6e03b3ac751def2 instance=00c61b117c4af04d988479170a16be36f45bfc53 2014-07-23 03:20:49.676 / 200 121ms 2kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) module=default version=not_ver 46.118.159.53 - - [22/Jul/2014:18:20:49 -0700] GET / HTTP/1.1 200 2941 http://strigkaomsk.ru/; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) www.my_gae_app.com ms=122 cpu_ms=37 cpm_usd=0.000329 app_engine_release=1.9.7 trace_id=a96948af014e3acf42b2e474ba94f8c0 instance=00c61b117c4af04d988479170a16be36f45bfc53 2014-07-23 03:20:49.674 / 200 126ms 2kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) module=default version=not_ver 46.118.159.53 - - [22/Jul/2014:18:20:49 -0700] GET / HTTP/1.1 200 2941 http://gungamesz.com/; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) www.my_gae_app.com ms=127 cpu_ms=26 cpm_usd=0.000329 app_engine_release=1.9.7 trace_id=8b1ddb57a3a72cde4eadde30e3abfe20 instance=00c61b117c4af04d988479170a16be36f45bfc53 2014-07-23 03:20:49.118 / 200 72ms 2kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) module=default version=not_ver 46.118.159.53 - - [22/Jul/2014:18:20:49 -0700] GET / HTTP/1.1 200 2941 http://strigkaomsk.ru/; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) www.my_gae_app.com ms=73 cpu_ms=20 cpm_usd=0.000329 app_engine_release=1.9.7 trace_id=6fc9601cad4d75116f41a7e6b5c5726b instance=00c61b117c4af04d988479170a16be36f45bfc53 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: ndb, get_by_id can not get an entity with a parent key only by id?
are you sure?
i tested in two apps. the first is using query. when i do a query, the read
ops increases about 300. the second is using referenceproperty as above,
after 5 times showing, there is no obviously increase.
the articles in the second app is much more than the first. so i don't
think it run a query.
在 2014年7月22日星期二UTC+8下午11时20分14秒，Jay写道：
When you use that api in db, it is doing a query underneath the covers.
On Monday, July 21, 2014 11:27:08 PM UTC-5, saintthor wrote:
in db, i set such a property in model article:
auth = db.ReferenceProperty( author, collection_name=AuthSet )
and use author.AuthSet to get all articles written by the certain author.
there is no query. how can i do the same thing in ndb?
在 2014年7月22日星期二UTC+8上午3时55分28秒，Jay写道：
Responding to your first post ... that is right. The parent key is part
of the entity's key. When you use get_by_id, this is just shorthand for
Key('Foo', 99).get().
You cannot get an entity by key unless you know the key, the whole key.
You say you don't want to query, but your use case ... give me all
articles for user A ... sounds like a query.
articles = Article.query(Article.author == user_key)...
If you want to do something like most recent n articles then you can
store those keys in a 'secondary index' of sorts and use ndb.get_multi()
On Monday, July 21, 2014 8:11:20 AM UTC-5, saintthor wrote:
to query may cost too many ops. i don't query.
在 2014年7月21日星期一UTC+8下午8时53分29秒，Kaan Soral写道：
Not associated with google, also not an expert on ndb
As far as I know key groups are related with consistency, in my
opinion, you shouldn't put article's and author's in the same group,
there
is a 1/s write speed limit on entity groups, if you put them in the same
entity group, you could hit this limit more easily
You don't even have to use referenceproperty's - I would just put the
author key in an article StringProperty named author and query for it,
if
I were you
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Strange requests in my log
On Wed, Jul 23, 2014 at 8:41 AM, Matija matija.jerko...@gmail.com wrote: Does anybody know what are these strange requests in my log? What ' http://gungamesz.com/' and 'http://strigkaomsk.ru' means in them? 46.118.159.53 - - [22/Jul/2014:18:20:50 -0700] GET / HTTP/1.1 200 2941 http://gungamesz.com/; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) www.my_gae_app.com ms=74 cpu_ms=40 cpm_usd=0.000329 app_engine_release=1.9.7 trace_id=c4f6c3cf952affbcf665db101bdde952 instance=00c61b117c4af04d988479170a16be36f45bfc53 Those addresses are referer ( http://en.wikipedia.org/wiki/HTTP_referer ) headers. In short, those sites linked to your GAE website, a user clicked on the link, and went from that website to your website. See http://blog.notdot.net/2011/06/Demystifying-the-App-Engine-request-logs if you want an explanation of how request logs are structured. - -Vinny P Technology Media Consultant Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Trying to figure out why my app has unexpected overhead
On Wed, Jul 23, 2014 at 6:00 AM, Adrian Randall adrianj.rand...@gmail.com wrote: Thanks for the reply — that’s the lag I’m seeing, correct. However, I log around that pickling step to get time stamps and it looks like the step only takes ~20ms to do: I wouldn't trust the log data so quickly if I were you. Looking at your SO post, the last log line (which is just before the return statement) says *To get to writing output HTML took 0.725300073624 seconds*, which means the request should be taking roughly that amount of time (slightly more than that to account for cleanup duties). But AppStats claims the request actually took 1867 ms, which is a huge difference. Your first hypothesis is accurate: it looks almost as if there was a pause in execution during runtime, but your application itself didn't even recognize/experience the time difference. I would still stand by my original recommendation: try rerunning the index search operation and limiting it to only 5, 10, then 20 results and see if it makes a difference. If you see no difference, drop it down to a limit of 1 and retry. The reason I'm recommending this is that I think you're hitting a throttle somewhere, and I want to find the point where the throttle kicks in. There are a few other possible causes (mostly extraordinary situations such as machine failure, or periodic instance startup/shutdown) but I think this is the most probable. On Wed, Jul 23, 2014 at 6:00 AM, Adrian Randall adrianj.rand...@gmail.com wrote: The data I’m pickling is ~10kb. Is that pre-pickle or post-pickle? - -Vinny P Technology Media Consultant Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: ndb, get_by_id can not get an entity with a parent key only by id?
I refer you to line 3865 of __init__ in the ext.db package in the sdk. This
is on a _ReverseReferenceProperty that gets setup.
If I have time tomorrow I will write a quick test to demonstrate.
On Wednesday, July 23, 2014 11:11:31 AM UTC-5, saintthor wrote:
are you sure?
i tested in two apps. the first is using query. when i do a query, the
read ops increases about 300. the second is using referenceproperty as
above, after 5 times showing, there is no obviously increase.
the articles in the second app is much more than the first. so i don't
think it run a query.
在 2014年7月22日星期二UTC+8下午11时20分14秒，Jay写道：
When you use that api in db, it is doing a query underneath the covers.
On Monday, July 21, 2014 11:27:08 PM UTC-5, saintthor wrote:
in db, i set such a property in model article:
auth = db.ReferenceProperty( author, collection_name=AuthSet )
and use author.AuthSet to get all articles written by the certain
author. there is no query. how can i do the same thing in ndb?
在 2014年7月22日星期二UTC+8上午3时55分28秒，Jay写道：
Responding to your first post ... that is right. The parent key is part
of the entity's key. When you use get_by_id, this is just shorthand for
Key('Foo', 99).get().
You cannot get an entity by key unless you know the key, the whole key.
You say you don't want to query, but your use case ... give me all
articles for user A ... sounds like a query.
articles = Article.query(Article.author == user_key)...
If you want to do something like most recent n articles then you can
store those keys in a 'secondary index' of sorts and use ndb.get_multi()
On Monday, July 21, 2014 8:11:20 AM UTC-5, saintthor wrote:
to query may cost too many ops. i don't query.
在 2014年7月21日星期一UTC+8下午8时53分29秒，Kaan Soral写道：
Not associated with google, also not an expert on ndb
As far as I know key groups are related with consistency, in my
opinion, you shouldn't put article's and author's in the same group,
there
is a 1/s write speed limit on entity groups, if you put them in the same
entity group, you could hit this limit more easily
You don't even have to use referenceproperty's - I would just put the
author key in an article StringProperty named author and query for it,
if
I were you
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.
