[Zope-dev] Re: [Zope] highly available Zope thread; our hanging problem
On Tue, 6 Jun 2000 15:19:29 +0200 Marcus Collins <[EMAIL PROTECTED]> wrote: > Hi, > > I'd like to comment on this, and summarise some references below. Much of > this discussion took place on the zope-dev list (see references), so I'm > cc'ing the zope-dev list. You might also wish to add to the Wiki: > http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/. OK, that's a good suggestion! > > > -Original Message- > > From: Brian Takashi Hooper [mailto:[EMAIL PROTECTED]] > > Sent: 06 June 2000 12:11 > > To: [EMAIL PROTECTED] > > Subject: [Zope] highly available Zope thread; our hanging problem > > > > Hi all - > > > > I was looking at the discussion from April that was posted on the > > HighlyAvailableZope Wiki about problems with Zope hanging; we had a > > similar situation here at Digital Garage which seemed to be alleviated > > by changing the zombie_timeout to be really short (like, 1 minute). > > Before changing the zombie_timeout, the server would periodically hang > > and not give any responses to requests, sometimes recovering after a > > short time. > > Some questions at this point: > 1. Were you running with multiple threads, and if so, how many? Yes; Zope is set to run with 16 threads (-t 16), and we've increased the pool_size parameter in ZODB/DB.py to 16 also (guess this is all right... :-P ) > > 2. If you were using multiple threads, would *all* the threads periodically > hang, or was the hanging isolated to a single thread at a time? All the threads hang. One interesting thing is, we looked at vmstat and whenever the system is having trouble, the number of system calls drops dramatically, when the server is doing well it's normally up in the 1000s, but when it's in trouble there are like 20-30 system calls per second, and they're all either lwp_* or poll s. > > 3. Could you possibly comment on the operating system used? Solaris, 2.6, on netras. Our Zope is still v. 2.1.4. > > 4. Which zombie_timeout did you twiddle -- the one in the zhttp_channel in > ZServer.py, or that in http_channel in medusa/http_server.py? The one in zhttp_channel. As far as I can tell, since zhttp_channels are actually used instead of http_channels, the number in zhttp_channel is the one that matters. The kill_zombies method, and the code that calls it, is inherited from the medusa code... kill_zombies looks at the timeout value of all the channels in the select list, and since all of those instances happen to be zhttp_channels in the case of Zope, they all use the zhttp_channel timeout. > > > At this point, I don't have anything more than just an empirical > > observation - changing this parameter seemed to help our server. Has > > anyone else noticed anything similar, or can explain this observation? > > Concerning the zombie_timeout suggestion, here are some references when I > posed the question of whether reducing the value would be beneficial: > > Amos Lattier wrote in > http://lists.zope.org/pipermail/zope-dev/2000-April/004194.html: > > The ZServer zombie stuff is to get rid of zombie client > > connections, not zombie publishing threads. These are quite > > different beasts. > > Michel Pelletier wrote in > http://lists.zope.org/pipermail/zope-dev/2000-April/004229.html: > > What the Zombie timeout means is that after a publishing thread gets > > done answering a request, the socket may not go away. This many for a a > > number of reasons, the client 'hung' and is not 'putting down the phone > > after the converstation is over' (so to speak) or network troubles may > > prevent the connection from closing properly. This means that there is > > a 'zombie' connection laying around. This zombie will probably end up > > going away on its own, but if not, ZServer will kill it after a period > > of time. > > > > The only reasorce laying around during the life of a Zombie is an tiny > > little unused open socket, the Mack truck of a Zope thread that served > > the request for the zombie socket does not 'hang' for that entire period > > of time, but goes on after it has completed the request to serve other > > requests. > > > > Amos is correct in that these problems are almost always at the > > Application level, and not at the ZServer level. The fact that Pavlos > > can prevent hanging by inserting a print statement in the asyncore loop[*] > > is suspicious, but we do not have enough information yet to point > > fingers anywhere. > > [* references http://lists.zope.org/pipermail/zope/2000-April/023697.html] Yeah, I saw this... like I said, I
Re: [Zope-dev] ZPatterns alpha 3 released (was re: ZPatterns0.4.0a2 bug fixes)
On Tue, 20 Jun 2000 09:46:34 -0500
"Phillip J. Eby" <[EMAIL PROTECTED]> wrote:
[snip]
> >Thank you, Phillip. I've spent all the evening hunting these bugs.
> >By the way, how do you debug? Is there better way than inserting things
> >like
> >
> > print "objectChanging %s, has status=%d" % (
> > self, self.__dict__.has_key('_v_status_')
> > )
> >
>
> I haven't found one yet. I'm actually using zLOG.LOG calls, however, to a
> logfile, so I can look back at things easily (and because I didn't realize
> you could use print until you posted this... :) ).
I recommend pdb; there's a How-To klm wrote on the zope.org site called
'Debugging Zope Python Code', that recommends inserting something like
import pdb
pdb.set_trace()
at the part of the code that you want to have it drop down into the
debugger at, and then you can step through and find out the value of
various expressions as you're running things, call functions, etc.
Mr. Creosote is also my favorite - check out the MrCreosote subdirectory
of pcgi/ in your Zope installation; the nice thing about Mr. Creosote is
you can have it send debugging info to another machine (UDP packets)
--Brian Hooper
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] customizing Zope HTTP logs
Hi there - I want to customize the format of Zope's HTTP logs for a certain project, so, reading the code I can see that the log() method of medusa's http_request is used for this. In order to customize the log format, I can of course just edit http_server.py to change the http_request's log() method to do what I want, or alternately have code that defines a new log() method and then just changes medusa.http_server.http_request on startup (like, from z2.py). However, I was thinking that wouldn't it be nicer to be able to subclass http_request, say in ZServer/HTTPServer.py and customize the behavior of http_request that way...? It looks like the only things that would be affected would be a couple lines in medusa's http_channel class, where http_request is instantiated: ... r = http_request (self, request, None, None, None, join_h eaders(lines[1:])) r.error(400) return header = join_headers (lines[1:]) r = http_request (self, request, command, uri, version, header) ... Do you think it would be reasonable to change this to something similar to the way zhttp_channel/zhttp_server works, specifying request_class = zhttp_request and then changing the above request lines to call self.request_class instead? Then we can do (in HTTPServer.py): class zhttp_channel: ... request_class = zhttp_request class zhttp_request: def log(...) ... Which seems nicer to me. I guess this is really a request for a medusa change, is this a more appropriate topic for the medusa list? Does anyone else like this idea? --Brian Hooper Digital Garage, Inc. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] HTTP user agent?
Hi Zopistas,
For anyone that might know:
Is there a particular reason that the User-Agent header is not part of
the request data that ZServer sends in the environment to ZPublisher?
It looks like the user agent can also be provided to Zope's request
object just by adding a line for 'user-agent' to
ZServer.HTTPServer.header2env:
header2env={'content-length': 'CONTENT_LENGTH',
'content-type' : 'CONTENT_TYPE',
'connection': 'CONNECTION_TYPE',
'user-agent': 'HTTP_USER_AGENT'
}
Is there a particular reason this is not done?
--Brian Hooper
Digital Garage, Inc.
Tokyo, Japan
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] patches to make ZServer's http_request subclassable
Hi all - I made a couple of patches to ZServer (medusa, actually) to make medusa'a http_request objects subclassable... the motivation is to make customizing medusa / ZServer's logging easier to do just overriding http_request's log() method in a subclass. Patches are attached to this message, suggestions welcome! --Brian Hooper Tokyo, Japan medusa-2.1.6.patch medusa-2.2.0.patch
[Zope-dev] bug in ZSQL methods caching; new explicit 'nocache' feature for SQL Methods
Hi Zopistas,
I've been playing a bit with adding a feature to Z SQL Methods to allow
an explicit 'nocache' parameter to be sent to an SQL Method, so that a
particular query can be run pre-clearing the cache for that particular
query, while leaving the rest of the cache intact. So you for example
serve content from your SQL database with something like
in your display templates, and then say if you have an admin screen
where you edit that content in the DB, you can do
after editing, and that will clear the cache entry only for
content_id=1. I think there might be a lot of reasons this could be
good.
Anyways, when I was looking over Shared.DC.ZRDB.DA.DA._cached_result()
to find out how the caching worked, I noticed something that seems a
little funny; it looks like the cache clearing is done in _cached_result
by sorting through all the time values in tcache, which is an
_integer-keyed_ IOBTree which stores the times of each of the queries.
However, what happens when the same query is made several times in one
second? Only one time-query pair is stored in the tcache for each
unique integer time, so if in the same second I make 20 queries using
the same SQL method, then later when I try to clean the cache, I will
only be able to delete one of these values...
I added a little debugging output to _cached_result and was able to, by
making a lot of queries to the method in the same page, make the size of
the cache go above the supposed maximum size max_cache.
Admittedly this kind of usage is rare, but it probably should be
fixed... it would probably be improved by making tcache a regular Python
dict storing floats as the keys.
Finally, here's my code to add 'nocache' to SQLMethods; it would
definitely be nicer if it was integrated into DA.py instead, since that
would save having to generate the query twice, for example... in the
meantime, though here it is (ZSQLMethods/SQL.py, SQL.__call__).
def __call__(self, *args, **kw):
""" Add an option 'nocache' to avoid using the query cache for
a particular query. This way, SQL queries can be cached in general
but an application can decide if it wants to explicitly retrieve a
new value...
"""
scall = SQL.inheritedAttribute('__call__')
sargs = (self,)+args
if kw.has_key('nocache') and kw['nocache'] == 1:
# we only care about clearing the cache if there _is_ a cache
if hasattr(self,'_v_cache'):
cache=self._v_cache
cache, tcache = cache
srckw = kw.copy()
srckw['src__'] = 1
query = (apply(scall, sargs, srckw), self.max_rows_)
# clear the cache and tcache, if necessary
if cache.has_key(query):
t = cache[query][0]
t = int(t)
del cache[query]
if tcache.has_key(t) and tcache[t] == query:
del tcache[t]
return apply(scall, sargs, kw)
Do other people like this idea?
--Brian Hooper
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
