Re: [Zope-dev] TCP CLOSE_WAIT leaks

2006-03-29 Thread Alan Milligan
I managed to get a DeadlockDebugger trace on this thing, it made very
interesting reading:

Thread -1578087504 (GET
/VirtualHostBase/http/www.last-bastion.net:80/BastionLedger/ledgerlist/Mail_338253076):
  File /opt/zope2.8/lib/python/ZServer/PubCore/ZServerPublisher.py,
line 23, in __init__
response=response)
  File /opt/zope2.8/lib/python/ZPublisher/Publish.py, line 387, in
publish_module
environ, debug, request, response)
  File /opt/zope2.8/lib/python/ZPublisher/Publish.py, line 188, in
publish_module_standard
response = publish(request, module_name, after_list, debug=debug)
  File
/opt/zope2.8/lib/python/Products/PlacelessTranslationService/PatchStringIO.py,
line 51, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
  File /opt/zope2.8/lib/python/ZPublisher/Publish.py, line 114, in publish
request, bind=1)
  File /opt/zope2.8/lib/python/ZPublisher/mapply.py, line 88, in mapply
if debug is not None: return debug(object,args,context)
  File /opt/zope2.8/lib/python/ZPublisher/Publish.py, line 40, in
call_object
result=apply(object,args) # Type scr to step into published object.
  File /opt/zope2.8/lib/python/Shared/DC/Scripts/Bindings.py, line
311, in __call__
return self._bindAndExec(args, kw, None)
  File /opt/zope2.8/lib/python/Shared/DC/Scripts/Bindings.py, line
329, in _bindAndExec
def _bindAndExec(self, args, kw, caller_namespace):
  File /opt/zope2.8/lib/python/Shared/DC/Scripts/Bindings.py, line
348, in _bindAndExec
return self._exec(bound_data, args, kw)
  File /opt/zope2.8/lib/python/Products/CMFCore/FSPageTemplate.py,
line 195, in _exec
result = self.pt_render(extra_context=bound_names)
  File /opt/zope2.8/lib/python/Products/CMFCore/FSPageTemplate.py,
line 134, in pt_render
result = FSPageTemplate.inheritedAttribute('pt_render')(
  File /opt/zope2.8/lib/python/Products/PageTemplates/PageTemplate.py,
line 104, in pt_render
tal=not source, strictinsert=0)()
  File /opt/zope2.8/lib/python/TAL/TALInterpreter.py, line 202, in
__call__
def __call__(self):
  File /opt/zope2.8/lib/python/Products/PageTemplates/ZRPythonExpr.py,
line 47, in __call__
return eval(code, g, {})
  File Python expression putils.createTopLevelTabs(actions), line 1,
in expression
  File /opt/zope2.8/lib/python/Products/CMFPlone/PloneTool.py, line
701, in createTopLevelTabs
default=data['name'])
  File
/opt/zope2.8/lib/python/Products/CMFPlone/TranslationServiceTool.py,
line 35, in utranslate
return utranslate(*args, **kw)
  File /opt/zope2.8/lib/python/Products/CMFPlone/i18nl10n.py, line 53,
in utranslate
return service.utranslate(*args, **kw)
  File
/opt/zope2.8/lib/python/Products/PlacelessTranslationService/PlacelessTranslationService.py,
line 133, in utranslate
if not service: return default
  File /opt/zope2.8/lib/python/ZODB/Connection.py, line 704, in setstate
self._setstate(obj)
  File /opt/zope2.8/lib/python/ZODB/Connection.py, line 760, in _setstate
self._reader.setGhostState(obj, p)
  File /opt/zope2.8/lib/python/ZODB/serialize.py, line 495, in
setGhostState
state = self.getState(pickle)
  File /opt/zope2.8/lib/python/ZODB/serialize.py, line 488, in getState
return unpickler.load()
  File /opt/zope2.8/lib/python/ZODB/serialize.py, line 436, in
_persistent_load
return self._conn.get(oid)
  File /opt/zope2.8/lib/python/ZODB/Connection.py, line 216, in get
return obj
  File /opt/zope2.8/lib/python/ZEO/ClientStorage.py, line 746, in load
return self.loadEx(oid, version)[:2]
  File /opt/zope2.8/lib/python/ZEO/ClientStorage.py, line 781, in loadEx
return data, tid, ver

*every* thread was block-waiting on zeo (from a wide range of different
Zope/Plone types)!  It looks to me like Apache has timed out, clearing
down it's end, Zope however is still having to wait for zeo which is
completely borked.

I've consequently ditched zeo and everything is again well-behaved.

Alan

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] TCP CLOSE_WAIT leaks

2006-03-29 Thread Paul Winkler
On Thu, Mar 30, 2006 at 02:32:58AM +1000, Alan Milligan wrote:
 I managed to get a DeadlockDebugger trace on this thing, it made very
 interesting reading:
(snip)
   File /opt/zope2.8/lib/python/ZEO/ClientStorage.py, line 781, in loadEx
 return data, tid, ver
 
 *every* thread was block-waiting on zeo (from a wide range of different
 Zope/Plone types)!  It looks to me like Apache has timed out, clearing
 down it's end, Zope however is still having to wait for zeo which is
 completely borked.
 
 I've consequently ditched zeo and everything is again well-behaved.

Is your zeo server on a separate box? Is there a firewall between them?

The *only* time I've ever had problems like that was in the following
scenario:

* firewall between zope and zeo

* minimal traffic at times (it was a secondary system, most of its
  usage was when our primary data center was down for maintenance)

* firewall was of an evil type that tears down unused connections
  without either end being able to know it happened 

In this scenario, after suitably long period of no traffic between Zope
and Zeo, the firewall would disconnect them but they would still think
they were connected, and we would get a problem like yours.

Dieter Maurer observed the same thing and gave me the hint that this
might be the problem.  Implementing his suggested keepalive product 
was less trouble than arguing with the firewall administrators.
http://aspn.activestate.com/ASPN/Mail/Message/zope-list/2918584

I used something very close to that, I believe I just saved it as 
Products/ZeoKeepalive/__init__.py.
(I've changed jobs so I'm going by memory.)

-- 

Paul Winkler
http://www.slinkp.com
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )