Re: [Zope] ZODB Key error in root Zope database
FYI,
I was able to successfully remove the cross storage reference in the
root of my main Zope database by explicitly setting the missing database
connection to a hacked DummyConnection instance. I probably could have
dug deeper to create a real connection, but this worked.
from ZODB import FileStorage, DB
import transaction
storage = FileStorage.FileStorage('Data.fs')
app = DB(storage).open().root()['Application']
base_conn = app._p_jar
class DummyConnection:
""" implements enough of a connection to allow for
removing of the cross database reference """
def __init__(self):
self.connections = []
self._cache = {}
def get(self, key):
return None
def open(self, **kw):
return DummyConnection()
base_conn._db.databases['site1'] = DummyConnection()
app._objects = tuple([i for i in app._objects if i['id'] != 'badID'])
transaction.get().commit()
It's quite the hack, but allowed us to get the database back.
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
--
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
___
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] ZODB Key error in root Zope database
We've got a bit of a situation...
We run Zope 2.11 with a main ZODB and a few dozen separate file storages
for sub-sites.
Occasionally, we've encountered the problem where a cut-and-paste occurs
between storages and upon a pack results in a key error. Normally, we
avoid this because the FileStorages represent individual sites.
Last night something happened where a script was cut-and-pasted from one
of the sub-storages into the main, root-level, database.
Upon a restart we encountered a key error:
File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
line 361, in get_connection
new_con = self._db.databases[database_name].open(
KeyError: 'site1'
Normally, we've encountered these errors when a database is packed and
the solution is to revert to a pre-pack database and remove or fix the
offending object.
In this case, we can't even start Zope or mount the storage in a script
to remove the offending items.
Before I go on, I found that ZODB 3.9 provides options to disallow
cross-database references and we'll work to get to this version during
our next upgrade cycle.
In the mean time we've reverted to a previous backup prior to the
offending object being placed in the root database. This resulted in
about 200 MB of data loss. I would like a way to mount the corrupted
database and attempt to determine exactly what was lost.
What we've tried so far has gone like this:
from ZODB import FileStorage, DB
from OFS.Folder import Folder
storage = FileStorage.FileStorage('BadData.fs')
db = DB(storage)
conn = db.open()
root = conn.root()
app = root['Application']
app.manage_deleteObjects(['OffendingObject'])
# This throws the same exception so we tried working around the
# mechanics of manage_deleteObjects with this.
app._objects = tuple([i for i in app._objects if i['id'] !=
'OffendingObject'])
Same exception. Can't access the database. We've tried using _setOb to
replace the object.
Any suggestions? It seems as though it needs to be able to access
_objects before it mounts the mounted databases.
Thanks,
--
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
___
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Performance with Multiple Mount Points
Our goal with this configuration was to limit the ability of a single site to lock the entire database during a large transaction. Are ZEO Server transaction locks one per storage or one per server? If the locks are per storage rather than per server I believe we could accomplish our goal with less servers, in which case is there an upper limit on the number of storages a single server can reasonably handle? Thanks, -Brian Andreas Jung wrote: > Brian Brinegar wrote: >> We've recently moved to a Zope configuration with approximately 30 >> separate databases mounted at mount points within a main database. > >> Previously, we ran a single database which was approaching 300GB in >> size. We chose to split the database to reduce Data.fs size. We now run >> 30 separate ZEO Servers (on a single machine). > >> However, after switching to this configuration we've noticed some >> performance issues. > >> Initially, after a client is restarted and builds out it's caches the >> site runs very quickly. However as the memory usage grows the >> performance degrades. If the client gets into swap the client >> practically dies. > >> We had 10,000 object caches on each of the databases with 2 threads on >> the client. 30 x 10,000 x 2 = 600,000 objects in memory. We've since >> reduced the cache sizes to 1,000 or 30 x 1,000 x 2 = 60,000 objects. >> This seemed to extend the life of a client between restarts. > > No idea what's running onside your system and inside your Zope - 1000 > objects seems way to small for almost any case. Also 30 ZEO servers > seems kind of extreme. We are running similar a big applications and we > are running with five or six ZEO servers - not 30. > > > -aj > >> Generally, our client machines will hover around 97%-99% memory usage >> and 90%-100% CPU (on a 2 CPU machine). > >> We still experience periodic performance problems and are looking for >> any input that might help us address them. > >> Thanks for your input, > > > -- Brian Brinegar Web Services Coordinator Engineering Computer Network ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope Performance with Multiple Mount Points
We've recently moved to a Zope configuration with approximately 30 separate databases mounted at mount points within a main database. Previously, we ran a single database which was approaching 300GB in size. We chose to split the database to reduce Data.fs size. We now run 30 separate ZEO Servers (on a single machine). However, after switching to this configuration we've noticed some performance issues. Initially, after a client is restarted and builds out it's caches the site runs very quickly. However as the memory usage grows the performance degrades. If the client gets into swap the client practically dies. We had 10,000 object caches on each of the databases with 2 threads on the client. 30 x 10,000 x 2 = 600,000 objects in memory. We've since reduced the cache sizes to 1,000 or 30 x 1,000 x 2 = 60,000 objects. This seemed to extend the life of a client between restarts. Generally, our client machines will hover around 97%-99% memory usage and 90%-100% CPU (on a 2 CPU machine). We still experience periodic performance problems and are looking for any input that might help us address them. Thanks for your input, -- Brian Brinegar Web Services Coordinator Engineering Computer Network ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Dieter Mauer's Reference Product
Dieter,
You've just made my week! I'm glad that my failure to understand how all
of this works has shed some light on the problem.
Thank you,
Brian
Dieter Maurer wrote:
> Dieter Maurer wrote at 2010-3-16 17:42 +0100:
>> Brian Brinegar wrote at 2010-3-16 10:12 -0400:
>>> Our university relies heavily on a Zope product based on Dieter Maurer's
>>> "Reference" product. Recently, we upgraded from Zope 2.9.6 to Zope
>>> 2.11.x and found some changes in behavior.
>>>
>>> In short the Reference product creates a Symlink like pointer in the
>>> Zope hierarchy. Dieter's product can be found on his site at:
>>>
>>> http://www.dieter.handshake.de/pyprojects/zope/index.html#bct_sec_5.9
>>>
>>> First, the security machinery now prevents access to attributes of
>>> References through page template path notation. For example, the
>>> following fails:
>>>
>>> tal:content="container/MyReference/property_name"
>>>
>>> Traceback:
>>> ...
>>> * Module zope.tales.expressions, line 217, in __call__
>>> * Module Products.PageTemplates.Expressions, line 133, in _eval
>>> * Module zope.tales.expressions, line 124, in _eval
>>> * Module Products.PageTemplates.Expressions, line 82, in
>>> boboAwareZopeTraverse
>>> * Module OFS.Traversable, line 301, in restrictedTraverse
>>> * Module OFS.Traversable, line 232, in unrestrictedTraverse
>>>__traceback_info__: ([], 'property_name')
>>>
>>> Unauthorized: You are not allowed to access 'property_name' in this context
>> This is a bug/weakness in Zope which affects the "traversal" methods
>> (used for TALES path expressions):
>>
>> When a value is retrieved during traversal via
>> "__bobo_traverse__" which does not have its own
>> security declarations (impossible for a simple datatype),
>> then the traversal insists that it is the same object
>> (verified by object identity) than the object retrieved
>> via "getattr" ("guarded_getattr", to be precise).
>>
>> This drastically restricts the access to simple values
>> via traversal if "__bobo_traverse__" is defined.
>>
>>
>> "Reference" grew a "__bobo_traverse__" method to work
>> around a (apparent) Five bug as delivered with Zope 2.9.
>> Maybe, the "__bobo_traverse__" method is not longer necessary
>> for Zope 2.11. Try to comment it out.
>>
>>> ...
>>> Second, through path notation or URL traversal, References under the
>>> previous version of Zope would default to using methods / objects within
>>> the target before falling back to acquisition. Under Zope 2.11 acquired
>>> methods/objects take priority (only when traversed).
>>>
>>> For example, assuming there is an index_html in the root as well as in
>>> the target, and using the following code:
>>>
>>> tal:content="container/MyReference/index_html/absolute_url_path"
>>>
>>> Zope 2.11 yields the path to the acquired index_html:
>>>
>>> /index_html
>>>
>>> Zope 2.9.6 yields the path to the index_html in the target:
>>>
>>> /Path/To/Target/index_html
>>>
>>> Again, through python, both yield the second, desired output.
>> This sounds strange -- almost unbelievable.
>>
>> I will look into it within the next few days and report back.
>
>
> Thanks to your problem report, I have much better understood
> the problem reported by J Cameron Cooper for Zope 2.9.
>
> The problem has not been a Five problem. Instead, it was
> caused by a confusion whether the traversal methods
> should be resolved with respect to the reference or its target.
> The primary implementation resolved them with respect to the reference
> and then could not traverse with respect to the target -- J Cameron's problem.
>
> The "__bobo_traverse__" method partially fixed this again using
> an explicit proxy (which takes into account both reference and target)
> but triggered the security weakness in Zope's traversal for
> simple values.
> A bug in its implementation (a missing "aq_base(...)")
> caused the wrong acquisition context.
>
>
> After the improved understanding, I can handle traversal
> methods without a need for "__bobo_traverse__".
> This fixes both of the problems you have observed.
>
> I will write some tests and then publish "References" as
> "Products.References" on PyPI in the next days.
>
>
> Thank you for your problem report!
>
>
>
>
> --
> Dieter
>
--
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
___
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Dieter Mauer's Reference Product
Our university relies heavily on a Zope product based on Dieter Maurer's "Reference" product. Recently, we upgraded from Zope 2.9.6 to Zope 2.11.x and found some changes in behavior. In short the Reference product creates a Symlink like pointer in the Zope hierarchy. Dieter's product can be found on his site at: http://www.dieter.handshake.de/pyprojects/zope/index.html#bct_sec_5.9 First, the security machinery now prevents access to attributes of References through page template path notation. For example, the following fails: tal:content="container/MyReference/property_name" Traceback: ... * Module zope.tales.expressions, line 217, in __call__ * Module Products.PageTemplates.Expressions, line 133, in _eval * Module zope.tales.expressions, line 124, in _eval * Module Products.PageTemplates.Expressions, line 82, in boboAwareZopeTraverse * Module OFS.Traversable, line 301, in restrictedTraverse * Module OFS.Traversable, line 232, in unrestrictedTraverse __traceback_info__: ([], 'property_name') Unauthorized: You are not allowed to access 'property_name' in this context Interestingly, the same access via dot notation works: tal:content="python:container.MyReference.property_name There were substantial changes to Traversable.py between versions which seem to cause the problem. Any suggestion on how to fix this would be greatly appreciated. Second, through path notation or URL traversal, References under the previous version of Zope would default to using methods / objects within the target before falling back to acquisition. Under Zope 2.11 acquired methods/objects take priority (only when traversed). For example, assuming there is an index_html in the root as well as in the target, and using the following code: tal:content="container/MyReference/index_html/absolute_url_path" Zope 2.11 yields the path to the acquired index_html: /index_html Zope 2.9.6 yields the path to the index_html in the target: /Path/To/Target/index_html Again, through python, both yield the second, desired output. I realize this is an obscure product, and the changes seem to have to do with the Five implementation. At this point we are looking for options to restore the desired functionality which does not require perpetually running Zope 2.9.x One option may be to change everything to dot notation, however I would at least like to understand why this change occurred. Thanks for reading, -- Brian Brinegar Web Services Coordinator Engineering Computer Network ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Maintain Last Modified during Copy
I'm looking for a way to copy content from one database to another without updating modification times. I believe the modification times are tied to transaction IDs which may mean this isn't possible. Moving objects does not update their modification times, but, it doesn't seem to be possible to move objects from one storage to another. One option that works is to copy the entire database, remove everything I don't want, and then pack it. But this isn't ideal. Any suggestions? -- Brian Brinegar Web Services Coordinator Engineering Computer Network Purdue University ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Error in Copy/Import during Migration
I'm fairly well versed on the inner workings of Zope, but yesterday I've
run into a problem that I can grasp.
First, some background on what I'm doing.
I've got a 200GB Data.fs file which I'm splitting up into multiple
storages. The way I'm doing this is to create a zope instance which
mounts a new storage and the old storage and then a python script
connects to the application and uses a modified version of manage_clone
(which maintains ownership) to copy content into the new storage.
This works correctly for most things, but yesterday I tried to move a
folder with about 3.1GB of stuff in it.
On a side note, I ran into an issue with running out of /tmp space
during the clone. The _getCopy method of CopySourse exports an object to
a tempfile and then imports it to create the copy. When copying
something larger than available tmp space zope crashes. I worked around
this by explicitly setting tempfile.tempdir to somewhere with plenty of
free space. There may be an environment variable to do this as well.
Just an FYI.
Anyway, here's what I see. The copy starts, a tempfile is created (on
the client) and grows to the appropriate size (3.1GB). Then, a
Data.fs.temp appears on the new server and grows to the appropriate
size. However, when my script tries to commit the transaction I get an
EOF error:
Traceback (most recent call last):
File "./migrate_site.py", line 45, in ?
transaction.get().commit()
File
"/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
line 325, in commit
self._commitResources()
File
"/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
line 427, in _commitResources
rm.tpc_vote(self)
File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
line 744, in tpc_vote
s = vote(transaction)
File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py",
line 1004, in tpc_vote
self._server.vote(id(txn))
File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ServerStub.py",
line 262, in vote
return self.rpc.call('vote', trans_id)
File
"/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/zrpc/connection.py", line
699, in call
raise inst # error raised by server
EOFError
I don't seem to have any space issues on any devices, and the process
works for smaller site. Any input would be greatly appreciated.
Thanks,
--
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
___
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Connecting New Zope to old ZEO server
As part of a migration plan we would like to have a Zope client running Zope 2.11.4 connect to a ZEO server running Zope 2.9.6 ZODB 3.6.2 When trying to connect we get an error the following error: 2009-12-10T10:09:48 INFO ZEO.ClientStorage (6145) Testing connection -- 2009-12-10T10:09:48 ERROR ZEO.zrpc.Connection(C) (192.168.1.51:17900) bad handshake 'Z303' Is it possible to have a new Zope connect to an old ZODB? Thanks, -- Brian Brinegar Web Services Coordinator Engineering Computer Network ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] method call pre-attribute access
I've got product where it would be very beneficial to me if I could have
a method called prior to any attribute access. For example:
thing.foo
thing.getProperty('foo')
Either of these should trigger a call to a method which could allow or
deny access to the attribute.
Any thoughts?
--
Brian Brinegar
Web Systems Developer
Engineering Computer Network
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Client Disconnected
Occasionally a client in our ZEO setup will lose it's connection to the ZEO Server. I suspect some user is trying to commit something that is taking excessively long, perhaps uploading a very large file. Anyway, when a client is disconnected should it not try to reconnect? The min-disconnect-poll and max-disconnect-poll are set to the default. It seems that the clients NEVER reconnect until Zope is restarted. Any suggestions on how to automate this? -- Brian Brinegar Web Systems Developer Engineering Computer Network ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: debugging a wayward zope process
http://www.zopelabs.com/cookbook/1073504990
In addition to the information on this page I have found that you can
also run the following command at a (gdb) prompt to get a list of URLs
being processed by the misbehaving thread:
call PyRun_SimpleString("import string,sys,os,ZServer;
sys.stdout=open('/tmp/urls','w',0); print string.join(map(lambda
a:str(getattr(a,'env',{}).get('PATH_INFO','')),
sys.modules['asyncore'].socket_map.values()), '\\n')")
It's doing a bunch of stuff to try to format the file nicely into a file
/tmp/urls, it can probably be simplified.
There may also be an altogether better way of approaching it. Just
thought I'd throw this in. And looking at it, I'm not sure that you need
to import ZServer or os, I came up with this and didn't really refine it
once it worked.
--
Brian Brinegar
Web Systems Developer
Engineering Computer Network
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Integrating METAL into Zope Product
I've got a Zope product which will render itself using standard_html_header and standard_html_footer if they exist. I would like this same product to look for a specific Page Template (ptMaster) and render itself using a macro and filling a slot with it's contents. Any pointers/examples on integrating METAL into a product? Thanks, -- Brian Brinegar Web Systems Developer Engineering Computer Network ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope 2.9.5 -> 2.9.6 DateTime Change
Today I started testing Zope 2.9.6. After the upgrade I noticed that
some calendar data generated from Oracle was wrong. After some
investigation I found that Zope 2.9.6 seems to assign a timezone of
GMT+0 when none is specified rather than using the server's timezone.
For example, the following SQL query to an oracle database:
select TO_DATE('12-13-2006 01:00','MM-DD- HH:MI') from dual
Under 2.9.5 returns:
2006/12/13 01:00:00 US/Eastern
Under 2.9.6 returns:
2006/12/13 01:00:00 GMT+0
There was clearly no time zone defined in the query, and it's the same
Oracle server.
Is this an intended change? We have hundreds of developers working
independently within our system so it's not possible to adapt each
application to account for this change.
Any advice on moving to Zope 2.9.6? In the mean time I my try to back
port specific bug fixes into 2.9.5.
Thank you,
--
Brian Brinegar
Web Systems Developer
Engineering Computer Network
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope 2.6.4 -> 2.10.0 Migration (PageTemplates, Unicode, etc.)
So... We decided to move from Zope 2.6.4 to something newer (currently targetting Zope 2.10.0). I immediately found that none of our PageTemplates work we get an error on PageTemplates/Expressions.py line 92 that call_with_ns is not defined. ob = call_with_ns(ob, ns, 2) Changing it as follows fixes the problem, I think this is a bug. ob = ZRPythonExpr.call_with_ns(ob, ns, 2) Next... we have lots of PageTemplates which raise the following exception: Module zope.pagetemplate.pagetemplate, line 118, in pt_render Module StringIO, line 271, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 93: ordinal not in range(128) I suspect there are non-ascii characters in the file, but I'm not sure how to fix this, or how I would know what to replace it with. Is there a way to turn off Unicode support? Any other issues I might run into during this upgrade? Should we move to an older version (2.8? 2.9?) first and then onto 2.10? Thanks, -- Brian Brinegar Web Systems Developer Engineering Computer Network ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: ZEO with Multiple Storages
> > One ZEO server can deal with multiple storages just fine. > > Transactions involving multiple storages commit just fine. > > Caveat: You buy nothing by having a single ZEO process serve several > ZODBs when you're trying to decrease commit times. You should run a > ZEO process per ZODB you're serving out, that way the writes can be > segregated and parallelized. That also makes it easier to move > processes/databases around disks/hosts to spread I/O load. Just to clarify, you are saying that a single ZEO server would not be able to deal with multiple storages simultaneously and that a transaction in one storage would block transactions in all other storages. However, if we were to run multiple ZEO processes this would not be the case? -Brian ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] ZEO with Multiple Storages
We currently have a ZEO environment with FileStorage (data.fs) over 100 gigs, which continues to grow. Currently we are approaching a major hardware and software upgrade. All new hardware and moving from Zope 2.6 to Zope 2.9. Our server hosts sites for several different schools and departments within our college, what I have considered doing is trying to split up the data.fs into multiple smaller storages, possibly one per school. I suspect if this is possible it could potentially have several benefits. The first being that we would not have to deal with 100 gig files which are very difficult to move around backup/restore, etc. Secondly, we often have a very large number of users updating content at a given time and will run into slowdowns where a transaction is blocked while another commits. I hope that multiple storages could allow for a transaction to commit to one storage at the same time as another, though I am not sure. Along this line can a single transaction commit to multiple storages? First off, does this make any sense? Is it worth pursuing? Secondly, I assume that where an object is stored does not limit what other objects it can access in a transaction (e.g. object A could add a property to object B if they were in different storages) because all changes would occur in a client. (this ties into the single transaction committing to multiple storages) If this does makes sense is there any documentation, recommendations, best practices for how to set something like this up? Lastly, any thoughts on how we could split up our existing data.fs file? One thought I had was to attempt to import/export data. Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network Purdue University ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Preventing Zope Spinning?
Jonathan, We have debated this, however the time necessary to revalidate the cache can be several minutes (maybe more) at startup for a 2 gig cache. If only one client was going down at a time this would not be a problem, however in the event that someone clicks reload until all of the clients are dead it would cause complete down time while the clients startup. We made the decision that having the server be slow was better than having it be completely unavailable for an extended period of time. I would like to prevent the clients from ever needing to be restarted. Brian Brinegar Web Systems Developer Engineering Computer Network Jonathan wrote: > How about setting up persistent caches for your zeo clients, so that > when the zeo server goes down you save some cache-reloading time when it > restarts? > > > Jonathan > > > - Original Message - From: "Brian Brinegar" > <[EMAIL PROTECTED]> > To: > Cc: "David S Carmichael" <[EMAIL PROTECTED]>; "Andrew T Sydelko" > <[EMAIL PROTECTED]>; "Christopher N Deckard" <[EMAIL PROTECTED]> > Sent: Thursday, May 25, 2006 10:49 AM > Subject: [Zope] Preventing Zope Spinning? > > >> The majority of the Purdue University Engineering web presence is >> provided via a cluster running ZEO. We offer hosting for every school, >> department, faculty, staff, and student in the College. Because of this >> we have a large number of content maintainers/developers on our system. >> We are running into problems with users writing bad code which spins or >> uploading huge files which seems to tie up the database for long periods >> of time. >> >> We end up in a situation where something will spin a client and a user >> will repeatedly resubmit the request until all of our zeo clients are >> spinning. Our clients are eventually killed, usually manually, and >> quickly come back up. We drop the zeo cache on a restart to improve >> startup speed, so when the clients do come back up we have 100% cache >> misses and the zeo server gets pounded resulting in slow performance >> until the client caches repopulate. >> >> Occasionally we can track down the offending URL and correct the >> problem, sometimes we cannot. >> >> Perhaps these issues will be addressed in future versions of Zope, we >> are currently running Zope 2.6.4 >> >> What I would like is some sort of timeout for requests, however I do not >> want to punish users with slow connections. Perhaps a way to kill off a >> specific request that is consuming excessive resources, without killing >> the entire client. >> >> Below is some information on our setup: >> >> 1 zeo server (Solaris) >> - 82 gig datafs >> - transaction time out of 120 seconds >> >> 2 load balanced zeoclients (Linux) >> - 2 gig zeo cache >> - Database Cache 3 objects >> - 4 threads >> >> 2 failover apaches (Linux) >> - using pydirector for load balancing >> >> We are receiving appoximately 1 million hits per day, which from what >> I've read is not all that much. We probably have a higher number of DB >> writes than usual because of the number of developers/maintainers. Can >> anyone make suggestions for providing a more stable environment? >> >> Thank you, >> Brian Brinegar >> Web Systems Developer >> Engineering Computer Network >> ___ >> Zope maillist - Zope@zope.org >> http://mail.zope.org/mailman/listinfo/zope >> ** No cross posts or HTML encoding! ** >> (Related lists - >> http://mail.zope.org/mailman/listinfo/zope-announce >> http://mail.zope.org/mailman/listinfo/zope-dev ) >> ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Preventing Zope Spinning?
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time. We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate. Occasionally we can track down the offending URL and correct the problem, sometimes we cannot. Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4 What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client. Below is some information on our setup: 1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds 2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 3 objects - 4 threads 2 failover apaches (Linux) - using pydirector for load balancing We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment? Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] request.set('AUTHENTICATED_USER',...
We are running Zope 2.6.x and I noticed yesterday that I could do the
following:
acl_users = container.acl_users
user = acl_users.getUser('test_user')
request.set('AUTHENTICATED_USER',user)
print request.AUTHENTICATED_USER.getUserName()
This isn't a huge deal since it doesn't seem to change the permissions
available to the user. But many of our scripts rely on
AUTHENTICATED_USER.getUserName() to return the actual logged in user. Is
this addressed in later versions of Zope? Is there a better way to get
the current user's user name?
We allow untrusted developers on our Zope server and this may allow them
to exploit certain systems.
-Brian
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
