RE: [Zope-dev] How to make Zope fail nicely under high load?
On Thu, 2004-02-12 at 07:19, Bjorn Stabell wrote: Toby wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) Okay, for example, since it's a BBS we need to have quick access to: the latest poster the latest posting the latest registered member Create a zope product that doesn't keep a history (otherwise the ZODB will grow like nuts) and store these values in it whenever there is a new port/registration. Then just read them when you need to. Anyone see anything wrong with that approach? All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time. Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO. In effect we want to do something like this: def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster def setLatestPoster(self, poster): self._v_latestPoster = poster But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads: http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html Regards, -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Thu, 2004-02-12 at 10:06, Jan-Wijbrand Kolman wrote: Bjorn Stabell wrote: (..)For ZEO, my understanding is that we would still have to store the values in the ZODB, though. You are able to store *and* share data over ZEO Clients in a TemporaryStorage by making the ZSS serve this TemporaryStorage to its ZCs. You could mount this storage as temp_folder I guess, but any other mount point would work too. Cool, didn't know that. Not sure, but I'd expect so, whether this has significant perfomance advantages over a 'normal' ZODB served by the ZSS. regards, jw -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Wed, 2004-02-11 at 13:09, Jamie Heilman wrote: Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the images are fluf, and the user knows they are fluf, then *maybe* they won't reload, and your pages will simply appear ugly. But then you have to ask yourself, why am I sending fluf images that degrades the overall user experience of my application? Clearly there's more optimization that could be done to your application. Without knowing the app I'd probably say there are many ways that it can be made more efficient. For instance are your image urls being generated using Zope's acquisition? If so, don't do that. Another alternative is to move the static/fluf images out of zope and just let apache serve them. That's just small thing that you can do to reduce the load on zope. Things like squid, zeo, better hardware, re-analyze long running routines, etc are all possible without having to do the other gymnastics that have been discussed. -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope] Proposed installation changes for review
On Tue, 2003-03-11 at 08:48, Chris McDonough wrote: On Tue, 2003-03-11 at 00:24, Edward Muller wrote: Once zope is installed in /opt/zope-2.7.0 can it be moved without damaging the install say to /home/virtual/some.host.name/opt/zope-2.7.0 ? Yes. Its location is only meaningful to the instance files that need to find it. In our hosting setup some things get run in a chroot, some things can't... Currently zope get's installed in a chroot environment for anyone who wants a zope install. It must be a complete install since when the user restarts it he will be in his chroot environment. So I'd ideally like to install zope in a way where all of the core of zope is in one place ... say ... /opt/zope/version # (/opt/zope/2.7.0, /opt/zope/2.7.1, etc...) This I can hardlink/symlink into each chroot and make permissions 755 root/root. I think this will work. The only thing that might be a little weird is tracebacks generated by pyc files, as they may report the filenames of the Python modules where they were originally installed, instead of where they live now. There is some contention about whether this happens under Python 2.2, but I know it's true for Python 2.1 and prior. Well I can install zope in /opt/zope/2.7.1 (in the real root) and then when I symlink/hardlink it into a virtual host I can link it into that hosts /opt/zope/2.7.1 ... So that's not a biggie From there I would like to be able to install an 'instance', which is ... in my case meaning the data.fs, /Products directory, log files, etc, etc. The stuff that make this users instance theirs. When the install is happening, the script executing it would most likely be outside of the chroot ... but I guess it could be configured to chroot as well.. You would need to chroot the run of makeinstance currently as it encodes paths to software within the instance files that start Zope. So if you ran it outside the chroot it would work, but when the user logged in to the chroot, the paths to the software would be wrong. That's not a problem ... at least IIRC. I can chroot when creating the account in a shell script and execute custom setup scripts. I think this might be made configurable with a switch to mkzopeinstance (--sw_location=/some/path), though. I will add this to the tentative TODO, thanks. all thought that would be nice. I already have start/stop scripts to go through the users that have a zope install and chroot into that users 'host' and then start zope as that 'hosts' administrative user. These scripts will unfortunately need to change for Zope 2.7 unless we create some sort of backwards compatibilty layer for startup. Yeah. Oh well. They aren't that complex. :-) I wouldn't worry about the backward compatibility layer myself. I don't know if there is a great value add to it, aside from keeping users from going 'WTF happened?' :-) -- Edward Muller Interlix - President Web Hosting - PC Service Support Custom Programming - Network Service Support Phone: 417-862-0573 Cell: 417-844-2435 Fax: 417-862-0572 http://www.interlix.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] Proposed installation changes for review
new-install-branch -f Zope The installation instructions in doc/INSTALL.txt has been updated on the branch; that file provides detailed information on creating an installation and instance. (A copy has been attached for convenience.) A sample installation/startup runthrough --- The below is the output from a sample install session. During this session, we install the Zope software to /tmp/zopehome and install an instance home to /tmp/insthome. [EMAIL PROTECTED] software]$ cd NewInstallBranch/ [EMAIL PROTECTED] NewInstallBranch]$ ./configure --prefix=/tmp/zopehome Configuring Zope installation Testing for an acceptable Python interpreter... Python version 2.1.3 found at /home/chrism/bin/python Python version 2.1.3 found at /home/chrism/bin/python2.1 Python version 2.2.2 found at /home/chrism/bin/python2.2 The optimimum Python version (2.2.2) was found at /home/chrism/bin/python2.2. - Zope top-level binary directory will be /tmp/zopehome. - Makefile written. Next, run make. [EMAIL PROTECTED] NewInstallBranch]$ make /home/chrism/bin/python2.2 inst/setup.py -q build_ext -i ... compilation output elided ... Zope built. Next, do 'make install' (or 'make instance' to run a Zope instance directly from the build directory). [EMAIL PROTECTED] NewInstallBranch]$ make install ... installation output elided ... Zope binaries installed successfully. Now run '/tmp/zopehome/bin/mkzopeinstance' [EMAIL PROTECTED] NewInstallBranch]$ /tmp/zopehome/bin/mkzopeinstance mkzopeinstance requires exactly one argument mkzopeinstance: Create a Zope instance home. usage: mkzopeinstance [options] directory Options: -h/--help -- print this help text -u/--user NAME:PASSWORD -- set the user name and password of the initial user [EMAIL PROTECTED] NewInstallBranch]$ /tmp/zopehome/bin/mkzopeinstance \ --user=admin:123 /tmp/inst [EMAIL PROTECTED] NewInstallBranch]$ cd /tmp/inst/ [EMAIL PROTECTED] inst]$ ls bin etc Extensions import inituser log Products README.txt var [EMAIL PROTECTED] inst]$ bin/zopectl start . daemon process started, pid=2947 [EMAIL PROTECTED] inst]$ bin/zopectl stop . daemon process stopped Changing the configuration via the config file -- The Zope configuration file uses the ZConfig API and file format. See the ZConfig documentation PDF at http://cvs.zope.org/Packages/ZConfig/doc/zconfig.pdf for more information. The default Zope configuration file will live in the instance_home's /etc directory and will be named 'zope.conf'. The Zope configuration file allows you to change the following configuration variables: instance home client home debug mode effective user production installation (on/off) locale number of zserver threads the python check interval (thread switch timing) zserver read only mode pid file name lock file name structured text header level publisher profile file (debugging) cgi environment dns server (hostname resolving in logfiles) ip address (for zserver bindings) http realm automatic quotation of dtml request data security manager stack size security policy implementation (C, Python) skip authentication checking skip ownership checking max number of session objects session add notify script path session delete notify script path session timeout minutes access rule suppress site root suppression database quota size read only database zeo client name logging (access, event, debug, filenames, to syslog, etc). network servers (http, dav, ftp, monitor, etc) databases (storages, database classes, etc.) ___ Zope maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Edward Muller Interlix - President Web Hosting - PC Service Support Custom Programming - Network Service Support Phone: 417-862-0573 Cell: 417-844-2435 Fax: 417-862-0572 http://www.interlix.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Proposed installation changes for review
On Mon, 2003-03-10 at 17:07, Paul Winkler wrote: A few questions / concerns listed below, otherwise it looks fine to me... On Mon, Mar 10, 2003 at 04:41:48PM -0500, Chris McDonough wrote: debug mode does this still toggle a whole bunch of things? production installation (on/off) what's this mean? network servers (http, dav, ftp, monitor, etc) is this where you set the ports? One thing I didn't see in your list is a way to extend the products path (currently I do that with $PRODUCTS_PATH). We need that. One more question: Does zopectl.py always detach from the terminal (booo) or not (yay) ? Or does it behave like z2.py and this depends on debug mode (booo) ? Actually I like the way z2.py detaches or doesn't detach. Perhaps a separate config option would be good to control this. -- Edward Muller Interlix - President Web Hosting - PC Service Support Custom Programming - Network Service Support Phone: 417-862-0573 Cell: 417-844-2435 Fax: 417-862-0572 http://www.interlix.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Precompiled w/Apache?
Even if you compiled zope...on a PII 450 it only took a few minutes... and the process was (minus the apache stuff)... get source tar.gz unpack in a location run python w_pcgi (or python wo_pcgi) (wo or w is dependant on if you want pcgi or not...you probably want wo_pcgi) chown -R user.group of the account who will use zope ./start done... (I probably forgot a step...but the README/docs are very clear on how to set it up) On Wednesday 20 December 2000 12:44 pm, Jonathan \(ListServ Account\) wrote: Well, I found it but - It's compiled against an older glibc... Oh well. Looks like I'll miss the deadline to get the demo up, Unless someone has some magic to pull somehow. Uncle Jonathan's Magic Zope Recipe is here ;) - Get a binary release of Zope and unpack into /usr/local for example. - Install Apache if you haven't got it already. - Do a search for 'apache' on Zope.org and locate the proxypass howto. - Read and implement proxypass. Shouldn't take more than about 30 minutes. Magic no? ;) Cya Jonathan ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) -- Edward Muller - [EMAIL PROTECTED] - [EMAIL PROTECTED] 'It's because crappy programs offend me.' --Eric Raymond ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Large XML files
Definetly consider a stripped down linux install for this machine instead. just make sure almost nothing else is running as well (A lots ditros start all sorts of stuff up for you). Edward Muller - [EMAIL PROTECTED] - [EMAIL PROTECTED] Grow a ponytail -- view it as your telepathic antenna to other Linux Kernel Developers. -- Jeff V. Merkey On Tue, 5 Dec 2000, Jim Hebert wrote: You could check into the ExtFile product (search on zope.org) and see if that makes a perf difference. That said, just about anything on that setup is going to be slow!! Please excuse the os advocacy, but given that this is likely all you'd dream of running on this server anyways (ie it's not like it's also going to be your Exchange server), you might consider putting something that will leave more spare RAM and cpu time for Zope on that box, e.g. Linux or some other OS more suited towards the low end server hardware market. Or buy a nicer box to run this on. If you don't have existing unix skills in-house then throwing money at hardware might be less expensive than investing in human resources to get a Linux box going. (Whew, being even-handed is so much harder than being an OS bigot...;-) Sorry that's probably not what you wanted to hear. =) jim On Tue, 5 Dec 2000 [EMAIL PROTECTED] wrote: I am testing the possibilities of delivering the content of XML Documents through the Zope environment. Unfortunately, some of the proposed file are rather large (up to 760KB) and just uploading them and viewing them on our current Zope server is prohibitively slow. Our server, running Z2 is a blazing P133 running NT 4.0 with 32 MB of RAM (I get the bottom feeders). Is the bottleneck the hardware; is there something I can do software-wise to improve performance; or is development not yet advanced enough to handle this scenario efficiently? Any opinions on this? Thanks, P. Johnson ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] Access to Python arrays
I am working on a product in which one of the objects keeps an array. I've
already overcome the problems with making the array persistent (i.e. by not
using .append, but intead re-assigning the array each time). Anyway...I
digress..
So I have this object that has an array..A folder may contain several of these
items, so I loop over them in DTML like this:
dtml-in "objectValues('My Object')"
dtml-var id
/dtml-in
This prints out the id of each of my objects. But I ALSO want to print out the
contents of the array, indented after each instances id. What is the best way to
do this in DTML? Is there as way? Lets say for arguments sake that each array
element is a string, how do I iterate through the array and print the
strings. Lets say each element in the array is anothe array (of 5 elements), How
do I loop over the array and pull out the sub arrays and print those five
elements?
Any help/direction would be appreciated, I've checked the docs and refered to
some sample products but I either missed something obvious (probable) or It's
hard..or something :-)
--
...EAM...
[EMAIL PROTECTED]
[EMAIL PROTECTED]
---
'It's because crappy programs offend me.' --Eric Raymond
Grow a ponytail -- view it as your telepathic antenna to other Linux Kernel
Developers. -- Jeff V. Merkey
___
Zope maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] External Methods
I'm a little new to the python sde of zope and haven't done much with the other
side (dtml,etc), but am getting back into it...So here is how I think you can do
it (note: I've done this myself recently...I'm writing a product, that started
off as a collection of external methods)...
So you have a html form with a submit button. that calls a url
(http://yoursite/folder/externalmethodname)
lets say your form has two valiables it wants to pass to the method. Let's say
the variables are 'foo' and 'foo2'.
now define you external method in a file in the zope path/Extension
directory..
lets call the file bar.py
okay define your method in the file
def imamethod(self,REQUEST=non):
dosomethingwith(REQUEST['foo']) #variable foo from the form
dosomethingwith(REQUEST['foo2']) #other variable from the form
REQUEST.RESPONSE.setHeader('content-type','text/html')
REQUEST.RESPONSE.body('HTMLBODYDone!/BODY/HTML')
That's it I think..
This is from memory so I am sure there are some things wrong with it. If anyone
cares to comment (i.e. the zope community) please let me know
On Tue, 21 Nov 2000, Kini Natekar wrote:
Hi,
I have got a html form which accepts an input from
the user. This input has to be passed to a Python
script, as a parameter, which is added as an external
method in zope.
Is there any way to do this ?
Regards,
Kini.
__
Do You Yahoo!?
Yahoo! Shopping - Thousands of Stores. Millions of Products.
http://shopping.yahoo.com/
___
Zope maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope-dev )
--
...EAM...
[EMAIL PROTECTED]
[EMAIL PROTECTED]
---
'It's because crappy programs offend me.' --Eric Raymond
Grow a ponytail -- view it as your telepathic antenna to other Linux Kernel
Developers. -- Jeff V. Merkey
___
Zope maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Gadfly
On Wed, 22 Nov 2000, Chris Withers wrote: Cees de Groot wrote: Tim Cook [EMAIL PROTECTED] said: You problems will go away if you move to MySQL, PostGres, Oracle, etc... or stop using RDBMS'es altogether... I agree. Unless there is a compelling reason, such as sharing tables with another application. Why not just use ZODB. Life can't be simplier! g Will life stay simple (robust, performant) after you've dumped a couple of hundredthousand records into ZODB? I find using mySQL a PITA compared to ZODB, but I'm not sure how well it keeps with lots of data... Shane Hathaway mentioned having a ZODB that was happily running at 11GB, that's a lot of data in my books ;-) Quickly question then... How would you record data in the ZODB? Would you have to write a Product for each thing you want to store? If you can provide some links it would be great! Thanks, -- ...EAM... [EMAIL PROTECTED] [EMAIL PROTECTED] --- 'It's because crappy programs offend me.' --Eric Raymond Grow a ponytail -- view it as your telepathic antenna to other Linux Kernel Developers. -- Jeff V. Merkey ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] Help! Zope Crash
I am running a Zwiki site at http://www.handhelds.org:8080/wiki. This site has become invaluable to our growing handhelds.org community as a place to collect information. But, sadly our zope server keeps dyingNo idea why Sometimes the zope server can be up for a few days, sometimes only a few hours. I am not sure what is causing this or how to go about tracking it down. Is there anything I can do? Any help would be greatly appreciated. -- ...EAM... [EMAIL PROTECTED] [EMAIL PROTECTED] --- 'It's because crappy programs offend me.' --Eric Raymond ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
