Re: [Zope-dev] file descriptors on Solaris - IT'S A BUG, PEOPLE!
[cc'ed to zope-dev so people can check this information] On Thursday 04 October 2001 13:47, Joseph Wayne Norton wrote: > One related question that I have regarding cgi.py and zope. > > Would there be any compatibility problem of applying the following > older cgi hotfix to python's cgi.py file and usage with Zope 2.4.1 and > higher? > > python cgi hotfix: > http://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=4 >43120 > > zope hotfix: > http://www.zope.org/Products/Zope/Hotfix_2001-07-25/README.txt > > My understanding is that the zope hotfix is not required for version > 2.4.1 and higher. Zope 2.4.1 includes the hotfix. The hotfix does not fix the "too many open files" problem. I don't believe there's any compatibility problems (we're using both OK here). Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris - IT'S A BUG, PEOPLE!
On Wednesday 03 October 2001 22:00, R. David Murray wrote: > On Tue, 2 Oct 2001, John Ziniti wrote: > > Yeah ... something tells me it's a little more complicated than that. > > Like recompiling the kernel, quite possibly. On FreeBSD there's > a sysctl, although you may still have to recompile the kernel in > some cases I think; on Linux you can zap a variable in proc. On > solariswho knows . > > > Any advice on the other front? If I can reduce the number of files > > Zope needs to process this request, I'd grumpily agree to do that, > > is Zope opening a file for every ? > > Will using help? > > I don't see why zope should need so many file descriptors. But > you may have to start tracing code . As I mentioned yesterday, this is a _bug_ in cgi.py which is fixable by getting the patch from: http://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=231249 Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris
On Tue, 2 Oct 2001, John Ziniti wrote: > Yeah ... something tells me it's a little more complicated than that. Like recompiling the kernel, quite possibly. On FreeBSD there's a sysctl, although you may still have to recompile the kernel in some cases I think; on Linux you can zap a variable in proc. On solariswho knows . > Any advice on the other front? If I can reduce the number of files > Zope needs to process this request, I'd grumpily agree to do that, > is Zope opening a file for every ? > Will using help? I don't see why zope should need so many file descriptors. But you may have to start tracing code . I have previously noticed a related phenomenon: on FreeBSD when I start zope, *something* tries to open every possible port number using a separate file descriptor. They must then be closed, because after startup only the ports requested are still open. I determined this by running lsof and capturing the output. In this case, it appears to be benign; the only negative consequence is the appearance of the "Too many open file descriptors" message in /var/log/messages. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris [SUMMARY]
On Wednesday 03 October 2001 04:50, John Ziniti wrote: > DIAGNOSIS: > It appears that this only happens when the form is specified with > "enctype=multipart/form-data". In that case, Zope (or, more accurately, > the cgi module), tries to create temporary file for each form , > no matter what type the input has ... (I think but I can't be 100% > sure about that). This seems a little weird. Why do we have to open > a file for each "part" just because it *may* contain a file? This is a bug in python's cgi module that has been patched (but not in python 2.1.1). There's a patch on sourceforge for it in any case. http://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=231249 No fiddling with max open files required. Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris [SUMMARY]
- Original Message - From: "John Ziniti" <[EMAIL PROTECTED]> To: "Andreas Jung" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Tuesday, October 02, 2001 15:57 Subject: Re: [Zope-dev] file descriptors on Solaris [SUMMARY] > > > > > > > >This is nonsense. Solaris allows of course to use more than 256 FDs. > >I don't know how they are stored inside the kernel but I have been using > >Solaris in projects where we used 1024 FDs and more. Zope does not > >increase the number of allowed FDs (resource module) but inherits > >the settings from the environment where Zope gets started. > > > >The number of FDs are set in Solaris in /etc/system: > > > >set rlim_fd_max = 4096 > >set rlim_fd_cur = 1024 > > > >Solaris 7+ allows up to 65536 FDs. > > > Solaris is definitely not my area of expertise, but to the best of my > UNIX-hunting-around- > looking-for-the-answer ability, that's all I can find. If Zope is > started in a shell with > $ ulimit -n 200 > and I POST the offending form, I get a "too many open files" error. But > if ulimit -n 512, > then I get "No such file or directory" when $ZOPE/lib/python/tempfile.py > tries to fdopen a > file descriptor higher than 255 (line 155). > > my /etc/system doesn't have anything other than some shm settings. When u dont have the settings then you are using the default settings (maybe 256). Modify the settings according to your needs. > Can you create a form: > > > > > value=""> > > I am running Linux here with 1024 FDs. Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris [SUMMARY]
> > > >This is nonsense. Solaris allows of course to use more than 256 FDs. >I don't know how they are stored inside the kernel but I have been using >Solaris in projects where we used 1024 FDs and more. Zope does not >increase the number of allowed FDs (resource module) but inherits >the settings from the environment where Zope gets started. > >The number of FDs are set in Solaris in /etc/system: > >set rlim_fd_max = 4096 >set rlim_fd_cur = 1024 > >Solaris 7+ allows up to 65536 FDs. > Solaris is definitely not my area of expertise, but to the best of my UNIX-hunting-around- looking-for-the-answer ability, that's all I can find. If Zope is started in a shell with $ ulimit -n 200 and I POST the offending form, I get a "too many open files" error. But if ulimit -n 512, then I get "No such file or directory" when $ZOPE/lib/python/tempfile.py tries to fdopen a file descriptor higher than 255 (line 155). my /etc/system doesn't have anything other than some shm settings. Can you create a form: and tell me if you get the OSError? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris [SUMMARY]
- Original Message - From: "John Ziniti" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, October 02, 2001 14:50 Subject: Re: [Zope-dev] file descriptors on Solaris [SUMMARY] > > PROGNOSIS: > The problem (on Solaris) is not very easy to fix, since it lies in the > system-wide definition of a file descriptor, which uses only one byte > to store the fd value (i.e., anything higher than 256 is meaningless, > and truncated??). Changing this struct is not easy. The problem is > not that Zope is exceeding the *allowed* number of FD's (usually > policed by the shell), but that Zope is exceeding the *meaningful* > number of FD's. This sucks. :-) This is nonsense. Solaris allows of course to use more than 256 FDs. I don't know how they are stored inside the kernel but I have been using Solaris in projects where we used 1024 FDs and more. Zope does not increase the number of allowed FDs (resource module) but inherits the settings from the environment where Zope gets started. The number of FDs are set in Solaris in /etc/system: set rlim_fd_max = 4096 set rlim_fd_cur = 1024 Solaris 7+ allows up to 65536 FDs. Cheers, Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris [SUMMARY]
I'd just like to summarize for the list some additional findings, questions and clarifications. DIAGNOSIS: It appears that this only happens when the form is specified with "enctype=multipart/form-data". In that case, Zope (or, more accurately, the cgi module), tries to create temporary file for each form , no matter what type the input has ... (I think but I can't be 100% sure about that). This seems a little weird. Why do we have to open a file for each "part" just because it *may* contain a file? PROGNOSIS: The problem (on Solaris) is not very easy to fix, since it lies in the system-wide definition of a file descriptor, which uses only one byte to store the fd value (i.e., anything higher than 256 is meaningless, and truncated??). Changing this struct is not easy. The problem is not that Zope is exceeding the *allowed* number of FD's (usually policed by the shell), but that Zope is exceeding the *meaningful* number of FD's. This sucks. :-) SUMMARY: If you're planning on using large forms in Zope on Solaris (version??), you'll have to move your file uploads to another page, since specifying "multipart/form-data" causes cgi.py to open a tempfile.TemporaryFile for each input on the form, which causes problems if the number of inputs is greater than the file descriptor address space. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris
Yeah ... something tells me it's a little more complicated than that. Any advice on the other front? If I can reduce the number of files Zope needs to process this request, I'd grumpily agree to do that, is Zope opening a file for every ? Will using help? Thanks. Andreas Jung wrote: >- Original Message - >From: "John Ziniti" <[EMAIL PROTECTED]> >To: <[EMAIL PROTECTED]> >Sent: Tuesday, October 02, 2001 12:01 >Subject: Re: [Zope-dev] file descriptors on Solaris > > >>:( I was hoping against hope that this wouldn't be the answer ... >> >>I think that the hard limit onSolaris must be 256, because ulimit -n 200 >>seems to have the appropriate effect of making Zope complain about >>"too many open files" ... but anything higher than 256 gets >>a "File not found" ... >> > >Just ask your administrator to increase the number of file descriptors. >Usually this requires *only* a reboot of the machine :-) > >Andreas > > >___ >Zope-Dev maillist - [EMAIL PROTECTED] >http://lists.zope.org/mailman/listinfo/zope-dev >** No cross posts or HTML encoding! ** >(Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris
- Original Message - From: "John Ziniti" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, October 02, 2001 12:01 Subject: Re: [Zope-dev] file descriptors on Solaris > :( I was hoping against hope that this wouldn't be the answer ... > > I think that the hard limit onSolaris must be 256, because ulimit -n 200 > seems to have the appropriate effect of making Zope complain about > "too many open files" ... but anything higher than 256 gets > a "File not found" ... Just ask your administrator to increase the number of file descriptors. Usually this requires *only* a reboot of the machine :-) Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris
:( I was hoping against hope that this wouldn't be the answer ... I think that the hard limit onSolaris must be 256, because ulimit -n 200 seems to have the appropriate effect of making Zope complain about "too many open files" ... but anything higher than 256 gets a "File not found" ... Let's try this from another angle ... why does Zope need to open so many files just to process a *large-but-not-really-large* POST? Can I change something about the way the form is set up (I use a lot of ":records") to circumvent the problem? Andreas Jung wrote: >There is always a hard limit of filedescriptors (based on the kernel >configuration). But usually you can change the number of descriptors >using "limit descriptors " under tcsh/csh. Under Bash I think you >must call "ulimit". > >Andreas >- Original Message - >From: "John Ziniti" <[EMAIL PROTECTED]> >To: <[EMAIL PROTECTED]> >Sent: Tuesday, October 02, 2001 11:46 >Subject: [Zope-dev] file descriptors on Solaris > > >>I am running into a problem where Zope is trying to open too many >>file descriptors (256) in order to process a POST. >> >>I'm not sure how to phrase this question, but my reading has suggested >>that this limit may be set by the FILE struct in /usr/incldue/stdio.h. >> >>Does anyone know if this is used in Python/Zope? >> >>My first impression was that this problem would be easy to solve -- >>just up the number of FD's, but now, I'm not so sure. Any >>suggestions? >> >>Here's a traceback -- but I don't think it's much help >> >>Traceback (innermost last): >> File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, >> >line 223, in publish_module > >> File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, >> >line 187, in publish > >> File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/Zope/__init__.py, line >> >226, in zpublisher_exception_hook > >>(Object: ApplicationDefaultPermissions) >> File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, >> >line 136, in publish > >> File >> >/u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/HTTPRequest.py, line >405, in processInputs > >> File /ilocal/lib/python2.1/cgi.py, line 517, in __init__ >> File /ilocal/lib/python2.1/cgi.py, line 606, in read_multi >> File /ilocal/lib/python2.1/cgi.py, line 519, in __init__ >> File /ilocal/lib/python2.1/cgi.py, line 616, in read_single >> File /ilocal/lib/python2.1/cgi.py, line 636, in read_lines >> File /ilocal/lib/python2.1/cgi.py, line 723, in make_file >> File /ilocal/opt/Zope-2.4.0-src/lib/python/tempfile.py, line 155, in >> >TemporaryFile > >>OSError: [Errno 2] No such file or directory >> >> >> >> >>___ >>Zope-Dev maillist - [EMAIL PROTECTED] >>http://lists.zope.org/mailman/listinfo/zope-dev >>** No cross posts or HTML encoding! ** >>(Related lists - >> http://lists.zope.org/mailman/listinfo/zope-announce >> http://lists.zope.org/mailman/listinfo/zope ) >> > > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] file descriptors on Solaris
There is always a hard limit of filedescriptors (based on the kernel configuration). But usually you can change the number of descriptors using "limit descriptors " under tcsh/csh. Under Bash I think you must call "ulimit". Andreas - Original Message - From: "John Ziniti" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, October 02, 2001 11:46 Subject: [Zope-dev] file descriptors on Solaris > I am running into a problem where Zope is trying to open too many > file descriptors (256) in order to process a POST. > > I'm not sure how to phrase this question, but my reading has suggested > that this limit may be set by the FILE struct in /usr/incldue/stdio.h. > > Does anyone know if this is used in Python/Zope? > > My first impression was that this problem would be easy to solve -- > just up the number of FD's, but now, I'm not so sure. Any > suggestions? > > Here's a traceback -- but I don't think it's much help > > Traceback (innermost last): > File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 223, in publish_module > File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 187, in publish > File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/Zope/__init__.py, line 226, in zpublisher_exception_hook > (Object: ApplicationDefaultPermissions) > File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 136, in publish > File /u05/ilocal/opt/Zope-2.4.0-src/lib/python/ZPublisher/HTTPRequest.py, line 405, in processInputs > File /ilocal/lib/python2.1/cgi.py, line 517, in __init__ > File /ilocal/lib/python2.1/cgi.py, line 606, in read_multi > File /ilocal/lib/python2.1/cgi.py, line 519, in __init__ > File /ilocal/lib/python2.1/cgi.py, line 616, in read_single > File /ilocal/lib/python2.1/cgi.py, line 636, in read_lines > File /ilocal/lib/python2.1/cgi.py, line 723, in make_file > File /ilocal/opt/Zope-2.4.0-src/lib/python/tempfile.py, line 155, in TemporaryFile > OSError: [Errno 2] No such file or directory > > > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
