Re: More efficient StringField and Field classes
It has been awhile, but I finally made the patch.
Advice and feedback is welcome.
What it does:
- Simplifies the creation of StringField objects. This was already
marked as a TODO in the 3.2.5b code.
- Does not create a file object (cStringIO) for each and every field.
- FieldStorage.get() will always return the same instance(s) for any
given name.
- FieldStorage.get() is very cheap now (does not create new objects,
which is expensive in Python)
It may break:
- Code that creates an instance of Field or StringField directly. This
can be fixed by extending the __init__() routine, but I don't think it's
needed.
It works:
- Perfect on all of my projects.
Attached:
- Patch created by SVN
--
Mike Looijmans
Philips Natlab / Topic Automation
Index: util.py
===
--- util.py (revision 348746)
+++ util.py (working copy)
@@ -48,19 +48,8 @@
class Field:
- def __init__(self, name, file, ctype, type_options,
-disp, disp_options, headers = {}):
+ def __init__(self, name):
self.name = name
- self.file = file
- self.type = ctype
- self.type_options = type_options
- self.disposition = disp
- self.disposition_options = disp_options
- if disp_options.has_key(filename):
- self.filename = disp_options[filename]
- else:
- self.filename = None
- self.headers = headers
def __repr__(self):
Return printable representation.
@@ -81,13 +70,30 @@
self.file.close()
class StringField(str):
-This class is basically a string with
- a value attribute for compatibility with std lib cgi.py
-
+ This class is basically a string with
+added attributes for compatibility with std lib cgi.py. Basically, this
+works the opposite of Field, as it stores its data in a string, but creates
+a file on demand. Field creates a value on demand and stores data in a
file.
+
+filename = None
+headers = {}
+ctype = text/plain
+type_options = {}
+disposition = None
+disp_options = None
+
+# I wanted __init__(name, value) but that does not work (apparently, you
+# cannot subclass str with a constructor that takes 1 argument)
+def __init__(self, value):
+'''Create StringField instance. You'll have to set name yourself.'''
+str.__init__(self, value)
+self.value = value
- def __init__(self, str=):
- str.__init__(self, str)
- self.value = self.__str__()
+def __getattr__(self, name):
+if name != 'file':
+raise AttributeError, name
+self.file = cStringIO.StringIO(self.value)
+return self.file
class FieldStorage:
@@ -103,8 +109,7 @@
if req.args:
pairs = parse_qsl(req.args, keep_blank_values)
for pair in pairs:
- file = cStringIO.StringIO(pair[1])
- self.list.append(Field(pair[0], file, text/plain, {}, None,
{}))
+ self.add_field(pair[0], pair[1])
if req.method != POST:
return
@@ -123,9 +128,7 @@
if ctype.startswith(application/x-www-form-urlencoded):
pairs = parse_qsl(req.read(clen), keep_blank_values)
for pair in pairs:
- # TODO : isn't this a bit heavyweight just for form fields ?
- file = cStringIO.StringIO(pair[1])
- self.list.append(Field(pair[0], file, text/plain, {}, None,
{}))
+ self.add_field(pair[0], pair[1])
return
if not ctype.startswith(multipart/):
@@ -205,33 +208,44 @@
else:
name = None
+ # create a file object
# is this a file?
if disp_options.has_key(filename):
if file_callback and callable(file_callback):
file = file_callback(disp_options[filename])
else:
- file = self.make_file()
+ file = tempfile.TemporaryFile(w+b)
else:
if field_callback and callable(field_callback):
file = field_callback()
else:
- file = self.make_field()
+ file = cStringIO.StringIO()
# read it in
- end_of_stream = self.read_to_boundary(req, boundary, file)
+ self.read_to_boundary(req, boundary, file)
file.seek(0)
-
+
# make a Field
- field = Field(name, file, ctype, type_options, disp, disp_options,
headers)
-
+ if disp_options.has_key(filename):
+ field = Field(name)
+ field.filename = disp_options[filename]
+ else:
+ field = StringField(file.read())
+ field.name = name
+ field.file = file
+ field.type = ctype
+ field.type_options = type_options
+ field.disposition = disp
+
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
Jim, I am not totally surprised... I am afraid this is a platform specific issue as we are running mod_python on Tru64. Something like a 64 bits issue. Does it sound a reasonnable possibility ? How to progress in troubleshooting ? Michel --On jeudi 24 novembre 2005 17:41 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
Michel Jouvin wrote: Jim, I am not totally surprised... I am afraid this is a platform specific issue as we are running mod_python on Tru64. Something like a 64 bits issue. Does it sound a reasonnable possibility ? I have no idea what may be going on, but that seems as likely as anything else. How to progress in troubleshooting ? Again, no clue. :(. Hopefully some of the bigger brains that hang out around here will chime in. I know Indrek Järve tested 3.2.2b on SuSE Linux 9.2 (x86-64). Perhaps he or someone else with a 64-bit platform could try and reproduce the problem. That would tell us if it's 64-bit related or Tru64 related. I've attached my test script if anyone wants to mess with it. I'm sure I don't need to tell you to *not* run it on a production machine. ;) You'll likely want to change the PAUSE variable to something less than 30 seconds, which is the time between the kill calls. I was testing using qemu, and it needs lots of time for things to happen. usage: ./killchildren # number of loops Jim Michel --On jeudi 24 novembre 2005 17:41 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * killchildren.sh Description: application/shellscript
Re: More efficient StringField and Field classes
Hi Mike, I don't have time to dig into this tonight, but your patch causes one of the unit tests to fail. FAIL: test_util_fieldstorage (__main__.PerRequestTestCase) -- Traceback (most recent call last): File test.py, line 1117, in test_util_fieldstorage self.fail(`rsp`) File /usr/lib/python2.3/unittest.py, line 270, in fail raise self.failureException, msg AssertionError: ['1', '2', '3', '4'] Jim
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
for myself, I have not had problems on ubuntu 5.10 amd64. Nick Jim Gallacher wrote: Michel Jouvin wrote: Jim, I am not totally surprised... I am afraid this is a platform specific issue as we are running mod_python on Tru64. Something like a 64 bits issue. Does it sound a reasonnable possibility ? I have no idea what may be going on, but that seems as likely as anything else. How to progress in troubleshooting ? Again, no clue. :(. Hopefully some of the bigger brains that hang out around here will chime in. I know Indrek Järve tested 3.2.2b on SuSE Linux 9.2 (x86-64). Perhaps he or someone else with a 64-bit platform could try and reproduce the problem. That would tell us if it's 64-bit related or Tru64 related. I've attached my test script if anyone wants to mess with it. I'm sure I don't need to tell you to *not* run it on a production machine. ;) You'll likely want to change the PAUSE variable to something less than 30 seconds, which is the time between the kill calls. I was testing using qemu, and it needs lots of time for things to happen. usage: ./killchildren # number of loops Jim Michel --On jeudi 24 novembre 2005 17:41 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
performing a security analsysis on the Tomcat software
Dear tomcat developers, BSI, the german Federal Office for Information Security -- Bundesamt fur Sicherheit in der Informationstechnik http://www.bsi.de, e-mail: [EMAIL PROTECTED] endorses the use of Open Source software and has contracted T-Systems to perform a security check on Tomcat. The Federal Office for Information Security (BSI) is the central IT security service provider for the German government. By our basic research within the area of IT security we take responsibility for the security of our society, and are thus indispensable to the internal security of Germany. Our services and products are aimed at the users and manufacturers of information technology products. Those are primarily the public administration at federal, state and municipal level, in addition companies and private users. As Germanys National Security Agency, it is our goal to promote IT security in Germany so that everyone can make the most of the opportunities opened up by the information society. As part of its activities, BSI has contracted the security engineering group at T-Systems International to perform security-related testing of the open source Tomcat software. These activities comprise the following: + installation documentation checks, + a source code review of mod_jk and selected parts of Tomcat, + penetration testing. BSI is going to make the results of the analysis publicly available on internet, so people will be able to download the study from their site. Please contact [EMAIL PROTECTED] for any questions related to the analysis, or feel free to mail me at [EMAIL PROTECTED] The analysis has already started. I think I owe you people an apology for already having posted two bugreports (#37322 and #37332) prior to this announcement of our activity to the mailing list. We sincerely hope that our analysis will contribute to make Tomcat even more robust and easy to deploy. So far, we are very pleased with what we see, which gives us a good impression of the software. Our goal is to publish to the bugtracker individual and separable items which can be classified as bugs. We'll alert [EMAIL PROTECTED] for any serious security vulnerabilities we find (which is what Bugzilla recommends). And finally, I plan to send a general summary of findings to this mailing list when we'll have finished. These will be the kind of findings and remarks that do not fit into individual methods and modules but rather concern the software as a whole. Regards, Jorg Hohle. Solution Service Center Testfactory Security T-Systems International GmbH Postal address: Deutsche-Telekom-Allee 7, 64295 Darmstadt Tel. ++49 6151 937-6913
Re: performing a security analsysis on the Tomcat software
On Thursday 24 November 2005 09:25, Hoehle, Joerg-Cyril wrote: Dear tomcat developers, This is the httpd development list. You won't find tomcat developers here, except by coincidence (those who are interested in both projects). -- Nick Kew
Re: How to determine if a graceful restart is initiated from within a module
I wanted to add that I am working om mod_cgid and thats where I want to check if it is a graceful restart. I did some more investigation and found that listener_may_exit is not set during a graceful restart for cgid which is why ap_graceful_stop_signalled() does not return 1. Is there any way I can find if it is a graceful restart in mod_cgid ? Thanks and Regards, Kiran Kiran Mendonce wrote: Hi, I'd like to know how a module can determine if a graceful restart has been issued. I tried using the ap_mpm_query() and the ap_graceful_stop_signalled() APIs but I have'nt really been able to get what I want. ap_mpm_query() always returns 0 on graceful/restart/stop conditions and ap_graceful_stop_signalled() does not return 1 on graceful restart. So I'm confused what to do. Any suggestions ? Thanks and Regards, Kiran
Re: 2.1 Live on Proudction Sites was Re: [VOTE] 2.1.10 as GA
On Mon, Nov 21, 2005 at 12:53:01PM +, Colm MacCarthaigh wrote: On Sun, Nov 20, 2005 at 10:44:05AM +, Nick Kew wrote: FWIW, www.apache.org has been running 2.1.9 since early November. I've been running it exclusively live since 2.1.9 (and previously 2.1.8 quietly on a high port). I expect Colm should be able to report on it running in an ultra-busy environment. It's been running on ftp.heanet.ie for about 3 hours now, no problems so far, but I'll check back in in a few days. o.k., 76 hours later, a few dozen million requests, 10 Terabytes, a peak of 480Mbit/sec later and a concurrency of just over 5000 later, no problems. +1 on GA, ignore my previous -1, which as Justin pointed out has a workaround. The build process still needs refinement, but we need more user feedback too, and it looks like GA is when we're going to get it. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Suggest renaming mod_authz_host to mod_access_host
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I would like to put the case for renaming mod_authz_host to mod_access_host. There is a small but significant division between: access control: is this request permitted, based on where it is being made from and authorization: is this request permitted, based on who is making it. This division is modelled by the separate access_checker/auth_checker hooks, but it is also user-visible, through the Satisfy Any|All directive. For a server administrator to properly understand the Satisfy directive's behaviour, they need to understand this division. For all access and authz modules to share mod_authz_ names, is misleading, and hinders user understanding of this point. Therefore, I think it would be best for mod_authz_host to be renamed mod_access_host. The previous thread on this topic, titled mod_access vs mod_authz_host in http://mail-archives.apache.org/mod_mbox/httpd-dev/200511.mbox/browser seemed to be heading in this direction, but seems to have died out without any action being taken, so I am reviving this discussion, hoping to get this resolved before GA happens. Max. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Cygwin) iD8DBQFDhiebfFNSmcDyxYARAtXnAJ9HsztbY3xGq17PXwryYVvdwSQsEwCfTaLi oxlzY7ESShhdBM9Y3GVVj5g= =6bdq -END PGP SIGNATURE-
Re: Suggest renaming mod_authz_host to mod_access_host
access control: is this request permitted, based on where it is being made from In other words, is the host from which the request comes, authorised to make this request? Hence mod_authz_host. Joost
Re: Suggest renaming mod_authz_host to mod_access_host
Joost de Heer wrote: access control: is this request permitted, based on where it is being made from In other words, is the host from which the request comes, authorised to make this request? Hence mod_authz_host. I tend to agree that the new name is confusing. It is theoretically possible to have host authz integrated into our system (Require host .apache.org, for example), but it isn't designed that way. But the new name seems to suggest that it is part of the same system. And note that any name involving host is also a little deceptive since the module allows control based on arbitrary env variables, and hence arbitrary request headers. But this discussion is moot if if 2.1.10 beta becomes 2.2.0 because we shouldn't change a module name after release. Joshua.
