Re: mod_python 3.2.8 available for testing
+1 FreeBSD 6.0, apache 2.0.55 mpm-prefork, python 2.4.2
Re: 3.2.6 or not
Jim Gallacher wrote: +1 trunk rev 374588 Debian (sid), Apache 2.0.55-prefork, Python 2.3.5 +1 trunk rev 374588 Debian (sarge), Apache 2.0.54-worker, Python 2.3.5 +1 trunk rev 374588 Debian (sarge), Apache 2.0.54-prefork, Python 2.3.5 If I can get just one more test from FreeBSD 5 or 6, I'll produce the 3.2.7 tarball. +1 trunk rev 374709 FreeBSD 6.0 Apache 2.0.55-prefork, Python 2.4.2 This is a machine that always had trouble with that connectionhandler test before. Ran the entire unittest 5 times in a row with no trouble. Barry
Re: Segfaults in ConnectionHander FreeBSD (was Re: 3.2.6 test period - how long do we wait?)
Jim Gallacher wrote: Dang, it's frustrating not being able to reproduce this bug in Linux. I suppose it's maybe something to do with different malloc implementations or such. I haven't seen any +1s for OpenBSD, which would be interesting to see since they added some stuff in 3.8 to help catch problems with this sort of thing http://kerneltrap.org/node/5584 Anyone been able to use valgrind or similar with mod_python? I Googled and found a couple old messages from '02 and '04 mentioning attempts to use this, but doesn't sound like much came out of it. I think there's a valgrind port on FreeBSD, so I may give that a try. Barry
Re: 3.2.6 test period - how long do we wait?
I don't know if this is the answer to the problem, but it looks like a
bug anyway. In connobject.c starting at line 133:
/* time to grow destination string? */
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer += HUGE_STRING_LEN;
bufsize += HUGE_STRING_LEN;
}
It looks like we've just set the buffer pointer to an address somewhere
inside the buffer. That can't be good. The buffer pointer should be set
to the bytes_read position. Perhaps one of you FreeBSD heads could try
the attached patch.
Jim
Index: src/connobject.c
===
--- src/connobject.c(revision 369511)
+++ src/connobject.c(working copy)
@@ -135,7 +135,7 @@
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
-buffer += HUGE_STRING_LEN;
+buffer += bytes_read;
bufsize += HUGE_STRING_LEN;
}
Sorry, that doesn't seem to fix it. I did a fresh extraction of
mod_python-3.2.6.tgz, applied the patch, did ./configure, make, su, make
install, exit su, cd test, ran test.py - got the same result as before,
with the same core dump apparently.
I think this is the general kind of thing we're looking for though, with
some mistaken pointer/memory operation.
---
As I mentioned in another message, I did some experimenting with
disabling other unittests and found if you disable just
"test_fileupload", all the remaining tests including
"test_connectionhandler" pass.
If you disable everything except "test_fileupload" and
"test_connectionhandler", then "test_connectionhandler" still crashes.
So I suspect that it's code involved with running "test_fileupload"
(Testing 1 MB file upload support) that's really the source of the
problem, and it's screwing up some part of memory thats only tripped
over later later during the connectionhandler test.
Barry
Re: mod_python 3.2.6 (Final!) available for testing
Still seeing a failure - seems to be the same thing I saw back on 3.2.5b http://www.mail-archive.com/python-dev@httpd.apache.org/msg00750.html and suspiciously similar to this report on Mac OSX http://issues.apache.org/jira/browse/MODPYTHON-102 FreeBSD 6.0 Apache 2.0.55 (prefork) port built WITH_THREADS=1 Python 2.4.2 built from ports with these port options THREADS HUGE_STACK_SIZE UCS4 PYMALLOC IPV6 == ERROR: test_connectionhandler (__main__.PerRequestTestCase) -- Traceback (most recent call last): File "test.py", line 1336, in test_connectionhandler f = urllib.urlopen(url) File "/usr/local/lib/python2.4/urllib.py", line 77, in urlopen return opener.open(url) File "/usr/local/lib/python2.4/urllib.py", line 185, in open return getattr(self, name)(url) File "/usr/local/lib/python2.4/urllib.py", line 317, in open_http return self.http_error(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 334, in http_error return self.http_error_default(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 574, in http_error_default return addinfourl(fp, headers, "http:" + url) File "/usr/local/lib/python2.4/urllib.py", line 863, in __init__ addbase.__init__(self, fp) File "/usr/local/lib/python2.4/urllib.py", line 813, in __init__ self.read = self.fp.read AttributeError: 'NoneType' object has no attribute 'read' -- Ran 45 tests in 65.420s FAILED (errors=1) F Stopping Apache... /usr/local/sbin/httpd -k stop -f /home/barryp/mod_python-3.2.6/test/conf/test.conf == FAIL: testPerRequestTests (__main__.PerInstanceTestCase) -- Traceback (most recent call last): File "test.py", line 1928, in testPerRequestTests self.failUnless(result.wasSuccessful()) AssertionError -- Ran 6 tests in 121.462s FAILED (failures=1) GDB backtrace -- #0 0x0058 in ?? () #1 0x2849b4f5 in _conn_read () from /home/barryp/mod_python-3.2.6/src/mod_python.so #2 0x2849b6c9 in conn_readline () from /home/barryp/mod_python-3.2.6/src/mod_python.so #3 0x284e4ef2 in PyEval_EvalFrame () from /home/barryp/mod_python-3.2.6/src/mod_python.so #4 0x284e5091 in PyEval_EvalFrame () from /home/barryp/mod_python-3.2.6/src/mod_python.so #5 0x284e56e4 in PyEval_EvalCodeEx () from /home/barryp/mod_python-3.2.6/src/mod_python.so #6 0x2851ede2 in function_call () from /home/barryp/mod_python-3.2.6/src/mod_python.so #7 0x284a5f44 in PyObject_Call () from /home/barryp/mod_python-3.2.6/src/mod_python.so #8 0x284ab986 in instancemethod_call () from /home/barryp/mod_python-3.2.6/src/mod_python.so #9 0x284a5f44 in PyObject_Call () from /home/barryp/mod_python-3.2.6/src/mod_python.so #10 0x284a6119 in PyObject_CallMethod () from /home/barryp/mod_python-3.2.6/src/mod_python.so #11 0x284a35bf in PythonConnectionHandler () from /home/barryp/mod_python-3.2.6/src/mod_python.so #12 0x0807025a in ap_run_process_connection () #13 0x08066285 in child_main () #14 0x080664af in make_child () #15 0x08066540 in startup_children () #16 0x08066bc3 in ap_mpm_run () #17 0x0806be77 in main () --- I'm suspicious about whether the problem is actually in the connection handler code - if I strip the test.py down to just run test_connectionhandler, it works fine. But after some trial and error adding things back, it seems the simplest test combination that causes the problem is to run test_fileupload, and then test_connectionhandler. So I'm basically just running --- [EMAIL PROTECTED]:~/mod_python-3.2.6/test>python test2.py * Running the per-request test suite... Creating config listen port: 57772 Starting Apache /usr/local/sbin/httpd -k start -f /home/barryp/mod_python-3.2.6/test/conf/test.conf * Testing 1 MB file upload support --> Send + process + receive took 0.577 s . * Testing PythonConnectionHandler E * Testing internally (status messages go to error_log) . - So I wonder if some non-connection-handling code is stomping over some structure that doesn't happen to be used til the connection-handler is exercised? In that case it maybe we can't just shrug it off figuring it's "...hardly likely that anyone would use connection handlers with mod_python for anything meaningful." Barry
Re: [jira] Created: (MODPYTHON-102) Bus error with connection handler on secondary port.
Graham Dumpleton (JIRA) wrote: Bus error with connection handler on secondary port. Key: MODPYTHON-102 URL: http://issues.apache.org/jira/browse/MODPYTHON-102 Project: mod_python Type: Bug Components: core Versions: 3.2 Environment: Mas OS X 10.3 (Panther) Reporter: Graham Dumpleton Logging this one for posterity as hardly likely that anyone would use connection handlers with mod_python for anything meaningful. Basic problem is that use of connection handler on secondary listener port is resulting in a bus error deep in Apache code when it tries to read data from the connection. The problem initially only seems to be present on Mac OS X 10.3. The problem could not be duplicated on Linux. On Mac OS X the problem occurred with both Apache 2.0.51 and 2.0.55 in both prefork and worker MPM modes. ... Output from gdb is below. Note that if the connection handler simply returns apache.DECLINED, everything is okay and Apache will serve up pages fine, only when doing read of data from connection handler does it decide to crash. (gdb) cont Continuing. Program received signal EXC_BAD_ACCESS, Could not access memory. [Switching to process 2342 thread 0xc03] 0x0058 in ?? () (gdb) where #0 0x0058 in ?? () Cannot access memory at address 0x58 Cannot access memory at address 0x58 #1 0x006068f8 in _conn_read (c=0x1933014, mode=AP_MODE_READBYTES, len=0) at connobject.c:117 #2 0x00606b04 in conn_read (self=0x471da0, args=0xf0101140) at connobject.c:177 That's interesting in that it's exactly the same crash I was seeing on FreeBSD 6.0 (x86) when running the test_connectionhandler unittest. Even the invalid access to 0x58. So it's not just a MacOS thing. I posted the details in a message to this list back on 11/17/2005 (are there archives? I'm not finding any). Barry
Re: mod_python 3.2.5b - FreeBSD 6.0 segfault
Barry Pederson wrote: >> > FreeBSD 6.0 >> > Apache 2.0.55 port built WITH_THREADS=1 >> > Python 2.4.2 == ERROR: test_connectionhandler (__main__.PerRequestTestCase) -- Traceback (most recent call last): File "test.py", line 1220, in test_connectionhandler f = urllib.urlopen(url) File "/usr/local/lib/python2.4/urllib.py", line 77, in urlopen return opener.open(url) File "/usr/local/lib/python2.4/urllib.py", line 185, in open return getattr(self, name)(url) File "/usr/local/lib/python2.4/urllib.py", line 317, in open_http return self.http_error(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 334, in http_error return self.http_error_default(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 574, in http_error_default return addinfourl(fp, headers, "http:" + url) File "/usr/local/lib/python2.4/urllib.py", line 863, in __init__ addbase.__init__(self, fp) File "/usr/local/lib/python2.4/urllib.py", line 813, in __init__ self.read = self.fp.read AttributeError: 'NoneType' object has no attribute 'read' -- Looking at my logs/error_log I see [Mon Nov 14 23:10:23 2005] [notice] child pid 53034 exit signal Segmentation fault (11), possible coredump in /home/barryp/mod_python-3.2.5b/test I rebuilt apache and mod_python with debugging turned on, went into gdb with "gdb /usr/local/sbin/httpd httpd.core", did a "bt", and got: -- #0 0x0058 in ?? () #1 0x284b5ad6 in _conn_read (c=0x8258128, mode=AP_MODE_GETLINE, len=0) at connobject.c:117 #2 0x284b5d63 in conn_readline (self=0x86b6780, args=0x819002c) at connobject.c:193 #3 0x285014fa in PyEval_EvalFrame () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #4 0x28501699 in PyEval_EvalFrame () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #5 0x28501cec in PyEval_EvalCodeEx () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #6 0x2853b3ea in function_call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #7 0x284c254c in PyObject_Call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #8 0x284c7f8e in instancemethod_call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #9 0x284c254c in PyObject_Call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #10 0x284c2721 in PyObject_CallMethod () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #11 0x284bea7d in python_connection (con=0x8258128) at mod_python.c:1281 #12 0x284bfbd5 in PythonConnectionHandler (con=0x8258128) at mod_python.c:1929 #13 0x08075546 in ap_run_process_connection (c=0x8258128) at connection.c:43 #14 0x08075940 in ap_process_connection (c=0x8258128, csd=0x8258050) at connection.c:176 #15 0x08068971 in child_main (child_num_arg=0) at prefork.c:610 #16 0x08068afb in make_child (s=0x80b5df8, slot=0) at prefork.c:704 #17 0x08068b71 in startup_children (number_to_start=3) at prefork.c:722 #18 0x08068f6e in ap_mpm_run (_pconf=0x80b4018, plog=0x80e0018, s=0x80b5df8) at prefork.c:941 #19 0x0806fd5e in main (argc=5, argv=0xbfbfea90) at main.c:638 I've been looking into this crash, learning a little gdb along the way, and adding some ap_log_cerror calls to the program, and have only been able to figure out that in connobject.c, around line 75 - Py_BEGIN_ALLOW_THREADS; rc = ap_get_brigade(c->input_filters, bb, mode, APR_BLOCK_READ, bufsize); Py_END_ALLOW_THREADS; --- after this point, the bucket-brigade "bb" seems to contain a corrupt bucket (later on pointed to by "b"), that at the very least has its type field pointing at some garbage, instead of what I'm guessing should be apr_bucket_type_heap. b->type->read has a value of 0x58, which explains the first line of that backtrace. I thought this might be a threading problem, and removed those "ALLOW_THREADS" macros around the ap_get_brigade call, but that didn't change the result at all. So I guess I'm a bit stumped as to what's messing up this bucket. Barry
Re: mod_python 3.2.5b available for testing
Graham Dumpleton wrote: I can't find the old mail about this, but Grisha suggested that this can occur in virtual hosting environments, eg, OpenVPS. No, no virtual hosting, jails, or other obviously unusual stuff. >> > FreeBSD 6.0 >> > Apache 2.0.55 port built WITH_THREADS=1 >> > Python 2.4.2 == ERROR: test_connectionhandler (__main__.PerRequestTestCase) -- Traceback (most recent call last): File "test.py", line 1220, in test_connectionhandler f = urllib.urlopen(url) File "/usr/local/lib/python2.4/urllib.py", line 77, in urlopen return opener.open(url) File "/usr/local/lib/python2.4/urllib.py", line 185, in open return getattr(self, name)(url) File "/usr/local/lib/python2.4/urllib.py", line 317, in open_http return self.http_error(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 334, in http_error return self.http_error_default(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 574, in http_error_default return addinfourl(fp, headers, "http:" + url) File "/usr/local/lib/python2.4/urllib.py", line 863, in __init__ addbase.__init__(self, fp) File "/usr/local/lib/python2.4/urllib.py", line 813, in __init__ self.read = self.fp.read AttributeError: 'NoneType' object has no attribute 'read' -- Looking at my logs/error_log I see [Mon Nov 14 23:10:23 2005] [notice] child pid 53034 exit signal Segmentation fault (11), possible coredump in /home/barryp/mod_python-3.2.5b/test I rebuilt apache and mod_python with debugging turned on, went into gdb with "gdb /usr/local/sbin/httpd httpd.core", did a "bt", and got: -- #0 0x0058 in ?? () #1 0x284b5ad6 in _conn_read (c=0x8258128, mode=AP_MODE_GETLINE, len=0) at connobject.c:117 #2 0x284b5d63 in conn_readline (self=0x86b6780, args=0x819002c) at connobject.c:193 #3 0x285014fa in PyEval_EvalFrame () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #4 0x28501699 in PyEval_EvalFrame () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #5 0x28501cec in PyEval_EvalCodeEx () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #6 0x2853b3ea in function_call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #7 0x284c254c in PyObject_Call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #8 0x284c7f8e in instancemethod_call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #9 0x284c254c in PyObject_Call () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #10 0x284c2721 in PyObject_CallMethod () from /home/barryp/mod_python-3.2.5b/src/mod_python.so #11 0x284bea7d in python_connection (con=0x8258128) at mod_python.c:1281 #12 0x284bfbd5 in PythonConnectionHandler (con=0x8258128) at mod_python.c:1929 #13 0x08075546 in ap_run_process_connection (c=0x8258128) at connection.c:43 #14 0x08075940 in ap_process_connection (c=0x8258128, csd=0x8258050) at connection.c:176 #15 0x08068971 in child_main (child_num_arg=0) at prefork.c:610 #16 0x08068afb in make_child (s=0x80b5df8, slot=0) at prefork.c:704 #17 0x08068b71 in startup_children (number_to_start=3) at prefork.c:722 #18 0x08068f6e in ap_mpm_run (_pconf=0x80b4018, plog=0x80e0018, s=0x80b5df8) at prefork.c:941 #19 0x0806fd5e in main (argc=5, argv=0xbfbfea90) at main.c:638 Not sure where to look next. Barry
Re: mod_python 3.2.5b available for testing
Barry Pederson wrote: I've got failures that seem to be caused by the tests themselves, but with a bit of tweaking they pass. FreeBSD 6.0 Apache 2.0.55 port built WITH_THREADS=1 Python 2.4.2 DOH! nevermind - just realized I missed this part of Jim's very clear instructions: - Then (as non-root user!) $ cd test $ python test.py - However I get some other errors now: == ERROR: test_connectionhandler (__main__.PerRequestTestCase) -- Traceback (most recent call last): File "test.py", line 1220, in test_connectionhandler f = urllib.urlopen(url) File "/usr/local/lib/python2.4/urllib.py", line 77, in urlopen return opener.open(url) File "/usr/local/lib/python2.4/urllib.py", line 185, in open return getattr(self, name)(url) File "/usr/local/lib/python2.4/urllib.py", line 317, in open_http return self.http_error(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 334, in http_error return self.http_error_default(url, fp, errcode, errmsg, headers) File "/usr/local/lib/python2.4/urllib.py", line 574, in http_error_default return addinfourl(fp, headers, "http:" + url) File "/usr/local/lib/python2.4/urllib.py", line 863, in __init__ addbase.__init__(self, fp) File "/usr/local/lib/python2.4/urllib.py", line 813, in __init__ self.read = self.fp.read AttributeError: 'NoneType' object has no attribute 'read' -- Ran 43 tests in 16.807s FAILED (errors=1) F Stopping Apache... /usr/local/sbin/httpd -k stop -f /home/barryp/mod_python-3.2.5b/test/conf/test.conf == FAIL: testPerRequestTests (__main__.PerInstanceTestCase) -- Traceback (most recent call last): File "test.py", line 1805, in testPerRequestTests self.failUnless(result.wasSuccessful()) AssertionError -- Ran 6 tests in 63.714s FAILED (failures=1) -- Not sure why running under a different userid is causing this - I cleaned out my /tmp of things owned by www. Will keep looking at this. Barry
Re: mod_python 3.2.5b available for testing
I've got failures that seem to be caused by the tests themselves, but
with a bit of tweaking they pass.
FreeBSD 6.0
Apache 2.0.55 port built WITH_THREADS=1
Python 2.4.2
The error_log shows:
--
[Mon Nov 14 19:38:15 2005] [notice] mod_python: Creating 8 session
mutexes based on 256 max processes and 0 max threads.
[Mon Nov 14 19:38:15 2005] [alert] (2)No such file or directory:
getpwuid: couldn't determine user name from uid 4294967295, you probably
need to modify the User directive
[Mon Nov 14 19:38:15 2005] [notice] Apache/2.0.55 (FreeBSD)
mod_python/3.2.5b Python/2.4.2 configured -- resuming normal operations
[Mon Nov 14 19:38:15 2005] [info] Server built: Nov 12 2005 23:05:22
[Mon Nov 14 19:38:15 2005] [debug] prefork.c(956): AcceptMutex: flock
(default: flock)
[Mon Nov 14 19:38:15 2005] [alert] Child 9492 returned a Fatal error...
Apache is exiting!
[Mon Nov 14 19:38:15 2005] [emerg] (2)No such file or directory:
Couldn't initialize cross-process lock in child
[Mon Nov 14 19:38:15 2005] [emerg] (2)No such file or directory:
Couldn't initialize cross-process lock in child
Googling that last message comes up with a suggesting that you specify a
"User" in the http config.
With the attached patch, the tests run httpd with a "User www"
directive, and pass.
Barry
--- mod_python-3.2.5b-old/test/httpdconf.py Tue Sep 13 15:35:57 2005
+++ mod_python-3.2.5b/test/httpdconf.py Mon Nov 14 19:43:07 2005
@@ -264,6 +264,10 @@
def __init__(self, val='Off'):
Directive.__init__(self, self.__class__.__name__, val)
+class User(Directive):
+def __init__(self, val='www'):
+Directive.__init__(self, self.__class__.__name__, val)
+
class VirtualHost(ContainerTag):
def __init__(self, addr, *args):
ContainerTag.__init__(self, self.__class__.__name__, addr, args)
--- mod_python-3.2.5b-old/test/test.py Mon Nov 14 12:09:49 2005
+++ mod_python-3.2.5b/test/test.py Mon Nov 14 19:56:03 2005
@@ -229,6 +229,7 @@
IfModule("!mod_dir.c",
LoadModule("dir_module %s" %
quoteIfSpace(os.path.join(modpath,
"mod_dir.so",
+User("www"),
ServerRoot(SERVER_ROOT),
ErrorLog("logs/error_log"),
LogLevel("debug"),
