RE: mod_perl2: nmake test crashes apache
Greetings.
[...]
do you have a simple test case to reproduce the problem?
No, I do not. I hope I will be able get/build/test the latest CVS this week,
and perhaps I'll be able to work on making the problem
reproducible/debuggable which may well mean
I'll have to build a debugperl.
Some facts of the threads allocation are making me curious though
(euphemistic to say I don't really understand them). I am running a test
with a counter:
[...]
foreach my $count (0..20) {
$r-puts(lib$count/b/li\n);
sleep(1);
}
[...]
which I alias to two handlers (say /count1/ and /count2/).
If I then call it from the same browser in three separate frames like this:
| Frame 1 | Frame 2 | Frame 3
| htpp://localhost/count1/ | htpp://localhost/count2/ |
htpp://localhost/count1/
The execution order turns out to be: 1+2 and *then* 3. It looks like a
thread is allocated to this (client,handler) pair, so Frame 1 and 3 are
running in the same thread, separate from the thread that's running 2.
Is this reading correct, and is this behavior amenable to configuration?
(And is this a modperl question or an Apache question?)
iib) I then set out to build a debug version. That ain't
easy I finally
this has been fixed in cvs, MP_DEBUG=1 should do the right things now.
Great - I'll try it soon.
Cheers,
alf
RE: mod_perl2: nmake test crashes apache
On Tue, 21 May 2002, Alessandro Forghieri wrote: The execution order turns out to be: 1+2 and *then* 3. It looks like a thread is allocated to this (client,handler) pair, so Frame 1 and 3 are running in the same thread, separate from the thread that's running 2. there should never be multiple requests being served concurrently by the same thread. if this is happening, this is a major bug in the winnt mpm. the same thread may handle request #1, then #3 when it is done serving #1, but not both at the same time.
RE: mod_perl2: nmake test crashes apache
On Tue, 14 May 2002, Alessandro Forghieri wrote: ii) It does however crash on my testbed app (which runs as standard CGI, FastCGI and moperl1). The crash itself appears to happen when a number of nearly-simultaneous requests arrive to the server and is fatal to modperl (but the static-serving part of apache appears to survive) do you have a simple test case to reproduce the problem? iib) I then set out to build a debug version. That ain't easy I finally this has been fixed in cvs, MP_DEBUG=1 should do the right things now.
RE: mod_perl2: nmake test crashes apache
Greetings. A few updates... i) modperl-2.0_20020514042137.tar.gz Now tests cleanly on: WinNT SP6/MSDEV 6 SP3/AS perl 5.6.1 b630/httpd-2.0_20020506161223.tar.gz ii) It does however crash on my testbed app (which runs as standard CGI, FastCGI and moperl1). The crash itself appears to happen when a number of nearly-simultaneous requests arrive to the server and is fatal to modperl (but the static-serving part of apache appears to survive) iia) On non-debug modperl in post-mortem examination within msdev I have been able to observe a seemingly incorrect r-pool thingy (casted to something else within table related something_palloc) triggering the segfault. iib) I then set out to build a debug version. That ain't easy I finally got it with semiautomatic Makefile mangling: perl Makefile.PL MP_INST_APACHE2=1 MP_AP_PREFIX=D:\Apache2 MP_GENERATE_XS=1 MP_DEBUG=1 MP_TRACE=1 find . -name Makefile -print | xargs perl -spi.bak -e s/-O1//g;s/-DNDEBUG//g;s/-release/-DEBUG -PDB:vc60.pdb/g; nmake [Explanation: the generated Makefiles have sequences of -O1 -Od -O1 *AND* the linkline misses the critical -DEBUG option. the PDB and -DNDEBUG things may not be needed.] After this the crash location moves (darn) and the debugger brings me in a location with nothing obviously wrong in sight. It is wedged on modperl_get_perl_module_config, called from XS_Apache_RequestRec_send_cgi_header etc. etc. Note: After switching to the debug version, I have started seeing apache errors - Your script did not blah blah - that were not there before, coupled to diagnostics from ModPerl::Registry complaining about not being able to call send_cgi_header on an undefined value. This may be significant. I will sendo more info when I have it... Cheers, alf
RE: mod_perl2: nmake test crashes apache
Greetings
Doug McEachern wrote:
[...]
avoiding use of PL_sv_no has cured all segfaults on win32.
the fixes have
been checked into cvs.
there are still a few tests that fail, but none that trigger
a segfault.
With the latest nodperl CVS snapshot the tally of the tests is much
improved.
There is a residual crash at dir_config.t (WinNT SP6 - MS Visual
Studio 6 SP3).
The debugger shows apache (debug version) going through the
ap_invoke_handler sequence before dumping me in the middle of
mod_perl's assembly. When it segfaults the stack (not cleared, this
time) shows a call into modperl the a trip into perl56.dll followed by
a call into request_util then modperl again (where it dies).
I re-ran the test suite without dir_config and all crashes go away,
though a few tests still fail.
At this point, I made a debug version of mod_perl (now possible - the
compiler warns about replacements of /O1 with /Od) and - lo and
behold! - the crash disappeared (darn). Also a few of the previously
failing tests now succeed
The tallies (with/without dir_config.t, and nmake test DEBUG) are
appendend below. Note that, in the first one, all tests coming after
dir_config fail ('coz apache is deeply unsettled, I think)
Cheers,
alf
Using the following snapshot:
modperl-2.0_20020513043636.tar.gz
httpd-2.0_20020506161223.tar.gz
apr-util_20020506162813.tar.gz
apr_20020506162759.tar.gz
-- output of nmake test --
D:\Apache2/bin/Apache.exe -d D:/builds/modperl-2.0/t -f D:/builds/mo
/conf/httpd.conf -DAPACHE2 -DPERL_USEITHREADS
using Apache/2.0.37-dev (winnt MPM)
waiting for server to start: 00:00[Mon May 13 11:42:50 2002] [info] 1
modules loaded
[Mon May 13 11:42:50 2002] [info] 5 APR:: modules loaded
[Mon May 13 11:42:50 2002] [info] base server + 6 vhosts ready to run
[Mon May 13 11:42:50 2002] [info] 11 Apache:: modules loaded
[Mon May 13 11:42:50 2002] [info] 5 APR:: modules loaded
[Mon May 13 11:42:50 2002] [info] base server + 6 vhosts ready to run
END in modperl_extra.pl, pid=309
END in modperl_extra.pl, pid=309
END in modperl_extra.pl, pid=309
END in modperl_extra.pl, pid=309
END in modperl_extra.pl, pid=309
END in modperl_extra.pl, pid=309
END in modperl_extra.pl, pid=309
waiting for server to start: ok (waited 1 secs)
server localhost:8529 started
server localhost:8530 listening (TestDirective::perlmodule)
server localhost:8531 listening (TestDirective::perlrequire)
server localhost:8532 listening (TestProtocol::echo)
server localhost:8533 listening (TestProtocol::echo_filter)
server localhost:8534 listening (TestProtocol::eliza)
server localhost:8535 listening (TestFilter::input_msg)
apache\cgihandlerok
apache\compatok
apache\compat2...ok
apache\conftree..ok
apache\constants.ok
apache\post..ok
apache\read..ok
apache\scanhdrs..ok
apache\subprocessskipped: perl 5.7.3, cannot find module 'A
apache\write.ok
api\access...ok
api\aplogok
api\conn_rec.ok
api\lookup_uri...ok
api\lookup_uri2..ok
api\module...FAILED test 3
Failed 1/13 tests, 92.31% okay
api\r_subclass...ok
api\request_rec..ok
api\response.ok
api\rutilok
api\send_fd..ok
api\sendfile.ok
api\server_rec...ok
api\server_util..ok
api\uri..ok
apr\base64...ok
apr\constantsok
apr\date.ok
apr\netlib...ok
apr\perlio...skipped: iolayers is not available with this
Perl
apr\pool.ok
apr\string...ok
apr\tableok
apr\util.ok
apr\uuid.ok
directive\envFAILED tests 1-3
Failed 3/4 tests, 25.00% okay
directive\perlmodule.ok
directive\perlrequireok
directive\setupenv...ok
filter\api...ok
filter\buckets...ok
filter\input_bodyok
filter\input_msg.ok
filter\lcok
filter\reverse...ok
hooks\access.ok
hooks\authen.ok
hooks\authz..ok
hooks\fixup..ok
hooks\headerparser...ok
hooks\init...ok
hooks\trans..ok
modperl\dir_config...FAILED before any test output arrived
modperl\endavFAILED before any test output arrived
modperl\env..FAILED before any test output arrived
modperl\exit.FAILED before any test output arrived
modperl\getc.FAILED tests 1-2
Failed 2/2 tests, 0.00% okay
modperl\pnotes...FAILED before any test output arrived
modperl\printFAILED before any test output arrived
modperl\printf...FAILED before any test output arrived
modperl\readline.FAILED tests 1-2
Failed 2/2 tests, 0.00% okay
modperl\sameinterp...must pass an interpreter id to work with at
odperl-2.0/Apache-Test/lib/Apache/TestRequest.pm line
RE: mod_perl2: nmake test crashes apache
On Mon, 13 May 2002, Alessandro Forghieri wrote:
There is a residual crash at dir_config.t (WinNT SP6 - MS Visual
Studio 6 SP3).
all tests pass for me with both 5.6.1 and bleedperl, httpd-2.0 from cvs on
xp with msdev 6.0. i also tried 5.6.1 with no debug symbols, still all
pass. would help to know which test causes the crash. you could try
turning on autoflush in the test (patch below), start the server with
't/TEST -start' and run it through a browser by opening:
http://localhost:8529/TestModperl::dir_config
Index: t/response/TestModperl/dir_config.pm
===
RCS file: /home/cvs/modperl-2.0/t/response/TestModperl/dir_config.pm,v
retrieving revision 1.3
diff -u -r1.3 dir_config.pm
--- t/response/TestModperl/dir_config.pm11 Apr 2002 11:08:44 - 1.3
+++ t/response/TestModperl/dir_config.pm13 May 2002 16:27:03 -
-16,6 +16,8
sub handler {
my $r = shift;
+$| = 1;
+
plan $r, tests = 12;
#Apache::RequestRec::dir_config tests
Re: mod_perl2: nmake test crashes apache
On Mon, 13 May 2002, Alessandro Forghieri wrote: I think apache may be (sometimes?) picking up whatever mod_perl.so is under SERVER_ROOT/modules during the test run. this is fixed in cvs now. So, disregard my previous message, my failed line is now: as is this.
RE: mod_perl2: nmake test crashes apache
Greetings. [...] Apache builds and installs fine, mod_perl2 builds fine but running `nmake test` crashes the instance of apache2 created by the TEST script. [...] I have exactly the same behavior with May 6 snapshots (Visual Studio 6SP3, WinNT SP6) - quoting myself from a different thread: [RE: mod_perl failing on win32 (CVS snapshots)] [...] Yup. My 'nmake test' now segfaults at conftree.t, with an access violation somewhere in perl56.dll By running restricted test suites, I found out that the culprit appears to hide somewhere in compat2.t, i.e the sequence: perl -Mblib t\test t/apache/compat2.t t/apache/conftree.t wedges, whereas: perl -Mblib t\test t/apache/cgihandler.t t/apache/compat.t t/apache/conftree.t perl -Mblib t\test t/apache/cgihandler.t t/apache/conftree.t perl -Mblib t\test t/apache/compat.t t/apache/conftree.t are clean. (But note that for instance: perl -Mblib t\test t/apache/compat2.t t/apache/post.t also works. Combinatorial explosion, anyone?) [...] My test output is exactly the same as yours down to the memory addresses in the segfault dialog box. From hints in the above mentioned thread, looks like people running XP are experiencing fewer or different problems in the test suite. Cheers, alf
Re: mod_perl2: nmake test crashes apache
By running restricted test suites, I found out that the culprit appears to hide somewhere in compat2.t, i.e the sequence: perl -Mblib t\test t/apache/compat2.t t/apache/conftree.t wedges, whereas: perl -Mblib t\test t/apache/cgihandler.t t/apache/compat.t t/apache/conftree.t perl -Mblib t\test t/apache/cgihandler.t t/apache/conftree.t perl -Mblib t\test t/apache/compat.t t/apache/conftree.t are clean. (But note that for instance: perl -Mblib t\test t/apache/compat2.t t/apache/post.t also works. Combinatorial explosion, anyone?) [...] My test output is exactly the same as yours down to the memory addresses in the segfault dialog box. From hints in the above mentioned thread, looks like people running XP are experiencing fewer or different problems in the test suite. Hi about combinatorial I think not only compat2 is involved here in test suites I have ran on win2k (i have the same problem mentionned) with apache 2.035 or 2.036 modeperl.1.99-02-dev here is a resume of previous post with perl t/test dirtest : dirtest=api: api/access :ok api/aplog : apache segfault dirtest=apr apr/base34 :ok apr/constants : ok apr/date apache segfault dirtest=apache apache/cgihandler : ok apache/compat: ok apache/compat2:ok apache/conftree : apache segfault dirtest=directive directive/env failed 1-3, 4 ok directive/prelmodule : ok directive/perlrequire : ok directive/setupenv : apache segfault dirtest=filter all tests succesfull dirtest=hooks hooks/access : ok hooks/authen : ok 2/4 and 3 segfault dirtest=modperl modperl/dirconfig : failed tests 6-12 failed 7/12 tests (this reminds me my previous post about dir_config not working) and apache segfaults dirtest=protocol protocol/echo ok protocol/filter ok protocol/eliza skipped dirtest=module modules/cgi ok 2/5 and apache segfaults... - also running tests with perl t\test -start-httpd perl -Mblib t\test t/apache/conftree.t perl -Mblib t\test t/apache/compat2.t segfaults at compat2 perl t\test -start-httpd perl -Mblib t\test t/apache/conftree.t perl -Mblib t\test t/apache/compat.t segfaults at compat perl t\test -start-httpd perl -Mblib t\test t/apache/write.t perl -Mblib t\test t/apache/compat.t perl -Mblib t\test t/apache/compat2.t perl -Mblib t\test t/apache/conftree.t segfault at conftree perl t\test -start-httpd perl -Mblib t\test t/apache/compat.t perl -Mblib t\test t/apache/conftree.t perl -Mblib t\test t/apache/compat2.t segfault at compat2 perl t\test -start-httpd perl -Mblib t\test t/apache/compat2.t perl -Mblib t\test t/apache/conftree.t segfault at conftree --- also all test run individually perl t/test some_test_alone pass EXCEPT : modules/cgi produces apache segfault modperl/dir_config fails with message : error examine t\logs\error_log, which log says the well known error with dir_config : [Mon May 06 21:38:18 2002] [error] [client 62.147.163.25] Attempt to free unreferenced scalar at C:\modperl-2.0\t\response/TestModperl/dir_config.pm line 71. best regards pascal
RE: mod_perl2: nmake test crashes apache
Greetings, [...] Hi about combinatorial I think not only compat2 is involved here in test suites I have ran [...] Wow. this is great detective work you have done Pascal. I have tried to go deeper and actually debug the thing. However, while I can get a debug version of apache I am stumped on a few counts: o) mod_perl does not compile with MP_DEBUG=1 (complains about an undefined _MP_debug_level) - and I doubt it is doing THE RIGHT THING anyways, because it adds a truely gcc-ish '-g' flag to the command line... o) mod_perl does not build against 5.007003 bleadperl (I wanted to test whether bleadperl had the same problem) o) the crash erases the call stack so all I can do is staring at mov edi,dword ptr [esi] Not very useful. And for the Curioser and curioser! department: I happened to run the test suite with a shell anvironment containing DEBUG=0 and TRACE=0 (ask not why...). Under these circumstances the test suite spins endlessly on conftree.t (rather than crashing). Folklore, perhaps, and hardly an improvement, but I am wondering who is checking those two variables? If I remove them from my environment, I get my old friend the crash back. Cheers, alf
Re: mod_perl2: nmake test crashes apache
Alessandro Forghieri wrote: about combinatorial I think not only compat2 is involved here in test suites I have ran Wow. this is great detective work you have done Pascal. Actually you didn't have to do the detective work. Apache::Test comes with a detective of its own. Just run t/SMOKE and it'll find all the combinations that fail. Though I cannot help you on WinFU :( Hopefully Randy or Doug will hear you. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_perl2: nmake test crashes apache
On Fri, 10 May 2002, Stas Bekman wrote: Alessandro Forghieri wrote: about combinatorial I think not only compat2 is involved here in test suites I have ran Wow. this is great detective work you have done Pascal. Actually you didn't have to do the detective work. Apache::Test comes with a detective of its own. Just run t/SMOKE and it'll find all the combinations that fail. Though I cannot help you on WinFU :( Hopefully Randy or Doug will hear you. I haven't been able to come up with much more either ... Running individual tests as, eg, perl -Mblib t/TEST t/apache/post.t shows most pass (some, like dir_config.t, partially fail, but the error log, or running in verbose mode, shows why). However, when running groups of tests, like nmake test or perl -Mblib t/TEST t/apache things hang, or crash, after about the 2nd or 3rd test in the group. It's hard to see which particular one's at fault, though, as running the tests randomly: perl -Mblib t/TEST t/apache -order=random still hangs after about the 2nd or 3rd test, whichever one that happens to be. Running in verbose mode shows that, when a test hangs, it doesn't even get to the 1st subtest. I tried running the tests by, rather than using Win32::Process, having Apache-Test install and start a service, but there was no difference in the results. As well, this seems to be a problem particular to the mod_perl tests, as such behaviour doesn't arise with the httpd-test suite in testing pure httpd-2. However, in some limited random manual testing, I haven't seen an analogous problem outside of the Apache-Test framework - does anyone have such an example? best regards, randy
Re: mod_perl2: nmake test crashes apache
the issue with all segfaults on win32 is related to the use of the internal perl variable PL_sv_no. not sure what the real problem is, but avoiding use of PL_sv_no has cured all segfaults on win32. the fixes have been checked into cvs. there are still a few tests that fail, but none that trigger a segfault.
Re: mod_perl2: nmake test crashes apache
* Doug MacEachern wrote: the issue with all segfaults on win32 is related to the use of the internal perl variable PL_sv_no. not sure what the real problem is, but avoiding use of PL_sv_no has cured all segfaults on win32. the fixes have been checked into cvs. I now get apache\cgihandler...ok apache\compat...ok apache\compat2..ok apache\conftree.ok apache\constantsok apache\post.ok apache\read.ok apache\scanhdrs.ok apache\subprocess...skipped: perl 5.7.3 apache\writeok api\access..ok api\aplog...ok api\conn_recok api\lookup_uri..ok api\lookup_uri2.ok api\module..FAILED test 2 Failed 1/13 tests, 92.31% okay api\r_subclass..ok api\request_rec.ok api\responseok api\rutil...ok api\send_fd.ok api\sendfileok api\server_rec..ok api\server_util.ok api\uri.ok apr\base64..ok apr\constants...ok apr\dateok apr\netlib..FAILED test 3 Failed 1/3 tests, 66.67% okay apr\perlio..skipped: iolayers is not available with this version of Perl apr\poolok apr\string..ok apr\table...ok apr\utilok apr\uuidok directive\env...FAILED tests 1-3 Failed 3/4 tests, 25.00% okay directive\perlmodule.ok directive\perlrequire.ok directive\setupenv..ok filter\api..ok filter\buckets..ok filter\input_body...ok filter\input_msgok filter\lc...ok filter\reverse..ok hooks\accessok hooks\authenok hooks\authz.ok hooks\fixup.ok hooks\headerparser..ok hooks\init..ok hooks\trans.ok modperl\dir_config..FAILED tests 6-12 Failed 7/12 tests, 41.67% okay modperl\endav...ok modperl\env.ok modperl\exitok modperl\getcFAILED test 2 Failed 1/2 tests, 50.00% okay modperl\pnotes..ok modperl\print...ok modperl\printf..ok modperl\readlineok modperl\sameinterp..ok modules\cgi.ok modules\cgiupload...ok modules\include.ok protocol\echo...ok protocol\echo_filter.ok protocol\eliza..skipped: cannot find module 'Chatbot::Eliza' Failed Test Status Wstat Total Fail Failed List of Failed api\module.t 131 7.69% 2 apr\netlib.t 31 33.33% 3 directive\env.t43 75.00% 1-3 modperl\dir_config.t 127 58.33% 6-12 modperl\getc.t 21 50.00% 2 3 tests skipped. !!! : error running tests (please examine C:t\logs\error_log) NMAKE : fatal error U1077: 'C:\Perl\bin\perl.exe' : return code '0x1' Stop. Where error_log is [Sat May 11 03:14:49 2002] [info] 12 Apache:: modules loaded [Sat May 11 03:14:49 2002] [info] 5 APR:: modules loaded [Sat May 11 03:14:49 2002] [info] base server + 6 vhosts ready to run tests [Sat May 11 03:14:49 2002] [info] 11 Apache:: modules loaded [Sat May 11 03:14:49 2002] [info] 5 APR:: modules loaded [Sat May 11 03:14:49 2002] [info] base server + 6 vhosts ready to run tests [Sat May 11 03:14:50 2002] [notice] Child 2552: Child process is running [Sat May 11 03:14:50 2002] [notice] Child 2552: Acquired the start mutex. [Sat May 11 03:14:50 2002] [notice] Child 2552: Starting 10 worker threads. [Sat May 11 03:15:02 2002] [info] [client 192.168.0.128] TestAPI::aplog test in progress [Sat May 11 03:15:02 2002] [debug] C:\b\modperl-2.0\t\response/TestAPI/aplog.pm(43): log_serror test ok [Sat May 11 03:15:02 2002] [debug] C:\b\modperl-2.0\t\response/TestAPI/aplog.pm(46): (20007)No time was provided and one was required.: log_serror test 2 ok [Sat May 11 03:15:03 2002] [debug] C:\b\modperl-2.0\t\response/TestAPI/aplog.pm(49): [client 192.168.0.128] log_rerror test ok [Sat May 11 03:15:03 2002] [error] $r-log_error test ok [Sat May 11 03:15:03 2002] [error] $s-log_error test ok [Sat May 11 03:15:03 2002] [debug] C:\b\modperl-2.0\t\response/TestAPI/aplog.pm(63): TestAPI::aplog test done [Sat May 11 03:15:33 2002] [error] [client 192.168.0.128] File does not exist: C:/b/modperl-2.0/t/htdocs/nope [Sat May 11 03:15:34 2002] [error] [client 192.168.0.128] Attempt to free unreferenced scalar at C:\b\modperl-2.0\t\response/TestModperl/dir_config.pm line 71. [Sat May 11 03:15:37 2002] [error] [client 192.168.0.128] Attempt to free unreferenced scalar at C:\b\modperl-2.0\t\response/TestModperl/getc.pm line 23. there are still a few tests that fail, but none that trigger a segfault. Exactly. Thanks.
