Hello,
We are running Tomcat 6.0.26 on Windows Server 2008 R2 Enterprise 64 Bit using
IIS and isapi redirector version 1.2.30 (just upgraded to 1.2.31). The 6
months or so we have been using Tomcat have been plagued with intermittent
periods crashes and instability. Specifically, tomcat requests are resulting
in crashes and the application log shows numerous (over 1200) instances of
error 1000:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp:
0x4a5bd0eb
Faulting module name: ntdll.dll, version: 6.1.7600.16385, time stamp: 0x4a5be02b
Exception code: 0xc0000005
Fault offset: 0x000000000004d174
Faulting process id: 0x1254
Faulting application start time: 0x01cbc19aceedc9cd
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 0d413cc2-2d8e-11e0-92d0-005056340288
We installed debug diag on the machine and opened a case with Microsoft since
it seemed like it was a MS problem. After opening a case and sending MS
several dump files for analysis we were told:
We are failing during a call into isapi_redirect64. It appears to be passing a
bad value for a critical section to NTDLL (0x00000000`00000003) which in turn
is causing the Access Violation to occur.
Here is the complete email response from MS with some of the data from the
dumps:
All of the dumps show the same issue:
Child-SP RetAddr Call Site
00000000`019bd040 00000000`7734d0a2 ntdll!RtlpWaitOnCriticalSection+0xb4
00000000`019bd0f0 00000000`6a6d1c41 ntdll!RtlEnterCriticalSection+0xab
00000000`019bd120 00000000`6a6bd40c isapi_redirect64!jk_md5+0x1c1
00000000`019bd150 00000000`6a6ebbb7 isapi_redirect64!ajp_service+0xb4c
00000000`019bd1e0 00000000`6a6c679a isapi_redirect64!uri_worker_map_open+0x2c7
00000000`019bd240 000007fe`f31bc540
isapi_redirect64!read_registry_init_data+0x89a
00000000`019be520 000007fe`f31c4007 isapi!ProcessIsapiRequest+0x13c
00000000`019be580 000007fe`f31bb421 isapi!W3_ISAPI_HANDLER::DoWork+0x4fb
00000000`019be690 000007fe`f31bb964 isapi!RequestDoWork+0x40d 00000000`019be720
000007fe`f3db2fa7 isapi!CIISHttpModule::OnExecuteRequestHandler+0x1c
00000000`019be760 000007fe`f3db464e
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x233
00000000`019be7a0 000007fe`f3db4465
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x17f
00000000`019be890 000007fe`f3db5e56
iiscore!NOTIFICATION_CONTEXT::CallModules+0x25
00000000`019be8e0 000007fe`f3de51bd iiscore!W3_CONTEXT::DoWork+0x3a6
00000000`019bec00 000007fe`f3de55df
iiscore!W3_MAIN_CONTEXT::StartNotificationLoop+0x5d
00000000`019bec30 000007fe`f319551e iiscore!W3_CONTEXT::ExecuteRequest+0x2bf
00000000`019becb0 000007fe`f3191060 filter!RequestDoWork+0x40be
00000000`019bf360 000007fe`f3db6ac1
filter!CIISHttpModule::OnPostAuthenticateRequest+0x20
00000000`019bf3a0 000007fe`f3db464e
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x20a
00000000`019bf3e0 000007fe`f3db4465
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x17f
00000000`019bf4d0 000007fe`f3db593d
iiscore!NOTIFICATION_CONTEXT::CallModules+0x25
00000000`019bf520 000007fe`f3db8ede iiscore!W3_CONTEXT::DoWork+0x326
00000000`019bf840 000007fe`f3da10d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1ea
00000000`019bf870 000007fe`f3da109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
00000000`019bf8d0 000007fe`f4581fba w3dt!OverlappedCompletionRoutine+0x1c
00000000`019bf900 000007fe`f4582024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a
00000000`019bf950 000007fe`f45820a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
00000000`019bf980 00000000`770ff56d
w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
00000000`019bf9b0 00000000`77333281 kernel32!BaseThreadInitThunk+0xd
00000000`019bf9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
We are failing during a call into isapi_redirect64. It appears to be passing a
bad value for a critical section to NTDLL (0x00000000`00000003) which in turn
is causing the Access Violation to occur.
This is a TomCat redirector from the Apache Foundation:
0:004> lmvm isapi_redirect64
start end module name
00000000`6a6b0000 00000000`6a714000 isapi_redirect64 (export symbols)
isapi_redirect64.dll
Loaded symbol image file: isapi_redirect64.dll
Image path: D:\isapiRedirect\isapi_redirect64.dll
Image name: isapi_redirect64.dll
Timestamp: Thu Sep 09 18:04:53 2010 (4C895A05)
CheckSum: 00062701
ImageSize: 00064000
File version: 1.2.30.0
Product version: 1.2.30.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Apache Software Foundation
ProductName: Apache Tomcat Connectors project
InternalName: isapi_redirect-1.2.30
OriginalFilename: isapi_redirect-1.2.30.dll
ProductVersion: 1.2.30
FileVersion: 1.2.30
FileDescription: Apache Tomcat IIS Redirector
You will need to follow up with the vendor of that product for further details
on what is causing the error.
----end MS response-----------
We have not been able to figure out what is causing this problem. It is
intermittent - we have gone as long as a month between incidents and had it
happen 5 or 6 times in a day. Searching the list archives we found several
instances of people with similar problems, but none with solutions. We also
found an one issue in ASF Bugzilla which appears similar:
https://issues.apache.org/bugzilla/show_bug.cgi?id=45063
However, we are using the same identity ("ApplicationPoolIdentity") to run all
of our app pools, so that one doesn't quite match. We are also on different
version of windows/IIS.
We have just upgraded the redirector to version 1.2.31. Based on the change
log for that version we don't expect that it will fix the problem but we will
monitor and report if it does.
Any help would be appreciated.
