Hello, We are running Tomcat 6.0.26 on Windows Server 2008 R2 Enterprise 64 Bit using IIS and isapi redirector version 1.2.30 (just upgraded to 1.2.31). The 6 months or so we have been using Tomcat have been plagued with intermittent periods crashes and instability. Specifically, tomcat requests are resulting in crashes and the application log shows numerous (over 1200) instances of error 1000: Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp: 0x4a5bd0eb Faulting module name: ntdll.dll, version: 6.1.7600.16385, time stamp: 0x4a5be02b Exception code: 0xc0000005 Fault offset: 0x000000000004d174 Faulting process id: 0x1254 Faulting application start time: 0x01cbc19aceedc9cd Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 0d413cc2-2d8e-11e0-92d0-005056340288
We installed debug diag on the machine and opened a case with Microsoft since it seemed like it was a MS problem. After opening a case and sending MS several dump files for analysis we were told: We are failing during a call into isapi_redirect64. It appears to be passing a bad value for a critical section to NTDLL (0x00000000`00000003) which in turn is causing the Access Violation to occur. Here is the complete email response from MS with some of the data from the dumps: All of the dumps show the same issue: Child-SP RetAddr Call Site 00000000`019bd040 00000000`7734d0a2 ntdll!RtlpWaitOnCriticalSection+0xb4 00000000`019bd0f0 00000000`6a6d1c41 ntdll!RtlEnterCriticalSection+0xab 00000000`019bd120 00000000`6a6bd40c isapi_redirect64!jk_md5+0x1c1 00000000`019bd150 00000000`6a6ebbb7 isapi_redirect64!ajp_service+0xb4c 00000000`019bd1e0 00000000`6a6c679a isapi_redirect64!uri_worker_map_open+0x2c7 00000000`019bd240 000007fe`f31bc540 isapi_redirect64!read_registry_init_data+0x89a 00000000`019be520 000007fe`f31c4007 isapi!ProcessIsapiRequest+0x13c 00000000`019be580 000007fe`f31bb421 isapi!W3_ISAPI_HANDLER::DoWork+0x4fb 00000000`019be690 000007fe`f31bb964 isapi!RequestDoWork+0x40d 00000000`019be720 000007fe`f3db2fa7 isapi!CIISHttpModule::OnExecuteRequestHandler+0x1c 00000000`019be760 000007fe`f3db464e iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x233 00000000`019be7a0 000007fe`f3db4465 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x17f 00000000`019be890 000007fe`f3db5e56 iiscore!NOTIFICATION_CONTEXT::CallModules+0x25 00000000`019be8e0 000007fe`f3de51bd iiscore!W3_CONTEXT::DoWork+0x3a6 00000000`019bec00 000007fe`f3de55df iiscore!W3_MAIN_CONTEXT::StartNotificationLoop+0x5d 00000000`019bec30 000007fe`f319551e iiscore!W3_CONTEXT::ExecuteRequest+0x2bf 00000000`019becb0 000007fe`f3191060 filter!RequestDoWork+0x40be 00000000`019bf360 000007fe`f3db6ac1 filter!CIISHttpModule::OnPostAuthenticateRequest+0x20 00000000`019bf3a0 000007fe`f3db464e iiscore!NOTIFICATION_CONTEXT::RequestDoWork+0x20a 00000000`019bf3e0 000007fe`f3db4465 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+0x17f 00000000`019bf4d0 000007fe`f3db593d iiscore!NOTIFICATION_CONTEXT::CallModules+0x25 00000000`019bf520 000007fe`f3db8ede iiscore!W3_CONTEXT::DoWork+0x326 00000000`019bf840 000007fe`f3da10d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1ea 00000000`019bf870 000007fe`f3da109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126 00000000`019bf8d0 000007fe`f4581fba w3dt!OverlappedCompletionRoutine+0x1c 00000000`019bf900 000007fe`f4582024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a 00000000`019bf950 000007fe`f45820a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34 00000000`019bf980 00000000`770ff56d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61 00000000`019bf9b0 00000000`77333281 kernel32!BaseThreadInitThunk+0xd 00000000`019bf9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d We are failing during a call into isapi_redirect64. It appears to be passing a bad value for a critical section to NTDLL (0x00000000`00000003) which in turn is causing the Access Violation to occur. This is a TomCat redirector from the Apache Foundation: 0:004> lmvm isapi_redirect64 start end module name 00000000`6a6b0000 00000000`6a714000 isapi_redirect64 (export symbols) isapi_redirect64.dll Loaded symbol image file: isapi_redirect64.dll Image path: D:\isapiRedirect\isapi_redirect64.dll Image name: isapi_redirect64.dll Timestamp: Thu Sep 09 18:04:53 2010 (4C895A05) CheckSum: 00062701 ImageSize: 00064000 File version: 1.2.30.0 Product version: 1.2.30.0 File flags: 0 (Mask 3F) File OS: 40004 NT Win32 File type: 1.0 App File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Apache Software Foundation ProductName: Apache Tomcat Connectors project InternalName: isapi_redirect-1.2.30 OriginalFilename: isapi_redirect-1.2.30.dll ProductVersion: 1.2.30 FileVersion: 1.2.30 FileDescription: Apache Tomcat IIS Redirector You will need to follow up with the vendor of that product for further details on what is causing the error. ----end MS response----------- We have not been able to figure out what is causing this problem. It is intermittent - we have gone as long as a month between incidents and had it happen 5 or 6 times in a day. Searching the list archives we found several instances of people with similar problems, but none with solutions. We also found an one issue in ASF Bugzilla which appears similar: https://issues.apache.org/bugzilla/show_bug.cgi?id=45063 However, we are using the same identity ("ApplicationPoolIdentity") to run all of our app pools, so that one doesn't quite match. We are also on different version of windows/IIS. We have just upgraded the redirector to version 1.2.31. Based on the change log for that version we don't expect that it will fix the problem but we will monitor and report if it does. Any help would be appreciated.