Hi Chris, I know what is going on here (it is quite clear in the debugger). I'll take you through it one step at a time in this email, but the issue is complicated, so I'll try to be thorough.
Would you be willing to try the newly-minted 8.0.21, or 8.0.20, to see if it makes a difference? Yes, I would! I just tested/debugged in 8.0.20, and the issue still occurs. I don't see 8.0.21 on the tomcat download site, so I checked out trunk, and the code I believe is responsible for this bug (I'll explain below) hasn't been modified since 8.0.20. Are there any other log messages in either your application's log, the > localhost_*.log, or catalina.out that come *before* this > ClassNotFoundException? Nope. In your original message, did you post the full and complete stack trace? I believe so, but I might as well post updated versions from 8.0.20. 26-Mar-2015 14:38:46.838 SEVERE [http-nio-8443-exec-5] org.apache.catalina.core.StandardContext.listenerStop Exception sending context destroyed event to listener instance of class com.avadyne.TestcaseContextListener java.lang.NoClassDefFoundError: com/avadyne/ThisClassNotFound at com.avadyne.TestcaseContextListener.contextDestroyed(TestcaseContextListener.java:27) at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4775) at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5385) at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3739) at org.apache.catalina.startup.HostConfig.reload(HostConfig.java:1304) at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1236) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1491) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:300) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at org.apache.catalina.manager.ManagerServlet.check(ManagerServlet.java:1460) at org.apache.catalina.manager.ManagerServlet.deploy(ManagerServlet.java:906) at org.apache.catalina.manager.ManagerServlet.doGet(ManagerServlet.java:344) at javax.servlet.http.HttpServlet.service(HttpServlet.java:618) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:613) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: com.avadyne.ThisClassNotFound at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1305) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157) ... 45 more Here is the stack where the ClassLoader gives up on loading the class (8.0.20). The WebResourceSet it would have used to load the class from (in the method StandardRoot.getResourceInternal) is a DirResourceSet pointed to the exploded directory path - e.g. /path/to/tomcat/webapps/ClassNotFoundGenerator/. I can see it in the debugger in the allResources member variable. The path it generates in DirResourceSet.getResource is /path/to/tomcat/webapps/ClassNotFoundGenerator/WEB-INF/classes/com/avadyne/ThisClassNotFound.class - the path is correct. The file doesn't exist, and it returns a new EmptyResource(root, path, f). Daemon Thread [http-nio-8443-exec-31] (Suspended) owns: WebappClassLoader (id=3483) owns: TestcaseContextListener (id=3488) owns: StandardContext (id=3324) owns: HostConfig (id=3323) owns: SecureNioChannel (id=3313) StandardRoot.getResourceInternal(String, boolean) line: 302 Cache.getResource(String, boolean) line: 65 StandardRoot.getResource(String, boolean, boolean) line: 216 StandardRoot.getClassLoaderResource(String) line: 225 WebappClassLoader(WebappClassLoaderBase).findResourceInternal(String, String) line: 2548 WebappClassLoader(WebappClassLoaderBase).findClassInternal(String) line: 2405 WebappClassLoader(WebappClassLoaderBase).findClass(String) line: 854 WebappClassLoader(WebappClassLoaderBase).loadClass(String, boolean) line: 1274 WebappClassLoader(WebappClassLoaderBase).loadClass(String) line: 1157 TestcaseContextListener.contextDestroyed(ServletContextEvent) line: 27 StandardContext.listenerStop() line: 4775 StandardContext.stopInternal() line: 5385 StandardContext(LifecycleBase).stop() line: 232 StandardContext.reload() line: 3739 HostConfig.reload(HostConfig$DeployedApplication) line: 1304 HostConfig.checkResources(HostConfig$DeployedApplication) line: 1236 HostConfig.check(String) line: 1491 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43 Method.invoke(Object, Object...) line: 483 BaseModelMBean.invoke(String, Object[], String[]) line: 300 DefaultMBeanServerInterceptor.invoke(ObjectName, String, Object[], String[]) line: 819 JmxMBeanServer.invoke(ObjectName, String, Object[], String[]) line: 801 ManagerServlet.check(String) line: 1460 ManagerServlet.deploy(PrintWriter, String, ContextName, String, boolean, StringManager) line: 906 ManagerServlet.doGet(HttpServletRequest, HttpServletResponse) line: 344 ManagerServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 618 ManagerServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 725 ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 291 ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206 WsFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 52 ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 239 ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206 SetCharacterEncodingFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 108 ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 239 ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206 StandardWrapperValve.invoke(Request, Response) line: 219 StandardContextValve.invoke(Request, Response) line: 106 BasicAuthenticator(AuthenticatorBase).invoke(Request, Response) line: 613 StandardHostValve.invoke(Request, Response) line: 142 ErrorReportValve.invoke(Request, Response) line: 79 AccessLogValve(AbstractAccessLogValve).invoke(Request, Response) line: 610 StandardEngineValve.invoke(Request, Response) line: 88 CoyoteAdapter.service(Request, Response) line: 516 Http11NioProcessor(AbstractHttp11Processor).process(SocketWrapper<S>) line: 1086 Http11NioProtocol$Http11ConnectionHandler(AbstractProtocol$AbstractConnectionHandler).process(SocketWrapper<S>, SocketStatus) line: 659 Http11NioProtocol$Http11ConnectionHandler.process(SocketWrapper<NioChannel>, SocketStatus) line: 223 NioEndpoint$SocketProcessor.doRun(SelectionKey, NioEndpoint$KeyAttachment) line: 1558 NioEndpoint$SocketProcessor.run() line: 1515 ThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1142 ThreadPoolExecutor$Worker.run() line: 617 TaskThread$WrappingRunnable.run() line: 61 TaskThread(Thread).run() line: 745 Taking a look at Tomcat 8 trunk, this code has undergone a large > refactoring since 8.0.12 so the line numbers aren't correct relative > to your stack trace. I looked in 8.0.12 and the line in question > (WebappClassLoader.java:1320) is at the end of the loadClass method, > when all options for loading the class have been exhausted. It just > gives up and says "sorry, I can't find the class". Yes, that is what I saw in the debugger on 8.0.12. It goes all the way down to findClass, findClassInternal and findResourceInternal before giving up. 8.0.12 takes a slightly different path than 8.0.20, but 8.0.20 still fails to find the class and throws a ClassNotFoundError. It's possible that Tomcat's reload process first notifies the > WebappClassLoader that it shouldn't load any new classes during > shutdown, but that sounds like it would be a recipe for disaster -- > the kind of disaster that you are experiencing, here. That would trigger a ClassNotFoundError in WebappClassLoader.checkStateForClassLoading or checkStateForResourceLoading, right? I can see in the debugger the loadClass(String,boolean) method continues after checkStateForClassLoading. I have a better explanation, though, so keep reading :) In this case, I would expect that the class is actually missing. Are > you sure it's there? Where is it located? At first I was thinking that WCL was objecting to loading the class, > but it's not: it's actually not finding it. I believe I figured this part out in the debugger. When the WebappClassLoader attempts to load the class, the .class file (and the entire exploded directory) is not on the disk. First, let me quote a few frames of the ClassNotFoundError stack trace (once again, 8.0.20). java.lang.NoClassDefFoundError: com/avadyne/ThisClassNotFound ... at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3739) at org.apache.catalina.startup.HostConfig.reload(HostConfig.java:1304) at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1236) ... HostConfig.checkResources:1231 deletes the exploded directory HostConfig.checkResources:1236 calls into StandardContext.reload StandardContext.reload:3739 eventually invokes the ServletContextListeners Once the contextDestroyed listeners are executed, the .class file doesn't exist (HostConfig.checkResources deleted it). I have walked (in the debugger) all the way from HostConfig.checkResources to the SevletContextListener/WebappClassLoader.loadClass call, and there doesn't seem to be any other code that causes trouble. It is definitely unsafe for HostConfig.checkResources to delete the unpacked directory before the context is stopped. However, WebappClassLoader is correct to throw a ClassNotFoundError, since the class really doesn't exist on disk. Now, Tomcat 8 uses a more > complicated resource-loading framework than previous Tomcats, and it's > possible that the resource-loading framework is being taken down > earlier than your code is running, which would make certain classes > /libraries inaccessible. I haven't read enough code to know if that's > the case or not. IMO it shouldn't be the case, and probably isn't > because the smarter Tomcat devs tend to think about things like that > before coding them ;) Okay. IIRC, during ServletContextListener.contextDestroyed, the > WebappClassLoader should be fully-functional. That is, it shouldn't > have complained about anything. I wouldn't blame anyone for not noticing this bug. And, I dream of a fully functional ClassLoader :D The bug requires very specific conditions. Deploy a WAR through the Tomcat manager, and configure it to explode the war, and create a listener that references a class on shutdown that was never referenced prior... Unfortunately, the impact (uncaught java.lang.Error leaving uncleaned resources/threads) is pretty severe. It is also unavoidable if you need to execute lots of code in the shutdown handler, or you need to wait for Threads to stop. I need to do both... > In my first example, the ClassNotFoundError is thrown when > > enabled=false. When enabled=true, the error doesn't occur. > When you say enabled=true, you mean that's what's showing in the > Manager's web interface? Don't worry too much about the enabled flag, it doesn't have anything to do with the actual issue. I was trying to explain why a class wouldn't be loaded during contextInitialized (in this case - when initialization is conditional). The "new" WebappClassLoader shouldn't even be involved, yet, since the > shutdown is evidently not completing successfully, so the reload never > actually gets to the "load" part. Well, yes and no. The WebappClassLoader which throws the ClassNotFoundError is definitely the one on the 'stop' side of the reload. I can tell from the cached classes. The ClassNotFoundError gets caught and logged by StandardContext.listenerStop(), which catches Throwable. Tomcat completes the reload without any issue. When you do the reload(), are you shutting-down a properly-running web > application, or one that failed to load properly in the first place? Yes, the app is up and running just fine. All servlets/listeners are mounted, and no errors were reported during startup. > I actually have reworked my app's top-level initialization layer to > > preload classes on startup. I am considering preloading all > > classes in WEB-INF/classes at server startup. Unfortunately, my > > init layer needs to call into code deeper in the application, which > > may call into bundled JARs (WEB-INF/lib). Including JARs, that > > would be a lot of code to preload. It also shouldn't be necessary. I think something else weird might be > going on. It's a temporary workaround, but it works OK. I get occasional NPEs from the ClassLoader when it needs to load from JARs in WEB-INF/lib, but I can live with that until the Tomcat code is fixed. Actually, Guava's ClassPath API made preloading surprisingly easy.
