[ https://issues.apache.org/jira/browse/HBASE-27327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
fengxianjing resolved HBASE-27327. ---------------------------------- Resolution: Not A Bug > Class missing at runtime > ------------------------ > > Key: HBASE-27327 > URL: https://issues.apache.org/jira/browse/HBASE-27327 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 2.3.4 > Reporter: fengxianjing > Priority: Critical > > We found that some class cannot be found after the regionserver has been > running for a period of time(more than a month). And more than half of the > machines in our cluster have this problem. Some are still running normally, > but _/rs-status_ cannot be opened,and others have various problems(such as > rit, replication failed, abort failed) > Some exceptions are as follows: > {code:java} > java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl > at > org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:49) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) > at > org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.hbase.http.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:52) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1495) > {code} > {code:java} > 2022-08-24 19:22:52,536 ERROR [RS_CLOSE_REGION-regionserver/xxxx:26020-1] > regionserver.HRegionServer: ***** ABORTING region server > 10.x.x.x,26020,1659357427101: Replay of WAL required. Forcing server shutdown > ***** > org.apache.hadoop.hbase.DroppedSnapshotException: region: > xxxx:xxxx,xxxxxx,1661298619920.943104cbcf4a74db9896fd4abd051411. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2906) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2575) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2547) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2538) > at > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1652) > at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1591) > at > org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:114) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker$DeleteResult > at > org.apache.hadoop.hbase.regionserver.querymatcher.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:108) > at > org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.checkDeleted(ScanQueryMatcher.java:209) > at > org.apache.hadoop.hbase.regionserver.querymatcher.MinorCompactionScanQueryMatcher.match(MinorCompactionScanQueryMatcher.java:54) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:627) > at > org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:127) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:69) > at > org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1067) > at > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2442) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2842) > ... 10 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.regionserver.querymatcher.DeleteTracker$DeleteResult > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 19 more > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)