fengxianjing created HBASE-27327:
------------------------------------
Summary: Class missing at runtime
Key: HBASE-27327
URL: https://issues.apache.org/jira/browse/HBASE-27327
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.3.4
Reporter: fengxianjing
We found that some class cannot be found after the regionserver has been
running for a period of time(more than a month). And more than half of the
machines in our cluster have this problem. Some are still running normally, but
_/rs-status_ cannot be opened，and others have various problems(such as rit,
replication failed, abort failed)
Some exceptions are as follows:
{code:java}
java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl
at
org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:49)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
at
org.apache.hadoop.hbase.http.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
at
org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:52)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
at
org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1495)
{code}
{code:java}
2022-08-24 19:22:52,536 ERROR [RS_CLOSE_REGION-regionserver/xxxx:26020-1]
regionserver.HRegionServer: ***** ABORTING region server
10.x.x.x,26020,1659357427101: Replay of WAL required. Forcing server shutdown
*****
org.apache.hadoop.hbase.DroppedSnapshotException: region:
xxxx:xxxx,xxxxxx,1661298619920.943104cbcf4a74db9896fd4abd051411.
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2906)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2575)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2547)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2538)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1652)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1591)
at
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:114)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker$DeleteResult
at
org.apache.hadoop.hbase.regionserver.querymatcher.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:108)
at
org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.checkDeleted(ScanQueryMatcher.java:209)
at
org.apache.hadoop.hbase.regionserver.querymatcher.MinorCompactionScanQueryMatcher.match(MinorCompactionScanQueryMatcher.java:54)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:627)
at
org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:127)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:69)
at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1067)
at
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2442)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2842)
... 10 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.regionserver.querymatcher.DeleteTracker$DeleteResult
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 19 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
