[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873982#comment-13873982 ]
Hudson commented on HBASE-6873: ------------------------------- SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #78 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/78/]) HBASE-6873. Clean up Coprocessor loading failure handling (apurtell: rev 1558870) * /hbase/branches/0.98/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.98/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java > Clean up Coprocessor loading failure handling > --------------------------------------------- > > Key: HBASE-6873 > URL: https://issues.apache.org/jira/browse/HBASE-6873 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, regionserver > Affects Versions: 0.98.0 > Reporter: David Arthur > Assignee: Andrew Purtell > Priority: Blocker > Fix For: 0.98.0, 0.99.0 > > Attachments: 6873.patch, 6873.patch, 6873.patch, 6873.patch, > 6873.patch, 6873.patch, 6873.patch > > > When registering a coprocessor with a missing dependency, the regionserver > gets stuck in an infinite fail loop. Restarting the regionserver and/or > master has no affect. > E.g., > Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) > that is not included with HBase. > {code} > 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => > 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', > ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking > as FAILED_OPEN in ZK > 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Attempting to transition node > 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to > RS_ZK_REGION_FAILED_OPEN > 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Successfully transitioned node > 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to > RS_ZK_REGION_FAILED_OPEN > 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open > region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b. > 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Attempting to transition node > 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to > RS_ZK_REGION_OPENING > 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Successfully transitioned node > 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to > RS_ZK_REGION_OPENING > 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => > 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', > ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} > 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor > config now ... > 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class > com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded > from a file - file:/path/to/my-coproc.jar. > 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of > region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting > to roll back the global memstore size. > java.lang.IllegalStateException: Could not instantiate a region instance. > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592) > ... 7 more > Caused by: java.lang.NoClassDefFoundError: > kafka/common/NoBrokersForPartitionException > at java.lang.Class.getDeclaredConstructors0(Native Method) > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) > at java.lang.Class.getConstructor0(Class.java:2699) > at java.lang.Class.newInstance0(Class.java:326) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254) > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126) > at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417) > ... 11 more > Caused by: java.lang.ClassNotFoundException: > kafka.common.NoBrokersForPartitionException > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 21 more > 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => > 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', > ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking > as FAILED_OPEN in ZK > 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Attempting to transition node > 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to > RS_ZK_REGION_FAILED_OPEN > 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Successfully transitioned node > 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to > RS_ZK_REGION_FAILED_OPEN > 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open > region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b. > 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Attempting to transition node > 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to > RS_ZK_REGION_OPENING > 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: > regionserver:60020-0x139f43af2a70043 Successfully transitioned node > 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to > RS_ZK_REGION_OPENING > 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => > 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', > ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} > 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor > config now ... > 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class > com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded > from a file - file:/path/to/my-coproc.jar. > 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of > region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting > to roll back the global memstore size. > java.lang.IllegalStateException: Could not instantiate a region instance. > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592) > ... 7 more > Caused by: java.lang.NoClassDefFoundError: > kafka/common/NoBrokersForPartitionException > at java.lang.Class.getDeclaredConstructors0(Native Method) > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) > at java.lang.Class.getConstructor0(Class.java:2699) > at java.lang.Class.newInstance0(Class.java:326) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254) > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126) > at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417) > ... 11 more > Caused by: java.lang.ClassNotFoundException: > kafka.common.NoBrokersForPartitionException > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 21 more > {code} > Ad infinitum. > It seems that upon failing to open a region after adding a coprocessor, that > coprocessor should be unregister or at least disabled/blacklisted. -- This message was sent by Atlassian JIRA (v6.1.5#6160)