Hello all, Thank you all very much for trying to help! I am too ashamed to share my *stupid* mistake.
Just now, I had a time to dig more, so I did the remote debugging again. Even though the phoenix-core jar was in the classpath of the region server, CoprocessorHost was not able to load MetaDataEndpointImpl.class. Holy cow, the unthinkable was happening right in front of my eyes. WTH? And I remembered what my mama told me, 'Computers don't lie'. So, I had to think back what I had done wrong in my past, and the eventual enlightenment came through. 'Let's unjar the jar.' As you expected, that just did not work with error saying 'Dude, this file's broken as your mind.' OUCH, I uploaded my jar to FTP without using 'bin' option, the obvious 'bin'! I am really sorry, folks. It just works great! (even with phoenix built with 0.94.16 on 0.94.15 HBase since it only depends on it) Thanks, Haebin P.S. Please, check your jar file first when phoenix doesn't work. Don't be like me. :( 2014-03-03 7:16 GMT+09:00 Abe Weinograd <[email protected]>: > Frederick, > > Were you able to get this to work? > > Abe > > > On Sun, Mar 2, 2014 at 10:58 AM, Stan Campbell > <[email protected]>wrote: > >> You mean 0.94.15 and .16 right? No, those are not compatible. >> >> Sounds like maybe you're using CDH? You can easily upgrade 15 to 16 by >> just replacing the HBase jars. >> On Mar 2, 2014 1:05 AM, "Frederick Haebin Na" <[email protected]> >> wrote: >> >>> Oh shucks, >>> Never mind about my solution. >>> Still not working. >>> >>> Will get back to the thread after making it working. >>> >>> Sorry, folks. >>> >>> >>> 2014-03-02 11:47 GMT+09:00 Frederick Haebin Na <[email protected]>: >>> >>>> Hey James, >>>> >>>> I figured out the source of the problem by remote debugging HBase & >>>> Phoenix. >>>> It turned out that I am using HBase version 0.9.15 and the >>>> 3.0.0-SNAPSHOT depends on 0.9.16. >>>> >>>> So, I built the Phoenix with HBase 0.9.15 by modifying the root pom.xml >>>> and it worked just fine. >>>> Hope this helped for all the other folks with similar problems. >>>> >>>> Best, >>>> Haebin >>>> >>>> >>>> >>>> >>>> 2014-03-02 10:05 GMT+09:00 James Taylor <[email protected]>: >>>> >>>> Try manually disabling and dropping the SYSTEM.TABLE and/or >>>>> SYSTEM.CATALOG from the HBase shell. Make sure you only have one phoenix >>>>> jar in the hbase lib dir and make sure it's not the one with "client" in >>>>> the name. >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> >>>>> On Sat, Mar 1, 2014 at 4:50 PM, Frederick Haebin Na < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Stan, >>>>>> >>>>>> Thank you for your suggestion. >>>>>> Yet, I tried all the stuffs recommend here. >>>>>> >>>>>> https://groups.google.com/forum/#!topic/phoenix-hbase-user/gvkHbjZKmtU >>>>>> >>>>>> He had the same problem, but didn't mention how he solved he. Sigh :( >>>>>> >>>>>> Best, >>>>>> Haebin >>>>>> >>>>>> >>>>>> >>>>>> 2014-03-01 2:51 GMT+09:00 Stan Campbell <[email protected]>: >>>>>> >>>>>> Sounds like maybe you have multiple versions of the jars in your >>>>>>> hbase lib directory? >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 27, 2014 at 11:41 PM, Frederick Haebin Na < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Forgot to add my settings' info. >>>>>>>> >>>>>>>> CentOS 6.3 >>>>>>>> cdh4.6.0 >>>>>>>> HBase 0.94.15 >>>>>>>> >>>>>>>> >>>>>>>> Do I need to add the following to hbase-site.xml? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>hbase.coprocessor.user.region.classes</name> >>>>>>>> <value>???</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Haebin >>>>>>>> >>>>>>>> >>>>>>>> 2014-02-28 16:19 GMT+09:00 Frederick Haebin Na <[email protected] >>>>>>>> >: >>>>>>>> >>>>>>>> Hell all, >>>>>>>>> >>>>>>>>> The document says that put phoenix-<version>.jar in hbase/lib >>>>>>>>> directory but there isn't phoenix-<version>.jar like the previous >>>>>>>>> versions. >>>>>>>>> (2.2.2, 2.2.3) >>>>>>>>> >>>>>>>>> So, I just put the core file, phoenix-core-3.0.0-SNAPSHOT.jar, in >>>>>>>>> hbase/lib directory, yet it does not seem to work. >>>>>>>>> sqlline.py (not .sh) produces an error like the following. >>>>>>>>> >>>>>>>>> Setting property: [isolation, TRANSACTION_READ_COMMITTED] >>>>>>>>> issuing: !connect jdbc:phoenix:localhost none none >>>>>>>>> org.apache.phoenix.jdbc.PhoenixDriver >>>>>>>>> Connecting to jdbc:phoenix:localhost >>>>>>>>> Error: ERROR 2006 (INT08): Incompatible jars detected between >>>>>>>>> client and server. Ensure that phoenix.jar is put on the classpath of >>>>>>>>> HBase >>>>>>>>> in every region server: null (state=INT08,code=2006) >>>>>>>>> java.sql.SQLException: ERROR 2006 (INT08): Incompatible jars >>>>>>>>> detected between client and server. Ensure that phoenix.jar is put on >>>>>>>>> the >>>>>>>>> classpath of HBase in every region server: null >>>>>>>>> at >>>>>>>>> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:308) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:133) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:761) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:641) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:884) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1129) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:409) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:174) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:226) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:912) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1171) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:100) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:108) >>>>>>>>> at sqlline.SqlLine$DatabaseConnection.connect(SqlLine.java:4650) >>>>>>>>> at >>>>>>>>> sqlline.SqlLine$DatabaseConnection.getConnection(SqlLine.java:4701) >>>>>>>>> at sqlline.SqlLine$Commands.connect(SqlLine.java:3942) >>>>>>>>> at sqlline.SqlLine$Commands.connect(SqlLine.java:3851) >>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>> at >>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>> at >>>>>>>>> sqlline.SqlLine$ReflectiveCommandHandler.execute(SqlLine.java:2810) >>>>>>>>> at sqlline.SqlLine.dispatch(SqlLine.java:817) >>>>>>>>> at sqlline.SqlLine.initArgs(SqlLine.java:633) >>>>>>>>> at sqlline.SqlLine.begin(SqlLine.java:680) >>>>>>>>> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441) >>>>>>>>> at sqlline.SqlLine.main(SqlLine.java:424) >>>>>>>>> Caused by: java.lang.reflect.UndeclaredThrowableException >>>>>>>>> at com.sun.proxy.$Proxy7.getVersion(Unknown Source) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:736) >>>>>>>>> at >>>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:733) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1605) >>>>>>>>> at >>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> at java.lang.Thread.run(Thread.java:724) >>>>>>>>> Caused by: >>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: >>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No >>>>>>>>> matching >>>>>>>>> handler for protocol org.apache.phoenix.coprocessor.MetaDataProtocol >>>>>>>>> in >>>>>>>>> region SYSTEM.CATALOG,,1393571709854.ff2ff7eaea2d82e6f4bbb892d323307b. >>>>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>>>>>> Method) >>>>>>>>> at >>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:256) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79) >>>>>>>>> ... 9 more >>>>>>>>> Caused by: org.apache.hadoop.ipc.RemoteException: >>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No >>>>>>>>> matching >>>>>>>>> handler for protocol org.apache.phoenix.coprocessor.MetaDataProtocol >>>>>>>>> in >>>>>>>>> region SYSTEM.CATALOG,,1393571709854.ff2ff7eaea2d82e6f4bbb892d323307b. >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:5591) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3918) >>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>> at >>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87) >>>>>>>>> at com.sun.proxy.$Proxy6.execCoprocessor(Unknown Source) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1.call(ExecRPCInvoker.java:75) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1.call(ExecRPCInvoker.java:73) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164) >>>>>>>>> ... 10 more >>>>>>>>> sqlline version 1.1.2 >>>>>>>>> 0: jdbc:phoenix:localhost> >>>>>>>>> >>>>>>>>> >>>>>>>>> Funny thing is that I tried all the other versions, such as, 2.2.2 >>>>>>>>> and 2.2.3, but none of them worked with similar error. (Incompatible >>>>>>>>> version, please deploy phoenix.jar in hbase/lib) >>>>>>>>> >>>>>>>>> Can anyone guide me what to do? >>>>>>>>> >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Haebin >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >
