Please note that there is a difference between Phoenix Tracing and the
TRACE log4j level.
It appears that you're using a version of Phoenix which is incompatible
with the version of HBase/Hadoop that you're running. The implementation
of PhoenixMetricsSink is incompatible with the interface/abstract-class
that HBase/Hadoop is expecting.
This may be a classpath or Phoenix version issue, or you may have
stumbled onto a bug.
On 11/8/17 6:33 AM, Mallieswari Dineshbabu wrote:
Hi All,
I am working with HBase-Phoenix, /everything works fine/. In addition
trying to enable Tracing <https://phoenix.apache.org/tracing.html> in
Phoenix with the following steps,
1. Copy ‘hadoop-metrics2-hbase.PROPERTIES’ from Phoenix package to
HBase conf folder.
2. ‘hadoop-metrics2-phoenix.PROPERTIES’ file will be in ‘Phoenix/bin’
location by default. So I left it as it is.
3. Add the following property to phoenix configuration,
<property>
<name>phoenix.trace.frequency</name>
<value>always</value>
</property>
After doing the above, HBase’s HMaster fails to start with the following
exception; Please tell if you have any suggestion on this,
2017-11-08 16:46:56,118 INFO [main] regionserver.RSRpcServices:
master/Selfuser-VirtualBox/172.16.203.117:60000
<http://172.16.203.117:60000> server-side HConnection retries=140
2017-11-08 16:46:56,520 INFO [main] ipc.SimpleRpcScheduler: Using
deadline as user call queue, count=3
2017-11-08 16:46:56,554 INFO [main] ipc.RpcServer:
master/Selfuser-VirtualBox/192.16.203.117:60000
<http://192.16.203.117:60000>: started 10 reader(s) listening on port=60000
*2017-11-08 16:46:56,839 INFO [main] impl.MetricsConfig: loaded
properties from hadoop-metrics2-hbase.properties*
*2017-11-08 16:46:56,926 INFO [main] trace.PhoenixMetricsSink: Writing
tracing metrics to phoenix table*
*2017-11-08 16:46:56,933 ERROR [main] master.HMasterCommandLine: Master
exiting*
*java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster. *
*at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2512)*
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:231)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2522)
Caused by: java.lang.AbstractMethodError:
org.apache.phoenix.trace.PhoenixMetricsSink.init(Lorg/apache/commons/configuration/SubsetConfiguration;)V
at
org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:199)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:530)
at
org.apache.hadoop.metrics2.impl.MetricsSdrddystemImpl.configureSinks(MetricsSystemImpl.java:502)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:481)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:189)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:164)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.hbase.metrics.BaseSourceImpl$DefaultMetricsSystemInitializer.init(BaseSourceImpl.java:49)
at
org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:72)
at
org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceImpl.<init>(MetricsHBaseServerSourceImpl.java:66)
at
org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceFactoryImpl.getSource(MetricsHBaseServerSourceFactoryImpl.java:48)
at
org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceFactoryImpl.create(MetricsHBaseServerSourceFactoryImpl.java:38)
at
org.apache.hadoop.hbase.ipc.MetricsHBaseServer.<init>(MetricsHBaseServer.java:40)
at org.apache.hadoop.hbase.ipc.RpcServer.<init>(RpcServer.java:2040)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:930)
at
org.apache.hadoop.hbase.master.MasterRpcServices.<init>(MasterRpcServices.java:231)
at
org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:572)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:412)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2505)
... 5 more
Note: Additionally based on this Hortonworks reference
<https://community.hortonworks.com/articles/92371/trace-logging-for-phoenix.html>,
did the following changes in log4j.PROPERTIES of Phoenix. Still facing
same issue;
log4j.threshold=TRACE
log4j.logger.org.apache.phoenix=TRACE
log4j.logger.org.apache.hadoop.hbase.ipc=TRACE
log4j.logger.org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel=DEBUG
--
Thanks and regards
D.Mallieswari
