Re: Error while HBase from source code
Hi Sanjiv, Are you looking to run HBase in standalone mode (all daemons in one JVM) or pseudo-distributed mode (each with its own process, but on one host)? Have you tried following the instructions on http://hbase.apache.org/book.html regarding configurations to switch between the two? All the best, Dima On Mon, Sep 1, 2014 at 10:53 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Any help on this ...issue is still not resolved. Regards Sanjiv Singh Mob : +091 9990-447-339 On Mon, Sep 1, 2014 at 5:31 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Matteo, Following your steps, I was able to build binary package hbase-2. 0.0-SNAPSHOT-bin.tar.gz ..today I used the same to run hbase locally it is giving error when i tried to start with start-hbase.sh I am totally lost with it , as don't have clue about the error. Please help me on this. Following are logs generated .in logs directory : 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-01 17:22:46,109 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impad$ 2014-09-01 17:22:46,121 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-01 17:22:46,660 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 ] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:34636 2014-09-01 17:22:46,667 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 ] server.NIOServerCnxn: Processing stat command from /127.0.0.1:34636 2014-09-01 17:22:46,672 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-01 17:22:46,673 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:34636 (no session established for client) 2014-09-01 17:22:46,673 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-01 17:22:47,067 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-01 17:22:47,221 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-01 17:22:47,234 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-01 17:22:47,297 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-01 17:22:47,428 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasternull at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:214) at org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:152) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:185) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1764) Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.security.Groups.init(Groups.java:64) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:309) at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:303) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.isSecurityEnabled(User.java:349) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:340) at
Re: Error while HBase from source code
Hi Dima, It's standalone mode where all daemons in one JVM. I have not changed any single configuration , i tried to start hbase with all default configuration. Let me know if need info to debug. Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 1:21 PM, Dima Spivak dspi...@cloudera.com wrote: Hi Sanjiv, Are you looking to run HBase in standalone mode (all daemons in one JVM) or pseudo-distributed mode (each with its own process, but on one host)? Have you tried following the instructions on http://hbase.apache.org/book.html regarding configurations to switch between the two? All the best, Dima On Mon, Sep 1, 2014 at 10:53 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Any help on this ...issue is still not resolved. Regards Sanjiv Singh Mob : +091 9990-447-339 On Mon, Sep 1, 2014 at 5:31 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Matteo, Following your steps, I was able to build binary package hbase-2. 0.0-SNAPSHOT-bin.tar.gz ..today I used the same to run hbase locally it is giving error when i tried to start with start-hbase.sh I am totally lost with it , as don't have clue about the error. Please help me on this. Following are logs generated .in logs directory : 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-01 17:22:46,109 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impad$ 2014-09-01 17:22:46,121 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-01 17:22:46,660 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:34636 2014-09-01 17:22:46,667 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:34636 2014-09-01 17:22:46,672 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-01 17:22:46,673 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:34636 (no session established for client) 2014-09-01 17:22:46,673 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-01 17:22:47,067 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-01 17:22:47,221 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-01 17:22:47,234 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-01 17:22:47,297 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-01 17:22:47,428 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasternull at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:214) at org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:152) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:185) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1764) Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.security.Groups.init(Groups.java:64) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:309) at
Re: Error while HBase from source code
The default configuration should work well. Check if you set $HADOOP_HOME and $HBASE_HOME or not. On Tue, Sep 2, 2014 at 5:00 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Dima, It's standalone mode where all daemons in one JVM. I have not changed any single configuration , i tried to start hbase with all default configuration. Let me know if need info to debug. Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 1:21 PM, Dima Spivak dspi...@cloudera.com wrote: Hi Sanjiv, Are you looking to run HBase in standalone mode (all daemons in one JVM) or pseudo-distributed mode (each with its own process, but on one host)? Have you tried following the instructions on http://hbase.apache.org/book.html regarding configurations to switch between the two? All the best, Dima On Mon, Sep 1, 2014 at 10:53 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Any help on this ...issue is still not resolved. Regards Sanjiv Singh Mob : +091 9990-447-339 On Mon, Sep 1, 2014 at 5:31 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Matteo, Following your steps, I was able to build binary package hbase-2. 0.0-SNAPSHOT-bin.tar.gz ..today I used the same to run hbase locally it is giving error when i tried to start with start-hbase.sh I am totally lost with it , as don't have clue about the error. Please help me on this. Following are logs generated .in logs directory : 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-01 17:22:46,109 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impad$ 2014-09-01 17:22:46,121 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-01 17:22:46,660 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:34636 2014-09-01 17:22:46,667 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:34636 2014-09-01 17:22:46,672 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-01 17:22:46,673 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:34636 (no session established for client) 2014-09-01 17:22:46,673 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-01 17:22:47,067 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-01 17:22:47,221 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-01 17:22:47,234 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-01 17:22:47,297 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-01 17:22:47,428 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasternull at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:214) at org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:152) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:185) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1764) Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.security.Groups.init(Groups.java:64) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240) at
Re: Error while HBase from source code
I just checked, following are defined for these. # pointing to hadoop 1.x HADOOP_HOME=/usr/local/hadoop/hadoop-1.2.1/ # directory which is created after extracting hbase-2.0.0-SNAPSHOT-bin.tar.gz HBASE_HOME=/usr/local/hbase/hbase-2.0.0-SNAPSHOT/ Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 2:35 PM, tobe tobeg3oo...@gmail.com wrote: The default configuration should work well. Check if you set $HADOOP_HOME and $HBASE_HOME or not. On Tue, Sep 2, 2014 at 5:00 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Dima, It's standalone mode where all daemons in one JVM. I have not changed any single configuration , i tried to start hbase with all default configuration. Let me know if need info to debug. Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 1:21 PM, Dima Spivak dspi...@cloudera.com wrote: Hi Sanjiv, Are you looking to run HBase in standalone mode (all daemons in one JVM) or pseudo-distributed mode (each with its own process, but on one host)? Have you tried following the instructions on http://hbase.apache.org/book.html regarding configurations to switch between the two? All the best, Dima On Mon, Sep 1, 2014 at 10:53 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Any help on this ...issue is still not resolved. Regards Sanjiv Singh Mob : +091 9990-447-339 On Mon, Sep 1, 2014 at 5:31 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Matteo, Following your steps, I was able to build binary package hbase-2. 0.0-SNAPSHOT-bin.tar.gz ..today I used the same to run hbase locally it is giving error when i tried to start with start-hbase.sh I am totally lost with it , as don't have clue about the error. Please help me on this. Following are logs generated .in logs directory : 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-01 17:22:46,109 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impad$ 2014-09-01 17:22:46,121 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-01 17:22:46,660 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:34636 2014-09-01 17:22:46,667 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:34636 2014-09-01 17:22:46,672 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-01 17:22:46,673 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:34636 (no session established for client) 2014-09-01 17:22:46,673 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-01 17:22:47,067 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-01 17:22:47,221 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-01 17:22:47,234 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-01 17:22:47,297 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-01 17:22:47,428 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasternull at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:214) at org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:152) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:185) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1764) Caused by: java.lang.RuntimeException:
Re: Error while HBase from source code
I think you should NOT set HADOOP_HOME and HBASE_HOME, which have made some trouble for me. On Tue, Sep 2, 2014 at 5:48 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: I just checked, following are defined for these. # pointing to hadoop 1.x HADOOP_HOME=/usr/local/hadoop/hadoop-1.2.1/ # directory which is created after extracting hbase-2.0.0-SNAPSHOT-bin.tar.gz HBASE_HOME=/usr/local/hbase/hbase-2.0.0-SNAPSHOT/ Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 2:35 PM, tobe tobeg3oo...@gmail.com wrote: The default configuration should work well. Check if you set $HADOOP_HOME and $HBASE_HOME or not. On Tue, Sep 2, 2014 at 5:00 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Dima, It's standalone mode where all daemons in one JVM. I have not changed any single configuration , i tried to start hbase with all default configuration. Let me know if need info to debug. Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 1:21 PM, Dima Spivak dspi...@cloudera.com wrote: Hi Sanjiv, Are you looking to run HBase in standalone mode (all daemons in one JVM) or pseudo-distributed mode (each with its own process, but on one host)? Have you tried following the instructions on http://hbase.apache.org/book.html regarding configurations to switch between the two? All the best, Dima On Mon, Sep 1, 2014 at 10:53 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Any help on this ...issue is still not resolved. Regards Sanjiv Singh Mob : +091 9990-447-339 On Mon, Sep 1, 2014 at 5:31 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Matteo, Following your steps, I was able to build binary package hbase-2. 0.0-SNAPSHOT-bin.tar.gz ..today I used the same to run hbase locally it is giving error when i tried to start with start-hbase.sh I am totally lost with it , as don't have clue about the error. Please help me on this. Following are logs generated .in logs directory : 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-01 17:22:46,096 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-01 17:22:46,109 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impad$ 2014-09-01 17:22:46,121 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-01 17:22:46,660 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:34636 2014-09-01 17:22:46,667 INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:34636 2014-09-01 17:22:46,672 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-01 17:22:46,673 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:34636 (no session established for client) 2014-09-01 17:22:46,673 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-01 17:22:47,067 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-01 17:22:47,221 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-01 17:22:47,234 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-01 17:22:47,297 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-01 17:22:47,324 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-01 17:22:47,428 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasternull at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:214) at org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:152) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:185) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at
Re: Error while HBase from source code
Hi Tobe, I have removed HADOOP_HOME and HBASE_HOME , now i am getting different error like below \ Caused by: java.lang.IllegalArgumentException: Don't understand io engine name for cache - prefix with file:, heap or offheap at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:302) See below for complete startup logs : 2014-09-02 15:38:10,084 INFO [main] util.VersionInfo: Compiled by impadmin on Tue Sep 2 15:03:37 IST 2014 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:host.name=IMPETUS-NL147centos.impetus.co.in 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.version=1.7.0 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.home=/usr/java/jdk1.7.0/jre 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.class.path=/usr/local/hbase/hbase-2.0.0-SNAPSHOT/bin/../conf:/usr/java/jdk1.7.0/lib/tools.jar:/usr/local/hbase/hbase-2.0.0-SNAPSHOT/bin/..:/usr/local/hbase/hbase-2.0.0- 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.compiler=NA 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.name=Linux 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.arch=amd64 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.version=2.6.32-358.el6.x86_64 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-02 15:38:10,657 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impadmin/zookeeper/zookeeper_0/version-2 snapdir /tmp/hbase-impadmin/zookeeper/zookeeper_0/version-2 2014-09-02 15:38:10,704 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-02 15:38:11,291 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:49144 2014-09-02 15:38:11,296 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:49144 2014-09-02 15:38:11,301 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-02 15:38:11,302 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:49144 (no session established for client) 2014-09-02 15:38:11,302 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-02 15:38:12,066 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-02 15:38:12,517 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-02 15:38:12,543 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-02 15:38:12,646 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-02 15:38:12,710 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-02 15:38:12,710 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-02 15:38:12,887 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-09-02 15:38:12,915 INFO [main] hfile.CacheConfig: Allocating LruBlockCache size=386.70 MB, blockSize=64 KB 2014-09-02 15:38:12,949 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:214) at org.apache.hadoop.hbase.LocalHBaseCluster.init(LocalHBaseCluster.java:152) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:185) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at
Re: Error while HBase from source code
By default, HBase use onheap LruBlockCache which doesn't require to configure bucketcache.ioengine. Please double check your configuration file, hbase-site.xml. It would be ok if it doesn't have any property. On Tue, Sep 2, 2014 at 6:29 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Looking at the source-code , it seems like hbase.bucketcache.ioengine must be set to one of these file: , offheap , heap. it might help in debugging the issue . Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 3:45 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Tobe, I have removed HADOOP_HOME and HBASE_HOME , now i am getting different error like below \ Caused by: java.lang.IllegalArgumentException: Don't understand io engine name for cache - prefix with file:, heap or offheap at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:302) See below for complete startup logs : 2014-09-02 15:38:10,084 INFO [main] util.VersionInfo: Compiled by impadmin on Tue Sep 2 15:03:37 IST 2014 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:host.name=IMPETUS-NL147centos.impetus.co.in 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.version=1.7.0 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.home=/usr/java/jdk1.7.0/jre 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.class.path=/usr/local/hbase/hbase-2.0.0-SNAPSHOT/bin/../conf:/usr/java/jdk1.7.0/lib/tools.jar:/usr/local/hbase/hbase-2.0.0-SNAPSHOT/bin/..:/usr/local/hbase/hbase-2.0.0- 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.compiler=NA 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.name=Linux 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.arch=amd64 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.version=2.6.32-358.el6.x86_64 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-02 15:38:10,657 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impadmin/zookeeper/zookeeper_0/version-2 snapdir /tmp/hbase-impadmin/zookeeper/zookeeper_0/version-2 2014-09-02 15:38:10,704 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-02 15:38:11,291 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:49144 2014-09-02 15:38:11,296 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:49144 2014-09-02 15:38:11,301 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2014-09-02 15:38:11,302 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:49144 (no session established for client) 2014-09-02 15:38:11,302 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZK Cluster and connect 1 ZK server on client port: 2181 2014-09-02 15:38:12,066 INFO [main] regionserver.RSRpcServices: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0 server-side HConnection retries=350 2014-09-02 15:38:12,517 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2014-09-02 15:38:12,543 INFO [main] ipc.RpcServer: master/ IMPETUS-NL147centos.impetus.co.in/127.0.1.1:0: started 10 reader(s). 2014-09-02 15:38:12,646 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2014-09-02 15:38:12,710 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2014-09-02 15:38:12,710 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2014-09-02 15:38:12,887 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-09-02 15:38:12,915 INFO [main] hfile.CacheConfig: Allocating LruBlockCache size=386.70 MB, blockSize=64 KB 2014-09-02 15:38:12,949
[jira] [Created] (HBASE-11877) Make TableSplit more readable
Liu Shaohui created HBASE-11877:
---
Summary: Make TableSplit more readable
Key: HBASE-11877
URL: https://issues.apache.org/jira/browse/HBASE-11877
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Priority: Minor
When debugging MR jobs reading from hbase table, it's import to figure out
which region a map task is reading from.
But the table split object is hard to read.
eg:
{code}
2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split: lg-hadoop-prc-st40.bj:,0
{code}
We should make it more readable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Error while HBase from source code
I checked at both hbase-default.xml and hbase-site.xml in my code base. There is nothing defined in hbase-site.xml for this property and no value defined for property hbase.bucketcache.ioengine in hbase-default.xml. I am not sure what's wrong with it. See for default value : https://github.com/apache/hbase/blob/master/hbase-common/src/main/resources/hbase-default.xml when I changed it to 'heap' in my hbase-site.xml , now getting different error : Exception in thread main java.lang.RuntimeException: Current heap configuration for MemStore and BlockCache exceeds the threshold required for successful cluster operation. The combined value cannot exceed 0.8. Please check the settings for hbase.regionserver.global.memstore.size and hfile.block.cache.size in your configuration. hbase.regionserver.global.memstore.size is 0.4 hfile.block.cache.size is 68.09002 at org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil.checkForClusterFreeMemoryLimit(HeapMemorySizeUtil.java:64) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:82) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:92) at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39) Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 5:09 PM, tobe tobeg3oo...@gmail.com wrote: By default, HBase use onheap LruBlockCache which doesn't require to configure bucketcache.ioengine. Please double check your configuration file, hbase-site.xml. It would be ok if it doesn't have any property. On Tue, Sep 2, 2014 at 6:29 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Looking at the source-code , it seems like hbase.bucketcache.ioengine must be set to one of these file: , offheap , heap. it might help in debugging the issue . Regards Sanjiv Singh Mob : +091 9990-447-339 On Tue, Sep 2, 2014 at 3:45 PM, @Sanjiv Singh sanjiv.is...@gmail.com wrote: Hi Tobe, I have removed HADOOP_HOME and HBASE_HOME , now i am getting different error like below \ Caused by: java.lang.IllegalArgumentException: Don't understand io engine name for cache - prefix with file:, heap or offheap at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:302) See below for complete startup logs : 2014-09-02 15:38:10,084 INFO [main] util.VersionInfo: Compiled by impadmin on Tue Sep 2 15:03:37 IST 2014 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:host.name=IMPETUS-NL147centos.impetus.co.in 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.version=1.7.0 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.home=/usr/java/jdk1.7.0/jre 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.class.path=/usr/local/hbase/hbase-2.0.0-SNAPSHOT/bin/../conf:/usr/java/jdk1.7.0/lib/tools.jar:/usr/local/hbase/hbase-2.0.0-SNAPSHOT/bin/..:/usr/local/hbase/hbase-2.0.0- 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:java.compiler=NA 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.name=Linux 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.arch=amd64 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:os.version=2.6.32-358.el6.x86_64 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.name=impadmin 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/impadmin 2014-09-02 15:38:10,626 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase/hbase-2.0.0-SNAPSHOT 2014-09-02 15:38:10,657 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4 datadir /tmp/hbase-impadmin/zookeeper/zookeeper_0/version-2 snapdir /tmp/hbase-impadmin/zookeeper/zookeeper_0/version-2 2014-09-02 15:38:10,704 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2014-09-02 15:38:11,291 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from / 127.0.0.1:49144 2014-09-02 15:38:11,296 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:49144 2014-09-02 15:38:11,301 INFO
Re: [VOTE] [FAILED] Re: The 1st HBase 0.98.6 release candidate (RC0) is available, vote closing 9/2/2014
Thanks to Jean-Marc's assistance, the cause
for
TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
failure has been found.
I logged HBASE-11878 and attached a preliminary patch.
Cheers
On Mon, Sep 1, 2014 at 10:41 AM, Ted Yu yuzhih...@gmail.com wrote:
Searching for 'hbase.ResourceChecker(147): before' in
http://server.distparser.com:81/hbase/with_teds_patch/node8/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
showed that testAddVisibilityLabelsOnRSRestart was the first sub-test to
be run:
2014-09-01 08:41:12,266 INFO [pool-1-thread-1] hbase.ResourceChecker(147):
before:
security.visibility.TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
Thread=295, OpenFileDescriptor=380, MaxFileDescriptor=65536,
What Ram described was different from the problem JMS reported - since ABC
and XYZ were not written to hbase:labels table, the test would surely fail.
Cheers
On Mon, Sep 1, 2014 at 9:53 AM, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com wrote:
To reproduce just ensured that
in TestVisibilityLabelsWithDefaultVisLabelService the testAddLabels and
then testVddAddVisibilityLabelsOnRSRestart runs.
This ensures that the total no of labels is 17 and not 13.
So may be when it runs in Jenkins the order of execution determines the
way
the labels are added or deleted.
String[] labels = { SECRET, TOPSECRET, CONFIDENTIAL, PUBLIC, PRIVATE,
COPYRIGHT,
ACCENT, UNICODE_VIS_TAG, UC1, UC2 };
static String[] labels1 = { L1, SECRET, L2, invalid~, L3 };
static String[] labels2 = { SECRET, CONFIDENTIAL, PRIVATE, ABC, XYZ
};
Count this excluding the repeated confidential, secret and private it
gives
17. So may be we can seperate the test cases?
Regards
Ram
On Mon, Sep 1, 2014 at 9:55 PM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
Cool! Glad to see I'm not the only one ;)
Since it passed on Jenkins does it mean we need to add some additional
tests?
Le 2014-09-01 12:20, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com a écrit :
Am able to reproduce this locally. Will try to come up with a patch.
On Mon, Sep 1, 2014 at 9:43 PM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
So I guess that's why we don't see the log you have added? It loops
and
timeout and never reach that point?
What's next? Anything else I can run/do/test/check? I can even
install
Jenkins on those 4 servers
2014-09-01 11:59 GMT-04:00 Ted Yu yuzhih...@gmail.com:
The patch I sent to Jean-Marc added a loop when retrieving labels.
Looking at
http://server.distparser.com:81/hbase/with_teds_patch/node8/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
, there was no log from DefaultVisibilityLabelServiceImpl w.r.t.
labels
ABC
and XYZ.
The timeout of test was due to these two labels not written to
hbase:labels
table.
Cheers
On Mon, Sep 1, 2014 at 8:26 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
Ted sent me another patch to test.
Everything is here:
http://server.distparser.com:81/hbase/with_teds_patch/
This specific test did not failed on hbasetest1
Only this one failed on node8:
Tests in error:
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay):
test timed out after 6 milliseconds
Tests run: 1284, Failures: 0, Errors: 1, Skipped: 8
And many tests failed on node1 and t430s. But all
TestVisibilityLabelsWithDistributedLogReplay failed with
timeout.
Again, I have copied everything on the server so you look at
whatever
can
be interesting.
http://server.distparser.com:81/hbase/node1/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
Feel free to send me anything you want me to test. Also, I can
share
all
required information, like CPU, etc. if that helps.
JM
2014-09-01 0:06 GMT-04:00 Anoop John anoop.hb...@gmail.com:
The addition of the 2 new labels would have failed (?) May
be we
log
the
return value of this addition in test and see the trace?
-Anoop-
On Mon, Sep 1, 2014 at 5:59 AM, Ted Yu yuzhih...@gmail.com
wrote:
In the output of successful test run, I saw:
2014-08-30 11:24:12,139 DEBUG
[B.defaultRpcServer.handler=1,queue=0,port=56716]
Re: [VOTE] [FAILED] Re: The 1st HBase 0.98.6 release candidate (RC0) is available, vote closing 9/2/2014
Tried Ted's patch on 3 servers. The 3 passed.
2014-09-02 10:40 GMT-04:00 Ted Yu yuzhih...@gmail.com:
Thanks to Jean-Marc's assistance, the cause
for
TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
failure has been found.
I logged HBASE-11878 and attached a preliminary patch.
Cheers
On Mon, Sep 1, 2014 at 10:41 AM, Ted Yu yuzhih...@gmail.com wrote:
Searching for 'hbase.ResourceChecker(147): before' in
http://server.distparser.com:81/hbase/with_teds_patch/node8/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
showed that testAddVisibilityLabelsOnRSRestart was the first sub-test to
be run:
2014-09-01 08:41:12,266 INFO [pool-1-thread-1]
hbase.ResourceChecker(147): before:
security.visibility.TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
Thread=295, OpenFileDescriptor=380, MaxFileDescriptor=65536,
What Ram described was different from the problem JMS reported - since
ABC and XYZ were not written to hbase:labels table, the test would
surely fail.
Cheers
On Mon, Sep 1, 2014 at 9:53 AM, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com wrote:
To reproduce just ensured that
in TestVisibilityLabelsWithDefaultVisLabelService the testAddLabels and
then testVddAddVisibilityLabelsOnRSRestart runs.
This ensures that the total no of labels is 17 and not 13.
So may be when it runs in Jenkins the order of execution determines the
way
the labels are added or deleted.
String[] labels = { SECRET, TOPSECRET, CONFIDENTIAL, PUBLIC, PRIVATE,
COPYRIGHT,
ACCENT, UNICODE_VIS_TAG, UC1, UC2 };
static String[] labels1 = { L1, SECRET, L2, invalid~, L3 };
static String[] labels2 = { SECRET, CONFIDENTIAL, PRIVATE, ABC,
XYZ
};
Count this excluding the repeated confidential, secret and private it
gives
17. So may be we can seperate the test cases?
Regards
Ram
On Mon, Sep 1, 2014 at 9:55 PM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
Cool! Glad to see I'm not the only one ;)
Since it passed on Jenkins does it mean we need to add some additional
tests?
Le 2014-09-01 12:20, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com a écrit :
Am able to reproduce this locally. Will try to come up with a patch.
On Mon, Sep 1, 2014 at 9:43 PM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
So I guess that's why we don't see the log you have added? It
loops
and
timeout and never reach that point?
What's next? Anything else I can run/do/test/check? I can even
install
Jenkins on those 4 servers
2014-09-01 11:59 GMT-04:00 Ted Yu yuzhih...@gmail.com:
The patch I sent to Jean-Marc added a loop when retrieving
labels.
Looking at
http://server.distparser.com:81/hbase/with_teds_patch/node8/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
, there was no log from DefaultVisibilityLabelServiceImpl w.r.t.
labels
ABC
and XYZ.
The timeout of test was due to these two labels not written to
hbase:labels
table.
Cheers
On Mon, Sep 1, 2014 at 8:26 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
Ted sent me another patch to test.
Everything is here:
http://server.distparser.com:81/hbase/with_teds_patch/
This specific test did not failed on hbasetest1
Only this one failed on node8:
Tests in error:
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay):
test timed out after 6 milliseconds
Tests run: 1284, Failures: 0, Errors: 1, Skipped: 8
And many tests failed on node1 and t430s. But all
TestVisibilityLabelsWithDistributedLogReplay failed with
timeout.
Again, I have copied everything on the server so you look at
whatever
can
be interesting.
http://server.distparser.com:81/hbase/node1/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
Feel free to send me anything you want me to test. Also, I can
share
all
required information, like CPU, etc. if that helps.
JM
2014-09-01 0:06 GMT-04:00 Anoop John anoop.hb...@gmail.com:
The addition of the 2 new labels would have failed (?) May
be we
log
the
return value of this addition in test and see the trace?
Re: [VOTE] The 1st hbase 0.94.23 release candidate is available for download
+1 Sums and signature looks good Tarball structure looks good Built from source Unit tests pass Ran LoadTestTool, no unusual warnings or errors in the logs On Tue, Aug 26, 2014 at 7:38 PM, lars hofhansl la...@apache.org wrote: The 1st 0.94.23 RC is available for download at http://people.apache.org/~larsh/hbase-0.94.23-rc0/ Signed with my code signing key: C7CFE328 HBase 0.94.23 is a small bug fix release with 12 fixes: [HBASE-9746] - RegionServer can't start when replication tries to replicate to an unknown host [HBASE-10834] - Better error messaging on issuing grant commands in non-authz mode [HBASE-11232] - Add MultiRowMutation tests. [HBASE-11536] - Puts of region location to Meta may be out of order which causes inconsistent of region location [HBASE-11641] - TestDistributedLogSplitting.testMasterStartsUpWithLogSplittingWork fails frequently [HBASE-11652] - Port HBASE-3270 and HBASE-11650 to 0.94 - create cluster id and version file in a tmp location and move it into place [HBASE-11767] - [0.94] Unnecessary garbage produced by schema metrics during scanning [HBASE-11667] - Comment ClientScanner logic for NSREs. [HBASE-11754] - [Shell] Record table property SPLITS_FILE in descriptor [HBASE-11690] - Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 [HBASE-11691] - Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 [HBASE-11693] - Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 TThe list of changes is also available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12327363 Here's the test run for this RC: https://builds.apache.org/job/HBase-0.94.23/39/ Please try out the RC, check out the doc, take it for a spin, etc, and vote +1/-1 by EOD September 2nd on whether we should release this as 0.94.23. Thanks. -- Lars -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-11879) Change TableInputFormatBase to take interface arguments
Carter created HBASE-11879:
--
Summary: Change TableInputFormatBase to take interface arguments
Key: HBASE-11879
URL: https://issues.apache.org/jira/browse/HBASE-11879
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
As part of the ongoing interface abstraction work, I'm now investigating
{{TableInputFormatBase}}, which has two methods that break encapsulation:
{code}
protected HTable getHTable();
protected void setHTable(HTable table);
{code}
While these are protected methods, the base @InterfaceAudience.Public is
abstract, meaning that it supports extension by user code.
I propose deprecating these two methods and replacing them with these four,
once the Table interface is merged:
{code}
protected Table getTable();
protected void setTable(Table table);
protected RegionLocator getRegionLocator();
protected void setRegionLocator(RegionLocator regionLocator);
{code}
Since users will frequently call {{setTable}} and {{setRegionLocator}}
together, it probably also makes sense to add the following convenience method:
{code}
protected void setTableAndRegionLocator(Table table, RegionLocator
regionLocator);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[REMINDER] 0.99.0 RC0 on Sep 8
Just a friendly reminder that I'll try to cut the first RC Monday next week (Sep 8). Please let me know (in this thread or jira) if you need a patch in before the cut. Enis
[jira] [Created] (HBASE-11880) NPE in MasterStatusServlet
Jimmy Xiang created HBASE-11880:
---
Summary: NPE in MasterStatusServlet
Key: HBASE-11880
URL: https://issues.apache.org/jira/browse/HBASE-11880
Project: HBase
Issue Type: Bug
Components: master
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
{quote}
2014-09-02 10:03:04,807 ERROR [82061158@qtp-1041323834-0] mortbay.log:
/master-status
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.MasterStatusServlet.getMetaLocationOrNull(MasterStatusServlet.java:91)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1345)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] [FAILED] Re: The 1st HBase 0.98.6 release candidate (RC0) is available, vote closing 9/2/2014
Thanks J-M
Your effort is valuable.
On Tue, Sep 2, 2014 at 11:16 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org wrote:
Tried Ted's patch on 3 servers. The 3 passed.
2014-09-02 10:40 GMT-04:00 Ted Yu yuzhih...@gmail.com:
Thanks to Jean-Marc's assistance, the cause
for
TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
failure has been found.
I logged HBASE-11878 and attached a preliminary patch.
Cheers
On Mon, Sep 1, 2014 at 10:41 AM, Ted Yu yuzhih...@gmail.com wrote:
Searching for 'hbase.ResourceChecker(147): before' in
http://server.distparser.com:81/hbase/with_teds_patch/node8/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
showed that testAddVisibilityLabelsOnRSRestart was the first sub-test
to
be run:
2014-09-01 08:41:12,266 INFO [pool-1-thread-1]
hbase.ResourceChecker(147): before:
security.visibility.TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
Thread=295, OpenFileDescriptor=380, MaxFileDescriptor=65536,
What Ram described was different from the problem JMS reported - since
ABC and XYZ were not written to hbase:labels table, the test would
surely fail.
Cheers
On Mon, Sep 1, 2014 at 9:53 AM, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com wrote:
To reproduce just ensured that
in TestVisibilityLabelsWithDefaultVisLabelService the testAddLabels
and
then testVddAddVisibilityLabelsOnRSRestart runs.
This ensures that the total no of labels is 17 and not 13.
So may be when it runs in Jenkins the order of execution determines
the
way
the labels are added or deleted.
String[] labels = { SECRET, TOPSECRET, CONFIDENTIAL, PUBLIC, PRIVATE,
COPYRIGHT,
ACCENT, UNICODE_VIS_TAG, UC1, UC2 };
static String[] labels1 = { L1, SECRET, L2, invalid~, L3 };
static String[] labels2 = { SECRET, CONFIDENTIAL, PRIVATE, ABC,
XYZ
};
Count this excluding the repeated confidential, secret and private it
gives
17. So may be we can seperate the test cases?
Regards
Ram
On Mon, Sep 1, 2014 at 9:55 PM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
Cool! Glad to see I'm not the only one ;)
Since it passed on Jenkins does it mean we need to add some
additional
tests?
Le 2014-09-01 12:20, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com a écrit :
Am able to reproduce this locally. Will try to come up with a
patch.
On Mon, Sep 1, 2014 at 9:43 PM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
So I guess that's why we don't see the log you have added? It
loops
and
timeout and never reach that point?
What's next? Anything else I can run/do/test/check? I can even
install
Jenkins on those 4 servers
2014-09-01 11:59 GMT-04:00 Ted Yu yuzhih...@gmail.com:
The patch I sent to Jean-Marc added a loop when retrieving
labels.
Looking at
http://server.distparser.com:81/hbase/with_teds_patch/node8/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
, there was no log from DefaultVisibilityLabelServiceImpl
w.r.t.
labels
ABC
and XYZ.
The timeout of test was due to these two labels not written to
hbase:labels
table.
Cheers
On Mon, Sep 1, 2014 at 8:26 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org
wrote:
Ted sent me another patch to test.
Everything is here:
http://server.distparser.com:81/hbase/with_teds_patch/
This specific test did not failed on hbasetest1
Only this one failed on node8:
Tests in error:
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay):
test timed out after 6 milliseconds
Tests run: 1284, Failures: 0, Errors: 1, Skipped: 8
And many tests failed on node1 and t430s. But all
TestVisibilityLabelsWithDistributedLogReplay failed with
timeout.
Again, I have copied everything on the server so you look at
whatever
can
be interesting.
http://server.distparser.com:81/hbase/node1/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
Feel free to send me anything you want me to test. Also, I
can
share
all
required information, like
[jira] [Created] (HBASE-11881) Improve the swift structures for KeyValue lists
Manukranth Kolloju created HBASE-11881: -- Summary: Improve the swift structures for KeyValue lists Key: HBASE-11881 URL: https://issues.apache.org/jira/browse/HBASE-11881 Project: HBase Issue Type: Bug Components: Thrift Affects Versions: 0.89-fb Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 0.89-fb With introduction of swift we have to build composite data structures that can represent a list of KeyValues in a more efficient manner. This can be used both in Put and the Result objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [REMINDER] 0.99.0 RC0 on Sep 8
Enis, I will have a patch for HBASE-11072. Working on the rebase. Hopefully it will be available tomorrow. Thanks, Sergey On Tue, Sep 2, 2014 at 10:28 PM, Enis Söztutar e...@apache.org wrote: Just a friendly reminder that I'll try to cut the first RC Monday next week (Sep 8). Please let me know (in this thread or jira) if you need a patch in before the cut. Enis
Re: [VOTE] The 1st hbase 0.94.23 release candidate is available for download
Done with the perfs tests too. 0.94.23 is overall very similar to 0.94.22, slightly faster. Compared all those tests: FilteredScanTest RandomReadTest RandomSeekScanTestRandomScanWithRange10Test RandomScanWithRange100Test RandomScanWithRange1000Test SequentialReadTest SequentialWriteTest RandomWriteTest GaussianRandomReadBenchmark SequentialReadBenchmarkSequentialWriteBenchmark UniformRandomReadBenchmark UniformRandomSmallScan LoadTestToolIntegrationTestLoadAnVerify HLogPerformanceEvaluation IntegrationTestBigLinkedList So still +1 for me. JM 2014-09-02 14:18 GMT-04:00 Andrew Purtell apurt...@apache.org: +1 Sums and signature looks good Tarball structure looks good Built from source Unit tests pass Ran LoadTestTool, no unusual warnings or errors in the logs On Tue, Aug 26, 2014 at 7:38 PM, lars hofhansl la...@apache.org wrote: The 1st 0.94.23 RC is available for download at http://people.apache.org/~larsh/hbase-0.94.23-rc0/ Signed with my code signing key: C7CFE328 HBase 0.94.23 is a small bug fix release with 12 fixes: [HBASE-9746] - RegionServer can't start when replication tries to replicate to an unknown host [HBASE-10834] - Better error messaging on issuing grant commands in non-authz mode [HBASE-11232] - Add MultiRowMutation tests. [HBASE-11536] - Puts of region location to Meta may be out of order which causes inconsistent of region location [HBASE-11641] - TestDistributedLogSplitting.testMasterStartsUpWithLogSplittingWork fails frequently [HBASE-11652] - Port HBASE-3270 and HBASE-11650 to 0.94 - create cluster id and version file in a tmp location and move it into place [HBASE-11767] - [0.94] Unnecessary garbage produced by schema metrics during scanning [HBASE-11667] - Comment ClientScanner logic for NSREs. [HBASE-11754] - [Shell] Record table property SPLITS_FILE in descriptor [HBASE-11690] - Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 [HBASE-11691] - Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 [HBASE-11693] - Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 TThe list of changes is also available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12327363 Here's the test run for this RC: https://builds.apache.org/job/HBase-0.94.23/39/ Please try out the RC, check out the doc, take it for a spin, etc, and vote +1/-1 by EOD September 2nd on whether we should release this as 0.94.23. Thanks. -- Lars -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-11882) Row level consistency may not be maintained with bulk load and compaction
Jerry He created HBASE-11882: Summary: Row level consistency may not be maintained with bulk load and compaction Key: HBASE-11882 URL: https://issues.apache.org/jira/browse/HBASE-11882 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 1.0.0, 2.0.0 Reporter: Jerry He Priority: Critical Fix For: 1.0.0, 2.0.0 While looking into the TestHRegionServerBulkLoad failure for HBASE-11772, I found the root cause is that row level atomicity may not be maintained with bulk load together with compation. TestHRegionServerBulkLoad is used to test bulk load atomicity. The test uses multiple threads to do bulk load and scan continuously and do compactions periodically. It verifies row level data is always consistent across column families. After HBASE-11591, we added readpoint checks for bulkloaded data using the seqId at the time of bulk load. Now a scanner will not see the data from a bulk load if the scanner's readpoint is earlier than the bulk load seqId. Previously, the atomic bulk load result is visible immediately to all scanners. The problem is with compaction after bulk load. Compaction does not lock the region and it is done one store (column family) at a time. It also compact away the seqId marker of bulk load. Here is an event sequence where the row level consistency is broken. 1. A scanner is started to scan a region with cf1 and cf2. The readpoint is 10. 2. There is a bulk load that loads into cf1 and cf2. The bulk load seqId is 11. Bulk load is guarded by region write lock. So it is atomic. 3. There is a compaction that compacts cf1. It compacts away the seqId marker of the bulk load. 4. The scanner tries to next to row-1001. It gets the bulk load data for cf1 since there is no seqId preventing it. It does not get the bulk load data for cf2 since the scanner's readpoint (10) is less than the bulk load seqId (11). Now we row level consistency is broken in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11883) Refactoring of ZookeeperWrapper
Yi Deng created HBASE-11883: --- Summary: Refactoring of ZookeeperWrapper Key: HBASE-11883 URL: https://issues.apache.org/jira/browse/HBASE-11883 Project: HBase Issue Type: Bug Components: IPC/RPC Affects Versions: 0.89-fb Reporter: Yi Deng Priority: Minor Fix For: 0.89-fb ZookeeperWrapper contains a lot of confusing methods. Combine and redefine them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Thrift client authentication skipped for HBase thrift server
Kashif, HBASE-11349/HBASE-11474 is indeed to authenticate Thrift clients using Kerberos. Is this what you are looking for? For Thrift server authentication, it is already there. Please refer to the hbase book http://hbase.apache.org/book/security.html#hbase.secure.configuration Section 8.1.4 for more details. Thanks, Jimmy On Sun, Aug 31, 2014 at 11:28 PM, Kashif Jawed Siddiqui kashi...@huawei.com wrote: Hi All, As per current implementation done for https://issues.apache.org/jira/i#browse/HBASE-11349 https://issues.apache.org/jira/i#browse/HBASE-11474 , The authentication mechanism using Kerberos principal for Thrift server with HBase is perfectly fine. But for clients communicating to HBase via thrift server does not handle the security mechanism. Any unauthenticated client can access HBase via thrift server. The thrift sever can act as a backdoor entry for skipping the security authentication. It will be better if thrift clients can also be authenticated through some mechanism like Kerberos or IP restriction,etc Let us discuss on mechanism for thrift client authentication that can be implemented. *** This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
Re: [REMINDER] 0.99.0 RC0 on Sep 8
Enis, I will have a patch for HBASE-10576. I will upload it tomorrow mostly. Thanks, Rajeshbabu. On Wed, Sep 3, 2014 at 12:18 AM, Sergey Soldatov sergeysolda...@gmail.com wrote: Enis, I will have a patch for HBASE-11072. Working on the rebase. Hopefully it will be available tomorrow. Thanks, Sergey On Tue, Sep 2, 2014 at 10:28 PM, Enis Söztutar e...@apache.org wrote: Just a friendly reminder that I'll try to cut the first RC Monday next week (Sep 8). Please let me know (in this thread or jira) if you need a patch in before the cut. Enis
[jira] [Resolved] (HBASE-11878) TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController not yet initialized
[
https://issues.apache.org/jira/browse/HBASE-11878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-11878.
Resolution: Fixed
TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
sometimes fails due to VisibilityController not yet initialized
---
Key: HBASE-11878
URL: https://issues.apache.org/jira/browse/HBASE-11878
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 2.0.0, 0.98.6
Attachments: 11878-v1.txt, 11878-v2.txt, 11878-v2.txt, 11878-v3.txt,
11878-v4.txt, 11878-v5.txt
In the thread w.r.t. first RC of 0.98.6,
http://search-hadoop.com/m/DHED4p2rw81 , Jean-Marc reported that
TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart
sometimes failed on his machines.
From
http://server.distparser.com:81/hbase/with_teds_patch2/hbasetest1/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt
:
{code}
result {
exception {
name: org.apache.hadoop.hbase.coprocessor.CoprocessorException
value: org.apache.hadoop.hbase.coprocessor.CoprocessorException:
VisibilityController not yet initialized\n\tat
org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:638)\n\tat
org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)\n\tat
org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)\n\tat
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5794)\n\tat
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1608)\n\tat
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1590)\n\tat
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30088)\n\tat
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2014)\n\tat
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)\n\tat
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)\n\tat
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)\n\tat
java.lang.Thread.run(Thread.java:744)\n
}
}
{code}
The above exception revealed a race condition: writing of labels ABC and
XYZ took place when VisibilityController was not yet initialized.
The test writes the labels only once, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-11854) MetricsRegion.updateScanNext takes 20% of overall RegionScannerImpl.nextRaw call
[ https://issues.apache.org/jira/browse/HBASE-11854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11854. Resolution: Duplicate Fix Version/s: (was: 0.98.7) (was: 2.0.0) (was: 0.99.0) Assignee: (was: Vladimir Rodionov) I'm going to resolve this as a dup of HBASE-11876 because there's a patch available on that issue and it seems to subsume this. MetricsRegion.updateScanNext takes 20% of overall RegionScannerImpl.nextRaw call Key: HBASE-11854 URL: https://issues.apache.org/jira/browse/HBASE-11854 Project: HBase Issue Type: Bug Components: metrics, Performance, Scanners Affects Versions: 0.98.6 Reporter: Vladimir Rodionov From our internal profiling we were surprised by amount of time spent in this call. Further investigation is needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11884) RegionServer Thrift RPC queue for control messages, to prevent blocking on read/write queues
Alex Goltman created HBASE-11884: Summary: RegionServer Thrift RPC queue for control messages, to prevent blocking on read/write queues Key: HBASE-11884 URL: https://issues.apache.org/jira/browse/HBASE-11884 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 0.89-fb Reporter: Alex Goltman Priority: Minor Fix For: 0.89-fb Enable out-of-band path for control messages such as update servers assignment and others, otherwise when server is busy can get blocked by read/write requests and cause timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11230) Remove getRowOrBefore from HTableInterface and HTable
[
https://issues.apache.org/jira/browse/HBASE-11230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-11230.
---
Resolution: Implemented
Implemented as a part of HBASE-11797
Remove getRowOrBefore from HTableInterface and HTable
-
Key: HBASE-11230
URL: https://issues.apache.org/jira/browse/HBASE-11230
Project: HBase
Issue Type: Task
Reporter: Andrew Purtell
Fix For: 0.99.0, 2.0.0
The javadoc for HTable#getRowOrBefore says:
{code}
* @deprecated As of version 0.92 this method is deprecated without
* replacement.
* getRowOrBefore is used internally to find entries in hbase:meta and makes
* various assumptions about the table (which are true for hbase:meta but
not
* in general) to be efficient.
{code}
It's been deprecated long enough. We should remove it from the client API and
mark it as private.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-11863) WAL files are not archived and stays in the WAL directory after splitting
[ https://issues.apache.org/jira/browse/HBASE-11863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-11863. --- Resolution: Fixed Hadoop Flags: Reviewed WAL files are not archived and stays in the WAL directory after splitting -- Key: HBASE-11863 URL: https://issues.apache.org/jira/browse/HBASE-11863 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Blocker Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: hbase-11863_v1-0.98.patch, hbase-11863_v1.patch, hbase-11863_v2.patch, hbase-11863_v3-0.98.patch, hbase-11863_v3-0.99.patch, hbase-11863_v3.patch In patch HBASE-11094, it seems that we changed the constructor we are using for SplitLogManager, which does not archive the log files after splitting is done to the archive folder. The log files stays in the splitting directory forever and re-split every time the master restarts. It is surprising that our unit tests are passing (since 0.94.4) without any issues. Part of the reason is that the split is actually carried, but the WAL is not moved and thus the -splitting directory never gets deleted. It seems critical to fix in 0.98.6, [~andrew.purt...@gmail.com] FYI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-11868) Data loss in hlog when the hdfs is unavailable
[
https://issues.apache.org/jira/browse/HBASE-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-11868:
This change causes reproducible test failures in TestHRegion and
TestHregionBusyWait. Reverting for .6. We can try again for .7
Data loss in hlog when the hdfs is unavailable
--
Key: HBASE-11868
URL: https://issues.apache.org/jira/browse/HBASE-11868
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.5
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Blocker
Fix For: 0.98.6
Attachments: HBASE-11868-0.98-v1.diff
When using the new thread model in hbase 0.98, we found a bug which may cause
data loss when the the hdfs is unavailable.
When writing wal Edits to hlog in doMiniBatchMutation of HRegion, the hlog
first call appendNoSync to write the edits to hlog and then call sync with
txid.
Assumed that the txid of current write is 10, and the syncedTillHere in hlog
is 9 and the failedTxid is 0. When the the hdfs is unavailable, the
AsyncWriter or AsyncSyncer will fail to apend the edits or sync, then they
will update the syncedTillHere to 10 and the failedTxid to 10.
When the hlog calls the sync with txid :10, the failedTxid will nerver be
checked for txid equals with syncedTillHere. The client thinks the write
success , but the data only be writtten to memstore not hlog. If the
regionserver is down later before the memstore is flushed, the data will be
lost.
See: FSHLog.java #1348
{code}
// sync all transactions upto the specified txid
private void syncer(long txid) throws IOException {
synchronized (this.syncedTillHere) {
while (this.syncedTillHere.get() txid) {
try {
this.syncedTillHere.wait();
if (txid = this.failedTxid.get()) {
assert asyncIOE != null :
current txid is among(under) failed txids, but asyncIOE is
null!;
throw asyncIOE;
}
} catch (InterruptedException e) {
LOG.debug(interrupted while waiting for notification from
AsyncNotifier);
}
}
}
}
{code}
We can fix this issue by moving the comparing of txid and failedTxid outside
the while block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source
Dima Spivak created HBASE-11885: --- Summary: Provide a Dockerfile to easily build and run HBase from source Key: HBASE-11885 URL: https://issues.apache.org/jira/browse/HBASE-11885 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak [A recent email to dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E] highlighted the difficulty that new users can face in getting HBase compiled from source and running locally. I'd like to provide a Dockerfile that would allow anyone with Docker running on a machine with a reasonably current Linux kernel to do so with ease. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
The 2nd HBase 0.98.6 release candidate (RC1) is available, vote closing 9/9/2014
The 2nd HBase 0.98.6 release candidate (RC1) is available for download at https://people.apache.org/~apurtell/0.98.6RC1/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1034 Signed with my code signing key D5365CCD. The issues resolved in this release can be found here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12327376 This candidate contains a significant backported change, ZK-less region assignment (HBASE-11546), thanks to the heavy lifting of Virag Kothari (and original contributors Jimmy Xiang and Virag Kothari). The new code will not be active unless the configuration setting hbase.assignment.usezk is set to 'false'. If possible please consider testing with this configuration change on a cluster and/or with the integration tests. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on September 9 on whether or not we should release this as 0.98.6. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-11886) The creator of the table should have all permissions on the table
Devaraj Das created HBASE-11886: --- Summary: The creator of the table should have all permissions on the table Key: HBASE-11886 URL: https://issues.apache.org/jira/browse/HBASE-11886 Project: HBase Issue Type: Bug Affects Versions: 0.98.6, 0.98.3 Reporter: Devaraj Das Priority: Critical Fix For: 1.0.0 In our testing of 0.98.4 with security ON, we found that table creator doesn't have RWXCA on the created table. Instead, the user representing the HBase daemon gets all permissions. Due to this the table creator can't write to the table he just created. I am suspecting HBASE-11275 introduced the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: The 2nd HBase 0.98.6 release candidate (RC1) is available, vote closing 9/9/2014
I meant to send this mail earlier in the evening but wasn't able to.. Apologies. Andrew could you take a look at HBASE-11886 and see if it should be fixed in this release. I think it's a serious usability issue but will defer to your call. On Tue, Sep 2, 2014 at 10:17 PM, Andrew Purtell apurt...@apache.org wrote: The 2nd HBase 0.98.6 release candidate (RC1) is available for download at https://people.apache.org/~apurtell/0.98.6RC1/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1034 Signed with my code signing key D5365CCD. The issues resolved in this release can be found here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12327376 This candidate contains a significant backported change, ZK-less region assignment (HBASE-11546), thanks to the heavy lifting of Virag Kothari (and original contributors Jimmy Xiang and Virag Kothari). The new code will not be active unless the configuration setting hbase.assignment.usezk is set to 'false'. If possible please consider testing with this configuration change on a cluster and/or with the integration tests. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on September 9 on whether or not we should release this as 0.98.6. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Thrift client authentication skipped for HBase thrift server
Hi Jimmy, HBASE-11349/HBASE-11474 is to authenticate the connection from thrift server to HBase. Here by the term thrift clients, I mean clients trying to access HBase via thrift server. For example, clients in any unsecure machine will connect to thrift server port 9090 , and in turn thrift will connect to HBase. Client -(unauthenticated connect via port 9090) Thrift - (Kerberos authenticated connection) HBase As portrayed above, the Thrift will authenticate with HBase using Kerberos. But clients connecting to thrift server via 9090 do not get authentictaed *** This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! -Original Message- From: Jimmy Xiang [mailto:jxi...@cloudera.com] Sent: 03 September 2014 03:06 To: dev@hbase.apache.org Subject: Re: Thrift client authentication skipped for HBase thrift server Kashif, HBASE-11349/HBASE-11474 is indeed to authenticate Thrift clients using Kerberos. Is this what you are looking for? For Thrift server authentication, it is already there. Please refer to the hbase book http://hbase.apache.org/book/security.html#hbase.secure.configuration Section 8.1.4 for more details. Thanks, Jimmy On Sun, Aug 31, 2014 at 11:28 PM, Kashif Jawed Siddiqui kashi...@huawei.com wrote: Hi All, As per current implementation done for https://issues.apache.org/jira/i#browse/HBASE-11349 https://issues.apache.org/jira/i#browse/HBASE-11474 , The authentication mechanism using Kerberos principal for Thrift server with HBase is perfectly fine. But for clients communicating to HBase via thrift server does not handle the security mechanism. Any unauthenticated client can access HBase via thrift server. The thrift sever can act as a backdoor entry for skipping the security authentication. It will be better if thrift clients can also be authenticated through some mechanism like Kerberos or IP restriction,etc Let us discuss on mechanism for thrift client authentication that can be implemented. ** * This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
