[jira] [Updated] (HBASE-6970) hbase-deamon.sh creates/updates pid file even when that start failed.
[
https://issues.apache.org/jira/browse/HBASE-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-6970:
---
Assignee: nkeywal
> hbase-deamon.sh creates/updates pid file even when that start failed.
> -
>
> Key: HBASE-6970
> URL: https://issues.apache.org/jira/browse/HBASE-6970
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: nkeywal
>
> We just ran into a strange issue where could neither start nor stop services
> with hbase-deamon.sh.
> The problem is this:
> {code}
> nohup nice -n $HBASE_NICENESS "$HBASE_HOME"/bin/hbase \
> --config "${HBASE_CONF_DIR}" \
> $command "$@" $startStop > "$logout" 2>&1 < /dev/null &
> echo $! > $pid
> {code}
> So the pid file is created or updated even when the start of the service
> failed. The next stop command will then fail, because the pid file has the
> wrong pid in it.
> Edit: Spelling and more spelling errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6970) hbase-deamon.sh creates/updates pid file even when that start failed.
[
https://issues.apache.org/jira/browse/HBASE-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503690#comment-13503690
]
nkeywal commented on HBASE-6970:
I plan to work on it this week.
> hbase-deamon.sh creates/updates pid file even when that start failed.
> -
>
> Key: HBASE-6970
> URL: https://issues.apache.org/jira/browse/HBASE-6970
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: nkeywal
>
> We just ran into a strange issue where could neither start nor stop services
> with hbase-deamon.sh.
> The problem is this:
> {code}
> nohup nice -n $HBASE_NICENESS "$HBASE_HOME"/bin/hbase \
> --config "${HBASE_CONF_DIR}" \
> $command "$@" $startStop > "$logout" 2>&1 < /dev/null &
> echo $! > $pid
> {code}
> So the pid file is created or updated even when the start of the service
> failed. The next stop command will then fail, because the pid file has the
> wrong pid in it.
> Edit: Spelling and more spelling errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-4709:
---
Attachment: 4709_workaround.v1.patch
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
>
>
> Key: HBASE-4709
> URL: https://issues.apache.org/jira/browse/HBASE-4709
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.92.0, 0.94.0, 0.94.1
>Reporter: Gary Helmling
>Priority: Minor
> Attachments: 4709_workaround.v1.patch
>
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been
> getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
> jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
> to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
> SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir:
> /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
> system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
> Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered
> with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> at
> com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> at
> com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> at
> com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by
> default, when running in a mini cluster. The errors themselves seem to be
> harmless -- they're not breaking any tests -- but we should figure out what
> configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-4709:
---
Fix Version/s: 0.96.0
Status: Patch Available (was: Open)
A workaround, proposed on the mailing list Friday 23th November.
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
>
>
> Key: HBASE-4709
> URL: https://issues.apache.org/jira/browse/HBASE-4709
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.94.1, 0.94.0, 0.92.0
>Reporter: Gary Helmling
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 4709_workaround.v1.patch
>
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been
> getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
> jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
> to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
> SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir:
> /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
> system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
> Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered
> with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> at
> com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> at
> com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> at
> com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by
> default, when running in a mini cluster. The errors themselves seem to be
> harmless -- they're not breaking any tests -- but we should figure out what
> configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503727#comment-13503727
]
Hadoop QA commented on HBASE-4709:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554834/4709_workaround.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3400//console
This message is automatically generated.
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
>
>
> Key: HBASE-4709
> URL: https://issues.apache.org/jira/browse/HBASE-4709
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.92.0, 0.94.0, 0.94.1
>Reporter: Gary Helmling
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 4709_workaround.v1.patch
>
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been
> getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
> jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
> to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
> SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir:
> /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
> system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
> Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered
> with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> at
> com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> at
> com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> at
> com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> at
> com.sun.jmx.interceptor.DefaultMBea
[jira] [Created] (HBASE-7217) Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set
Neil Ferguson created HBASE-7217: Summary: Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set Key: HBASE-7217 URL: https://issues.apache.org/jira/browse/HBASE-7217 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.94.2 Reporter: Neil Ferguson Priority: Minor When short-circuit reads are enabled (dfs.client.read.shortcircuit = true), reading with checksums enabled (dfs.client.read.shortcircuit.skip.checksum = false) follows a completely different, and much slower, path to reading with checksums disabled (BlockReaderLocal uses something called a "slow buffer", which is, unsurprisingly, slow). My tests show that this path is actually slower than having short-circuit reads disabled. Therefore, I think section 11.5.1 of the HBase documentation should recommend that hbase.regionserver.checksum.verify be set to true when using short-circuit reads. I'd suggest the following: "For optimal performance when short-circuit reads are enabled, it is recommended that HDFS checksums are disabled. To maintain data integrity with HDFS checksums disabled, HBase can be configured to write its own checksums into its datablocks and verify against these. See Section 11.4.9. "hbase.regionserver.checksum.verify". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7206) External Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7206: -- Status: Patch Available (was: Open) > External Exception framework v2 (simplifies and replaces HBASE-6571) > > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7206) External Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7206: -- Attachment: hbase-7206.v3.patch Uploaded v3 from review board. > External Exception framework v2 (simplifies and replaces HBASE-6571) > > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7206) External Exception framework v2 (simplifies and replaces HBASE-6571)
[
https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503784#comment-13503784
]
Hadoop QA commented on HBASE-7206:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554852/hbase-7206.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3401//console
This message is automatically generated.
> External Exception framework v2 (simplifies and replaces HBASE-6571)
>
>
> Key: HBASE-7206
> URL: https://issues.apache.org/jira/browse/HBASE-7206
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
> Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch
>
>
> This provides a way of sending exceptions from 'external' threads/processes
> (not the main executing thread) to others that poll cooperatively for
> external exceptions. Some examples of how this can be used include: having a
> separate timeout thread that injects an exception when a time limit has
> elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an
> exception from an separate process delivered to a local thread.
> This simplified version is centered around the ExternalException class.
> Instead of using generics and ErrorListener interfaces, this more
> straight-forward implementation eliminates many of the builders/factories and
> generics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7218) Rename Snapshot
Matteo Bertozzi created HBASE-7218: -- Summary: Rename Snapshot Key: HBASE-7218 URL: https://issues.apache.org/jira/browse/HBASE-7218 Project: HBase Issue Type: Sub-task Components: snapshots Affects Versions: hbase-6055 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: hbase-6055 Add the ability to rename a snapshot. HBaseAdmin.renameSnapshot(oldName, newName) shell: snapshot_rename 'oldName', 'newName' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7218) Rename Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7218: --- Attachment: HBASE-7218-v0.patch https://reviews.apache.org/r/8226/ > Rename Snapshot > --- > > Key: HBASE-7218 > URL: https://issues.apache.org/jira/browse/HBASE-7218 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: hbase-6055 > > Attachments: HBASE-7218-v0.patch > > > Add the ability to rename a snapshot. > HBaseAdmin.renameSnapshot(oldName, newName) > shell: snapshot_rename 'oldName', 'newName' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7218) Rename Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7218: --- Status: Patch Available (was: Open) > Rename Snapshot > --- > > Key: HBASE-7218 > URL: https://issues.apache.org/jira/browse/HBASE-7218 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: hbase-6055 > > Attachments: HBASE-7218-v0.patch > > > Add the ability to rename a snapshot. > HBaseAdmin.renameSnapshot(oldName, newName) > shell: snapshot_rename 'oldName', 'newName' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7218) Rename Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503821#comment-13503821
]
Hadoop QA commented on HBASE-7218:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554858/HBASE-7218-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3402//console
This message is automatically generated.
> Rename Snapshot
> ---
>
> Key: HBASE-7218
> URL: https://issues.apache.org/jira/browse/HBASE-7218
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: hbase-6055
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: hbase-6055
>
> Attachments: HBASE-7218-v0.patch
>
>
> Add the ability to rename a snapshot.
> HBaseAdmin.renameSnapshot(oldName, newName)
> shell: snapshot_rename 'oldName', 'newName'
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7218) Rename Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7218: --- Attachment: HBASE-7218-v1.patch > Rename Snapshot > --- > > Key: HBASE-7218 > URL: https://issues.apache.org/jira/browse/HBASE-7218 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: hbase-6055 > > Attachments: HBASE-7218-v0.patch, HBASE-7218-v1.patch > > > Add the ability to rename a snapshot. > HBaseAdmin.renameSnapshot(oldName, newName) > shell: snapshot_rename 'oldName', 'newName' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7218) Rename Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503911#comment-13503911
]
Hadoop QA commented on HBASE-7218:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554882/HBASE-7218-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3403//console
This message is automatically generated.
> Rename Snapshot
> ---
>
> Key: HBASE-7218
> URL: https://issues.apache.org/jira/browse/HBASE-7218
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: hbase-6055
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: hbase-6055
>
> Attachments: HBASE-7218-v0.patch, HBASE-7218-v1.patch
>
>
> Add the ability to rename a snapshot.
> HBaseAdmin.renameSnapshot(oldName, newName)
> shell: snapshot_rename 'oldName', 'newName'
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7219) Make it so can connect remotely to a standalone hbase
stack created HBASE-7219: Summary: Make it so can connect remotely to a standalone hbase Key: HBASE-7219 URL: https://issues.apache.org/jira/browse/HBASE-7219 Project: HBase Issue Type: Bug Reporter: stack Should be able to connect from a remote client to a standalone instance. HBase has 'localhost' in regionservers file and will write 'localhost' to znode for master location which remote client can't use. Fix. This comes up on mailing list w/ some frequency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7203) Move example CoProcessor into hbase-examples
[ https://issues.apache.org/jira/browse/HBASE-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7203: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed revision 1413751. > Move example CoProcessor into hbase-examples > > > Key: HBASE-7203 > URL: https://issues.apache.org/jira/browse/HBASE-7203 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: HBASE-7203-0.patch, HBASE-7203-1.patch > > > Move the example co-processor into the hbase-examples module. Also move the > protobuf definition files into the module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-74) [performance] When a get or scan request spans multiple columns, execute the reads in parallel
[ https://issues.apache.org/jira/browse/HBASE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-74: -- Affects Version/s: 0.96.0 > [performance] When a get or scan request spans multiple columns, execute the > reads in parallel > -- > > Key: HBASE-74 > URL: https://issues.apache.org/jira/browse/HBASE-74 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: Jim Kellerman > Fix For: 0.96.0 > > > When a get or scan request spans multiple columns, execute the reads in > parallel and use a CountDownLatch to wait for them to complete before > returning the results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-74) [performance] When a get or scan request spans multiple columns, execute the reads in parallel
[ https://issues.apache.org/jira/browse/HBASE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-74: -- Fix Version/s: 0.96.0 > [performance] When a get or scan request spans multiple columns, execute the > reads in parallel > -- > > Key: HBASE-74 > URL: https://issues.apache.org/jira/browse/HBASE-74 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: Jim Kellerman > Fix For: 0.96.0 > > > When a get or scan request spans multiple columns, execute the reads in > parallel and use a CountDownLatch to wait for them to complete before > returning the results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-74) [performance] When a get or scan request spans multiple columns, execute the reads in parallel
[ https://issues.apache.org/jira/browse/HBASE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503958#comment-13503958 ] Sergey Shelukhin commented on HBASE-74: --- Set. Thanks! > [performance] When a get or scan request spans multiple columns, execute the > reads in parallel > -- > > Key: HBASE-74 > URL: https://issues.apache.org/jira/browse/HBASE-74 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: Jim Kellerman > Fix For: 0.96.0 > > > When a get or scan request spans multiple columns, execute the reads in > parallel and use a CountDownLatch to wait for them to complete before > returning the results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503973#comment-13503973 ] Sergey Shelukhin commented on HBASE-7109: - Looks like compat project exclusion checking is not working. I'll check why, probably issues with absolute path in the package. If it's that I'll add jar filter... Any reason why compat project tests shouldn't be categorized? > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-squashed.patch, HBASE-7109-v2-squashed.patch, > HBASE-7109-v3-squashed.patch, HBASE-7109-v4-squashed.patch, > HBASE-7109-v5.patch, HBASE-7109-v5.patch, HBASE-7109-v6-0.94.patch, > HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503984#comment-13503984 ] Elliott Clark commented on HBASE-7109: -- The compat modules don't depend on hbase-common so they don't have the annotations. I was worried about getting the dependency graph too complex. > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-squashed.patch, HBASE-7109-v2-squashed.patch, > HBASE-7109-v3-squashed.patch, HBASE-7109-v4-squashed.patch, > HBASE-7109-v5.patch, HBASE-7109-v5.patch, HBASE-7109-v6-0.94.patch, > HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7190) Add an option to hbck to check only meta and assignment
[ https://issues.apache.org/jira/browse/HBASE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503985#comment-13503985 ] Hudson commented on HBASE-7190: --- Integrated in HBase-TRUNK #3564 (See [https://builds.apache.org/job/HBase-TRUNK/3564/]) HBASE-7190 Add an option to hbck to check only meta and assignment (Revision 1413762) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Add an option to hbck to check only meta and assignment > --- > > Key: HBASE-7190 > URL: https://issues.apache.org/jira/browse/HBASE-7190 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-7190.patch > > > Currently, hbck loads region info from HDFS for each run. It may take some > time if there are many regions. > We need an option to not check HDFS, i.e. just checking meta and assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7203) Move example CoProcessor into hbase-examples
[ https://issues.apache.org/jira/browse/HBASE-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503986#comment-13503986 ] Hudson commented on HBASE-7203: --- Integrated in HBase-TRUNK #3564 (See [https://builds.apache.org/job/HBase-TRUNK/3564/]) HBASE-7203 Move example CoProcessor into hbase-examples (Revision 1413751) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-examples/pom.xml * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/BulkDeleteEndpoint.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/ZooKeeperScanPolicyObserver.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/BulkDeleteProtos.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/ExampleProtos.java * /hbase/trunk/hbase-examples/src/main/protobuf * /hbase/trunk/hbase-examples/src/main/protobuf/BulkDelete.proto * /hbase/trunk/hbase-examples/src/main/protobuf/Examples.proto * /hbase/trunk/hbase-examples/src/test * /hbase/trunk/hbase-examples/src/test/java * /hbase/trunk/hbase-examples/src/test/java/org * /hbase/trunk/hbase-examples/src/test/java/org/apache * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRowCountEndpoint.java * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestZooKeeperScanPolicyObserver.java * /hbase/trunk/hbase-protocol/src/main/protobuf/BulkDelete.proto * /hbase/trunk/hbase-protocol/src/main/protobuf/Examples.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/example * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/example > Move example CoProcessor into hbase-examples > > > Key: HBASE-7203 > URL: https://issues.apache.org/jira/browse/HBASE-7203 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: HBASE-7203-0.patch, HBASE-7203-1.patch > > > Move the example co-processor into the hbase-examples module. Also move the > protobuf definition files into the module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
nkeywal created HBASE-7220:
--
Summary: Creating a table with 3000 regions on 2 nodes fails after
1 hour
Key: HBASE-7220
URL: https://issues.apache.org/jira/browse/HBASE-7220
Project: HBase
Issue Type: Bug
Components: metrics, Performance, regionserver
Affects Versions: 0.96.0
Environment: small test cluster. not reproducible on a local cluster
(needs a network).
Reporter: nkeywal
I'm trying to create a table with 3000 regions on two regions servers, from the
shell.
It's ok on trunk a standalone config.
It's ok on 0.94
It's not ok on trunk: it fails after around 1 hour.
If I remove all the code related to metrics in HRegion, the 3000 regions are
created in 3 minutes (twice faster than the 0.94).
On trunk, the region server spends its time in "waitForWork", while the master
is in the tcp connection related code. It's a 1Gb network.
I haven't looked at the metric code itself.
Patch used to remove the metrics from HRegion:
{noformat}
index c70e9ab..6677e65 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
private HTableDescriptor htableDescriptor = null;
private RegionSplitPolicy splitPolicy;
- private final MetricsRegion metricsRegion;
+ private final MetricsRegion metricsRegion = null;
/**
* Should only be used for testing purposes
@@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
this.coprocessorHost = null;
this.scannerReadPoints = new ConcurrentHashMap();
-this.metricsRegion = new MetricsRegion(new MetricsRegionWrapperImpl(this));
+//this.metricsRegion = new MetricsRegion(new
MetricsRegionWrapperImpl(this));
}
/**
@@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
this.scannerReadPoints = new ConcurrentHashMap();
-this.metricsRegion = new MetricsRegion(new MetricsRegionWrapperImpl(this));
+//this.metricsRegion = new MetricsRegion(new
MetricsRegionWrapperImpl(this));
/*
* timestamp.slop provides a server-side constraint on the timestamp. This
@@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // , Writable{
status.setStatus("Running coprocessor post-close hooks");
this.coprocessorHost.postClose(abort);
}
- this.metricsRegion.close();
+ //this.metricsRegion.close();
status.markComplete("Closed");
LOG.info("Closed " + this);
return result;
@@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
Writable{
if (noOfPuts > 0) {
// There were some Puts in the batch.
double noOfMutations = noOfPuts + noOfDeletes;
-this.metricsRegion.updatePut();
+//this.metricsRegion.updatePut();
}
if (noOfDeletes > 0) {
// There were some Deletes in the batch.
-this.metricsRegion.updateDelete();
+//this.metricsRegion.updateDelete();
}
if (!success) {
for (int i = firstIndex; i < lastIndexExclusive; i++) {
@@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // , Writable{
// do after lock
-this.metricsRegion.updateGet();
+//this.metricsRegion.updateGet();
return results;
}
@@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // , Writable{
closeRegionOperation();
}
-this.metricsRegion.updateAppend();
+//this.metricsRegion.updateAppend();
if (flush) {
@@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // , Writable{
mvcc.completeMemstoreInsert(w);
}
closeRegionOperation();
- this.metricsRegion.updateIncrement();
+ //this.metricsRegion.updateIncrement();
}
if (flush) {
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark reassigned HBASE-7220:
Assignee: Elliott Clark
> Creating a table with 3000 regions on 2 nodes fails after 1 hour
>
>
> Key: HBASE-7220
> URL: https://issues.apache.org/jira/browse/HBASE-7220
> Project: HBase
> Issue Type: Bug
> Components: metrics, Performance, regionserver
>Affects Versions: 0.96.0
> Environment: small test cluster. not reproducible on a local cluster
> (needs a network).
>Reporter: nkeywal
>Assignee: Elliott Clark
>
> I'm trying to create a table with 3000 regions on two regions servers, from
> the shell.
> It's ok on trunk a standalone config.
> It's ok on 0.94
> It's not ok on trunk: it fails after around 1 hour.
> If I remove all the code related to metrics in HRegion, the 3000 regions are
> created in 3 minutes (twice faster than the 0.94).
> On trunk, the region server spends its time in "waitForWork", while the
> master is in the tcp connection related code. It's a 1Gb network.
> I haven't looked at the metric code itself.
> Patch used to remove the metrics from HRegion:
> {noformat}
> index c70e9ab..6677e65 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
>private HTableDescriptor htableDescriptor = null;
>private RegionSplitPolicy splitPolicy;
>
> - private final MetricsRegion metricsRegion;
> + private final MetricsRegion metricsRegion = null;
>
>/**
> * Should only be used for testing purposes
> @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.coprocessorHost = null;
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>}
>
>/**
> @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>
> /*
> * timestamp.slop provides a server-side constraint on the timestamp.
> This
> @@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> status.setStatus("Running coprocessor post-close hooks");
> this.coprocessorHost.postClose(abort);
>}
> - this.metricsRegion.close();
> + //this.metricsRegion.close();
>status.markComplete("Closed");
>LOG.info("Closed " + this);
>return result;
> @@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
> Writable{
>if (noOfPuts > 0) {
> // There were some Puts in the batch.
> double noOfMutations = noOfPuts + noOfDeletes;
> -this.metricsRegion.updatePut();
> +//this.metricsRegion.updatePut();
>}
>if (noOfDeletes > 0) {
> // There were some Deletes in the batch.
> -this.metricsRegion.updateDelete();
> +//this.metricsRegion.updateDelete();
>}
>if (!success) {
> for (int i = firstIndex; i < lastIndexExclusive; i++) {
> @@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>
> // do after lock
>
> -this.metricsRegion.updateGet();
> +//this.metricsRegion.updateGet();
>
> return results;
>}
> @@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>closeRegionOperation();
> }
>
> -this.metricsRegion.updateAppend();
> +//this.metricsRegion.updateAppend();
>
>
> if (flush) {
> @@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> mvcc.completeMemstoreInsert(w);
>}
>closeRegionOperation();
> - this.metricsRegion.updateIncrement();
> + //this.metricsRegion.updateIncrement();
> }
>
> if (flush) {
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7220:
-
Environment: (was: small test cluster. not reproducible on a local
cluster (needs a network).)
> Creating a table with 3000 regions on 2 nodes fails after 1 hour
>
>
> Key: HBASE-7220
> URL: https://issues.apache.org/jira/browse/HBASE-7220
> Project: HBase
> Issue Type: Bug
> Components: metrics, Performance, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: Elliott Clark
>
> I'm trying to create a table with 3000 regions on two regions servers, from
> the shell.
> It's ok on trunk a standalone config.
> It's ok on 0.94
> It's not ok on trunk: it fails after around 1 hour.
> If I remove all the code related to metrics in HRegion, the 3000 regions are
> created in 3 minutes (twice faster than the 0.94).
> On trunk, the region server spends its time in "waitForWork", while the
> master is in the tcp connection related code. It's a 1Gb network.
> I haven't looked at the metric code itself.
> Patch used to remove the metrics from HRegion:
> {noformat}
> index c70e9ab..6677e65 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
>private HTableDescriptor htableDescriptor = null;
>private RegionSplitPolicy splitPolicy;
>
> - private final MetricsRegion metricsRegion;
> + private final MetricsRegion metricsRegion = null;
>
>/**
> * Should only be used for testing purposes
> @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.coprocessorHost = null;
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>}
>
>/**
> @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>
> /*
> * timestamp.slop provides a server-side constraint on the timestamp.
> This
> @@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> status.setStatus("Running coprocessor post-close hooks");
> this.coprocessorHost.postClose(abort);
>}
> - this.metricsRegion.close();
> + //this.metricsRegion.close();
>status.markComplete("Closed");
>LOG.info("Closed " + this);
>return result;
> @@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
> Writable{
>if (noOfPuts > 0) {
> // There were some Puts in the batch.
> double noOfMutations = noOfPuts + noOfDeletes;
> -this.metricsRegion.updatePut();
> +//this.metricsRegion.updatePut();
>}
>if (noOfDeletes > 0) {
> // There were some Deletes in the batch.
> -this.metricsRegion.updateDelete();
> +//this.metricsRegion.updateDelete();
>}
>if (!success) {
> for (int i = firstIndex; i < lastIndexExclusive; i++) {
> @@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>
> // do after lock
>
> -this.metricsRegion.updateGet();
> +//this.metricsRegion.updateGet();
>
> return results;
>}
> @@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>closeRegionOperation();
> }
>
> -this.metricsRegion.updateAppend();
> +//this.metricsRegion.updateAppend();
>
>
> if (flush) {
> @@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> mvcc.completeMemstoreInsert(w);
>}
>closeRegionOperation();
> - this.metricsRegion.updateIncrement();
> + //this.metricsRegion.updateIncrement();
> }
>
> if (flush) {
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503993#comment-13503993
]
Elliott Clark commented on HBASE-7220:
--
Just tried this locally and the reason this is taking so long is that when
creating a new MetricsRegion, the whole metrics system is stopped and then
started (to clear all jmx caches). This is taking a long time, and there is no
need to do this on the main thread. All that matters is that the metrics
system is restarted some time after the regions are added or removed.
I'll add a patch soon that moves this off of the main threads and into the
metrics executor.
Additionally we can merge several of the reset requests.
> Creating a table with 3000 regions on 2 nodes fails after 1 hour
>
>
> Key: HBASE-7220
> URL: https://issues.apache.org/jira/browse/HBASE-7220
> Project: HBase
> Issue Type: Bug
> Components: metrics, Performance, regionserver
>Affects Versions: 0.96.0
> Environment: small test cluster. not reproducible on a local cluster
> (needs a network).
>Reporter: nkeywal
>Assignee: Elliott Clark
>
> I'm trying to create a table with 3000 regions on two regions servers, from
> the shell.
> It's ok on trunk a standalone config.
> It's ok on 0.94
> It's not ok on trunk: it fails after around 1 hour.
> If I remove all the code related to metrics in HRegion, the 3000 regions are
> created in 3 minutes (twice faster than the 0.94).
> On trunk, the region server spends its time in "waitForWork", while the
> master is in the tcp connection related code. It's a 1Gb network.
> I haven't looked at the metric code itself.
> Patch used to remove the metrics from HRegion:
> {noformat}
> index c70e9ab..6677e65 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
>private HTableDescriptor htableDescriptor = null;
>private RegionSplitPolicy splitPolicy;
>
> - private final MetricsRegion metricsRegion;
> + private final MetricsRegion metricsRegion = null;
>
>/**
> * Should only be used for testing purposes
> @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.coprocessorHost = null;
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>}
>
>/**
> @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>
> /*
> * timestamp.slop provides a server-side constraint on the timestamp.
> This
> @@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> status.setStatus("Running coprocessor post-close hooks");
> this.coprocessorHost.postClose(abort);
>}
> - this.metricsRegion.close();
> + //this.metricsRegion.close();
>status.markComplete("Closed");
>LOG.info("Closed " + this);
>return result;
> @@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
> Writable{
>if (noOfPuts > 0) {
> // There were some Puts in the batch.
> double noOfMutations = noOfPuts + noOfDeletes;
> -this.metricsRegion.updatePut();
> +//this.metricsRegion.updatePut();
>}
>if (noOfDeletes > 0) {
> // There were some Deletes in the batch.
> -this.metricsRegion.updateDelete();
> +//this.metricsRegion.updateDelete();
>}
>if (!success) {
> for (int i = firstIndex; i < lastIndexExclusive; i++) {
> @@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>
> // do after lock
>
> -this.metricsRegion.updateGet();
> +//this.metricsRegion.updateGet();
>
> return results;
>}
> @@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>closeRegionOperation();
> }
>
> -this.metricsRegion.updateAppend();
> +//this.metricsRegion.updateAppend();
>
>
> if (flush) {
> @@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> mvcc.completeMemstoreInsert(w);
>}
>closeRegionOperation()
[jira] [Updated] (HBASE-7217) Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set
[ https://issues.apache.org/jira/browse/HBASE-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7217: - Attachment: 7217.txt Patch w/ Neil's fix (Move the 'HDFS Configuration' section under the 'HDFS' section that comes later too). > Documentation: Update section 11.5.1 to recommend that > hbase.regionserver.checksum.verify is set > > > Key: HBASE-7217 > URL: https://issues.apache.org/jira/browse/HBASE-7217 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.94.2 >Reporter: Neil Ferguson >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7217.txt > > > When short-circuit reads are enabled (dfs.client.read.shortcircuit = true), > reading with checksums enabled (dfs.client.read.shortcircuit.skip.checksum = > false) follows a completely different, and much slower, path to reading with > checksums disabled (BlockReaderLocal uses something called a "slow buffer", > which is, unsurprisingly, slow). My tests show that this path is actually > slower than having short-circuit reads disabled. > Therefore, I think section 11.5.1 of the HBase documentation should recommend > that hbase.regionserver.checksum.verify be set to true when using > short-circuit reads. I'd suggest the following: > "For optimal performance when short-circuit reads are enabled, it is > recommended that HDFS checksums are disabled. To maintain data integrity with > HDFS checksums disabled, HBase can be configured to write its own checksums > into its datablocks and verify against these. See Section 11.4.9. > "hbase.regionserver.checksum.verify". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7217) Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set
[ https://issues.apache.org/jira/browse/HBASE-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-7217. -- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Committed. Thanks for the fixup Neil. > Documentation: Update section 11.5.1 to recommend that > hbase.regionserver.checksum.verify is set > > > Key: HBASE-7217 > URL: https://issues.apache.org/jira/browse/HBASE-7217 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.94.2 >Reporter: Neil Ferguson >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7217.txt > > > When short-circuit reads are enabled (dfs.client.read.shortcircuit = true), > reading with checksums enabled (dfs.client.read.shortcircuit.skip.checksum = > false) follows a completely different, and much slower, path to reading with > checksums disabled (BlockReaderLocal uses something called a "slow buffer", > which is, unsurprisingly, slow). My tests show that this path is actually > slower than having short-circuit reads disabled. > Therefore, I think section 11.5.1 of the HBase documentation should recommend > that hbase.regionserver.checksum.verify be set to true when using > short-circuit reads. I'd suggest the following: > "For optimal performance when short-circuit reads are enabled, it is > recommended that HDFS checksums are disabled. To maintain data integrity with > HDFS checksums disabled, HBase can be configured to write its own checksums > into its datablocks and verify against these. See Section 11.4.9. > "hbase.regionserver.checksum.verify". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504010#comment-13504010
]
stack commented on HBASE-4709:
--
+1 if the TestHCM is not because of this patch. Thanks for doing this N.
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
>
>
> Key: HBASE-4709
> URL: https://issues.apache.org/jira/browse/HBASE-4709
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.92.0, 0.94.0, 0.94.1
>Reporter: Gary Helmling
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 4709_workaround.v1.patch
>
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been
> getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
> jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
> to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
> SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir:
> /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
> system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
> Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered
> with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> at
> com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> at
> com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> at
> com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by
> default, when running in a mini cluster. The errors themselves seem to be
> harmless -- they're not breaking any tests -- but we should figure out what
> configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7190) Add an option to hbck to check only meta and assignment
[ https://issues.apache.org/jira/browse/HBASE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7190: --- Resolution: Fixed Fix Version/s: 0.96.0 0.94.3 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Add an option to hbck to check only meta and assignment > --- > > Key: HBASE-7190 > URL: https://issues.apache.org/jira/browse/HBASE-7190 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.94.3, 0.96.0 > > Attachments: trunk-7190.patch > > > Currently, hbck loads region info from HDFS for each run. It may take some > time if there are many regions. > We need an option to not check HDFS, i.e. just checking meta and assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504023#comment-13504023 ] Lars Hofhansl commented on HBASE-7215: -- How about I make a patch with that, and then we can see how to improve it. I assume we want to avoid importing generated protobuf classes in Put/Delete (because they are in the client interface), which complicates things further. > Put, Delete, Increment, and Result still implement Writable > --- > > Key: HBASE-7215 > URL: https://issues.apache.org/jira/browse/HBASE-7215 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Blocker > Fix For: 0.96.0 > > > Making blocker as suggested by Stack. > At least the following still use Put/Delete as writables. > * IdentityTableReduce.java > * MultiPut.java > * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504025#comment-13504025 ] stack commented on HBASE-7215: -- I thought the MR requirement that we do Writables was undone by HBASE-7024? My guess is there is more work to be done there still > Put, Delete, Increment, and Result still implement Writable > --- > > Key: HBASE-7215 > URL: https://issues.apache.org/jira/browse/HBASE-7215 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Blocker > Fix For: 0.96.0 > > > Making blocker as suggested by Stack. > At least the following still use Put/Delete as writables. > * IdentityTableReduce.java > * MultiPut.java > * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504028#comment-13504028
]
Elliott Clark commented on HBASE-4709:
--
Should the static logger definitions be moved into hbase-hadoop{1|2}-compat?
Though right now it works because hadoop 1 and 2 both happen to have that class
in the same location.
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
>
>
> Key: HBASE-4709
> URL: https://issues.apache.org/jira/browse/HBASE-4709
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.92.0, 0.94.0, 0.94.1
>Reporter: Gary Helmling
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 4709_workaround.v1.patch
>
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been
> getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
> jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
> to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
> SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir:
> /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
> system not started: Cannot locate configuration: tried
> hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
> Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered
> with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> at
> com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> at
> com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> at
> com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:280)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by
> default, when running in a mini cluster. The errors themselves seem to be
> harmless -- they're not breaking any tests -- but we should figure out what
> configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504032#comment-13504032 ] stack commented on HBASE-7213: -- Yes > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7220:
-
Attachment: HBASE-7220-0.patch
This patch for me moved creating a table with 9k regions from 110 seconds to 52
seconds.
* Created a cache buster runnable
* async clears the jmx cache
* merges any clears that happen close together.
> Creating a table with 3000 regions on 2 nodes fails after 1 hour
>
>
> Key: HBASE-7220
> URL: https://issues.apache.org/jira/browse/HBASE-7220
> Project: HBase
> Issue Type: Bug
> Components: metrics, Performance, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: Elliott Clark
> Attachments: HBASE-7220-0.patch
>
>
> I'm trying to create a table with 3000 regions on two regions servers, from
> the shell.
> It's ok on trunk a standalone config.
> It's ok on 0.94
> It's not ok on trunk: it fails after around 1 hour.
> If I remove all the code related to metrics in HRegion, the 3000 regions are
> created in 3 minutes (twice faster than the 0.94).
> On trunk, the region server spends its time in "waitForWork", while the
> master is in the tcp connection related code. It's a 1Gb network.
> I haven't looked at the metric code itself.
> Patch used to remove the metrics from HRegion:
> {noformat}
> index c70e9ab..6677e65 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
>private HTableDescriptor htableDescriptor = null;
>private RegionSplitPolicy splitPolicy;
>
> - private final MetricsRegion metricsRegion;
> + private final MetricsRegion metricsRegion = null;
>
>/**
> * Should only be used for testing purposes
> @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.coprocessorHost = null;
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>}
>
>/**
> @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>
> /*
> * timestamp.slop provides a server-side constraint on the timestamp.
> This
> @@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> status.setStatus("Running coprocessor post-close hooks");
> this.coprocessorHost.postClose(abort);
>}
> - this.metricsRegion.close();
> + //this.metricsRegion.close();
>status.markComplete("Closed");
>LOG.info("Closed " + this);
>return result;
> @@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
> Writable{
>if (noOfPuts > 0) {
> // There were some Puts in the batch.
> double noOfMutations = noOfPuts + noOfDeletes;
> -this.metricsRegion.updatePut();
> +//this.metricsRegion.updatePut();
>}
>if (noOfDeletes > 0) {
> // There were some Deletes in the batch.
> -this.metricsRegion.updateDelete();
> +//this.metricsRegion.updateDelete();
>}
>if (!success) {
> for (int i = firstIndex; i < lastIndexExclusive; i++) {
> @@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>
> // do after lock
>
> -this.metricsRegion.updateGet();
> +//this.metricsRegion.updateGet();
>
> return results;
>}
> @@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>closeRegionOperation();
> }
>
> -this.metricsRegion.updateAppend();
> +//this.metricsRegion.updateAppend();
>
>
> if (flush) {
> @@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> mvcc.completeMemstoreInsert(w);
>}
>closeRegionOperation();
> - this.metricsRegion.updateIncrement();
> + //this.metricsRegion.updateIncrement();
> }
>
> if (flush) {
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7214) CleanerChore logs too much, so much so it obscures all else that is going on
[
https://issues.apache.org/jira/browse/HBASE-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504037#comment-13504037
]
stack commented on HBASE-7214:
--
+1 on commit. On commit, I'd wrap the log messages in and if (TRACE) to avoid
our making objects we won't use.
I find the logging annoying. Would be +1 on committing to 0.94 for 0.94.4 (not
enough to sink an RC IMO).
> CleanerChore logs too much, so much so it obscures all else that is going on
>
>
> Key: HBASE-7214
> URL: https://issues.apache.org/jira/browse/HBASE-7214
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3
>Reporter: stack
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.4
>
> Attachments: hbase-7214-v0.patch
>
>
> Testing 0.94.3RC0, I see loads of this in logs:
> {code}
> 2012-11-23 13:39:40,488 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
> file:/tmp/hbase-stack/hbase/.oldlogs/192.168.1.73%2C61033%2C1353705880078.1353706219757
> is not deletable according t#
> 14 2012-11-23 13:39:40,488 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
> file:/tmp/hbase-stack/hbase/.oldlogs/192.168.1.73%2C61033%2C1353705880078.1353706221210
> is not deletable according t#
> 1
> {code}
> There is too much of it. Can we cut back some?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7214) CleanerChore logs too much, so much so it obscures all else that is going on
[
https://issues.apache.org/jira/browse/HBASE-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504039#comment-13504039
]
stack commented on HBASE-7214:
--
You going to commit then Jesse? Mr Committer?
> CleanerChore logs too much, so much so it obscures all else that is going on
>
>
> Key: HBASE-7214
> URL: https://issues.apache.org/jira/browse/HBASE-7214
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3
>Reporter: stack
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.4
>
> Attachments: hbase-7214-v0.patch
>
>
> Testing 0.94.3RC0, I see loads of this in logs:
> {code}
> 2012-11-23 13:39:40,488 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
> file:/tmp/hbase-stack/hbase/.oldlogs/192.168.1.73%2C61033%2C1353705880078.1353706219757
> is not deletable according t#
> 14 2012-11-23 13:39:40,488 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
> file:/tmp/hbase-stack/hbase/.oldlogs/192.168.1.73%2C61033%2C1353705880078.1353706221210
> is not deletable according t#
> 1
> {code}
> There is too much of it. Can we cut back some?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7155) Excessive usage of InterruptedException where it can't be thrown
[ https://issues.apache.org/jira/browse/HBASE-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504041#comment-13504041 ] stack commented on HBASE-7155: -- bq. The problem is that the IE is being suppressed in ZKUtil, which swallows the IEs in all its methods. Which is kinda broke I'd say. bq. If we were to expose all those IEs the change would be huge. Yes. Agree. An ugly big change for what likes little value. bq. If you want to go that route I suggest opening a new ticket and closing this one. What you think we should do D? > Excessive usage of InterruptedException where it can't be thrown > > > Key: HBASE-7155 > URL: https://issues.apache.org/jira/browse/HBASE-7155 > Project: HBase > Issue Type: Bug >Reporter: Daniel Gómez Ferro >Assignee: Daniel Gómez Ferro > Attachments: HBASE-7155.patch > > > RootRegionTracker.getRootRegionLocation() declares that it can throw an > InterruptedException, but it can't. This exception is rethrown by many other > functions reaching the HBaseAdmin API. > If we remove the throws statement from the HBaseAdmin API libraries already > compiled will work fine, but if the user is trying to catch an > InterruptedException around one of those methods the compiler will complain. > Should we clean this up? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7214) CleanerChore logs too much, so much so it obscures all else that is going on
[
https://issues.apache.org/jira/browse/HBASE-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504042#comment-13504042
]
Jesse Yates commented on HBASE-7214:
[~saint@gmail.com] I'll get on it... this afternoon
> CleanerChore logs too much, so much so it obscures all else that is going on
>
>
> Key: HBASE-7214
> URL: https://issues.apache.org/jira/browse/HBASE-7214
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3
>Reporter: stack
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.4
>
> Attachments: hbase-7214-v0.patch
>
>
> Testing 0.94.3RC0, I see loads of this in logs:
> {code}
> 2012-11-23 13:39:40,488 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
> file:/tmp/hbase-stack/hbase/.oldlogs/192.168.1.73%2C61033%2C1353705880078.1353706219757
> is not deletable according t#
> 14 2012-11-23 13:39:40,488 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
> file:/tmp/hbase-stack/hbase/.oldlogs/192.168.1.73%2C61033%2C1353705880078.1353706221210
> is not deletable according t#
> 1
> {code}
> There is too much of it. Can we cut back some?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7220:
-
Status: Patch Available (was: Open)
> Creating a table with 3000 regions on 2 nodes fails after 1 hour
>
>
> Key: HBASE-7220
> URL: https://issues.apache.org/jira/browse/HBASE-7220
> Project: HBase
> Issue Type: Bug
> Components: metrics, Performance, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: Elliott Clark
> Attachments: HBASE-7220-0.patch
>
>
> I'm trying to create a table with 3000 regions on two regions servers, from
> the shell.
> It's ok on trunk a standalone config.
> It's ok on 0.94
> It's not ok on trunk: it fails after around 1 hour.
> If I remove all the code related to metrics in HRegion, the 3000 regions are
> created in 3 minutes (twice faster than the 0.94).
> On trunk, the region server spends its time in "waitForWork", while the
> master is in the tcp connection related code. It's a 1Gb network.
> I haven't looked at the metric code itself.
> Patch used to remove the metrics from HRegion:
> {noformat}
> index c70e9ab..6677e65 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
>private HTableDescriptor htableDescriptor = null;
>private RegionSplitPolicy splitPolicy;
>
> - private final MetricsRegion metricsRegion;
> + private final MetricsRegion metricsRegion = null;
>
>/**
> * Should only be used for testing purposes
> @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.coprocessorHost = null;
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>}
>
>/**
> @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>
> /*
> * timestamp.slop provides a server-side constraint on the timestamp.
> This
> @@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> status.setStatus("Running coprocessor post-close hooks");
> this.coprocessorHost.postClose(abort);
>}
> - this.metricsRegion.close();
> + //this.metricsRegion.close();
>status.markComplete("Closed");
>LOG.info("Closed " + this);
>return result;
> @@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
> Writable{
>if (noOfPuts > 0) {
> // There were some Puts in the batch.
> double noOfMutations = noOfPuts + noOfDeletes;
> -this.metricsRegion.updatePut();
> +//this.metricsRegion.updatePut();
>}
>if (noOfDeletes > 0) {
> // There were some Deletes in the batch.
> -this.metricsRegion.updateDelete();
> +//this.metricsRegion.updateDelete();
>}
>if (!success) {
> for (int i = firstIndex; i < lastIndexExclusive; i++) {
> @@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>
> // do after lock
>
> -this.metricsRegion.updateGet();
> +//this.metricsRegion.updateGet();
>
> return results;
>}
> @@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
>closeRegionOperation();
> }
>
> -this.metricsRegion.updateAppend();
> +//this.metricsRegion.updateAppend();
>
>
> if (flush) {
> @@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // ,
> Writable{
> mvcc.completeMemstoreInsert(w);
>}
>closeRegionOperation();
> - this.metricsRegion.updateIncrement();
> + //this.metricsRegion.updateIncrement();
> }
>
> if (flush) {
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7217) Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set
[ https://issues.apache.org/jira/browse/HBASE-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504057#comment-13504057 ] Hudson commented on HBASE-7217: --- Integrated in HBase-TRUNK #3565 (See [https://builds.apache.org/job/HBase-TRUNK/3565/]) HBASE-7217 Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set (Revision 1413793) Result = FAILURE stack : Files : * /hbase/trunk/src/docbkx/performance.xml > Documentation: Update section 11.5.1 to recommend that > hbase.regionserver.checksum.verify is set > > > Key: HBASE-7217 > URL: https://issues.apache.org/jira/browse/HBASE-7217 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.94.2 >Reporter: Neil Ferguson >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7217.txt > > > When short-circuit reads are enabled (dfs.client.read.shortcircuit = true), > reading with checksums enabled (dfs.client.read.shortcircuit.skip.checksum = > false) follows a completely different, and much slower, path to reading with > checksums disabled (BlockReaderLocal uses something called a "slow buffer", > which is, unsurprisingly, slow). My tests show that this path is actually > slower than having short-circuit reads disabled. > Therefore, I think section 11.5.1 of the HBase documentation should recommend > that hbase.regionserver.checksum.verify be set to true when using > short-circuit reads. I'd suggest the following: > "For optimal performance when short-circuit reads are enabled, it is > recommended that HDFS checksums are disabled. To maintain data integrity with > HDFS checksums disabled, HBase can be configured to write its own checksums > into its datablocks and verify against these. See Section 11.4.9. > "hbase.regionserver.checksum.verify". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7190) Add an option to hbck to check only meta and assignment
[ https://issues.apache.org/jira/browse/HBASE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504085#comment-13504085 ] Hudson commented on HBASE-7190: --- Integrated in HBase-0.94 #601 (See [https://builds.apache.org/job/HBase-0.94/601/]) HBASE-7190 Add an option to hbck to check only meta and assignment (Revision 1413800) Result = FAILURE jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Add an option to hbck to check only meta and assignment > --- > > Key: HBASE-7190 > URL: https://issues.apache.org/jira/browse/HBASE-7190 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.94.3, 0.96.0 > > Attachments: trunk-7190.patch > > > Currently, hbck loads region info from HDFS for each run. It may take some > time if there are many regions. > We need an option to not check HDFS, i.e. just checking meta and assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504098#comment-13504098
]
Hadoop QA commented on HBASE-7220:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554899/HBASE-7220-0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 26 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3404//console
This message is automatically generated.
> Creating a table with 3000 regions on 2 nodes fails after 1 hour
>
>
> Key: HBASE-7220
> URL: https://issues.apache.org/jira/browse/HBASE-7220
> Project: HBase
> Issue Type: Bug
> Components: metrics, Performance, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: Elliott Clark
> Attachments: HBASE-7220-0.patch
>
>
> I'm trying to create a table with 3000 regions on two regions servers, from
> the shell.
> It's ok on trunk a standalone config.
> It's ok on 0.94
> It's not ok on trunk: it fails after around 1 hour.
> If I remove all the code related to metrics in HRegion, the 3000 regions are
> created in 3 minutes (twice faster than the 0.94).
> On trunk, the region server spends its time in "waitForWork", while the
> master is in the tcp connection related code. It's a 1Gb network.
> I haven't looked at the metric code itself.
> Patch used to remove the metrics from HRegion:
> {noformat}
> index c70e9ab..6677e65 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
>private HTableDescriptor htableDescriptor = null;
>private RegionSplitPolicy splitPolicy;
>
> - private final MetricsRegion metricsRegion;
> + private final MetricsRegion metricsRegion = null;
>
>/**
> * Should only be used for testing purposes
> @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.coprocessorHost = null;
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
> +//this.metricsRegion = new MetricsRegion(new
> MetricsRegionWrapperImpl(this));
>}
>
>/**
> @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
> this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> this.scannerReadPoints = new ConcurrentHashMap();
>
> -this.metricsRegion = new MetricsRegion(new
>
[jira] [Updated] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7109: Attachment: HBASE-7109-fix.patch Jar absolute path is merely a package path... added filtering on actual jar path and c/p-ed tests. > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HBASE-7109: - breaks mvn package > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7109: Status: Patch Available (was: Reopened) > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504128#comment-13504128 ] stack commented on HBASE-7215: -- In hadoop, there is a http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/io/serializer/SerializationFactory.html It defaults Writables. We'd make it default pb in 0.96. > Put, Delete, Increment, and Result still implement Writable > --- > > Key: HBASE-7215 > URL: https://issues.apache.org/jira/browse/HBASE-7215 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Blocker > Fix For: 0.96.0 > > > Making blocker as suggested by Stack. > At least the following still use Put/Delete as writables. > * IdentityTableReduce.java > * MultiPut.java > * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504157#comment-13504157 ] Lars Hofhansl commented on HBASE-7215: -- I see, cool. Makes we want to delete my comment above, but I'll refrain :) > Put, Delete, Increment, and Result still implement Writable > --- > > Key: HBASE-7215 > URL: https://issues.apache.org/jira/browse/HBASE-7215 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Blocker > Fix For: 0.96.0 > > > Making blocker as suggested by Stack. > At least the following still use Put/Delete as writables. > * IdentityTableReduce.java > * MultiPut.java > * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7221) RowKey utility class for rowkey construction
Doug Meil created HBASE-7221:
Summary: RowKey utility class for rowkey construction
Key: HBASE-7221
URL: https://issues.apache.org/jira/browse/HBASE-7221
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
A common question in the dist-lists is how to construct rowkeys, particularly
composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
you to sensibly populate that byte-array, and that's where things tend to go
off the rails.
The intent of this RowKey utility class isn't meant to add functionality into
Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
Example:
{code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Meil updated HBASE-7221:
-
Status: Patch Available (was: Open)
> RowKey utility class for rowkey construction
>
>
> Key: HBASE-7221
> URL: https://issues.apache.org/jira/browse/HBASE-7221
> Project: HBase
> Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBASE_7221.patch
>
>
> A common question in the dist-lists is how to construct rowkeys, particularly
> composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
> you to sensibly populate that byte-array, and that's where things tend to go
> off the rails.
> The intent of this RowKey utility class isn't meant to add functionality into
> Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
> Example:
> {code}
>RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
>key.addHash(a);
>key.add(b);
>byte bytes[] = key.getBytes();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Meil updated HBASE-7221:
-
Attachment: HBASE_7221.patch
> RowKey utility class for rowkey construction
>
>
> Key: HBASE-7221
> URL: https://issues.apache.org/jira/browse/HBASE-7221
> Project: HBase
> Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBASE_7221.patch
>
>
> A common question in the dist-lists is how to construct rowkeys, particularly
> composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
> you to sensibly populate that byte-array, and that's where things tend to go
> off the rails.
> The intent of this RowKey utility class isn't meant to add functionality into
> Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
> Example:
> {code}
>RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
>key.addHash(a);
>key.add(b);
>byte bytes[] = key.getBytes();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7168) [dev] in the script called 'hbase', we don't check for errors when generating the classpath with mvn
[ https://issues.apache.org/jira/browse/HBASE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7168: - Attachment: HBASE-7168-ADD-0.patch Seems like the maven_opts things were messing me up. Also as I was adding more modules to hbase this was messed up as the ordering was not the one required. So here's a little addendum that fixed things for me. > [dev] in the script called 'hbase', we don't check for errors when generating > the classpath with mvn > > > Key: HBASE-7168 > URL: https://issues.apache.org/jira/browse/HBASE-7168 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.96.0 > Environment: dev >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7168.v1.patch, HBASE-7168-ADD-0.patch > > > When it happens, it's difficult to guess. Let's fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7168) [dev] in the script called 'hbase', we don't check for errors when generating the classpath with mvn
[ https://issues.apache.org/jira/browse/HBASE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reopened HBASE-7168: -- > [dev] in the script called 'hbase', we don't check for errors when generating > the classpath with mvn > > > Key: HBASE-7168 > URL: https://issues.apache.org/jira/browse/HBASE-7168 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.96.0 > Environment: dev >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7168.v1.patch, HBASE-7168-ADD-0.patch > > > When it happens, it's difficult to guess. Let's fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7168) [dev] in the script called 'hbase', we don't check for errors when generating the classpath with mvn
[ https://issues.apache.org/jira/browse/HBASE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504183#comment-13504183 ] stack commented on HBASE-7168: -- +1 on this patch after talking to Elliott about 'portability' -- the paste command is probably not present on many *nx -- but after chatting, this fix is only for devs, lets just get this in for now. Can address portability in another issue. > [dev] in the script called 'hbase', we don't check for errors when generating > the classpath with mvn > > > Key: HBASE-7168 > URL: https://issues.apache.org/jira/browse/HBASE-7168 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.96.0 > Environment: dev >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7168.v1.patch, HBASE-7168-ADD-0.patch > > > When it happens, it's difficult to guess. Let's fix this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7211) Improve hbase ref guide for the testing part.
[ https://issues.apache.org/jira/browse/HBASE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504186#comment-13504186 ] stack commented on HBASE-7211: -- These look like great changes N. On IT stuff, thats HBASE-6302 > Improve hbase ref guide for the testing part. > - > > Key: HBASE-7211 > URL: https://issues.apache.org/jira/browse/HBASE-7211 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > > Here is some stuff I saw. I will propose a fix in a week or so, please add > the comment or issues you have in mind. > ??15.6.1. Apache HBase Modules?? > => We should be able to use categories in all modules. The default should be > small; but any test manipulating the time needs to be in a specific jvm > (hence medium), so it's not always related to minicluster. > ??15.6.3.6. hbasetests.sh?? > => We can remove this chapter, and the script > The script is not totally useless, but I think nobody actually uses it. > => Add a chapter on flakiness. > Some tests are, unfortunately, flaky. While there number decreases, we still > have some. Rules are: > - don't write flaky tests! :-) > - small tests cannot be flaky, as it blocks other test execution. Corollary: > if you have an issue with a small test, it's either your environment either a > severe issue. > - rerun the test a few time to validate, check the ports and file descriptors > used. > ??mvn test -P localTests -Dtest=MyTest?? > => We could actually activate the localTests profile whenever -Dtest is used. > If we do that, we can remove the reference from localTests in the doc. > ??mvn test -P runSmallTests?? ??mvn test -P runMediumTests?? > => I'm not sure it's actually used. We could remove them from the pom.xml > (and the doc). > ??The HBase build uses a patched version of the maven surefire plugin?? > => Hopefully, we will be able to remove this soon :-) > ??Integration tests are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION?? > => Should be documented -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504190#comment-13504190 ] Jonathan Hsieh commented on HBASE-7208: --- This looks like a big patch but most of it is mechanical due to a renames. Highlights of what has been done: * Removed old error handling framework, many renames (HBaseException -> ExtenralException, failOnError -> rethrowException) * Moved the body of snapshot functions into the SnapshotManager. Better encapsulation and makes exception layering much clearer (HMaster throws SerivceExceptions, SnapshotManager throws IO and HBaseSnapshot exn). * Added cancel method to SnapshotSentinel * Refactored DisabledTableSnapshotHandler into TakeSnapshotHandler (This is the online-scaffolding patch essentially) * Converted *Task to Callable instead of Runnable so that Exceptions can be thrown normally if called in line and centralized use of the ExternalException infra to only where necessary. Compared to previous version -- there are no generics, fewer classes, more straightforward errorhandling, and better encapsulation of snapshotting methods within the SnapshotManager. > Transition Offline Snapshots to ExternalExceptions > -- > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504191#comment-13504191 ] Jonathan Hsieh commented on HBASE-7208: --- Reviewboard available here https://reviews.apache.org/r/8229/ > Transition Offline Snapshots to ExternalExceptions > -- > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7208: -- Attachment: hbase-7208.v6.patch > Transition Offline Snapshots to ExternalExceptions > -- > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7208: -- Attachment: pre-hbase-7208.v6.patch > Transition Offline Snapshots to ExternalExceptions > -- > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch, pre-hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7208: -- Status: Patch Available (was: Open) > Transition Offline Snapshots to ExternalExceptions > -- > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch, pre-hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504195#comment-13504195 ] Jonathan Hsieh commented on HBASE-7208: --- Patch build ontop of Jesse's snapshots branch hash 8e58f35f98. pre-hbase-7208.v6 goes on top of a particular apache hbase commit, and then the hbase-7208.v6.patch can be applied on top of it. This is also available here: https://github.com/jmhsieh/hbase jmhsieh/snapshot-work-1126-2 hash 25da4512 (one parent is working version, the other is the review ready version). > Transition Offline Snapshots to ExternalExceptions > -- > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch, pre-hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7212) Globally Barriered Procedure mechanism
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh reassigned HBASE-7212: - Assignee: Jonathan Hsieh > Globally Barriered Procedure mechanism > -- > > Key: HBASE-7212 > URL: https://issues.apache.org/jira/browse/HBASE-7212 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > > This is a simplified version of what was proposed in HBASE-6573. Instead of > claiming to be a 2pc or 3pc implementation (which implies logging at each > actor, and recovery operations) this is just provides a best effort global > barrier mechanism called a Procedure. > Users need only to implement a methods to acquireBarrier, to act when > insideBarrier, and to releaseBarrier that use the ExternalException > cooperative error checking mechanism. > Globally consistent snapshots require the ability to quiesce writes to a set > of region servers before a the snapshot operation is executed. Also if any > node fails, it needs to be able to notify them so that they abort. > The first cut of other online snapshots don't need the fully barrier but may > still use this for its error propagation mechanisms. > This version removes the extra layer incurred in the previous implementation > due to the use of generics, separates the coordinator and members, and > reduces the amount of inheritance used in favor of composition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6573) Distributed Three-Phase Commit framework.
[ https://issues.apache.org/jira/browse/HBASE-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-6573. --- Resolution: Incomplete Superceded by HBASE-7212 > Distributed Three-Phase Commit framework. > - > > Key: HBASE-6573 > URL: https://issues.apache.org/jira/browse/HBASE-6573 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0 >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: hbase-6055 > > Attachments: java_HBASE-6573-v0.patch, Procedure coordination.pdf > > > For HBASE-6055 (snapshots), we do two-phase commit in several places. This is > a generally useful paradigm for a distributed system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[
https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504197#comment-13504197
]
Hadoop QA commented on HBASE-7109:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554917/HBASE-7109-fix.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3405//console
This message is automatically generated.
> integration tests on cluster are not getting picked up from distribution
>
>
> Key: HBASE-7109
> URL: https://issues.apache.org/jira/browse/HBASE-7109
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch,
> HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch,
> HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch,
> HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch
>
>
> The method of finding test classes only works on local build (or its full
> copy), not if the distribution is used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7208) Transition Offline Snapshots to ExternalExceptions
[
https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504198#comment-13504198
]
Hadoop QA commented on HBASE-7208:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554928/pre-hbase-7208.v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3407//console
This message is automatically generated.
> Transition Offline Snapshots to ExternalExceptions
> --
>
> Key: HBASE-7208
> URL: https://issues.apache.org/jira/browse/HBASE-7208
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: hbase-7208.v6.patch, pre-hbase-7208.v6.patch
>
>
> This will eliminate the old errorhandling code, and update existing code to
> use the ExternalException mechanisms.
> I'd like this to be done before attempt merging to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3533) Allow HBASE_LIBRARY_PATH env var to specify extra locations of native libs
[ https://issues.apache.org/jira/browse/HBASE-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504202#comment-13504202 ] stack commented on HBASE-3533: -- @Andres It seems like trunk has changed again from what you cite in the above. What would fix it for you? Seems like we want to get hadoop additions in there if available. You have a patch? Thanks. > Allow HBASE_LIBRARY_PATH env var to specify extra locations of native libs > -- > > Key: HBASE-3533 > URL: https://issues.apache.org/jira/browse/HBASE-3533 > Project: HBase > Issue Type: Improvement >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hbase-3533.txt > > > Would be handy when you have native libs at other spots on the system (eg I > often want to test hadoop-lzo changes directly out of its build dir) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504208#comment-13504208 ] stack commented on HBASE-7109: -- Your patch breaks mvn package Sergey? What about the above failure? Thanks. > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7110: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch Sergey. > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-v6-squashed.patch, HBASE-7110-v7.patch, HBASE-7110-v8.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504210#comment-13504210 ] Sergey Shelukhin commented on HBASE-7109: - original patch breaks mvn package, as it finds compat tests in the jar and determines they have no annotation; that is due to there being no module path inside package; without path compat couldn't be filtered out. The above test passed on local run... > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504216#comment-13504216
]
Hadoop QA commented on HBASE-7221:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554921/HBASE_7221.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 27 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3406//console
This message is automatically generated.
> RowKey utility class for rowkey construction
>
>
> Key: HBASE-7221
> URL: https://issues.apache.org/jira/browse/HBASE-7221
> Project: HBase
> Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBASE_7221.patch
>
>
> A common question in the dist-lists is how to construct rowkeys, particularly
> composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
> you to sensibly populate that byte-array, and that's where things tend to go
> off the rails.
> The intent of this RowKey utility class isn't meant to add functionality into
> Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
> Example:
> {code}
>RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
>key.addHash(a);
>key.add(b);
>byte bytes[] = key.getBytes();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504217#comment-13504217 ] Sergey Shelukhin commented on HBASE-7110: - um, this shouldn't have been committed to trunk, I still haven't looked at TestHeapSize failure, I will probably do today. Sorry for confusion. > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-v6-squashed.patch, HBASE-7110-v7.patch, HBASE-7110-v8.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504222#comment-13504222 ] stack commented on HBASE-7109: -- Why make below protected? - private static final Log LOG = LogFactory.getLog(ClassFinder.class); + protected static final Log LOG = LogFactory.getLog(ClassFinder.class); So an Interface implementation has access? Make its own logger instead? For future, we usually write this kinda thing 'null == this.resourcePathFilter', the other way round. Otherwise +1 on commit > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7211) Improve hbase ref guide for the testing part.
[ https://issues.apache.org/jira/browse/HBASE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504221#comment-13504221 ] Enis Soztutar commented on HBASE-7211: -- bq. => We should be able to use categories in all modules. The default should be small; but any test manipulating the time needs to be in a specific jvm (hence medium), so it's not always related to minicluster. Right now, the only exception is hadoop-compat modules. See Elliot's comment: https://issues.apache.org/jira/browse/HBASE-7109?focusedCommentId=13503984&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13503984 bq. => Add a chapter on flakiness. +1. bq. => We could actually activate the localTests profile whenever -Dtest is used. If we do that, we can remove the reference from localTests in the doc. +1. -PlocalTets is not intuitive, and I was burned a couple of times by this. By having active in profile, this should be easy to do. bq. => Integration tests are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION We can just commit HBASE-6302. > Improve hbase ref guide for the testing part. > - > > Key: HBASE-7211 > URL: https://issues.apache.org/jira/browse/HBASE-7211 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > > Here is some stuff I saw. I will propose a fix in a week or so, please add > the comment or issues you have in mind. > ??15.6.1. Apache HBase Modules?? > => We should be able to use categories in all modules. The default should be > small; but any test manipulating the time needs to be in a specific jvm > (hence medium), so it's not always related to minicluster. > ??15.6.3.6. hbasetests.sh?? > => We can remove this chapter, and the script > The script is not totally useless, but I think nobody actually uses it. > => Add a chapter on flakiness. > Some tests are, unfortunately, flaky. While there number decreases, we still > have some. Rules are: > - don't write flaky tests! :-) > - small tests cannot be flaky, as it blocks other test execution. Corollary: > if you have an issue with a small test, it's either your environment either a > severe issue. > - rerun the test a few time to validate, check the ports and file descriptors > used. > ??mvn test -P localTests -Dtest=MyTest?? > => We could actually activate the localTests profile whenever -Dtest is used. > If we do that, we can remove the reference from localTests in the doc. > ??mvn test -P runSmallTests?? ??mvn test -P runMediumTests?? > => I'm not sure it's actually used. We could remove them from the pom.xml > (and the doc). > ??The HBase build uses a patched version of the maven surefire plugin?? > => Hopefully, we will be able to remove this soon :-) > ??Integration tests are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION?? > => Should be documented -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7110: Attachment: HBASE-7110-v9.patch fixed size calculation. > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-v6-squashed.patch, HBASE-7110-v7.patch, HBASE-7110-v8.patch, > HBASE-7110-v9.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504227#comment-13504227 ] Enis Soztutar commented on HBASE-7109: -- bq. For future, we usually write this kinda thing 'null == this.resourcePathFilter', the other way round. That's called Yoda Conditionals :) http://www.codinghorror.com/blog/2012/07/new-programming-jargon.html > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504230#comment-13504230 ] Enis Soztutar commented on HBASE-7213: -- This can also allow us to directly deploy META without waiting for log splitting (if we have separate WAL for every META region, there is nothing to split). > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504245#comment-13504245 ] Hudson commented on HBASE-7110: --- Integrated in HBase-TRUNK #3566 (See [https://builds.apache.org/job/HBase-TRUNK/3566/]) HBASE-7110 refactor the compaction selection and config code similarly to 0.89-fb changes (Revision 1413912) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfiguration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreUtils.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-v6-squashed.patch, HBASE-7110-v7.patch, HBASE-7110-v8.patch, > HBASE-7110-v9.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504247#comment-13504247 ] Hudson commented on HBASE-7110: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #274 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/274/]) HBASE-7110 refactor the compaction selection and config code similarly to 0.89-fb changes (Revision 1413912) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfiguration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreUtils.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-v6-squashed.patch, HBASE-7110-v7.patch, HBASE-7110-v8.patch, > HBASE-7110-v9.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7190) Add an option to hbck to check only meta and assignment
[ https://issues.apache.org/jira/browse/HBASE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504248#comment-13504248 ] Hudson commented on HBASE-7190: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #274 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/274/]) HBASE-7190 Add an option to hbck to check only meta and assignment (Revision 1413762) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Add an option to hbck to check only meta and assignment > --- > > Key: HBASE-7190 > URL: https://issues.apache.org/jira/browse/HBASE-7190 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.94.3, 0.96.0 > > Attachments: trunk-7190.patch > > > Currently, hbck loads region info from HDFS for each run. It may take some > time if there are many regions. > We need an option to not check HDFS, i.e. just checking meta and assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7217) Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set
[ https://issues.apache.org/jira/browse/HBASE-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504250#comment-13504250 ] Hudson commented on HBASE-7217: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #274 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/274/]) HBASE-7217 Documentation: Update section 11.5.1 to recommend that hbase.regionserver.checksum.verify is set (Revision 1413793) Result = FAILURE stack : Files : * /hbase/trunk/src/docbkx/performance.xml > Documentation: Update section 11.5.1 to recommend that > hbase.regionserver.checksum.verify is set > > > Key: HBASE-7217 > URL: https://issues.apache.org/jira/browse/HBASE-7217 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.94.2 >Reporter: Neil Ferguson >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7217.txt > > > When short-circuit reads are enabled (dfs.client.read.shortcircuit = true), > reading with checksums enabled (dfs.client.read.shortcircuit.skip.checksum = > false) follows a completely different, and much slower, path to reading with > checksums disabled (BlockReaderLocal uses something called a "slow buffer", > which is, unsurprisingly, slow). My tests show that this path is actually > slower than having short-circuit reads disabled. > Therefore, I think section 11.5.1 of the HBase documentation should recommend > that hbase.regionserver.checksum.verify be set to true when using > short-circuit reads. I'd suggest the following: > "For optimal performance when short-circuit reads are enabled, it is > recommended that HDFS checksums are disabled. To maintain data integrity with > HDFS checksums disabled, HBase can be configured to write its own checksums > into its datablocks and verify against these. See Section 11.4.9. > "hbase.regionserver.checksum.verify". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7203) Move example CoProcessor into hbase-examples
[ https://issues.apache.org/jira/browse/HBASE-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504249#comment-13504249 ] Hudson commented on HBASE-7203: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #274 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/274/]) HBASE-7203 Move example CoProcessor into hbase-examples (Revision 1413751) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-examples/pom.xml * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/BulkDeleteEndpoint.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/ZooKeeperScanPolicyObserver.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/BulkDeleteProtos.java * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/ExampleProtos.java * /hbase/trunk/hbase-examples/src/main/protobuf * /hbase/trunk/hbase-examples/src/main/protobuf/BulkDelete.proto * /hbase/trunk/hbase-examples/src/main/protobuf/Examples.proto * /hbase/trunk/hbase-examples/src/test * /hbase/trunk/hbase-examples/src/test/java * /hbase/trunk/hbase-examples/src/test/java/org * /hbase/trunk/hbase-examples/src/test/java/org/apache * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRowCountEndpoint.java * /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestZooKeeperScanPolicyObserver.java * /hbase/trunk/hbase-protocol/src/main/protobuf/BulkDelete.proto * /hbase/trunk/hbase-protocol/src/main/protobuf/Examples.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/example * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/example > Move example CoProcessor into hbase-examples > > > Key: HBASE-7203 > URL: https://issues.apache.org/jira/browse/HBASE-7203 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: HBASE-7203-0.patch, HBASE-7203-1.patch > > > Move the example co-processor into the hbase-examples module. Also move the > protobuf definition files into the module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504281#comment-13504281 ] stack commented on HBASE-7109: -- That is an excellent link Enis. > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-squashed.patch, > HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, > HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, > HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7109: Attachment: HBASE-7109-fix-v2.patch Fixed the visibility; wrt conditionals - sure; C++ habit :) > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-fix-v2.patch, > HBASE-7109-squashed.patch, HBASE-7109-v2-squashed.patch, > HBASE-7109-v3-squashed.patch, HBASE-7109-v4-squashed.patch, > HBASE-7109-v5.patch, HBASE-7109-v5.patch, HBASE-7109-v6-0.94.patch, > HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7215: - Attachment: 7251-SKETCH.txt Sketch of a patch. The only testing I have done is making sure it compiles. * adds Serialization for Mutation and Result * rejigger out M/R support to use that * remove all Writable stuff from Put, Delete, Increment, and Result. Please have a careful look, this is new'ish territory for me. > Put, Delete, Increment, and Result still implement Writable > --- > > Key: HBASE-7215 > URL: https://issues.apache.org/jira/browse/HBASE-7215 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7251-SKETCH.txt > > > Making blocker as suggested by Stack. > At least the following still use Put/Delete as writables. > * IdentityTableReduce.java > * MultiPut.java > * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-7215: Assignee: Lars Hofhansl > Put, Delete, Increment, and Result still implement Writable > --- > > Key: HBASE-7215 > URL: https://issues.apache.org/jira/browse/HBASE-7215 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7251-SKETCH.txt > > > Making blocker as suggested by Stack. > At least the following still use Put/Delete as writables. > * IdentityTableReduce.java > * MultiPut.java > * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HBASE-7110: - > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-follow-up.patch, HBASE-7110-v6-squashed.patch, > HBASE-7110-v7.patch, HBASE-7110-v8.patch, HBASE-7110-v9.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7110) refactor the compaction selection and config code similarly to 0.89-fb changes
[ https://issues.apache.org/jira/browse/HBASE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7110: Attachment: HBASE-7110-follow-up.patch Patch was committed before test was fixed... attaching followup patch for convenience. > refactor the compaction selection and config code similarly to 0.89-fb changes > -- > > Key: HBASE-7110 > URL: https://issues.apache.org/jira/browse/HBASE-7110 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-v5-refactor-only-squashed.patch, > HBASE-7110-follow-up.patch, HBASE-7110-v6-squashed.patch, > HBASE-7110-v7.patch, HBASE-7110-v8.patch, HBASE-7110-v9.patch > > > Separate JIRA for refactoring changes from HBASE-7055 (and further ones after > code review) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[
https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504309#comment-13504309
]
Hadoop QA commented on HBASE-7109:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554943/HBASE-7109-fix-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
96 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 25 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3408//console
This message is automatically generated.
> integration tests on cluster are not getting picked up from distribution
>
>
> Key: HBASE-7109
> URL: https://issues.apache.org/jira/browse/HBASE-7109
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: HBASE-7109-fix.patch, HBASE-7109-fix-v2.patch,
> HBASE-7109-squashed.patch, HBASE-7109-v2-squashed.patch,
> HBASE-7109-v3-squashed.patch, HBASE-7109-v4-squashed.patch,
> HBASE-7109-v5.patch, HBASE-7109-v5.patch, HBASE-7109-v6-0.94.patch,
> HBASE-7109-v6.patch
>
>
> The method of finding test classes only works on local build (or its full
> copy), not if the distribution is used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504312#comment-13504312 ] Sergey Shelukhin commented on HBASE-7109: - test fails due to HBASE-7110 > integration tests on cluster are not getting picked up from distribution > > > Key: HBASE-7109 > URL: https://issues.apache.org/jira/browse/HBASE-7109 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-7109-fix.patch, HBASE-7109-fix-v2.patch, > HBASE-7109-squashed.patch, HBASE-7109-v2-squashed.patch, > HBASE-7109-v3-squashed.patch, HBASE-7109-v4-squashed.patch, > HBASE-7109-v5.patch, HBASE-7109-v5.patch, HBASE-7109-v6-0.94.patch, > HBASE-7109-v6.patch > > > The method of finding test classes only works on local build (or its full > copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7200) create integration test for balancing regions and killing region servers
[ https://issues.apache.org/jira/browse/HBASE-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7200: Attachment: HBASE-7200-v0.patch > create integration test for balancing regions and killing region servers > > > Key: HBASE-7200 > URL: https://issues.apache.org/jira/browse/HBASE-7200 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7200-v0.patch > > > See related JIRA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7200) create integration test for balancing regions and killing region servers
[ https://issues.apache.org/jira/browse/HBASE-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7200: Status: Patch Available (was: Open) Initial patch. Test passed at least once, but may be unstable, when parameters were more aggressive I've seen some put failures with no useful logging to explain them. > create integration test for balancing regions and killing region servers > > > Key: HBASE-7200 > URL: https://issues.apache.org/jira/browse/HBASE-7200 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7200-v0.patch > > > See related JIRA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7200) create integration test for balancing regions and killing region servers
[ https://issues.apache.org/jira/browse/HBASE-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504316#comment-13504316 ] Sergey Shelukhin commented on HBASE-7200: - (on minicluster) > create integration test for balancing regions and killing region servers > > > Key: HBASE-7200 > URL: https://issues.apache.org/jira/browse/HBASE-7200 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7200-v0.patch > > > See related JIRA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7200) create integration test for balancing regions and killing region servers
[
https://issues.apache.org/jira/browse/HBASE-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504335#comment-13504335
]
Hadoop QA commented on HBASE-7200:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554953/HBASE-7200-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
96 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 25 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3409//console
This message is automatically generated.
> create integration test for balancing regions and killing region servers
>
>
> Key: HBASE-7200
> URL: https://issues.apache.org/jira/browse/HBASE-7200
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7200-v0.patch
>
>
> See related JIRA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6639) Class.newInstance() can throw any checked exceptions and must be encapsulated with catching Exception
[ https://issues.apache.org/jira/browse/HBASE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-6639: - Attachment: HBASE-6639.patch Added a patch. Encapsulated with catching Exception, with fixing/refactoring some code. > Class.newInstance() can throw any checked exceptions and must be encapsulated > with catching Exception > - > > Key: HBASE-6639 > URL: https://issues.apache.org/jira/browse/HBASE-6639 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-6639.patch > > > There are some logics to call Class.newInstance() without catching Exception, > for example, in the method CoprocessorHost.loadInstance(). > Class.newInstance() is declared to throw InstantiationException and > IllegalAccessException but indeed the method can throw any checked exceptions > without declaration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7200) create integration test for balancing regions and killing region servers
[ https://issues.apache.org/jira/browse/HBASE-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504347#comment-13504347 ] Sergey Shelukhin commented on HBASE-7200: - looks like master tends to lose meta region somewhere... I'll take a more detailed look tomorrow or later, meanwhile patch can in principle be reviewed - this is probably a bug in code; unless the issue is due to unbalancer and there's something special about moving meta region via HBaseAdmin that I don't know about > create integration test for balancing regions and killing region servers > > > Key: HBASE-7200 > URL: https://issues.apache.org/jira/browse/HBASE-7200 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7200-v0.patch > > > See related JIRA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504352#comment-13504352
]
ramkrishna.s.vasudevan commented on HBASE-7221:
---
{code}
addHash(s.getBytes());
{code}
Can we make this too as 'addHash(Bytes.toBytes(s));?
> RowKey utility class for rowkey construction
>
>
> Key: HBASE-7221
> URL: https://issues.apache.org/jira/browse/HBASE-7221
> Project: HBase
> Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBASE_7221.patch
>
>
> A common question in the dist-lists is how to construct rowkeys, particularly
> composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
> you to sensibly populate that byte-array, and that's where things tend to go
> off the rails.
> The intent of this RowKey utility class isn't meant to add functionality into
> Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
> Example:
> {code}
>RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
>key.addHash(a);
>key.add(b);
>byte bytes[] = key.getBytes();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504356#comment-13504356
]
Elliott Clark commented on HBASE-7221:
--
Seems like this would be better in the hbase-examples module.
> RowKey utility class for rowkey construction
>
>
> Key: HBASE-7221
> URL: https://issues.apache.org/jira/browse/HBASE-7221
> Project: HBase
> Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBASE_7221.patch
>
>
> A common question in the dist-lists is how to construct rowkeys, particularly
> composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
> you to sensibly populate that byte-array, and that's where things tend to go
> off the rails.
> The intent of this RowKey utility class isn't meant to add functionality into
> Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
> Example:
> {code}
>RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
>key.addHash(a);
>key.add(b);
>byte bytes[] = key.getBytes();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6639) Class.newInstance() can throw any checked exceptions and must be encapsulated with catching Exception
[ https://issues.apache.org/jira/browse/HBASE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504360#comment-13504360 ] Hiroshi Ikeda commented on HBASE-6639: -- Review board: https://reviews.apache.org/r/8230/ > Class.newInstance() can throw any checked exceptions and must be encapsulated > with catching Exception > - > > Key: HBASE-6639 > URL: https://issues.apache.org/jira/browse/HBASE-6639 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-6639.patch > > > There are some logics to call Class.newInstance() without catching Exception, > for example, in the method CoprocessorHost.loadInstance(). > Class.newInstance() is declared to throw InstantiationException and > IllegalAccessException but indeed the method can throw any checked exceptions > without declaration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
