[jira] [Commented] (HBASE-7507) Make memstore flush be able to retry after exception
[ https://issues.apache.org/jira/browse/HBASE-7507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558722#comment-13558722 ] Hudson commented on HBASE-7507: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #364 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/]) HBASE-7507 Make memstore flush be able to retry after exception (Chunhui) (Revision 1436111) Result = FAILURE zjushch : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java > Make memstore flush be able to retry after exception > > > Key: HBASE-7507 > URL: https://issues.apache.org/jira/browse/HBASE-7507 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.5 > > Attachments: 7507-94.patch, 7507-trunk v1.patch, 7507-trunk v2.patch, > 7507-trunkv3.patch > > > We will abort regionserver if memstore flush throws exception. > I thinks we could do retry to make regionserver more stable because file > system may be not ok in a transient time. e.g. Switching namenode in the > NamenodeHA environment > {code} > HRegion#internalFlushcache(){ > ... > try { > ... > }catch(Throwable t){ > DroppedSnapshotException dse = new DroppedSnapshotException("region: " + > Bytes.toStringBinary(getRegionName())); > dse.initCause(t); > throw dse; > } > ... > } > MemStoreFlusher#flushRegion(){ > ... > region.flushcache(); > ... > try { > }catch(DroppedSnapshotException ex){ > server.abort("Replay of HLog required. Forcing server shutdown", ex); > } > ... > } > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7633) Add a metric that tracks the current number of used RPC threads on the regionservers
Joey Echeverria created HBASE-7633: -- Summary: Add a metric that tracks the current number of used RPC threads on the regionservers Key: HBASE-7633 URL: https://issues.apache.org/jira/browse/HBASE-7633 Project: HBase Issue Type: Improvement Reporter: Joey Echeverria One way to detect that you're hitting a "John Wayne" disk[1] would be if we could see when region servers exhausted their RPC handlers. This would also be useful when tuning the cluster for your workload to make sure that reads or writes were not starving the other operations out. [1] http://hbase.apache.org/book.html#bad.disk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6907) KeyValue equals and compareTo methods should match
[ https://issues.apache.org/jira/browse/HBASE-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558861#comment-13558861 ] Ted Yu commented on HBASE-6907: --- Integrated to trunk. Thanks for the review, Stack and Matt. > KeyValue equals and compareTo methods should match > -- > > Key: HBASE-6907 > URL: https://issues.apache.org/jira/browse/HBASE-6907 > Project: HBase > Issue Type: Bug > Components: util >Reporter: Matt Corgan >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6907-v1.txt, 6907-v2.txt, 6907-v3.txt, 6907-v4.txt, > 6907-v5.txt > > > KeyValue.KVComparator includes the memstoreTS when comparing, however the > KeyValue.equals() method ignores the memstoreTS. > The Comparator interface has always specified that comparator return 0 when > equals would return true and vice versa. Obeying that rule has been sort of > optional in the past, but Java 7 introduces a new default collection sorting > algorithm called Tim Sort which relies on that behavior. > http://bugs.sun.com/view_bug.do?bug_id=6804124 > Possible problem spots: > * there's a Collections.sort(KeyValues) in > RedundantKVGenerator.generateTestKeyValues(..) > * TestColumnSeeking compares two collections of KeyValues using the > containsAll method. It is intentionally ignoring memstoreTS, so will need an > alternative method for comparing the two collections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
Gabriel Reid created HBASE-7634: --- Summary: Replication handling of changes to peer clusters is inefficient Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.96.0 Reporter: Gabriel Reid Attachments: HBASE-7634.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated HBASE-7634: Attachment: HBASE-7634.patch Initial patch to resolve this issue. Adds a watcher to replication peer's list of region servers to respond to changes in the list of region servers. Also changes checking randomly-chosen peer regionservers with the ability to report a "bad" peer regionserver. When a bad peer regionserver has been reported three times, it is no longer used for replication until the list of replication peer regionservers is refreshed. > Replication handling of changes to peer clusters is inefficient > --- > > Key: HBASE-7634 > URL: https://issues.apache.org/jira/browse/HBASE-7634 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.96.0 >Reporter: Gabriel Reid > Attachments: HBASE-7634.patch > > > The current handling of changes to the region servers in a replication peer > cluster is currently quite inefficient. The list of region servers that are > being replicated to is only updated if there are a large number of issues > encountered while replicating. > This can cause it to take quite a while to recognize that a number of the > regionserver in a peer cluster are no longer available. A potentially bigger > problem is that if a replication peer cluster is started with a small number > of regionservers, and then more region servers are added after replication > has started, the additional region servers will never be used for replication > (unless there are failures on the in-use regionservers). > Part of the current issue is that the retry code in > ReplicationSource#shipEdits checks a randomly-chosen replication peer > regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a > replication write has failed on a different randonly-chosen replication peer. > If the peer is seen as not down, another randomly-chosen peer is used for > writing. > A second part of the issue is that changes to the list of region servers in a > peer cluster are not detected at all, and are only picked up if a certain > number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7635) HFileSystem should implement Closeable
Ted Yu created HBASE-7635: - Summary: HFileSystem should implement Closeable Key: HBASE-7635 URL: https://issues.apache.org/jira/browse/HBASE-7635 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 >From >https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : {code} 2013-01-21 11:49:26,141 ERROR [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1792081] server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not Closeable or does not provide closeable invocation handler class $Proxy20 at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) at org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) at org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) at org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Attachment: 7635.txt Straightforward patch. > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Status: Patch Available (was: Open) > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Baugher updated HBASE-3996: - Attachment: 3996-v12.txt Update to latest trunk which had a conflict > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Bryan Baugher >Priority: Critical > Fix For: 0.96.0, 0.94.5 > > Attachments: 3996-v10.txt, 3996-v11.txt, 3996-v12.txt, 3996-v2.txt, > 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, 3996-v8.txt, > 3996-v9.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558889#comment-13558889 ] stack commented on HBASE-7635: -- HFileSystem implements FilterFileSystem which implements FileSystem which implement Closeable (am I missing something?). Given this, why we need this patch? > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558890#comment-13558890 ] Bryan Baugher commented on HBASE-3996: -- I believe there are two possible questions left unanswered as well as some +1's still needed, * The changes to TableSplit would not allow a new version of it to be deserialized by an old server. Is that OK for a M/R job? * It has been mentioned to scope this to scans (of a single table) rather then multiple tables. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Bryan Baugher >Priority: Critical > Fix For: 0.96.0, 0.94.5 > > Attachments: 3996-v10.txt, 3996-v11.txt, 3996-v12.txt, 3996-v2.txt, > 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, 3996-v8.txt, > 3996-v9.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-3996: -- Status: Patch Available (was: Open) @Bryan: Can you upload patch to review board ? Thanks > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Bryan Baugher >Priority: Critical > Fix For: 0.96.0, 0.94.5 > > Attachments: 3996-v10.txt, 3996-v11.txt, 3996-v12.txt, 3996-v2.txt, > 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, 3996-v8.txt, > 3996-v9.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558899#comment-13558899 ] Bryan Baugher commented on HBASE-3996: -- Done, https://reviews.apache.org/r/9042/diff/ > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Bryan Baugher >Priority: Critical > Fix For: 0.96.0, 0.94.5 > > Attachments: 3996-v10.txt, 3996-v11.txt, 3996-v12.txt, 3996-v2.txt, > 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, 3996-v8.txt, > 3996-v9.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Status: Open (was: Patch Available) > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Attachment: (was: 7635.txt) > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-5664: -- Attachment: HBASE-5664_Trunk.patch Patch for Trunk. Let me try against HadoopQA > CP hooks in Scan flow for fast forward when filter filters out a row > > > Key: HBASE-5664 > URL: https://issues.apache.org/jira/browse/HBASE-5664 > Project: HBase > Issue Type: Improvement > Components: Coprocessors >Affects Versions: 0.92.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-5664_94.patch, HBASE-5664_Trunk.patch > > > In HRegion.nextInternal(int limit, String metric) > We have while(true) loop so as to fetch a next result which satisfies > filter condition. When Filter filters out the current fetched row we call > nextRow(byte [] currentRow) before going with the next row. > {code} > if (results.isEmpty() || filterRow()) { > // this seems like a redundant step - we already consumed the row > // there're no left overs. > // the reasons for calling this method are: > // 1. reset the filters. > // 2. provide a hook to fast forward the row (used by subclasses) > nextRow(currentRow); > {code} > // 2. provide a hook to fast forward the row (used by subclasses) > We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-5664: -- Status: Patch Available (was: Open) > CP hooks in Scan flow for fast forward when filter filters out a row > > > Key: HBASE-5664 > URL: https://issues.apache.org/jira/browse/HBASE-5664 > Project: HBase > Issue Type: Improvement > Components: Coprocessors >Affects Versions: 0.92.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-5664_94.patch, HBASE-5664_Trunk.patch > > > In HRegion.nextInternal(int limit, String metric) > We have while(true) loop so as to fetch a next result which satisfies > filter condition. When Filter filters out the current fetched row we call > nextRow(byte [] currentRow) before going with the next row. > {code} > if (results.isEmpty() || filterRow()) { > // this seems like a redundant step - we already consumed the row > // there're no left overs. > // the reasons for calling this method are: > // 1. reset the filters. > // 2. provide a hook to fast forward the row (used by subclasses) > nextRow(currentRow); > {code} > // 2. provide a hook to fast forward the row (used by subclasses) > We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-5664: -- Component/s: Filters > CP hooks in Scan flow for fast forward when filter filters out a row > > > Key: HBASE-5664 > URL: https://issues.apache.org/jira/browse/HBASE-5664 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, Filters >Affects Versions: 0.92.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-5664_94.patch, HBASE-5664_Trunk.patch > > > In HRegion.nextInternal(int limit, String metric) > We have while(true) loop so as to fetch a next result which satisfies > filter condition. When Filter filters out the current fetched row we call > nextRow(byte [] currentRow) before going with the next row. > {code} > if (results.isEmpty() || filterRow()) { > // this seems like a redundant step - we already consumed the row > // there're no left overs. > // the reasons for calling this method are: > // 1. reset the filters. > // 2. provide a hook to fast forward the row (used by subclasses) > nextRow(currentRow); > {code} > // 2. provide a hook to fast forward the row (used by subclasses) > We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7633) Add a metric that tracks the current number of used RPC threads on the regionservers
[ https://issues.apache.org/jira/browse/HBASE-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-7633: Assignee: Elliott Clark We have anything like this currently Mr Metric? > Add a metric that tracks the current number of used RPC threads on the > regionservers > > > Key: HBASE-7633 > URL: https://issues.apache.org/jira/browse/HBASE-7633 > Project: HBase > Issue Type: Improvement >Reporter: Joey Echeverria >Assignee: Elliott Clark > > One way to detect that you're hitting a "John Wayne" disk[1] would be if we > could see when region servers exhausted their RPC handlers. This would also > be useful when tuning the cluster for your workload to make sure that reads > or writes were not starving the other operations out. > [1] http://hbase.apache.org/book.html#bad.disk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3170: - Attachment: 3170v5.txt v5 factors out a bit of common code into a method. Patch looks good to me. We could probably just throw an exception if you try and pass a Get a null row too... put the query out of its misery earlier rather than later. Was wondering about the test. We spin up a mini cluster instance to do the null key test. Should this test just be added to another test that has already put up a cluster? Good stuff lads. > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-v2.patch, > 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0550 seconds > {code} > This isn't a parsing problem with the command-line of the shell. I can > reproduce this behavior both with plain Java code and with my asynchbase > client. > Since I don't actually have a row with an empty row key, I expected that the > first {{get}} would return nothing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Status: Patch Available (was: Open) > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Attachment: 7635.txt Thanks for the review, Stack. See if this patch is better. > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6907) KeyValue equals and compareTo methods should match
[ https://issues.apache.org/jira/browse/HBASE-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558927#comment-13558927 ] Hudson commented on HBASE-6907: --- Integrated in HBase-TRUNK #3773 (See [https://builds.apache.org/job/HBase-TRUNK/3773/]) HBASE-6907 KeyValue equals and compareTo methods should match (Ted Yu) (Revision 1436434) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueTestUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java > KeyValue equals and compareTo methods should match > -- > > Key: HBASE-6907 > URL: https://issues.apache.org/jira/browse/HBASE-6907 > Project: HBase > Issue Type: Bug > Components: util >Reporter: Matt Corgan >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6907-v1.txt, 6907-v2.txt, 6907-v3.txt, 6907-v4.txt, > 6907-v5.txt > > > KeyValue.KVComparator includes the memstoreTS when comparing, however the > KeyValue.equals() method ignores the memstoreTS. > The Comparator interface has always specified that comparator return 0 when > equals would return true and vice versa. Obeying that rule has been sort of > optional in the past, but Java 7 introduces a new default collection sorting > algorithm called Tim Sort which relies on that behavior. > http://bugs.sun.com/view_bug.do?bug_id=6804124 > Possible problem spots: > * there's a Collections.sort(KeyValues) in > RedundantKVGenerator.generateTestKeyValues(..) > * TestColumnSeeking compares two collections of KeyValues using the > containsAll method. It is intentionally ignoring memstoreTS, so will need an > alternative method for comparing the two collections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558928#comment-13558928 ] stack commented on HBASE-7635: -- Does it fix the issue? > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7636) TestDistributedLogSplitting#testThreeRSAbort fails against hadoop 2.0
Ted Yu created HBASE-7636: - Summary: TestDistributedLogSplitting#testThreeRSAbort fails against hadoop 2.0 Key: HBASE-7636 URL: https://issues.apache.org/jira/browse/HBASE-7636 Project: HBase Issue Type: Bug Reporter: Ted Yu Fix For: 0.96.0 >From >https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : {code} 2013-01-21 11:49:34,276 DEBUG [MASTER_SERVER_OPERATIONS-juno.apache.org,57966,1358768818594-0] client.HConnectionManager$HConnectionImplementation(956): Looked up root region location, connection=hconnection 0x12f19fe; serverName=juno.apache.org,55531,1358768819479 2013-01-21 11:49:34,278 INFO [MASTER_SERVER_OPERATIONS-juno.apache.org,57966,1358768818594-0] catalog.CatalogTracker(576): Failed verification of .META.,,1 at address=juno.apache.org,57582,1358768819456; org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: juno.apache.org/67.195.138.61:57582 {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558930#comment-13558930 ] Ted Yu commented on HBASE-7635: --- Yes. I issued the following command: mvn clean test -Dhadoop.profile=2.0 -Dtest=TestDistributedLogSplitting#testThreeRSAbort I kept monitoring / searching org.apache.hadoop.hbase.master.TestDistributedLogSplitting-output.txt for above mentioned exception message. There was none. > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558932#comment-13558932 ] stack commented on HBASE-7635: -- +1 > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558934#comment-13558934 ] Hadoop QA commented on HBASE-3996: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565806/3996-v12.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4108//console This message is automatically generated. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Bryan Baugher >Priority: Critical > Fix For: 0.96.0, 0.94.5 > > Attachments: 3996-v10.txt, 3996-v11.txt, 3996-v12.txt, 3996-v2.txt, > 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, 3996-v8.txt, > 3996-v9.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558937#comment-13558937 ] Ted Yu commented on HBASE-5664: --- {code} + * @return Returns whether more rows are available for the scanner or not. {code} Remove 'Returns' {code} +RegionScannerImpl(Scan scan, List additionalScanners, HRegion region) throws IOException { {code} Wrap long line above. {code} + * @param s + * @param currentRow + * @return + * @throws IOException + */ + public boolean postScannerFilterRow(final InternalScanner s, final byte[] currentRow) {code} Please complete javadoc. See if the test failures in TestFromClientSideWithCoprocessor and TestFromClientSide are related to this patch. > CP hooks in Scan flow for fast forward when filter filters out a row > > > Key: HBASE-5664 > URL: https://issues.apache.org/jira/browse/HBASE-5664 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, Filters >Affects Versions: 0.92.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-5664_94.patch, HBASE-5664_Trunk.patch > > > In HRegion.nextInternal(int limit, String metric) > We have while(true) loop so as to fetch a next result which satisfies > filter condition. When Filter filters out the current fetched row we call > nextRow(byte [] currentRow) before going with the next row. > {code} > if (results.isEmpty() || filterRow()) { > // this seems like a redundant step - we already consumed the row > // there're no left overs. > // the reasons for calling this method are: > // 1. reset the filters. > // 2. provide a hook to fast forward the row (used by subclasses) > nextRow(currentRow); > {code} > // 2. provide a hook to fast forward the row (used by subclasses) > We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one
[ https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558938#comment-13558938 ] Himanshu Vashishtha commented on HBASE-2611: [~lhofhansl]: Yes, I followed the same approach in the attached patch. > Handle RS that fails while processing the failure of another one > > > Key: HBASE-2611 > URL: https://issues.apache.org/jira/browse/HBASE-2611 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Jean-Daniel Cryans >Assignee: Himanshu Vashishtha > Fix For: 0.96.0, 0.94.5 > > Attachments: HBase-2611-upstream-v1.patch, HBASE-2611-v2.patch > > > HBASE-2223 doesn't manage region servers that fail while doing the transfer > of HLogs queues from other region servers that failed. Devise a reliable way > to do it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558939#comment-13558939 ] Devaraj Das commented on HBASE-3170: bq. We could probably just throw an exception if you try and pass a Get a null row too.. Was wondering about the test. Ok [~stack], will review the patch from that point of view.. > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-v2.patch, > 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0550 seconds > {code} > This isn't a parsing problem with the command-line of the shell. I can > reproduce this behavior both with plain Java code and with my asynchbase > client. > Since I don't actually have a row with an empty row key, I expected that the > first {{get}} would return nothing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
nkeywal created HBASE-7637: -- Summary: hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 Key: HBASE-7637 URL: https://issues.apache.org/jira/browse/HBASE-7637 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.96.0 Reporter: nkeywal Priority: Critical I'm unclear on the root cause / fix. Here is the scenario: {noformat} mvn clean package install -Dhadoop.profile=2.0 -DskipTests bin/start-hbase.sh {noformat} fails with {noformat} Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.lib.MetricMutable at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) {noformat} doing {noformat} rm -rf hbase-hadoop1-compat/target/ {noformat} makes it work. In the pom.xml, we never reference hadoop2-compat. But doing so does not help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558941#comment-13558941 ] Hadoop QA commented on HBASE-3170: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565810/3170v5.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestLocalHBaseCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4110//console This message is automatically generated. > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-v2.patch, > 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo
[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558942#comment-13558942 ] Hadoop QA commented on HBASE-5664: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565809/HBASE-5664_Trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.security.access.TestAccessController org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol org.apache.hadoop.hbase.TestLocalHBaseCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4109//console This message is automatically generated. > CP hooks in Scan flow for fast forward when filter filters out a row > > > Key: HBASE-5664 > URL: https://issues.apache.org/jira/browse/HBASE-5664 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, Filters >Affects Versions: 0.92.1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-5664_94.patch, HBASE-5664_Trunk.patch > > > In HRegion.nextInternal(int limit, String metric) > We have while(true) loop so as to fetch a next result which satisfies > filter condition. When Filter filters out the current fetched row we call > nextRow(byte [] currentRow) before going with the next row. > {code} > if (results.isEmpty() || filterRow()) { > // this seems like a redundant step - we already consumed the row > // there're no left overs. > // the reasons for calling this method are: > // 1. reset the filters. > // 2. provide a hook to fast forward the row (used by subclasses) > nextRow(currentRow); > {code} > // 2. provide a hook to fast forward the row (used by subclasses) > We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558964#comment-13558964 ] Ted Yu commented on HBASE-3170: --- How about adding the new test to: {code} public class TestFromClientSide3 { {code} > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-v2.patch, > 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0550 seconds > {code} > This isn't a parsing problem with the command-line of the shell. I can > reproduce this behavior both with plain Java code and with my asynchbase > client. > Since I don't actually have a row with an empty row key, I expected that the > first {{get}} would return nothing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7637: -- Fix Version/s: 0.96.0 > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558968#comment-13558968 ] Hadoop QA commented on HBASE-7635: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565811/7635.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestHLog Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//console This message is automatically generated. > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7516: --- Attachment: trunk-7516.patch Attached trunk-7516.patch, which supports level compaction and other pluggable compaction policies. > Make compaction policy pluggable > > > Key: HBASE-7516 > URL: https://issues.apache.org/jira/browse/HBASE-7516 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Sergey Shelukhin > Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, > HBASE-7516-v2.patch, trunk-7516.patch > > > Currently, the compaction selection is pluggable. It will be great to make > the compaction algorithm pluggable too so that we can implement and play with > other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558973#comment-13558973 ] Sergey Shelukhin commented on HBASE-7329: - bq. Have you done much testing of this patch SS? I've run mvn tests; I will run some integration tests. bq. Nit: Below looks like it should be class comment rather than internal implementation comment or do you think otherwise? It's actually an implementation comment since it describes the implementation :) Class and method javadoc describe the external stuff. bq.Initially, the number of operations is 1. bq. Is it right having it at 1 when we construct the class? Should we wait on first beginOp call? Having 1 at start allows us to maintain a simple invariant that decrement to 0 is always the last. bq. It is not your fault but that is sure an ugly name on a method, getCompleteCacheFlushSequenceId. From its name you would not know what it is for. bq. In fact, if you want to remove it it looks like you could since 'TransactionalRegion' is a facility that no longer exists and going forward if you wanted to do this kinda thing, you'd do it via a coprocessor. Removed. bq. Is it right getting seqid before we advance memstore? We used to do it other way around. Fixed. bq. I wonder if the log of the roll should be outside of the lock...? Could be a bit sloppy but maybe it does not have to be too precise? You mean rollWriter? The only reason it's inside the lock is to prevent two concurrent rolls. It can be done differently e.g. we could lock around a boolean check and set, and bail duplicates. However current logic worked as having rollWriter-s queue up, not sure if anything relies on that. I will submit another patch, then run some integration tests on real cluster. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558975#comment-13558975 ] Ted Yu commented on HBASE-7635: --- Looks like QA environment issue: https://builds.apache.org/job/PreCommit-HBASE-Build/4111//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ {code} java.net.BindException: Problem binding to localhost/127.0.0.1:57829 : Address already in use at org.apache.hadoop.ipc.Server.bind(Server.java:228) at org.apache.hadoop.ipc.Server$Listener.(Server.java:302) at org.apache.hadoop.ipc.Server.(Server.java:1488) at org.apache.hadoop.ipc.RPC$Server.(RPC.java:560) at org.apache.hadoop.ipc.RPC.getServer(RPC.java:521) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:295) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:529) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1403) {code} I ran the test against both hadoop 1.0 and 2.0: 1528 ~/runtest.sh 4 TestHLog 1529 mvn clean test -Dhadoop.profile=2.0 -Dtest=TestHLog They all passed. Will integrate later today if there is no further review comment. > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) HFileSystem should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Hadoop Flags: Reviewed > HFileSystem should implement Closeable > -- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558976#comment-13558976 ] Jimmy Xiang commented on HBASE-7516: I put my patch on RB https://reviews.apache.org/r/9044/. It is very different from Sergey's patch. Instead just making the configuration pluggable, I made the compactor part of the policy. I assume the configuration is tightly related to the compactor for each compaction algorithm. > Make compaction policy pluggable > > > Key: HBASE-7516 > URL: https://issues.apache.org/jira/browse/HBASE-7516 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Sergey Shelukhin > Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, > HBASE-7516-v2.patch, trunk-7516.patch > > > Currently, the compaction selection is pluggable. It will be great to make > the compaction algorithm pluggable too so that we can implement and play with > other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558984#comment-13558984 ] stack commented on HBASE-7329: -- bq. You mean rollWriter? I meant the fat log message we make under the log. Maybe this log could be down outside the lock? I think running integration tests a good idea. The spaghetti locking and state management that is in place around flush/close and log roll was hard won. Changing it around may bite in unexpected ways. Regards the internal comment, it talks about calling beginOp then you must call...I'm not sure. looks like two possibilities. I was thinking the class comment would include how you would use the class? This is a nit. Not important. Thanks for persisting w/ this one Sergey. Its a nice patch. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558985#comment-13558985 ] Hadoop QA commented on HBASE-7516: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565819/trunk-7516.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 12 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestCatalogJanitor org.apache.hadoop.hbase.regionserver.TestBlocksScanned org.apache.hadoop.hbase.regionserver.TestResettingCounters org.apache.hadoop.hbase.regionserver.TestScanWithBloomError org.apache.hadoop.hbase.regionserver.TestColumnSeeking org.apache.hadoop.hbase.regionserver.TestHBase7051 org.apache.hadoop.hbase.regionserver.TestSplitTransaction org.apache.hadoop.hbase.filter.TestColumnPrefixFilter org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection org.apache.hadoop.hbase.client.TestIntraRowPagination org.apache.hadoop.hbase.filter.TestDependentColumnFilter org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking org.apache.hadoop.hbase.regionserver.TestHRegionInfo org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter org.apache.hadoop.hbase.regionserver.TestKeepDeletes org.apache.hadoop.hbase.regionserver.TestMinVersions org.apache.hadoop.hbase.filter.TestFilter org.apache.hadoop.hbase.regionserver.TestScanner org.apache.hadoop.hbase.regionserver.TestWideScanner org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4112//console This message is automatically generated. > Make compaction policy pluggable > > > Key: HBASE-7516 > URL: https://issues.apache.org/jira/browse/HBASE-7516 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Sergey Shelukhin > Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, > HBASE-7516-v2.patch, trunk-7516.patch > > > Currently, the compaction selection is pluggable. It will be great to make > the compaction algorithm pluggable too so that we can implement and play with > other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558992#comment-13558992 ] Sergey Shelukhin commented on HBASE-7516: - I took a cursory look; this patch makes compactor also pluggable and makes it return multiple files, and moves the default compaction policy stuff off the base class. If compactionPolicy returns, I wonder if (while Compactor is separate for reuse) it makes sense to make compactionPolicy interface simpler, and just let it compact. E.g. if Store has no default selection stuff anymore it doesn't make sense for it to get selection and feed it into compactor, right? Then, does it make sense to change CP interface to be called for all files post compaction instead of one file? I am not sure what use-cases it has, but there's no way to tell apart different compaction algorithms. My main concern is that this patch does not allow us to implement level compaction as described (see txt in level compaction issue). We can implement different algos which will allow for gradual compaction at the cost of IO, but not level algorithm, because that will remove the file ordering by seqNum, break the heuristic for determining mid-point for split, and other things. > Make compaction policy pluggable > > > Key: HBASE-7516 > URL: https://issues.apache.org/jira/browse/HBASE-7516 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Sergey Shelukhin > Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, > HBASE-7516-v2.patch, trunk-7516.patch > > > Currently, the compaction selection is pluggable. It will be great to make > the compaction algorithm pluggable too so that we can implement and play with > other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7329: Attachment: HBASE-7329-v6.patch Changes from review by stack (minor). > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558996#comment-13558996 ] nkeywal commented on HBASE-7637: I will need to fix it, as it's blocks some MTTR tests. May be I'm actually the cause of it , with stuff on the pom and cached path. But if someone already knows exactly what to do, I'm interested :-). > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7559) Add additional Snapshots Unit Test Coverage
[ https://issues.apache.org/jira/browse/HBASE-7559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7559: -- Status: Open (was: Patch Available) > Add additional Snapshots Unit Test Coverage > --- > > Key: HBASE-7559 > URL: https://issues.apache.org/jira/browse/HBASE-7559 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Fix For: 0.96.0 > > Attachments: 7559-v7.txt, aleks-snapshots.patch > > > Add additional testing for Snapshots. In particular, we should add tests to > verify that operations on cloned tables do not affect the original (and vice > versa). Also, we should do testing on table describes before and after > snapshot/restore operations. Finally, we should add testing for the HBase > shell. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7559) Add additional Snapshots Unit Test Coverage
[ https://issues.apache.org/jira/browse/HBASE-7559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7559: -- Attachment: 7559-v7.txt Patch v7 fixes the format of license header in the newly added files. > Add additional Snapshots Unit Test Coverage > --- > > Key: HBASE-7559 > URL: https://issues.apache.org/jira/browse/HBASE-7559 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Fix For: 0.96.0 > > Attachments: 7559-v7.txt, aleks-snapshots.patch > > > Add additional testing for Snapshots. In particular, we should add tests to > verify that operations on cloned tables do not affect the original (and vice > versa). Also, we should do testing on table describes before and after > snapshot/restore operations. Finally, we should add testing for the HBase > shell. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-3170: --- Attachment: 3170-5.patch This puts in the test in TestFromClientSide3. On the handling of null row key, the RPC handler will throw a NPE immediately since the ProtoBufUtil tries to convert the PB request to a regular request and that will throw NPE. I think handling that should be outside the scope of this jira since there could be other such nulls in requests. [~stack] what do you think? > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-5.patch, > 3170-v2.patch, 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0550 seconds > {code} > This isn't a parsing problem with the command-line of the shell. I can > reproduce this behavior both with plain Java code and with my asynchbase > client. > Since I don't actually have a row with an empty row key, I expected that the > first {{get}} would return nothing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7559) Add additional Snapshots Unit Test Coverage
[ https://issues.apache.org/jira/browse/HBASE-7559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559009#comment-13559009 ] Ted Yu commented on HBASE-7559: --- New tests pass: Running org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence 2013-01-21 11:19:49.812 java[98894:1203] Unable to load realm info from SCDynamicStore Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.802 sec Running org.apache.hadoop.hbase.client.TestSnapshotMetadata 2013-01-21 11:20:59.189 java[98909:1203] Unable to load realm info from SCDynamicStore Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.716 sec I think patch v7 should be good. > Add additional Snapshots Unit Test Coverage > --- > > Key: HBASE-7559 > URL: https://issues.apache.org/jira/browse/HBASE-7559 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Fix For: 0.96.0 > > Attachments: 7559-v7.txt, aleks-snapshots.patch > > > Add additional testing for Snapshots. In particular, we should add tests to > verify that operations on cloned tables do not affect the original (and vice > versa). Also, we should do testing on table describes before and after > snapshot/restore operations. Finally, we should add testing for the HBase > shell. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7329: Attachment: HBASE-7329-v6.patch A little more CR feedback... not technically related but good point. Moved log message outside of updateLock and cleaned it up a bit > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559020#comment-13559020 ] Ted Yu commented on HBASE-7637: --- I think this should be related to line 906 in pom.xml: hbase-hadoop1-compat > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559024#comment-13559024 ] nkeywal commented on HBASE-7637: Yes, I tried to change it (and I guess it must be done), but compat1 is still built and is found in the path. > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559029#comment-13559029 ] Elliott Clark commented on HBASE-7637: -- [~yuzhih...@gmail.com] Nope. That just sets the property. You'll see that the Hadoop 2.0 profile sets that property to use hbase-hadoop2-compat. I tested it and got hbase-hadoop2-compat on the class path correctly. Though I still got the error. I'm not sure what the root cause of the error is. > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559034#comment-13559034 ] nkeywal commented on HBASE-7637: it's not in cached_classpath.txt But if you do bin/hbase classpath, it shows up. investigating... > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559036#comment-13559036 ] Sergey Shelukhin commented on HBASE-6466: - ping? > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0 > > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, > HBASE-6466v3.1.patch, HBASE-6466v3.patch, HBASE-6466-v4.patch, > HBASE-6466-v4.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? > Some performance test data for reference, > 1.test environment ๏ผ > random writting๏ผupper memstore limit 5.6GB;lower memstore limit 4.8GB;400 > regions per regionserver๏ผrow len=50 bytes, value len=1024 bytes;5 > regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler > per client for writing > 2.test results: > one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per > regionserver, appears many aboveGlobalMemstoreLimit blocking > two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per > regionserver, > 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per > regionserver, Flush:18.6MB/s per regionserver -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559037#comment-13559037 ] nkeywal commented on HBASE-7637: Here is the guilty, in bin/hbase add_maven_main_classes_to_classpath() { # assumes all modules are named hbase-* in the top level directory IFS=$ORIG_IFS for module in `ls $HBASE_HOME | grep 'hbase-*'` do add_to_cp_if_exists "$HBASE_HOME/$module/target/classes" done } > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559038#comment-13559038 ] Elliott Clark commented on HBASE-6466: -- Tested this on a small cluster and everything seemed to work pretty well. Nothing strange happened and there were no pauses. Code in trunk has changed quite a lot since this patch started, so I can't really pinpoint what was happening. > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0 > > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, > HBASE-6466v3.1.patch, HBASE-6466v3.patch, HBASE-6466-v4.patch, > HBASE-6466-v4.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? > Some performance test data for reference, > 1.test environment ๏ผ > random writting๏ผupper memstore limit 5.6GB;lower memstore limit 4.8GB;400 > regions per regionserver๏ผrow len=50 bytes, value len=1024 bytes;5 > regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler > per client for writing > 2.test results: > one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per > regionserver, appears many aboveGlobalMemstoreLimit blocking > two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per > regionserver, > 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per > regionserver, Flush:18.6MB/s per regionserver -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559040#comment-13559040 ] nkeywal commented on HBASE-7637: The best option would be to build only the compat module we need. I don't know if it's possible with maven, but I will try (tomorrow :-)/ > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Attachment: 7594-1.patch Treat the symptom. > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559046#comment-13559046 ] stack commented on HBASE-7594: -- go for it > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Attachment: (was: 7594-1.patch) > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Status: Patch Available (was: Open) > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Attachment: 7594-1.patch > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-7268: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed v9. Thanks Sergey. To be clear, this bug also affects 0.94, but we won't backport this. The reason is that this is a relatively rare corner case. Correct me if I am wrong. > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7268-v6.patch, 7268-v8.patch, HBASE-7268-v0.patch, > HBASE-7268-v0.patch, HBASE-7268-v1.patch, HBASE-7268-v2.patch, > HBASE-7268-v2-plus-masterTs.patch, HBASE-7268-v2-plus-masterTs.patch, > HBASE-7268-v3.patch, HBASE-7268-v4.patch, HBASE-7268-v5.patch, > HBASE-7268-v6.patch, HBASE-7268-v7.patch, HBASE-7268-v8.patch, > HBASE-7268-v9.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3170: - Resolution: Fixed Release Note: If no row specified by a Get, we no longer return first row in the table. Now we fail. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks lads. > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-5.patch, > 3170-v2.patch, 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0550 seconds > {code} > This isn't a parsing problem with the command-line of the shell. I can > reproduce this behavior both with plain Java code and with my asynchbase > client. > Since I don't actually have a row with an empty row key, I expected that the > first {{get}} would return nothing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559054#comment-13559054 ] Hadoop QA commented on HBASE-7329: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565825/HBASE-7329-v6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 11 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestLocalHBaseCluster {color:red}-1 core zombie tests{color}. There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4113//console This message is automatically generated. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559055#comment-13559055 ] Sergey Shelukhin commented on HBASE-7329: - Integration tests have run without problems... I can see a lot of rolling/flushing in the logs, although of course this is not a rigorous test for this patch in particular. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3170) RegionServer confused about empty row keys
[ https://issues.apache.org/jira/browse/HBASE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559056#comment-13559056 ] Hadoop QA commented on HBASE-3170: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565830/3170-5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4114//console This message is automatically generated. > RegionServer confused about empty row keys > -- > > Key: HBASE-3170 > URL: https://issues.apache.org/jira/browse/HBASE-3170 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.89.20100621, 0.89.20100924, 0.90.0, 0.90.1, 0.90.2, > 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1 >Reporter: Benoit Sigoure >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 3170-1.patch, 3170-3.patch, 3170-4.patch, 3170-5.patch, > 3170-v2.patch, 3170-v3.patch, 3170-v3.patch, 3170v5.txt > > > I'm no longer sure about the expected behavior when using an empty row key > (e.g. a 0-byte long byte array). I assumed that this was a legitimate row > key, just like having an empty column qualifier is allowed. But it seems > that the RegionServer considers the empty row key to be whatever the first > row key is. > {code} > Version: 0.89.20100830, r0da2890b242584a8a5648d83532742ca7243346b, Sat Sep 18 > 15:30:09 PDT 2010 > hbase(main):001:0> scan 'tsdb-uid', {LIMIT => 1} > ROW COLUMN+CELL > > \x00 column=id:metrics, timestamp=1288375187699, > value=foo > \x00 column=id:tagk, timestamp=1287522021046, > value=bar > \x00 column=id:tagv, timestamp=1288111387685, > value=qux > 1 row(s) in 0.4610 seconds > hbase(main):002:0> get 'tsdb-uid', '' > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timestamp=1287522021046, value=bar > > id:tagv timestamp=1288111387685, value=qux > > 3 row(s) in 0.0910 seconds > hbase(main):003:0> get 'tsdb-uid', "\000" > COLUMNCELL > > id:metrics timestamp=1288375187699, value=foo > > id:tagk timest
[jira] [Commented] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559066#comment-13559066 ] Sergey Shelukhin commented on HBASE-7268: - I think part of this bug would affect 0.94 (removal from cache only), and it wouldn't require anything involved to fix, I will create a JIRA. Main part is not necessary because the redirection logic is not in 94 > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7268-v6.patch, 7268-v8.patch, HBASE-7268-v0.patch, > HBASE-7268-v0.patch, HBASE-7268-v1.patch, HBASE-7268-v2.patch, > HBASE-7268-v2-plus-masterTs.patch, HBASE-7268-v2-plus-masterTs.patch, > HBASE-7268-v3.patch, HBASE-7268-v4.patch, HBASE-7268-v5.patch, > HBASE-7268-v6.patch, HBASE-7268-v7.patch, HBASE-7268-v8.patch, > HBASE-7268-v9.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7638) [0.94] region cache entry should only be removed on error if the error is from the server currently in cache
Sergey Shelukhin created HBASE-7638: --- Summary: [0.94] region cache entry should only be removed on error if the error is from the server currently in cache Key: HBASE-7638 URL: https://issues.apache.org/jira/browse/HBASE-7638 Project: HBase Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor See HBASE-7268. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559067#comment-13559067 ] Hadoop QA commented on HBASE-7329: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565832/HBASE-7329-v6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 11 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestLocalHBaseCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4115//console This message is automatically generated. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7635) Proxy created by HFileSystem#createReorderingProxy() should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559071#comment-13559071 ] Ted Yu commented on HBASE-7635: --- Integrated to trunk. Thanks for the review, Stack. > Proxy created by HFileSystem#createReorderingProxy() should implement > Closeable > --- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7635) Proxy created by HFileSystem#createReorderingProxy() should implement Closeable
[ https://issues.apache.org/jira/browse/HBASE-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7635: -- Summary: Proxy created by HFileSystem#createReorderingProxy() should implement Closeable (was: HFileSystem should implement Closeable) > Proxy created by HFileSystem#createReorderingProxy() should implement > Closeable > --- > > Key: HBASE-7635 > URL: https://issues.apache.org/jira/browse/HBASE-7635 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7635.txt > > > From > https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/364/testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testThreeRSAbort/ > : > {code} > 2013-01-21 11:49:26,141 ERROR [Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081] > server.NIOServerCnxnFactory$1(44): Thread Thread[Shutdown of > org.apache.hadoop.hbase.fs.HFileSystem@1792081,5,main] died > org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not > Closeable or does not provide closeable invocation handler class $Proxy20 > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624) > at > org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:638) > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:696) > at > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > at > org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:404) > at org.apache.hadoop.hbase.fs.HFileSystem.close(HFileSystem.java:148) > at > org.apache.hadoop.hbase.MiniHBaseCluster$SingleFileSystemShutdownThread.run(MiniHBaseCluster.java:187) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559072#comment-13559072 ] Hadoop QA commented on HBASE-7594: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565838/7594-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFile Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4116//console This message is automatically generated. > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559078#comment-13559078 ] Elliott Clark commented on HBASE-7637: -- Do we even need that add_maven_main_classes_to_classpath any more ? Seems like cached_classpath contained all of the jars that I would expect. > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7633) Add a metric that tracks the current number of used RPC threads on the regionservers
[ https://issues.apache.org/jira/browse/HBASE-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559080#comment-13559080 ] Elliott Clark commented on HBASE-7633: -- In 0.94 there's: * callQueueLen In trunk there are a few more metrics: * numCallsInGeneralQueue * numCallsInPriorityQueue * numCallsInReplicationQueue While the don't tell you how many threads are currently running they do hint at if things are backing up. > Add a metric that tracks the current number of used RPC threads on the > regionservers > > > Key: HBASE-7633 > URL: https://issues.apache.org/jira/browse/HBASE-7633 > Project: HBase > Issue Type: Improvement >Reporter: Joey Echeverria >Assignee: Elliott Clark > > One way to detect that you're hitting a "John Wayne" disk[1] would be if we > could see when region servers exhausted their RPC handlers. This would also > be useful when tuning the cluster for your workload to make sure that reads > or writes were not starving the other operations out. > [1] http://hbase.apache.org/book.html#bad.disk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Status: Open (was: Patch Available) > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7637: - Attachment: HBASE-7637-0.patch Something like this works for me. > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > Attachments: HBASE-7637-0.patch > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7633) Add a metric that tracks the current number of used RPC threads on the regionservers
[ https://issues.apache.org/jira/browse/HBASE-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559103#comment-13559103 ] stack commented on HBASE-7633: -- Given what Elliott says, can we close this [~fwiffo]? > Add a metric that tracks the current number of used RPC threads on the > regionservers > > > Key: HBASE-7633 > URL: https://issues.apache.org/jira/browse/HBASE-7633 > Project: HBase > Issue Type: Improvement >Reporter: Joey Echeverria >Assignee: Elliott Clark > > One way to detect that you're hitting a "John Wayne" disk[1] would be if we > could see when region servers exhausted their RPC handlers. This would also > be useful when tuning the cluster for your workload to make sure that reads > or writes were not starving the other operations out. > [1] http://hbase.apache.org/book.html#bad.disk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559104#comment-13559104 ] stack commented on HBASE-7329: -- [~sershe] Thanks for running tests. +1 on commit. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7633) Add a metric that tracks the current number of used RPC threads on the regionservers
[ https://issues.apache.org/jira/browse/HBASE-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559106#comment-13559106 ] Joey Echeverria commented on HBASE-7633: callQueueLen is close I think, but I'm not sure how that translates into tuning hbase.regionserver.handler.count. Do we have a good example of interpreting that value? > Add a metric that tracks the current number of used RPC threads on the > regionservers > > > Key: HBASE-7633 > URL: https://issues.apache.org/jira/browse/HBASE-7633 > Project: HBase > Issue Type: Improvement >Reporter: Joey Echeverria >Assignee: Elliott Clark > > One way to detect that you're hitting a "John Wayne" disk[1] would be if we > could see when region servers exhausted their RPC handlers. This would also > be useful when tuning the cluster for your workload to make sure that reads > or writes were not starving the other operations out. > [1] http://hbase.apache.org/book.html#bad.disk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7637: -- Status: Patch Available (was: Open) > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > Attachments: HBASE-7637-0.patch > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7622) Add table descriptor verification after snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7622: --- Attachment: HBASE-7622-v2.patch v2 is equals to v1 but adds table.close() to TestRestoreFlush (similar to TestRestore) > Add table descriptor verification after snapshot restore > > > Key: HBASE-7622 > URL: https://issues.apache.org/jira/browse/HBASE-7622 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: hbase-6055 > > Attachments: HBASE-7622-v0.patch, HBASE-7622-v1.patch, > HBASE-7622-v2.patch > > > Add the schema verification not only based on disk data, but also on the > HTableDescriptor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7633) Add a metric that tracks the current number of used RPC threads on the regionservers
[ https://issues.apache.org/jira/browse/HBASE-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559114#comment-13559114 ] stack commented on HBASE-7633: -- bq. ...translates into tuning hbase.regionserver.handler.count. Do we have a good example of interpreting that value? Not really other than if they are backed up frequently and we're not blocked on cpu or io, then bump them up (Is this for another issue [~fwiffo]?) Thanks boss. > Add a metric that tracks the current number of used RPC threads on the > regionservers > > > Key: HBASE-7633 > URL: https://issues.apache.org/jira/browse/HBASE-7633 > Project: HBase > Issue Type: Improvement >Reporter: Joey Echeverria >Assignee: Elliott Clark > > One way to detect that you're hitting a "John Wayne" disk[1] would be if we > could see when region servers exhausted their RPC handlers. This would also > be useful when tuning the cluster for your workload to make sure that reads > or writes were not starving the other operations out. > [1] http://hbase.apache.org/book.html#bad.disk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559122#comment-13559122 ] Ted Yu commented on HBASE-7329: --- HBASE-7268 has been checked in. I checked in and then reverted patch v6 because of compilation error. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7329: -- Attachment: 7329-v7.txt > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559137#comment-13559137 ] Ted Yu commented on HBASE-7329: --- Patch v7 compiles based on latest trunk. TestHLog passes. Let Hadoop QA tell us the result. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6907) KeyValue equals and compareTo methods should match
[ https://issues.apache.org/jira/browse/HBASE-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6907: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > KeyValue equals and compareTo methods should match > -- > > Key: HBASE-6907 > URL: https://issues.apache.org/jira/browse/HBASE-6907 > Project: HBase > Issue Type: Bug > Components: util >Reporter: Matt Corgan >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6907-v1.txt, 6907-v2.txt, 6907-v3.txt, 6907-v4.txt, > 6907-v5.txt > > > KeyValue.KVComparator includes the memstoreTS when comparing, however the > KeyValue.equals() method ignores the memstoreTS. > The Comparator interface has always specified that comparator return 0 when > equals would return true and vice versa. Obeying that rule has been sort of > optional in the past, but Java 7 introduces a new default collection sorting > algorithm called Tim Sort which relies on that behavior. > http://bugs.sun.com/view_bug.do?bug_id=6804124 > Possible problem spots: > * there's a Collections.sort(KeyValues) in > RedundantKVGenerator.generateTestKeyValues(..) > * TestColumnSeeking compares two collections of KeyValues using the > containsAll method. It is intentionally ignoring memstoreTS, so will need an > alternative method for comparing the two collections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559147#comment-13559147 ] Andrew Purtell commented on HBASE-7594: --- The instantiation check flagged TestHFile as doing something bogus. This is interesting. TestHFile#testComparator creates an anonymous class which does not have a nullary constructor so cannot be instantiated. See this disassembly with javap: {noformat} Compiled from "TestHFile.java" class org.apache.hadoop.hbase.io.hfile.TestHFile$2 extends org.apache.hadoop.hbase.KeyValue$KeyComparator{ final org.apache.hadoop.hbase.io.hfile.TestHFile this$0; Signature: Lorg/apache/hadoop/hbase/io/hfile/TestHFile; org.apache.hadoop.hbase.io.hfile.TestHFile$2(org.apache.hadoop.hbase.io.hfile.TestHFile); Signature: (Lorg/apache/hadoop/hbase/io/hfile/TestHFile;)V public int compare(byte[], int, int, byte[], int, int); Signature: ([BII[BII)I public int compare(byte[], byte[]); Signature: ([B[B)I public int compare(java.lang.Object, java.lang.Object); Signature: (Ljava/lang/Object;Ljava/lang/Object;)I } {noformat} Note the constructor. This minor change to TestHFile fixes the problem locally: {noformat} --- hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java (revision 1436569) +++ hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java (working copy) @@ -347,21 +347,25 @@ assertTrue(Compression.Algorithm.LZ4.ordinal() == 4); } + // This can't be an anonymous class because the compiler will not generate + // a nullary constructor for it. + static class CustomKeyComparator extends KeyComparator { +@Override +public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, +int l2) { + return -Bytes.compareTo(b1, s1, l1, b2, s2, l2); +} +@Override +public int compare(byte[] o1, byte[] o2) { + return compare(o1, 0, o1.length, o2, 0, o2.length); +} + } + public void testComparator() throws IOException { if (cacheConf == null) cacheConf = new CacheConfig(conf); Path mFile = new Path(ROOT_DIR, "meta.tfile"); FSDataOutputStream fout = createFSOutput(mFile); -KeyComparator comparator = new KeyComparator() { - @Override - public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, - int l2) { -return -Bytes.compareTo(b1, s1, l1, b2, s2, l2); - } - @Override - public int compare(byte[] o1, byte[] o2) { -return compare(o1, 0, o1.length, o2, 0, o2.length); - } -}; +KeyComparator comparator = new CustomKeyComparator(); Writer writer = HFile.getWriterFactory(conf, cacheConf) .withOutputStream(fout) .withBlockSize(minBlockSize) {noformat} Will update the patch to include this and try again. > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$S
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Attachment: 7594-2.patch > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch, 7594-2.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7594: -- Status: Patch Available (was: Open) > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch, 7594-2.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559151#comment-13559151 ] Sergey Shelukhin commented on HBASE-7516: - This code will also have to change depending on compation algo, in particular for level. {code} // exclude all files older than the newest file we're currently // compacting. this allows us to preserve contiguity (HBASE-2856) StoreFile last = filesCompacting.get(filesCompacting.size() - 1); int idx = candidates.indexOf(last); Preconditions.checkArgument(idx != -1); candidates.subList(0, idx + 1).clear(); }{code} I will try to get a prototype patch for store files into the refactor jira by tomorrow, tomorrow we can discuss the necessary changes > Make compaction policy pluggable > > > Key: HBASE-7516 > URL: https://issues.apache.org/jira/browse/HBASE-7516 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Sergey Shelukhin > Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, > HBASE-7516-v2.patch, trunk-7516.patch > > > Currently, the compaction selection is pluggable. It will be great to make > the compaction algorithm pluggable too so that we can implement and play with > other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7594) TestLocalHBaseCluster failing on ubuntu2
[ https://issues.apache.org/jira/browse/HBASE-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559156#comment-13559156 ] stack commented on HBASE-7594: -- Nice find Andrew. +1 > TestLocalHBaseCluster failing on ubuntu2 > > > Key: HBASE-7594 > URL: https://issues.apache.org/jira/browse/HBASE-7594 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 7594-1.patch, 7594-2.patch > > > {noformat} > java.io.IOException: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:612) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:533) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4092) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4042) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:427) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:130) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.io.IOException: > java.lang.InstantiationException: org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:450) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:215) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3060) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:585) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:583) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:607) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:615) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:115) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:564) > at > org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:599) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1294) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:628) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:426) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:422) > ... 8 more > Caused by: java.lang.InstantiationException: > org.apache.hadoop.io.RawComparator > at java.lang.Class.newInstance0(Class.java:340) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:605) > ... 17 more > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6669) Add BigDecimalColumnInterpreter for doing aggregations using AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6669: -- Attachment: 6669-0.94-v5.txt Patch v5 addresses latest review comments. Running org.apache.hadoop.hbase.coprocessor.TestBigDecimalColumnInterpreter 2013-01-21 14:06:27.730 java[99778:1203] Unable to load realm info from SCDynamicStore Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.557 sec > Add BigDecimalColumnInterpreter for doing aggregations using AggregationClient > -- > > Key: HBASE-6669 > URL: https://issues.apache.org/jira/browse/HBASE-6669 > Project: HBase > Issue Type: New Feature > Components: Client, Coprocessors >Affects Versions: 0.94.3 >Reporter: Anil Gupta >Priority: Minor > Labels: client, coprocessors > Fix For: 0.94.5 > > Attachments: 6669-0.94-v4.txt, 6669-0.94-v5.txt, > BigDecimalColumnInterpreter.java, BigDecimalColumnInterpreter.patch, > BigDecimalColumnInterpreter.patch, HBASE-6669.patch, HBASE-6669-v2.patch, > HBASE-6669-v3.patch, TestBDAggregateProtocol.patch, > TestBigDecimalColumnInterpreter.java > > > I recently created a Class for doing aggregations(sum,min,max,std) on values > stored as BigDecimal in HBase. I would like to commit the > BigDecimalColumnInterpreter into HBase. In my opinion this class can be used > by a wide variety of users. Please let me know if its not appropriate to add > this class in HBase. > Thanks, > Anil Gupta > Software Engineer II, Intuit, Inc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7637) hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0
[ https://issues.apache.org/jira/browse/HBASE-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559163#comment-13559163 ] Hadoop QA commented on HBASE-7637: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565849/HBASE-7637-0.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestLocalHBaseCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4117//console This message is automatically generated. > hbase-hadoop1-compat conflicts with -Dhadoop.profile=2.0 > > > Key: HBASE-7637 > URL: https://issues.apache.org/jira/browse/HBASE-7637 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Critical > Fix For: 0.96.0 > > Attachments: HBASE-7637-0.patch > > > I'm unclear on the root cause / fix. Here is the scenario: > {noformat} > mvn clean package install -Dhadoop.profile=2.0 -DskipTests > bin/start-hbase.sh > {noformat} > fails with > {noformat} > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.metrics2.lib.MetricMutable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > {noformat} > doing > {noformat} > rm -rf hbase-hadoop1-compat/target/ > {noformat} > makes it work. > In the pom.xml, we never reference hadoop2-compat. But doing so does not > help: hadoop1-compat is compiled and takes precedence over hadoop2... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7639) Enable online schema update by default
Enis Soztutar created HBASE-7639: Summary: Enable online schema update by default Key: HBASE-7639 URL: https://issues.apache.org/jira/browse/HBASE-7639 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Enis Soztutar Assignee: Enis Soztutar After we get HBASE-7305 and HBASE-7546, things will become stable enough to enable online schema update to be enabled by default. {code} hbase.online.schema.update.enable false Set true to enable online schema changes. This is an experimental feature.ยทยท There are known issues modifying table schemas at the same time a region split is happening so your table needs to be quiescent or else you have to be running with splits disabled. {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7268: - Release Note: On region open, save the edit seqId and then write it into META. On region move, the region is opened again with a greater seqId somewhere else. Pass the client the seqid to the client when asks about locations. Client can reason about cache invalidation with seqids (if seqid for new location is < its current seqid, it can recognize the new location stale). Does this fix mean that hbase-it works again? > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: 7268-v6.patch, 7268-v8.patch, HBASE-7268-v0.patch, > HBASE-7268-v0.patch, HBASE-7268-v1.patch, HBASE-7268-v2.patch, > HBASE-7268-v2-plus-masterTs.patch, HBASE-7268-v2-plus-masterTs.patch, > HBASE-7268-v3.patch, HBASE-7268-v4.patch, HBASE-7268-v5.patch, > HBASE-7268-v6.patch, HBASE-7268-v7.patch, HBASE-7268-v8.patch, > HBASE-7268-v9.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7640) FSUtils#getTableStoreFilePathMap should ignore non-hfiles.
Jimmy Xiang created HBASE-7640: -- Summary: FSUtils#getTableStoreFilePathMap should ignore non-hfiles. Key: HBASE-7640 URL: https://issues.apache.org/jira/browse/HBASE-7640 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Priority: Trivial ERROR: Found lingering reference file hdfs://node3:9000/hbase/entry_proposed/fbd1735591467005e53f48645278b006/recovered.edits/00091843039.temp recovered.edits is not a column family. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira