[jira] [Resolved] (HBASE-16406) import data to hbase error by using hbase api
[
https://issues.apache.org/jira/browse/HBASE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak resolved HBASE-16406.
-
Resolution: Invalid
> import data to hbase error by using hbase api
> -
>
> Key: HBASE-16406
> URL: https://issues.apache.org/jira/browse/HBASE-16406
> Project: HBase
> Issue Type: Bug
>Reporter: xiboliyalangwolf
>
> The code i wrote belows:
> public static void addRow(String tableName, String row,String
> columnFamily, String column, String value) throws Exception {
> HTable table = new HTable(conf,tableName);
> Put put = new Put(Bytes.toBytes(row));
> put.add(Bytes.toBytes(columnFamily),
> Bytes.toBytes(column),Bytes.toBytes(value));
> table.put(put);
> table.close();
> }
> public static void main(String[] args){
> String tableName = "test1";
> String[] columnFamilys = { "info", "course" };
> //createTable(tableName,columnFamilys);
> try {
> addRow(tableName, "tht", "info", "age", "20");
> //addRow(tableName, "tht", "info", "sex", "boy");
> addRow(tableName, "tht", "course", "china", "97");
> //addRow(tableName, "tht", "course", "math", "128");
> //addRow(tableName, "tht", "course", "english", "85");
> } catch (Exception e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> And The error:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: test1: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1700)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1449)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1040)
> at hbase2.testCreate.addRow(testCreate.java:45)
> at hbase2.testCreate.main(testCreate.java:53)
> How can i solve it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-16406) import data to hbase error by using hbase api
[
https://issues.apache.org/jira/browse/HBASE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak reopened HBASE-16406:
-
> import data to hbase error by using hbase api
> -
>
> Key: HBASE-16406
> URL: https://issues.apache.org/jira/browse/HBASE-16406
> Project: HBase
> Issue Type: Bug
>Reporter: xiboliyalangwolf
>
> The code i wrote belows:
> public static void addRow(String tableName, String row,String
> columnFamily, String column, String value) throws Exception {
> HTable table = new HTable(conf,tableName);
> Put put = new Put(Bytes.toBytes(row));
> put.add(Bytes.toBytes(columnFamily),
> Bytes.toBytes(column),Bytes.toBytes(value));
> table.put(put);
> table.close();
> }
> public static void main(String[] args){
> String tableName = "test1";
> String[] columnFamilys = { "info", "course" };
> //createTable(tableName,columnFamilys);
> try {
> addRow(tableName, "tht", "info", "age", "20");
> //addRow(tableName, "tht", "info", "sex", "boy");
> addRow(tableName, "tht", "course", "china", "97");
> //addRow(tableName, "tht", "course", "math", "128");
> //addRow(tableName, "tht", "course", "english", "85");
> } catch (Exception e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> And The error:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: test1: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1700)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1449)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1040)
> at hbase2.testCreate.addRow(testCreate.java:45)
> at hbase2.testCreate.main(testCreate.java:53)
> How can i solve it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16406) import data to hbase error by using hbase api
[
https://issues.apache.org/jira/browse/HBASE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak resolved HBASE-16406.
-
Resolution: Fixed
Hi [~xiboliyalangwolf]! This issue tracker is intended only to track HBase bugs
and this looks more like a problem that you'd get help on by sending a message
to the [HBase user mailing list|mailto:u...@hbase.apache.org]. Thanks!
> import data to hbase error by using hbase api
> -
>
> Key: HBASE-16406
> URL: https://issues.apache.org/jira/browse/HBASE-16406
> Project: HBase
> Issue Type: Bug
>Reporter: xiboliyalangwolf
>
> The code i wrote belows:
> public static void addRow(String tableName, String row,String
> columnFamily, String column, String value) throws Exception {
> HTable table = new HTable(conf,tableName);
> Put put = new Put(Bytes.toBytes(row));
> put.add(Bytes.toBytes(columnFamily),
> Bytes.toBytes(column),Bytes.toBytes(value));
> table.put(put);
> table.close();
> }
> public static void main(String[] args){
> String tableName = "test1";
> String[] columnFamilys = { "info", "course" };
> //createTable(tableName,columnFamilys);
> try {
> addRow(tableName, "tht", "info", "age", "20");
> //addRow(tableName, "tht", "info", "sex", "boy");
> addRow(tableName, "tht", "course", "china", "97");
> //addRow(tableName, "tht", "course", "math", "128");
> //addRow(tableName, "tht", "course", "english", "85");
> } catch (Exception e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> And The error:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: test1: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1700)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1449)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1040)
> at hbase2.testCreate.addRow(testCreate.java:45)
> at hbase2.testCreate.main(testCreate.java:53)
> How can i solve it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[
https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418417#comment-15418417
]
yang ming edited comment on HBASE-16395 at 8/12/16 6:33 AM:
h3.HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358)
at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364)
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Try reading via
the datanode on /192.168.7.139:50010
h3.HDFS log(no error log):
2016-08-12 14:25:49,902 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55856, bytes: 66048, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
{color:red}blk_-4614053595081055029_54130616{color}, duration: 252426
2016-08-12 14:25:49,964 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55858, bytes: 198144, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
blk_-4614053595081055029_54130616, duration: 329364
was (Author: yangming860101):
h3.HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171
[jira] [Comment Edited] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[
https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418417#comment-15418417
]
yang ming edited comment on HBASE-16395 at 8/12/16 6:30 AM:
h3.HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358)
at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364)
h3.HDFS log(no error log):
2016-08-12 14:25:49,902 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55856, bytes: 66048, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
{color:red}blk_-4614053595081055029_54130616{color}, duration: 252426
2016-08-12 14:25:49,964 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55858, bytes: 198144, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
blk_-4614053595081055029_54130616, duration: 329364
was (Author: yangming860101):
h3.HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358)
at org.apache.hadoop.h
[jira] [Comment Edited] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[
https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418417#comment-15418417
]
yang ming edited comment on HBASE-16395 at 8/12/16 6:30 AM:
h3.HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358)
at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364)
HDFS log(no error log):
2016-08-12 14:25:49,902 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55856, bytes: 66048, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
{color:red}blk_-4614053595081055029_54130616{color}, duration: 252426
2016-08-12 14:25:49,964 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55858, bytes: 198144, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
blk_-4614053595081055029_54130616, duration: 329364
was (Author: yangming860101):
*strong*HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358)
at org.apache.hadoop
[jira] [Commented] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[
https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418417#comment-15418417
]
yang ming commented on HBASE-16395:
---
HBase log:
2016-08-12 14:25:56,738 DEBUG
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log
for replication dn7%2C60020%2C1470216726863.1470983077165 at 28175754
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal
requested with incorrect offset: Offset 0 and length 28186400 don't match
block blk_-4614053595081055029_54130616 ( blockLen 28175754 )
2016-08-12 14:25:56,740 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-4614053595081055029_54130616 from cache because
local file /sdf/hdfs/dfs/data/blocksBeingWritten/blk_-4614053595081055029 could
not be opened.
2016-08-12 14:25:56,740 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block {color:red}blk_-4614053595081055029_54130616{color} on local
machinejava.io.IOException: Offset 0 and length 28186400 don't match block
{color:red}blk_-4614053595081055029_54130616{color} ( blockLen 28175754 )
at
org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358)
at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364)
HDFS log(no error log):
2016-08-12 14:25:49,902 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55856, bytes: 66048, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
{color:red}blk_-4614053595081055029_54130616{color}, duration: 252426
2016-08-12 14:25:49,964 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/192.168.7.139:50010, dest: /192.168.7.139:55858, bytes: 198144, op: HDFS_READ,
cliID: DFSClient_hb_rs_dn7,60020,1470216726863, offset: 0, srvID:
DS-1014379950-192.168.7.139-50010-1416802300444, blockid:
blk_-4614053595081055029_54130616, duration: 329364
> ShortCircuitLocalReads Failed when enabled replication
> --
>
> Key: HBASE-16395
> URL: https://issues.apache.org/jira/browse/HBASE-16395
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.94.20
>Reporter: yang ming
>Priority: Critical
>
> I had sended an email to u...@hbase.apache.org,but received no help.
> The cluster enabled shortCircuitLocalReads.
>
> dfs.client.read.shortcircuit
> true
>
> When enabled replication,we found a large number of error logs.
> 1.shortCircuitLocalReads(fail everytime).
> 2.Try reading via the datanode on targetAddr(success).
> How to make shortCircuitLocalReads successfully when enabled replication?
> 2016-08-03 10:46:21,721 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
> log for replication dn7%2C60020%2C1470136216957.1470192327030 at 16999670
> 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient:
> BlockReaderLocal requested with incorrect offset: Offset 0 and length
> 17073479 don't match block blk_4137524355009640437_53760530 ( blockLen
> 16999670 )
> 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient:
> BlockReaderLocal: Remo
[jira] [Updated] (HBASE-16404) Make DeleteBloomFilters work with BloomContext
[ https://issues.apache.org/jira/browse/HBASE-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16404: --- Status: Patch Available (was: Open) > Make DeleteBloomFilters work with BloomContext > -- > > Key: HBASE-16404 > URL: https://issues.apache.org/jira/browse/HBASE-16404 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16404.patch > > > Since we have added BloomFiltercontext to work with RowBlooms the same can be > applied for DeleteBloomFilters also. Should be a simple change considering we > have the infrastructure in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16404) Make DeleteBloomFilters work with BloomContext
[ https://issues.apache.org/jira/browse/HBASE-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16404: --- Priority: Minor (was: Major) > Make DeleteBloomFilters work with BloomContext > -- > > Key: HBASE-16404 > URL: https://issues.apache.org/jira/browse/HBASE-16404 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16404.patch > > > Since we have added BloomFiltercontext to work with RowBlooms the same can be > applied for DeleteBloomFilters also. Should be a simple change considering we > have the infrastructure in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16404) Make DeleteBloomFilters work with BloomContext
[ https://issues.apache.org/jira/browse/HBASE-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16404: --- Attachment: HBASE-16404.patch Simple patch. > Make DeleteBloomFilters work with BloomContext > -- > > Key: HBASE-16404 > URL: https://issues.apache.org/jira/browse/HBASE-16404 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16404.patch > > > Since we have added BloomFiltercontext to work with RowBlooms the same can be > applied for DeleteBloomFilters also. Should be a simple change considering we > have the infrastructure in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15554) StoreFile$Writer.appendGeneralBloomFilter generates extra KV
[ https://issues.apache.org/jira/browse/HBASE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418414#comment-15418414 ] Hudson commented on HBASE-15554: FAILURE: Integrated in HBase-Trunk_matrix #1400 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1400/]) HBASE-15554 - StoreFile$Writer.appendGeneralBloomFilter generates extra (ramkrishna: rev 285e104176bf848dc81657ac5358ad3beb49cc57) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomContext.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/HashKey.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteArrayHashKey.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterWriter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/Hash.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/MurmurHash.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/MurmurHash3.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CompoundBloomFilter.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/RowColBloomHashKey.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/CellHashKey.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/RowColBloomContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CompoundBloomFilterWriter.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/JenkinsHash.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/RowBloomHashKey.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/RowBloomContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterChunk.java > StoreFile$Writer.appendGeneralBloomFilter generates extra KV > > > Key: HBASE-15554 > URL: https://issues.apache.org/jira/browse/HBASE-15554 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Vladimir Rodionov >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15554.patch, HBASE-15554_10.patch, > HBASE-15554_11.patch, HBASE-15554_12.patch, HBASE-15554_13.patch, > HBASE-15554_14.patch, HBASE-15554_3.patch, HBASE-15554_4.patch, > HBASE-15554_6.patch, HBASE-15554_7.patch, HBASE-15554_9.patch > > > Accounts for 10% memory allocation in compaction thread when BloomFilterType > is ROWCOL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16403) Upgrade guava version to the lastest one(19.0)
[ https://issues.apache.org/jira/browse/HBASE-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418411#comment-15418411 ] Anoop Sam John commented on HBASE-16403: Ya our shading is not having the package renaming to avoid possible conflicts? I was also thinking we do have this way.. cc [~saint@gmail.com] > Upgrade guava version to the lastest one(19.0) > -- > > Key: HBASE-16403 > URL: https://issues.apache.org/jira/browse/HBASE-16403 > Project: HBase > Issue Type: Improvement > Components: API, Client >Affects Versions: 1.1.4 >Reporter: lihongzhe > > HBase client use the guava version 12.0.1, while elasticsearch sdk use the > version 18+. It makes conflict, cause the constructor of StopWatch class is > not public now. > We have to choose the es rest api to resolve the conflict. Why not try to > upgrade our dependency versions? > thx. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[ https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak resolved HBASE-16395. - Resolution: Invalid Let's keep discussion of this on user@ so that others can benefit from it. > ShortCircuitLocalReads Failed when enabled replication > -- > > Key: HBASE-16395 > URL: https://issues.apache.org/jira/browse/HBASE-16395 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.20 >Reporter: yang ming >Priority: Critical > > I had sended an email to u...@hbase.apache.org,but received no help. > The cluster enabled shortCircuitLocalReads. > > dfs.client.read.shortcircuit > true > > When enabled replication,we found a large number of error logs. > 1.shortCircuitLocalReads(fail everytime). > 2.Try reading via the datanode on targetAddr(success). > How to make shortCircuitLocalReads successfully when enabled replication? > 2016-08-03 10:46:21,721 DEBUG > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening > log for replication dn7%2C60020%2C1470136216957.1470192327030 at 16999670 > 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient: > BlockReaderLocal requested with incorrect offset: Offset 0 and length > 17073479 don't match block blk_4137524355009640437_53760530 ( blockLen > 16999670 ) > 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient: > BlockReaderLocal: Removing blk_4137524355009640437_53760530 from cache > because local file > /sdd/hdfs/dfs/data/blocksBeingWritten/blk_4137524355009640437 could not be > opened. > 2016-08-03 10:46:21,724 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read > block blk_4137524355009640437_53760530 on local machinejava.io.IOException: > Offset 0 and length 17073479 don't match block > blk_4137524355009640437_53760530 ( blockLen 16999670 ) > at org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287) > at > org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171) > at org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358) > at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224) > at java.io.DataInputStream.read(DataInputStream.java:149) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178) > at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364) > 2016-08-03 10:46:21,724 INFO org.apache.hadoop.hdfs.DFSClient: Try reading > via the datanode on /192.168.7.139:50010 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16229) Cleaning up size and heapSize calculation
[
https://issues.apache.org/jira/browse/HBASE-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418395#comment-15418395
]
Anoop Sam John commented on HBASE-16229:
Gone through it.. That is what exactly this is trying to do. Each class is
responsible for its own size. Also added a keySize() and size/heapSize. In
segment also we have this 2 variants. Former just tracks the size of the
cells. heapSize will be like keySize() + heap overhead.
The keySize includes cells heapsize. - We plan to split this also.. (I
guess you are suggesting that too)- This will be one part will have cell data
size (key size + val size + tags size) and other will be heap overhead. Even
at global level also we will have this split. This is required when we have off
heap memstore (The MSLAB will be off heap based). So there we will need clear
separate tracking for what is memstore size in off heap area and what is heap
overhead. The flush decisions to consider both. ( Pls see
https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMuMaJqaMNRI86C4k5S1rQ/edit
for discussion abt this. See the section "Memstore sizing" )
Once 14921 is in, we will go ahead with this jira and later with the split of
cell data size and heap overhead split.
> Cleaning up size and heapSize calculation
> -
>
> Key: HBASE-16229
> URL: https://issues.apache.org/jira/browse/HBASE-16229
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16229.patch, HBASE-16229_V2.patch,
> HBASE-16229_V3.patch
>
>
> It is bit ugly now. For eg:
> AbstractMemStore
> {code}
> public final static long FIXED_OVERHEAD = ClassSize.align(
> ClassSize.OBJECT +
> (4 * ClassSize.REFERENCE) +
> (2 * Bytes.SIZEOF_LONG));
> public final static long DEEP_OVERHEAD = ClassSize.align(FIXED_OVERHEAD +
> (ClassSize.ATOMIC_LONG + ClassSize.TIMERANGE_TRACKER +
> ClassSize.CELL_SKIPLIST_SET + ClassSize.CONCURRENT_SKIPLISTMAP));
> {code}
> We include the heap overhead of Segment also here. It will be better the
> Segment contains its overhead part and the Memstore impl uses the heap size
> of all of its segments to calculate its size.
> Also this
> {code}
> public long heapSize() {
> return getActive().getSize();
> }
> {code}
> HeapSize to consider all segment's size not just active's. I am not able to
> see an override method in CompactingMemstore.
> This jira tries to solve some of these.
> When we create a Segment, we seems pass some initial heap size value to it.
> Why? The segment object internally has to know what is its heap size not
> like some one else dictate it.
> More to add when doing this cleanup
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16405) Remove byte[] based hash() APIs.
[ https://issues.apache.org/jira/browse/HBASE-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16405: --- Assignee: Anoop Sam John (was: ramkrishna.s.vasudevan) > Remove byte[] based hash() APIs. > > > Key: HBASE-16405 > URL: https://issues.apache.org/jira/browse/HBASE-16405 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > > Currently the read path though it works with Cells, still does go with the > assumption that we need to copy the Cells to KeyValue's key format and then > use the byte[] version of hash() function to calculate the Bloom hash. In > case of incoming cell's being KeyValue type we try to use the > KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will > get rid of all those and things will work with Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418386#comment-15418386 ] ramkrishna.s.vasudevan commented on HBASE-14921: bq.1. Size arithmetic. I'll be publishing a design proposal for this issue here and on HBASE-16229 soon, let's manage the rest of the discussion there. There seems to be an elegant solution, which I even have a partial code for. Can post a suggested patch immediately after HBASE-14921 is committed. Yes. We are infact waiting for gettig in HBASE-16229 after this JIRA is done. Size arithmetic is important for us for the offheap accounting also. bq.The current patch circumvents the "too many open files" with a simple heuristic that prevents too many segments from piling up. However, we realize that this issue might require more discussion, so let's open a new jira for it. Makes sense. So were you able to observe the problems that we highlighted in the previous discussions - We got two problems one is the invalid negative size for memstore and the 'Too many open files'? Were you able to get this patch running with the PE tool and the configs that we had updated in the earlier discussions? bq.And finally, the crown jewel - the CellChunkMap project. Its main value is providing platform for the Off-Heap Memory project. Here too, let us create a new jira, focusing on the following: Yes. Once you can get the CellChunkMap code into it - we can help you with the new type of Cell to avoid chunk copying and the MSLAB chunk creation related code which we have maintained internally. bq.The Big Cells issue. +1 to address this. Need to see what best can be done here. May be discuss what we call as a BigCell. > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch, > HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, > MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418380#comment-15418380
]
Hadoop QA commented on HBASE-15691:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 17s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 18s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 112m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestClusterId |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-12 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823396/HBASE-15691.v2-branch-1.patch
|
| JIRA Issue | HBASE-15691 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux fd805cbf558e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014
[jira] [Created] (HBASE-16405) Remove byte[] based hash() APIs.
ramkrishna.s.vasudevan created HBASE-16405: -- Summary: Remove byte[] based hash() APIs. Key: HBASE-16405 URL: https://issues.apache.org/jira/browse/HBASE-16405 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Currently the read path though it works with Cells, still does go with the assumption that we need to copy the Cells to KeyValue's key format and then use the byte[] version of hash() function to calculate the Bloom hash. In case of incoming cell's being KeyValue type we try to use the KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will get rid of all those and things will work with Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16404) Make DeleteBloomFilters work with BloomContext
[ https://issues.apache.org/jira/browse/HBASE-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-16404: -- Assignee: ramkrishna.s.vasudevan > Make DeleteBloomFilters work with BloomContext > -- > > Key: HBASE-16404 > URL: https://issues.apache.org/jira/browse/HBASE-16404 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > Since we have added BloomFiltercontext to work with RowBlooms the same can be > applied for DeleteBloomFilters also. Should be a simple change considering we > have the infrastructure in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16404) Make DeleteBloomFilters work with BloomContext
ramkrishna.s.vasudevan created HBASE-16404: -- Summary: Make DeleteBloomFilters work with BloomContext Key: HBASE-16404 URL: https://issues.apache.org/jira/browse/HBASE-16404 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15554) StoreFile$Writer.appendGeneralBloomFilter generates extra KV
[ https://issues.apache.org/jira/browse/HBASE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15554: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > StoreFile$Writer.appendGeneralBloomFilter generates extra KV > > > Key: HBASE-15554 > URL: https://issues.apache.org/jira/browse/HBASE-15554 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Vladimir Rodionov >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15554.patch, HBASE-15554_10.patch, > HBASE-15554_11.patch, HBASE-15554_12.patch, HBASE-15554_13.patch, > HBASE-15554_14.patch, HBASE-15554_3.patch, HBASE-15554_4.patch, > HBASE-15554_6.patch, HBASE-15554_7.patch, HBASE-15554_9.patch > > > Accounts for 10% memory allocation in compaction thread when BloomFilterType > is ROWCOL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15554) StoreFile$Writer.appendGeneralBloomFilter generates extra KV
[ https://issues.apache.org/jira/browse/HBASE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15554: --- Attachment: HBASE-15554_14.patch This is the patch that was pushed. Pushed to 2.0. Thanks for the nice reviews to get this in - [~anoop.hbase] and [~saint@gmail.com]. > StoreFile$Writer.appendGeneralBloomFilter generates extra KV > > > Key: HBASE-15554 > URL: https://issues.apache.org/jira/browse/HBASE-15554 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Vladimir Rodionov >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15554.patch, HBASE-15554_10.patch, > HBASE-15554_11.patch, HBASE-15554_12.patch, HBASE-15554_13.patch, > HBASE-15554_14.patch, HBASE-15554_3.patch, HBASE-15554_4.patch, > HBASE-15554_6.patch, HBASE-15554_7.patch, HBASE-15554_9.patch > > > Accounts for 10% memory allocation in compaction thread when BloomFilterType > is ROWCOL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418366#comment-15418366
]
Andrew Purtell commented on HBASE-16402:
Pushed
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16402:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[ https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418345#comment-15418345 ] yang ming commented on HBASE-16395: --- Thanks! [~dimaspivak] But why short circuit read fail only for log files everytime? > ShortCircuitLocalReads Failed when enabled replication > -- > > Key: HBASE-16395 > URL: https://issues.apache.org/jira/browse/HBASE-16395 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.20 >Reporter: yang ming >Priority: Critical > > I had sended an email to u...@hbase.apache.org,but received no help. > The cluster enabled shortCircuitLocalReads. > > dfs.client.read.shortcircuit > true > > When enabled replication,we found a large number of error logs. > 1.shortCircuitLocalReads(fail everytime). > 2.Try reading via the datanode on targetAddr(success). > How to make shortCircuitLocalReads successfully when enabled replication? > 2016-08-03 10:46:21,721 DEBUG > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening > log for replication dn7%2C60020%2C1470136216957.1470192327030 at 16999670 > 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient: > BlockReaderLocal requested with incorrect offset: Offset 0 and length > 17073479 don't match block blk_4137524355009640437_53760530 ( blockLen > 16999670 ) > 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient: > BlockReaderLocal: Removing blk_4137524355009640437_53760530 from cache > because local file > /sdd/hdfs/dfs/data/blocksBeingWritten/blk_4137524355009640437 could not be > opened. > 2016-08-03 10:46:21,724 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read > block blk_4137524355009640437_53760530 on local machinejava.io.IOException: > Offset 0 and length 17073479 don't match block > blk_4137524355009640437_53760530 ( blockLen 16999670 ) > at org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287) > at > org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171) > at org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358) > at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224) > at java.io.DataInputStream.read(DataInputStream.java:149) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178) > at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364) > 2016-08-03 10:46:21,724 INFO org.apache.hadoop.hdfs.DFSClient: Try reading > via the datanode on /192.168.7.139:50010 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16395) ShortCircuitLocalReads Failed when enabled replication
[ https://issues.apache.org/jira/browse/HBASE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418328#comment-15418328 ] yang ming commented on HBASE-16395: --- There is no similar fail,just only for log files. Configuration is as follows: dfs.client.read.shortcircuit true hbase.replication true replication.source.ratio 1 replication.executor.workers 5 replication.source.sleepforretries 5000 > ShortCircuitLocalReads Failed when enabled replication > -- > > Key: HBASE-16395 > URL: https://issues.apache.org/jira/browse/HBASE-16395 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.20 >Reporter: yang ming >Priority: Critical > > I had sended an email to u...@hbase.apache.org,but received no help. > The cluster enabled shortCircuitLocalReads. > > dfs.client.read.shortcircuit > true > > When enabled replication,we found a large number of error logs. > 1.shortCircuitLocalReads(fail everytime). > 2.Try reading via the datanode on targetAddr(success). > How to make shortCircuitLocalReads successfully when enabled replication? > 2016-08-03 10:46:21,721 DEBUG > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening > log for replication dn7%2C60020%2C1470136216957.1470192327030 at 16999670 > 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient: > BlockReaderLocal requested with incorrect offset: Offset 0 and length > 17073479 don't match block blk_4137524355009640437_53760530 ( blockLen > 16999670 ) > 2016-08-03 10:46:21,723 WARN org.apache.hadoop.hdfs.DFSClient: > BlockReaderLocal: Removing blk_4137524355009640437_53760530 from cache > because local file > /sdd/hdfs/dfs/data/blocksBeingWritten/blk_4137524355009640437 could not be > opened. > 2016-08-03 10:46:21,724 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read > block blk_4137524355009640437_53760530 on local machinejava.io.IOException: > Offset 0 and length 17073479 don't match block > blk_4137524355009640437_53760530 ( blockLen 16999670 ) > at org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:287) > at > org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:171) > at org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:358) > at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:74) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2073) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2224) > at java.io.DataInputStream.read(DataInputStream.java:149) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1486) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1470) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:55) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178) > at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:574) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:364) > 2016-08-03 10:46:21,724 INFO org.apache.hadoop.hdfs.DFSClient: Try reading > via the datanode on /192.168.7.139:50010 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16403) Upgrade guava version to the lastest one(19.0)
[ https://issues.apache.org/jira/browse/HBASE-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418296#comment-15418296 ] Duo Zhang commented on HBASE-16403: --- I believe there is a relocation when shading so the guava inside the jar is not under the original package? > Upgrade guava version to the lastest one(19.0) > -- > > Key: HBASE-16403 > URL: https://issues.apache.org/jira/browse/HBASE-16403 > Project: HBase > Issue Type: Improvement > Components: API, Client >Affects Versions: 1.1.4 >Reporter: lihongzhe > > HBase client use the guava version 12.0.1, while elasticsearch sdk use the > version 18+. It makes conflict, cause the constructor of StopWatch class is > not public now. > We have to choose the es rest api to resolve the conflict. Why not try to > upgrade our dependency versions? > thx. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418294#comment-15418294 ] Stephen Yuan Jiang commented on HBASE-15691: The order of the change in branch-1 is different from master branch. The V2 patches preserves the HBASE-14624 change in branch-1. > Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to > branch-1 > - > > Key: HBASE-15691 > URL: https://issues.apache.org/jira/browse/HBASE-15691 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1, 1.2.3 > > Attachments: HBASE-15691-branch-1.patch, HBASE-15691.v2-branch-1.patch > > > HBASE-10205 was committed to trunk and 0.98 branches only. To preserve > continuity we should commit it to branch-1. The change requires more than > nontrivial fixups so I will attach a backport of the change from trunk to > current branch-1 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418294#comment-15418294 ] Stephen Yuan Jiang edited comment on HBASE-15691 at 8/12/16 3:14 AM: - The order of the change in branch-1 is different from master branch. The V2 patch preserves the HBASE-14624 change in branch-1. was (Author: syuanjiang): The order of the change in branch-1 is different from master branch. The V2 patches preserves the HBASE-14624 change in branch-1. > Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to > branch-1 > - > > Key: HBASE-15691 > URL: https://issues.apache.org/jira/browse/HBASE-15691 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1, 1.2.3 > > Attachments: HBASE-15691-branch-1.patch, HBASE-15691.v2-branch-1.patch > > > HBASE-10205 was committed to trunk and 0.98 branches only. To preserve > continuity we should commit it to branch-1. The change requires more than > nontrivial fixups so I will attach a backport of the change from trunk to > current branch-1 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418295#comment-15418295 ] Phil Yang commented on HBASE-16388: --- It is configurable (default is 2147483647 which means disabled) and configuring the threshold should consider you concerns. The main objective is prevent slow region server blocking all client threads, so I think this threshold should be much higher than average number of threads accessing to one server, and is mainly depend on the number of threads in client. For example, if we have 100 threads in client, and we have 5 region servers. The threshold should be much higher than 20 to prevent failing on unbalanced-but-working servers, we can set the threshold to 50, so if a server is slow, we still have half of threads working well. If we have only one server, maybe this threshold is meaningless and we have to disable it. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-v1.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-15691: --- Attachment: HBASE-15691.v2-branch-1.patch > Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to > branch-1 > - > > Key: HBASE-15691 > URL: https://issues.apache.org/jira/browse/HBASE-15691 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1, 1.2.3 > > Attachments: HBASE-15691-branch-1.patch, HBASE-15691.v2-branch-1.patch > > > HBASE-10205 was committed to trunk and 0.98 branches only. To preserve > continuity we should commit it to branch-1. The change requires more than > nontrivial fixups so I will attach a backport of the change from trunk to > current branch-1 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16403) Upgrade guava version to the lastest one(19.0)
[ https://issues.apache.org/jira/browse/HBASE-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418291#comment-15418291 ] lihongzhe commented on HBASE-16403: --- shaded client is just package all dependent class into one jar. So, it's also have the old version guava class. We will also meet the conflict when we dependent newer version guava. Am i right? > Upgrade guava version to the lastest one(19.0) > -- > > Key: HBASE-16403 > URL: https://issues.apache.org/jira/browse/HBASE-16403 > Project: HBase > Issue Type: Improvement > Components: API, Client >Affects Versions: 1.1.4 >Reporter: lihongzhe > > HBase client use the guava version 12.0.1, while elasticsearch sdk use the > version 18+. It makes conflict, cause the constructor of StopWatch class is > not public now. > We have to choose the es rest api to resolve the conflict. Why not try to > upgrade our dependency versions? > thx. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16403) Upgrade guava version to the lastest one(19.0)
[ https://issues.apache.org/jira/browse/HBASE-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418283#comment-15418283 ] Duo Zhang commented on HBASE-16403: --- You can try the hbase-shaded-client. http://mvnrepository.com/artifact/org.apache.hbase/hbase-shaded-client > Upgrade guava version to the lastest one(19.0) > -- > > Key: HBASE-16403 > URL: https://issues.apache.org/jira/browse/HBASE-16403 > Project: HBase > Issue Type: Improvement > Components: API, Client >Affects Versions: 1.1.4 >Reporter: lihongzhe > > HBase client use the guava version 12.0.1, while elasticsearch sdk use the > version 18+. It makes conflict, cause the constructor of StopWatch class is > not public now. > We have to choose the es rest api to resolve the conflict. Why not try to > upgrade our dependency versions? > thx. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418234#comment-15418234 ] Allan Yang edited comment on HBASE-16388 at 8/12/16 1:37 AM: - what if there is only one region server serving? or what if the region on the regionserver is not balanced, some region server have a lot more regions than others. So threads to this server will naturally a lot more than others. was (Author: allan163): I don't think throttle the thread to one server is reasonable, what if there is only one region server serving? or what if the region on the regionserver is not balanced, some region server have a lot more regions than others. So threads to this server will naturally a lot more than others. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-v1.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418234#comment-15418234 ] Allan Yang commented on HBASE-16388: I don't think throttle the thread to one server is reasonable, what if there is only one region server serving? or what if the region on the regionserver is not balanced, some region server have a lot more regions than others. So threads to this server will naturally a lot more than others. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-v1.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418224#comment-15418224 ] binlijin edited comment on HBASE-16393 at 8/12/16 1:20 AM: --- Still need it when this is a fresh cluster startup, or there is many regionservers down? was (Author: aoxiang): Yes, still need it when this is a fresh cluster startup, or there is many regionservers down? > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418224#comment-15418224 ] binlijin commented on HBASE-16393: -- Yes, still need it when this is a fresh cluster startup, or there is many regionservers down? > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16197) Enchance backup delete command
[
https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418223#comment-15418223
]
Hadoop QA commented on HBASE-16197:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16197 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823387/HBASE-16197-v6.patch |
| JIRA Issue | HBASE-16197 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3073/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Enchance backup delete command
> --
>
> Key: HBASE-16197
> URL: https://issues.apache.org/jira/browse/HBASE-16197
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch,
> HBASE-16197-v4.patch, HBASE-16197-v5.patch, HBASE-16197-v6.patch
>
>
> Currently, backup delete command deletes only physical files/directories in
> backup destination. It does not update backup system table (incremental
> backup table set) and it does not delete related backup images (previous
> ones), thus creating a hole in dependency chain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16197) Enchance backup delete command
[ https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16197: -- Attachment: HBASE-16197-v6.patch > Enchance backup delete command > -- > > Key: HBASE-16197 > URL: https://issues.apache.org/jira/browse/HBASE-16197 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch, > HBASE-16197-v4.patch, HBASE-16197-v5.patch, HBASE-16197-v6.patch > > > Currently, backup delete command deletes only physical files/directories in > backup destination. It does not update backup system table (incremental > backup table set) and it does not delete related backup images (previous > ones), thus creating a hole in dependency chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418210#comment-15418210
]
Andrew Purtell commented on HBASE-16402:
Thanks Matteo. Running tests. Assuming a perfectly fine outcome. Will commit
soon
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16402:
---
Assignee: Guanghao Zhang
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16402:
---
Status: Patch Available (was: Open)
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16400:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 0.98.22
Status: Resolved (was: Patch Available)
Thanks for the review, Andrew.
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.22
>
> Attachments: 16400-0.98.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418191#comment-15418191
]
Matteo Bertozzi commented on HBASE-16402:
-
+1
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418186#comment-15418186
]
Hadoop QA commented on HBASE-16400:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 25s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionServerMetrics |
| | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823359/
[jira] [Updated] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16402:
---
Attachment: HBASE-16402-0.98.patch
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16402:
---
Description:
We found NPE in our 0.98 production cluster because RegionServerCoprocessorHost
is not initialized before RpcServer start service.
{code}
// Try and register with the Master; tell it we are here. Break if
// server is stopped or the clusterup flag is down or hdfs went wacky.
while (keepLooping()) {
RegionServerStartupResponse w = reportForDuty();
if (w == null) {
LOG.warn("reportForDuty failed; sleeping and then retrying.");
this.sleeper.sleep();
} else {
handleReportForDutyResponse(w);
break;
}
}
// Initialize the RegionServerCoprocessorHost now that our ephemeral
// node was created by reportForDuty, in case any coprocessors want
// to use ZooKeeper
this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
{code}
RpcServer start service in handleReportForDutyResponse(), then it can serve rpc
call replicateWALEntry(). But the RegionServerCoprocessorHost is not
initialized and it is used in replicateWALEntry, so it will throw a NPE.
was:
>From [~zghaobac] on HBASE-10338
{quote}
We found NPE in our 0.98 production cluster because RegionServerCoprocessorHost
is not initialized before RpcServer start service.
{code}
// Try and register with the Master; tell it we are here. Break if
// server is stopped or the clusterup flag is down or hdfs went wacky.
while (keepLooping()) {
RegionServerStartupResponse w = reportForDuty();
if (w == null) {
LOG.warn("reportForDuty failed; sleeping and then retrying.");
this.sleeper.sleep();
} else {
handleReportForDutyResponse(w);
break;
}
}
// Initialize the RegionServerCoprocessorHost now that our ephemeral
// node was created by reportForDuty, in case any coprocessors want
// to use ZooKeeper
this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
{code}
RpcServer start service in handleReportForDutyResponse(), then it can serve rpc
call replicateWALEntry(). But the RegionServerCoprocessorHost is not
initialized and it is used in replicateWALEntry, so it will throw a NPE.
{quote}
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
> Fix For: 0.98.22
>
> Attachments: HBASE-16402-0.98.patch
>
>
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16402) RegionServerCoprocessorHost should be initialized before RpcServer starts
[
https://issues.apache.org/jira/browse/HBASE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16402:
---
Reporter: Guanghao Zhang (was: Andrew Purtell)
> RegionServerCoprocessorHost should be initialized before RpcServer starts
> -
>
> Key: HBASE-16402
> URL: https://issues.apache.org/jira/browse/HBASE-16402
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
> Fix For: 0.98.22
>
>
> From [~zghaobac] on HBASE-10338
> {quote}
> We found NPE in our 0.98 production cluster because
> RegionServerCoprocessorHost is not initialized before RpcServer start service.
> {code}
> // Try and register with the Master; tell it we are here. Break if
> // server is stopped or the clusterup flag is down or hdfs went wacky.
> while (keepLooping()) {
> RegionServerStartupResponse w = reportForDuty();
> if (w == null) {
> LOG.warn("reportForDuty failed; sleeping and then retrying.");
> this.sleeper.sleep();
> } else {
> handleReportForDutyResponse(w);
> break;
> }
> }
> // Initialize the RegionServerCoprocessorHost now that our ephemeral
> // node was created by reportForDuty, in case any coprocessors want
> // to use ZooKeeper
> this.rsHost = new RegionServerCoprocessorHost(this, this.conf);
> {code}
> RpcServer start service in handleReportForDutyResponse(), then it can serve
> rpc call replicateWALEntry(). But the RegionServerCoprocessorHost is not
> initialized and it is used in replicateWALEntry, so it will throw a NPE.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418154#comment-15418154 ] Andrew Purtell commented on HBASE-10338: Thanks [~zghaobac]. Let's move this to new issue HBASE-16402 > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338-0.98-addendum.patch, HBASE-10338.0.patch, > HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418150#comment-15418150
]
Andrew Purtell commented on HBASE-16400:
+1
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 16400-0.98.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16196) Update jruby to a newer version.
[
https://issues.apache.org/jira/browse/HBASE-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418130#comment-15418130
]
Hadoop QA commented on HBASE-16196:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 5s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} branch-1 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s
{color} | {color:green} branch-1 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 26s
{color} | {color:red} The patch generated 44 new + 1760 unchanged - 52 fixed =
1804 total (was 1812) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 18s
{color} | {color:red} The patch generated 15 new + 1123 unchanged - 16 fixed =
1138 total (was 1139) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 7s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 9s
{color} | {color:red} root in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:g
[jira] [Commented] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418129#comment-15418129
]
Hadoop QA commented on HBASE-15691:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 6s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 30s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800155/HBASE-15691-branch-1.patch
|
| JIRA Issue | HBASE-15691 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c8f0c479def0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:5
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418107#comment-15418107 ] Ted Yu commented on HBASE-14450: At the end of BackupdistCp#execute(), distcp job was marked unsuccessful. Yet in IncrementalTableBackupProcedure#incrementalCopy(), the return value from copyService.copy() was 0. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16255) Backup/Restore IT
[ https://issues.apache.org/jira/browse/HBASE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418092#comment-15418092 ] Dima Spivak commented on HBASE-16255: - Reviewed! > Backup/Restore IT > - > > Key: HBASE-16255 > URL: https://issues.apache.org/jira/browse/HBASE-16255 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: HBASE-16255-v1.patch, HBASE-16255-v2.patch, > HBASE-16255-v3.patch > > > Integration test for backup restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[
https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418078#comment-15418078
]
Ted Yu commented on HBASE-14450:
{code}
2016-08-11 15:43:58,313 DEBUG [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService$BackupDistCp(272): Counters: 20
File System Counters
FILE: Number of bytes read=94091779
FILE: Number of bytes written=94381659
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=73436
HDFS: Number of bytes written=2236101
HDFS: Number of read operations=528
HDFS: Number of large read operations=0
HDFS: Number of write operations=263
Map-Reduce Framework
Map input records=1
Map output records=0
Input split bytes=261
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=1215823872
File Input Format Counters
Bytes Read=2056
File Output Format Counters
Bytes Written=0
2016-08-11 15:43:58,313 DEBUG [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService(319): list of
hdfs://localhost:56032/backupUT/backup_1470955428847/WALs for distcp
{code}
The counters didn't include any FAIL counter. Yet the destination directory was
empty after distcp job exited.
> HBase Backup/Restore Phase 3: Multiwal support
> --
>
> Key: HBASE-14450
> URL: https://issues.apache.org/jira/browse/HBASE-14450
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14450.v2.txt,
> org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt
>
>
> We need to support multiwal configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418054#comment-15418054 ] Andrew Purtell commented on HBASE-15691: No plans [~syuanjiang], please proceed if you like > Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to > branch-1 > - > > Key: HBASE-15691 > URL: https://issues.apache.org/jira/browse/HBASE-15691 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1, 1.2.3 > > Attachments: HBASE-15691-branch-1.patch > > > HBASE-10205 was committed to trunk and 0.98 branches only. To preserve > continuity we should commit it to branch-1. The change requires more than > nontrivial fixups so I will attach a backport of the change from trunk to > current branch-1 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-16400:
--
Assignee: Ted Yu
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 16400-0.98.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16400:
---
Attachment: 16400-0.98.v1.txt
With the patch, TestHBaseFsck#testQuarantineMissingRegionDir passes against
hadoop-1.1
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16400-0.98.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418041#comment-15418041
]
Ted Yu commented on HBASE-16400:
TestHBaseFsck also passes with patch against hadoop-1.1.
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16400-0.98.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16400:
---
Status: Patch Available (was: Open)
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16400-0.98.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
[
https://issues.apache.org/jira/browse/HBASE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418003#comment-15418003
]
Andrew Purtell commented on HBASE-16400:
{noformat}
$ mvn clean install -Dhadoop.version=1.1 -DskipITs test=TestHBaseFsck
Running org.apache.hadoop.hbase.util.TestHBaseFsck
Tests run: 47, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 233.387 sec -
in org.apache.hadoop.hbase.util.TestHBaseFsck
Results :
Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
{noformat}
Would you be willing to look into this since you can reproduce it [~ted_yu]?
> TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
> -
>
> Key: HBASE-16400
> URL: https://issues.apache.org/jira/browse/HBASE-16400
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.21
>Reporter: Ted Yu
>Priority: Minor
>
> From
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
> at
> org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> {code}
> Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15691) Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417994#comment-15417994 ] Stephen Yuan Jiang commented on HBASE-15691: [~apurtell], do you plan to commit it in branch-1 any time soon? I can make another patch if you are busy. We have a customer hitting this issue in 1.x line. (it is strange that HBASE-10205 claims to commit in 0.99, but not in branch-1). > Port HBASE-10205 (ConcurrentModificationException in BucketAllocator) to > branch-1 > - > > Key: HBASE-15691 > URL: https://issues.apache.org/jira/browse/HBASE-15691 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1, 1.2.3 > > Attachments: HBASE-15691-branch-1.patch > > > HBASE-10205 was committed to trunk and 0.98 branches only. To preserve > continuity we should commit it to branch-1. The change requires more than > nontrivial fixups so I will attach a backport of the change from trunk to > current branch-1 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16196) Update jruby to a newer version.
[
https://issues.apache.org/jira/browse/HBASE-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Mullins updated HBASE-16196:
-
Attachment: hbase-16196.v3.branch-1.patch
The fix for the previous {{$LOAD_PATH}} confusion was simple — Ruby 1.9+ adds
{{require_relative}}, so I've used that for {{region_mover.rb}}.
I have manually tested {{region_mover.rb}}, {{draining_servers.rb}},
{{get-active-master.rb}}, {{region_status.rb}} and
{{shutdown_regionserver.rb}}, and I haven't noticed any problems. Since I also
touched {{taskmonitor.rb}}, I specifically verified that the shell's
{{processlist}} command does still function and does not have trouble finding
Jackson classes.
> Update jruby to a newer version.
>
>
> Key: HBASE-16196
> URL: https://issues.apache.org/jira/browse/HBASE-16196
> Project: HBase
> Issue Type: Bug
> Components: dependencies, shell
>Reporter: Elliott Clark
>Assignee: Matt Mullins
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 0001-Update-to-JRuby-9.1.2.0-and-JLine-2.12.patch,
> hbase-16196.branch-1.patch, hbase-16196.v2.branch-1.patch,
> hbase-16196.v3.branch-1.patch
>
>
> Ruby 1.8.7 is no longer maintained.
> The TTY library in the old jruby is bad. The newer one is less bad.
> Since this is only a dependency on the hbase-shell module and not on
> hbase-client or hbase-server this should be a pretty simple thing that
> doesn't have any backwards compat issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417917#comment-15417917 ] Ted Yu commented on HBASE-14450: See my comment 4 hours ago. With 'FSHLog$SyncRunner(1275): syncing writer' log, I identified two WAL files which were written after the full backup was done. These two files didn't appear in the 'Mismatch in length' errors. Since actual map progress was 16% when the job stopped, looks like some WAL files weren't even copied. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417912#comment-15417912 ] Vladimir Rodionov commented on HBASE-14450: --- Is it possible that we do not close WAL files correctly before distcp starts? DistCp copies file to destination than (for some reason) checks src/dest file lengths. They do not match meaning that src file length changed? > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417895#comment-15417895 ] Ted Yu commented on HBASE-14450: Please look at attached log. WAL files in src were listed under the line with: backup.BackupInfo(313): setting incr backup file list > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417885#comment-15417885 ] Vladimir Rodionov commented on HBASE-14450: --- [~ted yu] can you post the list of WAL files in src and backup destination? Files come from different folder in src, can this be the problem in DistCp? > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417873#comment-15417873 ] Ted Yu commented on HBASE-14450: Manually verified that the 6 WAL files passed to setIncrBackupFileList() appeared in the argument list of distcp. The 16% map progress seemed to imply that only one WAL file was actually copied. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[
https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417830#comment-15417830
]
Hadoop QA commented on HBASE-16094:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 3s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 56s {color}
| {color:red} hbase-procedure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.procedure2.store.wal.TestWALProcedureStore |
| | hadoop.hbase.procedure2.TestProcedureRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823325/HBASE-16094.master.001.patch
|
| JIRA Issue | HBASE-16094 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9cf978918010 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreC
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[
https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417826#comment-15417826
]
Ted Yu commented on HBASE-14450:
This is related code in BackupDistCp#execute() when job.isComplete() becomes
true:
{code}
// update the progress data after copy job complete
float newProgress =
progressDone + job.mapProgress() * subTaskPercntgInWholeTask * (1 -
INIT_PROGRESS);
BigDecimal progressData =
new BigDecimal(newProgress * 100).setScale(1,
BigDecimal.ROUND_HALF_UP);
{code}
Here is related output:
{code}
2016-08-11 12:39:26,658 INFO [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService$BackupDistCp(242): Progress: 24.5%
subTask: 1.0 mapProgress: 0.16099072
{code}
As you can see, mapProgress was 16% when the distcp stopped.
I wonder why mapProgress was so low.
> HBase Backup/Restore Phase 3: Multiwal support
> --
>
> Key: HBASE-14450
> URL: https://issues.apache.org/jira/browse/HBASE-14450
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14450.v2.txt,
> org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt
>
>
> We need to support multiwal configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16255) Backup/Restore IT
[ https://issues.apache.org/jira/browse/HBASE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417804#comment-15417804 ] Vladimir Rodionov commented on HBASE-16255: --- Here it is: https://reviews.apache.org/r/51003/ > Backup/Restore IT > - > > Key: HBASE-16255 > URL: https://issues.apache.org/jira/browse/HBASE-16255 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: HBASE-16255-v1.patch, HBASE-16255-v2.patch, > HBASE-16255-v3.patch > > > Integration test for backup restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16197) Enchance backup delete command
[
https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417801#comment-15417801
]
Hadoop QA commented on HBASE-16197:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16197 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823332/HBASE-16197-v5.patch |
| JIRA Issue | HBASE-16197 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3068/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Enchance backup delete command
> --
>
> Key: HBASE-16197
> URL: https://issues.apache.org/jira/browse/HBASE-16197
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch,
> HBASE-16197-v4.patch, HBASE-16197-v5.patch
>
>
> Currently, backup delete command deletes only physical files/directories in
> backup destination. It does not update backup system table (incremental
> backup table set) and it does not delete related backup images (previous
> ones), thus creating a hole in dependency chain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16197) Enchance backup delete command
[ https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16197: -- Attachment: HBASE-16197-v5.patch v5. Added new unit test. > Enchance backup delete command > -- > > Key: HBASE-16197 > URL: https://issues.apache.org/jira/browse/HBASE-16197 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch, > HBASE-16197-v4.patch, HBASE-16197-v5.patch > > > Currently, backup delete command deletes only physical files/directories in > backup destination. It does not update backup system table (incremental > backup table set) and it does not delete related backup images (previous > ones), thus creating a hole in dependency chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16401) Enable HeapMemoryManager by default in 2.0
[ https://issues.apache.org/jira/browse/HBASE-16401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417779#comment-15417779 ] stack commented on HBASE-16401: --- [~anoop.hbase] Any suggestion for what ranges to use as defaults? Thanks boss. > Enable HeapMemoryManager by default in 2.0 > -- > > Key: HBASE-16401 > URL: https://issues.apache.org/jira/browse/HBASE-16401 > Project: HBase > Issue Type: Improvement >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > Back in HBASE-5349, on the end of the issue, we talked about enabling > HeapMemoryManager by default. Lets do it for 2.0 with some conservative > boundaries. Do it now so we have some experience running it before release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16401) Enable HeapMemoryManager by default in 2.0
[ https://issues.apache.org/jira/browse/HBASE-16401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417750#comment-15417750 ] stack commented on HBASE-16401: --- Make sure the metrics in HBASE-14743 help when we enable HeapMemoryManager as our default. > Enable HeapMemoryManager by default in 2.0 > -- > > Key: HBASE-16401 > URL: https://issues.apache.org/jira/browse/HBASE-16401 > Project: HBase > Issue Type: Improvement >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > Back in HBASE-5349, on the end of the issue, we talked about enabling > HeapMemoryManager by default. Lets do it for 2.0 with some conservative > boundaries. Do it now so we have some experience running it before release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16401) Enable HeapMemoryManager by default in 2.0
stack created HBASE-16401: - Summary: Enable HeapMemoryManager by default in 2.0 Key: HBASE-16401 URL: https://issues.apache.org/jira/browse/HBASE-16401 Project: HBase Issue Type: Improvement Reporter: stack Priority: Critical Fix For: 2.0.0 Back in HBASE-5349, on the end of the issue, we talked about enabling HeapMemoryManager by default. Lets do it for 2.0 with some conservative boundaries. Do it now so we have some experience running it before release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[ https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16094: - Assignee: Appy Status: Patch Available (was: Open) > Procedure v2 - Improve cleaning up of proc wals > --- > > Key: HBASE-16094 > URL: https://issues.apache.org/jira/browse/HBASE-16094 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16094.master.001.patch > > > Avoid accumulating too many wals. > We remove logs in 3 cases: > - No procedures are running. We can remove all the logs. > - All procedures are updated/written in the last log. We can remove all the > logs, aside the active one. > - Remove log if does not contains “active” procIds > https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 > The last point, Remove log if does not contains “active” procIds, needs to be > improved. Basically, even if a log contains state of an active proc, we can > delete it if a later log contains a newer state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[ https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16094: - Attachment: HBASE-16094.master.001.patch > Procedure v2 - Improve cleaning up of proc wals > --- > > Key: HBASE-16094 > URL: https://issues.apache.org/jira/browse/HBASE-16094 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16094.master.001.patch > > > Avoid accumulating too many wals. > We remove logs in 3 cases: > - No procedures are running. We can remove all the logs. > - All procedures are updated/written in the last log. We can remove all the > logs, aside the active one. > - Remove log if does not contains “active” procIds > https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 > The last point, Remove log if does not contains “active” procIds, needs to be > improved. Basically, even if a log contains state of an active proc, we can > delete it if a later log contains a newer state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[ https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16094: - Description: Avoid accumulating too many wals. We remove logs in 3 cases: - No procedures are running. We can remove all the logs. - All procedures are updated/written in the last log. We can remove all the logs, aside the active one. - Remove log if does not contains “active” procIds https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 The last point, Remove log if does not contains “active” procIds, needs to be improved. Basically, even if a log contains state of an active proc, we can delete it if a later log contains a newer state. was: Avoid accumulating too many wals. We remove logs in 3 cases: - No procedures are running. We can remove all the logs. - All procedures are updated/written in the last log. We can remove all the logs, aside the active one. - Remove log if does not contains “active” procIds https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 Remove log if does not contains “active” procIds requires to be improved, since those procedures can be in earlier wals. An example that can be unit-tested is: Insert 10 proc, Roll The Wal, Update the 10 proc At the end of the 10 updates the removeInactiveLogs() should be able to remove the first WAL. We can probably do a diff on the updated tracker bitmap between wals. > Procedure v2 - Improve cleaning up of proc wals > --- > > Key: HBASE-16094 > URL: https://issues.apache.org/jira/browse/HBASE-16094 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Appy > Fix For: 2.0.0 > > > Avoid accumulating too many wals. > We remove logs in 3 cases: > - No procedures are running. We can remove all the logs. > - All procedures are updated/written in the last log. We can remove all the > logs, aside the active one. > - Remove log if does not contains “active” procIds > https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 > The last point, Remove log if does not contains “active” procIds, needs to be > improved. Basically, even if a log contains state of an active proc, we can > delete it if a later log contains a newer state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16255) Backup/Restore IT
[ https://issues.apache.org/jira/browse/HBASE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417722#comment-15417722 ] Dima Spivak commented on HBASE-16255: - [~vrodionov] could you please upload the patch to RB? I have some minor comments and it's a pain to make them here on JIRA. I'll also get a run of this underway today and then +1 it on my end. Thanks so much for doing this! > Backup/Restore IT > - > > Key: HBASE-16255 > URL: https://issues.apache.org/jira/browse/HBASE-16255 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: HBASE-16255-v1.patch, HBASE-16255-v2.patch, > HBASE-16255-v3.patch > > > Integration test for backup restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417594#comment-15417594 ] Ted Yu edited comment on HBASE-14450 at 8/11/16 5:58 PM: - I plan to dig some more into BackupDistCp code. But what confuses me was that the two WAL files in 'FSHLog$SyncRunner(1275): syncing writer' logs in the earlier snippet didn't encounter the Mismatch in length error. Meaning, even without the 'Mismatch in length', likely the test would still fail. was (Author: yuzhih...@gmail.com): I plan to dig some more into distcp code - likely the copy succeeded on subsequent attempt. But what confuses me was that the two WAL files in 'FSHLog$SyncRunner(1275): syncing writer' logs in the earlier snippet didn't encounter the Mismatch in length error. Meaning, even without the 'Mismatch in length', likely the test would still fail. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16255) Backup/Restore IT
[ https://issues.apache.org/jira/browse/HBASE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417673#comment-15417673 ] Ted Yu commented on HBASE-16255: [~dimaspivak]: Is patch v3 good by you ? > Backup/Restore IT > - > > Key: HBASE-16255 > URL: https://issues.apache.org/jira/browse/HBASE-16255 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: HBASE-16255-v1.patch, HBASE-16255-v2.patch, > HBASE-16255-v3.patch > > > Integration test for backup restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[
https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417670#comment-15417670
]
Ted Yu commented on HBASE-14450:
>From the same test:
{code}
2016-08-10 15:46:24,236 WARN [Thread-2327] mapred.LocalJobRunner$Job(560):
job_local1643260168_0005
java.lang.Exception: java.io.IOException: File copy failed:
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.
34%2C56226%2C1470869103454.regiongroup-1.1470869108161 -->
hdfs://localhost:56218/backupUT/backup_1470869176664/WALs/10.22.16.34%2C56226%2C1470869103454.regiongroup-1.1470869108161
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.IOException: File copy failed:
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-1.1470869108161
-->
hdfs://localhost:56218/backupUT/backup_1470869176664/WALs/10.22.16.34%2C56226%2C1470869103454.regiongroup-1.1470869108161
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:285)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.
34%2C56226%2C1470869103454.regiongroup-1.1470869108161 to
hdfs://localhost:56218/backupUT/backup_1470869176664/WALs/10.22.16.34%2C56226%2C1470869103454.regiongroup-1.1470869108161
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:282)
... 11 more
Caused by: java.io.IOException: Mismatch in length of
source:hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.
34%2C56226%2C1470869103454.regiongroup-1.1470869108161 and
target:hdfs://localhost:56218/backupUT/backup_1470869176664/WALs/.distcp.tmp.attempt_local1643260168_0005_m_00_0
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 12 more
2016-08-10 15:46:24,391 INFO
[org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@5c64f59]
blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:56219
to delete [blk_1073741892_1068, blk_1073741893_1069]
2016-08-10 15:46:24,622 INFO [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService$BackupDistCp(242): Progress: 10.0%
2016-08-10 15:46:24,622 DEBUG [ProcedureExecutor-5]
impl.BackupSystemTable(122): update backup status in hbase:backup for:
backup_1470869176664 set status=RUNNING
2016-08-10 15:46:24,625 DEBUG [sync.0] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-1.1470869110496
2016-08-10 15:46:24,626 DEBUG [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService(135): Backup progress data "10%" has been
updated to hbase:backup for backup_1470869176664
2016-08-10 15:46:24,626 DEBUG [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService$BackupDistCp(250): Backup progress data
updated to hbase:backup: "Progress: 10.0% - 514 bytes copied."
2016-08-10 15:46:24,626 DEBUG [ProcedureExecutor-5]
mapreduce.MapReduceBackupCopyService$BackupDistCp(262): DistCp job-id:
job_local1643260168_0005
{code}
Note the distcp job failed at 10% progress.
There was no other distcp job launched after the failed one.
Ran the test with had
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417630#comment-15417630 ] Elliott Clark commented on HBASE-16393: --- So that one computes the calls after the first one in parallel. There is still a lot of work needed to make the first invocation happen in parallel. > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417594#comment-15417594 ] Ted Yu edited comment on HBASE-14450 at 8/11/16 5:13 PM: - I plan to dig some more into distcp code - likely the copy succeeded on subsequent attempt. But what confuses me was that the two WAL files in 'FSHLog$SyncRunner(1275): syncing writer' logs in the earlier snippet didn't encounter the Mismatch in length error. Meaning, even without the 'Mismatch in length', likely the test would still fail. was (Author: yuzhih...@gmail.com): I plan to dig some more into distcp code. But what confuses me was that the two WAL files in 'FSHLog$SyncRunner(1275): syncing writer' logs in the earlier snippet didn't encounter the Mismatch in length error. Meaning, even without the 'Mismatch in length', likely the test would still fail. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417594#comment-15417594 ] Ted Yu commented on HBASE-14450: I plan to dig some more into distcp code. But what confuses me was that the two WAL files in 'FSHLog$SyncRunner(1275): syncing writer' logs in the earlier snippet didn't encounter the Mismatch in length error. Meaning, even without the 'Mismatch in length', likely the test would still fail. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417589#comment-15417589 ] Vladimir Rodionov commented on HBASE-14450: --- [~tedyu], is this reproducible? Seems like distcp/hdfs issue. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16383) TableMapReduceUtil should allow using JobConf, not Job
[ https://issues.apache.org/jira/browse/HBASE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolai Grigoriev resolved HBASE-16383. --- Resolution: Invalid > TableMapReduceUtil should allow using JobConf, not Job > -- > > Key: HBASE-16383 > URL: https://issues.apache.org/jira/browse/HBASE-16383 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Nikolai Grigoriev >Priority: Minor > > I was recently trying to set up an MR job initialized with TableMapReduceUtil > to be ran by Oozie. Oozie offers a way to configure the job by providing an > implementation of org.apache.oozie.action.hadoop.OozieActionConfigurator. > Unfortunately, this interface is using org.apache.hadoop.mapred.JobConf, not > Job. I may be wrong but I believe that everything TableMapReduceUtil does > actually maps to the job configuration so it does not really need the Job > itself. If this is true, probably it would be more appropriate to use JobConf > in TableMapReduceUtil instead of Job? Or provide the methods for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16383) TableMapReduceUtil should allow using JobConf, not Job
[ https://issues.apache.org/jira/browse/HBASE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417577#comment-15417577 ] Nikolai Grigoriev commented on HBASE-16383: --- OK, I think I see the point. I have just realized that other API classes in mapreduce package use Job. I agree with what you said about Oozie. Thanks. > TableMapReduceUtil should allow using JobConf, not Job > -- > > Key: HBASE-16383 > URL: https://issues.apache.org/jira/browse/HBASE-16383 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Nikolai Grigoriev >Priority: Minor > > I was recently trying to set up an MR job initialized with TableMapReduceUtil > to be ran by Oozie. Oozie offers a way to configure the job by providing an > implementation of org.apache.oozie.action.hadoop.OozieActionConfigurator. > Unfortunately, this interface is using org.apache.hadoop.mapred.JobConf, not > Job. I may be wrong but I believe that everything TableMapReduceUtil does > actually maps to the job configuration so it does not really need the Job > itself. If this is true, probably it would be more appropriate to use JobConf > in TableMapReduceUtil instead of Job? Or provide the methods for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[
https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417539#comment-15417539
]
Ted Yu commented on HBASE-14450:
In test output, I saw:
{code}
2016-08-10 15:46:19,174 ERROR [LocalJobRunner Map Task Executor #0]
util.RetriableCommand(89): Failure in Retriable command: Copying
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-
39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-1.1470869108161
to hdfs://localhost:56218/backupUT/backup_1470869176664/WALs/10.22.16.
34%2C56226%2C1470869103454.regiongroup-1.1470869108161
java.io.IOException: Mismatch in length of
source:hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-1.1470869108161
and
target:hdfs://localhost:56218/backupUT/backup_1470869176664/WALs/.distcp.tmp.attempt_local1643260168_0005_m_00_0
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:282)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
{code}
> HBase Backup/Restore Phase 3: Multiwal support
> --
>
> Key: HBASE-14450
> URL: https://issues.apache.org/jira/browse/HBASE-14450
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14450.v2.txt,
> org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt
>
>
> We need to support multiwal configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16383) TableMapReduceUtil should allow using JobConf, not Job
[
https://issues.apache.org/jira/browse/HBASE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417506#comment-15417506
]
Sean Busbey commented on HBASE-16383:
-
Both the {{mapred}} and {{mapreduce}} APIs are supported upstream in Hadoop
last I checked. We provide implementations for both, but in each we stick to
the conventions of that API. In the {{mapreduce}} API the {{Job}} object is the
submitters' view of the submittal / configuration.
If Oozie is only providing the mechanisms from the {{mapred}} API, why not just
stick with that? Or ask them to provide a {{mapreduce}} oriented version?
> TableMapReduceUtil should allow using JobConf, not Job
> --
>
> Key: HBASE-16383
> URL: https://issues.apache.org/jira/browse/HBASE-16383
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Nikolai Grigoriev
>Priority: Minor
>
> I was recently trying to set up an MR job initialized with TableMapReduceUtil
> to be ran by Oozie. Oozie offers a way to configure the job by providing an
> implementation of org.apache.oozie.action.hadoop.OozieActionConfigurator.
> Unfortunately, this interface is using org.apache.hadoop.mapred.JobConf, not
> Job. I may be wrong but I believe that everything TableMapReduceUtil does
> actually maps to the job configuration so it does not really need the Job
> itself. If this is true, probably it would be more appropriate to use JobConf
> in TableMapReduceUtil instead of Job? Or provide the methods for both.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417487#comment-15417487
]
Xiang Li edited comment on HBASE-14882 at 8/11/16 4:00 PM:
---
Got that, it is a good idea to use anonymous Cell! Thanks Stack!
was (Author: water):
Got that, anonymous Cell is a good idea! Thanks stack!
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Xiang Li
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417487#comment-15417487
]
Xiang Li commented on HBASE-14882:
--
Got that, anonymous Cell is a good idea! Thanks stack!
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Xiang Li
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiang Li updated HBASE-14882:
-
Description:
In the Put API, we have addImmutable()
{code}
/**
* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
* that the underlying arrays won't change. It's intended
* for usage internal HBase to and for advanced client applications.
*/
public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
{code}
But in the implementation, the family, qualifier and value are still being
copied locally to create kv.
Hopefully we should provide an API that truly uses immutable family, qualifier
and value.
was:
In the Put API, we have addImmutable()
{code}
/**
* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
* that the underlying arrays won't change. It's intended
* for usage internal HBase to and for advanced client applications.
*/
public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
{code}
But in the implementation the row, family. qualifier and value are still being
copied locally to create kv.
Hopefully we should provide an API that truely uses immutable family, qualifier
and value.
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Xiang Li
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16383) TableMapReduceUtil should allow using JobConf, not Job
[ https://issues.apache.org/jira/browse/HBASE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417476#comment-15417476 ] Nikolai Grigoriev commented on HBASE-16383: --- I know but I always assumed "mapred" package is for old API and mapreduce is for new one. Thus, given the choice, I would go with new API for anything new. So with this improvement request I am challenging the move from JobConf (old API) to Job (new API). I was trying to search for the origin of that move but could not find any traces. > TableMapReduceUtil should allow using JobConf, not Job > -- > > Key: HBASE-16383 > URL: https://issues.apache.org/jira/browse/HBASE-16383 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Nikolai Grigoriev >Priority: Minor > > I was recently trying to set up an MR job initialized with TableMapReduceUtil > to be ran by Oozie. Oozie offers a way to configure the job by providing an > implementation of org.apache.oozie.action.hadoop.OozieActionConfigurator. > Unfortunately, this interface is using org.apache.hadoop.mapred.JobConf, not > Job. I may be wrong but I believe that everything TableMapReduceUtil does > actually maps to the job configuration so it does not really need the Job > itself. If this is true, probably it would be more appropriate to use JobConf > in TableMapReduceUtil instead of Job? Or provide the methods for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16383) TableMapReduceUtil should allow using JobConf, not Job
[ https://issues.apache.org/jira/browse/HBASE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417426#comment-15417426 ] Sean Busbey commented on HBASE-16383: - The mapred version already uses JobConf. Take a look at [org.apache.hadoop.hbase.mapred.TableMapReduceUtil|https://github.com/apache/hbase/blob/rel/1.2.2/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java]. > TableMapReduceUtil should allow using JobConf, not Job > -- > > Key: HBASE-16383 > URL: https://issues.apache.org/jira/browse/HBASE-16383 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Nikolai Grigoriev >Priority: Minor > > I was recently trying to set up an MR job initialized with TableMapReduceUtil > to be ran by Oozie. Oozie offers a way to configure the job by providing an > implementation of org.apache.oozie.action.hadoop.OozieActionConfigurator. > Unfortunately, this interface is using org.apache.hadoop.mapred.JobConf, not > Job. I may be wrong but I believe that everything TableMapReduceUtil does > actually maps to the job configuration so it does not really need the Job > itself. If this is true, probably it would be more appropriate to use JobConf > in TableMapReduceUtil instead of Job? Or provide the methods for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16400) TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
Ted Yu created HBASE-16400:
--
Summary: TestHBaseFsck fails on HBase-0.98-on-Hadoop-1.1 build
Key: HBASE-16400
URL: https://issues.apache.org/jira/browse/HBASE-16400
Project: HBase
Issue Type: Test
Affects Versions: 0.98.21
Reporter: Ted Yu
Priority: Minor
>From
>https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1259/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testQuarantineMissingRegionDir/
> :
{code}
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker.checkRegionDir(HFileCorruptionChecker.java:201)
at
org.apache.hadoop.hbase.util.TestHBaseFsck$3$1.checkRegionDir(TestHBaseFsck.java:2013)
at
org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:293)
at
org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker$RegionDirChecker.call(HFileCorruptionChecker.java:284)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
{code}
Local run with hadoop-1.1 reproduced the above test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16267:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
TestLogLevel passes in master branch.
Thanks for the review, Stack.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v15.txt, 16267.v2.txt,
> 16267.v4.txt, 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[ https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15199: Fix Version/s: 1.4.0 2.0.0 > Move jruby jar so only on hbase-shell module classpath; currently globally > available > > > Key: HBASE-15199 > URL: https://issues.apache.org/jira/browse/HBASE-15199 > Project: HBase > Issue Type: Task > Components: dependencies, jruby, shell >Reporter: stack > Fix For: 2.0.0, 1.4.0 > > > A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) > was to move the scope of the jruby include down so it is only a dependency > for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time > for example) which can clash with the includes of others. Our Sean suggests > that could be good to shut down exploit possibilities if jruby was not > globally available. Only downside I can think is that it may no longer be > available to our bin/*rb scripts if we move the jar but perhaps these can be > changed so they can find the ruby jar in new location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[ https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15199: Component/s: dependencies > Move jruby jar so only on hbase-shell module classpath; currently globally > available > > > Key: HBASE-15199 > URL: https://issues.apache.org/jira/browse/HBASE-15199 > Project: HBase > Issue Type: Task > Components: dependencies, jruby, shell >Reporter: stack >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > > A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) > was to move the scope of the jruby include down so it is only a dependency > for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time > for example) which can clash with the includes of others. Our Sean suggests > that could be good to shut down exploit possibilities if jruby was not > globally available. Only downside I can think is that it may no longer be > available to our bin/*rb scripts if we move the jar but perhaps these can be > changed so they can find the ruby jar in new location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
