[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548148#comment-14548148 ] stack commented on HBASE-13071: --- [~eshcar] Thanks for finding issue. Please open new issue. This one is dense enough already. Thank you (FYI, you do not need to clean up old patches -- thanks). Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071-0_98.patch, HBASE-13071-BRANCH-1.patch, HBASE-13071-trunk-bug-fix.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13705) MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false
[ https://issues.apache.org/jira/browse/HBASE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547926#comment-14547926 ] Aleksandr Maksymenko commented on HBASE-13705: -- I'm not sure if I understand you. If startRowInclusive is false, then startRow should NOT be included. So, included rows should be strictly greater then startRow. MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false --- Key: HBASE-13705 URL: https://issues.apache.org/jira/browse/HBASE-13705 Project: HBase Issue Type: Bug Affects Versions: 1.1.0 Reporter: Aleksandr Maksymenko I've found the issue during code review, so I don't have tests and I even didn't test this case manualy. So I'll try to describe it in words. Pre-condition: we're using scan with MultiRowRangeFilter with some RowRange's with startRowInclusive = false. This means that we want to include all rows that are strictly greater than startRow (and less then stopRow, but it doesn't matter for now). What happens in MultiRowRangeFilter.filterRowKey (worth case is described): 1. Line 91: Check if current range contains a row. Lets follow the case when it doesn't. 2. Line 94: Search for the next RowRange in method getNextRangeIndex. 3. Line 238: We've found a RowRange, check if startRowInclusive == false and set EXCLUSIVE = true. This variable indicates if next row should be excluded. 4. Line 105: Check if EXCLUSIVE == true, if so skip this row. The problem: we've skipped first row we got in this range, but we never checked if this row is a RowRange.startRow . In distributed system may not get RowRange.startRow on current instance, so we may exclude some another row. Moreover, we may not have RowRange.startRow at all in the DB, we will exclude some rows that are (possible) close to RowRange.startRow, but not equals to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Jindal updated HBASE-13693:
Attachment: HBASE-13693-hbase-11339.patch
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693-hbase-11339.patch, HBASE-13693.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Jindal updated HBASE-13693:
Attachment: (was: HBASE-13693.patch)
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693-hbase-11339.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548160#comment-14548160
]
Anoop Sam John commented on HBASE-13693:
Any chance for a test case?
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693-hbase-11339.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13703) ReplicateContext should not be a member of ReplicationSource
[ https://issues.apache.org/jira/browse/HBASE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548152#comment-14548152 ] stack commented on HBASE-13703: --- +1 ReplicateContext should not be a member of ReplicationSource Key: HBASE-13703 URL: https://issues.apache.org/jira/browse/HBASE-13703 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 13703.txt The ReplicateContext object is created once per ReplicationSource and then reused when we have something to ship to the sinks. This is a misguided optimization. ReplicateContext is very lightweight (definitely compared to the all the work and copying the ReplicationSource is doing) and, crucially, it prevent the the entries array from being collected after it was successfully copied to the sink, wasting potentially a lot of heap. The entries array itself holds reference to WAL entries on the heap, that now also cannot be collected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Malaska updated HBASE-13677:
Attachment: HBASE-13677.2.patch
Made changed requested by Sean B
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 1.1.1
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Jindal updated HBASE-13693:
Status: Patch Available (was: Open)
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Jindal updated HBASE-13693:
Attachment: HBASE-13693.patch
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13675) ProcedureExecutor completion report should be at DEBUG log level
[
https://issues.apache.org/jira/browse/HBASE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547977#comment-14547977
]
Matteo Bertozzi commented on HBASE-13675:
-
+1 on the patch and on adding the elapsedTime() log under isDebugEnabled().
ProcedureExecutor completion report should be at DEBUG log level
Key: HBASE-13675
URL: https://issues.apache.org/jira/browse/HBASE-13675
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-13675.patch
Example:
{noformat}
2015-05-12 12:05:03,445 INFO [ProcedureExecutorThread-0]
procedure2.ProcedureExecutor: Procedure completed in 838msec:
EnableTableProcedure (table=cluster_test) user=apurtell (auth:SIMPLE) id=5
state=FINISHED
{noformat}
The procedures themselves are already emitting task specific information at
INFO level. This doesn't add much information besides the elapsed time, so
probably should be at DEBUG level. I assume if a procedure fails there will
be additional logging at WARN level, but if not, that could be a
consideration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547976#comment-14547976
]
Ashutosh Jindal commented on HBASE-13693:
-
Attached a simple patch. Please review
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13704) Hbase throws OutOfOrderScannerNextException exception when MultiRowRangeFilter is used.
[
https://issues.apache.org/jira/browse/HBASE-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547931#comment-14547931
]
ramkrishna.s.vasudevan commented on HBASE-13704:
[~jiajia]
Another one to check out for.
Hbase throws OutOfOrderScannerNextException exception when
MultiRowRangeFilter is used.
---
Key: HBASE-13704
URL: https://issues.apache.org/jira/browse/HBASE-13704
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 1.1.0
Reporter: Aleksandr Maksymenko
When using filter MultiRowRangeFilter with ranges closed to each other that
there are no rows between ranges, then OutOfOrderScannerNextException is
throwed.
In filterRowKey method when range is switched to the next range,
currentReturnCode is set to SEEK_NEXT_USING_HINT (MultiRowRangeFilter: 118 in
v1.1.0). But if new range is already contain this row, then we should include
this row, not to seek for another one.
Replacing line 118 to this code seems to be working fine:
{code}
if (range.contains(buffer, offset, length)) {
currentReturnCode = ReturnCode.INCLUDE;
} else {
currentReturnCode = ReturnCode.SEEK_NEXT_USING_HINT;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547984#comment-14547984
]
Hadoop QA commented on HBASE-13693:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733542/HBASE-13693.patch
against master branch at commit f49111e5f8a2db8f3065188f03c7ad6d4411bd10.
ATTACHMENT ID: 12733542
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14075//console
This message is automatically generated.
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13705) MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false
[ https://issues.apache.org/jira/browse/HBASE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547921#comment-14547921 ] ramkrishna.s.vasudevan commented on HBASE-13705: So you mean to say check for greater than equal to startRow rather than saying strictly greater than startRow, am I correct? MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false --- Key: HBASE-13705 URL: https://issues.apache.org/jira/browse/HBASE-13705 Project: HBase Issue Type: Bug Affects Versions: 1.1.0 Reporter: Aleksandr Maksymenko I've found the issue during code review, so I don't have tests and I even didn't test this case manualy. So I'll try to describe it in words. Pre-condition: we're using scan with MultiRowRangeFilter with some RowRange's with startRowInclusive = false. This means that we want to include all rows that are strictly greater than startRow (and less then stopRow, but it doesn't matter for now). What happens in MultiRowRangeFilter.filterRowKey (worth case is described): 1. Line 91: Check if current range contains a row. Lets follow the case when it doesn't. 2. Line 94: Search for the next RowRange in method getNextRangeIndex. 3. Line 238: We've found a RowRange, check if startRowInclusive == false and set EXCLUSIVE = true. This variable indicates if next row should be excluded. 4. Line 105: Check if EXCLUSIVE == true, if so skip this row. The problem: we've skipped first row we got in this range, but we never checked if this row is a RowRange.startRow . In distributed system may not get RowRange.startRow on current instance, so we may exclude some another row. Moreover, we may not have RowRange.startRow at all in the DB, we will exclude some rows that are (possible) close to RowRange.startRow, but not equals to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13705) MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false
[
https://issues.apache.org/jira/browse/HBASE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547943#comment-14547943
]
Aleksandr Maksymenko commented on HBASE-13705:
--
After one more quick look at code, I see another issue. This time it's
connected to stopRow.
Lets say we've RowRange with startRowInclusive = true and stopRowInclusive =
false. It's a pretty common use case.
The issue may appear if we have only one record with row that is exactly the
same as stopRow. In method filterRowKey we look for a row range by calling
getNextRangeIndex, the row range described above should be found. Despite
RowRange is found, actual row should not be included. But it will, as I can see
in code.
It seems to easies solution is to fix getNextRangeIndex method by replacing:
{code}
// the row key equals one of the start keys, and the the range exclude the
start key
if(rangeList.get(index).startRowInclusive == false) {
EXCLUSIVE = true;
}
{code}
by something like this (attention, I didn't ceck if it's even compiled):
{code}
if(!rangeList.get(index).contains(rowKey)) {
EXCLUSIVE = true;
}
{code}
It seems that it should resolve both issues, but let's someone else to check it
(and test if possible).
MultiRowRangeFilter seems to be working incorrect if
RowRange.startRowInclusive = false
---
Key: HBASE-13705
URL: https://issues.apache.org/jira/browse/HBASE-13705
Project: HBase
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Aleksandr Maksymenko
I've found the issue during code review, so I don't have tests and I even
didn't test this case manualy. So I'll try to describe it in words.
Pre-condition: we're using scan with MultiRowRangeFilter with some RowRange's
with startRowInclusive = false. This means that we want to include all rows
that are strictly greater than startRow (and less then stopRow, but it
doesn't matter for now).
What happens in MultiRowRangeFilter.filterRowKey (worth case is described):
1. Line 91: Check if current range contains a row. Lets follow the case when
it doesn't.
2. Line 94: Search for the next RowRange in method getNextRangeIndex.
3. Line 238: We've found a RowRange, check if startRowInclusive == false and
set EXCLUSIVE = true. This variable indicates if next row should be excluded.
4. Line 105: Check if EXCLUSIVE == true, if so skip this row.
The problem: we've skipped first row we got in this range, but we never
checked if this row is a RowRange.startRow . In distributed system may not
get RowRange.startRow on current instance, so we may exclude some another
row. Moreover, we may not have RowRange.startRow at all in the DB, we will
exclude some rows that are (possible) close to RowRange.startRow, but not
equals to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13705) MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false
[ https://issues.apache.org/jira/browse/HBASE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547936#comment-14547936 ] ramkrishna.s.vasudevan commented on HBASE-13705: I think I get you now. You mean that though the rowKey is greater than the start row, we assume that to be the startRow and we exclude it where as we should have included it as the rowKey is than the startRow. MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false --- Key: HBASE-13705 URL: https://issues.apache.org/jira/browse/HBASE-13705 Project: HBase Issue Type: Bug Affects Versions: 1.1.0 Reporter: Aleksandr Maksymenko I've found the issue during code review, so I don't have tests and I even didn't test this case manualy. So I'll try to describe it in words. Pre-condition: we're using scan with MultiRowRangeFilter with some RowRange's with startRowInclusive = false. This means that we want to include all rows that are strictly greater than startRow (and less then stopRow, but it doesn't matter for now). What happens in MultiRowRangeFilter.filterRowKey (worth case is described): 1. Line 91: Check if current range contains a row. Lets follow the case when it doesn't. 2. Line 94: Search for the next RowRange in method getNextRangeIndex. 3. Line 238: We've found a RowRange, check if startRowInclusive == false and set EXCLUSIVE = true. This variable indicates if next row should be excluded. 4. Line 105: Check if EXCLUSIVE == true, if so skip this row. The problem: we've skipped first row we got in this range, but we never checked if this row is a RowRange.startRow . In distributed system may not get RowRange.startRow on current instance, so we may exclude some another row. Moreover, we may not have RowRange.startRow at all in the DB, we will exclude some rows that are (possible) close to RowRange.startRow, but not equals to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-13686) Fail to limit rate in RateLimiter
[
https://issues.apache.org/jira/browse/HBASE-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi reassigned HBASE-13686:
-
Assignee: Ashish Singhi
Fail to limit rate in RateLimiter
-
Key: HBASE-13686
URL: https://issues.apache.org/jira/browse/HBASE-13686
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 1.1.0
Reporter: Guanghao Zhang
Assignee: Ashish Singhi
Priority: Minor
While using the patch in HBASE-11598 , I found that RateLimiter can't to
limit the rate right.
{code}
/**
* given the time interval, are there enough available resources to allow
execution?
* @param now the current timestamp
* @param lastTs the timestamp of the last update
* @param amount the number of required resources
* @return true if there are enough available resources, otherwise false
*/
public synchronized boolean canExecute(final long now, final long lastTs,
final long amount) {
return avail = amount ? true : refill(now, lastTs) = amount;
}
{code}
When avail = amount, avail can't be refill. But in the next time to call
canExecute, lastTs maybe update. So avail will waste some time to refill.
Even we use smaller rate than the limit, the canExecute will return false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: (was: HBASE-13071_trunk_2.patch) Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071_trunk_5.patch, HBASE-13071_trunk_6.patch, HBASE-13071_trunk_7.patch, HBASE-13071_trunk_8.patch, HBASE-13071_trunk_9.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: (was: HBASE-13071_trunk_4.patch) Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071_trunk_5.patch, HBASE-13071_trunk_6.patch, HBASE-13071_trunk_7.patch, HBASE-13071_trunk_8.patch, HBASE-13071_trunk_9.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: (was: HBASE-13071_trunk_1.patch) Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071_trunk_5.patch, HBASE-13071_trunk_6.patch, HBASE-13071_trunk_7.patch, HBASE-13071_trunk_8.patch, HBASE-13071_trunk_9.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: (was: HBASE-13071_trunk_3.patch) Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071_trunk_5.patch, HBASE-13071_trunk_6.patch, HBASE-13071_trunk_7.patch, HBASE-13071_trunk_8.patch, HBASE-13071_trunk_9.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: (was: HBASE-13071_98_1.patch) Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071_trunk_1.patch, HBASE-13071_trunk_10.patch, HBASE-13071_trunk_2.patch, HBASE-13071_trunk_3.patch, HBASE-13071_trunk_4.patch, HBASE-13071_trunk_5.patch, HBASE-13071_trunk_6.patch, HBASE-13071_trunk_7.patch, HBASE-13071_trunk_8.patch, HBASE-13071_trunk_9.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Attachment: HBASE-13158_V2.patch When client supports CellBlock, return the result Cells as controller payload for get(Get) API also --- Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13158.patch, HBASE-13158_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Fix Version/s: 1.2.0 When client supports CellBlock, return the result Cells as controller payload for get(Get) API also --- Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13158.patch, HBASE-13158_V2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: HBASE-13071-0_98.patch Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Assignee: Eshcar Hillel Fix For: 2.0.0 Attachments: 99.eshcar.png, HBASE-13071-0_98.patch, HBASE-13071-BRANCH-1.patch, HBASE-13071-trunk-bug-fix.patch, HBASE-13071_trunk_rebase_1.0.patch, HBASE-13071_trunk_rebase_2.0.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, HbaseStreamingScanEvaluationwithMultipleClients.pdf, Releasenote-13071.txt, gc.delay.png, gc.eshcar.png, gc.png, hits.delay.png, hits.eshcar.png, hits.png, latency.delay.png, latency.png, network.png A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13510) Purge ByteBloomFilter
[
https://issues.apache.org/jira/browse/HBASE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547663#comment-14547663
]
Hadoop QA commented on HBASE-13510:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733444/HBASE-13510_6.patch
against master branch at commit f49111e5f8a2db8f3065188f03c7ad6d4411bd10.
ATTACHMENT ID: 12733444
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.activemq.transport.mqtt.MQTTTest.testPacketIdGeneratorCleanSession(MQTTTest.java:913)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14073//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14073//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14073//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14073//console
This message is automatically generated.
Purge ByteBloomFilter
-
Key: HBASE-13510
URL: https://issues.apache.org/jira/browse/HBASE-13510
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0
Attachments: HBASE-13510_1.patch, HBASE-13510_2.patch,
HBASE-13510_3.patch, HBASE-13510_5.patch, HBASE-13510_6.patch
In order to address the comments over in HBASE-10800 related to comparing
Cell with a serialized KV's key we had some need for that in Bloom filters.
After discussing with Anoop, we found that it may be possible to
remove/modify some of the APIs in the BloomFilter interfaces and for doing
that we can purge ByteBloomFilter.
I read the code and found that ByteBloomFilter was getting used in V1 version
only. Now as it is obsolete we can remove this code and move some of the
static APIs in ByteBloomFilter to some other util class or bloom related
classes which will help us in refactoring the code too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13693) [HBase MOB] Mob files are not encrypting.
[
https://issues.apache.org/jira/browse/HBASE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548244#comment-14548244
]
Hadoop QA commented on HBASE-13693:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733546/HBASE-13693-hbase-11339.patch
against hbase-11339 branch at commit f49111e5f8a2db8f3065188f03c7ad6d4411bd10.
ATTACHMENT ID: 12733546
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14076//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14076//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14076//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14076//console
This message is automatically generated.
[HBase MOB] Mob files are not encrypting.
-
Key: HBASE-13693
URL: https://issues.apache.org/jira/browse/HBASE-13693
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: hbase-11339
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashutosh Jindal
Fix For: hbase-11339
Attachments: HBASE-13693-hbase-11339.patch
Mob HFiles are not encrypting.
steps to reproduce:
===
1.create a table and for column family with mob enabled and enable AES
encryption for the column family.
2. Insert mob data into the table.
3. Flush the mob table.
4. check hfiles for mob data is created or not.
5. check hfiles in hdfs is encrypted or not using hfile tool.
{code}
hfile tool output for mob reference hfile meta
Block index size as per heapsize: 392
reader=/hbase/data/default/mobTest/1587e00c3e257969c48d9872994ce57c/mobcf/8c33ab9e8201449e9ac709eb9e4263d6,
Trailer:
fileinfoOffset=527,
loadOnOpenDataOffset=353,
dataIndexCount=1,
metaIndexCount=0,
totalUncomressedBytes=5941,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=34,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=PRESENT,
majorVersion=3,
minorVersion=0
{code}
{code}
hfile tool output for mob hfile meta
Block index size as per heapsize: 872
reader=/hbase/mobdir/data/default/mobTest/46844d8b9f699e175a4d7bd57848c576/mobcf/d41d8cd98f00b204e9800998ecf8427e20150512bf18fa62a98c40d7bd6e810f790c6291,
Trailer:
fileinfoOffset=1018180,
loadOnOpenDataOffset=1017959,
dataIndexCount=9,
metaIndexCount=0,
totalUncomressedBytes=1552619,
entryCount=9,
compressionCodec=GZ,
uncompressedDataIndexSize=266,
numDataIndexLevels=1,
firstDataBlockOffset=0,
lastDataBlockOffset=904852,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
encryptionKey=NONE,
majorVersion=3,
minorVersion=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13694) CallQueueSize is incorrectly decremented until the response is sent
[
https://issues.apache.org/jira/browse/HBASE-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548296#comment-14548296
]
Esteban Gutierrez commented on HBASE-13694:
---
[~stack]: yeah I found a place where that can happen, I have a new patch were
basically we decrement callQueueSize immediately if the call is successful but
we left the decrement until the finally clause if there was an exception.
[~lhofhansl]: Not in 0.94 since we decrement callQueueSize after the call is
made but before sending the response:
{code}
CurCall.set(null);
callQueueSize.add(call.getSize() * -1);
// Set the response for undelayed calls and delayed calls with
// undelayed responses.
if (!call.isDelayed() || !call.isReturnValueDelayed()) {
call.setResponse(value,
errorClass == null? Status.SUCCESS: Status.ERROR,
errorClass, error);
}
call.sendResponseIfReady();
status.markComplete(Sent response);
{code}
However that still have the old issue that any exception before decrementing
the queue size will be miscount.
CallQueueSize is incorrectly decremented until the response is sent
---
Key: HBASE-13694
URL: https://issues.apache.org/jira/browse/HBASE-13694
Project: HBase
Issue Type: Bug
Components: master, regionserver, rpc
Affects Versions: 2.0.0, 1.1.0, 0.98.12, 1.0.2, 1.2.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Attachments:
0001-HBASE-13694-CallQueueSize-is-incorrectly-decremented.patch
We should decrement the CallQueueSize as soon as we no longer need the call
around, e.g. after {{RpcServer.CurCall.set(null)}} otherwise we will be only
pushing back other client requests while we send the response back to the
client that originated the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13694) CallQueueSize is incorrectly decremented until the response is sent
[
https://issues.apache.org/jira/browse/HBASE-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-13694:
--
Affects Version/s: 0.98.12
CallQueueSize is incorrectly decremented until the response is sent
---
Key: HBASE-13694
URL: https://issues.apache.org/jira/browse/HBASE-13694
Project: HBase
Issue Type: Bug
Components: master, regionserver, rpc
Affects Versions: 2.0.0, 1.1.0, 0.98.12, 1.0.2, 1.2.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Attachments:
0001-HBASE-13694-CallQueueSize-is-incorrectly-decremented.patch
We should decrement the CallQueueSize as soon as we no longer need the call
around, e.g. after {{RpcServer.CurCall.set(null)}} otherwise we will be only
pushing back other client requests while we send the response back to the
client that originated the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
[ https://issues.apache.org/jira/browse/HBASE-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548353#comment-14548353 ] Jerry He commented on HBASE-13706: -- We have a case where hive classes are used in the coprocessor jar. coprocessor would fail to load unless we explore other approaches. e.g copy the Hive jars into HBase lib or shade the Hive classes. CoprocessorClassLoader should not exempt Hive classes - Key: HBASE-13706 URL: https://issues.apache.org/jira/browse/HBASE-13706 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Reporter: Jerry He Priority: Minor CoprocessorClassLoader is used to load classes from the coprocessor jar. Certain classes are exempt from being loaded by this ClassLoader, which means they will be ignored in the coprocessor jar, but loaded from parent classpath instead. One problem is that we categorically exempt org.apache.hadoop. But it happens that Hive packages start with org.apache.hadoop. There is no reason to exclude hive classes from theCoprocessorClassLoader. HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548312#comment-14548312
]
Stephen Yuan Jiang commented on HBASE-13609:
+1
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
Jerry He created HBASE-13706: Summary: CoprocessorClassLoader should not exempt Hive classes Key: HBASE-13706 URL: https://issues.apache.org/jira/browse/HBASE-13706 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.12, 1.0.1, 2.0.0, 1.1.0 Reporter: Jerry He Priority: Minor CoprocessorClassLoader is used to load classes from the coprocessor jar. Certain classes are exempt from being loaded by this ClassLoader, which means they will be ignored in the coprocessor jar, but loaded from parent classpath instead. One problem is that we categorically exempt org.apache.hadoop. But it happens that Hive packages start with org.apache.hadoop. There is no reason to exclude hive classes from theCoprocessorClassLoader. HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13140) Version documentation to match software
[ https://issues.apache.org/jira/browse/HBASE-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548388#comment-14548388 ] Sean Busbey commented on HBASE-13140: - also need 1.1 docs now. Version documentation to match software --- Key: HBASE-13140 URL: https://issues.apache.org/jira/browse/HBASE-13140 Project: HBase Issue Type: Improvement Components: documentation, site Reporter: stack Its probably about time that we add in a 0.98 and 1.0 version of documentation. Currently, the doc shows as 2.0.0-SNAPSHOT which is a little disorientating. A user on mailing list just had trouble because doc talked of configs that are in hbase 1.0 but he was on 0.98. We have a manually made 0.94 version. It is time to do the work to make it so we can gen doc by version for the site and it is also similarly time to let doc versions diverge. (I think we've only talked of doing this in past but have not as yet filed issue... close if I have this wrong) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548216#comment-14548216
]
Ted Malaska commented on HBASE-13677:
-
Thanks guys.
I'll try to do something a little harder next time.
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-13677.
-
Resolution: Fixed
Fix Version/s: (was: 1.1.1)
1.2.0
0.98.13
2.0.0
+1. pushed to 0.98, branch-1, and master.
Thanks Ted! you should update your reviewboard posting now to close it as
submitted.
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548259#comment-14548259
]
Nick Dimiduk commented on HBASE-13677:
--
Thanks for grabbing this one [~ted.m].
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548442#comment-14548442
]
Hudson commented on HBASE-13677:
FAILURE: Integrated in HBase-TRUNK #6489 (See
[https://builds.apache.org/job/HBase-TRUNK/6489/])
HBASE-13677 changed RecoverableZooKeeper logs about retries from info and warn
to debug (busbey: rev a1c94c0bdfe31d2d2435a135fc2ff1d36a979e13)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539665#comment-14539665 ] Anoop Sam John edited comment on HBASE-13448 at 5/18/15 6:51 PM: - Table with CF and testing with 3 rows and 3 cells in each row Scan with out ExplicitColumnTracker and no stop row specified |Method|First cell in first row|First cell in other rows|Other cells| |getRowOffset|3|6|1| |getRowLength|6|9|4| |getFamilyOffset|2|2|2| |getFamilyLength|1|1|1| |getQualifierOffset|1|1|1| |getQualifierLength|1|1|1| |getTimestamp|1|1|1| |getTypeByte|2|2|2| |getValueLength|2|2|2| |#getKeyLength|6|6|6| Scan with ExplicitColumnTracker |Method|First cell in first row|First cell in other rows|Other cells| |getRowOffset|10|9|4| |getRowLength|20|19|15| |getFamilyOffset|8|8|8| |getFamilyLength|4|4|5| |getQualifierOffset|2|2|2| |getQualifierLength|2|2|3| |getTimestamp|1|1|2| |getTypeByte|2|2|2| |getValueLength|2|2|2| |getKeyLength|7|7|9| Similarly there are so many calls on getXXXOffset/Length on fake cells. (Like the oldest/latest cells on a row etc) was (Author: anoop.hbase): Table with CF and testing with 3 rows and 3 cells in each row Scan with out ExplicitColumnTracker and no stop row specified |Method|First cell in first row|First cell in other rows|Other cells| |getRowOffset|3|6|1| |getRowLength|6|9|4| |getFamilyOffset|2|2|2| |getFamilyLength|1|1|1| |getQualifierOffset|1|1|1| |getQualifierLength|1|1|1| |getTimestamp|1|1|1| |getTypeByte|2|2|2| |getValueLength|4|4|4| |#getKeyLength|6|6|6| Scan with ExplicitColumnTracker |Method|First cell in first row|First cell in other rows|Other cells| |getRowOffset|10|9|4| |getRowLength|20|19|15| |getFamilyOffset|8|8|8| |getFamilyLength|4|4|5| |getQualifierOffset|2|2|2| |getQualifierLength|2|2|3| |getTimestamp|1|1|2| |getTypeByte|2|2|2| |getValueLength|2|2|2| |getKeyLength|7|7|9| Similarly there are so many calls on getXXXOffset/Length on fake cells. (Like the oldest/latest cells on a row etc) New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13448.patch, HBASE-13448_V2.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548489#comment-14548489
]
Nick Dimiduk commented on HBASE-13609:
--
These maybe help? When I run this in a loop, I now get failures like this,
though only 1/20 or so. Could be that my machine is too idle to represent
jenkins env though.
{noformat}
Running org.apache.hadoop.hbase.client.TestFastFail
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 19.751 sec
FAILURE! - in org.apache.hadoop.hbase.client.TestFastFail
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed: 12.647
sec FAILURE!
java.lang.AssertionError: There should be atleast one thread that retried
instead of failing
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:271)
{noformat}
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548441#comment-14548441
]
Hudson commented on HBASE-13677:
SUCCESS: Integrated in HBase-1.2 #83 (See
[https://builds.apache.org/job/HBase-1.2/83/])
HBASE-13677 changed RecoverableZooKeeper logs about retries from info and warn
to debug (busbey: rev 00636cc998ff4e935977fa334db152bd7e7dd4f5)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13140) Version documentation to match software
[
https://issues.apache.org/jira/browse/HBASE-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548509#comment-14548509
]
Sean Busbey commented on HBASE-13140:
-
Great point on workaround from [~ndimiduk]:
{quote}
Nick Dimiduk via hbase.apache.org 1:34 PM (22 minutes ago) to hbase-user
You don't need to build from the src tgz, the bin tgz contains a docs
directory, wherein you'll find both public-facing (@Public annotated
classes) and full javadocs in apidocs and devapidocs respectively. The
whole site and book are there too, but our release policy is to copy site
and book from master. The javadocs are generated with this build of this
release though.
{quote}
Version documentation to match software
---
Key: HBASE-13140
URL: https://issues.apache.org/jira/browse/HBASE-13140
Project: HBase
Issue Type: Improvement
Components: documentation, site
Reporter: stack
Its probably about time that we add in a 0.98 and 1.0 version of
documentation. Currently, the doc shows as 2.0.0-SNAPSHOT which is a little
disorientating. A user on mailing list just had trouble because doc talked of
configs that are in hbase 1.0 but he was on 0.98.
We have a manually made 0.94 version. It is time to do the work to make it so
we can gen doc by version for the site and it is also similarly time to let
doc versions diverge.
(I think we've only talked of doing this in past but have not as yet filed
issue... close if I have this wrong)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13448: --- Attachment: HBASE-13448_V3.patch getRowLength() and getKeyLength() are getting called and data getting parsed more times. Caching these 2 items only. Any others come in later in profiling, we can add then. This can go in as a 1st step (?) New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13609:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to branch-1.0+. Didn't find a TestFastFail on 0.98. If this doesn't
help, I'll disable the test completely.
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549107#comment-14549107
]
Hudson commented on HBASE-13677:
FAILURE: Integrated in HBase-0.98 #994 (See
[https://builds.apache.org/job/HBase-0.98/994/])
HBASE-13677 changed RecoverableZooKeeper logs about retries from info and warn
to debug (busbey: rev 3352585160b6ff10069cb587ba1b9ddb1aedea24)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549119#comment-14549119 ] stack commented on HBASE-13448: --- +1 on v3 patch. Change name of class to SizeCachedKeyValue on commit. Maybe add to the class comment a pointer to here so [~lhofhansl] doesn't remove it (again). Fix this sentence on commit too: * Note: Please do not use these objects in write path and it will increase the heap space usage. New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13707) CellCounter uses to many counters
[ https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-13707: Labels: beginner newbie (was: newbie) CellCounter uses to many counters - Key: HBASE-13707 URL: https://issues.apache.org/jira/browse/HBASE-13707 Project: HBase Issue Type: Bug Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: beginner, newbie CellCounters creates a counter per row... So it quickly becomes to many. We should provide an option to drop the statistic per rows and count only cells overall for the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13673) WALProcedureStore procedure is chatty
[
https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Srungarapu updated HBASE-13673:
Attachment: HBASE-13673_v2.patch
Addressing feedback along with using {{LOG.isDebugEnabled()}} while using debug
level.
WALProcedureStore procedure is chatty
-
Key: HBASE-13673
URL: https://issues.apache.org/jira/browse/HBASE-13673
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-13673.patch, HBASE-13673_v2.patch
Lots of procedure logging at INFO level which should be at DEBUG, especially
Remove all state logs with ID less then 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13694) CallQueueSize is incorrectly decremented until the response is sent
[
https://issues.apache.org/jira/browse/HBASE-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-13694:
--
Attachment: HBASE-13694.2.patch
CallQueueSize is incorrectly decremented until the response is sent
---
Key: HBASE-13694
URL: https://issues.apache.org/jira/browse/HBASE-13694
Project: HBase
Issue Type: Bug
Components: master, regionserver, rpc
Affects Versions: 2.0.0, 1.1.0, 0.98.12, 1.0.2, 1.2.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Attachments:
0001-HBASE-13694-CallQueueSize-is-incorrectly-decremented.patch,
HBASE-13694.2.patch
We should decrement the CallQueueSize as soon as we no longer need the call
around, e.g. after {{RpcServer.CurCall.set(null)}} otherwise we will be only
pushing back other client requests while we send the response back to the
client that originated the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13705) MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false
[
https://issues.apache.org/jira/browse/HBASE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549358#comment-14549358
]
Ted Yu commented on HBASE-13705:
For the first issue, RowRange has this method:
{code}
public int compareTo(RowRange other) {
return Bytes.compareTo(this.startRow, other.startRow);
{code}
When Collections.binarySearch(rangeList, temp) finds the RowRange, we know
rowKey matches the start row of the RowRange.
Maybe I missed something ...
MultiRowRangeFilter seems to be working incorrect if
RowRange.startRowInclusive = false
---
Key: HBASE-13705
URL: https://issues.apache.org/jira/browse/HBASE-13705
Project: HBase
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Aleksandr Maksymenko
I've found the issue during code review, so I don't have tests and I even
didn't test this case manualy. So I'll try to describe it in words.
Pre-condition: we're using scan with MultiRowRangeFilter with some RowRange's
with startRowInclusive = false. This means that we want to include all rows
that are strictly greater than startRow (and less then stopRow, but it
doesn't matter for now).
What happens in MultiRowRangeFilter.filterRowKey (worth case is described):
1. Line 91: Check if current range contains a row. Lets follow the case when
it doesn't.
2. Line 94: Search for the next RowRange in method getNextRangeIndex.
3. Line 238: We've found a RowRange, check if startRowInclusive == false and
set EXCLUSIVE = true. This variable indicates if next row should be excluded.
4. Line 105: Check if EXCLUSIVE == true, if so skip this row.
The problem: we've skipped first row we got in this range, but we never
checked if this row is a RowRange.startRow . In distributed system may not
get RowRange.startRow on current instance, so we may exclude some another
row. Moreover, we may not have RowRange.startRow at all in the DB, we will
exclude some rows that are (possible) close to RowRange.startRow, but not
equals to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13675) ProcedureExecutor completion report should be at DEBUG log level
[
https://issues.apache.org/jira/browse/HBASE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Srungarapu updated HBASE-13675:
Attachment: HBASE-13675_v2.patch
ProcedureExecutor completion report should be at DEBUG log level
Key: HBASE-13675
URL: https://issues.apache.org/jira/browse/HBASE-13675
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-13675.patch, HBASE-13675_v2.patch
Example:
{noformat}
2015-05-12 12:05:03,445 INFO [ProcedureExecutorThread-0]
procedure2.ProcedureExecutor: Procedure completed in 838msec:
EnableTableProcedure (table=cluster_test) user=apurtell (auth:SIMPLE) id=5
state=FINISHED
{noformat}
The procedures themselves are already emitting task specific information at
INFO level. This doesn't add much information besides the elapsed time, so
probably should be at DEBUG level. I assume if a procedure fails there will
be additional logging at WARN level, but if not, that could be a
consideration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549386#comment-14549386 ] stack commented on HBASE-13448: --- [~lhofhansl] We are going this route to save CPU -- the length parses show in macro profiling as costly -- and to simplify code: all over our codebase we are doing carry-overs, passing a parsed length found in one method as input on other methods as we walk through the elements of a Cell/KeyValue. These latter won't always make sense as Cell implementations are different and rather than do them arbitrarily around the code base, rather let the Cell do the length caching. See https://issues.apache.org/jira/browse/HBASE-13448?focusedCommentId=14500517page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14500517 New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13707) CellCounter uses to many counters
[
https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549211#comment-14549211
]
Jean-Marc Spaggiari edited comment on HBASE-13707 at 5/18/15 8:56 PM:
--
Exactly. Very good for very wide tables with only few rows. You can still
increase the maximum number of counters but it become crazy quickly.
{code}
context.getCounter(QL_VERSIONS, currentRowKey + separator +
thisRowQualifierName).increment(1);
{code}
It creates one counter per row using the rowkey as the name to give you how
many cells and versions there is per row. It's still useful in some cases, but
generally speaking it creates to many counters...
I think we should simply add an option to not count per row, but only per
table, so just add a parameter and an if before this line.
was (Author: jmspaggi):
Exactly. Very good for very wide tables with only few rows. You can still
increase the maximum number of counters but it become crazy quickly.
[code]
context.getCounter(QL_VERSIONS, currentRowKey + separator +
thisRowQualifierName).increment(1);
[code]
It creates one counter per row using the rowkey as the name to give you how
many cells and versions there is per row. It's still useful in some cases, but
generally speaking it creates to many counters...
I think we should simply add an option to not count per row, but only per
table, so just add a parameter and an if before this line.
CellCounter uses to many counters
-
Key: HBASE-13707
URL: https://issues.apache.org/jira/browse/HBASE-13707
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.0.1
Reporter: Jean-Marc Spaggiari
Priority: Minor
Labels: beginner, newbie
CellCounters creates a counter per row... So it quickly becomes to many.
We should provide an option to drop the statistic per rows and count only
cells overall for the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13707) CellCounter uses to many counters
[ https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549211#comment-14549211 ] Jean-Marc Spaggiari commented on HBASE-13707: - Exactly. Very good for very wide tables with only few rows. You can still increase the maximum number of counters but it become crazy quickly. [code] context.getCounter(QL_VERSIONS, currentRowKey + separator + thisRowQualifierName).increment(1); [code] It creates one counter per row using the rowkey as the name to give you how many cells and versions there is per row. It's still useful in some cases, but generally speaking it creates to many counters... I think we should simply add an option to not count per row, but only per table, so just add a parameter and an if before this line. CellCounter uses to many counters - Key: HBASE-13707 URL: https://issues.apache.org/jira/browse/HBASE-13707 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: beginner, newbie CellCounters creates a counter per row... So it quickly becomes to many. We should provide an option to drop the statistic per rows and count only cells overall for the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549382#comment-14549382
]
Hudson commented on HBASE-13609:
FAILURE: Integrated in HBase-1.1 #491 (See
[https://builds.apache.org/job/HBase-1.1/491/])
HBASE-13609 TestFastFail is still failing (ndimiduk: rev
25cbb2840d51dbd67a64c95e0d4953a8aca220b9)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13707) CellCounter uses to many counters
Jean-Marc Spaggiari created HBASE-13707: --- Summary: CellCounter uses to many counters Key: HBASE-13707 URL: https://issues.apache.org/jira/browse/HBASE-13707 Project: HBase Issue Type: Bug Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor CellCounters creates a counter per row... So it quickly becomes to many. We should provide an option to drop the statistic per rows and count only cells overall for the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549347#comment-14549347 ] Lars Hofhansl commented on HBASE-13448: --- Please gimme a day or so to look through this issue. Want to make sure we're not falling into the micro-benchmark trap. In the past i have been _removing_ things like this. It looked bad in a profiler, but with a sampler it turned out not to be useful, and just waste HEAP, be cache line unfriendly, etc, etc. 2-3% does not seem to worth it on first glance. Are these end-to-end perf numbers? New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13708) PE - Add option to specify the range
Jean-Marc Spaggiari created HBASE-13708: --- Summary: PE - Add option to specify the range Key: HBASE-13708 URL: https://issues.apache.org/jira/browse/HBASE-13708 Project: HBase Issue Type: Bug Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor When specifying --rows=YYY for randomReads in PE, it will read YYY rows but only betweem 0 and YYY. We might want to read YYY rows randomly between 0 and ZZZ. YYY should only be the number of rows we want to ready. Not the high range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549337#comment-14549337
]
Hudson commented on HBASE-13609:
FAILURE: Integrated in HBase-1.0 #919 (See
[https://builds.apache.org/job/HBase-1.0/919/])
HBASE-13609 TestFastFail is still failing (ndimiduk: rev
c09730d1c5b1f40dffec719a7ba750aaeee07569)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549410#comment-14549410
]
Hudson commented on HBASE-13609:
FAILURE: Integrated in HBase-TRUNK #6490 (See
[https://builds.apache.org/job/HBase-TRUNK/6490/])
HBASE-13609 TestFastFail is still failing (ndimiduk: rev
92e66ef5222391d106bc8adb7afe7d96a7dafac2)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549145#comment-14549145 ] Anoop Sam John commented on HBASE-13448: Thanks Stack. Will fix the comments on commit. New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13707) CellCounter uses to many counters
[ https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-13707: - Component/s: mapreduce CellCounter uses to many counters - Key: HBASE-13707 URL: https://issues.apache.org/jira/browse/HBASE-13707 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: beginner, newbie CellCounters creates a counter per row... So it quickly becomes to many. We should provide an option to drop the statistic per rows and count only cells overall for the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13707) CellCounter uses to many counters
[ https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549203#comment-14549203 ] Nick Dimiduk commented on HBASE-13707: -- Counter per _row_? That sounds like a bug, or someone assumed wide rows only. We're generating counter names dynamically [~jmspaggi]? How do you think we should handle this? Option to specify what we're counting maybe -- count rows or count specific column occurrences in rows or found unique column names? CellCounter uses to many counters - Key: HBASE-13707 URL: https://issues.apache.org/jira/browse/HBASE-13707 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: beginner, newbie CellCounters creates a counter per row... So it quickly becomes to many. We should provide an option to drop the statistic per rows and count only cells overall for the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13376) Improvements to Stochastic load balancer
[
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549554#comment-14549554
]
Ted Yu edited comment on HBASE-13376 at 5/19/15 12:23 AM:
--
{code}
771 private ComparatorInteger localityComparator = new
ComparatorInteger() {
772 @Override
773 public int compare(Integer integer, Integer integer2) {
{code}
Can the parameters for compare() have better names, such as server1, server2 ?
{code}
796 i++;
797 lowestLocalServerIndex = serverIndicesSortedByLocality[i];
{code}
Better check for boundary before assigning lowestLocalServerIndex.
Test failure in TestRegionRebalancing might be related to patch, however
https://builds.apache.org/job/PreCommit-HBASE-Build/13602//testReport/ is no
longer accessible.
was (Author: yuzhih...@gmail.com):
{code}
771 private ComparatorInteger localityComparator = new
ComparatorInteger() {
772 @Override
773 public int compare(Integer integer, Integer integer2) {
{code}
Can the parameters for compare() have better names, such as server1, server2 ?
{code}
794 while (localityPerServer[lowestLocalServerIndex] == 0
795 (regionsPerServer[lowestLocalServerIndex].length == 0)) {
{code}
The above assumes 0 is the lowest locality. What if locality for
serverIndicesSortedByLocality[0] is greater than 0 but the region count on the
server is 0 ?
{code}
796 i++;
797 lowestLocalServerIndex = serverIndicesSortedByLocality[i];
{code}
Better check for boundary before assigning lowestLocalServerIndex.
Test failure in TestRegionRebalancing might be related to patch, however
https://builds.apache.org/job/PreCommit-HBASE-Build/13602//testReport/ is no
longer accessible.
Improvements to Stochastic load balancer
Key: HBASE-13376
URL: https://issues.apache.org/jira/browse/HBASE-13376
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: HBASE-13376_0.98.txt, HBASE-13376_0.txt,
HBASE-13376_98.patch
There are two things this jira tries to address:
1. The locality picker in the stochastic balancer does not pick regions with
least locality as candidates for swap/move. So when any user configures
locality cost in the configs, the balancer does not always seems to move
regions with bad locality.
2. When a cluster has equal number of loaded regions, it always picks the
first one. It should pick a random region on one of the equally loaded
servers. This improves a chance of finding a good candidate, when load picker
is invoked several times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13675) ProcedureExecutor completion report should be at DEBUG log level
[
https://issues.apache.org/jira/browse/HBASE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549593#comment-14549593
]
Hadoop QA commented on HBASE-13675:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733641/HBASE-13675_v2.patch
against master branch at commit 92e66ef5222391d106bc8adb7afe7d96a7dafac2.
ATTACHMENT ID: 12733641
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14080//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14080//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14080//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14080//console
This message is automatically generated.
ProcedureExecutor completion report should be at DEBUG log level
Key: HBASE-13675
URL: https://issues.apache.org/jira/browse/HBASE-13675
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-13675.patch, HBASE-13675_v2.patch
Example:
{noformat}
2015-05-12 12:05:03,445 INFO [ProcedureExecutorThread-0]
procedure2.ProcedureExecutor: Procedure completed in 838msec:
EnableTableProcedure (table=cluster_test) user=apurtell (auth:SIMPLE) id=5
state=FINISHED
{noformat}
The procedures themselves are already emitting task specific information at
INFO level. This doesn't add much information besides the elapsed time, so
probably should be at DEBUG level. I assume if a procedure fails there will
be additional logging at WARN level, but if not, that could be a
consideration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13709) Updates to meta table server columns may be eclipsed
Enis Soztutar created HBASE-13709: - Summary: Updates to meta table server columns may be eclipsed Key: HBASE-13709 URL: https://issues.apache.org/jira/browse/HBASE-13709 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1 HBASE-11536 fixes a case where on a very rare occasion, the meta updates may be processed out of order. The fix is to use the RS's timestamp for the server column in meta update, but that actually opens up a vulnerability for clock skew (see the discussions in the jira). For the region replicas case, we can reproduce a problem where the server name field is eclipsed by the masters earlier update because the RS is lagging behind. However, this is not specific to replicas, but occurs more frequently with it. One option that was discussed was to send the master's ts with open region RPC and use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11536) Puts of region location to Meta may be out of order which causes inconsistent of region location
[
https://issues.apache.org/jira/browse/HBASE-11536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549628#comment-14549628
]
Enis Soztutar commented on HBASE-11536:
---
bq. I agree with your that relying on timestamps, while it would work
9.% of the time, because of the murphy's law, the one time
it fails, it'd be a high profile situation and we'd all have to go home (smile).
Murphy loves me. We can reproduce a failure because of this on EC2 clusters a
lot more frequently than what you quote. This happens more for region replicas,
because the master writes null entries for server columns for replicas, and
then assigns those. When the replicas got assigned, if the RS's timestamp is
lagging behind the master by more than the amount of time the assignment took
place (a couple of seconds), then the server location becomes null for the
replica.
Anyway opened HBASE-13709 detailing the issue. Please take a look.
Puts of region location to Meta may be out of order which causes inconsistent
of region location
Key: HBASE-11536
URL: https://issues.apache.org/jira/browse/HBASE-11536
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Critical
Fix For: 0.99.0, 2.0.0, 0.94.23, 0.98.6
Attachments: 10.237.12.13.log, 10.237.12.15.log, 11536-trunk.txt,
HBASE-11536-0.94-v1.diff
In product hbase cluster, we found inconsistency of region location in the
meta table. Region cdfa2ed711bbdf054d9733a92fd43eb5 is onlined in
regionserver 10.237.12.13:11600 but the region location in Meta table is
10.237.12.15:11600.
This is because of the out-of-order puts for meta table.
# HMaster try to assign the region to 10.237.12.15:11600.
# RegionServer: 10.237.12.15:11600. During the opening the region, the put of
region location(10.237.12.15:11600) to meta table is timeout(60s) and the
htable retry for second time. (regionserver serving meta has got the request
of the put. The timeout is beause ther is a bad disk in this regionserver
and sync of hlog is very slow.
)
During the retry in htable, the OpenRegionHandler is timeout(100s) and the
PostOpenDeployTasksThread is interrupted. Through the htable is closed in the
MetaEditor finally, the share connection the htable used is not closed and
the call of put for meta table is on-flying in the connection. Assumed that
this on-flying call of put to meta is named call A.
# RegionServer: 10.237.12.15:11600. For the timeout of OpenRegionHandler, the
OpenRegionHandler marks the assign state of this region to FAILED_OPEN.
# HMaster watchs this event of FAILED_OPEN and assigns the region to another
regionserver: 10.237.12.13:11600
# RegionServer: 10.237.12.13:11600. This regionserver opens the region
successfully . Assumed that the put of region location(10.237.12.13:11600) to
meta table in this regionserver is named B.
There is no order guarantee for call A and B. If call A is processed after
call B in regionserver serving meta region, the region location in meta table
will be wrong.
From the raw scan of meta table we found:
{code}
scan '.META.', {RAW = true, LIMIT = 1, VERSIONS = 10, STARTROW =
'xxx.adfa2ed711bbdf054d9733a92fd43eb5.'}
{code}
{quote}
xxx.adfa2ed711bbdf054d9733a92fd43eb5. column=info:server,
timestamp=1404885460553(= Wed Jul 09 13:57:40 +0800 2014),
value=10.237.12.15:11600 -- Retry put from 10.237.12.15
xxx.adfa2ed711bbdf054d9733a92fd43eb5. column=info:server,
timestamp=1404885456731(= Wed Jul 09 13:57:36 +0800 2014),
value=10.237.12.13:11600 -- put from 10.237.12.13
xxx.adfa2ed711bbdf054d9733a92fd43eb5. column=info:server,
timestamp=1404885353122( Wed Jul 09 13:55:53 +0800 2014),
value=10.237.12.15:11600 -- First put from 10.237.12.15
{quote}
Related hbase log is attached in this issue and disscusions are welcomed.
For there is no order guarantee for puts from different htables, one solution
for this issue is to give an increased id for each assignment of a region and
use this id as the timestamp of put of region location to meta table. The
region location with large assign id will be got by hbase clients.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13709) Updates to meta table server columns may be eclipsed
[ https://issues.apache.org/jira/browse/HBASE-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13709: -- Status: Patch Available (was: Open) Updates to meta table server columns may be eclipsed Key: HBASE-13709 URL: https://issues.apache.org/jira/browse/HBASE-13709 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1 Attachments: hbase-13709_v1.patch HBASE-11536 fixes a case where on a very rare occasion, the meta updates may be processed out of order. The fix is to use the RS's timestamp for the server column in meta update, but that actually opens up a vulnerability for clock skew (see the discussions in the jira). For the region replicas case, we can reproduce a problem where the server name field is eclipsed by the masters earlier update because the RS is lagging behind. However, this is not specific to replicas, but occurs more frequently with it. One option that was discussed was to send the master's ts with open region RPC and use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13709) Updates to meta table server columns may be eclipsed
[ https://issues.apache.org/jira/browse/HBASE-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13709: -- Attachment: hbase-13709_v1.patch Attaching a patch which implements the scheme described in HBASE-11536. We send the masters wall clock with the open region RPC, and we use the max(masterSystemTime, localTime) from RS when updating the meta location. It adds new methods to RegionServerServices, but I think it should be fine for patch releases for 1.0+. Updates to meta table server columns may be eclipsed Key: HBASE-13709 URL: https://issues.apache.org/jira/browse/HBASE-13709 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1 Attachments: hbase-13709_v1.patch HBASE-11536 fixes a case where on a very rare occasion, the meta updates may be processed out of order. The fix is to use the RS's timestamp for the server column in meta update, but that actually opens up a vulnerability for clock skew (see the discussions in the jira). For the region replicas case, we can reproduce a problem where the server name field is eclipsed by the masters earlier update because the RS is lagging behind. However, this is not specific to replicas, but occurs more frequently with it. One option that was discussed was to send the master's ts with open region RPC and use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13698) Add RegionLocator methods to Thrift2 proxy.
[ https://issues.apache.org/jira/browse/HBASE-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13698: -- Fix Version/s: 1.2.0 2.0.0 Affects Version/s: 2.0.0 1.1.0 Status: Patch Available (was: Open) Add RegionLocator methods to Thrift2 proxy. --- Key: HBASE-13698 URL: https://issues.apache.org/jira/browse/HBASE-13698 Project: HBase Issue Type: Bug Affects Versions: 1.1.0, 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13698.patch Thrift2 doesn't provide the same functionality as the java client for getting region locations. We should change that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13698) Add RegionLocator methods to Thrift2 proxy.
[ https://issues.apache.org/jira/browse/HBASE-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13698: -- Attachment: HBASE-13698.patch Add RegionLocator methods to Thrift2 proxy. --- Key: HBASE-13698 URL: https://issues.apache.org/jira/browse/HBASE-13698 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 1.1.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13698.patch Thrift2 doesn't provide the same functionality as the java client for getting region locations. We should change that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13673) WALProcedureStore procedure is chatty
[ https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-13673: Resolution: Fixed Fix Version/s: 1.1.1 1.2.0 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews. Pushed to branch-1.1+ WALProcedureStore procedure is chatty - Key: HBASE-13673 URL: https://issues.apache.org/jira/browse/HBASE-13673 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 1.1.0 Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Priority: Minor Fix For: 2.0.0, 1.2.0, 1.1.1 Attachments: HBASE-13673.patch, HBASE-13673_v2.patch Lots of procedure logging at INFO level which should be at DEBUG, especially Remove all state logs with ID less then 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13675) ProcedureExecutor completion report should be at DEBUG log level
[
https://issues.apache.org/jira/browse/HBASE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Srungarapu updated HBASE-13675:
Resolution: Fixed
Fix Version/s: 1.1.1
1.2.0
2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the reviews. Pushed to branch-1.1+.
ProcedureExecutor completion report should be at DEBUG log level
Key: HBASE-13675
URL: https://issues.apache.org/jira/browse/HBASE-13675
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Fix For: 2.0.0, 1.2.0, 1.1.1
Attachments: HBASE-13675.patch, HBASE-13675_v2.patch
Example:
{noformat}
2015-05-12 12:05:03,445 INFO [ProcedureExecutorThread-0]
procedure2.ProcedureExecutor: Procedure completed in 838msec:
EnableTableProcedure (table=cluster_test) user=apurtell (auth:SIMPLE) id=5
state=FINISHED
{noformat}
The procedures themselves are already emitting task specific information at
INFO level. This doesn't add much information besides the elapsed time, so
probably should be at DEBUG level. I assume if a procedure fails there will
be additional logging at WARN level, but if not, that could be a
consideration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13704) Hbase throws OutOfOrderScannerNextException exception when MultiRowRangeFilter is used.
[
https://issues.apache.org/jira/browse/HBASE-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549736#comment-14549736
]
Jiajia Li commented on HBASE-13704:
---
I think if the range already contains row, it will from Line91 to Line121. Can
you provide your test more detailed, maybe I got it wrong, thanks.
Hbase throws OutOfOrderScannerNextException exception when
MultiRowRangeFilter is used.
---
Key: HBASE-13704
URL: https://issues.apache.org/jira/browse/HBASE-13704
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 1.1.0
Reporter: Aleksandr Maksymenko
When using filter MultiRowRangeFilter with ranges closed to each other that
there are no rows between ranges, then OutOfOrderScannerNextException is
throwed.
In filterRowKey method when range is switched to the next range,
currentReturnCode is set to SEEK_NEXT_USING_HINT (MultiRowRangeFilter: 118 in
v1.1.0). But if new range is already contain this row, then we should include
this row, not to seek for another one.
Replacing line 118 to this code seems to be working fine:
{code}
if (range.contains(buffer, offset, length)) {
currentReturnCode = ReturnCode.INCLUDE;
} else {
currentReturnCode = ReturnCode.SEEK_NEXT_USING_HINT;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13510) Purge ByteBloomFilter
[ https://issues.apache.org/jira/browse/HBASE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13510: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Purge ByteBloomFilter - Key: HBASE-13510 URL: https://issues.apache.org/jira/browse/HBASE-13510 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13510_1.patch, HBASE-13510_2.patch, HBASE-13510_3.patch, HBASE-13510_5.patch, HBASE-13510_6.patch, HBASE-13510_7.patch In order to address the comments over in HBASE-10800 related to comparing Cell with a serialized KV's key we had some need for that in Bloom filters. After discussing with Anoop, we found that it may be possible to remove/modify some of the APIs in the BloomFilter interfaces and for doing that we can purge ByteBloomFilter. I read the code and found that ByteBloomFilter was getting used in V1 version only. Now as it is obsolete we can remove this code and move some of the static APIs in ByteBloomFilter to some other util class or bloom related classes which will help us in refactoring the code too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13675) ProcedureExecutor completion report should be at DEBUG log level
[
https://issues.apache.org/jira/browse/HBASE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549824#comment-14549824
]
Hudson commented on HBASE-13675:
FAILURE: Integrated in HBase-1.1 #492 (See
[https://builds.apache.org/job/HBase-1.1/492/])
HBASE-13675 ProcedureExecutor completion report should be at DEBUG log level
(ssrungarapu: rev 71bfcfded7bcc261a426adf1288f1f484dc003cc)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
ProcedureExecutor completion report should be at DEBUG log level
Key: HBASE-13675
URL: https://issues.apache.org/jira/browse/HBASE-13675
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Fix For: 2.0.0, 1.2.0, 1.1.1
Attachments: HBASE-13675.patch, HBASE-13675_v2.patch
Example:
{noformat}
2015-05-12 12:05:03,445 INFO [ProcedureExecutorThread-0]
procedure2.ProcedureExecutor: Procedure completed in 838msec:
EnableTableProcedure (table=cluster_test) user=apurtell (auth:SIMPLE) id=5
state=FINISHED
{noformat}
The procedures themselves are already emitting task specific information at
INFO level. This doesn't add much information besides the elapsed time, so
probably should be at DEBUG level. I assume if a procedure fails there will
be additional logging at WARN level, but if not, that could be a
consideration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13673) WALProcedureStore procedure is chatty
[ https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549823#comment-14549823 ] Hudson commented on HBASE-13673: FAILURE: Integrated in HBase-1.1 #492 (See [https://builds.apache.org/job/HBase-1.1/492/]) HBASE-13673 WALProcedureStore procedure is chatty (ssrungarapu: rev 356fd25e84c4ad81d2f1b29ac397a9ddc0bd692a) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java WALProcedureStore procedure is chatty - Key: HBASE-13673 URL: https://issues.apache.org/jira/browse/HBASE-13673 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 1.1.0 Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Priority: Minor Fix For: 2.0.0, 1.2.0, 1.1.1 Attachments: HBASE-13673.patch, HBASE-13673_v2.patch Lots of procedure logging at INFO level which should be at DEBUG, especially Remove all state logs with ID less then 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13705) MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false
[ https://issues.apache.org/jira/browse/HBASE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549730#comment-14549730 ] Jiajia Li commented on HBASE-13705: --- For the second issue, suppose we have a range [10,12), and only one rowkey 12 in hbase table, steps of scan: 1. Line 91: Check if current range contains row 12 2. Line 94: Search for the next RowRange in method getNextRangeIndex 3. Line 223: index = -2 4. Line 225: insertionPosition = 1 5. Line 95: get the the index=1 will equal to rangList.size()=1, so return false, the row 12 will not be included. [~masyaman], I think this test is ok, please advise if I didn't get your point. MultiRowRangeFilter seems to be working incorrect if RowRange.startRowInclusive = false --- Key: HBASE-13705 URL: https://issues.apache.org/jira/browse/HBASE-13705 Project: HBase Issue Type: Bug Affects Versions: 1.1.0 Reporter: Aleksandr Maksymenko I've found the issue during code review, so I don't have tests and I even didn't test this case manualy. So I'll try to describe it in words. Pre-condition: we're using scan with MultiRowRangeFilter with some RowRange's with startRowInclusive = false. This means that we want to include all rows that are strictly greater than startRow (and less then stopRow, but it doesn't matter for now). What happens in MultiRowRangeFilter.filterRowKey (worth case is described): 1. Line 91: Check if current range contains a row. Lets follow the case when it doesn't. 2. Line 94: Search for the next RowRange in method getNextRangeIndex. 3. Line 238: We've found a RowRange, check if startRowInclusive == false and set EXCLUSIVE = true. This variable indicates if next row should be excluded. 4. Line 105: Check if EXCLUSIVE == true, if so skip this row. The problem: we've skipped first row we got in this range, but we never checked if this row is a RowRange.startRow . In distributed system may not get RowRange.startRow on current instance, so we may exclude some another row. Moreover, we may not have RowRange.startRow at all in the DB, we will exclude some rows that are (possible) close to RowRange.startRow, but not equals to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13510) Purge ByteBloomFilter
[ https://issues.apache.org/jira/browse/HBASE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549759#comment-14549759 ] ramkrishna.s.vasudevan commented on HBASE-13510: Any chance of +1s? Purge ByteBloomFilter - Key: HBASE-13510 URL: https://issues.apache.org/jira/browse/HBASE-13510 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13510_1.patch, HBASE-13510_2.patch, HBASE-13510_3.patch, HBASE-13510_5.patch, HBASE-13510_6.patch In order to address the comments over in HBASE-10800 related to comparing Cell with a serialized KV's key we had some need for that in Bloom filters. After discussing with Anoop, we found that it may be possible to remove/modify some of the APIs in the BloomFilter interfaces and for doing that we can purge ByteBloomFilter. I read the code and found that ByteBloomFilter was getting used in V1 version only. Now as it is obsolete we can remove this code and move some of the static APIs in ByteBloomFilter to some other util class or bloom related classes which will help us in refactoring the code too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13510) Purge ByteBloomFilter
[ https://issues.apache.org/jira/browse/HBASE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549764#comment-14549764 ] stack commented on HBASE-13510: --- +1 nit: In CellUtil, you add an unused import: 36 import org.apache.hadoop.hbase.util.Pair; .. fix on commit. Ditto for @VisibleForTesting additions. Purge ByteBloomFilter - Key: HBASE-13510 URL: https://issues.apache.org/jira/browse/HBASE-13510 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13510_1.patch, HBASE-13510_2.patch, HBASE-13510_3.patch, HBASE-13510_5.patch, HBASE-13510_6.patch In order to address the comments over in HBASE-10800 related to comparing Cell with a serialized KV's key we had some need for that in Bloom filters. After discussing with Anoop, we found that it may be possible to remove/modify some of the APIs in the BloomFilter interfaces and for doing that we can purge ByteBloomFilter. I read the code and found that ByteBloomFilter was getting used in V1 version only. Now as it is obsolete we can remove this code and move some of the static APIs in ByteBloomFilter to some other util class or bloom related classes which will help us in refactoring the code too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13709) Updates to meta table server columns may be eclipsed
[ https://issues.apache.org/jira/browse/HBASE-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549732#comment-14549732 ] Liu Shaohui commented on HBASE-13709: - [~enis] The system time of backup master may be less than that of active master. So the master system time may decrease during master failover. In the extreme case, the updates to meta table also will be eclipsed. Updates to meta table server columns may be eclipsed Key: HBASE-13709 URL: https://issues.apache.org/jira/browse/HBASE-13709 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1 Attachments: hbase-13709_v1.patch HBASE-11536 fixes a case where on a very rare occasion, the meta updates may be processed out of order. The fix is to use the RS's timestamp for the server column in meta update, but that actually opens up a vulnerability for clock skew (see the discussions in the jira). For the region replicas case, we can reproduce a problem where the server name field is eclipsed by the masters earlier update because the RS is lagging behind. However, this is not specific to replicas, but occurs more frequently with it. One option that was discussed was to send the master's ts with open region RPC and use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13709) Updates to meta table server columns may be eclipsed
[ https://issues.apache.org/jira/browse/HBASE-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549749#comment-14549749 ] Elliott Clark commented on HBASE-13709: --- The other option that I've always thought would solve some of the races in meta is to always do a check and put. Updates to meta table server columns may be eclipsed Key: HBASE-13709 URL: https://issues.apache.org/jira/browse/HBASE-13709 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1 Attachments: hbase-13709_v1.patch HBASE-11536 fixes a case where on a very rare occasion, the meta updates may be processed out of order. The fix is to use the RS's timestamp for the server column in meta update, but that actually opens up a vulnerability for clock skew (see the discussions in the jira). For the region replicas case, we can reproduce a problem where the server name field is eclipsed by the masters earlier update because the RS is lagging behind. However, this is not specific to replicas, but occurs more frequently with it. One option that was discussed was to send the master's ts with open region RPC and use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13510) Purge ByteBloomFilter
[ https://issues.apache.org/jira/browse/HBASE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13510: --- Attachment: HBASE-13510_7.patch Patch that was pushed to master. Thanks for the reviews [~anoop.hbase] and [~saint@gmail.com]. Purge ByteBloomFilter - Key: HBASE-13510 URL: https://issues.apache.org/jira/browse/HBASE-13510 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13510_1.patch, HBASE-13510_2.patch, HBASE-13510_3.patch, HBASE-13510_5.patch, HBASE-13510_6.patch, HBASE-13510_7.patch In order to address the comments over in HBASE-10800 related to comparing Cell with a serialized KV's key we had some need for that in Bloom filters. After discussing with Anoop, we found that it may be possible to remove/modify some of the APIs in the BloomFilter interfaces and for doing that we can purge ByteBloomFilter. I read the code and found that ByteBloomFilter was getting used in V1 version only. Now as it is obsolete we can remove this code and move some of the static APIs in ByteBloomFilter to some other util class or bloom related classes which will help us in refactoring the code too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13673) WALProcedureStore procedure is chatty
[ https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549792#comment-14549792 ] Hudson commented on HBASE-13673: FAILURE: Integrated in HBase-TRUNK #6491 (See [https://builds.apache.org/job/HBase-TRUNK/6491/]) HBASE-13673 WALProcedureStore procedure is chatty (ssrungarapu: rev bc189d0497e4052159242b9a358e8d6e2f4a2f2b) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java WALProcedureStore procedure is chatty - Key: HBASE-13673 URL: https://issues.apache.org/jira/browse/HBASE-13673 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 1.1.0 Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Priority: Minor Fix For: 2.0.0, 1.2.0, 1.1.1 Attachments: HBASE-13673.patch, HBASE-13673_v2.patch Lots of procedure logging at INFO level which should be at DEBUG, especially Remove all state logs with ID less then 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13675) ProcedureExecutor completion report should be at DEBUG log level
[
https://issues.apache.org/jira/browse/HBASE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549793#comment-14549793
]
Hudson commented on HBASE-13675:
FAILURE: Integrated in HBase-TRUNK #6491 (See
[https://builds.apache.org/job/HBase-TRUNK/6491/])
HBASE-13675 ProcedureExecutor completion report should be at DEBUG log level
(ssrungarapu: rev 901714d75dd6da9c3b7dca3f48046119c6f08905)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
ProcedureExecutor completion report should be at DEBUG log level
Key: HBASE-13675
URL: https://issues.apache.org/jira/browse/HBASE-13675
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Fix For: 2.0.0, 1.2.0, 1.1.1
Attachments: HBASE-13675.patch, HBASE-13675_v2.patch
Example:
{noformat}
2015-05-12 12:05:03,445 INFO [ProcedureExecutorThread-0]
procedure2.ProcedureExecutor: Procedure completed in 838msec:
EnableTableProcedure (table=cluster_test) user=apurtell (auth:SIMPLE) id=5
state=FINISHED
{noformat}
The procedures themselves are already emitting task specific information at
INFO level. This doesn't add much information besides the elapsed time, so
probably should be at DEBUG level. I assume if a procedure fails there will
be additional logging at WARN level, but if not, that could be a
consideration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13698) Add RegionLocator methods to Thrift2 proxy.
[
https://issues.apache.org/jira/browse/HBASE-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549811#comment-14549811
]
Hadoop QA commented on HBASE-13698:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733691/HBASE-13698.patch
against master branch at commit 92e66ef5222391d106bc8adb7afe7d96a7dafac2.
ATTACHMENT ID: 12733691
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1899 checkstyle errors (more than the master's current 1898 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ exists_call method_call = new exists_call(table, tget,
resultHandler, this, ___protocolFactory, ___transport);
+ get_call method_call = new get_call(table, tget, resultHandler, this,
___protocolFactory, ___transport);
+ getMultiple_call method_call = new getMultiple_call(table, tgets,
resultHandler, this, ___protocolFactory, ___transport);
+ put_call method_call = new put_call(table, tput, resultHandler, this,
___protocolFactory, ___transport);
+ checkAndPut_call method_call = new checkAndPut_call(table, row, family,
qualifier, value, tput, resultHandler, this, ___protocolFactory, ___transport);
+ putMultiple_call method_call = new putMultiple_call(table, tputs,
resultHandler, this, ___protocolFactory, ___transport);
+ deleteSingle_call method_call = new deleteSingle_call(table, tdelete,
resultHandler, this, ___protocolFactory, ___transport);
+ deleteMultiple_call method_call = new deleteMultiple_call(table,
tdeletes, resultHandler, this, ___protocolFactory, ___transport);
+ checkAndDelete_call method_call = new checkAndDelete_call(table, row,
family, qualifier, value, tdelete, resultHandler, this, ___protocolFactory,
___transport);
+ increment_call method_call = new increment_call(table, tincrement,
resultHandler, this, ___protocolFactory, ___transport);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFastFail
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:287)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:261)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:255)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:239)
at
org.apache.hadoop.hbase.procedure.TestZKProcedure.testMultiCohortWithMemberTimeoutDuringPrepare(TestZKProcedure.java:311)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14082//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14082//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14082//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14082//console
This message is automatically generated.
Add RegionLocator methods to Thrift2 proxy.
---
Key: HBASE-13698
URL: https://issues.apache.org/jira/browse/HBASE-13698
Project: HBase
Issue
[jira] [Commented] (HBASE-13707) CellCounter uses to many counters
[ https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549826#comment-14549826 ] Lars Hofhansl commented on HBASE-13707: --- What the... This seems wrong. I do not think anybody wants this, we should remove it, and replace the code with a per total counter. Not a fan of a new option to keep senseless existing behavior. CellCounter uses to many counters - Key: HBASE-13707 URL: https://issues.apache.org/jira/browse/HBASE-13707 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 1.0.1 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: beginner, newbie CellCounters creates a counter per row... So it quickly becomes to many. We should provide an option to drop the statistic per rows and count only cells overall for the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13710) Remove use of Hadoop's ReflectionUtil in favor of our own.
[ https://issues.apache.org/jira/browse/HBASE-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13710: Priority: Minor (was: Major) Remove use of Hadoop's ReflectionUtil in favor of our own. -- Key: HBASE-13710 URL: https://issues.apache.org/jira/browse/HBASE-13710 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor HttpServer makes use of Hadoop's ReflectionUtil instead of our own. AFAICT it's using 1 extra method. Just copy that one over to our own ReflectionUtil. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13710) Remove use of Hadoop's ReflectionUtil in favor of our own.
Sean Busbey created HBASE-13710: --- Summary: Remove use of Hadoop's ReflectionUtil in favor of our own. Key: HBASE-13710 URL: https://issues.apache.org/jira/browse/HBASE-13710 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey HttpServer makes use of Hadoop's ReflectionUtil instead of our own. AFAICT it's using 1 extra method. Just copy that one over to our own ReflectionUtil. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13709) Updates to meta table server columns may be eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549728#comment-14549728
]
Hadoop QA commented on HBASE-13709:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733687/hbase-13709_v1.patch
against master branch at commit 92e66ef5222391d106bc8adb7afe7d96a7dafac2.
ATTACHMENT ID: 12733687
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1901 checkstyle errors (more than the master's current 1898 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14081//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14081//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14081//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14081//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14081//console
This message is automatically generated.
Updates to meta table server columns may be eclipsed
Key: HBASE-13709
URL: https://issues.apache.org/jira/browse/HBASE-13709
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: hbase-13709_v1.patch
HBASE-11536 fixes a case where on a very rare occasion, the meta updates may
be processed out of order. The fix is to use the RS's timestamp for the
server column in meta update, but that actually opens up a vulnerability for
clock skew (see the discussions in the jira).
For the region replicas case, we can reproduce a problem where the server
name field is eclipsed by the masters earlier update because the RS is
lagging behind. However, this is not specific to replicas, but occurs more
frequently with it.
One option that was discussed was to send the master's ts with open region
RPC and use it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13510) Purge ByteBloomFilter
[ https://issues.apache.org/jira/browse/HBASE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549768#comment-14549768 ] ramkrishna.s.vasudevan commented on HBASE-13510: Sure. Thank you Stack bq.Ditto for @VisibleForTesting additions. This got missed out in the patch that I uploaded. Purge ByteBloomFilter - Key: HBASE-13510 URL: https://issues.apache.org/jira/browse/HBASE-13510 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13510_1.patch, HBASE-13510_2.patch, HBASE-13510_3.patch, HBASE-13510_5.patch, HBASE-13510_6.patch In order to address the comments over in HBASE-10800 related to comparing Cell with a serialized KV's key we had some need for that in Bloom filters. After discussing with Anoop, we found that it may be possible to remove/modify some of the APIs in the BloomFilter interfaces and for doing that we can purge ByteBloomFilter. I read the code and found that ByteBloomFilter was getting used in V1 version only. Now as it is obsolete we can remove this code and move some of the static APIs in ByteBloomFilter to some other util class or bloom related classes which will help us in refactoring the code too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13673) WALProcedureStore procedure is chatty
[ https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549773#comment-14549773 ] Hudson commented on HBASE-13673: SUCCESS: Integrated in HBase-1.2 #85 (See [https://builds.apache.org/job/HBase-1.2/85/]) HBASE-13673 WALProcedureStore procedure is chatty (ssrungarapu: rev 25106221082bfdf1695b9ef01b8513a7bfc6ebde) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java WALProcedureStore procedure is chatty - Key: HBASE-13673 URL: https://issues.apache.org/jira/browse/HBASE-13673 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 1.1.0 Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Priority: Minor Fix For: 2.0.0, 1.2.0, 1.1.1 Attachments: HBASE-13673.patch, HBASE-13673_v2.patch Lots of procedure logging at INFO level which should be at DEBUG, especially Remove all state logs with ID less then 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13531) After 4/18/15 merge, flakey failures of TestAcidGuarantees#testMobScanAtomicity
[ https://issues.apache.org/jira/browse/HBASE-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549797#comment-14549797 ] Jingcheng Du commented on HBASE-13531: -- How about the latest patch? [~jmhsieh], [~anoopsamjohn], [~ram_krish], do you want to look at it? Thanks! After 4/18/15 merge, flakey failures of TestAcidGuarantees#testMobScanAtomicity --- Key: HBASE-13531 URL: https://issues.apache.org/jira/browse/HBASE-13531 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Affects Versions: hbase-11339 Reporter: Jonathan Hsieh Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-13531-V2.diff, HBASE-13531-V3.diff, HBASE-13531-V4.diff, HBASE-13531-V5.diff, HBASE-13531.diff After the merge of master from 4/18/15 with hbase-11339 branch, we encounter some atomicity violations. We want to fix before calling merge to trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13673) WALProcedureStore procedure is chatty
[
https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549569#comment-14549569
]
Hadoop QA commented on HBASE-13673:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733640/HBASE-13673_v2.patch
against master branch at commit 92e66ef5222391d106bc8adb7afe7d96a7dafac2.
ATTACHMENT ID: 12733640
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14079//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14079//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14079//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14079//console
This message is automatically generated.
WALProcedureStore procedure is chatty
-
Key: HBASE-13673
URL: https://issues.apache.org/jira/browse/HBASE-13673
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Andrew Purtell
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-13673.patch, HBASE-13673_v2.patch
Lots of procedure logging at INFO level which should be at DEBUG, especially
Remove all state logs with ID less then 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13709) Updates to meta table server columns may be eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549622#comment-14549622
]
Enis Soztutar edited comment on HBASE-13709 at 5/19/15 1:32 AM:
Here are the details from a test failure:
We create a table with 8 replicas, and end of test, we verify that all the
replicas are assigned. For that we do a meta scan, which does not see replica
id 5 (notice that the server value is null):
{code}
2015-05-15
09:03:06,982|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server, timestamp=1431680376535,
value=ip-172-31-41-12.ec2.internal:16020
2015-05-15
09:03:06,984|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0001, timestamp=1431680377792,
value=ip-172-31-41-10.ec2.internal:16020
2015-05-15
09:03:06,985|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0002, timestamp=1431680376517,
value=ip-172-31-41-12.ec2.internal:16020
2015-05-15
09:03:06,987|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0003, timestamp=1431680377792,
value=ip-172-31-41-10.ec2.internal:16020
2015-05-15
09:03:06,989|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0004, timestamp=1431680377259,
value=ip-172-31-41-6.ec2.internal:16020
2015-05-15
09:03:06,990|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0005, timestamp=1431680375996, value=
2015-05-15
09:03:06,992|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0006, timestamp=1431680376986,
value=ip-172-31-41-11.ec2.internal:16020
2015-05-15
09:03:06,994|beaver.machine|INFO|30851|140131703838464|MainThread|tableone,,1431680375568.d3059f4bf8e7a0a07660d7cf01217189.
column=info:server_0007, timestamp=1431680377359,
value=ip-172-31-41-7.ec2.internal:16020
{code}
Looking at the master and region server logs, it seems that the region replica
with name {{tableone,,1431680375568_0005.68af88e9274633ba67b926b142b7ff4c}} is
indeed assigned, and it actually did write to meta:
{code}
2015-05-15 08:59:35,473 INFO [PriorityRpcServer.handler=10,queue=0,port=16020]
regionserver.RSRpcServices: Open
tableone,,1431680375568_0005.68af88e9274633ba67b926b142b7ff4c.
2015-05-15 08:59:35,494 INFO [StoreOpener-68af88e9274633ba67b926b142b7ff4c-1]
hfile.CacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=459272,
freeSize=420648856, maxSize=421108128, heapSi
2015-05-15 08:59:35,495 INFO [StoreOpener-68af88e9274633ba67b926b142b7ff4c-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.20; off-peak ratio 5
2015-05-15 08:59:35,499 INFO [StoreOpener-68af88e9274633ba67b926b142b7ff4c-1]
hfile.CacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=459272,
freeSize=420648856, maxSize=421108128, heapSi
2015-05-15 08:59:35,500 INFO [StoreOpener-68af88e9274633ba67b926b142b7ff4c-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.20; off-peak ratio 5
2015-05-15 08:59:35,504 INFO [StoreOpener-68af88e9274633ba67b926b142b7ff4c-1]
hfile.CacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=459272,
freeSize=420648856, maxSize=421108128, heapSi
2015-05-15 08:59:35,504 INFO [StoreOpener-68af88e9274633ba67b926b142b7ff4c-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.20; off-peak ratio 5
2015-05-15 08:59:35,507 INFO [RS_OPEN_REGION-ip-172-31-41-8:16020-2]
regionserver.HRegion: Onlined 68af88e9274633ba67b926b142b7ff4c; next
sequenceid=2
2015-05-15 08:59:35,513 INFO
[PostOpenDeployTasks:68af88e9274633ba67b926b142b7ff4c]
regionserver.HRegionServer: Post open deploy tasks for
tableone,,1431680375568_0005.68af88e9274633ba67b926b142b7ff4c.
2015-05-15 08:59:35,522 INFO
[PostOpenDeployTasks:68af88e9274633ba67b926b142b7ff4c] hbase.MetaTableAccessor:
Updated row tableone,,1431680375568_0005.68af88e9274633ba67b926b142b7ff4c. with
server=ip-17
{code}
As you can see from the meta scan results, the cell:
{code}
column=info:server_0005, timestamp=1431680375996, value=
{code}
has timestamp of 1431680375996, which is {{Fri, 15 May 2015 08:59:35 GMT}} and
the ms is 996, however the meta update from the RS uses it's local timestamp
because of HBASE-11536. As you can see from the log:
{code}
2015-05-15 08:59:35,522 INFO
[PostOpenDeployTasks:68af88e9274633ba67b926b142b7ff4c]
[jira] [Commented] (HBASE-13376) Improvements to Stochastic load balancer
[
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549682#comment-14549682
]
Ted Yu commented on HBASE-13376:
The test failure can be reproduced:
{code}
testRebalanceOnRegionServerNumberChange[1](org.apache.hadoop.hbase.TestRegionRebalancing)
Time elapsed: 8.44 sec ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster$2.compare(BaseLoadBalancer.java:772)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster$2.compare(BaseLoadBalancer.java:769)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:324)
at java.util.TimSort.sort(TimSort.java:189)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.sortServersByLocality(BaseLoadBalancer.java:762)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLowestLocalityRegionServer(BaseLoadBalancer.java:788)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityBasedCandidateGenerator.pickLowestLocalityServer(StochasticLoadBalancer.java:625)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityBasedCandidateGenerator.generate(StochasticLoadBalancer.java:597)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:266)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1241)
at
org.apache.hadoop.hbase.TestRegionRebalancing.testRebalanceOnRegionServerNumberChange(TestRegionRebalancing.java:119)
{code}
The NPE seems to come from this line in BaseLoadBalancer$Cluster$2.compare :
{code}
float locality1 = getLocality(integer);
{code}
Improvements to Stochastic load balancer
Key: HBASE-13376
URL: https://issues.apache.org/jira/browse/HBASE-13376
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: HBASE-13376_0.98.txt, HBASE-13376_0.txt,
HBASE-13376_98.patch
There are two things this jira tries to address:
1. The locality picker in the stochastic balancer does not pick regions with
least locality as candidates for swap/move. So when any user configures
locality cost in the configs, the balancer does not always seems to move
regions with bad locality.
2. When a cluster has equal number of loaded regions, it always picks the
first one. It should pick a random region on one of the equally loaded
servers. This improves a chance of finding a good candidate, when load picker
is invoked several times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13677) RecoverableZookeeper WARNs on expected events
[
https://issues.apache.org/jira/browse/HBASE-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549247#comment-14549247
]
Hudson commented on HBASE-13677:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #945 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/945/])
HBASE-13677 changed RecoverableZooKeeper logs about retries from info and warn
to debug (busbey: rev 3352585160b6ff10069cb587ba1b9ddb1aedea24)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
RecoverableZookeeper WARNs on expected events
-
Key: HBASE-13677
URL: https://issues.apache.org/jira/browse/HBASE-13677
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Ted Malaska
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 0.98.13, 1.2.0
Attachments: HBASE-13677.2.patch, HBASE-13677.patch
Noticed in the logs while verifying an RC, running a simple LTT in local mode
resulted in half a dozen lines like
{noformat}
2015-05-12 13:58:42,949 WARN [M:0;10.0.0.110:50866]
zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
quorum=localhost:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
{noformat}
A class called RecoverableZooKeeper should be expecting possibly
transient events relating to ZooKeeper. Let's drop this to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13609) TestFastFail is still failing
[
https://issues.apache.org/jira/browse/HBASE-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549377#comment-14549377
]
Hudson commented on HBASE-13609:
SUCCESS: Integrated in HBase-1.2 #84 (See
[https://builds.apache.org/job/HBase-1.2/84/])
HBASE-13609 TestFastFail is still failing (ndimiduk: rev
8614d86243ef8bd7cf27633e6e314b278cef0ab7)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
TestFastFail is still failing
-
Key: HBASE-13609
URL: https://issues.apache.org/jira/browse/HBASE-13609
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1
Attachments: 13609.branch-1.patch, 13609.patch, 13609.patch
{noformat}
testFastFail(org.apache.hadoop.hbase.client.TestFastFail) Time elapsed:
13.106 sec FAILURE!
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:15 threads that
retried : 2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:288)
{noformat}
This is failing consistently for me locally. Sometimes it's 15, sometimes
it's 5, sometimes 26.
We've seen this one before, HBASE-12771, HBASE-12881.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13694) CallQueueSize is incorrectly decremented until the response is sent
[
https://issues.apache.org/jira/browse/HBASE-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549577#comment-14549577
]
Hadoop QA commented on HBASE-13694:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12733636/HBASE-13694.2.patch
against master branch at commit 92e66ef5222391d106bc8adb7afe7d96a7dafac2.
ATTACHMENT ID: 12733636
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14078//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14078//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14078//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14078//console
This message is automatically generated.
CallQueueSize is incorrectly decremented until the response is sent
---
Key: HBASE-13694
URL: https://issues.apache.org/jira/browse/HBASE-13694
Project: HBase
Issue Type: Bug
Components: master, regionserver, rpc
Affects Versions: 2.0.0, 1.1.0, 0.98.12, 1.0.2, 1.2.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Attachments:
0001-HBASE-13694-CallQueueSize-is-incorrectly-decremented.patch,
HBASE-13694.2.patch
We should decrement the CallQueueSize as soon as we no longer need the call
around, e.g. after {{RpcServer.CurCall.set(null)}} otherwise we will be only
pushing back other client requests while we send the response back to the
client that originated the call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13376) Improvements to Stochastic load balancer
[
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549554#comment-14549554
]
Ted Yu commented on HBASE-13376:
{code}
771 private ComparatorInteger localityComparator = new
ComparatorInteger() {
772 @Override
773 public int compare(Integer integer, Integer integer2) {
{code}
Can the parameters for compare() have better names, such as server1, server2 ?
{code}
794 while (localityPerServer[lowestLocalServerIndex] == 0
795 (regionsPerServer[lowestLocalServerIndex].length == 0)) {
{code}
The above assumes 0 is the lowest locality. What if locality for
serverIndicesSortedByLocality[0] is greater than 0 but the region count on the
server is 0 ?
{code}
796 i++;
797 lowestLocalServerIndex = serverIndicesSortedByLocality[i];
{code}
Better check for boundary before assigning lowestLocalServerIndex.
Test failure in TestRegionRebalancing might be related to patch, however
https://builds.apache.org/job/PreCommit-HBASE-Build/13602//testReport/ is no
longer accessible.
Improvements to Stochastic load balancer
Key: HBASE-13376
URL: https://issues.apache.org/jira/browse/HBASE-13376
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: HBASE-13376_0.98.txt, HBASE-13376_0.txt,
HBASE-13376_98.patch
There are two things this jira tries to address:
1. The locality picker in the stochastic balancer does not pick regions with
least locality as candidates for swap/move. So when any user configures
locality cost in the configs, the balancer does not always seems to move
regions with bad locality.
2. When a cluster has equal number of loaded regions, it always picks the
first one. It should pick a random region on one of the equally loaded
servers. This improves a chance of finding a good candidate, when load picker
is invoked several times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13673) WALProcedureStore procedure is chatty
[ https://issues.apache.org/jira/browse/HBASE-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549589#comment-14549589 ] Matteo Bertozzi commented on HBASE-13673: - +1 WALProcedureStore procedure is chatty - Key: HBASE-13673 URL: https://issues.apache.org/jira/browse/HBASE-13673 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 1.1.0 Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-13673.patch, HBASE-13673_v2.patch Lots of procedure logging at INFO level which should be at DEBUG, especially Remove all state logs with ID less then 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
