[jira] [Created] (HBASE-8167) make RPC select ops multi-threaded
Liang Xie created HBASE-8167: Summary: make RPC select ops multi-threaded Key: HBASE-8167 URL: https://issues.apache.org/jira/browse/HBASE-8167 Project: HBase Issue Type: Improvement Components: IPC/RPC Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie Current RPC framework is single listener thread with multiple Reader thread, this is pretty similar with thrift's THsHaServer design. And from thrift0.8, TThreadedSelectorServer was introduced to meet network-io-bound scenario better. There's a benchmark report on it: https://github.com/m1ch1/mapkeeper/wiki/Thrift-Java-Servers-Compared I think we could make HBaseServer's listener multi-thread as well. Just a raw thought now, no patch available yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-7403: Attachment: hbase-7403-trunkv30.patch rebased on the trunk > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.95.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, > 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, > hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, > hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, > hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, > hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, > hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, > hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, > hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, > hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, > hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, > hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf > > > Support executing region merge transaction on Regionserver, similar with > split transaction > Process of merging two regions: > a.client sends RPC (dispatch merging regions) to master > b.master moves the regions together (on the same regionserver where the more > heavily loaded region resided) > c.master sends RPC (merge regions) to this regionserver > d.Regionserver executes the region merge transaction in the thread pool > e.the above b,c,d run asynchronously > Process of region merge transaction: > a.Construct a new region merge transaction. > b.prepare for the merge transaction, the transaction will be canceled if it > is unavailable, > e.g. two regions don't belong to same table; two regions are not adjacent in > a non-compulsory merge; region is closed or has reference > c.execute the transaction as the following: > /** > * Set region as in transition, set it into MERGING state. > */ > SET_MERGING_IN_ZK, > /** > * We created the temporary merge data directory. > */ > CREATED_MERGE_DIR, > /** > * Closed the merging region A. > */ > CLOSED_REGION_A, > /** > * The merging region A has been taken out of the server's online regions > list. > */ > OFFLINED_REGION_A, > /** > * Closed the merging region B. > */ > CLOSED_REGION_B, > /** > * The merging region B has been taken out of the server's online regions > list. > */ > OFFLINED_REGION_B, > /** > * Started in on creation of the merged region. > */ > STARTED_MERGED_REGION_CREATION, > /** > * Point of no return. If we got here, then transaction is not recoverable > * other than by crashing out the regionserver. > */ > PONR > d.roll back if step c throws exception > Usage: > HBaseAdmin#mergeRegions > See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608727#comment-13608727 ] chunhui shen commented on HBASE-8163: - [~te...@apache.org] The gap can be got through the metrics. e.g ganglia > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-8163v1.patch > > > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608760#comment-13608760
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574745/hbase-7403-trunkv30.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//console
This message is automatically generated.
> Online Merge
>
>
> Key: HBASE-7403
> URL: https://issues.apache.org/jira/browse/HBASE-7403
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.95.0
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.95.0, 0.98.0
>
> Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
> 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
> hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
> hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
> hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
> hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
> hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
> hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
> hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
> hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch,
> hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
> hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
>
>
> Support executing region merge transaction on Regionserver, similar with
> split transaction
> Process of merging two regions:
> a.client sends RPC (dispatch merging regions) to master
> b.master moves the regions together (on the same regionserver where the more
> heavily loaded region resided)
> c.master sends RPC (merge regions) to this regionserver
> d.Regionserver executes the region merge transaction in the thread pool
> e.the above b,c,d run asynchronously
> Process of region merge transaction:
> a.Construct a new region merge transaction.
> b.prepare for the merge transaction, the transaction will be canceled if it
> is unavailable,
> e.g. two regions don't belong to same table; two regions are not adjacent in
> a non-compulsory merge; region is closed or has reference
> c.execute the transaction as the following:
> /**
> * Set region as in transition, set it into MERGING state.
> */
> SET_MERGING_IN_ZK,
> /**
> * We created the temporary merge data di
[jira] [Updated] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-8163: Attachment: hbase-8163v2.patch > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-8163v1.patch, hbase-8163v2.patch > > > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-7403: Attachment: hbase-7403-trunkv31.patch fix javadoc warning > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.95.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, > 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, > hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, > hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, > hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, > hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, > hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, > hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, > hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, > hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, > hbase-7403-trunkv31.patch, hbase-7403-trunkv5.patch, > hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, > hbase-7403-trunkv9.patch, merge region.pdf > > > Support executing region merge transaction on Regionserver, similar with > split transaction > Process of merging two regions: > a.client sends RPC (dispatch merging regions) to master > b.master moves the regions together (on the same regionserver where the more > heavily loaded region resided) > c.master sends RPC (merge regions) to this regionserver > d.Regionserver executes the region merge transaction in the thread pool > e.the above b,c,d run asynchronously > Process of region merge transaction: > a.Construct a new region merge transaction. > b.prepare for the merge transaction, the transaction will be canceled if it > is unavailable, > e.g. two regions don't belong to same table; two regions are not adjacent in > a non-compulsory merge; region is closed or has reference > c.execute the transaction as the following: > /** > * Set region as in transition, set it into MERGING state. > */ > SET_MERGING_IN_ZK, > /** > * We created the temporary merge data directory. > */ > CREATED_MERGE_DIR, > /** > * Closed the merging region A. > */ > CLOSED_REGION_A, > /** > * The merging region A has been taken out of the server's online regions > list. > */ > OFFLINED_REGION_A, > /** > * Closed the merging region B. > */ > CLOSED_REGION_B, > /** > * The merging region B has been taken out of the server's online regions > list. > */ > OFFLINED_REGION_B, > /** > * Started in on creation of the merged region. > */ > STARTED_MERGED_REGION_CREATION, > /** > * Point of no return. If we got here, then transaction is not recoverable > * other than by crashing out the regionserver. > */ > PONR > d.roll back if step c throws exception > Usage: > HBaseAdmin#mergeRegions > See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8168) HMaster is not Running.
Nallapati T Krishna created HBASE-8168: -- Summary: HMaster is not Running. Key: HBASE-8168 URL: https://issues.apache.org/jira/browse/HBASE-8168 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.94.4 Environment: Ubuntu 12.4 Reporter: Nallapati T Krishna Fix For: 0.94.4 When I used to start Hbase, for the first time Hmaster is running. If i type JPS for second consecutive time, Hmaster is not working. It is showing some thing like the below. rg.apache.zookeeper.ClientCnxn: Opening socket connection to server ubuntu-1/172.16.78.122:. Will not attempt to authenticate using SASL (unknown error) 2013-03-20 22:04:33,491 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ubuntu-1/172.16.78.122:, initiating session 2013-03-20 22:04:33,492 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-03-20 22:04:33,592 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 2013-03-20 22:04:33,593 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 4000ms before retry #2... 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to authenticate using SASL (unknown error) 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to CLBBLR-6196/172.16.78.91:, initiating session 2013-03-20 22:04:33,761 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server CLBBLR-6176/172.16.78.136:. Will not attempt to authenticate using SASL (unknown error) 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to CLBBLR-6176/172.16.78.136:, initiating session 2013-03-20 22:04:34,841 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-03-20 22:04:36,026 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ubuntu-1/172.16.78.122:. Will not attempt to authenticate using SASL (unknown error) 2013-03-20 22:04:36,027 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ubuntu-1/172.16.78.122:, initiating session 2013-03-20 22:04:36,028 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-03-20 22:04:36,570 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to authenticate using SASL (unknown error) 2013-03-20 22:04:36,571 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to CLBBLR-6196/172.16.78.91:, initiating session 2013-03-20 22:04:36,572 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-03-20 22:04:37,505 INFO org.apache.zookeeper.ClientCnxn: Op As discussed.Hbase is starting properly without any issue. When we are trying to list the tables in Hbase or create the table, HMaster is not running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8168) HMaster is not Running.
[ https://issues.apache.org/jira/browse/HBASE-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nallapati T Krishna updated HBASE-8168: --- Attachment: Hbase Error.png > HMaster is not Running. > --- > > Key: HBASE-8168 > URL: https://issues.apache.org/jira/browse/HBASE-8168 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 0.94.4 > Environment: Ubuntu 12.4 >Reporter: Nallapati T Krishna > Fix For: 0.94.4 > > Attachments: Hbase Error.png > > > When I used to start Hbase, for the first time Hmaster is running. If i type > JPS for second consecutive time, Hmaster is not working. It is showing some > thing like the below. > rg.apache.zookeeper.ClientCnxn: Opening socket connection to server > ubuntu-1/172.16.78.122:. Will not attempt to authenticate using SASL > (unknown error) > 2013-03-20 22:04:33,491 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ubuntu-1/172.16.78.122:, initiating session > 2013-03-20 22:04:33,492 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:33,592 WARN > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient > ZooKeeper exception: > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss for /hbase > 2013-03-20 22:04:33,593 INFO org.apache.hadoop.hbase.util.RetryCounter: > Sleeping 4000ms before retry #2... > 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6196/172.16.78.91:, initiating session > 2013-03-20 22:04:33,761 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6176/172.16.78.136:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6176/172.16.78.136:, initiating session > 2013-03-20 22:04:34,841 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:36,026 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server ubuntu-1/172.16.78.122:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:36,027 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ubuntu-1/172.16.78.122:, initiating session > 2013-03-20 22:04:36,028 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:36,570 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:36,571 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6196/172.16.78.91:, initiating session > 2013-03-20 22:04:36,572 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:37,505 INFO org.apache.zookeeper.ClientCnxn: Op > As discussed.Hbase is starting properly without any issue. When we are trying > to list the tables in Hbase or create the table, HMaster is not running. > > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8168) HMaster is not Running.
[ https://issues.apache.org/jira/browse/HBASE-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608798#comment-13608798 ] ramkrishna.s.vasudevan commented on HBASE-8168: --- Pls do not raise JIRA for this. Send out a mail to the User list. Folks will help you out. BTW the problem here is i think the MAster is not able to connect to Zookeeper. Check if there is any firewall and your configs. The first time when you do JPS the process starts but it gets killed which means there is a setup/config related issue. > HMaster is not Running. > --- > > Key: HBASE-8168 > URL: https://issues.apache.org/jira/browse/HBASE-8168 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 0.94.4 > Environment: Ubuntu 12.4 >Reporter: Nallapati T Krishna > Fix For: 0.94.4 > > Attachments: Hbase Error.png > > > When I used to start Hbase, for the first time Hmaster is running. If i type > JPS for second consecutive time, Hmaster is not working. It is showing some > thing like the below. > rg.apache.zookeeper.ClientCnxn: Opening socket connection to server > ubuntu-1/172.16.78.122:. Will not attempt to authenticate using SASL > (unknown error) > 2013-03-20 22:04:33,491 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ubuntu-1/172.16.78.122:, initiating session > 2013-03-20 22:04:33,492 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:33,592 WARN > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient > ZooKeeper exception: > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss for /hbase > 2013-03-20 22:04:33,593 INFO org.apache.hadoop.hbase.util.RetryCounter: > Sleeping 4000ms before retry #2... > 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6196/172.16.78.91:, initiating session > 2013-03-20 22:04:33,761 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6176/172.16.78.136:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6176/172.16.78.136:, initiating session > 2013-03-20 22:04:34,841 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:36,026 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server ubuntu-1/172.16.78.122:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:36,027 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ubuntu-1/172.16.78.122:, initiating session > 2013-03-20 22:04:36,028 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:36,570 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:36,571 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6196/172.16.78.91:, initiating session > 2013-03-20 22:04:36,572 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:37,505 INFO org.apache.zookeeper.ClientCnxn: Op > As discussed.Hbase is starting properly without any issue. When we are trying > to list the tables in Hbase or create the table, HMaster is not running. > > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8168) HMaster is not Running.
[ https://issues.apache.org/jira/browse/HBASE-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-8168. --- Resolution: Invalid Pls raise in User list. > HMaster is not Running. > --- > > Key: HBASE-8168 > URL: https://issues.apache.org/jira/browse/HBASE-8168 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 0.94.4 > Environment: Ubuntu 12.4 >Reporter: Nallapati T Krishna > Fix For: 0.94.4 > > Attachments: Hbase Error.png > > > When I used to start Hbase, for the first time Hmaster is running. If i type > JPS for second consecutive time, Hmaster is not working. It is showing some > thing like the below. > rg.apache.zookeeper.ClientCnxn: Opening socket connection to server > ubuntu-1/172.16.78.122:. Will not attempt to authenticate using SASL > (unknown error) > 2013-03-20 22:04:33,491 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ubuntu-1/172.16.78.122:, initiating session > 2013-03-20 22:04:33,492 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:33,592 WARN > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient > ZooKeeper exception: > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss for /hbase > 2013-03-20 22:04:33,593 INFO org.apache.hadoop.hbase.util.RetryCounter: > Sleeping 4000ms before retry #2... > 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:33,760 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6196/172.16.78.91:, initiating session > 2013-03-20 22:04:33,761 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6176/172.16.78.136:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:34,840 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6176/172.16.78.136:, initiating session > 2013-03-20 22:04:34,841 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:36,026 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server ubuntu-1/172.16.78.122:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:36,027 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to ubuntu-1/172.16.78.122:, initiating session > 2013-03-20 22:04:36,028 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:36,570 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server CLBBLR-6196/172.16.78.91:. Will not attempt to > authenticate using SASL (unknown error) > 2013-03-20 22:04:36,571 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to CLBBLR-6196/172.16.78.91:, initiating session > 2013-03-20 22:04:36,572 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2013-03-20 22:04:37,505 INFO org.apache.zookeeper.ClientCnxn: Op > As discussed.Hbase is starting properly without any issue. When we are trying > to list the tables in Hbase or create the table, HMaster is not running. > > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608812#comment-13608812
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574758/hbase-7403-trunkv31.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//console
This message is automatically generated.
> Online Merge
>
>
> Key: HBASE-7403
> URL: https://issues.apache.org/jira/browse/HBASE-7403
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.95.0
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.95.0, 0.98.0
>
> Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
> 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
> hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
> hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
> hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
> hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
> hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
> hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
> hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
> hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch,
> hbase-7403-trunkv31.patch, hbase-7403-trunkv5.patch,
> hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
> hbase-7403-trunkv9.patch, merge region.pdf
>
>
> Support executing region merge transaction on Regionserver, similar with
> split transaction
> Process of merging two regions:
> a.client sends RPC (dispatch merging regions) to master
> b.master moves the regions together (on the same regionserver where the more
> heavily loaded region resided)
> c.master sends RPC (merge regions) to this regionserver
> d.Regionserver executes the region merge transaction in the thread pool
> e.the above b,c,d run asynchronously
> Process of region merge transaction:
> a.Construct a new region merge transaction.
> b.prepare for the merge transaction, the transaction will be canceled if it
> is unavailable,
> e.g. two regions don't belong to same table; two regions are not adjacent in
> a non-compulsory merge; region is closed or has reference
> c.execute the transaction as the following:
> /**
> * Set region as in transition, set it into MERGING
[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1
[ https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608818#comment-13608818 ] Andrew Purtell commented on HBASE-8165: --- Is Hadoop using 2.5? Do we need two protocs installed to build Hadoop and HBase ? Will a 2.5 shipped with HBase collide with 2.4 library shipped with the HDFS client on the classpath? > Update our protobuf to 2.5 from 2.4.1 > - > > Key: HBASE-8165 > URL: https://issues.apache.org/jira/browse/HBASE-8165 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.95.0 > > Attachments: 8165.txt > > > Update to new 2.5 pb. Some speedups and a new PARSER idiom that bypasses > making a builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7948) client doesn't need to refresh meta while the region is opening
[ https://issues.apache.org/jira/browse/HBASE-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608856#comment-13608856 ] Hudson commented on HBASE-7948: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #457 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/457/]) HBASE-7948 client doesn't need to refresh meta while the region is opening (Revision 1459130) Result = FAILURE sershe : Files : * /hbase/trunk/conf/log4j.properties * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RegionOpeningException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHRegionLocation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java * /hbase/trunk/hbase-server/src/test/resources/log4j.properties > client doesn't need to refresh meta while the region is opening > --- > > Key: HBASE-7948 > URL: https://issues.apache.org/jira/browse/HBASE-7948 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7948-v0.patch, HBASE-7948-v1.patch, > HBASE-7948-v1.patch, HBASE-7948-v2.patch, HBASE-7948-v3.patch, > HBASE-7948-v4.patch, HBASE-7948-v5-.patch, HBASE-7948-v5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[ https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7953: --- Status: Open (was: Patch Available) > Some HFilePerformanceEvaluation tests might fail because of scanner.getValue > when there is no more rows > --- > > Key: HBASE-7953 > URL: https://issues.apache.org/jira/browse/HBASE-7953 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch, > HBASE-7953-v2-trunk.patch > > > There is 2 places in the code (Both trunk and 0.94) where we are trying to > get the row value even when scanner.next() returned null. Need to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[ https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7953: --- Status: Patch Available (was: Open) Really need to get this in. Launching Hadoop QA again... > Some HFilePerformanceEvaluation tests might fail because of scanner.getValue > when there is no more rows > --- > > Key: HBASE-7953 > URL: https://issues.apache.org/jira/browse/HBASE-7953 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch, > HBASE-7953-v2-trunk.patch > > > There is 2 places in the code (Both trunk and 0.94) where we are trying to > get the row value even when scanner.next() returned null. Need to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7848) Use ZK-based read/write lock to make flush-type snapshot robust against table enable/disable/alter
[ https://issues.apache.org/jira/browse/HBASE-7848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608947#comment-13608947 ] Jonathan Hsieh commented on HBASE-7848: --- Then I'm concerned that all the other handlers may have the same problem. Other operations (ZK nodes in splitting for example) have had a history of assuming exn means failure (when it could mean success but temporary network partition). Looking at https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableLockManager.java#L223 it seem that when interruptions and timeouts exceptions are thrown, the lock could be taken but reported as an exception. Can you give me a pointer to some tests that exercies the timeout and/or interrupt path? A quick solution may be to use the is a slight modification of the canonical java lock pattern -- put the acquire inside the try block, and make sure the release is called and is able to differentiate between the different lock acquire failure modes or can release when acquire returned in the questionable state) > Use ZK-based read/write lock to make flush-type snapshot robust against table > enable/disable/alter > -- > > Key: HBASE-7848 > URL: https://issues.apache.org/jira/browse/HBASE-7848 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.96.0 >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 7848-v1.txt, 7848-v2.txt, hbase-7848_v2.patch > > > Current region split following flush would fail snapshot. > We can utilize ZK-based read/write lock to make flush-type snapshot robust -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8160) HMaster#move doesn't check if master initialized
[
https://issues.apache.org/jira/browse/HBASE-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608960#comment-13608960
]
Jonathan Hsieh commented on HBASE-8160:
---
The other uses of checkInitialized just wrap with ServiceException (no extra
UnknownRegionException wrapping on init check fails). Stick to the pattern
(and remove comment)?
Other methods have a this structural pattern, consider using it in moveRegion
as well?
{code}
ProtoResponse method(ProtoRequest) {
...
actualMethod()
...
}
retType actualMethod(arg, arg, arg) {
..
checkInitialized()
.. do stuff
}
{code}
> HMaster#move doesn't check if master initialized
>
>
> Key: HBASE-8160
> URL: https://issues.apache.org/jira/browse/HBASE-8160
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Trivial
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 0.94-8160.patch, trunk-8160.patch
>
>
> Before the master is initialized, if a move request comes in, it could cause
> the master to abort, if the region was already in transition before the
> master started up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8156) Support for Namenode HA for non-idempotent operations
[ https://issues.apache.org/jira/browse/HBASE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609005#comment-13609005 ] Himanshu Vashishtha commented on HBASE-8156: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat passes for me on local. > Support for Namenode HA for non-idempotent operations > - > > Key: HBASE-8156 > URL: https://issues.apache.org/jira/browse/HBASE-8156 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Affects Versions: 0.95.0 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.98.0 > > Attachments: HBase-8156-trunk-v1.patch, HBase-8156-trunk-v2.patch > > > In hadoop 2 HA, non-idempotent operations are not retried at the hdfs side. > This is by design as retrying a non-idempotent operation might not be a good > design choice for some use case. > HBase needs to handle the retries for such operations at its end. > With HBase-7806, there is already some work going on for file system > abstractions. There, HReginFileSystem sits as an abstraction between region > and FS. This jira is a move in the same direction, where it adds retry > functionality for non-idempotent calls such as create, rename and delete in > HRegionFileSystem class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7878: -- Attachment: 7878-trunk-v14.txt > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v14.txt, 7878-trunk-v14.txt, > 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, > 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, > 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609063#comment-13609063 ] Ted Yu commented on HBASE-8163: --- Does it make more sense to define maxsize as percentage of (total) memstore size ? > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-8163v1.patch, hbase-8163v2.patch > > > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7878: -- Attachment: (was: 7878-trunk-v14.txt) > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7878: -- Attachment: (was: 7878-trunk-v14.txt) > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7878: -- Attachment: (was: 7878-trunk-v14.txt) > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7878: -- Attachment: 7878-trunk-v14.txt > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7848) Use ZK-based read/write lock to make flush-type snapshot robust against table enable/disable/alter
[ https://issues.apache.org/jira/browse/HBASE-7848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609072#comment-13609072 ] ramkrishna.s.vasudevan commented on HBASE-7848: --- HBASE-8131 modifies the way CreateTableHandler releases the lock. It was me who changed that. So even incase of exceptions while creating table the lock would be released. Otherwise the user was unable to perform the create table operation once again. Pls do correct me if that was wrong. > Use ZK-based read/write lock to make flush-type snapshot robust against table > enable/disable/alter > -- > > Key: HBASE-7848 > URL: https://issues.apache.org/jira/browse/HBASE-7848 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.96.0 >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 7848-v1.txt, 7848-v2.txt, hbase-7848_v2.patch > > > Current region split following flush would fail snapshot. > We can utilize ZK-based read/write lock to make flush-type snapshot robust -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[
https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609082#comment-13609082
]
Ted Yu commented on HBASE-7953:
---
Can you use the following instead of calling System.out.println() ?
{code}
static final Log LOG =
LogFactory.getLog(HFilePerformanceEvaluation.class.getName());
{code}
{code}
+ if (scanner.seekTo(b) < 0) {
+System.out.println("Not able to seekTo");
{code}
Include contents of b above.
{code}
+ if (scanner.seekTo(getGaussianRandomRowBytes()) < 0) {
+System.out.println("Not able to seekTo");
{code}
Store getGaussianRandomRowBytes() in a variable and include the contents of the
variable in log, please.
> Some HFilePerformanceEvaluation tests might fail because of scanner.getValue
> when there is no more rows
> ---
>
> Key: HBASE-7953
> URL: https://issues.apache.org/jira/browse/HBASE-7953
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch,
> HBASE-7953-v2-trunk.patch
>
>
> There is 2 places in the code (Both trunk and 0.94) where we are trying to
> get the row value even when scanner.next() returned null. Need to fix that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8160) HMaster#move doesn't check if master initialized
[ https://issues.apache.org/jira/browse/HBASE-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609086#comment-13609086 ] Jimmy Xiang commented on HBASE-8160: I noticed that pattern. Sure, will do. > HMaster#move doesn't check if master initialized > > > Key: HBASE-8160 > URL: https://issues.apache.org/jira/browse/HBASE-8160 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 0.94-8160.patch, trunk-8160.patch > > > Before the master is initialized, if a move request comes in, it could cause > the master to abort, if the region was already in transition before the > master started up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8137) Add failed to open/close region state
[ https://issues.apache.org/jira/browse/HBASE-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8137: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add failed to open/close region state > - > > Key: HBASE-8137 > URL: https://issues.apache.org/jira/browse/HBASE-8137 > Project: HBase > Issue Type: Improvement > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-8137.patch, trunk-8137_v2.patch > > > Since we are going to remove assignment timeout monitor, we should add some > state to show if a region failed to open/close after max retries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index key
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609093#comment-13609093 ] Jean-Marc Spaggiari commented on HBASE-7845: [~xieliang007] I ran 10 times the test in Trunk without your patch. I'm now running with your patch. I also used HBASE-7953 to make sure tests are not failing. I will post the results today. > optimize hfile index key > > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 0.98.0 > > Attachments: HBASE-7845.txt, HBASE-7845-v2.txt, HBASE-7845-v3.txt, > HBASE-7845-v4.txt, HBASE-7845-v5.txt, HBASE-7845-v6.txt, HBASE-7845-v7.txt, > HBASE-7845-v8.txt > > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1
[ https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609099#comment-13609099 ] stack commented on HBASE-8165: -- [~andrew.purt...@gmail.com] Hadoop is not using 2.5. Doubt hadoop1 will ever get pb2.5. Ditto hadoop2. Regards protoc, yeah, if you were going to gen pb classes, you might be in awkward position where you would different protocs. You think this a killer (2.5 has some purported speedups). > Update our protobuf to 2.5 from 2.4.1 > - > > Key: HBASE-8165 > URL: https://issues.apache.org/jira/browse/HBASE-8165 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.95.0 > > Attachments: 8165.txt > > > Update to new 2.5 pb. Some speedups and a new PARSER idiom that bypasses > making a builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8024) Make Store flush algorithm pluggable
[
https://issues.apache.org/jira/browse/HBASE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609103#comment-13609103
]
Ted Yu commented on HBASE-8024:
---
Javadoc doesn't match actual parameter list:
{code}
+ * @param path The pathname of the tmp file into which the store was flushed
+ * @param logCacheFlushId
+ * @return StoreFile created.
+ * @throws IOException
+ */
+ public StoreFile commitStoreFile(final Path path,
{code}
{code}
+ * Default implementation of StoreFlusher.
+ */
+public class DefaultStoreFlusher implements StoreFlusher {
{code}
Add annotation for audience.
> Make Store flush algorithm pluggable
>
>
> Key: HBASE-8024
> URL: https://issues.apache.org/jira/browse/HBASE-8024
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 0.95.0, 0.96.0, 0.94.5
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Attachments: HBASE-8024-trunk.patch, HBASE-8024.v2.patch
>
>
> The idea is to make "StoreFlusher" an interface instead of an implementation
> class, and have the original StoreFlusher as the default store flush impl.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1
[ https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609105#comment-13609105 ] Ted Yu commented on HBASE-8165: --- Harsh opened HADOOP-9346 for Hadoop to compile with protoc 2.5 > Update our protobuf to 2.5 from 2.4.1 > - > > Key: HBASE-8165 > URL: https://issues.apache.org/jira/browse/HBASE-8165 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.95.0 > > Attachments: 8165.txt > > > Update to new 2.5 pb. Some speedups and a new PARSER idiom that bypasses > making a builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1
[ https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609107#comment-13609107 ] Andrew Purtell commented on HBASE-8165: --- bq. Regards protoc, yeah, if you were going to gen pb classes, you might be in awkward position where you would different protocs. You think this a killer (2.5 has some purported speedups). I worry about collisions between different versions of the protobuf library jar on the classpath wrt. HBase and the HDFS client. Needing distinct versions of protoc for building HBase and Hadoop is not a big deal (for me). > Update our protobuf to 2.5 from 2.4.1 > - > > Key: HBASE-8165 > URL: https://issues.apache.org/jira/browse/HBASE-8165 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.95.0 > > Attachments: 8165.txt > > > Update to new 2.5 pb. Some speedups and a new PARSER idiom that bypasses > making a builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1
[ https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609108#comment-13609108 ] Andrew Purtell commented on HBASE-8165: --- Hmm, HADOOP-9346 may indicate that if the PB 2.5 library is seen first on the classpath then code generated by 2.4 (e.g. HDFS client) will fail to link. > Update our protobuf to 2.5 from 2.4.1 > - > > Key: HBASE-8165 > URL: https://issues.apache.org/jira/browse/HBASE-8165 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.95.0 > > Attachments: 8165.txt > > > Update to new 2.5 pb. Some speedups and a new PARSER idiom that bypasses > making a builder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[ https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7953: --- Status: Open (was: Patch Available) > Some HFilePerformanceEvaluation tests might fail because of scanner.getValue > when there is no more rows > --- > > Key: HBASE-7953 > URL: https://issues.apache.org/jira/browse/HBASE-7953 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch, > HBASE-7953-v2-trunk.patch, HBASE-7953-v3-trunk.patch > > > There is 2 places in the code (Both trunk and 0.94) where we are trying to > get the row value even when scanner.next() returned null. Need to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[ https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7953: --- Attachment: HBASE-7953-v3-trunk.patch > Some HFilePerformanceEvaluation tests might fail because of scanner.getValue > when there is no more rows > --- > > Key: HBASE-7953 > URL: https://issues.apache.org/jira/browse/HBASE-7953 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch, > HBASE-7953-v2-trunk.patch, HBASE-7953-v3-trunk.patch > > > There is 2 places in the code (Both trunk and 0.94) where we are trying to > get the row value even when scanner.next() returned null. Need to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[ https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7953: --- Status: Patch Available (was: Open) Thanks for the comments Ted. I have also updated the existing System.out.printl() calls to use log instead. > Some HFilePerformanceEvaluation tests might fail because of scanner.getValue > when there is no more rows > --- > > Key: HBASE-7953 > URL: https://issues.apache.org/jira/browse/HBASE-7953 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch, > HBASE-7953-v2-trunk.patch, HBASE-7953-v3-trunk.patch > > > There is 2 places in the code (Both trunk and 0.94) where we are trying to > get the row value even when scanner.next() returned null. Need to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[
https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609146#comment-13609146
]
Hadoop QA commented on HBASE-7878:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574820/7878-trunk-v14.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4942//console
This message is automatically generated.
> recoverFileLease does not check return value of recoverLease
>
>
> Key: HBASE-7878
> URL: https://issues.apache.org/jira/browse/HBASE-7878
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Eric Newton
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2,
> 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt,
> 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt,
> 7878-trunk-v14.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt,
> 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt,
> 7878-trunk-v9.txt, 7878-trunk-v9.txt
>
>
> I think this is a problem, so I'm opening a ticket so an HBase person takes a
> look.
> Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease
> recovery for Accumulo after HBase's lease recovery. During testing, we
> experienced data loss. I found it is necessary to wait until recoverLease
> returns true to know that the file has been truly closed. In FSHDFSUtils,
> the return result of recoverLease is not checked. In the unit tests created
> to check lease recovery in HBASE-2645, the return result of recoverLease is
> always checked.
> I think FSHDFSUtils should be modified to check the return result, and wait
> until it returns true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[
https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609147#comment-13609147
]
Nick Dimiduk commented on HBASE-8158:
-
That would be ideal, yes. Trouble is, JarFinder is from hadoop-common-test,
which means it's likely not on the classpath for prod deployments. The comment
in that bit of code suggests it was originally introduced only for the benefit
of MRv2 test cases.
{noformat}
$ find . -name JarFinder.java
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java
{noformat}
> Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested
> inside another MR job"
> --
>
> Key: HBASE-8158
> URL: https://issues.apache.org/jira/browse/HBASE-8158
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 0.94.7
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 8158-port-jarfinder-0.94.patch,
> 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch
>
>
> Assess the stability of the JarFinder util class from Hadoop-2, as a
> candidate for backporting to 0.94.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7953) Some HFilePerformanceEvaluation tests might fail because of scanner.getValue when there is no more rows
[ https://issues.apache.org/jira/browse/HBASE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609161#comment-13609161 ] Ted Yu commented on HBASE-7953: --- Patch v3 looks good. > Some HFilePerformanceEvaluation tests might fail because of scanner.getValue > when there is no more rows > --- > > Key: HBASE-7953 > URL: https://issues.apache.org/jira/browse/HBASE-7953 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7953-v0-trunk.patch, HBASE-7953-v1-trunk.patch, > HBASE-7953-v2-trunk.patch, HBASE-7953-v3-trunk.patch > > > There is 2 places in the code (Both trunk and 0.94) where we are trying to > get the row value even when scanner.next() returned null. Need to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8163: -- Status: Patch Available (was: Open) > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-8163v1.patch, hbase-8163v2.patch > > > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609162#comment-13609162 ] ramkrishna.s.vasudevan commented on HBASE-7878: --- I went thro the patch and also the different errors that you got Ted. First thanks for your patience and persistence. https://issues.apache.org/jira/browse/HBASE-7878?focusedCommentId=13608535&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13608535 - what was the error that you faced here Ted? I can see that the juliet HLog creation should have failed the split right? Did the OrphanHLogAfterSplitException was thrown? I see that though the patch was simple now this JIRA is more of testcase correction. I just rebased the List to SEt for processedLog and corruptedLogs. See whether you are ok with it. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v15.patch, 7878-trunk-v2.txt, > 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, 7878-trunk-v6.txt, > 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-7878: -- Attachment: 7878-trunk-v15.patch > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v15.patch, 7878-trunk-v2.txt, > 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, 7878-trunk-v6.txt, > 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8160) HMaster#move doesn't check if master initialized
[ https://issues.apache.org/jira/browse/HBASE-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8160: --- Status: Open (was: Patch Available) > HMaster#move doesn't check if master initialized > > > Key: HBASE-8160 > URL: https://issues.apache.org/jira/browse/HBASE-8160 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 0.94-8160.patch, trunk-8160.patch > > > Before the master is initialized, if a move request comes in, it could cause > the master to abort, if the region was already in transition before the > master started up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8147) Add Integration Test for "The HCat Scenario"
[ https://issues.apache.org/jira/browse/HBASE-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-8147: --- Assignee: Nick Dimiduk > Add Integration Test for "The HCat Scenario" > > > Key: HBASE-8147 > URL: https://issues.apache.org/jira/browse/HBASE-8147 > Project: HBase > Issue Type: Test > Components: mapreduce, test >Affects Versions: 0.95.0, 0.98.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: HBASE-8147-WIP.patch > > > HBASE-8140 needs an integration test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609193#comment-13609193 ] Ted Yu commented on HBASE-7878: --- Do we need to use Set for processedLogs and corruptedLogs ? Each log would be either in processed or corrupted state. I think there wouldn't be duplicate entry in the List representation. bq. that the juliet HLog creation should have failed the split right? As I said above, that didn't happen due to check for orphan hlog missing the juliet HLog. bq. I see that though the patch was simple now this JIRA is more of testcase correction. I think my patch exposed different behavior in tests (w.r.t. ZombieNewLogWriterRegionServer) due to timing. I am quite certain that patch v14 would make tests consistently pass. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v15.patch, 7878-trunk-v2.txt, > 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, 7878-trunk-v6.txt, > 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7848) Use ZK-based read/write lock to make flush-type snapshot robust against table enable/disable/alter
[
https://issues.apache.org/jira/browse/HBASE-7848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609194#comment-13609194
]
Enis Soztutar commented on HBASE-7848:
--
bq. Looking at
https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableLockManager.java#L223
it seem that when interruptions and timeouts exceptions are thrown, the lock
could be taken but reported as an exception.
On timeout, we delete the lock znode already, ZKInterProcessLockBase, line 199.
In case of interruptions, I think we should ensure that the lock znode is
deleted before throwing the exception. This seems to be the model taken by
ReentrantLock in java. I'll open a jira for that.
bq. Can you give me a pointer to some tests that exercies the timeout and/or
interrupt path?
There are timeout tests in TestTableLockManager, and
TestZKInterProcessReadWriteLock.
bq. A quick solution may be to use the is a slight modification of the
canonical java lock pattern – put the acquire inside the try block
>From the javadoc
>http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReentrantLock.html,
> the canonical model is not to put acquire() in try, but rest of the code
>after acquire().
{code}
public void m() {
lock.lock(); // block until condition holds
try {
// ... method body
} finally {
lock.unlock()
}
}
{code}
Though you are right that we have to ensure lock is properly cleaned up on
Interruption. This has not been a problem, since currently only the server
shutdown interrupts the lock waiting, and zk cleans the znodes since they are
EPHEMERAL.
bq. HBASE-8131 modifies the way CreateTableHandler releases the lock. It was me
who changed that.
Yes, in CreateTableHandler, we should have guarded against the whole process(),
instead of the subcall handleCreateTable(). Your patch fixes the condition
where lock is acquired(). What Jonathan is pointing to is inside the acquire()
call.
> Use ZK-based read/write lock to make flush-type snapshot robust against table
> enable/disable/alter
> --
>
> Key: HBASE-7848
> URL: https://issues.apache.org/jira/browse/HBASE-7848
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 0.96.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 7848-v1.txt, 7848-v2.txt, hbase-7848_v2.patch
>
>
> Current region split following flush would fail snapshot.
> We can utilize ZK-based read/write lock to make flush-type snapshot robust
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5583: -- Attachment: HBASE-5583_new_1.patch > Master restart on create table with splitkeys does not recreate table with > all the splitkey regions > --- > > Key: HBASE-5583 > URL: https://issues.apache.org/jira/browse/HBASE-5583 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.95.0 > > Attachments: HBASE-5583_new_1.patch > > > -> Create table using splitkeys > -> MAster goes down before all regions are added to meta > -> On master restart the table is again enabled but with less number of > regions than specified in splitkeys > Anyway client will get an exception if i had called sync create table. But > table exists or not check will say table exists. > Is this scenario to be handled by client only or can we have some mechanism > on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8147) Add Integration Test for "The HCat Scenario"
[ https://issues.apache.org/jira/browse/HBASE-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8147: Issue Type: Sub-task (was: Test) Parent: HBASE-8084 > Add Integration Test for "The HCat Scenario" > > > Key: HBASE-8147 > URL: https://issues.apache.org/jira/browse/HBASE-8147 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, test >Affects Versions: 0.95.0, 0.98.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: HBASE-8147-WIP.patch > > > HBASE-8140 needs an integration test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index key
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609196#comment-13609196 ] Jean-Marc Spaggiari commented on HBASE-7845: Here we go! jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep Gaus 13/03/21 11:39:36 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 120260ms. 13/03/21 11:44:42 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119153ms. 13/03/21 11:49:49 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119066ms. 13/03/21 11:54:55 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 118865ms. 13/03/21 12:00:02 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119450ms. 13/03/21 12:05:11 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 122048ms. 13/03/21 12:10:14 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119197ms. 13/03/21 12:15:22 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119977ms. 13/03/21 12:20:32 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 121030ms. 13/03/21 12:25:38 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119880ms. jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep Gaus 13/03/21 12:56:07 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119765ms. 13/03/21 13:01:14 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119730ms. 13/03/21 13:06:20 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119945ms. 13/03/21 13:11:24 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119283ms. 13/03/21 13:16:29 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 118535ms. 13/03/21 13:21:30 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 117409ms. 13/03/21 13:26:37 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119458ms. 13/03/21 13:31:43 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119178ms. 13/03/21 13:36:48 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 118427ms. 13/03/21 13:41:55 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100 rows took 119580ms. jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep Write 13/03/21 11:35:42 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1307ms. 13/03/21 11:40:51 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1333ms. 13/03/21 11:45:58 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1328ms. 13/03/21 11:51:04 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1313ms. 13/03/21 11:56:09 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1302ms. 13/03/21 12:01:15 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1322ms. 13/03/21 12:06:23 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1304ms. 13/03/21 12:11:29 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1299ms. 13/03/21 12:16:37 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1329ms. 13/03/21 12:21:45 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1354ms. jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep Write 13/03/21 12:52:14 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1261ms. 13/03/21 12:57:21 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1286ms. 13/03/21 13:02:27 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1338ms. 13/03/21 13:07:33 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1309ms. 13/03/21 13:12:39 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 1317ms. 13/03/21 13:17:41 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100 rows took 130
[jira] [Commented] (HBASE-7845) optimize hfile index key
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609206#comment-13609206 ] Ted Yu commented on HBASE-7845: --- Thanks Jean-Marc for doing the comparison. Based on the above, I give +1 for Liang's patch. > optimize hfile index key > > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 0.98.0 > > Attachments: HBASE-7845.txt, HBASE-7845-v2.txt, HBASE-7845-v3.txt, > HBASE-7845-v4.txt, HBASE-7845-v5.txt, HBASE-7845-v6.txt, HBASE-7845-v7.txt, > HBASE-7845-v8.txt > > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7995) Export$Exporter could be replaced with IdentityTableMapper
[
https://issues.apache.org/jira/browse/HBASE-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609209#comment-13609209
]
Nick Dimiduk commented on HBASE-7995:
-
bump.
> Export$Exporter could be replaced with IdentityTableMapper
> --
>
> Key: HBASE-7995
> URL: https://issues.apache.org/jira/browse/HBASE-7995
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 0.95.0, 0.98.0
>
> Attachments:
> 0001-HBASE-7995-Replace-Export-Exporter-with-IdentityTabl.patch
>
>
> The Mapper implementation in Export looks identical to IdentityTableMapper,
> except for the (poor) exception handling. I'd like to consolidate on
> IdentityTableMapper, but I'm looking for some historical perspective before
> doing so (cc [~stack])
> Export$Exporter's map method:
> {code}
> public void map(ImmutableBytesWritable row, Result value, Context context)
> throws IOException {
> try {
> context.write(row, value);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> }
> {code}
> IdentityTableMapper's map method:
> {code}
> public void map(ImmutableBytesWritable key, Result value, Context context)
> throws IOException, InterruptedException {
> context.write(key, value);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6739) Single put should avoid batch overhead when autoflush is on
[ https://issues.apache.org/jira/browse/HBASE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-6739: --- Assignee: (was: Nick Dimiduk) > Single put should avoid batch overhead when autoflush is on > > > Key: HBASE-6739 > URL: https://issues.apache.org/jira/browse/HBASE-6739 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Priority: Minor > > Currently, even when autoflush is on, a single put is handled the same way as > if autoflush is off: convert the put to multi-action, create a callable, hand > it to an executor to process, wait for it to complete. > We can avoid this overhead for single put if autoflush is on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609212#comment-13609212 ] ramkrishna.s.vasudevan commented on HBASE-5583: --- I am attaching a patch here with the current state of work. This is not the final one. Still Exception handling and testing is pending. I would like to get myself clarified whether am going in the right direction. My first question would be -> In order to maintain different states at which we are in while creating table should we go for a special node under the table znode or can we add new states to the existing table states like (DISABLING, ENABLING, DISABLED etc).? BAsed on this i can rebase the patch. Also pls note that the ZooKeeperProto.java is showing as a complete change with out which this patch should be comparitively smaller. -> This patch basically tries to handle table creation on master failure and restart. -> Adds a new state CREATINGTABLE instead of ENABLING so that we know that this table is partially created. -> Adds a status node under /table to maintain various states of the table creation process (is this seperate node needed or not is my doubt). -> Table lock related things needs to be checked. Even after we do all this HBaseAdmin.isTableAvailable(tableName, splitKeys) should be used by the client to check if all the regions are created. Feed back is appreciated. I can do more testing and handle ZK exceptions in a better way in my further patches. Thanks. > Master restart on create table with splitkeys does not recreate table with > all the splitkey regions > --- > > Key: HBASE-5583 > URL: https://issues.apache.org/jira/browse/HBASE-5583 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.95.0 > > Attachments: HBASE-5583_new_1.patch > > > -> Create table using splitkeys > -> MAster goes down before all regions are added to meta > -> On master restart the table is again enabled but with less number of > regions than specified in splitkeys > Anyway client will get an exception if i had called sync create table. But > table exists or not check will say table exists. > Is this scenario to be handled by client only or can we have some mechanism > on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index key
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609226#comment-13609226 ] ramkrishna.s.vasudevan commented on HBASE-7845: --- JM , thanks for your work. > optimize hfile index key > > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 0.98.0 > > Attachments: HBASE-7845.txt, HBASE-7845-v2.txt, HBASE-7845-v3.txt, > HBASE-7845-v4.txt, HBASE-7845-v5.txt, HBASE-7845-v6.txt, HBASE-7845-v7.txt, > HBASE-7845-v8.txt > > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8137) Add failed to open/close region state
[ https://issues.apache.org/jira/browse/HBASE-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609227#comment-13609227 ] Hudson commented on HBASE-8137: --- Integrated in HBase-TRUNK #3981 (See [https://builds.apache.org/job/HBase-TRUNK/3981/]) HBASE-8137 Add failed to open/close region state (Revision 1459384) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterStatusProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/ClusterStatus.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/Mocking.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java > Add failed to open/close region state > - > > Key: HBASE-8137 > URL: https://issues.apache.org/jira/browse/HBASE-8137 > Project: HBase > Issue Type: Improvement > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-8137.patch, trunk-8137_v2.patch > > > Since we are going to remove assignment timeout monitor, we should add some > state to show if a region failed to open/close after max retries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609228#comment-13609228 ] ramkrishna.s.vasudevan commented on HBASE-7878: --- bq.I am quite certain that patch v14 would make tests consistently pass. I agree. Changing to Set from List is upto you. Just thought would help you from adding all the files once and again removing it in two for loops after that. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt, > 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt, > 7878-trunk-v14.txt, 7878-trunk-v15.patch, 7878-trunk-v2.txt, > 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, 7878-trunk-v6.txt, > 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8169) TestMaster#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions may randomly be assigned to a RS
Jeffrey Zhong created HBASE-8169: Summary: TestMaster#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions may randomly be assigned to a RS Key: HBASE-8169 URL: https://issues.apache.org/jira/browse/HBASE-8169 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.94.7 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.94.7 In the test case, we use "master.assignRegion(hri);" to assign regions to live RS or dead RS(will be aborted during the test). While master.assignRegion may not always successfully assign a region to the expected RS due to following error: 1) We firstly try to open a region in region server, let's say RS1 2) For some reason the region open takes more than a time out period 3) AM tries to reassign it and force a new plan due to the time out 4) the Region is re-assigned to a random RS Then it cause the test case failure due to regions setup isn't in expected state. I attached a log with extra added logging info and some stack traces(with (new Exception()).printStackTrace() so you can ignore java.lang.Exception). You can search region 36716be644bc9c78b3a96e0d4a43c183 to see master.assignRegion assigned a region to a randome RS. I saw the test failed once in 5 times in a loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[
https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609237#comment-13609237
]
Anoop Sam John commented on HBASE-5583:
---
Ram
Started going through the patch. It looks so big change but is it
really? There generated file is fully rewritten!
ZooKeeper.proto
{code}
ENABLING = 3;
+CREATINGTABLE = 4;
{code}
Can we make CREATINGTABLE -> CREATING?
{code}
+ enum State {
+CREATED_TD = 0;
+CREATED_REGIONINFO = 1;
+MOVED_TO_ORIG_LOCATION = 2;
+ADD_TO_META = 3;
+ASSIGN_USER_REGIONS = 4;
+DONE = 4;
+ }
{code}
Repeated ordinal. Typo?
Will continue..
> Master restart on create table with splitkeys does not recreate table with
> all the splitkey regions
> ---
>
> Key: HBASE-5583
> URL: https://issues.apache.org/jira/browse/HBASE-5583
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.0
>
> Attachments: HBASE-5583_new_1.patch
>
>
> -> Create table using splitkeys
> -> MAster goes down before all regions are added to meta
> -> On master restart the table is again enabled but with less number of
> regions than specified in splitkeys
> Anyway client will get an exception if i had called sync create table. But
> table exists or not check will say table exists.
> Is this scenario to be handled by client only or can we have some mechanism
> on the master side for this? Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8169) TestMaster#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions may randomly be assigned to a RS
[ https://issues.apache.org/jira/browse/HBASE-8169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8169: - Attachment: RegionAssignedRandomly.txt > TestMaster#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions > may randomly be assigned to a RS > --- > > Key: HBASE-8169 > URL: https://issues.apache.org/jira/browse/HBASE-8169 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.98.0, 0.94.7 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong >Priority: Minor > Fix For: 0.94.7 > > Attachments: RegionAssignedRandomly.txt > > > In the test case, we use "master.assignRegion(hri);" to assign regions to > live RS or dead RS(will be aborted during the test). While > master.assignRegion may not always successfully assign a region to the > expected RS due to following error: > 1) We firstly try to open a region in region server, let's say RS1 > 2) For some reason the region open takes more than a time out period > 3) AM tries to reassign it and force a new plan due to the time out > 4) the Region is re-assigned to a random RS > Then it cause the test case failure due to regions setup isn't in expected > state. > I attached a log with extra added logging info and some stack traces(with > (new Exception()).printStackTrace() so you can ignore java.lang.Exception). > You can search region 36716be644bc9c78b3a96e0d4a43c183 to see > master.assignRegion assigned a region to a randome RS. I saw the test failed > once in 5 times in a loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609245#comment-13609245 ] Anoop Sam John commented on HBASE-5583: --- bq.In order to maintain different states at which we are in while creating table should we go for a special node under the table znode or can we add new states to the existing table states like (DISABLING, ENABLING, DISABLED etc) IMO go with separate node. > Master restart on create table with splitkeys does not recreate table with > all the splitkey regions > --- > > Key: HBASE-5583 > URL: https://issues.apache.org/jira/browse/HBASE-5583 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.95.0 > > Attachments: HBASE-5583_new_1.patch > > > -> Create table using splitkeys > -> MAster goes down before all regions are added to meta > -> On master restart the table is again enabled but with less number of > regions than specified in splitkeys > Anyway client will get an exception if i had called sync create table. But > table exists or not check will say table exists. > Is this scenario to be handled by client only or can we have some mechanism > on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8151) Decode memstoreTS in HFileReaderV2 only when necessary
[ https://issues.apache.org/jira/browse/HBASE-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-8151. -- Resolution: Fixed Hadoop Flags: Reviewed Committed the first patch (handling #1) to 0.94, 0.95, and 0.96. > Decode memstoreTS in HFileReaderV2 only when necessary > -- > > Key: HBASE-8151 > URL: https://issues.apache.org/jira/browse/HBASE-8151 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8151-0.94.txt, 8151-0.94-v2.txt, 8151-0.96.txt, > 8151-0.96-v2.txt, 8151-0.96-v3.txt > > > HFiles V2 store the memstoreTS of each KV. > In many cases all the KVs in an HFile will have a memstoreTS of 0 (that is > the case when at the time the HFile was written there are no KVs that were > created after the oldest still active scanner - which is frequently the case). > In that case we: > # do not need to decode the memstoreTS (a vlong), since we know its value is > 0 and its length is 1 byte. > # when we compact HFiles and all of the involved files have only KVs with > memstoreTS = 0 we know ahead of time that all KVs meet this condition and we > do not need to store the memstoreTS in the new HFile. > This issue will cover the first part. The performance improvement will be > modest as it is fairly cheap to decode vlongs of size 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8169) TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions randomly assigned to a RS
[ https://issues.apache.org/jira/browse/HBASE-8169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8169: -- Summary: TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions randomly assigned to a RS (was: TestMaster#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions may randomly be assigned to a RS) > TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS may fail due to > regions randomly assigned to a RS > > > Key: HBASE-8169 > URL: https://issues.apache.org/jira/browse/HBASE-8169 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.98.0, 0.94.7 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong >Priority: Minor > Fix For: 0.94.7 > > Attachments: hbase-8169.patch, RegionAssignedRandomly.txt > > > In the test case, we use "master.assignRegion(hri);" to assign regions to > live RS or dead RS(will be aborted during the test). While > master.assignRegion may not always successfully assign a region to the > expected RS due to following error: > 1) We firstly try to open a region in region server, let's say RS1 > 2) For some reason the region open takes more than a time out period > 3) AM tries to reassign it and force a new plan due to the time out > 4) the Region is re-assigned to a random RS > Then it cause the test case failure due to regions setup isn't in expected > state. > I attached a log with extra added logging info and some stack traces(with > (new Exception()).printStackTrace() so you can ignore java.lang.Exception). > You can search region 36716be644bc9c78b3a96e0d4a43c183 to see > master.assignRegion assigned a region to a randome RS. I saw the test failed > once in 5 times in a loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8169) TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS may fail due to regions randomly assigned to a RS
[ https://issues.apache.org/jira/browse/HBASE-8169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8169: - Attachment: hbase-8169.patch I think a region can be assigned randomly by design in a time out situation so I came up a patch for the issue in case someone else also stumble on this. > TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS may fail due to > regions randomly assigned to a RS > > > Key: HBASE-8169 > URL: https://issues.apache.org/jira/browse/HBASE-8169 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.98.0, 0.94.7 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong >Priority: Minor > Fix For: 0.94.7 > > Attachments: hbase-8169.patch, RegionAssignedRandomly.txt > > > In the test case, we use "master.assignRegion(hri);" to assign regions to > live RS or dead RS(will be aborted during the test). While > master.assignRegion may not always successfully assign a region to the > expected RS due to following error: > 1) We firstly try to open a region in region server, let's say RS1 > 2) For some reason the region open takes more than a time out period > 3) AM tries to reassign it and force a new plan due to the time out > 4) the Region is re-assigned to a random RS > Then it cause the test case failure due to regions setup isn't in expected > state. > I attached a log with extra added logging info and some stack traces(with > (new Exception()).printStackTrace() so you can ignore java.lang.Exception). > You can search region 36716be644bc9c78b3a96e0d4a43c183 to see > master.assignRegion assigned a region to a randome RS. I saw the test failed > once in 5 times in a loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[ https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8166: - Attachment: 8166-0.94.txt 0.94 patch (same patch as -v2 in HBASE-8155, with the already committed part removed) > Avoid writing the memstoreTS into HFiles when possible > -- > > Key: HBASE-8166 > URL: https://issues.apache.org/jira/browse/HBASE-8166 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8166-0.94.txt > > > See discussion in HBASE-8151. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609258#comment-13609258 ] Himanshu Vashishtha commented on HBASE-8028: Ping... :) > Append, Increment: Adding rollback support > -- > > Key: HBASE-8028 > URL: https://issues.apache.org/jira/browse/HBASE-8028 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0 > > Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, > HBase-8028-with-Increments-v1.patch > > > In case there is an exception while doing the log-sync, the memstore is not > rollbacked, while the mvcc is _always_ forwarded to the writeentry created at > the beginning of the operation. This may lead to scanners seeing results > which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[ https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8166: - Attachment: 8166-0.96.txt Same for the 0.96 version. > Avoid writing the memstoreTS into HFiles when possible > -- > > Key: HBASE-8166 > URL: https://issues.apache.org/jira/browse/HBASE-8166 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8166-0.94.txt, 8166-0.96.txt > > > See discussion in HBASE-8151. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[ https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8166: - Description: See discussion in HBASE-8151. Upon compaction we can check all the involved HFiles. If all of them have a MAX_MEMSTORE_TS < the smallest scannner readpoint, we create a writer that does not write the memstoreTS. was:See discussion in HBASE-8151. > Avoid writing the memstoreTS into HFiles when possible > -- > > Key: HBASE-8166 > URL: https://issues.apache.org/jira/browse/HBASE-8166 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8166-0.94.txt, 8166-0.96.txt > > > See discussion in HBASE-8151. > Upon compaction we can check all the involved HFiles. If all of them have a > MAX_MEMSTORE_TS < the smallest scannner readpoint, we create a writer that > does not write the memstoreTS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[ https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8166: - Status: Patch Available (was: Open) > Avoid writing the memstoreTS into HFiles when possible > -- > > Key: HBASE-8166 > URL: https://issues.apache.org/jira/browse/HBASE-8166 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8166-0.94.txt, 8166-0.96.txt > > > See discussion in HBASE-8151. > Upon compaction we can check all the involved HFiles. If all of them have a > MAX_MEMSTORE_TS < the smallest scannner readpoint, we create a writer that > does not write the memstoreTS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609265#comment-13609265
]
Jeffrey Zhong commented on HBASE-8127:
--
[~rajesh23] If you run the test in a loop, you most likely will see it fail
about once in 5 times due to the reason I logged in
https://issues.apache.org/jira/browse/HBASE-8169.
After I apply your fix + repro_hang + the fix of hbase-8169 on the test case, I
was able to run the test successfully 20 times in a loop.(I saw it failed once
due to RS could not start, which I think it's normal to other test cases as
well).
If you like, you can reassign the JIRA to yourself.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609273#comment-13609273 ] Sergey Shelukhin commented on HBASE-8163: - some r comments > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-8163v1.patch, hbase-8163v2.patch > > > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609276#comment-13609276 ] Sergey Shelukhin commented on HBASE-7055: - Hmm... I don't think it makes sense to put description of all config params into release notes. There's separate doc in parent jira for that, we can add it to comments or book. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes) > -- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.95.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch, > HBASE-6371-v4-refactor-only-squashed.patch, > HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, > HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, > HBASE-7055-v4.patch, HBASE-7055-v5.patch, HBASE-7055-v6.patch, > HBASE-7055-v7.patch, HBASE-7055-v7.patch > > > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[
https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609285#comment-13609285
]
Ted Yu commented on HBASE-7878:
---
I see.
{code}
+ if (currFiles.length > processedLogs.size()
+ corruptedLogs.size()) {
{code}
Since the for loops are in the above if block which would lead to
OrphanHLogAfterSplitException being thrown, I think the cost at runtime
wouldn't be noticeable.
> recoverFileLease does not check return value of recoverLease
>
>
> Key: HBASE-7878
> URL: https://issues.apache.org/jira/browse/HBASE-7878
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Eric Newton
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2,
> 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt,
> 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt,
> 7878-trunk-v14.txt, 7878-trunk-v15.patch, 7878-trunk-v2.txt,
> 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, 7878-trunk-v6.txt,
> 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, 7878-trunk-v9.txt
>
>
> I think this is a problem, so I'm opening a ticket so an HBase person takes a
> look.
> Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease
> recovery for Accumulo after HBase's lease recovery. During testing, we
> experienced data loss. I found it is necessary to wait until recoverLease
> returns true to know that the file has been truly closed. In FSHDFSUtils,
> the return result of recoverLease is not checked. In the unit tests created
> to check lease recovery in HBASE-2645, the return result of recoverLease is
> always checked.
> I think FSHDFSUtils should be modified to check the return result, and wait
> until it returns true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
Edward C. Skoviak created HBASE-8170: Summary: HbaseAdmin.createTable cannot handle creating three regions Key: HBASE-8170 URL: https://issues.apache.org/jira/browse/HBASE-8170 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Edward C. Skoviak Priority: Minor createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int numRegions) (line #370) dictates that you must specify a minimum of three regions, however is not able to handle being fed a value three. This is a result of line #379 where it attempts to create the key splits, and calls Bytes.Split with a value of 0 for the third parameter. createTable should instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8171) Perf issue with multiple client threads trying to prefetch Region Cache from META
Amitanand Aiyer created HBASE-8171: -- Summary: Perf issue with multiple client threads trying to prefetch Region Cache from META Key: HBASE-8171 URL: https://issues.apache.org/jira/browse/HBASE-8171 Project: HBase Issue Type: Bug Affects Versions: 0.94.5, 0.94.3, 0.89-fb Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer HBaseClient maintains a cache of Regions to region locations that gets cleared whenever the RS that the client talks to is unresponsive. This will result in the Client having to talk to META to fetch the region location before doing the get/put. https://issues.apache.org/jira/browse/HBASE-2468 introduces the feature to prewarm the cache, where the client gets some rows from META to warm the cache. This is a costly operation. And, it looks like all client threads that block on the regionLockObject, will go in and do the cache warmup; although the cache might have been already warmed up by an earlier thread. When there is a network issue going on, we see lots of Client threads blocking in locateRegionInMeta. The fix would be to check the cache again; before we do the prefetch after grabbing the lock. Currently, the client threads look in the cache at the begining of the method. But a lot could have changed while waiting for the regionLockObject. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7533) Write an RPC Specification for 0.96
[ https://issues.apache.org/jira/browse/HBASE-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7533: - Attachment: addendum.txt Addendum that comes of review comments made by Jon Hsieh up on the google docs (thanks Jon). > Write an RPC Specification for 0.96 > --- > > Key: HBASE-7533 > URL: https://issues.apache.org/jira/browse/HBASE-7533 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 0.95.0 > > Attachments: 7533_proto_sketch.txt, 7533.txt, 7533v4.txt, > addendum.txt, rpc_spec3.txt, rpc_spec3.txt, rpc_spec.txt > > > RPC format is changing for 0.96 to accomodate our protobufing all around. > Here is a first cut. Please shred: > https://docs.google.com/document/d/1-1RJMLXzYldmHgKP7M7ynK6euRpucD03fZ603DlZfGI/edit -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609291#comment-13609291 ] Jean-Marc Spaggiari commented on HBASE-8170: Sound like the split should use numRegions - 1 instead of numRegions - 3... Have you tried with other numbers, like 4 regions or more? Are you going to provide a patch? > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Priority: Minor > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7517) Preemptive fast fail exception should not be processed multiple times in the stack
[ https://issues.apache.org/jira/browse/HBASE-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-7517: --- Description: Preemptive fast fail exception is a mechanism to prevent client threads from blocking on talking to a unresponsive server. This was introduced in HBASE-6629. However, if a PFFE is thrown from an inner function (say locateRegionInMeta), then the outer functions should not process this exception as a failure in talking to the RS. We should just percolate this exception back to the client. > Preemptive fast fail exception should not be processed multiple times in the > stack > -- > > Key: HBASE-7517 > URL: https://issues.apache.org/jira/browse/HBASE-7517 > Project: HBase > Issue Type: Bug >Reporter: Amitanand Aiyer >Priority: Minor > > Preemptive fast fail exception is a mechanism to prevent client threads from > blocking on talking to a unresponsive server. > This was introduced in HBASE-6629. > However, if a PFFE is thrown from an inner function (say locateRegionInMeta), > then the outer functions should not process this exception as a failure in > talking to the RS. We should just percolate this exception back to the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[
https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609306#comment-13609306
]
Hadoop QA commented on HBASE-7878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574836/7878-trunk-v15.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestRollingRestart
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4945//console
This message is automatically generated.
> recoverFileLease does not check return value of recoverLease
>
>
> Key: HBASE-7878
> URL: https://issues.apache.org/jira/browse/HBASE-7878
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Eric Newton
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2,
> 7878-trunk.addendum, 7878-trunk.addendum2, 7878-trunk-v10.txt,
> 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, 7878-trunk-v13.txt,
> 7878-trunk-v14.txt, 7878-trunk-v15.patch, 7878-trunk-v2.txt,
> 7878-trunk-v3.txt, 7878-trunk-v4.txt, 7878-trunk-v5.txt, 7878-trunk-v6.txt,
> 7878-trunk-v7.txt, 7878-trunk-v8.txt, 7878-trunk-v9.txt, 7878-trunk-v9.txt
>
>
> I think this is a problem, so I'm opening a ticket so an HBase person takes a
> look.
> Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease
> recovery for Accumulo after HBase's lease recovery. During testing, we
> experienced data loss. I found it is necessary to wait until recoverLease
> returns true to know that the file has been truly closed. In FSHDFSUtils,
> the return result of recoverLease is not checked. In the unit tests created
> to check lease recovery in HBASE-2645, the return result of recoverLease is
> always checked.
> I think FSHDFSUtils should be modified to check the return result, and wait
> until it returns true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[
https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609308#comment-13609308
]
Hadoop QA commented on HBASE-8163:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574757/hbase-8163v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testHbckFixOrphanTable(TestHBaseFsck.java:427)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4944//console
This message is automatically generated.
> MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
> --
>
> Key: HBASE-8163
> URL: https://issues.apache.org/jira/browse/HBASE-8163
> Project: HBase
> Issue Type: New Feature
> Components: regionserver
>Reporter: chunhui shen
>Assignee: chunhui shen
> Attachments: hbase-8163v1.patch, hbase-8163v2.patch
>
>
> *Background*：
> When we use mslab,we will copy the keyvalue together in a structure called
> *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment.
> *Problem*：
> When one chunk is full, we will create a new chunk, and then the old chunk
> will be reclaimed by JVM if no reference to it.
> Mostly the chunk object would be promoted when doing Young GC, cause
> increasing the cost of YGC
> When a Chunk object has no reference? It should meet two following condition:
> 1.Memstore which this chunk belongs to is flushed
> 2.No scanner is opening on the memstore which this chunk belongs to
> *Solution:*
> 1.Create a chunk pool to manage the no-reference chunks, instead of being
> reclaimed by JVM
> 2.When a Chunk has no reference, put it back to the pool
> 3.The pool has a max capacity, it will skip the chunks when achieve the max
> size
> 4.When we need new Chunk to store KeyValue, get it from the pool if exists,
> else create new one by pool, so we could be able to reuse the old chunks
> *Test results:*
> Environment:
> hbase-version:0.94
> -Xms4G -Xmx4G -Xmn2G
> Row size=50 bytes, Value size=1024 bytes
> 50 concurrent theads per client, insert 10,000,000 rows
> Before:
> Avg write request per second:12953
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT
> 747 36.503 48 2.492 38.995
> After:
> Avg write request per second:14025
> After testing, final result of jstat -gcutil :
> YGC YGCT FGC FGCT GCT
> 711 20.344 4 0.284 20
[jira] [Commented] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609311#comment-13609311 ] Jean-Marc Spaggiari commented on HBASE-8170: I just did some tests, and split returns n+2 splits. So to get 3 splits, we need to ask for 1 and not for 0. So numRegions - 2 should be passed. > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Priority: Minor > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8137) Add failed to open/close region state
[ https://issues.apache.org/jira/browse/HBASE-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609320#comment-13609320 ] Hudson commented on HBASE-8137: --- Integrated in hbase-0.95 #93 (See [https://builds.apache.org/job/hbase-0.95/93/]) HBASE-8137 Add failed to open/close region state (Revision 1459385) Result = FAILURE jxiang : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterStatusProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/ClusterStatus.proto * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/Mocking.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java > Add failed to open/close region state > - > > Key: HBASE-8137 > URL: https://issues.apache.org/jira/browse/HBASE-8137 > Project: HBase > Issue Type: Improvement > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-8137.patch, trunk-8137_v2.patch > > > Since we are going to remove assignment timeout monitor, we should add some > state to show if a region failed to open/close after max retries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609321#comment-13609321 ] stack commented on HBASE-8163: -- [~zjushch] Can we guarantee that pool size is 0 when memstores are full or near full? [~lhofhansl] "If it is too big you'll waste space, if it is too small you'll just offset the GC for a bit." Agree that pool should not be above and beyond memstore allocation. Regards your comment that we just offset GC for a bit, don't we offset it period if successful reuse? > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: hbase-8163v1.patch, hbase-8163v2.patch > > > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index key
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609327#comment-13609327 ] stack commented on HBASE-7845: -- [~jmspaggi] Nice. Thanks. > optimize hfile index key > > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 0.98.0 > > Attachments: HBASE-7845.txt, HBASE-7845-v2.txt, HBASE-7845-v3.txt, > HBASE-7845-v4.txt, HBASE-7845-v5.txt, HBASE-7845-v6.txt, HBASE-7845-v7.txt, > HBASE-7845-v8.txt > > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609329#comment-13609329 ] Edward C. Skoviak commented on HBASE-8170: -- Perhaps I am wrong, but as I understand it each splitKey defines the border where the split happens. So in this scenario the three regions would be everything before startKey, startKey to endKey, and then everything after endKey. The other createTable() function, which is called at the end of this function, works that way. This can be verified by calling it with two keys and calling HTable.getRegionLocations().size() and getting a return value of 3. So if we did numRegions - 2 we would always be getting an additional region (so in this case, 4). > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Priority: Minor > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Assignee: rajeshbabu (was: Jeffrey Zhong)
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[
https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609332#comment-13609332
]
Edward C. Skoviak commented on HBASE-8170:
--
This is confirmed here[1]. {quote}The start key specified will become the end
key of the first region of the table, and the end key specified will become the
start key of the last region of the table (the first region has a null start
key and the last region has a null end key). BigInteger math will be used to
divide the key range specified into enough segments to make the required number
of total regions.{quote}
[1]
"http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#createTable(org.apache.hadoop.hbase.HTableDescriptor,
byte[], byte[], int)"
> HbaseAdmin.createTable cannot handle creating three regions
> ---
>
> Key: HBASE-8170
> URL: https://issues.apache.org/jira/browse/HBASE-8170
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 0.94.5
>Reporter: Edward C. Skoviak
>Priority: Minor
>
> createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int
> numRegions) (line #370) dictates that you must specify a minimum of three
> regions, however is not able to handle being fed a value three. This is a
> result of line #379 where it attempts to create the key splits, and calls
> Bytes.Split with a value of 0 for the third parameter. createTable should
> instead just create a byte[][] with the startKey and endKey in this scenario.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index key
[
https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609345#comment-13609345
]
stack commented on HBASE-7845:
--
[~xieliang007] I don't think you answered all my review questions above.
Thanks.
getFakedKey is a bad name for what you are about. How about
getShortMidpointKey? Also, in the comment, say what the method is trying to
achieve. You point at the issue but that is being a little (lazy) and besides
the issue subject is 'optimize' which is pretty general.
Do you need to do this copy?
+ if (leftKey == null) {
+return Arrays.copyOf(rightKey, rightKey.length);
+ }
I suppose it is safer.
This define should be in HFile rather than in HConstants so it is beside where
it is used:
HConstants.FAKE_INDEX_KEY_MAGIC
Also, 'FAKE' has negative connotations whereas what you are doing here is
clever in a good way; would suggest you drop use of 'fake' and just say
INDEX_KEY_MAGIC (ditto for method names).
Why this now:
{code}
- // Expected block reads: 2. [HBASE-4443]
- kvs = getData(FAMILY, "row", Arrays.asList("col5"), 2);
+ // Expected block reads: 1. [HBASE-4443]&[HBASE-7845]
+ kvs = getData(FAMILY, "row", Arrays.asList("col5"), 1);
{code}
You have a good few tests of new functionality. That is good. I'd think we
should try this as part of a long-running job before commit. I can volunteer
to do that...
> optimize hfile index key
>
>
> Key: HBASE-7845
> URL: https://issues.apache.org/jira/browse/HBASE-7845
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 0.98.0
>
> Attachments: HBASE-7845.txt, HBASE-7845-v2.txt, HBASE-7845-v3.txt,
> HBASE-7845-v4.txt, HBASE-7845-v5.txt, HBASE-7845-v6.txt, HBASE-7845-v7.txt,
> HBASE-7845-v8.txt
>
>
> Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() &
> FindShortSuccessor() to reduce index key size, it would be helpful under
> special conditions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[
https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609362#comment-13609362
]
Hadoop QA commented on HBASE-8166:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574846/8166-0.96.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4946//console
This message is automatically generated.
> Avoid writing the memstoreTS into HFiles when possible
> --
>
> Key: HBASE-8166
> URL: https://issues.apache.org/jira/browse/HBASE-8166
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 8166-0.94.txt, 8166-0.96.txt
>
>
> See discussion in HBASE-8151.
> Upon compaction we can check all the involved HFiles. If all of them have a
> MAX_MEMSTORE_TS < the smallest scannner readpoint, we create a writer that
> does not write the memstoreTS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[
https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609373#comment-13609373
]
Jean-Marc Spaggiari commented on HBASE-8170:
The method you are calling is using the Bytes.split function to split the keys
and then is calling HBaseAdmin.createTable(final HTableDescriptor desc, byte
[][] splitKeys)
In this method you can see this in the documentation: The total number of
regions created will be the number of split keys plus one.
Which mean that each key you are passing is defining the end or/and the
beginning of a region, without any overlap.
As an example:
{code}
byte[] a = "a".getBytes();
byte[] b = "z".getBytes();
byte[][] splits = Bytes.split(a, b, 1);
{code}
Will give you 3 splits. "a", "m" and "z".
If you give that to createTable, that will give you 4 regions. Something like
#00 to "a", "a" to "m", "m" to "z" and finally "z" to #ff. (but without the
overlaps).
So if you want to have only 3 regions, you need to call split (a, b, 0). Which
is why it's calling it with numRegions - 3.
But Bytes.split is not working with 0. So we have 2 options here.
1) We change (numRegions < 3) by (numRegions < 4) to force at least 4 regions.
That way split is called with at least 1.
2) We implement a special scenario when numRegions = 3 and use the 2 keys
Something like that:
{code}
public void createTable(HTableDescriptor desc, byte [] startKey,
byte [] endKey, int numRegions)
throws IOException {
HTableDescriptor.isLegalTableName(desc.getName());
if(numRegions < 3) {
throw new IllegalArgumentException("Must create at least three regions");
} else if(Bytes.compareTo(startKey, endKey) >= 0) {
throw new IllegalArgumentException("Start key must be smaller than end
key");
}
if (numRegions == 3) {
createTable(desc, new byte[][]{startKey, endKey});
return;
}
byte [][] splitKeys = Bytes.split(startKey, endKey, numRegions - 3);
if(splitKeys == null || splitKeys.length != numRegions - 1) {
throw new IllegalArgumentException("Unable to split key range into enough
regions");
}
createTable(desc, splitKeys);
}
{code}
> HbaseAdmin.createTable cannot handle creating three regions
> ---
>
> Key: HBASE-8170
> URL: https://issues.apache.org/jira/browse/HBASE-8170
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 0.94.5
>Reporter: Edward C. Skoviak
>Priority: Minor
>
> createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int
> numRegions) (line #370) dictates that you must specify a minimum of three
> regions, however is not able to handle being fed a value three. This is a
> result of line #379 where it attempts to create the key splits, and calls
> Bytes.Split with a value of 0 for the third parameter. createTable should
> instead just create a byte[][] with the startKey and endKey in this scenario.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8156) Support for Namenode HA for non-idempotent operations
[
https://issues.apache.org/jira/browse/HBASE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609375#comment-13609375
]
stack commented on HBASE-8156:
--
You do not check the return here:
+createDir(storeDir);
return storeDir;
Is it because it will either return true or throw an exception? Never false?
Stuff will run slower now we are doing more trips to NN I suppose:
+if (!fs.exists(familyDir)) return;
+if(!deleteDir(familyDir))
e.g. the addition of the exists check above (Not the fault of this patch...
just saying).
Again we do not check the return. Should we?
-fs.mkdirs(storeDir);
+createDir(storeDir);
Here the model is a little different in that we check the return and if not
what we want, then we throw exception:
-if (!fs.rename(buildPath, dstPath)) {
+
+if (!rename(buildPath, dstPath)) {
I suppose the createDir just throws an exception if it fails? That is probably
ok but different shape on these methods.
Yeah, do you need doc on this method saying how it works?
+ boolean createDir(Path dir) throws IOException {
You ahve this:
+ * @return
I'd think this'd generate a javadoc warning but doesn't look like it does.
Want to fill in what it means on rename, etc?
rename has doc. Others don't. Be consistent I'd say.
I don't think you need these kinda banners:
+ //
===
+ // Non-Idempotent calls to DFS, helpers for static calls
+ //
===
They tend to get ignored and methods that don't belong get inserted w/o
updating the banner scope
You have two sleepBeforeRetry ? Shouldn't one call the other? Why have two at
all?
Patch is looking good. Lets get it in.
> Support for Namenode HA for non-idempotent operations
> -
>
> Key: HBASE-8156
> URL: https://issues.apache.org/jira/browse/HBASE-8156
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Affects Versions: 0.95.0
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.98.0
>
> Attachments: HBase-8156-trunk-v1.patch, HBase-8156-trunk-v2.patch
>
>
> In hadoop 2 HA, non-idempotent operations are not retried at the hdfs side.
> This is by design as retrying a non-idempotent operation might not be a good
> design choice for some use case.
> HBase needs to handle the retries for such operations at its end.
> With HBase-7806, there is already some work going on for file system
> abstractions. There, HReginFileSystem sits as an abstraction between region
> and FS. This jira is a move in the same direction, where it adds retry
> functionality for non-idempotent calls such as create, rename and delete in
> HRegionFileSystem class.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[ https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609377#comment-13609377 ] Lars Hofhansl commented on HBASE-8166: -- Comments/Concerns? This is a straightforward and backward compatible change. > Avoid writing the memstoreTS into HFiles when possible > -- > > Key: HBASE-8166 > URL: https://issues.apache.org/jira/browse/HBASE-8166 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8166-0.94.txt, 8166-0.96.txt > > > See discussion in HBASE-8151. > Upon compaction we can check all the involved HFiles. If all of them have a > MAX_MEMSTORE_TS < the smallest scannner readpoint, we create a writer that > does not write the memstoreTS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609378#comment-13609378 ] Edward C. Skoviak commented on HBASE-8170: -- +1 to option #2 > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Priority: Minor > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8014) Backport HBASE-6915 to 0.94.
[ https://issues.apache.org/jira/browse/HBASE-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-8014. -- Resolution: Fixed > Backport HBASE-6915 to 0.94. > > > Key: HBASE-8014 > URL: https://issues.apache.org/jira/browse/HBASE-8014 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Critical > Fix For: 0.94.7 > > Attachments: HBASE-8014-v0-0.94.patch > > > JDK 1.7 changed some data size. Goal of this JIRA is to backport HBASE-6915 > to 0.94 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609379#comment-13609379
]
Lars Hofhansl commented on HBASE-8067:
--
Also failed in 0.94 again:
{code}
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:266)
{code}
(Just including the info from jenkins here, so we won't loose the context)
> TestHFileArchiving.testArchiveOnTableDelete sometimes fails
> ---
>
> Key: HBASE-8067
> URL: https://issues.apache.org/jira/browse/HBASE-8067
> Project: HBase
> Issue Type: Bug
> Components: Admin, master, test
>Affects Versions: 0.96.0, 0.94.6
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.95.0, 0.94.7
>
> Attachments: HBASE-8067-debug.patch, HBASE-8067-v0.patch
>
>
> it seems that testArchiveOnTableDelete() fails because the archiving in
> DeleteTableHandler is still in progress when admin.deleteTable() returns.
> {code}
> Error Message
> Archived files are missing some of the store files!
> Stacktrace
> java.lang.AssertionError: Archived files are missing some of the store files!
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
> {code}
> (Looking at the problem in a more generic way, we don't have any way to
> inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[
https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609379#comment-13609379
]
Lars Hofhansl edited comment on HBASE-8067 at 3/21/13 7:55 PM:
---
Also failed in 0.94 again.
Edit: Same exception as in description.
was (Author: lhofhansl):
Also failed in 0.94 again:
{code}
java.lang.AssertionError: Archived files are missing some of the store files!
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:266)
{code}
(Just including the info from jenkins here, so we won't loose the context)
> TestHFileArchiving.testArchiveOnTableDelete sometimes fails
> ---
>
> Key: HBASE-8067
> URL: https://issues.apache.org/jira/browse/HBASE-8067
> Project: HBase
> Issue Type: Bug
> Components: Admin, master, test
>Affects Versions: 0.96.0, 0.94.6
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.95.0, 0.94.7
>
> Attachments: HBASE-8067-debug.patch, HBASE-8067-v0.patch
>
>
> it seems that testArchiveOnTableDelete() fails because the archiving in
> DeleteTableHandler is still in progress when admin.deleteTable() returns.
> {code}
> Error Message
> Archived files are missing some of the store files!
> Stacktrace
> java.lang.AssertionError: Archived files are missing some of the store files!
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262)
> {code}
> (Looking at the problem in a more generic way, we don't have any way to
> inform the client when an async operation is completed)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8170: --- Attachment: HBASE-8170-v0-trunk.patch > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Priority: Minor > Attachments: HBASE-8170-v0-trunk.patch > > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8166) Avoid writing the memstoreTS into HFiles when possible
[ https://issues.apache.org/jira/browse/HBASE-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609381#comment-13609381 ] Ted Yu commented on HBASE-8166: --- +1 from me. > Avoid writing the memstoreTS into HFiles when possible > -- > > Key: HBASE-8166 > URL: https://issues.apache.org/jira/browse/HBASE-8166 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8166-0.94.txt, 8166-0.96.txt > > > See discussion in HBASE-8151. > Upon compaction we can check all the involved HFiles. If all of them have a > MAX_MEMSTORE_TS < the smallest scannner readpoint, we create a writer that > does not write the memstoreTS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8170: --- Assignee: Jean-Marc Spaggiari > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-8170-v0-trunk.patch > > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8170) HbaseAdmin.createTable cannot handle creating three regions
[ https://issues.apache.org/jira/browse/HBASE-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8170: --- Status: Patch Available (was: Open) Patch for option #2 attached. If fine, I will also upload it for 0.94. > HbaseAdmin.createTable cannot handle creating three regions > --- > > Key: HBASE-8170 > URL: https://issues.apache.org/jira/browse/HBASE-8170 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.94.5 >Reporter: Edward C. Skoviak >Priority: Minor > Attachments: HBASE-8170-v0-trunk.patch > > > createTable(HTableDescriptor desc, byte [] startKey, byte [] endKey, int > numRegions) (line #370) dictates that you must specify a minimum of three > regions, however is not able to handle being fed a value three. This is a > result of line #379 where it attempts to create the key splits, and calls > Bytes.Split with a value of 0 for the third parameter. createTable should > instead just create a byte[][] with the startKey and endKey in this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
