[jira] [Updated] (HBASE-10322) Strip tags from KV while sending back to client on reads
[ https://issues.apache.org/jira/browse/HBASE-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10322: --- Attachment: HBASE-10322_codec.patch Just attaching a patch where after we support only server side tags, with the above change for the replication side things would work. May be some one more familiar with replication can take a look. Replication testcases passes with this. Strip tags from KV while sending back to client on reads Key: HBASE-10322 URL: https://issues.apache.org/jira/browse/HBASE-10322 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.98.0, 0.99.0 Attachments: HBASE-10322.patch, HBASE-10322_V2.patch, HBASE-10322_codec.patch Right now we have some inconsistency wrt sending back tags on read. We do this in scan when using Java client(Codec based cell block encoding). But during a Get operation or when a pure PB based Scan comes we are not sending back the tags. So any of the below fix we have to do 1. Send back tags in missing cases also. But sending back visibility expression/ cell ACL is not correct. 2. Don't send back tags in any case. This will a problem when a tool like ExportTool use the scan to export the table data. We will miss exporting the cell visibility/ACL. 3. Send back tags based on some condition. It has to be per scan basis. Simplest way is pass some kind of attribute in Scan which says whether to send back tags or not. But believing some thing what scan specifies might not be correct IMO. Then comes the way of checking the user who is doing the scan. When a HBase super user doing the scan then only send back tags. So when a case comes like Export Tool's the execution should happen from a super user. So IMO we should go with #3. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10322) Strip tags from KV while sending back to client on reads
[ https://issues.apache.org/jira/browse/HBASE-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875584#comment-13875584 ] ramkrishna.s.vasudevan commented on HBASE-10322: If you do start using the 0.98 features which require cell tags, then your replication endpoints must all be upgrade to 0.98 first, and you must change a configuration setting. This would apply there. Strip tags from KV while sending back to client on reads Key: HBASE-10322 URL: https://issues.apache.org/jira/browse/HBASE-10322 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.98.0, 0.99.0 Attachments: HBASE-10322.patch, HBASE-10322_V2.patch, HBASE-10322_codec.patch Right now we have some inconsistency wrt sending back tags on read. We do this in scan when using Java client(Codec based cell block encoding). But during a Get operation or when a pure PB based Scan comes we are not sending back the tags. So any of the below fix we have to do 1. Send back tags in missing cases also. But sending back visibility expression/ cell ACL is not correct. 2. Don't send back tags in any case. This will a problem when a tool like ExportTool use the scan to export the table data. We will miss exporting the cell visibility/ACL. 3. Send back tags based on some condition. It has to be per scan basis. Simplest way is pass some kind of attribute in Scan which says whether to send back tags or not. But believing some thing what scan specifies might not be correct IMO. Then comes the way of checking the user who is doing the scan. When a HBase super user doing the scan then only send back tags. So when a case comes like Export Tool's the execution should happen from a super user. So IMO we should go with #3. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875587#comment-13875587 ] Bing Jiang commented on HBASE-10374: [~stack] I have a question, if we try to use the feature which Netty provides, I believe there are two choices, the one is to add dependence upon Netty, the other one is to implements the jemalloc by the help of their codes. I want to take your suggestion. add jemalloc into the choice of memstore allocation Key: HBASE-10374 URL: https://issues.apache.org/jira/browse/HBASE-10374 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.1.1 Reporter: Bing Jiang Priority: Minor https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead introduced that Netty used jemalloc to gain benefits from GC. It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10322) Strip tags from KV while sending back to client on reads
[ https://issues.apache.org/jira/browse/HBASE-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875600#comment-13875600 ] Anoop Sam John commented on HBASE-10322: How about adding new config hbase.replication.rpc.codec or going with existing config only but asking to configure xxxWithTags codec to be used? Ping [~stack] , [~apurtell]..I am fine with any of this. 1st one needs some code change as Ram attached while the other don't need any code changes. Strip tags from KV while sending back to client on reads Key: HBASE-10322 URL: https://issues.apache.org/jira/browse/HBASE-10322 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.98.0, 0.99.0 Attachments: HBASE-10322.patch, HBASE-10322_V2.patch, HBASE-10322_codec.patch Right now we have some inconsistency wrt sending back tags on read. We do this in scan when using Java client(Codec based cell block encoding). But during a Get operation or when a pure PB based Scan comes we are not sending back the tags. So any of the below fix we have to do 1. Send back tags in missing cases also. But sending back visibility expression/ cell ACL is not correct. 2. Don't send back tags in any case. This will a problem when a tool like ExportTool use the scan to export the table data. We will miss exporting the cell visibility/ACL. 3. Send back tags based on some condition. It has to be per scan basis. Simplest way is pass some kind of attribute in Scan which says whether to send back tags or not. But believing some thing what scan specifies might not be correct IMO. Then comes the way of checking the user who is doing the scan. When a HBase super user doing the scan then only send back tags. So when a case comes like Export Tool's the execution should happen from a super user. So IMO we should go with #3. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10322) Strip tags from KV while sending back to client on reads
[
https://issues.apache.org/jira/browse/HBASE-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875602#comment-13875602
]
Hadoop QA commented on HBASE-10322:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623779/HBASE-10322_codec.patch
against trunk revision .
ATTACHMENT ID: 12623779
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+this.conf.set(hbase.client.rpc.codec,
this.conf.get(hbase.replication.rpc.codec,org.apache.hadoop.hbase.codec.CellCodecV2));
+this.conf.set(hbase.client.rpc.codec,
this.conf.get(hbase.replication.rpc.codec,org.apache.hadoop.hbase.codec.CellCodecV2));
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestAcidGuarantees
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8468//console
This message is automatically generated.
Strip tags from KV while sending back to client on reads
Key: HBASE-10322
URL: https://issues.apache.org/jira/browse/HBASE-10322
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0, 0.99.0
Attachments: HBASE-10322.patch, HBASE-10322_V2.patch,
HBASE-10322_codec.patch
Right now we have some inconsistency wrt sending back tags on read. We do
this in scan when using Java client(Codec based cell block encoding). But
during a Get operation or when a pure PB based Scan comes we are not sending
back the tags. So any of the below fix we have to do
1. Send back tags in missing cases also. But sending back visibility
expression/ cell ACL is not correct.
2. Don't send back tags in any case. This will a problem when a tool like
ExportTool use the scan to export the table data. We will miss exporting the
cell visibility/ACL.
3. Send back tags based on some condition. It has to be per scan basis.
Simplest way is pass some kind of attribute in Scan which says whether to
send back tags or not. But believing some thing what scan specifies might not
be correct IMO. Then
[jira] [Commented] (HBASE-10349) Table became unusable when master balanced its region after table was dropped
[
https://issues.apache.org/jira/browse/HBASE-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875638#comment-13875638
]
Hudson commented on HBASE-10349:
FAILURE: Integrated in hbase-0.96-hadoop2 #179 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/179/])
HBASE-10349 Table became unusable when master balanced its region after table
was dropped (jxiang: rev 1559313)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
Table became unusable when master balanced its region after table was dropped
-
Key: HBASE-10349
URL: https://issues.apache.org/jira/browse/HBASE-10349
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.2, 0.99.0
Attachments:
10349-hadoop-hdfs-namenode-hor11n14.gq1.ygridcore.net.zip, 10349-output.log,
10349-v1.txt, 10349-v2.txt, HBASE-10349-meta-test-and-debug.patch,
hbase-10349.patch, hbase-10349_v2.patch,
hbase-hbase-master-hor15n05.gq1.ygridcore.net.log.tar.gz
0.98 was used.
This was sequence of events:
create 'tablethree_mod'
snapshot 'tablethree_mod', 'snapshot_tablethree_mod'
disable 'tablethree_mod'
2014-01-15 09:34:51,749 restore_snapshot 'snapshot_tablethree_mod'
2014-01-15 09:35:07,210 enable 'tablethree_mod'
2014-01-15 09:35:46,134 delete_snapshot 'snapshot_tablethree_mod'
2014-01-15 09:41:42,210 disable 'tablethree_mod'
2014-01-15 09:41:43,610 drop 'tablethree_mod'
create 'tablethree_mod'
For the last table creation request:
{code}
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'create 'tablethree_mod',
{NAME = 'f1', VERSIONS = 3}
,
{NAME = 'f2', VERSIONS = 3}
,
{NAME = 'f3', VERSIONS = 3}
'
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'exists 'tablethree_mod''
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'0', 'f1:q1', 'value-0', 10'
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'1', 'f1:q1', 'value-1', 20'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'2', 'f2:q2', 'value-2', 30'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'3', 'f3:q3', 'value-3', 40'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'4', 'f3:q3', 'value-4', 50'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO|Done writing commands to
file. Will execute them now.
2014-01-15 10:03:53,000|beaver.machine|INFO|RUNNING: /usr/lib/hbase/bin/hbase
shell /grid/0/tmp/hwqe/artifacts/tmp-471142
2014-01-15 10:03:55,878|beaver.machine|INFO|2014-01-15 10:03:55,878 INFO
[main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead,
use io.native.lib.available
2014-01-15 10:03:57,283|beaver.machine|INFO|2014-01-15 10:03:57,283 WARN
[main] conf.Configuration: hbase-site.xml:an attempt to override final
parameter: dfs.support.append; Ignoring.
2014-01-15 10:03:57,669|beaver.machine|INFO|2014-01-15 10:03:57,669 WARN
[main] conf.Configuration: hbase-site.xml:an attempt to override final
parameter: dfs.support.append; Ignoring.
2014-01-15 10:03:57,720|beaver.machine|INFO|2014-01-15 10:03:57,720 WARN
[main] conf.Configuration: hbase-site.xml:an attempt to override final
parameter: dfs.support.append; Ignoring.
2014-01-15 10:03:57,997|beaver.machine|INFO|
2014-01-15 10:03:57,997|beaver.machine|INFO|ERROR: Table already exists:
tablethree_mod!
2014-01-15 10:03:57,997|beaver.machine|INFO|
{code}
This was an intermittent issue after using Snapshots, a table is not properly
dropped / and not able to properly re-create with the same name. And a
HRegion is empty or null Error occurs. (When you try to drop the table it
says it does not exist, and when you try to create the table it says that it
does already exist).
{code}
2014-01-15 10:04:02,462|beaver.machine|INFO|ERROR: HRegionInfo was null or
empty in hbase:meta, row=keyvalues=
{tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/1389778905355/Put/vlen=32/mvcc=0,
[jira] [Comment Edited] (HBASE-10322) Strip tags from KV while sending back to client on reads
[ https://issues.apache.org/jira/browse/HBASE-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875571#comment-13875571 ] Anoop Sam John edited comment on HBASE-10322 at 1/18/14 6:50 PM: - My patch is like almost ready except Replication! ReplicationSource uses AdminService#ReplicateWALEntry() and call HRS#replicateWALEntry() via the same path of client to HRS communication.. ie via RpcClient.. So if we have to introduce 2 configs some sort of if checks etc will come which tells abt the context etc..:( Another simple way with out any code change I am thinking now is as follows: RpcClient looks at hbase.client.rpc.codec for getting the Codec class name. Well HBase client side looks at this config at client end and uses that Codec to write to server and also writes the codec class name in connection header. Server looks at the connection header to see the class name of the Codec which is used by the client and it instantiate that and uses the same to communicate with that client. Here we can do one thing. We can ask the user to configure two class names (When some one uses tags) at client and server. Say at client use KVCodec and at server configure it as KVCodecWithTags. Let KVCodecWithTags serialize tags also.At the peer cluster HRS also this codec class should be there and there, in connection header, the class name will be KVCodecWithTags and will use that :) This will work well. So for existing users there is no need to do any changes. When users use the tags (or visibility labels or cell level acl) they must make this change and make sure the 98 upgrade is done in all the clusters. Opinions pls. Edit : ReplicationSource already refers to this config hbase.client.rpc.codec as it uses RpcClient to talk with peer. was (Author: anoop.hbase): My patch is like almost ready except Replication! ReplicationSource uses AdminService#ReplicateWALEntry() and call HRS#replicateWALEntry() via the same path of client to HRS communication.. ie via RpcClient.. So if we have to introduce 2 configs some sort of if checks etc will come which tells abt the context etc..:( Another simple way with out any code change I am thinking now is as follows: RpcClient looks at hbase.client.rpc.codec for getting the Codec class name. Well HBase client side looks at this config at client end and uses that Codec to write to server and also writes the codec class name in connection header. Server looks at the connection header to see the class name of the Codec which is used by the client and it instantiate that and uses the same to communicate with that client. Here we can do one thing. We can ask the user to configure two class names (When some one uses tags) at client and server. Say at client use KVCodec and at server configure it as KVCodecWithTags. Let KVCodecWithTags serialize tags also.At the peer cluster HRS also this codec class should be there and there, in connection header, the class name will be KVCodecWithTags and will use that :) This will work well. So for existing users there is no need to do any changes. When users use the tags (or visibility labels or cell level acl) they must make this change and make sure the 98 upgrade is done in all the clusters. Opinions pls. Strip tags from KV while sending back to client on reads Key: HBASE-10322 URL: https://issues.apache.org/jira/browse/HBASE-10322 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.98.0, 0.99.0 Attachments: HBASE-10322.patch, HBASE-10322_V2.patch, HBASE-10322_codec.patch Right now we have some inconsistency wrt sending back tags on read. We do this in scan when using Java client(Codec based cell block encoding). But during a Get operation or when a pure PB based Scan comes we are not sending back the tags. So any of the below fix we have to do 1. Send back tags in missing cases also. But sending back visibility expression/ cell ACL is not correct. 2. Don't send back tags in any case. This will a problem when a tool like ExportTool use the scan to export the table data. We will miss exporting the cell visibility/ACL. 3. Send back tags based on some condition. It has to be per scan basis. Simplest way is pass some kind of attribute in Scan which says whether to send back tags or not. But believing some thing what scan specifies might not be correct IMO. Then comes the way of checking the user who is doing the scan. When a HBase super user doing the scan then only send back tags. So when a case comes like Export Tool's the execution should happen from a super user. So IMO we should go with #3. Patch
[jira] [Commented] (HBASE-10322) Strip tags from KV while sending back to client on reads
[ https://issues.apache.org/jira/browse/HBASE-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875757#comment-13875757 ] stack commented on HBASE-10322: --- It is wrong that replication is in the admin Service and that is in the server Service. Can we make a Replication Service? Would that help? Strip tags from KV while sending back to client on reads Key: HBASE-10322 URL: https://issues.apache.org/jira/browse/HBASE-10322 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 0.98.0, 0.99.0 Attachments: HBASE-10322.patch, HBASE-10322_V2.patch, HBASE-10322_codec.patch Right now we have some inconsistency wrt sending back tags on read. We do this in scan when using Java client(Codec based cell block encoding). But during a Get operation or when a pure PB based Scan comes we are not sending back the tags. So any of the below fix we have to do 1. Send back tags in missing cases also. But sending back visibility expression/ cell ACL is not correct. 2. Don't send back tags in any case. This will a problem when a tool like ExportTool use the scan to export the table data. We will miss exporting the cell visibility/ACL. 3. Send back tags based on some condition. It has to be per scan basis. Simplest way is pass some kind of attribute in Scan which says whether to send back tags or not. But believing some thing what scan specifies might not be correct IMO. Then comes the way of checking the user who is doing the scan. When a HBase super user doing the scan then only send back tags. So when a case comes like Export Tool's the execution should happen from a super user. So IMO we should go with #3. Patch coming soon. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
Himanshu Vashishtha created HBASE-10378: --- Summary: Divide HLog interface into User and Implementor specific interfaces Key: HBASE-10378 URL: https://issues.apache.org/jira/browse/HBASE-10378 Project: HBase Issue Type: Sub-task Components: wal Reporter: Himanshu Vashishtha HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as: 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it. 2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks. I'd like to propose some changes in HLog interface that would ease the multi WAL story: 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer). 2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL. 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting, 4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-9681) Basic codec negotiation
[ https://issues.apache.org/jira/browse/HBASE-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9681: -- Affects Version/s: (was: 0.98.0) 0.99.0 Fix Version/s: (was: 0.98.0) Moving out of 0.98 Basic codec negotiation --- Key: HBASE-9681 URL: https://issues.apache.org/jira/browse/HBASE-9681 Project: HBase Issue Type: Sub-task Affects Versions: 0.99.0 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Basic codec negotiation: There should be a default codec used for cell encoding over the RPC connection. This should be configurable in the site file. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for a list of supported cell codecs. An older server should simply send back an error because the request is invalid except to servers supporting this feature. A server supporting this feature should send back the requested information or an error indication if something went wrong. The client can optionally send a message, a manufactured call that would otherwise be invalid in some way, to the server asking for it to use a given codec for all further communication. Otherwise the server will continue to use the default codec. The server will send back a call response acknowledging the change or an error indication if the request cannot be honored. Server configuration should support mappings from one codec type to another. We need to handle the case where the server has a codec available that extends the requested type but overrides some behavior in the base class, and this is what should be used in lieu of the base type. It must also be possible to choose an alternate default codec which stands in for the default codec, is compatible with client expectations, but changes the server side behavior as needed in the absence of negotiation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-9351) Connection capability negotiation
[ https://issues.apache.org/jira/browse/HBASE-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9351: -- Affects Version/s: (was: 0.98.0) 0.99.0 Connection capability negotiation - Key: HBASE-9351 URL: https://issues.apache.org/jira/browse/HBASE-9351 Project: HBase Issue Type: Brainstorming Affects Versions: 0.99.0 Reporter: Andrew Purtell Would be useful to support negotiation at connection setup time beyond SASL. Consider: Start with a default baseline profile. Both client and server sides can begin communicating immediately (or after SASL completes if security is active), with a baseline set of messages and codecs. For more interesting use cases, support configuration messages that negotiate connection configuration going forward after both sides ack the changes: codec, configuration, compression. Any nack aborts the upgrade request and leaves both sides still in the default profile. Should be a pluggable implementation. For example, codec implementations should be automatically discovered at runtime if shipped with the client or server, and the set of available options should be communicated to the other side. Features like codecs should all be versioned themselves. Negotiation should be version aware, and decisionmaking on if a given pair of component versions are compatible should be delegated to the component. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
[ https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-10378: Attachment: 10378-1.patch Attaching an initial patch implementing the above mentioned changes. Please have a look at the overall approach. Thanks. For testing, mvn clean test -PrunAllTests passes for both hadoop1 and hadoop2. Also ran the patched version on a test cluster/killing nodes and doing log splitting for basic sanity test. It looks good. Divide HLog interface into User and Implementor specific interfaces --- Key: HBASE-10378 URL: https://issues.apache.org/jira/browse/HBASE-10378 Project: HBase Issue Type: Sub-task Components: wal Reporter: Himanshu Vashishtha Attachments: 10378-1.patch HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as: 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it. 2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks. I'd like to propose some changes in HLog interface that would ease the multi WAL story: 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer). 2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL. 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting, 4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
[ https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-10378: Attachment: 10378-1.patch Divide HLog interface into User and Implementor specific interfaces --- Key: HBASE-10378 URL: https://issues.apache.org/jira/browse/HBASE-10378 Project: HBase Issue Type: Sub-task Components: wal Reporter: Himanshu Vashishtha Attachments: 10378-1.patch, 10378-1.patch HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as: 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it. 2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks. I'd like to propose some changes in HLog interface that would ease the multi WAL story: 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer). 2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL. 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting, 4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10349) Table became unusable when master balanced its region after table was dropped
[
https://issues.apache.org/jira/browse/HBASE-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875787#comment-13875787
]
Hudson commented on HBASE-10349:
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #58 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/58/])
HBASE-10349 Table became unusable when master balanced its region after table
was dropped (jxiang: rev 1559311)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
Table became unusable when master balanced its region after table was dropped
-
Key: HBASE-10349
URL: https://issues.apache.org/jira/browse/HBASE-10349
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.2, 0.99.0
Attachments:
10349-hadoop-hdfs-namenode-hor11n14.gq1.ygridcore.net.zip, 10349-output.log,
10349-v1.txt, 10349-v2.txt, HBASE-10349-meta-test-and-debug.patch,
hbase-10349.patch, hbase-10349_v2.patch,
hbase-hbase-master-hor15n05.gq1.ygridcore.net.log.tar.gz
0.98 was used.
This was sequence of events:
create 'tablethree_mod'
snapshot 'tablethree_mod', 'snapshot_tablethree_mod'
disable 'tablethree_mod'
2014-01-15 09:34:51,749 restore_snapshot 'snapshot_tablethree_mod'
2014-01-15 09:35:07,210 enable 'tablethree_mod'
2014-01-15 09:35:46,134 delete_snapshot 'snapshot_tablethree_mod'
2014-01-15 09:41:42,210 disable 'tablethree_mod'
2014-01-15 09:41:43,610 drop 'tablethree_mod'
create 'tablethree_mod'
For the last table creation request:
{code}
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'create 'tablethree_mod',
{NAME = 'f1', VERSIONS = 3}
,
{NAME = 'f2', VERSIONS = 3}
,
{NAME = 'f3', VERSIONS = 3}
'
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'exists 'tablethree_mod''
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'0', 'f1:q1', 'value-0', 10'
2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'1', 'f1:q1', 'value-1', 20'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'2', 'f2:q2', 'value-2', 30'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'3', 'f3:q3', 'value-3', 40'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod',
'4', 'f3:q3', 'value-4', 50'
2014-01-15 10:03:53,000|beaver.component.hbase|INFO|Done writing commands to
file. Will execute them now.
2014-01-15 10:03:53,000|beaver.machine|INFO|RUNNING: /usr/lib/hbase/bin/hbase
shell /grid/0/tmp/hwqe/artifacts/tmp-471142
2014-01-15 10:03:55,878|beaver.machine|INFO|2014-01-15 10:03:55,878 INFO
[main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead,
use io.native.lib.available
2014-01-15 10:03:57,283|beaver.machine|INFO|2014-01-15 10:03:57,283 WARN
[main] conf.Configuration: hbase-site.xml:an attempt to override final
parameter: dfs.support.append; Ignoring.
2014-01-15 10:03:57,669|beaver.machine|INFO|2014-01-15 10:03:57,669 WARN
[main] conf.Configuration: hbase-site.xml:an attempt to override final
parameter: dfs.support.append; Ignoring.
2014-01-15 10:03:57,720|beaver.machine|INFO|2014-01-15 10:03:57,720 WARN
[main] conf.Configuration: hbase-site.xml:an attempt to override final
parameter: dfs.support.append; Ignoring.
2014-01-15 10:03:57,997|beaver.machine|INFO|
2014-01-15 10:03:57,997|beaver.machine|INFO|ERROR: Table already exists:
tablethree_mod!
2014-01-15 10:03:57,997|beaver.machine|INFO|
{code}
This was an intermittent issue after using Snapshots, a table is not properly
dropped / and not able to properly re-create with the same name. And a
HRegion is empty or null Error occurs. (When you try to drop the table it
says it does not exist, and when you try to create the table it says that it
does already exist).
{code}
2014-01-15 10:04:02,462|beaver.machine|INFO|ERROR: HRegionInfo was null or
empty in hbase:meta, row=keyvalues=
{tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/1389778905355/Put/vlen=32/mvcc=0,
[jira] [Updated] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
[ https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-10378: Attachment: (was: 10378-1.patch) Divide HLog interface into User and Implementor specific interfaces --- Key: HBASE-10378 URL: https://issues.apache.org/jira/browse/HBASE-10378 Project: HBase Issue Type: Sub-task Components: wal Reporter: Himanshu Vashishtha Attachments: 10378-1.patch HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as: 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it. 2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks. I'd like to propose some changes in HLog interface that would ease the multi WAL story: 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer). 2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL. 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting, 4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10327) Remove remove(K, V) from type PoolMapK,V
[ https://issues.apache.org/jira/browse/HBASE-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875788#comment-13875788 ] Hudson commented on HBASE-10327: SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #58 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/58/]) HBASE-10327. Remove remove(K, V) from type PoolMapK,V (Eric Charles) (apurtell: rev 1559288) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/util/PoolMap.java Remove remove(K, V) from type PoolMapK,V -- Key: HBASE-10327 URL: https://issues.apache.org/jira/browse/HBASE-10327 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Eric Charles Assignee: Eric Charles Fix For: 0.99.0 Attachments: HBASE-10327.patch I keep getting red cross in my eclipse, whatever the jdk (jdk6, jdk7, jdk8) Name clash: The method remove(K, V) of type PoolMapK,V has the same erasure as remove(Object, Object) of type MapK,V but does not override it maybe related to HBASE-10030 The solution I have is simply removing the deprecated method, and everything is fine. I am not sure of the backwards compatibility here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
[ https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-10378: Status: Patch Available (was: Open) Divide HLog interface into User and Implementor specific interfaces --- Key: HBASE-10378 URL: https://issues.apache.org/jira/browse/HBASE-10378 Project: HBase Issue Type: Sub-task Components: wal Reporter: Himanshu Vashishtha Attachments: 10378-1.patch HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as: 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it. 2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks. I'd like to propose some changes in HLog interface that would ease the multi WAL story: 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer). 2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL. 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting, 4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
[
https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875796#comment-13875796
]
Hadoop QA commented on HBASE-10378:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623840/10378-1.patch
against trunk revision .
ATTACHMENT ID: 12623840
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 128
new or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8469//console
This message is automatically generated.
Divide HLog interface into User and Implementor specific interfaces
---
Key: HBASE-10378
URL: https://issues.apache.org/jira/browse/HBASE-10378
Project: HBase
Issue Type: Sub-task
Components: wal
Reporter: Himanshu Vashishtha
Attachments: 10378-1.patch
HBASE-5937 introduces the HLog interface as a first step to support multiple
WAL implementations. This interface is a good start, but has some
limitations/drawbacks in its current state, such as:
1) There is no clear distinction b/w User and Implementor APIs, and it
provides APIs both for WAL users (append, sync, etc) and also WAL
implementors (Reader/Writer interfaces, etc). There are APIs which are very
much implementation specific (getFileNum, etc) and a user such as a
RegionServer shouldn't know about it.
2) There are about 14 methods in FSHLog which are not present in HLog
interface but are used at several places in the unit test code. These tests
typecast HLog to FSHLog, which makes it very difficult to test multiple WAL
implementations without doing some ugly checks.
I'd like to propose some changes in HLog interface that would ease the multi
WAL story:
1) Have two interfaces WAL and WALService. WAL provides APIs for
implementors. WALService provides APIs for users (such as RegionServer).
2) A skeleton implementation of the above two interface as the base class for
other WAL implementations (AbstractWAL). It provides required fields for all
subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods
and add this set in AbstractWAL.
3) HLogFactory returns a WALService reference when creating a WAL instance;
if a user need to access impl specific APIs (there are unit tests which get
WAL from a
