[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008289#comment-14008289
]
Anoop Sam John commented on HBASE-11234:
prefix-tree part of fix is required in Trunk also.
For 94 and 96 patches no changes are needed. 94 is reverted I think. (Which we
can commit again)
I am fine with attached fix. Would like to get Matt Corgan's opinion also.
FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
Key: HBASE-11234
URL: https://issues.apache.org/jira/browse/HBASE-11234
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4
Attachments: 11234-94.patch, 11234-96.patch,
11234-98-with-prefix-tree.txt, HBASE-11234.patch
As Ted found,
With this change:
{noformat}
Index:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
===
---
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
+++
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
@@ -51,6 +51,7 @@
import org.apache.hadoop.hbase.filter.FilterList.Operator;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
import org.apache.hadoop.hbase.io.hfile.CacheConfig;
import org.apache.hadoop.hbase.io.hfile.HFileContext;
import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
@@ -90,6 +91,7 @@
CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
HFileContextBuilder hcBuilder = new HFileContextBuilder();
hcBuilder.withBlockSize(2 * 1024);
+hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
HFileContext hFileContext = hcBuilder.build();
StoreFile.Writer writer = new StoreFile.WriterBuilder(
TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
{noformat}
I got:
java.lang.AssertionError:
expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5
but was:testRow0198/testCf:testQual/1400712260004/ Put/vlen=13/mvcc=0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
After debugging, it seems the method of
FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
hfilescanner#seekBefore returns wrong result.
The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11246) Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result'
[ https://issues.apache.org/jira/browse/HBASE-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John resolved HBASE-11246. Resolution: Invalid HBASE-11234 is in reopened state now. No need for this Jira. So closing Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result' --- Key: HBASE-11246 URL: https://issues.apache.org/jira/browse/HBASE-11246 Project: HBase Issue Type: Bug Reporter: Ted Yu Priority: Critical Fix For: 0.98.4 HBASE-11234 fixed bug in FastDiffDeltaEncoder. This issue is to backport the fix to earlier releases. For 0.96 / 0.98, the fix should also cover prefix tree encoding. This would allow TestReversibleScanners#testReversibleStoreFileScanner to iterate through all the data block encodings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HBASE-11246) Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result'
[ https://issues.apache.org/jira/browse/HBASE-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-11246: Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result' --- Key: HBASE-11246 URL: https://issues.apache.org/jira/browse/HBASE-11246 Project: HBase Issue Type: Bug Reporter: Ted Yu Priority: Critical Fix For: 0.98.4 HBASE-11234 fixed bug in FastDiffDeltaEncoder. This issue is to backport the fix to earlier releases. For 0.96 / 0.98, the fix should also cover prefix tree encoding. This would allow TestReversibleScanners#testReversibleStoreFileScanner to iterate through all the data block encodings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11246) Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result'
[ https://issues.apache.org/jira/browse/HBASE-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-11246. Resolution: Duplicate Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result' --- Key: HBASE-11246 URL: https://issues.apache.org/jira/browse/HBASE-11246 Project: HBase Issue Type: Bug Reporter: Ted Yu Priority: Critical Fix For: 0.98.4 HBASE-11234 fixed bug in FastDiffDeltaEncoder. This issue is to backport the fix to earlier releases. For 0.96 / 0.98, the fix should also cover prefix tree encoding. This would allow TestReversibleScanners#testReversibleStoreFileScanner to iterate through all the data block encodings. -- This message was sent by Atlassian JIRA (v6.2#6252)
Does anyone know how to read hbase hdfs file using java api?
Hi experts,
I create a table using this command: create 'test','cf1'.
then put some data there: put 'test','realRow','cf1','realvalue1'
now I am trying to read its data using hdfs java api instead of hbase api:
FileSystem fs = FileSystem.get(URI.create(uri), conf);
Path path = new Path(
hdfs://localhost:9000/apps/hbase/data/data/default/test/69c16a187661b1ea8dd904851b9e3bb0/cf1/111c243d1953442f93b4d653690abe20
);
FSDataInputStream in = fs.open(path);
String filename = 111c243d1953442f93b4d653690abe20;
BufferedOutputStream out = new BufferedOutputStream(
new FileOutputStream(new File(filename)));
byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) 0) {
out.write(b, 0, numBytes);
System.out.println(Bytes.toString(b));
}
But data come like this:
[image: Inline image 1]
Is there something wrong with my decoding code?
Thanks
[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008311#comment-14008311 ] Qiang Tian commented on HBASE-11096: Hi [~apurtell], it looks the shutdown method is not called..could you please check if the patch apply to the MasterCoprocessorHost.java correctly? I just run command patch -p0 ../backup/HBASE-11096-trunk-v3.patch to trunk, and the change is not applied correctly. ps when I firstly saw your update on 20/5, I merged the code with latest code using git merge command and the test running fine. today I used git rebase command to merge my local branch with the latest code (since you recommended rebase in mailing list), and I got the error ---after investigation the MasterCoprocessorHost.java was not merged correctly.. I did a search, it looks git rebase uses patch command to do the final merge. so if patch does not work correctly, the rebase will fail too... pps..this is my second time to see patch command fails...Did I use wrong option? stop method of Master and RegionServer coprocessor is not invoked -- Key: HBASE-11096 URL: https://issues.apache.org/jira/browse/HBASE-11096 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.94.19 Reporter: Qiang Tian Assignee: Qiang Tian Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.patch, HBASE-11096-trunk-v3.patch, TestCoprocessorStop-failed-output.txt the stop method of coprocessor specified by hbase.coprocessor.master.classes and hbase.coprocessor.regionserver.classes is not invoked. If coprocessor allocates OS resources, it could cause master/regionserver resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Does anyone know how to read hbase hdfs file using java api?
On Sat, May 24, 2014 at 11:23 PM, David Liu d...@gopivotal.com wrote:
Hi experts,
I create a table using this command: create 'test','cf1'.
then put some data there: put 'test','realRow','cf1','realvalue1'
now I am trying to read its data using hdfs java api instead of hbase api:
FileSystem fs = FileSystem.get(URI.create(uri), conf);
Path path = new Path(
hdfs://localhost:9000/apps/hbase/data/data/default/test/69c16a187661b1ea8dd904851b9e3bb0/cf1/111c243d1953442f93b4d653690abe20
);
FSDataInputStream in = fs.open(path);
String filename = 111c243d1953442f93b4d653690abe20;
BufferedOutputStream out = new BufferedOutputStream(
new FileOutputStream(new File(filename)));
byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) 0) {
out.write(b, 0, numBytes);
System.out.println(Bytes.toString(b));
}
But data come like this:
[image: Inline image 1]
Is there something wrong with my decoding code?
Your image did not come across (they are suppressed on this mailing list).
Better to put up a link in future.
You just want to read raw bytes or you want interpreted view on the
contents?
HBase writes hfiles: see http://hbase.apache.org/book.html#hfilev2
For an example reading them, start at the main method here
http://hbase.apache.org/xref/org/apache/hadoop/hbase/io/hfile/HFile.html#891and
follow through to the pretty print class.
St.Ack
[jira] [Created] (HBASE-11249) Missing null check in finally block of HRegion#processRowsWithLocks()
Ted Yu created HBASE-11249:
--
Summary: Missing null check in finally block of
HRegion#processRowsWithLocks()
Key: HBASE-11249
URL: https://issues.apache.org/jira/browse/HBASE-11249
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11249) Missing null check in finally block of HRegion#processRowsWithLocks()
[
https://issues.apache.org/jira/browse/HBASE-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11249:
---
Description:
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Exception would be thrown from checkFamily() if store is null.
In the finally block:
{code}
} finally {
if (!mutations.isEmpty() !walSyncSuccessful) {
LOG.warn(Wal sync failed. Roll back + mutations.size() +
memstore keyvalues for row(s): + StringUtils.byteToHexString(
processor.getRowsToLock().iterator().next()) + ...);
for (KeyValue kv : mutations) {
getStore(kv).rollback(kv);
}
{code}
There is no corresponding null check for store above, potentially leading to
partially rolled back state in memstore.
was:
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Missing null check in finally block of HRegion#processRowsWithLocks()
-
Key: HBASE-11249
URL: https://issues.apache.org/jira/browse/HBASE-11249
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Exception would be thrown from checkFamily() if store is null.
In the finally block:
{code}
} finally {
if (!mutations.isEmpty() !walSyncSuccessful) {
LOG.warn(Wal sync failed. Roll back + mutations.size() +
memstore keyvalues for row(s): + StringUtils.byteToHexString(
processor.getRowsToLock().iterator().next()) + ...);
for (KeyValue kv : mutations) {
getStore(kv).rollback(kv);
}
{code}
There is no corresponding null check for store above, potentially leading to
partially rolled back state in memstore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11249) Missing null check in finally block of HRegion#processRowsWithLocks()
[
https://issues.apache.org/jira/browse/HBASE-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11249:
---
Description:
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Exception would be thrown from checkFamily() if store is null.
In the finally block:
{code}
} finally {
if (!mutations.isEmpty() !walSyncSuccessful) {
LOG.warn(Wal sync failed. Roll back + mutations.size() +
memstore keyvalues for row(s): + StringUtils.byteToHexString(
processor.getRowsToLock().iterator().next()) + ...);
for (KeyValue kv : mutations) {
getStore(kv).rollback(kv);
}
{code}
There is no corresponding null check for return value of getStore() above,
potentially leading to partially rolled back state in memstore.
was:
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Exception would be thrown from checkFamily() if store is null.
In the finally block:
{code}
} finally {
if (!mutations.isEmpty() !walSyncSuccessful) {
LOG.warn(Wal sync failed. Roll back + mutations.size() +
memstore keyvalues for row(s): + StringUtils.byteToHexString(
processor.getRowsToLock().iterator().next()) + ...);
for (KeyValue kv : mutations) {
getStore(kv).rollback(kv);
}
{code}
There is no corresponding null check for store above, potentially leading to
partially rolled back state in memstore.
Missing null check in finally block of HRegion#processRowsWithLocks()
-
Key: HBASE-11249
URL: https://issues.apache.org/jira/browse/HBASE-11249
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Exception would be thrown from checkFamily() if store is null.
In the finally block:
{code}
} finally {
if (!mutations.isEmpty() !walSyncSuccessful) {
LOG.warn(Wal sync failed. Roll back + mutations.size() +
memstore keyvalues for row(s): + StringUtils.byteToHexString(
processor.getRowsToLock().iterator().next()) + ...);
for (KeyValue kv : mutations) {
getStore(kv).rollback(kv);
}
{code}
There is no corresponding null check for return value of getStore() above,
potentially leading to partially rolled back state in memstore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11249) Missing null check in finally block of HRegion#processRowsWithLocks() may lead to partially rolled back state in memstore.
[
https://issues.apache.org/jira/browse/HBASE-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11249:
---
Summary: Missing null check in finally block of
HRegion#processRowsWithLocks() may lead to partially rolled back state in
memstore. (was: Missing null check in finally block of
HRegion#processRowsWithLocks())
Missing null check in finally block of HRegion#processRowsWithLocks() may
lead to partially rolled back state in memstore.
--
Key: HBASE-11249
URL: https://issues.apache.org/jira/browse/HBASE-11249
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
At line 4883:
{code}
Store store = getStore(kv);
if (store == null) {
checkFamily(CellUtil.cloneFamily(kv));
// unreachable
}
{code}
Exception would be thrown from checkFamily() if store is null.
In the finally block:
{code}
} finally {
if (!mutations.isEmpty() !walSyncSuccessful) {
LOG.warn(Wal sync failed. Roll back + mutations.size() +
memstore keyvalues for row(s): + StringUtils.byteToHexString(
processor.getRowsToLock().iterator().next()) + ...);
for (KeyValue kv : mutations) {
getStore(kv).rollback(kv);
}
{code}
There is no corresponding null check for return value of getStore() above,
potentially leading to partially rolled back state in memstore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11234:
---
Attachment: 11234-trunk.addendum
Addendum for trunk which touches PrefixTreeArrayScanner.java
FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
Key: HBASE-11234
URL: https://issues.apache.org/jira/browse/HBASE-11234
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4
Attachments: 11234-94.patch, 11234-96.patch,
11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch
As Ted found,
With this change:
{noformat}
Index:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
===
---
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
+++
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
@@ -51,6 +51,7 @@
import org.apache.hadoop.hbase.filter.FilterList.Operator;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
import org.apache.hadoop.hbase.io.hfile.CacheConfig;
import org.apache.hadoop.hbase.io.hfile.HFileContext;
import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
@@ -90,6 +91,7 @@
CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
HFileContextBuilder hcBuilder = new HFileContextBuilder();
hcBuilder.withBlockSize(2 * 1024);
+hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
HFileContext hFileContext = hcBuilder.build();
StoreFile.Writer writer = new StoreFile.WriterBuilder(
TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
{noformat}
I got:
java.lang.AssertionError:
expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5
but was:testRow0198/testCf:testQual/1400712260004/ Put/vlen=13/mvcc=0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
After debugging, it seems the method of
FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
hfilescanner#seekBefore returns wrong result.
The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008447#comment-14008447 ] Hudson commented on HBASE-11161: SUCCESS: Integrated in HBase-0.98 #314 (See [https://builds.apache.org/job/HBase-0.98/314/]) HBASE-11161 Provide example of POJO encoding with protobuf (Nick Dimiduk) (apurtell: rev d73711c800415250e5543cf22db486074c8ce815) * hbase-examples/src/main/java/org/apache/hadoop/hbase/types/PBCell.java * hbase-common/src/main/java/org/apache/hadoop/hbase/types/PBType.java * hbase-examples/src/test/java/org/apache/hadoop/hbase/types/TestPBCell.java Amend HBASE-11161 Provide example of POJO encoding with protobuf; Add explicit dependency on protobuf-java to hbase-common for Hadoop 1 (apurtell: rev 503709fdf67d335fcb96f9a60a4b971de01479e1) * hbase-common/pom.xml Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161-addendum.patch, HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008443#comment-14008443 ] Hudson commented on HBASE-11137: SUCCESS: Integrated in HBase-0.98 #314 (See [https://builds.apache.org/job/HBase-0.98/314/]) Amend HBASE-11137 Add mapred.TableSnapshotInputFormat; add missing license headers on branch 0.98 (apurtell: rev 568bfe416c1c880c84dbb4a11cdbe61aea9717a2) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSnapshotInputFormat.java Add mapred.TableSnapshotInputFormat --- Key: HBASE-11137 URL: https://issues.apache.org/jira/browse/HBASE-11137 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Affects Versions: 0.98.0, 0.96.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11137.00-0.98.patch, HBASE-11137.00.patch, HBASE-11137.01.patch, HBASE-11137.01_rerun.patch, HBASE-11137.02-0.98.patch, HBASE-11137.02.patch We should have feature parity between mapreduce and mapred implementations. This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
[ https://issues.apache.org/jira/browse/HBASE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008445#comment-14008445 ] Hudson commented on HBASE-11227: SUCCESS: Integrated in HBase-0.98 #314 (See [https://builds.apache.org/job/HBase-0.98/314/]) HBASE-11227 Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring (Nick Dimiduk) (apurtell: rev 2aed26ae6f9f3b7e961d0d92aebad41ecd2b200c) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11227.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11149) Wire encryption is broken
[ https://issues.apache.org/jira/browse/HBASE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008444#comment-14008444 ] Hudson commented on HBASE-11149: SUCCESS: Integrated in HBase-0.98 #314 (See [https://builds.apache.org/job/HBase-0.98/314/]) HBASE-11149. Addendum for fixing unit test (apurtell: rev 52dd30727a35b1b56ef2c474065ae206811d4a09) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java Wire encryption is broken - Key: HBASE-11149 URL: https://issues.apache.org/jira/browse/HBASE-11149 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: 11149-1.txt, 11149-2-addendum.txt, 11149-2.txt Upon some testing with the QOP configuration (hbase.rpc.protection), discovered that RPC doesn't work with integrity and privacy values for the configuration key. I was using 0.98.x for testing but I believe the issue is there in trunk as well (haven't checked 0.96 and 0.94). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008442#comment-14008442 ] Hudson commented on HBASE-9857: --- SUCCESS: Integrated in HBase-0.98 #314 (See [https://builds.apache.org/job/HBase-0.98/314/]) Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev a260c862a71c6433213e6619c4de004250468c83) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008446#comment-14008446
]
Hudson commented on HBASE-11234:
SUCCESS: Integrated in HBase-0.98 #314 (See
[https://builds.apache.org/job/HBase-0.98/314/])
Revert HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong
result (apurtell: rev b985e4fd0ebdd920b5c3b33dd5549713d82fa16a)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
Key: HBASE-11234
URL: https://issues.apache.org/jira/browse/HBASE-11234
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4
Attachments: 11234-94.patch, 11234-96.patch,
11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch
As Ted found,
With this change:
{noformat}
Index:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
===
---
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
+++
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
@@ -51,6 +51,7 @@
import org.apache.hadoop.hbase.filter.FilterList.Operator;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
import org.apache.hadoop.hbase.io.hfile.CacheConfig;
import org.apache.hadoop.hbase.io.hfile.HFileContext;
import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
@@ -90,6 +91,7 @@
CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
HFileContextBuilder hcBuilder = new HFileContextBuilder();
hcBuilder.withBlockSize(2 * 1024);
+hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
HFileContext hFileContext = hcBuilder.build();
StoreFile.Writer writer = new StoreFile.WriterBuilder(
TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
{noformat}
I got:
java.lang.AssertionError:
expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5
but was:testRow0198/testCf:testQual/1400712260004/ Put/vlen=13/mvcc=0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
After debugging, it seems the method of
FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
hfilescanner#seekBefore returns wrong result.
The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11149) Wire encryption is broken
[ https://issues.apache.org/jira/browse/HBASE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008454#comment-14008454 ] Hudson commented on HBASE-11149: SUCCESS: Integrated in HBase-TRUNK #5140 (See [https://builds.apache.org/job/HBase-TRUNK/5140/]) HBASE-11149. Addendum for fixing unit test (ddas: rev e9017586ca16d428d2d596bbe816fbaf2467e20c) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java Wire encryption is broken - Key: HBASE-11149 URL: https://issues.apache.org/jira/browse/HBASE-11149 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: 11149-1.txt, 11149-2-addendum.txt, 11149-2.txt Upon some testing with the QOP configuration (hbase.rpc.protection), discovered that RPC doesn't work with integrity and privacy values for the configuration key. I was using 0.98.x for testing but I believe the issue is there in trunk as well (haven't checked 0.96 and 0.94). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008456#comment-14008456 ] Hudson commented on HBASE-11161: SUCCESS: Integrated in HBase-TRUNK #5140 (See [https://builds.apache.org/job/HBase-TRUNK/5140/]) HBASE-11161 Provide example of POJO encoding with protobuf (Nick Dimiduk) (apurtell: rev 8b145419edac5d5d29e0e7d5718f60635e355500) * hbase-examples/src/test/java/org/apache/hadoop/hbase/types/TestPBCell.java * hbase-examples/src/main/java/org/apache/hadoop/hbase/types/PBCell.java * hbase-common/src/main/java/org/apache/hadoop/hbase/types/PBType.java Amend HBASE-11161 Provide example of POJO encoding with protobuf; Add explicit dependency on protobuf-java to hbase-common for Hadoop 1 (apurtell: rev b85cbbdea920525f9c019c0ac9187590943fcc27) * hbase-common/pom.xml Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161-addendum.patch, HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008453#comment-14008453 ] Hudson commented on HBASE-9857: --- SUCCESS: Integrated in HBase-TRUNK #5140 (See [https://builds.apache.org/job/HBase-TRUNK/5140/]) Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev be85f89cd4508de5337820a2694d5262c6d69092) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
[ https://issues.apache.org/jira/browse/HBASE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008455#comment-14008455 ] Hudson commented on HBASE-11227: SUCCESS: Integrated in HBase-TRUNK #5140 (See [https://builds.apache.org/job/HBase-TRUNK/5140/]) HBASE-11227 Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring (Nick Dimiduk) (apurtell: rev 30aab8b5ea667b97f7a161893cf98b82d7f34fc6) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11227.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11238) Add info about SlabCache and BucketCache to Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008457#comment-14008457 ] Misty Stanley-Jones commented on HBASE-11238: - My thinking was that this is a config setting, not a development setting. I wouldn't normally look in Javadoc for a config setting. But maybe it's different in HBase-land? If you want to keep it in Javadoc, do you like my version or have I lost some of the meaning in the rewrite? Add info about SlabCache and BucketCache to Ref Guide - Key: HBASE-11238 URL: https://issues.apache.org/jira/browse/HBASE-11238 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11238.patch Upstream issues: HBASE-11171 and HBASE-11098. Could back port some of what is in these issues, the package-info.java class for instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
[ https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones reassigned HBASE-11250: --- Assignee: Misty Stanley-Jones (was: Francis Liu) Document: CLONE - Fix jersey serialization/deserialization of json objects -- Key: HBASE-11250 URL: https://issues.apache.org/jira/browse/HBASE-11250 Project: HBase Issue Type: Bug Components: documentation, REST Affects Versions: 0.92.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Blocker Stargate uses the default json marshaller/unmarshaller in natural mode. In this mode the unmarshaller has trouble unmarshalling json instances. This patch fixes this issue by using jackson as the marshaller/unmarshaller instead. I've also updated all the model unit tests to test json serialization/deserialization. Backwards compatibilty can be verified by modify the test base class to use the original marshaller/unmarshaller and see that model tests pass. The patch is backward compatible except for StorageClusterStatusModel, which is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
Misty Stanley-Jones created HBASE-11250: --- Summary: Document: CLONE - Fix jersey serialization/deserialization of json objects Key: HBASE-11250 URL: https://issues.apache.org/jira/browse/HBASE-11250 Project: HBase Issue Type: Bug Components: REST Reporter: Misty Stanley-Jones Assignee: Francis Liu Priority: Blocker Fix For: 0.98.0, 0.96.0 Stargate uses the default json marshaller/unmarshaller in natural mode. In this mode the unmarshaller has trouble unmarshalling json instances. This patch fixes this issue by using jackson as the marshaller/unmarshaller instead. I've also updated all the model unit tests to test json serialization/deserialization. Backwards compatibilty can be verified by modify the test base class to use the original marshaller/unmarshaller and see that model tests pass. The patch is backward compatible except for StorageClusterStatusModel, which is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
[ https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11250: Fix Version/s: (was: 0.96.0) (was: 0.98.0) Document: CLONE - Fix jersey serialization/deserialization of json objects -- Key: HBASE-11250 URL: https://issues.apache.org/jira/browse/HBASE-11250 Project: HBase Issue Type: Bug Components: documentation, REST Affects Versions: 0.92.2 Reporter: Misty Stanley-Jones Assignee: Francis Liu Priority: Blocker Stargate uses the default json marshaller/unmarshaller in natural mode. In this mode the unmarshaller has trouble unmarshalling json instances. This patch fixes this issue by using jackson as the marshaller/unmarshaller instead. I've also updated all the model unit tests to test json serialization/deserialization. Backwards compatibilty can be verified by modify the test base class to use the original marshaller/unmarshaller and see that model tests pass. The patch is backward compatible except for StorageClusterStatusModel, which is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
[ https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11250: Affects Version/s: 0.92.2 Document: CLONE - Fix jersey serialization/deserialization of json objects -- Key: HBASE-11250 URL: https://issues.apache.org/jira/browse/HBASE-11250 Project: HBase Issue Type: Bug Components: documentation, REST Affects Versions: 0.92.2 Reporter: Misty Stanley-Jones Assignee: Francis Liu Priority: Blocker Stargate uses the default json marshaller/unmarshaller in natural mode. In this mode the unmarshaller has trouble unmarshalling json instances. This patch fixes this issue by using jackson as the marshaller/unmarshaller instead. I've also updated all the model unit tests to test json serialization/deserialization. Backwards compatibilty can be verified by modify the test base class to use the original marshaller/unmarshaller and see that model tests pass. The patch is backward compatible except for StorageClusterStatusModel, which is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
[ https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11250: Component/s: documentation Document: CLONE - Fix jersey serialization/deserialization of json objects -- Key: HBASE-11250 URL: https://issues.apache.org/jira/browse/HBASE-11250 Project: HBase Issue Type: Bug Components: documentation, REST Affects Versions: 0.92.2 Reporter: Misty Stanley-Jones Assignee: Francis Liu Priority: Blocker Stargate uses the default json marshaller/unmarshaller in natural mode. In this mode the unmarshaller has trouble unmarshalling json instances. This patch fixes this issue by using jackson as the marshaller/unmarshaller instead. I've also updated all the model unit tests to test json serialization/deserialization. Backwards compatibilty can be verified by modify the test base class to use the original marshaller/unmarshaller and see that model tests pass. The patch is backward compatible except for StorageClusterStatusModel, which is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008467#comment-14008467 ] Hudson commented on HBASE-11226: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) HBASE-11226 Document and increase the default value for hbase.hstore.flusher.count (nkeywal: rev 996705c61de1c7b533024dcb41cc571531ebfbb9) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * hbase-common/src/main/resources/hbase-default.xml Document and increase the default value for hbase.hstore.flusher.count -- Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: 11226.v1.patch HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008471#comment-14008471 ] Hudson commented on HBASE-11161: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) HBASE-11161 Provide example of POJO encoding with protobuf (Nick Dimiduk) (apurtell: rev d73711c800415250e5543cf22db486074c8ce815) * hbase-common/src/main/java/org/apache/hadoop/hbase/types/PBType.java * hbase-examples/src/main/java/org/apache/hadoop/hbase/types/PBCell.java * hbase-examples/src/test/java/org/apache/hadoop/hbase/types/TestPBCell.java Amend HBASE-11161 Provide example of POJO encoding with protobuf; Add explicit dependency on protobuf-java to hbase-common for Hadoop 1 (apurtell: rev 503709fdf67d335fcb96f9a60a4b971de01479e1) * hbase-common/pom.xml Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161-addendum.patch, HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008466#comment-14008466 ] Hudson commented on HBASE-11137: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) Amend HBASE-11137 Add mapred.TableSnapshotInputFormat; add missing license headers on branch 0.98 (apurtell: rev 568bfe416c1c880c84dbb4a11cdbe61aea9717a2) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSnapshotInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java Add mapred.TableSnapshotInputFormat --- Key: HBASE-11137 URL: https://issues.apache.org/jira/browse/HBASE-11137 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Affects Versions: 0.98.0, 0.96.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11137.00-0.98.patch, HBASE-11137.00.patch, HBASE-11137.01.patch, HBASE-11137.01_rerun.patch, HBASE-11137.02-0.98.patch, HBASE-11137.02.patch We should have feature parity between mapreduce and mapred implementations. This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11149) Wire encryption is broken
[ https://issues.apache.org/jira/browse/HBASE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008468#comment-14008468 ] Hudson commented on HBASE-11149: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) HBASE-11149. Addendum for fixing unit test (apurtell: rev 52dd30727a35b1b56ef2c474065ae206811d4a09) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java Wire encryption is broken - Key: HBASE-11149 URL: https://issues.apache.org/jira/browse/HBASE-11149 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: 11149-1.txt, 11149-2-addendum.txt, 11149-2.txt Upon some testing with the QOP configuration (hbase.rpc.protection), discovered that RPC doesn't work with integrity and privacy values for the configuration key. I was using 0.98.x for testing but I believe the issue is there in trunk as well (haven't checked 0.96 and 0.94). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008465#comment-14008465 ] Hudson commented on HBASE-9857: --- SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) Amend HBASE-9857 Blockcache prefetch option; add missing license header to correct file this time (apurtell: rev a260c862a71c6433213e6619c4de004250468c83) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
[ https://issues.apache.org/jira/browse/HBASE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008469#comment-14008469 ] Hudson commented on HBASE-11227: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) HBASE-11227 Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring (Nick Dimiduk) (apurtell: rev 2aed26ae6f9f3b7e961d0d92aebad41ecd2b200c) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11227.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11209) Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4
[ https://issues.apache.org/jira/browse/HBASE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008464#comment-14008464 ] Hudson commented on HBASE-11209: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) HBASE-11209 Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4 (nkeywal: rev 5521dfe6f05c8f49f2ebfa1c54b80169f2296a83) * hbase-common/src/main/resources/hbase-default.xml Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4 - Key: HBASE-11209 URL: https://issues.apache.org/jira/browse/HBASE-11209 Project: HBase Issue Type: Brainstorming Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11209.patch On a YCSB test, I saw a 33% performance increase, both on the max latency and on the throughput. I'm convinced enough that this value is better that I think it makes sense to change it on 0.98 as well. More fundamentally, but outside of the scope of this patch, I think this parameter should be changed to something at the region server level: today, we have: - global memstore check: if we're other 40%, we flush the biggest memstore - local: no more than 2 (proposed: 4) memstore size per region. But if we have enough memory and a spike on a region, there is no reason for not taking the write. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008470#comment-14008470
]
Hudson commented on HBASE-11234:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/])
Revert HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong
result (apurtell: rev b985e4fd0ebdd920b5c3b33dd5549713d82fa16a)
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
Key: HBASE-11234
URL: https://issues.apache.org/jira/browse/HBASE-11234
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4
Attachments: 11234-94.patch, 11234-96.patch,
11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch
As Ted found,
With this change:
{noformat}
Index:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
===
---
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
+++
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
@@ -51,6 +51,7 @@
import org.apache.hadoop.hbase.filter.FilterList.Operator;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
import org.apache.hadoop.hbase.io.hfile.CacheConfig;
import org.apache.hadoop.hbase.io.hfile.HFileContext;
import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
@@ -90,6 +91,7 @@
CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
HFileContextBuilder hcBuilder = new HFileContextBuilder();
hcBuilder.withBlockSize(2 * 1024);
+hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
HFileContext hFileContext = hcBuilder.build();
StoreFile.Writer writer = new StoreFile.WriterBuilder(
TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
{noformat}
I got:
java.lang.AssertionError:
expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5
but was:testRow0198/testCf:testQual/1400712260004/ Put/vlen=13/mvcc=0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
After debugging, it seems the method of
FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
hfilescanner#seekBefore returns wrong result.
The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
[ https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008473#comment-14008473 ] Misty Stanley-Jones commented on HBASE-11250: - This should be updated at the Wiki: http://wiki.apache.org/hadoop/Hbase/Stargate Document: CLONE - Fix jersey serialization/deserialization of json objects -- Key: HBASE-11250 URL: https://issues.apache.org/jira/browse/HBASE-11250 Project: HBase Issue Type: Bug Components: documentation, REST Affects Versions: 0.92.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Blocker Stargate uses the default json marshaller/unmarshaller in natural mode. In this mode the unmarshaller has trouble unmarshalling json instances. This patch fixes this issue by using jackson as the marshaller/unmarshaller instead. I've also updated all the model unit tests to test json serialization/deserialization. Backwards compatibilty can be verified by modify the test base class to use the original marshaller/unmarshaller and see that model tests pass. The patch is backward compatible except for StorageClusterStatusModel, which is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11218) Data loss in HBase standalone mode
[ https://issues.apache.org/jira/browse/HBASE-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008526#comment-14008526 ] Liu Shaohui commented on HBASE-11218: - [~nkeywal] The comparison done in my dev machine is following ||case||trunk||HBASE-11218|| |small tests|6m32.725s|6m34.124s| |medium tests|91m23.238s|93m6.703s| |large tests|154m58.474s|150m42.585s| The test time increase is very little. Data loss in HBase standalone mode -- Key: HBASE-11218 URL: https://issues.apache.org/jira/browse/HBASE-11218 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Fix For: 0.99.0 Attachments: HBASE-11218-trunk-v1.diff Data loss in HBase standalone mode. *How to produce it* # Start HBase standalone mode. # Create a table using hbase shell. # Scan '.META.' and you will find data in meta table # Kill the HBase process with -9 option # Start the HBase agaion # Scan '.META.' and you will find nothing in meta table. *There are three main reasons.* # FSDataOutputStream.sync should call flush() if the underlying wrapped stream is not Syncable. See HADOOP-8861 # writeChecksum is ture in default LocalFileSystem and the ChecksumFSOutputSummer will buffer the data, which make the waledits are not written to os's filesystem with sync method immediately, and those edits will be lost in regionserver's failover. # The MiniZooKeeperCluster deletes the old zk data at startup which maye cause data loss in meta table. The failover procedure is: split pre root regionserver's hlog - assign root - split pre meta regionserver's hlog - assign meta - split all other regionservers' hlogs - assign other regions. If there is no data in zookeeper, we will get null for root regionserver and then assign root table. Some data in root table maybe be lost for some root's WalEdits have not been splited and replayed. So does the Meta table. I finished the patch for 0.94 and am working on the patch for trunk. Suggestions are welcomed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11218) Data loss in HBase standalone mode
[
https://issues.apache.org/jira/browse/HBASE-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008527#comment-14008527
]
Liu Shaohui commented on HBASE-11218:
-
[~enis]
{quote}
in pseudo distributed mode, where there is an actual RS, but still using the
local fs, there still will be data loss with this patch from my understanding
{quote}
Yes. but I think the data loss in pseudo distributed mode caused by hardware
failure is accepted, data loss caused by the HBase's implement is not accepted
and we should fix it.
Data loss in HBase standalone mode
--
Key: HBASE-11218
URL: https://issues.apache.org/jira/browse/HBASE-11218
Project: HBase
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Fix For: 0.99.0
Attachments: HBASE-11218-trunk-v1.diff
Data loss in HBase standalone mode.
*How to produce it*
# Start HBase standalone mode.
# Create a table using hbase shell.
# Scan '.META.' and you will find data in meta table
# Kill the HBase process with -9 option
# Start the HBase agaion
# Scan '.META.' and you will find nothing in meta table.
*There are three main reasons.*
# FSDataOutputStream.sync should call flush() if the underlying wrapped
stream is not Syncable. See HADOOP-8861
# writeChecksum is ture in default LocalFileSystem and the
ChecksumFSOutputSummer will buffer the data, which make the waledits are not
written to os's filesystem with sync method immediately, and those edits will
be lost in regionserver's failover.
# The MiniZooKeeperCluster deletes the old zk data at startup which maye
cause data loss in meta table. The failover procedure is: split pre root
regionserver's hlog - assign root - split pre meta regionserver's hlog -
assign meta - split all other regionservers' hlogs - assign other regions.
If there is no data in zookeeper, we will get null for root regionserver and
then assign root table. Some data in root table maybe be lost for some root's
WalEdits have not been splited and replayed. So does the Meta table.
I finished the patch for 0.94 and am working on the patch for trunk.
Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10771: --- Attachment: HBASE-10771_V4.patch Andy helped me to run a test for checking the inlining of put/getxxx() APIs. As per that org.apache.hadoop.hbase.util.Bytes#explainWrongLengthOrOffset () is too big for inlining. The patch changes the getxxx() API to not to use Bytes#putxxx API. That code is copy pasted to SimpleByteRange without the sanity checking. Also addressed comment about adding a test where lots of KV are written and read back from a PBR. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch, HBASE-10771_V4.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access
ramkrishna.s.vasudevan created HBASE-11251:
--
Summary: LoadTestTool should grant READ permission for the users
that are given READ access
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.3
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11251:
---
Summary: LoadTestTool should grant READ permission for the users that are
given READ access for specific cells (was: LoadTestTool should grant READ
permission for the users that are given READ access)
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.3
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11251:
---
Fix Version/s: 0.99.0
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11251:
---
Status: Patch Available (was: Open)
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11251_0.98.patch, HBASE-11251_trunk.patch
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11251:
---
Attachment: HBASE-11251_trunk.patch
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11251_0.98.patch, HBASE-11251_trunk.patch
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11251:
---
Attachment: HBASE-11251_0.98.patch
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11251_0.98.patch, HBASE-11251_trunk.patch
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008574#comment-14008574
]
Andrew Purtell commented on HBASE-11251:
+1
This change was made so the cell first strategy (which is not the default) can
be useful for the use case it is intended for - denying/overriding access. The
cell ACL IT was not working at the time so it was missed.
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11251_0.98.patch, HBASE-11251_trunk.patch
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11234:
---
Fix Version/s: (was: 0.98.4)
0.98.3
FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
Key: HBASE-11234
URL: https://issues.apache.org/jira/browse/HBASE-11234
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.99.0, 0.96.3, 0.98.3, 0.94.21
Attachments: 11234-94.patch, 11234-96.patch,
11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch
As Ted found,
With this change:
{noformat}
Index:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
===
---
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
+++
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
@@ -51,6 +51,7 @@
import org.apache.hadoop.hbase.filter.FilterList.Operator;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
import org.apache.hadoop.hbase.io.hfile.CacheConfig;
import org.apache.hadoop.hbase.io.hfile.HFileContext;
import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
@@ -90,6 +91,7 @@
CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
HFileContextBuilder hcBuilder = new HFileContextBuilder();
hcBuilder.withBlockSize(2 * 1024);
+hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
HFileContext hFileContext = hcBuilder.build();
StoreFile.Writer writer = new StoreFile.WriterBuilder(
TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
{noformat}
I got:
java.lang.AssertionError:
expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5
but was:testRow0198/testCf:testQual/1400712260004/ Put/vlen=13/mvcc=0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
After debugging, it seems the method of
FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
hfilescanner#seekBefore returns wrong result.
The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008594#comment-14008594
]
Anoop Sam John commented on HBASE-11251:
+ HTable table = new HTable(conf, tableName);
The table is just used to get the tabelName which we already have. We can
avoid this HTable creation and so the close in finally. This is there in 2
places in the patch.
{code}
-AccessControlClient.grant(conf, tableName, userOwner.getShortName(),
COLUMN_FAMILY,
+try {
+ AccessControlClient.grant(conf, table.getName(),
userOwner.getShortName(), null,
{code}
Any reason why the specific CF grant is changed in this patch?
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11251_0.98.patch, HBASE-11251_trunk.patch
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11251) LoadTestTool should grant READ permission for the users that are given READ access for specific cells
[
https://issues.apache.org/jira/browse/HBASE-11251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008596#comment-14008596
]
ramkrishna.s.vasudevan commented on HBASE-11251:
bq.Any reason why the specific CF grant is changed in this patch?
CF grant permission is not needed here. It was done previously. But while
testing I found specifically we need not grant permission on the CF level.
Also there is only one CF in the test.
bq.The table is just used to get the tabelName which we already have. We can
avoid this HTable creation and so the close in finally. This is there in 2
places in the patch.
Ok. I just ensured the close had to be done which was not done in the existing
code.
LoadTestTool should grant READ permission for the users that are given READ
access for specific cells
-
Key: HBASE-11251
URL: https://issues.apache.org/jira/browse/HBASE-11251
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11251_0.98.patch, HBASE-11251_trunk.patch
In 0.98.2 onwards the AccessControlFilter
{code}
case CHECK_CELL_FIRST: {
LOG.info(Am coming here for cell first strategy);
if (authManager.authorize(user, table, cell, Permission.Action.READ)
authManager.authorize(user, table, family, qualifier,
Permission.Action.READ)) {
LOG.info(Returning include);
return ReturnCode.INCLUDE;
}
{code}
expects a READ permission on the table for those Users that are granted READ
permission on the cell level.
In 0.98.1
{code}
if (authManager.authorize(user, table, cell, cellFirstStrategy,
Permission.Action.READ)) {
return ReturnCode.INCLUDE;
}
{code}
So from 0.98.2 onwards IntegrationTestIngestWithACL was failing. Hence this
JIRA is targeted to correct the behaviour and make the IT work again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
