[jira] [Commented] (HBASE-10060) Unsynchronized scanning
[
https://issues.apache.org/jira/browse/HBASE-10060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835984#comment-13835984
]
Hadoop QA commented on HBASE-10060:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616464/10060-trunk-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestStoreScanner
org.apache.hadoop.hbase.regionserver.TestMemStore
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
org.apache.hadoop.hbase.regionserver.TestStore
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8035//console
This message is automatically generated.
> Unsynchronized scanning
> ---
>
> Key: HBASE-10060
> URL: https://issues.apache.org/jira/browse/HBASE-10060
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: 10060-trunk-v2.txt, 10060-trunk.txt
>
>
> HBASE-10015 has some lengthy discussion. The solution there ended up
> replacing synchronized with ReentrantLock, which - somewhat surprisingly -
> yielded a non-trivial improvement for tall tables.
> The goal should be to avoid locking in StoreScanner at all. StoreScanner is
> only accessed by a single thread *except* when we have a concurrent flush or
> a compaction, which is rare (we'd acquire and release the lock millions of
> times per second, and compact/flush a few time an hour at the most).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Issue Comment Deleted] (HBASE-9489) Add cp hooks in online merge before and after setting PONR
[ https://issues.apache.org/jira/browse/HBASE-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9489: -- Comment: was deleted (was: +1 nit: Few more occurrence of white spaces in RegionMergeTransaction. Pls remove them while committing.) > Add cp hooks in online merge before and after setting PONR > -- > > Key: HBASE-9489 > URL: https://issues.apache.org/jira/browse/HBASE-9489 > Project: HBase > Issue Type: Sub-task >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: HBASE-9489.patch, HBASE-9489_v2.patch, > HBASE-9489_v3.patch, HBASE-9489_v4.patch > > > As we need to merge index region along with user region we need the hooks > before and after setting PONR in region merge transtion. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9489) Add cp hooks in online merge before and after setting PONR
[ https://issues.apache.org/jira/browse/HBASE-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836003#comment-13836003 ] Anoop Sam John commented on HBASE-9489: --- +1 nit: Few more occurrence of white spaces in RegionMergeTransaction. Pls remove them while committing. > Add cp hooks in online merge before and after setting PONR > -- > > Key: HBASE-9489 > URL: https://issues.apache.org/jira/browse/HBASE-9489 > Project: HBase > Issue Type: Sub-task >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: HBASE-9489.patch, HBASE-9489_v2.patch, > HBASE-9489_v3.patch, HBASE-9489_v4.patch > > > As we need to merge index region along with user region we need the hooks > before and after setting PONR in region merge transtion. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9489) Add cp hooks in online merge before and after setting PONR
[ https://issues.apache.org/jira/browse/HBASE-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836002#comment-13836002 ] Anoop Sam John commented on HBASE-9489: --- +1 nit: Few more occurrence of white spaces in RegionMergeTransaction. Pls remove them while committing. > Add cp hooks in online merge before and after setting PONR > -- > > Key: HBASE-9489 > URL: https://issues.apache.org/jira/browse/HBASE-9489 > Project: HBase > Issue Type: Sub-task >Reporter: rajeshbabu >Assignee: rajeshbabu > Attachments: HBASE-9489.patch, HBASE-9489_v2.patch, > HBASE-9489_v3.patch, HBASE-9489_v4.patch > > > As we need to merge index region along with user region we need the hooks > before and after setting PONR in region merge transtion. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836022#comment-13836022 ] Nicolas Liochon commented on HBASE-9535: bq. You have diff of GC use? Not really. I've looked at the gc through the profile, it was around 4 minor gc/s with CMS, 1/s with G1, w/ or w/o the patch. I don't think the patch makes a difference in this scenario, as with YCSB puts we have a few large allocations, instead of ismultiple tiny ones. So for this scenario I think the benefit comes from the array copy suppressed. > Try a pool of direct byte buffers handling incoming ipc requests > > > Key: HBASE-9535 > URL: https://issues.apache.org/jira/browse/HBASE-9535 > Project: HBase > Issue Type: Brainstorming >Reporter: stack >Assignee: Nicolas Liochon > Attachments: 9535.v1.patch, 9535.v2-trunk.patch, 9535.v2.patch, > 9535.v3.patch > > > ipc takes in a query by allocating a ByteBuffer of the size of the request > and then reading off the socket into this on-heap BB. > Experiment with keeping a pool of BBs so we have some buffer reuse to cut on > garbage generated. Could checkout from pool in RpcServer#Reader. Could > check back into the pool when Handler is done just before it queues the > response on the Responder's queue. We should be good since, at least for > now, kvs get copied up into MSLAB (not references) when data gets stuffed > into MemStore; this should make it so no references left over when we check > the BB back into the pool for use next time around. > If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes > time so if already allocated, should be good. GC of DBBs is a pain but if in > a pool, we shouldn't be wanting this to happen). The copy from socket to the > DBB will be off-heap (should be fast). > Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by > size (we might want to bypass the pool if an object is > size N). > DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when > short-circuit reading. It'd be a bummer if we had to allocate big objects > on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-10061) TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in thrown NPE
Amit Sela created HBASE-10061: - Summary: TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in thrown NPE Key: HBASE-10061 URL: https://issues.apache.org/jira/browse/HBASE-10061 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.94.12 Reporter: Amit Sela TableMapReduceUtil.findOrCreateJar line 596: jar = getJar(my_class); updateMap(jar, packagedClasses); In case getJar returns null, updateMap will throw NPE. Should check null==jar before calling updateMap. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10061) TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in thrown NPE
[ https://issues.apache.org/jira/browse/HBASE-10061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836055#comment-13836055 ] Amit Sela commented on HBASE-10061: --- Currently, findOrCreateJar throws IOException if jar path is null or empty. Ted suggested to add a configuration parameter that will allow to choose between logging a missing jar (like it used to be in previous versions, i.e., 0.94.2) or throwing Exception (as it is now). > TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in > thrown NPE > -- > > Key: HBASE-10061 > URL: https://issues.apache.org/jira/browse/HBASE-10061 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.12 >Reporter: Amit Sela > > TableMapReduceUtil.findOrCreateJar line 596: > jar = getJar(my_class); > updateMap(jar, packagedClasses); > In case getJar returns null, updateMap will throw NPE. > Should check null==jar before calling updateMap. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836076#comment-13836076 ] Himanshu Vashishtha commented on HBASE-10048: - +1 to the idea of exposing log numbers, and thanks for the patch. (I'd avoid doing the "numLogs +1" stuff, as it looks brittle). IMO, it is OKAY to show the rolled WALs and leave out the current one. And, name the metrics accordingly. What do you think [~lshmouse]? [~zjushch]: Pardon for chiming in late; but couldn't just we make an intelligent "guess" about log size rather than creating a separate metrics for the same. https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java#L360 > Add hlog number metric in regionserver > -- > > Key: HBASE-10048 > URL: https://issues.apache.org/jira/browse/HBASE-10048 > Project: HBase > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, > HBASE-10048-trunk-v1.diff, HBASE-10048-trunk-v2.diff, > HBASE-10048-trunk-v3.diff > > > Add hlog number metric in regionserver. > We can use this metric to alert about memstore flush because of too many > hlogs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9955) Make hadoop2 the default and deprecate hadoop1
[ https://issues.apache.org/jira/browse/HBASE-9955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9955: - Attachment: 9955v2.txt Patch had trash in it. > Make hadoop2 the default and deprecate hadoop1 > -- > > Key: HBASE-9955 > URL: https://issues.apache.org/jira/browse/HBASE-9955 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack > Fix For: 0.98.0 > > Attachments: 9955.txt, 9955v2.txt > > > See "Hadoop version trunk dependency?" on the dev mailing ilst. Consensus > seems to be forming to do the subject line (Recheck the mail thread before > going ahead). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9955) Make hadoop2 the default and deprecate hadoop1
[
https://issues.apache.org/jira/browse/HBASE-9955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836118#comment-13836118
]
Hadoop QA commented on HBASE-9955:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616486/9955v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:red}-1 hadoop2.0{color}. The patch failed to compile against the
hadoop 2.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8036//console
This message is automatically generated.
> Make hadoop2 the default and deprecate hadoop1
> --
>
> Key: HBASE-9955
> URL: https://issues.apache.org/jira/browse/HBASE-9955
> Project: HBase
> Issue Type: Task
>Reporter: stack
>Assignee: stack
> Fix For: 0.98.0
>
> Attachments: 9955.txt, 9955v2.txt
>
>
> See "Hadoop version trunk dependency?" on the dev mailing ilst. Consensus
> seems to be forming to do the subject line (Recheck the mail thread before
> going ahead).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-7091) support custom GC options in hbase-env.sh
[
https://issues.apache.org/jira/browse/HBASE-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836126#comment-13836126
]
Jesse Yates commented on HBASE-7091:
Is this in trunk or 0.94 that you have a concern Nicolas? I don't think it
erases it - it just doesn't apply it multiple times.
bin/hbase does this kind of thing:
{code}
HBASE_OPTS="$HBASE_OPTS $CLIENT_GC_OPTS"
{code}
So it doesn't erase it - just copies it.
> support custom GC options in hbase-env.sh
> -
>
> Key: HBASE-7091
> URL: https://issues.apache.org/jira/browse/HBASE-7091
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.94.4
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Labels: newbie
> Fix For: 0.94.4, 0.95.0
>
> Attachments: hbase-7091-v1.patch
>
>
> When running things like bin/start-hbase and bin/hbase-daemon.sh start
> [master|regionserver|etc] we end up setting HBASE_OPTS property a couple
> times via calling hbase-env.sh. This is generally not a problem for most
> cases, but when you want to set your own GC log properties, one would think
> you should set HBASE_GC_OPTS, which get added to HBASE_OPTS.
> NOPE! That would make too much sense.
> Running bin/hbase-daemons.sh will run bin/hbase-daemon.sh with the daemons it
> needs to start. Each time through hbase-daemon.sh we also call bin/hbase.
> This isn't a big deal except for each call to hbase-daemon.sh, we also source
> hbase-env.sh twice (once in the script and once in bin/hbase). This is
> important for my next point.
> Note that to turn on GC logging, you uncomment:
> {code}
> # export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps $HBASE_GC_OPTS"
> {code}
> and then to log to a gc file for each server, you then uncomment:
> {code}
> # export HBASE_USE_GC_LOGFILE=true
> {code}
> in hbase-env.sh
> On the first pass through hbase-daemon.sh, HBASE_GC_OPTS isn't set, so
> HBASE_OPTS doesn't get anything funky, but we set HBASE_USE_GC_LOGFILE, which
> then sets HBASE_GC_OPTS to the log file (-Xloggc:...). Then in bin/hbase we
> again run hbase-env.sh, which now hs HBASE_GC_OPTS set, adding the GC file.
> This isn't a general problem because HBASE_OPTS is set without prefixing the
> existing HBASE_OPTS (eg. HBASE_OPTS="$HBASE_OPTS ..."), allowing easy
> updating. However, GC OPTS don't work the same and this is really odd
> behavior when you want to set your own GC opts, which can include turning on
> GC log rolling (yes, yes, they really are jvm opts, but they ought to support
> their own param, to help minimize clutter).
> The simple version of this patch will just add an idempotent GC option to
> hbase-env.sh and some comments that uncommenting
> {code}
> # export HBASE_USE_GC_LOGFILE=true
> {code}
> will lead to a custom gc log file per server (along with an example name), so
> you don't need to set "-Xloggc".
> The more complex solution does the above and also solves the multiple calls
> to hbase-env.sh so we can be sane about how all this works. Note that to fix
> this, hbase-daemon.sh just needs to read in HBASE_USE_GC_LOGFILE after
> sourcing hbase-env.sh and then update HBASE_OPTS. Oh and also not source
> hbase-env.sh in bin/hbase.
> Even further, we might want to consider adding options just for cases where
> we don't need gc logging - i.e. the shell, the config reading tool, hcbk,
> etc. This is the hardest version to handle since the first couple will
> willy-nilly apply the gc options.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836177#comment-13836177 ] chunhui shen commented on HBASE-10048: -- bq,couldn't just we make an intelligent "guess" about log size rather than creating a separate metrics for the same. I know the hlog size is about 95% of block size in normal case, but we have seen the hlog file size much bigger than block size in abnormal case. I think such a metric could show the unhealthy scenario timely, avoiding the hlog size become very huge. Maybe we only need the metric for the max size of current hlog. [~liushaohui] Any thought about the log size metric? > Add hlog number metric in regionserver > -- > > Key: HBASE-10048 > URL: https://issues.apache.org/jira/browse/HBASE-10048 > Project: HBase > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, > HBASE-10048-trunk-v1.diff, HBASE-10048-trunk-v2.diff, > HBASE-10048-trunk-v3.diff > > > Add hlog number metric in regionserver. > We can use this metric to alert about memstore flush because of too many > hlogs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-4811:
Description:
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
And the output is:
ddd/c1:q1/value1
ddd/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to keep
them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the discussions
anywhere about it or why it would be infeasible.
was:
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
And the output is:
ddd/c1:q1/value1
ddd/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
NOTE: when setting reversed as true for a client scan, you must set the start
row, else will throw exception. Through {@link
Scan#createBiggestByteArray(int)},you could get a big enough byte array as the
start row
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to keep
them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the discussions
anywhere about it or why it would be infeasible.
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.96.1, 0.94.15
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
> 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
> HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
> hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
> hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
> hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
> hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
> hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
> hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
> hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
> hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
>
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836180#comment-13836180
]
chunhui shen commented on HBASE-4811:
-
Commited to trunk,
Thanks very much for the review, Ted, Lars,Jean-Marc,Andrew and others
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.96.1, 0.94.15
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
> 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
> HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
> hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
> hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
> hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
> hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
> hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
> hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
> hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
> hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-4811:
Resolution: Fixed
Fix Version/s: (was: 0.94.15)
(was: 0.96.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Moving out of 0.94.15, wait lars to confirm when backport to 0.94
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
> 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
> HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
> hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
> hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
> hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
> hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
> hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
> hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
> hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
> hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9614) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9614: -- Issue Type: Bug (was: Sub-task) Parent: (was: HBASE-9613) > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9614 > URL: https://issues.apache.org/jira/browse/HBASE-9614 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: stack > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9615) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9615: -- Issue Type: Bug (was: Sub-task) Parent: (was: HBASE-9613) > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9615 > URL: https://issues.apache.org/jira/browse/HBASE-9615 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9628) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9628: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9628 > URL: https://issues.apache.org/jira/browse/HBASE-9628 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9615) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9615: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9615 > URL: https://issues.apache.org/jira/browse/HBASE-9615 > Project: HBase > Issue Type: Bug >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9627) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9627: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9627 > URL: https://issues.apache.org/jira/browse/HBASE-9627 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10054) Add the default column compression option
[ https://issues.apache.org/jira/browse/HBASE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836187#comment-13836187 ] chunhui shen commented on HBASE-10054: -- In the above comment, I mean test the compression in CreateTable if add the default comression. In modifyTable addColumnFamily events, should also add the default compression ? >From the patch, I see the config "hbase.regionserver.codecs", what's the >difference with the new config? > Add the default column compression option > - > > Key: HBASE-10054 > URL: https://issues.apache.org/jira/browse/HBASE-10054 > Project: HBase > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: HBASE-10054-0.94-v1.diff, HBASE-10054-trunk-v1.diff > > > Add the default column compression option for cluster level. > If users don't set column compression for a column family, we should use the > default column compression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9621) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9621: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9621 > URL: https://issues.apache.org/jira/browse/HBASE-9621 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9624) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9624: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9624 > URL: https://issues.apache.org/jira/browse/HBASE-9624 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9618) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9618: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9618 > URL: https://issues.apache.org/jira/browse/HBASE-9618 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9617) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9617: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9617 > URL: https://issues.apache.org/jira/browse/HBASE-9617 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9625) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9625: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9625 > URL: https://issues.apache.org/jira/browse/HBASE-9625 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9626) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9626: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9626 > URL: https://issues.apache.org/jira/browse/HBASE-9626 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9620) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9620: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9620 > URL: https://issues.apache.org/jira/browse/HBASE-9620 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9616) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9616: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9616 > URL: https://issues.apache.org/jira/browse/HBASE-9616 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9623) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9623: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9623 > URL: https://issues.apache.org/jira/browse/HBASE-9623 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9619) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9619: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9619 > URL: https://issues.apache.org/jira/browse/HBASE-9619 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9622) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9622: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9622 > URL: https://issues.apache.org/jira/browse/HBASE-9622 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9614) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9614: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9614 > URL: https://issues.apache.org/jira/browse/HBASE-9614 > Project: HBase > Issue Type: Bug >Reporter: Liang Xie >Assignee: stack > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9613) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9613: -- Affects Version/s: (was: 0.98.0) No patch, unscheduling. > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9613 > URL: https://issues.apache.org/jira/browse/HBASE-9613 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10049: --- Resolution: Fixed Status: Resolved (was: Patch Available) Seems to already be committed, resolving issue. > Small improvments in region_mover.rb > > > Key: HBASE-10049 > URL: https://issues.apache.org/jira/browse/HBASE-10049 > Project: HBase > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 0.98.0, 0.94.15, 0.96.2 > > Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, > HBASE-10049-trunk-v1.diff > > > We use region_mover.rb in the graceful upgrade of hbase cluster. > Here are small improvements. > a. remove the table.close(), because the htable could be reused. > b. Add more info in the log of moving region. > c. Add 20s sleep in load command to make sure the rs finished initialization > of rpc server. There is a time gap between rs startup report and rpc server > initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9613) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836192#comment-13836192 ] Liang Xie commented on HBASE-9613: -- [~apurtell], it had been done at HBASE-9630 already. The dup issues were created during a JIRA outage... > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9613 > URL: https://issues.apache.org/jira/browse/HBASE-9613 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10005:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed, resolving
> TestVisibilityLabels fails occasionally
> ---
>
> Key: HBASE-10005
> URL: https://issues.apache.org/jira/browse/HBASE-10005
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Ted Yu
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 0.98.0
>
> Attachments: HBASE-10005.patch, HBASE-10005_addendum.patch,
> org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
>
>
> I got the following test failures running test suite on hadoop-2 where
> distributed log replay was turned on :
> {code}
> testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
> Time elapsed: 0.019 sec <<< FAILURE!
> java.lang.AssertionError: The count should be 8 expected:<8> but was:<6>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
> ...
> testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
> Time elapsed: 0.002 sec <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
> {code}
> Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836196#comment-13836196 ] Andrew Purtell commented on HBASE-1: Should this be committed? > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0 > > Attachments: 1-recover-ts-with-pb-2.txt, > 1-recover-ts-with-pb-2.txt, 1-v1.txt, 1-v4.txt, 1-v5.txt, > 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9991) Enhance test-patch.sh with post back of snippet of compilation error if any is detected
[
https://issues.apache.org/jira/browse/HBASE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-9991:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Already committed, resolving.
> Enhance test-patch.sh with post back of snippet of compilation error if any
> is detected
> ---
>
> Key: HBASE-9991
> URL: https://issues.apache.org/jira/browse/HBASE-9991
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9991-v1.txt, 9991-v2.txt, 9991.addendum
>
>
> Currently test-patch.sh would post back something like this:
> {code}
> -1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
> {code}
> Snippet of the compilation error should be included so that people can see
> what caused the error.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9994) ZeroCopyLiteralByteString.wrap() should be used in place of ByteString.copyFrom()
[ https://issues.apache.org/jira/browse/HBASE-9994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9994: -- Resolution: Fixed Status: Resolved (was: Patch Available) Already committed, resolving. > ZeroCopyLiteralByteString.wrap() should be used in place of > ByteString.copyFrom() > - > > Key: HBASE-9994 > URL: https://issues.apache.org/jira/browse/HBASE-9994 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0 > > Attachments: 9994-v1.txt > > > The following classes use ByteString.copyFrom() which should be replaced with > ZeroCopyLiteralByteString.wrap() : > hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java > hbase-server/src/main/java/org/apache/hadoop/hbase/codec/MessageCodec.java -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9978) The client retries even if the method is not present on the server
[ https://issues.apache.org/jira/browse/HBASE-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836203#comment-13836203 ] Andrew Purtell commented on HBASE-9978: --- You going to commit [~mbertozzi] ? > The client retries even if the method is not present on the server > -- > > Key: HBASE-9978 > URL: https://issues.apache.org/jira/browse/HBASE-9978 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.98.0, 0.96.1 > > Attachments: HBASE-9978-v0.patch > > > If the RpcServer is not able to find the method on the server throws an > UnsupportedOperationException, but since is not wrapped in a DoNotRetry the > client keeps retrying even if the operation doesn't exists. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each "newCaller"
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9990: -- Fix Version/s: (was: 0.96.1) (was: 0.98.0) Status: Open (was: Patch Available) Stale patch, cancelling and unscheduling. > HTable uses the conf for each "newCaller" > - > > Key: HBASE-9990 > URL: https://issues.apache.org/jira/browse/HBASE-9990 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0, 0.98.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Attachments: 9990.v1.patch > > > You can construct a RpcRetryingCallerFactory, but actually the conf is read > for each caller creation. Reading the conf is obviously expensive, and a > profiling session shows it. If we want to sent hundreds of thousands of > queries per second, we should not do that. > RpcRetryingCallerFactory.newCaller is called for each get, for example. > This is not a regression, we have something similar in 0.94. > On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after > a few thousand calls to "get". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836204#comment-13836204 ] Andrew Purtell commented on HBASE-9969: --- This should either be committed or unscheduled. > Improve KeyValueHeap using loser tree > - > > Key: HBASE-9969 > URL: https://issues.apache.org/jira/browse/HBASE-9969 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Chao Shi >Assignee: Chao Shi > Fix For: 0.98.0, 0.96.1 > > Attachments: 9969-0.94.txt, KeyValueHeapBenchmark_v1.ods, > KeyValueHeapBenchmark_v2.ods, hbase-9969-pq-v1.patch, hbase-9969-pq-v2.patch, > hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, > kvheap-benchmark.png, kvheap-benchmark.txt > > > LoserTree is the better data structure than binary heap. It saves half of the > comparisons on each next(), though the time complexity is on O(logN). > Currently A scan or get will go through two KeyValueHeaps, one is merging KVs > read from multiple HFiles in a single store, the other is merging results > from multiple stores. This patch should improve the both cases whenever CPU > is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). > All of the optimization work is done in KeyValueHeap and does not change its > public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836205#comment-13836205 ] Andrew Purtell commented on HBASE-9884: --- Looked at the REST parts of the patch, +1 > Add Thrift and REST support for Visibility Labels > - > > Key: HBASE-9884 > URL: https://issues.apache.org/jira/browse/HBASE-9884 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 0.98.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.98.0 > > Attachments: HBASE-9884.patch, HBASE-9884_1.patch, HBASE-9884_2.patch > > > In HBASE-7663 the REST and thrift support has been seperated out because the > patch is becoming bigger. This JIRA is to add the Thrift and REST part as a > seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836207#comment-13836207 ] Ted Yu commented on HBASE-1: This patch touches important code path. More review is welcome. > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0 > > Attachments: 1-recover-ts-with-pb-2.txt, > 1-recover-ts-with-pb-2.txt, 1-v1.txt, 1-v4.txt, 1-v5.txt, > 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9863) Intermittently TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs in admin#createTable() call
[
https://issues.apache.org/jira/browse/HBASE-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-9863:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Already committed, resolving.
> Intermittently
> TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry hangs in
> admin#createTable() call
> ---
>
> Key: HBASE-9863
> URL: https://issues.apache.org/jira/browse/HBASE-9863
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9863-v1.txt, 9863-v2.txt, 9863-v3.txt, 9863-v4.txt,
> 9863-v5.txt, 9863-v6.txt
>
>
> TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry sometimes
> hung.
> Here were two recent occurrences:
> https://builds.apache.org/job/PreCommit-HBASE-Build/7676/console
> https://builds.apache.org/job/PreCommit-HBASE-Build/7671/console
> There were 9 occurrences of the following in both stack traces:
> {code}
> "FifoRpcScheduler.handler1-thread-5" daemon prio=10 tid=0x09df8800 nid=0xc17
> waiting for monitor entry [0x6fdf8000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
> - waiting to lock <0x7f69b5f0> (a
> org.apache.hadoop.hbase.master.TableNamespaceManager)
> at
> org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3146)
> at
> org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3105)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1743)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1782)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38221)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1983)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
> {code}
> The test hung here:
> {code}
> "pool-1-thread-1" prio=10 tid=0x74f7b800 nid=0x5aa5 in Object.wait()
> [0x74efe000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xcc848348> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1436)
> - locked <0xcc848348> (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.createTable(MasterProtos.java:40372)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.createTable(HConnectionManager.java:1931)
> at org.apache.hadoop.hbase.client.HBaseAdmin$2.call(HBaseAdmin.java:598)
> at org.apache.hadoop.hbase.client.HBaseAdmin$2.call(HBaseAdmin.java:594)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
> - locked <0x7faa26d0> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
> - locked <0x7faa26d0> (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3124)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:594)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:485)
> at
> org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:486)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9856) Fix some findbugs Performance Warnings
[
https://issues.apache.org/jira/browse/HBASE-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836210#comment-13836210
]
Andrew Purtell commented on HBASE-9856:
---
Should this be committed?
> Fix some findbugs Performance Warnings
> --
>
> Key: HBASE-9856
> URL: https://issues.apache.org/jira/browse/HBASE-9856
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 9856-v1.txt
>
>
> These are the warnings to be fixed:
> {code}
> SIC Should org.apache.hadoop.hbase.regionserver.HRegion$RowLock be a _static_
> inner class?
> UPM Private method
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(String,
> String, Permission$Action[]) is never called
> WMI Method
> org.apache.hadoop.hbase.regionserver.wal.WALEditsReplaySink.replayEntries(List)
> makes inefficient use of keySet iterator instead of entrySet iterator
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9804) Startup option for holding user table deployment
[ https://issues.apache.org/jira/browse/HBASE-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9804: -- Affects Version/s: (was: 0.98.0) > Startup option for holding user table deployment > > > Key: HBASE-9804 > URL: https://issues.apache.org/jira/browse/HBASE-9804 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Purtell >Priority: Minor > > Introduce a boolean configuration option, false by default, that if set to > 'true' will cause the master to set all user tables to disabled state at > startup. From there, individual tables can be onlined as normal. Add a new > admin method HBA#enableAll for convenience, also a new HBA#disableAll for > symmetry. Add shell support for sending those new admin commands. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9856) Fix some findbugs Performance Warnings
[
https://issues.apache.org/jira/browse/HBASE-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9856:
--
Attachment: 9856-v2.txt
> Fix some findbugs Performance Warnings
> --
>
> Key: HBASE-9856
> URL: https://issues.apache.org/jira/browse/HBASE-9856
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 9856-v1.txt, 9856-v2.txt
>
>
> These are the warnings to be fixed:
> {code}
> SIC Should org.apache.hadoop.hbase.regionserver.HRegion$RowLock be a _static_
> inner class?
> UPM Private method
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(String,
> String, Permission$Action[]) is never called
> WMI Method
> org.apache.hadoop.hbase.regionserver.wal.WALEditsReplaySink.replayEntries(List)
> makes inefficient use of keySet iterator instead of entrySet iterator
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9856) Fix some findbugs Performance Warnings
[
https://issues.apache.org/jira/browse/HBASE-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836212#comment-13836212
]
Ted Yu commented on HBASE-9856:
---
The change in AccessController.java is no longer needed.
Integrated to trunk.
> Fix some findbugs Performance Warnings
> --
>
> Key: HBASE-9856
> URL: https://issues.apache.org/jira/browse/HBASE-9856
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 9856-v1.txt, 9856-v2.txt
>
>
> These are the warnings to be fixed:
> {code}
> SIC Should org.apache.hadoop.hbase.regionserver.HRegion$RowLock be a _static_
> inner class?
> UPM Private method
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(String,
> String, Permission$Action[]) is never called
> WMI Method
> org.apache.hadoop.hbase.regionserver.wal.WALEditsReplaySink.replayEntries(List)
> makes inefficient use of keySet iterator instead of entrySet iterator
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9978) The client retries even if the method is not present on the server
[ https://issues.apache.org/jira/browse/HBASE-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836214#comment-13836214 ] Elliott Clark commented on HBASE-9978: -- +1 > The client retries even if the method is not present on the server > -- > > Key: HBASE-9978 > URL: https://issues.apache.org/jira/browse/HBASE-9978 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 0.98.0, 0.96.1 > > Attachments: HBASE-9978-v0.patch > > > If the RpcServer is not able to find the method on the server throws an > UnsupportedOperationException, but since is not wrapped in a DoNotRetry the > client keeps retrying even if the operation doesn't exists. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9762) PutCombiners should add the OperationAttributes if present
[ https://issues.apache.org/jira/browse/HBASE-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836213#comment-13836213 ] Andrew Purtell commented on HBASE-9762: --- So sounds like we need OperationAttributeCombiners for the different attributes? > PutCombiners should add the OperationAttributes if present > -- > > Key: HBASE-9762 > URL: https://issues.apache.org/jira/browse/HBASE-9762 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0, 0.95.2, 0.94.12, 0.96.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-9762.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-9856) Fix some findbugs Performance Warnings
[
https://issues.apache.org/jira/browse/HBASE-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-9856.
---
Resolution: Fixed
Hadoop Flags: Reviewed
> Fix some findbugs Performance Warnings
> --
>
> Key: HBASE-9856
> URL: https://issues.apache.org/jira/browse/HBASE-9856
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 9856-v1.txt, 9856-v2.txt
>
>
> These are the warnings to be fixed:
> {code}
> SIC Should org.apache.hadoop.hbase.regionserver.HRegion$RowLock be a _static_
> inner class?
> UPM Private method
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(String,
> String, Permission$Action[]) is never called
> WMI Method
> org.apache.hadoop.hbase.regionserver.wal.WALEditsReplaySink.replayEntries(List)
> makes inefficient use of keySet iterator instead of entrySet iterator
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-9754:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
This has been committed to trunk, resolving.
Reopen as necessary when committing to other branches.
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94-v2.txt, 9754-0.94.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt,
> 9754-rp-hregion.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9721) meta assignment did not timeout
[
https://issues.apache.org/jira/browse/HBASE-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-9721:
--
Affects Version/s: 0.98.0
0.96.0
Fix Version/s: (was: 0.96.1)
(was: 0.98.0)
> meta assignment did not timeout
> ---
>
> Key: HBASE-9721
> URL: https://issues.apache.org/jira/browse/HBASE-9721
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> On a test cluster, this following events happened with ITBLL and CM leading
> to meta being unavailable until master is restarted.
> An RS carrying meta died, and master assigned the region to one of the RSs.
> {code}
> 2013-10-03 23:30:06,611 INFO
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-1]
> master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to
> gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820
> 2013-10-03 23:30:06,611 INFO
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-1]
> master.RegionStates: Transitioned {1588230740 state=OFFLINE,
> ts=1380843006601, server=null} to {1588230740 state=PENDING_OPEN,
> ts=1380843006611,
> server=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820}
> 2013-10-03 23:30:06,611 DEBUG
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-1]
> master.ServerManager: New admin connection to
> gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820
> {code}
> At the same time, the RS that meta recently got assigned also died (due to
> CM), and restarted:
> {code}
> 2013-10-03 23:30:07,636 DEBUG [RpcServer.handler=17,port=6]
> master.ServerManager: REPORT: Server
> gs-hdp2-secure-1380781860-hbase-8.cs1cloud.internal,60020,1380843002494 came
> back up, removed it from the dead servers list
> 2013-10-03 23:30:08,769 INFO [RpcServer.handler=18,port=6]
> master.ServerManager: Triggering server recovery; existingServer
> gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820 looks
> stale, new
> server:gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380843006362
> 2013-10-03 23:30:08,771 DEBUG [RpcServer.handler=18,port=6]
> master.AssignmentManager: Checking region=hbase:meta,,1.1588230740, zk
> server=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820
>
> current=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820,
> matches=true
> 2013-10-03 23:30:08,771 DEBUG [RpcServer.handler=18,port=6]
> master.ServerManager:
> Added=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820
> to dead servers, submitted shutdown handler to be executed meta=true
> 2013-10-03 23:30:08,771 INFO [RpcServer.handler=18,port=6]
> master.ServerManager: Registering
> server=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380843006362
> 2013-10-03 23:30:08,772 INFO
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-2]
> handler.MetaServerShutdownHandler: Splitting hbase:meta logs for
> gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820
> {code}
> AM/SSH sees that the RS that died was carrying meta, but the assignment RPC
> request was still not sent:
> {code}
> 2013-10-03 23:30:08,791 DEBUG
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-2]
> master.AssignmentManager: Checking region=hbase:meta,,1.1588230740, zk
> server=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820
>
> current=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820,
> matches=true
> 2013-10-03 23:30:08,791 INFO
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-2]
> handler.MetaServerShutdownHandler: Server
> gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820 was
> carrying META. Trying to assign.
> 2013-10-03 23:30:08,791 DEBUG
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-2]
> master.RegionStates: Offline 1588230740 with current state=PENDING_OPEN,
> expected state=OFFLINE/SPLITTING/MERGING
> 2013-10-03 23:30:08,791 INFO
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-2]
> master.RegionStates: Transitioned {1588230740 state=PENDING_OPEN,
> ts=1380843006611,
> server=gs-hdp2-secure-1380781860-hbase-5.cs1cloud.internal,60020,1380842900820}
> to {1588230740 state=OFFLINE, ts=1380843008791, server=null}
> 2013-10-03 23:30:09,809 INFO
> [MASTER_META_SERVER_OPERATIONS-gs-hdp2-secure-1380781860-hbase-12:6-2]
> zookeeper.ZooKeeperNodeTracker: Unsetting hbase:meta region location in
> ZooKeeper
> {code}
> Our first attempt at the assig
[jira] [Commented] (HBASE-10061) TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in thrown NPE
[ https://issues.apache.org/jira/browse/HBASE-10061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836236#comment-13836236 ] Anoop Sam John commented on HBASE-10061: bq.(like it used to be in previous versions, i.e., 0.94.2) You mean this behavior is changed from 94.2? Know which issue done that change any why. Trying to understand the reason for such a change if any. > TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in > thrown NPE > -- > > Key: HBASE-10061 > URL: https://issues.apache.org/jira/browse/HBASE-10061 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.12 >Reporter: Amit Sela > > TableMapReduceUtil.findOrCreateJar line 596: > jar = getJar(my_class); > updateMap(jar, packagedClasses); > In case getJar returns null, updateMap will throw NPE. > Should check null==jar before calling updateMap. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[ https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836234#comment-13836234 ] Hudson commented on HBASE-4811: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #859 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/859/]) HBASE-4811 Support reverse Scan (zjushch: rev 1546878) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/Filter.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonReversedNonLazyKeyValueScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedKeyValueHeap.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedRegionScannerImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/NoOpScanPolicyObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java > Support reverse Scan > > > Key: HBASE-4811 > URL: https://issues.apache.org/jira/browse/HBASE-4811 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 0.20.6, 0.94.7 >Reporter: John Carrino >Assignee: chunhui shen > Fix For: 0.98.0 > > Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt, > 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch, > HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94 > v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch, > hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch, > hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch, > hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch, > hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch, > hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch, > hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch, > hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch, > hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch, > hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch, > hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch, > hbase-481
[jira] [Updated] (HBASE-9718) Add a test scope dependency on org.slf4j:slf4j-api to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9718: -- Priority: Minor (was: Major) > Add a test scope dependency on org.slf4j:slf4j-api to hbase-client > -- > > Key: HBASE-9718 > URL: https://issues.apache.org/jira/browse/HBASE-9718 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.98.0, 0.96.1 > > Attachments: 9718.patch > > > hbase-client needs a test scope dependency on org.slf4j:slf4j-api in its POM. > Without this change at least Eclipse cannot resolve org.slf4j.Logger from > RecoverableZooKeeper - the ZooKeeper classes use it - and so the > 'hbase-client' project will not build. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-9718) Add a test scope dependency on org.slf4j:slf4j-api to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-9718. --- Resolution: Fixed Fix Version/s: 0.96.1 0.98.0 Hadoop Flags: Reviewed I still see this when setting up new Eclipse projects or regenerating them with 'mvn eclipse:eclipse'. Confirmed again the patch fixes the problem. Committed to trunk and 0.96 branch. > Add a test scope dependency on org.slf4j:slf4j-api to hbase-client > -- > > Key: HBASE-9718 > URL: https://issues.apache.org/jira/browse/HBASE-9718 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.0, 0.96.1 > > Attachments: 9718.patch > > > hbase-client needs a test scope dependency on org.slf4j:slf4j-api in its POM. > Without this change at least Eclipse cannot resolve org.slf4j.Logger from > RecoverableZooKeeper - the ZooKeeper classes use it - and so the > 'hbase-client' project will not build. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9631) add murmur3 hash
[ https://issues.apache.org/jira/browse/HBASE-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9631: -- Resolution: Fixed Fix Version/s: 0.98.0 Release Note: This change provides support for MurmurHash3, the successor to MurmurHash2. It is not selected by default but can be used by changing the configuration setting "hbase.hash.type" to "murmur3". For new installs only. You must not make this change to an existing cluster. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch [~xieliang007] > add murmur3 hash > > > Key: HBASE-9631 > URL: https://issues.apache.org/jira/browse/HBASE-9631 > Project: HBase > Issue Type: New Feature > Components: util >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 0.98.0 > > Attachments: HBase-9631-v2.txt, HBase-9631.txt > > > MurmurHash3 is the successor to MurmurHash2. It comes in 3 variants - a > 32-bit version that targets low latency for hash table use and two 128-bit > versions for generating unique identifiers for large blocks of data, one each > for x86 and x64 platforms. > several open source projects have added murmur3 already, like cassandra, > mahout, etc. > I just port the murmur3 from MAHOUT-862. due to compatibility, let's keep the > default Hash algo(murmur2) without changing. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9502) HStore.seekToScanner should handle magic value
[ https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836238#comment-13836238 ] Andrew Purtell commented on HBASE-9502: --- HBASE-9518 has been committed. Based on the above discussion and HadoopQA results, this patch is good to go. The patch applies with some fuzz. I'm going to regenerate it and submit it to HadoopQA again for another round of tests against current trunk. Assuming to commit afterward. > HStore.seekToScanner should handle magic value > -- > > Key: HBASE-9502 > URL: https://issues.apache.org/jira/browse/HBASE-9502 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 0.98.0, 0.95.2, 0.96.1 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HBASE-9502-v2.txt, HBASE-9502.txt > > > due to faked key, the seekTo probably reture -2, and HStore.seekToScanner > should handle this corner case. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9502) HStore.seekToScanner should handle magic value
[ https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9502: -- Status: Patch Available (was: Open) > HStore.seekToScanner should handle magic value > -- > > Key: HBASE-9502 > URL: https://issues.apache.org/jira/browse/HBASE-9502 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 0.98.0, 0.96.1 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: 9502-v2.patch, HBASE-9502-v2.txt, HBASE-9502.txt > > > due to faked key, the seekTo probably reture -2, and HStore.seekToScanner > should handle this corner case. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9502) HStore.seekToScanner should handle magic value
[ https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9502: -- Affects Version/s: (was: 0.95.2) Status: Open (was: Patch Available) > HStore.seekToScanner should handle magic value > -- > > Key: HBASE-9502 > URL: https://issues.apache.org/jira/browse/HBASE-9502 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 0.98.0, 0.96.1 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: 9502-v2.patch, HBASE-9502-v2.txt, HBASE-9502.txt > > > due to faked key, the seekTo probably reture -2, and HStore.seekToScanner > should handle this corner case. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[ https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836239#comment-13836239 ] Hudson commented on HBASE-4811: --- SUCCESS: Integrated in HBase-TRUNK #4705 (See [https://builds.apache.org/job/HBase-TRUNK/4705/]) HBASE-4811 Support reverse Scan (zjushch: rev 1546878) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/Filter.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonReversedNonLazyKeyValueScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedKeyValueHeap.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedRegionScannerImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/NoOpScanPolicyObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java > Support reverse Scan > > > Key: HBASE-4811 > URL: https://issues.apache.org/jira/browse/HBASE-4811 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 0.20.6, 0.94.7 >Reporter: John Carrino >Assignee: chunhui shen > Fix For: 0.98.0 > > Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt, > 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch, > HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94 > v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch, > hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch, > hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch, > hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch, > hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch, > hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch, > hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch, > hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch, > hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch, > hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch, > hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch, > hbase-4811-trunkv9.patch > > > Reversed
[jira] [Updated] (HBASE-9502) HStore.seekToScanner should handle magic value
[ https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9502: -- Attachment: 9502-v2.patch > HStore.seekToScanner should handle magic value > -- > > Key: HBASE-9502 > URL: https://issues.apache.org/jira/browse/HBASE-9502 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 0.98.0, 0.96.1 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: 9502-v2.patch, HBASE-9502-v2.txt, HBASE-9502.txt > > > due to faked key, the seekTo probably reture -2, and HStore.seekToScanner > should handle this corner case. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836285#comment-13836285 ] Lars Hofhansl edited comment on HBASE-9969 at 12/2/13 6:36 AM: --- This is not ready to be committed. In 0.94 I move issues like these to the next point release if I think they are generally useful for 0.94 but not quite ready yet, otherwise I unschedule them. Unscheduled issues tend to be forgotten. was (Author: lhofhansl): This is not ready to be committed. In 0.94 I generally move issues like these to the next point release if I think they are generally useful for 0.94 but not quite ready yet, otherwise I unschedule them. Unscheduled issues tend to be forgotten. > Improve KeyValueHeap using loser tree > - > > Key: HBASE-9969 > URL: https://issues.apache.org/jira/browse/HBASE-9969 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Chao Shi >Assignee: Chao Shi > Fix For: 0.98.0, 0.96.1 > > Attachments: 9969-0.94.txt, KeyValueHeapBenchmark_v1.ods, > KeyValueHeapBenchmark_v2.ods, hbase-9969-pq-v1.patch, hbase-9969-pq-v2.patch, > hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, > kvheap-benchmark.png, kvheap-benchmark.txt > > > LoserTree is the better data structure than binary heap. It saves half of the > comparisons on each next(), though the time complexity is on O(logN). > Currently A scan or get will go through two KeyValueHeaps, one is merging KVs > read from multiple HFiles in a single store, the other is merging results > from multiple stores. This patch should improve the both cases whenever CPU > is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). > All of the optimization work is done in KeyValueHeap and does not change its > public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836285#comment-13836285 ] Lars Hofhansl commented on HBASE-9969: -- This is not ready to be committed. In 0.94 I generally move issues like these to the next point release if I think they are generally useful for 0.94 but not quite ready yet, otherwise I unschedule them. Unscheduled issues tend to be forgotten. > Improve KeyValueHeap using loser tree > - > > Key: HBASE-9969 > URL: https://issues.apache.org/jira/browse/HBASE-9969 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Chao Shi >Assignee: Chao Shi > Fix For: 0.98.0, 0.96.1 > > Attachments: 9969-0.94.txt, KeyValueHeapBenchmark_v1.ods, > KeyValueHeapBenchmark_v2.ods, hbase-9969-pq-v1.patch, hbase-9969-pq-v2.patch, > hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, > kvheap-benchmark.png, kvheap-benchmark.txt > > > LoserTree is the better data structure than binary heap. It saves half of the > comparisons on each next(), though the time complexity is on O(logN). > Currently A scan or get will go through two KeyValueHeaps, one is merging KVs > read from multiple HFiles in a single store, the other is merging results > from multiple stores. This patch should improve the both cases whenever CPU > is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). > All of the optimization work is done in KeyValueHeap and does not change its > public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9502) HStore.seekToScanner should handle magic value
[
https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836286#comment-13836286
]
Hadoop QA commented on HBASE-9502:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616502/9502-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8037//console
This message is automatically generated.
> HStore.seekToScanner should handle magic value
> --
>
> Key: HBASE-9502
> URL: https://issues.apache.org/jira/browse/HBASE-9502
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Affects Versions: 0.98.0, 0.96.1
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: 9502-v2.patch, HBASE-9502-v2.txt, HBASE-9502.txt
>
>
> due to faked key, the seekTo probably reture -2, and HStore.seekToScanner
> should handle this corner case.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-1: --- Fix Version/s: (was: 0.98.0) Moving out of 0.98 > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 1-recover-ts-with-pb-2.txt, > 1-recover-ts-with-pb-2.txt, 1-v1.txt, 1-v4.txt, 1-v5.txt, > 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836316#comment-13836316
]
Liang Xie commented on HBASE-4811:
--
Glad to see it was in trunk now ! will revist our internal reverse scan bug
fixed list to verify weather existing in trunk version or not, possible need to
port a bit.
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
> 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
> HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
> hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
> hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
> hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
> hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
> hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
> hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
> hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
> hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836320#comment-13836320 ] Andrew Purtell commented on HBASE-9969: --- Ok. Moved out of 0.98.0. > Improve KeyValueHeap using loser tree > - > > Key: HBASE-9969 > URL: https://issues.apache.org/jira/browse/HBASE-9969 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Chao Shi >Assignee: Chao Shi > Fix For: 0.96.1, 0.98.1 > > Attachments: 9969-0.94.txt, KeyValueHeapBenchmark_v1.ods, > KeyValueHeapBenchmark_v2.ods, hbase-9969-pq-v1.patch, hbase-9969-pq-v2.patch, > hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, > kvheap-benchmark.png, kvheap-benchmark.txt > > > LoserTree is the better data structure than binary heap. It saves half of the > comparisons on each next(), though the time complexity is on O(logN). > Currently A scan or get will go through two KeyValueHeaps, one is merging KVs > read from multiple HFiles in a single store, the other is merging results > from multiple stores. This patch should improve the both cases whenever CPU > is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). > All of the optimization work is done in KeyValueHeap and does not change its > public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9613) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836322#comment-13836322 ] Andrew Purtell commented on HBASE-9613: --- Thanks [~xieliang007]. I will delete this one also. The issues are definitely weird, they don't allow me to resolve them, I can only delete them. > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9613 > URL: https://issues.apache.org/jira/browse/HBASE-9613 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9969: -- Fix Version/s: (was: 0.98.0) 0.98.1 > Improve KeyValueHeap using loser tree > - > > Key: HBASE-9969 > URL: https://issues.apache.org/jira/browse/HBASE-9969 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Chao Shi >Assignee: Chao Shi > Fix For: 0.96.1, 0.98.1 > > Attachments: 9969-0.94.txt, KeyValueHeapBenchmark_v1.ods, > KeyValueHeapBenchmark_v2.ods, hbase-9969-pq-v1.patch, hbase-9969-pq-v2.patch, > hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, > kvheap-benchmark.png, kvheap-benchmark.txt > > > LoserTree is the better data structure than binary heap. It saves half of the > comparisons on each next(), though the time complexity is on O(logN). > Currently A scan or get will go through two KeyValueHeaps, one is merging KVs > read from multiple HFiles in a single store, the other is merging results > from multiple stores. This patch should improve the both cases whenever CPU > is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). > All of the optimization work is done in KeyValueHeap and does not change its > public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Deleted] (HBASE-9613) Add thread which detects JVM pauses like HADOOP's
[ https://issues.apache.org/jira/browse/HBASE-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell deleted HBASE-9613: -- > Add thread which detects JVM pauses like HADOOP's > - > > Key: HBASE-9613 > URL: https://issues.apache.org/jira/browse/HBASE-9613 > Project: HBase > Issue Type: New Feature >Reporter: Liang Xie >Assignee: Liang Xie > > Todd adds daemon threads for dn&nn to indicate the VM or kernel caused pause > in application log, it's pretty handy for diagnose, i thought it's great to > have similar ability in HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836332#comment-13836332
]
Lars Hofhansl commented on HBASE-4811:
--
Cool.
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
> 4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
> HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
> hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
> hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
> hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
> hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
> hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
> hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
> hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
> hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-10062) Store the encrypted data length in the block encryption header instead of plaintext length
Andrew Purtell created HBASE-10062:
--
Summary: Store the encrypted data length in the block encryption
header instead of plaintext length
Key: HBASE-10062
URL: https://issues.apache.org/jira/browse/HBASE-10062
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.98.0
After HBASE-7544, if an HFile belongs to an encrypted family, it is encrypted
on a per block basis. The encrypted blocks include the following header:
{noformat}
// +--+
// | vint plaintext length|
// +--+
// | vint iv length |
// +--+
// | iv data ... |
// +--+
// | encrypted block data ... |
// +--+
{noformat}
The reason for storing the plaintext length is so we can create an encryption
stream over the encrypted block data and, no matter the internal details of the
crypto algorithm (whether it adds padding, etc.) after reading the expected
plaintext bytes we know the reader is finished. However my colleague Jerry Chen
pointed out today this construction mandates the block be processed exactly
that way. Storing and using the encrypted data length instead could provide
more implementation flexibility down the road.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10062) Store the encrypted data length in the block encryption header instead of plaintext length
[
https://issues.apache.org/jira/browse/HBASE-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10062:
---
Description:
After HBASE-7544, if an HFile belongs to an encrypted family, it is encrypted
on a per block basis. The encrypted blocks include the following header:
{noformat}
// +--+
// | vint plaintext length|
// +--+
// | vint iv length |
// +--+
// | iv data ... |
// +--+
// | encrypted block data ... |
// +--+
{noformat}
The reason for storing the plaintext length is so we can create an decryption
stream over the encrypted block data and, no matter the internal details of the
crypto algorithm (whether it adds padding, etc.) after reading the expected
plaintext bytes we know the reader is finished. However my colleague Jerry Chen
pointed out today this construction mandates the block be processed exactly
that way. Storing and using the encrypted data length instead could provide
more implementation flexibility down the road.
was:
After HBASE-7544, if an HFile belongs to an encrypted family, it is encrypted
on a per block basis. The encrypted blocks include the following header:
{noformat}
// +--+
// | vint plaintext length|
// +--+
// | vint iv length |
// +--+
// | iv data ... |
// +--+
// | encrypted block data ... |
// +--+
{noformat}
The reason for storing the plaintext length is so we can create an encryption
stream over the encrypted block data and, no matter the internal details of the
crypto algorithm (whether it adds padding, etc.) after reading the expected
plaintext bytes we know the reader is finished. However my colleague Jerry Chen
pointed out today this construction mandates the block be processed exactly
that way. Storing and using the encrypted data length instead could provide
more implementation flexibility down the road.
> Store the encrypted data length in the block encryption header instead of
> plaintext length
> --
>
> Key: HBASE-10062
> URL: https://issues.apache.org/jira/browse/HBASE-10062
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.0
>
>
> After HBASE-7544, if an HFile belongs to an encrypted family, it is encrypted
> on a per block basis. The encrypted blocks include the following header:
> {noformat}
> // +--+
> // | vint plaintext length|
> // +--+
> // | vint iv length |
> // +--+
> // | iv data ... |
> // +--+
> // | encrypted block data ... |
> // +--+
> {noformat}
> The reason for storing the plaintext length is so we can create an decryption
> stream over the encrypted block data and, no matter the internal details of
> the crypto algorithm (whether it adds padding, etc.) after reading the
> expected plaintext bytes we know the reader is finished. However my colleague
> Jerry Chen pointed out today this construction mandates the block be
> processed exactly that way. Storing and using the encrypted data length
> instead could provide more implementation flexibility down the road.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
