[jira] [Commented] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985218#comment-13985218 ] stack commented on HBASE-10403: --- +1 on notion; especially stuff like a percentage being applied to the XX:MaxDirectMemorySize for BC use. We'll need to make sure the old configs continue to work after instituting these new ones. On there being less options, I was going to suggest just drop slab cache? That'd be one less option. i've not looked closely at it but going by Nick blog post numbers, it does more work to get to same place at least when using DoubleBlockCache...and for CombinedBlockCache, BucketCache has more options so lets just concentrate on making it work and easy to use and just let go of SlabCache? On the other hand, it is looking like there'll be some more BC' implementations to play with before there will be less (HBASE-8894 and if we get around to acting on [~vrodionov] suggestions regards cache) so we should just deal but let us let go of options that have been superseded. Agree w/ [~zjushch] comments that exposing CBC and DBC will only confuse (Strike DBC since you can set up BucketCache to do L1/L2 as DBC does?). Maybe have configs: hbase.block.cache.l1=LruBlockCache hbase.block.cache.l2=BucketCache hbase.block.cache.policy=CBC ... or something like that. Thinking on it, even simpler is just have hbase.block.cache.impl and pass LRUBC, or BucketCache, or CBC (they all implement BlockCache interface?) Add a new class if you want to do a new combination. Keeps it simple. On the patch: Looks good. I like the refactorings. We should refactor that setVictim thing... its awkward. Hack. +1 moving bucket cache configs into bucket cache class. Restore the old param below: - * @param direct true if allocate direct buffer + * @param isDirect true if allocate isDirect buffer (isDirect is a method name, not a param name) +1 on moving all the sizing into the class the size applies to. Patch is great. Only thing to decide is what for configs? We have to support the old. > Simplify offheap cache configuration > > > Key: HBASE-10403 > URL: https://issues.apache.org/jira/browse/HBASE-10403 > Project: HBase > Issue Type: Bug > Components: io >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: HBASE-10403.0.patch > > > The BucketCache (HBASE-7404) is a very nice piece of functionality which is > hidden behind complex configuration. Enabling it currently requires manual > calculation of L1 cache. It'd be nice to make this easier to use and conform > better with the existing heap management tools we already have. > Turning it on currently requires explicitly setting LruBlockCache (L1) > instance size and IOEngine (L2) size, making sure that L1 size isn't too big > vs global memstore and total heap. This is further confused by > hbase.bucketcache.size accepting a percentage of total heap OR explicit size > in MB. Enabling SlabCache is slightly easier in that it just accepts whatever > LruBlockCache is provided. > Turning on BucketCache using off-heap mode could look like: > hbase-env.sh: > Set HBASE_REGIONSERVER_OPTS: > -Xmx5000m > -XX:MaxDirectMemorySize=15000m > hbase-site.xml: > - hbase.regionserver.global.memstore.size = 0.7 > - hbase.regionserver.onheap.blockcache.size = 0.1 > - hbase.regionserver.blockcache.impl = BucketCache > - hbase.bucketcache.ioengine = offheap > The result being a CombinedCache instance with 500m LruBlockCache + 15000m > ByteBufferIOEngine running in direct mode. > This example does a couple things (mostly for the admin): > - knows NOT to enable SlabCache > - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ > - maintains the validity of HBaseConfiguration's existing check that global > MemStore + LruBlockCache == 0.8 > - maps "BucketCache" into meaning "a CombinedCache instance with these > implementations for L1 and L2." > - Figures out appropriate values for hbase.bucketcache.size and > hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985191#comment-13985191
]
Mikhail Antonov commented on HBASE-11099:
-
[~jeffreyz] I can take second one (should have time to work on closer to this
weekend).
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985188#comment-13985188
]
Jeffrey Zhong commented on HBASE-11099:
---
[~saint@gmail.com]Sure, please go ahead. Thanks.
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10896) Listing processes, master+regionserver shows as HMaster, others as HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985187#comment-13985187 ] Mikhail Antonov commented on HBASE-10896: - My point is - I think process name should be not just informative, but also parseable by scripts (like _jps | grep blah-blah_) Something like: - hbase_rs - simple RS - hbase_m - simple active master - hbase_mrs - active master + rs You think backup master we don't need to display specifically? (also at some point.. we would have multiple active masters I guess, so backups would naturally go away :)) I can take this one. > Listing processes, master+regionserver shows as HMaster, others as > HRegionServer > > > Key: HBASE-10896 > URL: https://issues.apache.org/jira/browse/HBASE-10896 > Project: HBase > Issue Type: Task >Reporter: stack > > If I do a process listing I see the combined Master and RegionServer as > follows: > 30386 HMaster > ... and the regionserver only process as: > 16895 HRegionServer > Backup masters though they are serving regions show as 'HMaster'. > Should process now be called 'HBase' instead? > IntegratonTests IIRC, might kill based off process name so changing process > name would mess this up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10896) Listing processes, master+regionserver shows as HMaster, others as HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985175#comment-13985175 ] stack commented on HBASE-10896: --- I like your idea [~mantonov]. HMB probably should go away.. so hbase_MRS, hbase_RS or something? > Listing processes, master+regionserver shows as HMaster, others as > HRegionServer > > > Key: HBASE-10896 > URL: https://issues.apache.org/jira/browse/HBASE-10896 > Project: HBase > Issue Type: Task >Reporter: stack > > If I do a process listing I see the combined Master and RegionServer as > follows: > 30386 HMaster > ... and the regionserver only process as: > 16895 HRegionServer > Backup masters though they are serving regions show as 'HMaster'. > Should process now be called 'HBase' instead? > IntegratonTests IIRC, might kill based off process name so changing process > name would mess this up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11058) call_id seems never present in response
[
https://issues.apache.org/jira/browse/HBASE-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985174#comment-13985174
]
stack commented on HBASE-11058:
---
[~charlenehelios] You might want to let the HDP folks know... I was playing w/
apache trunk.
> call_id seems never present in response
> ---
>
> Key: HBASE-11058
> URL: https://issues.apache.org/jira/browse/HBASE-11058
> Project: HBase
> Issue Type: Bug
>Reporter: Charlene Sun
>
> version used: 0.98.1
> call_id seems never present in response, could someone take a look and help?
>
> message ResponseHeader {
> optional uint32 call_id = 1;
> message RequestHeader {
> // Monotonically increasing call_id to keep track of RPC requests and their
> response
> optional uint32 call_id = 1;
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HBASE-10965) Automate detection of presence of Filter#filterRow()
[
https://issues.apache.org/jira/browse/HBASE-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985161#comment-13985161
]
Ted Yu edited comment on HBASE-10965 at 4/30/14 4:50 AM:
-
Toward the tail of HBASE-11093, Anoop makes a case that as long as the
following code is present in HRegion, change in HBASE-11093 w.r.t.
FilterList#filterRow() cannot be applied:
{code}
private boolean filterRow() throws IOException {
// when hasFilterRow returns true, filter.filterRow() will be called
automatically inside
// filterRowCells(List kvs) so we skip that scenario here.
return filter != null && (!filter.hasFilterRow())
&& filter.filterRow();
}
{code}
See
https://issues.apache.org/jira/browse/HBASE-11093?focusedCommentId=13985149&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13985149
I think goal of this JIRA can be achieved without removing hasFilterRow().
New method for autodetecting presence of hasFilterRow() can be added to
FilterWrapper. We can rely on this new method in place where hasFilterRow() is
currently called in HRegion.
Post 1.0 release, we can remove hasFilterRow().
was (Author: yuzhih...@gmail.com):
Toward the tail of HBASE-11093, Anoop makes a case that as long as the
following code is present in HRegion, change in HBASE-11093 w.r.t.
FilterList#filterRow() cannot be applied:
{code}
private boolean filterRow() throws IOException {
// when hasFilterRow returns true, filter.filterRow() will be called
automatically inside
// filterRowCells(List kvs) so we skip that scenario here.
return filter != null && (!filter.hasFilterRow())
&& filter.filterRow();
}
{code}
See
https://issues.apache.org/jira/browse/HBASE-11093?focusedCommentId=13985149&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13985149
I think goal of this JIRA can be achieved without removing hasFilterRow().
New method for autodetecting presence of hasFilterRow() can be added to
FilterBase. We can rely on this new method in place where hasFilterRow() is
currently called in HRegion.
Post 1.0 release, we can remove hasFilterRow().
> Automate detection of presence of Filter#filterRow()
>
>
> Key: HBASE-10965
> URL: https://issues.apache.org/jira/browse/HBASE-10965
> Project: HBase
> Issue Type: Task
> Components: Filters
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 10965-v1.txt, 10965-v2.txt, 10965-v3.txt, 10965-v4.txt
>
>
> There is potential inconsistency between the return value of
> Filter#hasFilterRow() and presence of Filter#filterRow().
> Filters may override Filter#filterRow() while leaving return value of
> Filter#hasFilterRow() being false (inherited from FilterBase).
> Downside to purely depending on hasFilterRow() telling us whether custom
> filter overrides filterRow(List) or filterRow() is that the check below may
> be rendered ineffective:
> {code}
> if (nextKv == KV_LIMIT) {
> if (this.filter != null && filter.hasFilterRow()) {
> throw new IncompatibleFilterException(
> "Filter whose hasFilterRow() returns true is incompatible
> with scan with limit!");
> }
> {code}
> When user forgets to override hasFilterRow(), the above check becomes not
> useful.
> This JIRA aims to remove the inconsistency by automatically detecting the
> presence of overridden Filter#filterRow(). If filterRow() is implemented and
> not inherited from FilterBase, it is equivalent to having hasFilterRow()
> return true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11058) call_id seems never present in response
[
https://issues.apache.org/jira/browse/HBASE-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985163#comment-13985163
]
Charlene Sun commented on HBASE-11058:
--
I also see the matching call_id.
It appears the problem only exists in our newest installation which we
downloaded from http://hortonworks.com/hdp/downloads/ (HDP2.1) earlier this
month.
Following packets produced with Cloudera installation and call_id are fine:
request:
: 17 08 88 c3 05 1a 0f 49 73 4d 61 73 74 65 72 52 ...IsMasterR
0010 : 75 6e 6e 69 6e 67 20 01 00 unning ..
response:
: 00 00 00 08 04 08 88 c3 05 02 08 01
Thank you.
charlene
> call_id seems never present in response
> ---
>
> Key: HBASE-11058
> URL: https://issues.apache.org/jira/browse/HBASE-11058
> Project: HBase
> Issue Type: Bug
>Reporter: Charlene Sun
>
> version used: 0.98.1
> call_id seems never present in response, could someone take a look and help?
>
> message ResponseHeader {
> optional uint32 call_id = 1;
> message RequestHeader {
> // Monotonically increasing call_id to keep track of RPC requests and their
> response
> optional uint32 call_id = 1;
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10965) Automate detection of presence of Filter#filterRow()
[
https://issues.apache.org/jira/browse/HBASE-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985161#comment-13985161
]
Ted Yu commented on HBASE-10965:
Toward the tail of HBASE-11093, Anoop makes a case that as long as the
following code is present in HRegion, change in HBASE-11093 w.r.t.
FilterList#filterRow() cannot be applied:
{code}
private boolean filterRow() throws IOException {
// when hasFilterRow returns true, filter.filterRow() will be called
automatically inside
// filterRowCells(List kvs) so we skip that scenario here.
return filter != null && (!filter.hasFilterRow())
&& filter.filterRow();
}
{code}
See
https://issues.apache.org/jira/browse/HBASE-11093?focusedCommentId=13985149&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13985149
I think goal of this JIRA can be achieved without removing hasFilterRow().
New method for autodetecting presence of hasFilterRow() can be added to
FilterBase. We can rely on this new method in place where hasFilterRow() is
currently called in HRegion.
Post 1.0 release, we can remove hasFilterRow().
> Automate detection of presence of Filter#filterRow()
>
>
> Key: HBASE-10965
> URL: https://issues.apache.org/jira/browse/HBASE-10965
> Project: HBase
> Issue Type: Task
> Components: Filters
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 10965-v1.txt, 10965-v2.txt, 10965-v3.txt, 10965-v4.txt
>
>
> There is potential inconsistency between the return value of
> Filter#hasFilterRow() and presence of Filter#filterRow().
> Filters may override Filter#filterRow() while leaving return value of
> Filter#hasFilterRow() being false (inherited from FilterBase).
> Downside to purely depending on hasFilterRow() telling us whether custom
> filter overrides filterRow(List) or filterRow() is that the check below may
> be rendered ineffective:
> {code}
> if (nextKv == KV_LIMIT) {
> if (this.filter != null && filter.hasFilterRow()) {
> throw new IncompatibleFilterException(
> "Filter whose hasFilterRow() returns true is incompatible
> with scan with limit!");
> }
> {code}
> When user forgets to override hasFilterRow(), the above check becomes not
> useful.
> This JIRA aims to remove the inconsistency by automatically detecting the
> presence of overridden Filter#filterRow(). If filterRow() is implemented and
> not inherited from FilterBase, it is equivalent to having hasFilterRow()
> return true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[
https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985154#comment-13985154
]
Hadoop QA commented on HBASE-11092:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642570/HBASE-11092.diff
against trunk revision .
ATTACHMENT ID: 12642570
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 53 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9430//console
This message is automatically generated.
> Server interface should have method getConsensusProvider()
> --
>
> Key: HBASE-11092
> URL: https://issues.apache.org/jira/browse/HBASE-11092
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus
>Affects Versions: 0.99.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
> Fix For: 0.99.0
>
> Attachments: HBASE-11092.diff, HBASE-11092.patch
>
>
> As discussed in comments to HBASE-10915, we need to have a proper way to
> retrieve instance of consensus provider, and Server interface seems the right
> one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[ https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985149#comment-13985149 ] Anoop Sam John commented on HBASE-11093: Yes my point is as long as we have above snippet for the BC, we can not do the change in the patch. bq.We should make a decision as to which release drops this snippet. I am +1 for removing this in Trunk. Also remove the deprecated APIs in Filter. It is not so clean now.. Lets us clean it up for 1.0 Also +1 for removing hasFilterRow(). We should send out mail in dev@ and user@ for getting a consensus ? > FilterList#filterRow() iterates through its filters even though > FilterList#hasFilterRow() returns false > --- > > Key: HBASE-11093 > URL: https://issues.apache.org/jira/browse/HBASE-11093 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt > > > FilterList#hasFilterRow() returns false when hasFilterRow() of all of > constituent filters return false. > However, FilterList#filterRow() still iterates through its filters in this > scenario. > The iteration should be skipped when FilterList#hasFilterRow() returns false. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10896) Listing processes, master+regionserver shows as HMaster, others as HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985146#comment-13985146 ] Mikhail Antonov commented on HBASE-10896: - Just "HBase" may be little less informative, than, say, something like - "hbase (HMA=y,HMB=n,HRS=y)", where hma and hmb stand for active/backup masters? That probably looks ugly but more informative? > Listing processes, master+regionserver shows as HMaster, others as > HRegionServer > > > Key: HBASE-10896 > URL: https://issues.apache.org/jira/browse/HBASE-10896 > Project: HBase > Issue Type: Task >Reporter: stack > > If I do a process listing I see the combined Master and RegionServer as > follows: > 30386 HMaster > ... and the regionserver only process as: > 16895 HRegionServer > Backup masters though they are serving regions show as 'HMaster'. > Should process now be called 'HBase' instead? > IntegratonTests IIRC, might kill based off process name so changing process > name would mess this up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[ https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985145#comment-13985145 ] Ted Yu commented on HBASE-11093: User filter should follow Filter javadoc, which is clear since 0.96+, and implement hasFilterRow() accordingly. The snippet cited above makes filtering inefficient and somewhat hard to predict. We should make a decision as to which release drops this snippet. > FilterList#filterRow() iterates through its filters even though > FilterList#hasFilterRow() returns false > --- > > Key: HBASE-11093 > URL: https://issues.apache.org/jira/browse/HBASE-11093 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt > > > FilterList#hasFilterRow() returns false when hasFilterRow() of all of > constituent filters return false. > However, FilterList#filterRow() still iterates through its filters in this > scenario. > The iteration should be skipped when FilterList#hasFilterRow() returns false. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985141#comment-13985141 ] ramkrishna.s.vasudevan commented on HBASE-10885: Corrected the patch and now all cases passes. > Support visibility expressions on Deletes > - > > Key: HBASE-10885 > URL: https://issues.apache.org/jira/browse/HBASE-10885 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch > > > Accumulo can specify visibility expressions for delete markers. During > compaction the cells covered by the tombstone are determined in part by > matching the visibility expression. This is useful for the use case of data > set coalescing, where entries from multiple data sets carrying different > labels are combined into one common large table. Later, a subset of entries > can be conveniently removed using visibility expressions. > Currently doing the same in HBase would only be possible with a custom > coprocessor. Otherwise, a Delete will affect all cells covered by the > tombstone regardless of any visibility expression scoping. This is correct > behavior in that no data spill is possible, but certainly could be > surprising, and is only meant to be transitional. We decided not to support > visibility expressions on Deletes to control the complexity of the initial > implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10611) Description for hbase:acl table is wrong on master-status#catalogTables
[
https://issues.apache.org/jira/browse/HBASE-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985137#comment-13985137
]
Hudson commented on HBASE-10611:
ABORTED: Integrated in HBase-0.98-on-Hadoop-1.1 #285 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/285/])
HBASE-10611 Description for hbase:acl table is wrong on
master-status#catalogTables (Rekha Joshi) (tedyu: rev 1591069)
*
/hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
> Description for hbase:acl table is wrong on master-status#catalogTables
> ---
>
> Key: HBASE-10611
> URL: https://issues.apache.org/jira/browse/HBASE-10611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Ted Yu
>Assignee: Rekha Joshi
>Priority: Minor
> Labels: patch
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10611.1.patch, HBASE-10611.1.patch,
> HBASE-10611.2.patch
>
>
> On master-status#catalogTables, I see:
> hbase:acl The .NAMESPACE. table holds information about namespaces.
> In MasterStatusTmpl.jamon, we have:
> {code}
> if (tableName.equals(TableName.META_TABLE_NAME)){
> description = "The hbase:meta table holds references to all User
> Table regions";
> } else {
> description = "The .NAMESPACE. table holds information about
> namespaces.";
> }
> {code}
> The above check doesn't cover hbase:acl table.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10925) Do not OOME, throw RowTooBigException instead
[ https://issues.apache.org/jira/browse/HBASE-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985139#comment-13985139 ] Hudson commented on HBASE-10925: ABORTED: Integrated in HBase-TRUNK #5125 (See [https://builds.apache.org/job/HBase-TRUNK/5125/]) HBASE-10925 Do not OOME, throw RowTooBigException instead (Mikhail Antonov) (stack: rev 1591154) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowTooBigException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRowTooBig.java > Do not OOME, throw RowTooBigException instead > - > > Key: HBASE-10925 > URL: https://issues.apache.org/jira/browse/HBASE-10925 > Project: HBase > Issue Type: Improvement > Components: Usability >Affects Versions: 0.99.0 >Reporter: stack >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-10925.patch, HBASE-10925.patch, HBASE-10925.patch > > > If 10M columns in a row, throw a RowTooBigException rather than OOME when > Get'ing or Scanning w/o in-row scan flag set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10611) Description for hbase:acl table is wrong on master-status#catalogTables
[
https://issues.apache.org/jira/browse/HBASE-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985138#comment-13985138
]
Hudson commented on HBASE-10611:
ABORTED: Integrated in HBase-TRUNK #5125 (See
[https://builds.apache.org/job/HBase-TRUNK/5125/])
HBASE-10611 Description for hbase:acl table is wrong on
master-status#catalogTables (Rekha Joshi) (tedyu: rev 1591070)
*
/hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
> Description for hbase:acl table is wrong on master-status#catalogTables
> ---
>
> Key: HBASE-10611
> URL: https://issues.apache.org/jira/browse/HBASE-10611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Ted Yu
>Assignee: Rekha Joshi
>Priority: Minor
> Labels: patch
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10611.1.patch, HBASE-10611.1.patch,
> HBASE-10611.2.patch
>
>
> On master-status#catalogTables, I see:
> hbase:acl The .NAMESPACE. table holds information about namespaces.
> In MasterStatusTmpl.jamon, we have:
> {code}
> if (tableName.equals(TableName.META_TABLE_NAME)){
> description = "The hbase:meta table holds references to all User
> Table regions";
> } else {
> description = "The .NAMESPACE. table holds information about
> namespaces.";
> }
> {code}
> The above check doesn't cover hbase:acl table.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985135#comment-13985135 ] stack commented on HBASE-11077: --- Why undo these as static finals? That is passing a Strategy instead? + String EXEC_PERMISSION_CHECKS_KEY = "hbase.security.exec.permission.checks"; What is default for the above? You want to do this before commit? +// XXX: Compare in place, don't clone I skimmed the patch. Looks fine Andy. Nice test. Release note. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[
https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985134#comment-13985134
]
Anoop Sam John commented on HBASE-11093:
In HRegion#RegionScannerImpl
Wherever we call filterRowCells() or filterRow() we have hasFilterRow() checks
prior to those..
{code}
private boolean filterRow() throws IOException {
// when hasFilterRow returns true, filter.filterRow() will be called
automatically inside
// filterRowCells(List kvs) so we skip that scenario here.
return filter != null && (!filter.hasFilterRow())
&& filter.filterRow();
}
{code}
For BC we make above code. Here even if hasFilterRow is false we have to call
filterRow(). With this patch, we will fail to call filterRow() right?
> FilterList#filterRow() iterates through its filters even though
> FilterList#hasFilterRow() returns false
> ---
>
> Key: HBASE-11093
> URL: https://issues.apache.org/jira/browse/HBASE-11093
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt
>
>
> FilterList#hasFilterRow() returns false when hasFilterRow() of all of
> constituent filters return false.
> However, FilterList#filterRow() still iterates through its filters in this
> scenario.
> The iteration should be skipped when FilterList#hasFilterRow() returns false.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11088) Support Visibility Expression Deletes in Shell
[ https://issues.apache.org/jira/browse/HBASE-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985132#comment-13985132 ] ramkrishna.s.vasudevan commented on HBASE-11088: bq.I don't think _delete_internal() should take a variable number of arguments. Instead deal with variations in command forms with default args to command(). Let me check this. bq.we just assume whatever is the last string value enumerated in the hash is a visibility expression True, like the args are checked for args[VISIBILITY], need to check here also. Anyway if the string is not a VISIBILITY EXPRESSION the server would fail it. > Support Visibility Expression Deletes in Shell > -- > > Key: HBASE-11088 > URL: https://issues.apache.org/jira/browse/HBASE-11088 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-11058.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985128#comment-13985128
]
stack commented on HBASE-11099:
---
[~jeffreyz] Good one. Want me to take the first one as a subtask of this issue?
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985127#comment-13985127
]
stack commented on HBASE-8763:
--
You the man [~jeffreyz]
Spelling here waitForPreviousTransactoinsComplete
What is happening here?
syncOrDefer(txid, durability);
+// Get LogSequenceNumber from WAL Sync
+if(walEdit.getLogKey() != null) {
+ seqNumber.setValue(walEdit.getLogKey().getLogSeqNum());
+}
The seqNumber of the last wallEdit added to the WAL and sync'd? The seq number
could be well beyond this in actually? Does that matter? Should we let you
have access to last sync'd id out of WAL?
You import but don't use?
+import org.apache.commons.lang.mutable.MutableInt;
+import org.apache.commons.lang.mutable.MutableLong;
... maybe a few times.
Why you use the MutableLong instead of say a long or a volatile?
Will do a closer review later. So far looks great.
> [BRAINSTORM] Combine MVCC and SeqId
> ---
>
> Key: HBASE-8763
> URL: https://issues.apache.org/jira/browse/HBASE-8763
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Enis Soztutar
>Assignee: Jeffrey Zhong
>Priority: Critical
> Attachments: hbase-8736-poc.patch, hbase-8763-poc-v1.patch,
> hbase-8763_wip1.patch
>
>
> HBASE-8701 and a lot of recent issues include good discussions about mvcc +
> seqId semantics. It seems that having mvcc and the seqId complicates the
> comparator semantics a lot in regards to flush + WAL replay + compactions +
> delete markers and out of order puts.
> Thinking more about it I don't think we need a MVCC write number which is
> different than the seqId. We can keep the MVCC semantics, read point and
> smallest read points intact, but combine mvcc write number and seqId. This
> will allow cleaner semantics + implementation + smaller data files.
> We can do some brainstorming for 0.98. We still have to verify that this
> would be semantically correct, it should be so by my current understanding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10611) Description for hbase:acl table is wrong on master-status#catalogTables
[
https://issues.apache.org/jira/browse/HBASE-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985126#comment-13985126
]
Hudson commented on HBASE-10611:
SUCCESS: Integrated in HBase-0.98 #300 (See
[https://builds.apache.org/job/HBase-0.98/300/])
HBASE-10611 Description for hbase:acl table is wrong on
master-status#catalogTables (Rekha Joshi) (tedyu: rev 1591069)
*
/hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
> Description for hbase:acl table is wrong on master-status#catalogTables
> ---
>
> Key: HBASE-10611
> URL: https://issues.apache.org/jira/browse/HBASE-10611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Ted Yu
>Assignee: Rekha Joshi
>Priority: Minor
> Labels: patch
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10611.1.patch, HBASE-10611.1.patch,
> HBASE-10611.2.patch
>
>
> On master-status#catalogTables, I see:
> hbase:acl The .NAMESPACE. table holds information about namespaces.
> In MasterStatusTmpl.jamon, we have:
> {code}
> if (tableName.equals(TableName.META_TABLE_NAME)){
> description = "The hbase:meta table holds references to all User
> Table regions";
> } else {
> description = "The .NAMESPACE. table holds information about
> namespaces.";
> }
> {code}
> The above check doesn't cover hbase:acl table.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11006) Cannot create 'data' table
[
https://issues.apache.org/jira/browse/HBASE-11006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985118#comment-13985118
]
Hadoop QA commented on HBASE-11006:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642575/HBASE-11006-v4.patch
against trunk revision .
ATTACHMENT ID: 12642575
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9428//console
This message is automatically generated.
> Cannot create 'data' table
> --
>
> Key: HBASE-11006
> URL: https://issues.apache.org/jira/browse/HBASE-11006
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Hart
>Assignee: Gustavo Anatoly
> Labels: noob
> Fix For: 0.99.0
>
> Attachments: HBASE-11006-v2.patch, HBASE-11006-v3.patch,
> HBASE-11006-v4.patch, HBASE-11006.patch
>
>
> Starting with HBase 0.96, it is not possible to create a table named 'data'.
> When a user tries to create a table named 'data', the table is added to the
> metastore but the region server fails to open the region.
> To reproduce, I just run “hbase shell” and type in:
> create ‘data’, ‘cf’
> This was also verified on Apache HBase trunk by Enis from the Hortornworks
> forums. His comment on the bug:
> There has been a change in the directory layout in HBase 0.96 for introducing
> namespaces. The HBase root directory now keeps all table data under the
> sub-directory “data”. I suspect the error is somewhere there.
> More info can be found on the thread in the Hortornworks forums:
> http://hortonworks.com/community/forums/topic/cannot-create-data-table/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985080#comment-13985080
]
Enis Soztutar commented on HBASE-8763:
--
Great to see progress Jeffrey.
As talked offline, I think we do not want to allocate 2 objects on heap rather
than one:
{code}
- private long mvcc = 0; // this value is not part of a serialized KeyValue
(not in HFiles)
+ private MutableLong mvcc; // this value is not part of a serialized
KeyValue (not in HFiles)
{code}
Although, changing the mvcc long after object construction may require it to be
declared volatile.
I think you are using the technique of double incrementing the seqId, once at
trx start and once at end right? Did you try your "append 1B to seqId for
memstore" approach?
I think I also changed Cell.getMvcc() to Cell.getSeqId(). Maybe worth doing
once we get the patch fully running and performant.
> [BRAINSTORM] Combine MVCC and SeqId
> ---
>
> Key: HBASE-8763
> URL: https://issues.apache.org/jira/browse/HBASE-8763
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Enis Soztutar
>Assignee: Jeffrey Zhong
>Priority: Critical
> Attachments: hbase-8736-poc.patch, hbase-8763-poc-v1.patch,
> hbase-8763_wip1.patch
>
>
> HBASE-8701 and a lot of recent issues include good discussions about mvcc +
> seqId semantics. It seems that having mvcc and the seqId complicates the
> comparator semantics a lot in regards to flush + WAL replay + compactions +
> delete markers and out of order puts.
> Thinking more about it I don't think we need a MVCC write number which is
> different than the seqId. We can keep the MVCC semantics, read point and
> smallest read points intact, but combine mvcc write number and seqId. This
> will allow cleaner semantics + implementation + smaller data files.
> We can do some brainstorming for 0.98. We still have to verify that this
> would be semantically correct, it should be so by my current understanding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10841) Scan setters should consistently return this
[
https://issues.apache.org/jira/browse/HBASE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985074#comment-13985074
]
Enis Soztutar commented on HBASE-10841:
---
HBASE-10460 was the issue.
Doing this would break binary BC. I think we are fine with breaking BC between
0.98 and 1.0 as long as we retain wire compat.
> Scan setters should consistently return this
>
>
> Key: HBASE-10841
> URL: https://issues.apache.org/jira/browse/HBASE-10841
> Project: HBase
> Issue Type: Sub-task
> Components: Client, Usability
>Affects Versions: 0.99.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> While addressing review comments on HBASE-10818, I noticed that our {{Scan}}
> class is inconsistent with it's setter methods. Some of them return {{this}},
> other's don't. They should be consistent. I suggest making them all return
> {{this}}, to support chained invocation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8763: - Attachment: hbase-8763-poc-v1.patch This is updated POC version. All small/medium tests including TestAtomicOperation are passed except TestExportSnapshot. I'll try to fix it soon. Thanks. > [BRAINSTORM] Combine MVCC and SeqId > --- > > Key: HBASE-8763 > URL: https://issues.apache.org/jira/browse/HBASE-8763 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Enis Soztutar >Assignee: Jeffrey Zhong >Priority: Critical > Attachments: hbase-8736-poc.patch, hbase-8763-poc-v1.patch, > hbase-8763_wip1.patch > > > HBASE-8701 and a lot of recent issues include good discussions about mvcc + > seqId semantics. It seems that having mvcc and the seqId complicates the > comparator semantics a lot in regards to flush + WAL replay + compactions + > delete markers and out of order puts. > Thinking more about it I don't think we need a MVCC write number which is > different than the seqId. We can keep the MVCC semantics, read point and > smallest read points intact, but combine mvcc write number and seqId. This > will allow cleaner semantics + implementation + smaller data files. > We can do some brainstorming for 0.98. We still have to verify that this > would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10856) Prep for 1.0
[ https://issues.apache.org/jira/browse/HBASE-10856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-10856: -- Fix Version/s: 0.99.0 > Prep for 1.0 > > > Key: HBASE-10856 > URL: https://issues.apache.org/jira/browse/HBASE-10856 > Project: HBase > Issue Type: Umbrella >Reporter: stack > Fix For: 0.99.0 > > > Tasks for 1.0 copied here from our '1.0.0' mailing list discussion. Idea is > to file subtasks off this one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[
https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985069#comment-13985069
]
Qiang Tian commented on HBASE-11096:
Hi [~apurtell],
Thanks for response.
Yes, I will. the fix is simple.. this is part of work for HBASE-10289,
during my test I just find another small bug--the HRegionServer.stop is called
twice, the fix looks simple as well, but I better study the related code
first..:-)
the first time:
HRegionServer.run:
} else if (this.stopping) {
boolean allUserRegionsOffline = areAllUserRegionsOffline();
if (allUserRegionsOffline) {
// Set stopped if no requests since last time we went around the
loop.
// The remaining meta regions will be closed on our way out.
if (oldRequestCount == this.requestCount.get()) {
stop("Stopped; only catalog regions remaining online");
break;
}
the second time in shutdown hook:
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:273)
at
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoproces
sorHost.java:63)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1680)
at
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:114)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
2014-04-29 18:56:02,617 WARN
org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Not stopping coprocess
or org.apache.hadoop.hbase.jmxAgent because not active (state=STOPPED)
2014-04-29 18:56:02,617 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2014-04-29 18:56:02,617 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook:
Starting fs shutdown hoo
k thread.
> stop method of Master and RegionServer coprocessor is not invoked
> --
>
> Key: HBASE-11096
> URL: https://issues.apache.org/jira/browse/HBASE-11096
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.1, 0.94.19
>Reporter: Qiang Tian
>Assignee: Qiang Tian
> Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
>
>
> the stop method of coprocessor specified by
> "hbase.coprocessor.master.classes" and
> "hbase.coprocessor.regionserver.classes" is not invoked.
> If coprocessor allocates OS resources, it could cause master/regionserver
> resource leak or hang during exit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong reassigned HBASE-8763: Assignee: Jeffrey Zhong > [BRAINSTORM] Combine MVCC and SeqId > --- > > Key: HBASE-8763 > URL: https://issues.apache.org/jira/browse/HBASE-8763 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Enis Soztutar >Assignee: Jeffrey Zhong >Priority: Critical > Attachments: hbase-8736-poc.patch, hbase-8763_wip1.patch > > > HBASE-8701 and a lot of recent issues include good discussions about mvcc + > seqId semantics. It seems that having mvcc and the seqId complicates the > comparator semantics a lot in regards to flush + WAL replay + compactions + > delete markers and out of order puts. > Thinking more about it I don't think we need a MVCC write number which is > different than the seqId. We can keep the MVCC semantics, read point and > smallest read points intact, but combine mvcc write number and seqId. This > will allow cleaner semantics + implementation + smaller data files. > We can do some brainstorming for 0.98. We still have to verify that this > would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985062#comment-13985062
]
Jeffrey Zhong commented on HBASE-11099:
---
Yes, that's correct.
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-11099:
--
Fix Version/s: 0.99.0
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985060#comment-13985060
]
Enis Soztutar commented on HBASE-11099:
---
My understanding is that 1) only applies to trunk with the disruptor changes.
2) applies to 0.98 as well. Is that right Jeffrey ?
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11099) Two situations where we could open a region with smaller sequence number
Jeffrey Zhong created HBASE-11099:
-
Summary: Two situations where we could open a region with smaller
sequence number
Key: HBASE-11099
URL: https://issues.apache.org/jira/browse/HBASE-11099
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.99.0
Reporter: Jeffrey Zhong
Recently I happened to run into code where we potentially could open region
with smaller sequence number:
1) Inside function: HRegion#internalFlushcache. This is due to we change the
way WAL Sync where we use late binding(assign sequence number right before wal
sync).
The flushSeqId may less than the change sequence number included in the flush
which may cause later region opening code to use a smaller than expected
sequence number when we reopen the region.
{code}
flushSeqId = this.sequenceId.incrementAndGet();
...
mvcc.waitForRead(w);
{code}
2) HRegion#replayRecoveredEdits where we have following code:
{code}
...
if (coprocessorHost != null) {
status.setStatus("Running pre-WAL-restore hook in coprocessors");
if (coprocessorHost.preWALRestore(this.getRegionInfo(), key, val)) {
// if bypass this log entry, ignore it ...
continue;
}
}
...
currentEditSeqId = key.getLogSeqNum();
{code}
If coprocessor skip some tail WALEdits, then the function will return smaller
currentEditSeqId. In the end, a region may also open with a smaller sequence
number. This may cause data loss because Master may record a larger flushed
sequence Id and some WALEdits maybe skipped during recovery if the region fail
again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11084) Handle deletions and column tracking using internal filters rather than using DeleteTrackers and ColumnTrackers
[ https://issues.apache.org/jira/browse/HBASE-11084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985044#comment-13985044 ] Andrew Purtell commented on HBASE-11084: Interesting idea! > Handle deletions and column tracking using internal filters rather than using > DeleteTrackers and ColumnTrackers > --- > > Key: HBASE-11084 > URL: https://issues.apache.org/jira/browse/HBASE-11084 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > > See HBASE-11054 for discussion. > Currently the delete tracking is done by DeleteTracker and its subclasses. > column tracking and its versions are handled inside the ColumnTrackers and > its subclasses. > This JIRA aims at providing internal filters and attaching them to > scans/gets, including minor and major compaction scans so that all the logic > of deletes and version counting goes into it rather than having trackers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985041#comment-13985041 ] Andrew Purtell commented on HBASE-11096: Hi [~tianq], thanks for reporting the issue. Are you planning to provide a patch? > stop method of Master and RegionServer coprocessor is not invoked > -- > > Key: HBASE-11096 > URL: https://issues.apache.org/jira/browse/HBASE-11096 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.94.19 >Reporter: Qiang Tian >Assignee: Qiang Tian > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > > the stop method of coprocessor specified by > "hbase.coprocessor.master.classes" and > "hbase.coprocessor.regionserver.classes" is not invoked. > If coprocessor allocates OS resources, it could cause master/regionserver > resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11096: --- Affects Version/s: 0.96.2 0.98.1 0.94.19 Fix Version/s: 0.98.3 0.94.20 0.96.3 0.99.0 > stop method of Master and RegionServer coprocessor is not invoked > -- > > Key: HBASE-11096 > URL: https://issues.apache.org/jira/browse/HBASE-11096 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.94.19 >Reporter: Qiang Tian >Assignee: Qiang Tian > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > > the stop method of coprocessor specified by > "hbase.coprocessor.master.classes" and > "hbase.coprocessor.regionserver.classes" is not invoked. > If coprocessor allocates OS resources, it could cause master/regionserver > resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985037#comment-13985037 ] Andrew Purtell commented on HBASE-11077: Rolling the 0.98.2 RC0 tomorrow. Review would be very much welcomed because I think this is ready to go (modulo addressing review comments if any) and so will otherwise commit this tomorrow for the RC. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11088) Support Visibility Expression Deletes in Shell
[ https://issues.apache.org/jira/browse/HBASE-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11088: --- Priority: Blocker (was: Major) Fix Version/s: (was: 0.98.2) 0.98.3 Rolling the 0.98.2 RC0 tommorrow. This looks almost ready but not quite. No problem, making a blocker for 0.98.3, we can get in in there for sure. The integration test for deletes-with-visibility is slotted for 0.98.3 anyhow. > Support Visibility Expression Deletes in Shell > -- > > Key: HBASE-11088 > URL: https://issues.apache.org/jira/browse/HBASE-11088 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-11058.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10885: --- Priority: Blocker (was: Critical) Fix Version/s: (was: 0.98.2) 0.98.3 Rolling the 0.98.2 RC0 tommorrow. This looks almost ready but not quite. No problem, making a blocker for 0.98.3, we can get in in there for sure. The integration test for deletes-with-visibility is slotted for 0.98.3 anyhow. > Support visibility expressions on Deletes > - > > Key: HBASE-10885 > URL: https://issues.apache.org/jira/browse/HBASE-10885 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch > > > Accumulo can specify visibility expressions for delete markers. During > compaction the cells covered by the tombstone are determined in part by > matching the visibility expression. This is useful for the use case of data > set coalescing, where entries from multiple data sets carrying different > labels are combined into one common large table. Later, a subset of entries > can be conveniently removed using visibility expressions. > Currently doing the same in HBase would only be possible with a custom > coprocessor. Otherwise, a Delete will affect all cells covered by the > tombstone regardless of any visibility expression scoping. This is correct > behavior in that no data spill is possible, but certainly could be > surprising, and is only meant to be transitional. We decided not to support > visibility expressions on Deletes to control the complexity of the initial > implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985030#comment-13985030 ] Andrew Purtell commented on HBASE-10885: Will review patch after we have a clean run of o.a.h.h.security.visibility.* tests. > Support visibility expressions on Deletes > - > > Key: HBASE-10885 > URL: https://issues.apache.org/jira/browse/HBASE-10885 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch > > > Accumulo can specify visibility expressions for delete markers. During > compaction the cells covered by the tombstone are determined in part by > matching the visibility expression. This is useful for the use case of data > set coalescing, where entries from multiple data sets carrying different > labels are combined into one common large table. Later, a subset of entries > can be conveniently removed using visibility expressions. > Currently doing the same in HBase would only be possible with a custom > coprocessor. Otherwise, a Delete will affect all cells covered by the > tombstone regardless of any visibility expression scoping. This is correct > behavior in that no data spill is possible, but certainly could be > surprising, and is only meant to be transitional. We decided not to support > visibility expressions on Deletes to control the complexity of the initial > implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11088) Support Visibility Expression Deletes in Shell
[ https://issues.apache.org/jira/browse/HBASE-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985025#comment-13985025 ] Andrew Purtell commented on HBASE-11088: There are a couple of issues with argument handling. - Something called "timestamp" is not guaranteed to be a timestamp, it could be a hash meaning something entirely else. - If no timestamp is provided but the option hash is, then we don't check if the key is "VISIBILITY", we just assume whatever is the last string value enumerated in the hash is a visibility expression. That's not correct, but also I can see other delete options being added later via this new hash. I don't think _delete_internal() should take a variable number of arguments. Instead deal with variations in command forms with default args to command(). Then it's easy to fix the above mentioned issues. > Support Visibility Expression Deletes in Shell > -- > > Key: HBASE-11088 > URL: https://issues.apache.org/jira/browse/HBASE-11088 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11058.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11006) Cannot create 'data' table
[ https://issues.apache.org/jira/browse/HBASE-11006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gustavo Anatoly updated HBASE-11006: Attachment: HBASE-11006-v4.patch Failed test, corrected: TestSplitTransactionOnCluster#testSplitRegionWithNoStoreFiles > Cannot create 'data' table > -- > > Key: HBASE-11006 > URL: https://issues.apache.org/jira/browse/HBASE-11006 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Gregory Hart >Assignee: Gustavo Anatoly > Labels: noob > Fix For: 0.99.0 > > Attachments: HBASE-11006-v2.patch, HBASE-11006-v3.patch, > HBASE-11006-v4.patch, HBASE-11006.patch > > > Starting with HBase 0.96, it is not possible to create a table named 'data'. > When a user tries to create a table named 'data', the table is added to the > metastore but the region server fails to open the region. > To reproduce, I just run “hbase shell” and type in: > create ‘data’, ‘cf’ > This was also verified on Apache HBase trunk by Enis from the Hortornworks > forums. His comment on the bug: > There has been a change in the directory layout in HBase 0.96 for introducing > namespaces. The HBase root directory now keeps all table data under the > sub-directory “data”. I suspect the error is somewhere there. > More info can be found on the thread in the Hortornworks forums: > http://hortonworks.com/community/forums/topic/cannot-create-data-table/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985015#comment-13985015 ] Andrew Purtell commented on HBASE-11077: Latest patch includes new tests that verify the backwards compatible behavior toggle in HTableDescriptor and "cell first" ACL evaluation strategy under both regimes. All AccessController tests pass. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10935) support snapshot policy where flush memstore can be skipped to prevent production cluster freeze
[ https://issues.apache.org/jira/browse/HBASE-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-10935: --- Attachment: jira-10935.patch > support snapshot policy where flush memstore can be skipped to prevent > production cluster freeze > > > Key: HBASE-10935 > URL: https://issues.apache.org/jira/browse/HBASE-10935 > Project: HBase > Issue Type: New Feature > Components: shell, snapshots >Affects Versions: 0.94.7, 0.94.18 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 0.94.20 > > Attachments: jira-10935.patch > > > We are using snapshot feature to do HBase disaster recovery. We will do > snapshot in our production cluster periodically. The current flush snapshot > policy require all regions of the table to coordinate to prevent write and do > flush at the same time. Since we use WALPlayer to complete the data that is > not in the snapshot HFile, we don't need the snapshot to do coordinated > flush. The snapshot just recored all the HFile that are already there. > I added the parameter in the HBase shell. So people can choose to use the > NoFlush snapshot when they need, like below. Otherwise, the default flush > snpahot support is not impacted. > >snaphot 'TestTable', 'TestSnapshot', 'skipFlush' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10935) support snapshot policy where flush memstore can be skipped to prevent production cluster freeze
[ https://issues.apache.org/jira/browse/HBASE-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-10935: --- Attachment: (was: HBase-10935.patch) > support snapshot policy where flush memstore can be skipped to prevent > production cluster freeze > > > Key: HBASE-10935 > URL: https://issues.apache.org/jira/browse/HBASE-10935 > Project: HBase > Issue Type: New Feature > Components: shell, snapshots >Affects Versions: 0.94.7, 0.94.18 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 0.94.20 > > Attachments: jira-10935.patch > > > We are using snapshot feature to do HBase disaster recovery. We will do > snapshot in our production cluster periodically. The current flush snapshot > policy require all regions of the table to coordinate to prevent write and do > flush at the same time. Since we use WALPlayer to complete the data that is > not in the snapshot HFile, we don't need the snapshot to do coordinated > flush. The snapshot just recored all the HFile that are already there. > I added the parameter in the HBase shell. So people can choose to use the > NoFlush snapshot when they need, like below. Otherwise, the default flush > snpahot support is not impacted. > >snaphot 'TestTable', 'TestSnapshot', 'skipFlush' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10935) support snapshot policy where flush memstore can be skipped to prevent production cluster freeze
[ https://issues.apache.org/jira/browse/HBASE-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-10935: --- Attachment: (was: HBase-10935.patch) > support snapshot policy where flush memstore can be skipped to prevent > production cluster freeze > > > Key: HBASE-10935 > URL: https://issues.apache.org/jira/browse/HBASE-10935 > Project: HBase > Issue Type: New Feature > Components: shell, snapshots >Affects Versions: 0.94.7, 0.94.18 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 0.94.20 > > Attachments: jira-10935.patch > > > We are using snapshot feature to do HBase disaster recovery. We will do > snapshot in our production cluster periodically. The current flush snapshot > policy require all regions of the table to coordinate to prevent write and do > flush at the same time. Since we use WALPlayer to complete the data that is > not in the snapshot HFile, we don't need the snapshot to do coordinated > flush. The snapshot just recored all the HFile that are already there. > I added the parameter in the HBase shell. So people can choose to use the > NoFlush snapshot when they need, like below. Otherwise, the default flush > snpahot support is not impacted. > >snaphot 'TestTable', 'TestSnapshot', 'skipFlush' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11077: --- Attachment: HBASE-11077.patch > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11092: Attachment: HBASE-11092.diff updated patch, made base consensus provider abstract > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.diff, HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10925) Do not OOME, throw RowTooBigException instead
[ https://issues.apache.org/jira/browse/HBASE-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984999#comment-13984999 ] Mikhail Antonov commented on HBASE-10925: - Thanks! > Do not OOME, throw RowTooBigException instead > - > > Key: HBASE-10925 > URL: https://issues.apache.org/jira/browse/HBASE-10925 > Project: HBase > Issue Type: Improvement > Components: Usability >Affects Versions: 0.99.0 >Reporter: stack >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-10925.patch, HBASE-10925.patch, HBASE-10925.patch > > > If 10M columns in a row, throw a RowTooBigException rather than OOME when > Get'ing or Scanning w/o in-row scan flag set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10925) Do not OOME, throw RowTooBigException instead
[ https://issues.apache.org/jira/browse/HBASE-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10925: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch [~mantonov] > Do not OOME, throw RowTooBigException instead > - > > Key: HBASE-10925 > URL: https://issues.apache.org/jira/browse/HBASE-10925 > Project: HBase > Issue Type: Improvement > Components: Usability >Affects Versions: 0.99.0 >Reporter: stack >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-10925.patch, HBASE-10925.patch, HBASE-10925.patch > > > If 10M columns in a row, throw a RowTooBigException rather than OOME when > Get'ing or Scanning w/o in-row scan flag set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11098) Improve documentation around our blockcache options
[ https://issues.apache.org/jira/browse/HBASE-11098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11098: -- Attachment: 11098.txt WIP Adds doc to the config keys in CacheConfig explaining how each works. Add package-info for bucket cache package and the hfile package. Misc javadoc and formatting cleanup. Removed unused RandomSee test class and the unused SimpleBlockCache. > Improve documentation around our blockcache options > --- > > Key: HBASE-11098 > URL: https://issues.apache.org/jira/browse/HBASE-11098 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 11098.txt > > > Adding as a subtask on [~ndimiduk] necessary cleanup. Trying to make sense > of this stuff I started adding doc (package-info files, javadoc). I see this > as complementary to the parent task work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11098) Improve documentation around our blockcache options
stack created HBASE-11098: - Summary: Improve documentation around our blockcache options Key: HBASE-11098 URL: https://issues.apache.org/jira/browse/HBASE-11098 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Adding as a subtask on [~ndimiduk] necessary cleanup. Trying to make sense of this stuff I started adding doc (package-info files, javadoc). I see this as complementary to the parent task work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9003) TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar
[
https://issues.apache.org/jira/browse/HBASE-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-9003:
-
Attachment: HBASE-9003.v2.patch
New patch with [~ndimiduk] suggestions.
> TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar
> -
>
> Key: HBASE-9003
> URL: https://issues.apache.org/jira/browse/HBASE-9003
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 0.99.0
>
> Attachments: HBASE-9003.v0.patch, HBASE-9003.v1.patch,
> HBASE-9003.v2.patch
>
>
> This is the problem: {{TableMapReduceUtil#addDependencyJars}} relies on
> {{org.apache.hadoop.util.JarFinder}} if available to call {{getJar()}}.
> However {{getJar()}} uses File.createTempFile() to create a temporary file
> under {{hadoop.tmp.dir}}{{/target/test-dir}}. Due HADOOP-9737 the created jar
> and its content is not purged after the JVM is destroyed. Since most
> configurations point {{hadoop.tmp.dir}} under {{/tmp}} the generated jar
> files get purged by {{tmpwatch}} or a similar tool, but boxes that have
> {{hadoop.tmp.dir}} pointing to a different location not monitored by
> {{tmpwatch}} will pile up a collection of jars causing all kind of issues.
> Since {{JarFinder#getJar}} is not a public API from Hadoop (see [~tucu00]
> comment on HADOOP-9737) we shouldn't use that as part of
> {{TableMapReduceUtil}} in order to avoid this kind of issues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984979#comment-13984979 ] Ted Yu commented on HBASE-11092: The patch is straightforward. I don't have other comments. > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984967#comment-13984967 ] Mikhail Antonov commented on HBASE-11092: - Thanks for looking at it - [~tedyu] - any more feedback? I'll post updated patch soon in case there are more comments.. Would it better also go thru RB? > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984962#comment-13984962
]
Hadoop QA commented on HBASE-10915:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642511/HBASE-10915.patch
against trunk revision .
ATTACHMENT ID: 12642511
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testPreWALRestoreSkip(TestRegionObserverInterface.java:578)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9427//console
This message is automatically generated.
> Decouple region closing (HM and HRS) from ZK
>
>
> Key: HBASE-10915
> URL: https://issues.apache.org/jira/browse/HBASE-10915
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus, Zookeeper
>Affects Versions: 0.99.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
> Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch
>
>
> Decouple region closing from ZK.
> Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the
> code using (HRegionServer, RSRpcServices etc).
> May need small changes in AssignmentManager.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984961#comment-13984961 ] Mikhail Antonov commented on HBASE-11092: - Yes, I think it should. > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[
https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984955#comment-13984955
]
Ted Yu commented on HBASE-11092:
{code}
+@InterfaceAudience.Private
+public class BaseConsensusProvider implements ConsensusProvider {
{code}
Since BaseConsensusProvider would be overridden by concrete implementation,
should BaseConsensusProvider be abstract ?
> Server interface should have method getConsensusProvider()
> --
>
> Key: HBASE-11092
> URL: https://issues.apache.org/jira/browse/HBASE-11092
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus
>Affects Versions: 0.99.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
> Fix For: 0.99.0
>
> Attachments: HBASE-11092.patch
>
>
> As discussed in comments to HBASE-10915, we need to have a proper way to
> retrieve instance of consensus provider, and Server interface seems the right
> one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11092: Status: Patch Available (was: Open) > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11092: Attachment: HBASE-11092.patch Initial patch for review. > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11097) Allow 0.89-fb to work with ipv6 addresses
Amitanand Aiyer created HBASE-11097: --- Summary: Allow 0.89-fb to work with ipv6 addresses Key: HBASE-11097 URL: https://issues.apache.org/jira/browse/HBASE-11097 Project: HBase Issue Type: New Feature Affects Versions: 0.89-fb Reporter: Amitanand Aiyer -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11086) Add htrace support for PerfEval
[ https://issues.apache.org/jira/browse/HBASE-11086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984899#comment-13984899 ] Hudson commented on HBASE-11086: FAILURE: Integrated in HBase-TRUNK #5124 (See [https://builds.apache.org/job/HBase-TRUNK/5124/]) HBASE-11086 Add htrace support for PerfEval (ndimiduk: rev 1591001) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add htrace support for PerfEval > --- > > Key: HBASE-11086 > URL: https://issues.apache.org/jira/browse/HBASE-11086 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11086.00.patch, HBASE-11086.01.patch > > > Per the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11082) Potential unclosed TraceScope in FSHLog#replaceWriter()
[
https://issues.apache.org/jira/browse/HBASE-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984897#comment-13984897
]
Hudson commented on HBASE-11082:
FAILURE: Integrated in HBase-TRUNK #5124 (See
[https://builds.apache.org/job/HBase-TRUNK/5124/])
HBASE-11082 Potential unclosed TraceScope in FSHLog#replaceWriter() (ndimiduk:
rev 1590997)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
> Potential unclosed TraceScope in FSHLog#replaceWriter()
> ---
>
> Key: HBASE-11082
> URL: https://issues.apache.org/jira/browse/HBASE-11082
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Ted Yu
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-11082.00.patch
>
>
> In the finally block starting at line 924:
> {code}
> } finally {
> // Let the writer thread go regardless, whether error or not.
> if (zigzagLatch != null) {
> zigzagLatch.releaseSafePoint();
> // It will be null if we failed our wait on safe point above.
> if (syncFuture != null) blockOnSync(syncFuture);
> }
> scope.close();
> {code}
> If blockOnSync() throws IOException, the TraceScope would be left unclosed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-5697) Audit HBase for usage of deprecated hadoop 0.20.x property names.
[ https://issues.apache.org/jira/browse/HBASE-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984898#comment-13984898 ] Hudson commented on HBASE-5697: --- FAILURE: Integrated in HBase-TRUNK #5124 (See [https://builds.apache.org/job/HBase-TRUNK/5124/]) HBASE-5697 Audit HBase for usage of deprecated hadoop 0.20.x, 1.x property names (Srikanth Srungarapu) (jmhsieh: rev 1591045) * /hbase/trunk/bin/region_status.rb * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/compress/Compression.java * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestImportTsv.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Export.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/package-info.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithOperationAttributes.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithVisibilityLabels.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestJoinedScanners.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java * /hbase/trunk/src/main/docbkx/case_studies.xml * /hbase/trunk/src/main/docbkx/configuration.xml * /hbase/trunk/src/main/docbkx/ops_mgt.xml * /hbase/trunk/src/main/docbkx/troubleshooting.xml > Audit HBase for usage of deprecated hadoop 0.20.x property names. > - > > Key: HBASE-5697 > URL: https://issues.apache.org/jira/browse/HBASE-5697 > Project: HBase > Issue Type: Task >Reporter: Jonathan Hsieh >Assignee: Srikanth Srungarapu > Labels: noob > Fix Fo
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[ https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984895#comment-13984895 ] Ted Yu commented on HBASE-11093: Considering the responses I got on this issue (I do appreciate the opportunity that allows me to evaluate this in a bigger context), I think the priority should be at minor level. bq. Do you have numbers on savings this code change makes Not yet. The savings would vary depending on the number of KeyValues handed over to FilterList and the number of filters that are part of the FilterList. > FilterList#filterRow() iterates through its filters even though > FilterList#hasFilterRow() returns false > --- > > Key: HBASE-11093 > URL: https://issues.apache.org/jira/browse/HBASE-11093 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt > > > FilterList#hasFilterRow() returns false when hasFilterRow() of all of > constituent filters return false. > However, FilterList#filterRow() still iterates through its filters in this > scenario. > The iteration should be skipped when FilterList#hasFilterRow() returns false. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[ https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984893#comment-13984893 ] stack commented on HBASE-11093: --- [~ted_yu] Yeah. I think I get it. The takeaway seems to be lets do busy work until 1.0 on trivial issues -- this should be trivial priority, right? Do you have numbers on savings this code change makes -- that sucks in FOUR reviewers for a patch over code that should really be removed (IIUC). > FilterList#filterRow() iterates through its filters even though > FilterList#hasFilterRow() returns false > --- > > Key: HBASE-11093 > URL: https://issues.apache.org/jira/browse/HBASE-11093 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt > > > FilterList#hasFilterRow() returns false when hasFilterRow() of all of > constituent filters return false. > However, FilterList#filterRow() still iterates through its filters in this > scenario. > The iteration should be skipped when FilterList#hasFilterRow() returns false. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6192: - Fix Version/s: 0.99.0 > Document ACL matrix in the book > --- > > Key: HBASE-6192 > URL: https://issues.apache.org/jira/browse/HBASE-6192 > Project: HBase > Issue Type: Task > Components: documentation, security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Enis Soztutar > Labels: documentaion, security > Fix For: 0.99.0 > > Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL > Matrix.pdf, HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls, > HBase Security-ACL Matrix.xls, HBase Security-ACL Matrix.xls > > > We have an excellent matrix at > https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf > for ACL. Once the changes are done, we can adapt that and put it in the > book, also add some more documentation about the new authorization features. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10251) Restore API Compat for PerformanceEvaluation.generateValue()
[ https://issues.apache.org/jira/browse/HBASE-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984855#comment-13984855 ] Andrew Purtell commented on HBASE-10251: I'd be happy to review/support/commit such an effort. > Restore API Compat for PerformanceEvaluation.generateValue() > > > Key: HBASE-10251 > URL: https://issues.apache.org/jira/browse/HBASE-10251 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.99.0 >Reporter: Aleksandr Shulman > Labels: api_compatibility > > Observed: > A couple of my client tests fail to compile against trunk because the method > PerformanceEvaluation.generateValue was removed as part of HBASE-8496. > This is an issue because it was used in a number of places, including unit > tests. Since we did not explicitly label this API as private, it's ambiguous > as to whether this could/should have been used by people writing apps against > 0.96. If they used it, then they would be broken upon upgrade to 0.98 and > trunk. > Potential Solution: > The method was renamed to generateData, but the logic is still the same. We > can reintroduce it as deprecated in 0.98, as compat shim over generateData. > The patch should be a few lines. We may also consider doing so in trunk, but > I'd be just as fine with leaving it out. > More generally, this raises the question about what other code is in this > "grey-area", where it is public, is used outside of the package, but is not > explicitly labeled with an AudienceInterface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[ https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984856#comment-13984856 ] Ted Yu commented on HBASE-11093: [~stack]: Did I answer your question ? Kindly provide your suggestion. > FilterList#filterRow() iterates through its filters even though > FilterList#hasFilterRow() returns false > --- > > Key: HBASE-11093 > URL: https://issues.apache.org/jira/browse/HBASE-11093 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt > > > FilterList#hasFilterRow() returns false when hasFilterRow() of all of > constituent filters return false. > However, FilterList#filterRow() still iterates through its filters in this > scenario. > The iteration should be skipped when FilterList#hasFilterRow() returns false. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11093) FilterList#filterRow() iterates through its filters even though FilterList#hasFilterRow() returns false
[
https://issues.apache.org/jira/browse/HBASE-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984842#comment-13984842
]
Hadoop QA commented on HBASE-11093:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642477/11093-v3.txt
against trunk revision .
ATTACHMENT ID: 12642477
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestTableMapReduceBase.testMultiRegionTable(TestTableMapReduceBase.java:96)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9425//console
This message is automatically generated.
> FilterList#filterRow() iterates through its filters even though
> FilterList#hasFilterRow() returns false
> ---
>
> Key: HBASE-11093
> URL: https://issues.apache.org/jira/browse/HBASE-11093
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 11093-v1.txt, 11093-v2.txt, 11093-v3.txt
>
>
> FilterList#hasFilterRow() returns false when hasFilterRow() of all of
> constituent filters return false.
> However, FilterList#filterRow() still iterates through its filters in this
> scenario.
> The iteration should be skipped when FilterList#hasFilterRow() returns false.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10251) Restore API Compat for PerformanceEvaluation.generateValue()
[ https://issues.apache.org/jira/browse/HBASE-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984841#comment-13984841 ] Aleksandr Shulman commented on HBASE-10251: --- [~ndimiduk], I like the idea of having utility classes that have a compatibility story, from which the tests (which do not have compatibility considerations) can pull. > Restore API Compat for PerformanceEvaluation.generateValue() > > > Key: HBASE-10251 > URL: https://issues.apache.org/jira/browse/HBASE-10251 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.99.0 >Reporter: Aleksandr Shulman > Labels: api_compatibility > > Observed: > A couple of my client tests fail to compile against trunk because the method > PerformanceEvaluation.generateValue was removed as part of HBASE-8496. > This is an issue because it was used in a number of places, including unit > tests. Since we did not explicitly label this API as private, it's ambiguous > as to whether this could/should have been used by people writing apps against > 0.96. If they used it, then they would be broken upon upgrade to 0.98 and > trunk. > Potential Solution: > The method was renamed to generateData, but the logic is still the same. We > can reintroduce it as deprecated in 0.98, as compat shim over generateData. > The patch should be a few lines. We may also consider doing so in trunk, but > I'd be just as fine with leaving it out. > More generally, this raises the question about what other code is in this > "grey-area", where it is public, is used outside of the package, but is not > explicitly labeled with an AudienceInterface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10251) Restore API Compat for PerformanceEvaluation.generateValue()
[ https://issues.apache.org/jira/browse/HBASE-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984838#comment-13984838 ] Andrew Purtell commented on HBASE-10251: bq. We should consider how much code-reuse we want to make available for other tests vs stand-alone application, where a util library class resides, etc. Agreed [~ndimiduk], and I think the supporting classes that implement the functionality of LoadTestTool fall into the same bucket. > Restore API Compat for PerformanceEvaluation.generateValue() > > > Key: HBASE-10251 > URL: https://issues.apache.org/jira/browse/HBASE-10251 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.99.0 >Reporter: Aleksandr Shulman > Labels: api_compatibility > > Observed: > A couple of my client tests fail to compile against trunk because the method > PerformanceEvaluation.generateValue was removed as part of HBASE-8496. > This is an issue because it was used in a number of places, including unit > tests. Since we did not explicitly label this API as private, it's ambiguous > as to whether this could/should have been used by people writing apps against > 0.96. If they used it, then they would be broken upon upgrade to 0.98 and > trunk. > Potential Solution: > The method was renamed to generateData, but the logic is still the same. We > can reintroduce it as deprecated in 0.98, as compat shim over generateData. > The patch should be a few lines. We may also consider doing so in trunk, but > I'd be just as fine with leaving it out. > More generally, this raises the question about what other code is in this > "grey-area", where it is public, is used outside of the package, but is not > explicitly labeled with an AudienceInterface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[
https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984830#comment-13984830
]
Hadoop QA commented on HBASE-10885:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642441/HBASE-10885_2.patch
against trunk revision .
ATTACHMENT ID: 12642441
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOpWithDifferentUsersNoACL
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9426//console
This message is automatically generated.
> Support visibility expressions on Deletes
> -
>
> Key: HBASE-10885
> URL: https://issues.apache.org/jira/browse/HBASE-10885
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.98.1
>Reporter: Andrew Purtell
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch
>
>
> Accumulo can specify visibility expressions for delete markers. During
> compaction the cells covered by the tombstone are determined in part by
> matching the visibility expression. This is useful for the use case of data
> set coalescing, where entries from multiple data sets carrying different
> labels are combined into one common large table. Later, a subset of entries
> can be conveniently removed using visibility expressions.
> Currently doing the same in HBase would only be possible with a custom
> coprocessor. Otherwise, a Delete will affect all cells covered by the
> tombstone regardless of any visibility expression scoping. This is correct
> behavior in that no data spill is possible, but certainly could be
> surprising, and is only meant to be transitional. We decided not to support
> visibility expressions on Deletes to control the complexity of the initial
> implementation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10251) Restore API Compat for PerformanceEvaluation.generateValue()
[ https://issues.apache.org/jira/browse/HBASE-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984818#comment-13984818 ] Nick Dimiduk commented on HBASE-10251: -- [~apurtell] yes I recall the conversation; no I don't remember the context. I think it's a good idea, and is on my todo list. One of the prerequisites in my mind is that we add some unit test coverage around how the various features overlap in test configuration. Quasi-related, I hacked it up pretty badly in HBASE-10791 for an integration test on the HBASE-10070 branch. We should consider how much code-reuse we want to make available for other tests vs stand-alone application, where a util library class resides, etc. We already have PerformanceEvaluationCommons, which seems like an ideal place for the functionality [~aleksshulman] is referring to. > Restore API Compat for PerformanceEvaluation.generateValue() > > > Key: HBASE-10251 > URL: https://issues.apache.org/jira/browse/HBASE-10251 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.99.0 >Reporter: Aleksandr Shulman > Labels: api_compatibility > > Observed: > A couple of my client tests fail to compile against trunk because the method > PerformanceEvaluation.generateValue was removed as part of HBASE-8496. > This is an issue because it was used in a number of places, including unit > tests. Since we did not explicitly label this API as private, it's ambiguous > as to whether this could/should have been used by people writing apps against > 0.96. If they used it, then they would be broken upon upgrade to 0.98 and > trunk. > Potential Solution: > The method was renamed to generateData, but the logic is still the same. We > can reintroduce it as deprecated in 0.98, as compat shim over generateData. > The patch should be a few lines. We may also consider doing so in trunk, but > I'd be just as fine with leaving it out. > More generally, this raises the question about what other code is in this > "grey-area", where it is public, is used outside of the package, but is not > explicitly labeled with an AudienceInterface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10925) Do not OOME, throw RowTooBigException instead
[ https://issues.apache.org/jira/browse/HBASE-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10925: Affects Version/s: 0.99.0 > Do not OOME, throw RowTooBigException instead > - > > Key: HBASE-10925 > URL: https://issues.apache.org/jira/browse/HBASE-10925 > Project: HBase > Issue Type: Improvement > Components: Usability >Affects Versions: 0.99.0 >Reporter: stack >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-10925.patch, HBASE-10925.patch, HBASE-10925.patch > > > If 10M columns in a row, throw a RowTooBigException rather than OOME when > Get'ing or Scanning w/o in-row scan flag set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11095) Add ip restriction in user permissions
[
https://issues.apache.org/jira/browse/HBASE-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984812#comment-13984812
]
Andrew Purtell commented on HBASE-11095:
After considering the above if we would like to proceed, then I recommend a
reboot of the authorization code in the AccessController instead of hacking in
IP filtering.
> Add ip restriction in user permissions
> --
>
> Key: HBASE-11095
> URL: https://issues.apache.org/jira/browse/HBASE-11095
> Project: HBase
> Issue Type: New Feature
> Components: security
>Reporter: Liu Shaohui
>Priority: Minor
>
> For some sensitive data, users want to restrict the from ips of hbase users
> like mysql access control.
> One direct solution is to add the candidated ips when granting user
> permisions.
> {quote}
> grant [ [ [
> ] ] ]
> {quote}
> Any comments and suggestions are welcomed.
> [~apurtell]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-3045) Extend HBASE-3025 into a role based access control model using "HBase groups"
[ https://issues.apache.org/jira/browse/HBASE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-3045. --- Resolution: Later Assignee: (was: Eugene Koontz) Hadoop group mapping seems good enough for now. Resolving as Later. > Extend HBASE-3025 into a role based access control model using "HBase groups" > - > > Key: HBASE-3045 > URL: https://issues.apache.org/jira/browse/HBASE-3045 > Project: HBase > Issue Type: Sub-task > Components: security >Reporter: Andrew Purtell > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984808#comment-13984808 ] Sergey Soldatov commented on HBASE-11092: - The problem is that the Server interface is defined in hbase-client module, but most of consensuses requre classes from hbase-server module. One of the possible options is to define a basic consensus provider with default start/stop/init methods in the hbase-client module and subclass with all required consensus getters in hbase-server module and use explicit cast to this subclass when it's require. > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-5947) Check for valid user/table/family/qualifier and acl state
[ https://issues.apache.org/jira/browse/HBASE-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-5947: -- Assignee: (was: Matteo Bertozzi) > Check for valid user/table/family/qualifier and acl state > - > > Key: HBASE-5947 > URL: https://issues.apache.org/jira/browse/HBASE-5947 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Matteo Bertozzi > Labels: acl > > HBase Shell grant/revoke doesn't check for valid user or > table/family/qualifier so can you end up having rights for something that > doesn't exists. > We might also want to ensure, upon table/column creation, that no entries are > already stored at the acl table. We might still have residual acl entries if > something goes wrong, in postDeleteTable(), postDeleteColumn(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-5947) Check for valid user/table/family/qualifier and acl state
[ https://issues.apache.org/jira/browse/HBASE-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-5947: -- Issue Type: Improvement (was: Sub-task) Parent: (was: HBASE-5352) > Check for valid user/table/family/qualifier and acl state > - > > Key: HBASE-5947 > URL: https://issues.apache.org/jira/browse/HBASE-5947 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Labels: acl > > HBase Shell grant/revoke doesn't check for valid user or > table/family/qualifier so can you end up having rights for something that > doesn't exists. > We might also want to ensure, upon table/column creation, that no entries are > already stored at the acl table. We might still have residual acl entries if > something goes wrong, in postDeleteTable(), postDeleteColumn(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6106) Update documentation to reflect design and interface changes
[ https://issues.apache.org/jira/browse/HBASE-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6106. --- Resolution: Fixed Superseded by subsequent documentation updates. Retired with parent. > Update documentation to reflect design and interface changes > > > Key: HBASE-6106 > URL: https://issues.apache.org/jira/browse/HBASE-6106 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, documentation, security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Andrew Purtell > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6186) Assure thread safety
[ https://issues.apache.org/jira/browse/HBASE-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6186. --- Resolution: Invalid > Assure thread safety > > > Key: HBASE-6186 > URL: https://issues.apache.org/jira/browse/HBASE-6186 > Project: HBase > Issue Type: Sub-task > Components: security >Affects Versions: 0.94.0, 0.94.1, 0.95.2 >Reporter: Andrew Purtell > Labels: security > > Check data structures for necessary thread safety. > One known issue is as follows: > In TableAuthManager, we need to deal with concurrent grants and revokes now > that ADMIN permission can be delegated to perhaps multiple users. > Consider using a ReadWriteLock to protect for update. The current data > structure is an array multimap. It is also possible to obtain a synchronized > multimap from Guava with Multimaps#synchronizedListMultimap but by far the > most common access is read access, and concurrent read access is fine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6186) Assure thread safety
[ https://issues.apache.org/jira/browse/HBASE-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6186: -- Assignee: (was: Laxman) > Assure thread safety > > > Key: HBASE-6186 > URL: https://issues.apache.org/jira/browse/HBASE-6186 > Project: HBase > Issue Type: Sub-task > Components: security >Affects Versions: 0.94.0, 0.94.1, 0.95.2 >Reporter: Andrew Purtell > Labels: security > > Check data structures for necessary thread safety. > One known issue is as follows: > In TableAuthManager, we need to deal with concurrent grants and revokes now > that ADMIN permission can be delegated to perhaps multiple users. > Consider using a ReadWriteLock to protect for update. The current data > structure is an array multimap. It is also possible to obtain a synchronized > multimap from Guava with Multimaps#synchronizedListMultimap but by far the > most common access is read access, and concurrent read access is fine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-7333) Improve the security shell commands
[ https://issues.apache.org/jira/browse/HBASE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-7333. --- Resolution: Not a Problem Assignee: (was: Andrew Purtell) Superseded by a lot of commits. > Improve the security shell commands > --- > > Key: HBASE-7333 > URL: https://issues.apache.org/jira/browse/HBASE-7333 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, security, shell >Affects Versions: 0.94.4, 0.95.2 >Reporter: Andrew Purtell > > The shell commands for security are rudimentary and should be improved. The > commands we have need to be updated for the "AccessController v2" changes. > The distinction between the shell and the Java (admin) API is blurry because > our shell is JRuby but it makes sense to provide some convenient shortcuts > for common actions. > At a minimum the current set of commands should validate their arguments. > 'revoke' should be improved so all access for a given user can be > conveniently revoked with one command, as opposed to requiring a specific > revoke for every previous grant. This may involve interaction with a master > mediated transaction or barrier framework. > Once HBASE-6222 is in, it should be possible to conveniently construct ACLs > and add them to DML ops like put and delete; and there should be support for > dumping ACLs at the cell level too. > Also, I observed 'user_permission' fail with NPEs on a colleague's > workstation recently. It could have been the local environment, but I suspect > there may be some rot here. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6099) Secure ZooKeeper integration changes
[ https://issues.apache.org/jira/browse/HBASE-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6099. --- Resolution: Fixed "Sub-umbrella" retired with parent. > Secure ZooKeeper integration changes > > > Key: HBASE-6099 > URL: https://issues.apache.org/jira/browse/HBASE-6099 > Project: HBase > Issue Type: Sub-task > Components: Client, security, shell, Zookeeper >Affects Versions: 0.94.1, 0.95.2 >Reporter: Andrew Purtell > > Resolve linked issues. > Audit HBase client use of znodes and consider if we should tighten up access > anywhere. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7123) Refactor internal methods in AccessController
[
https://issues.apache.org/jira/browse/HBASE-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-7123:
--
Assignee: (was: Andrew Purtell)
> Refactor internal methods in AccessController
> -
>
> Key: HBASE-7123
> URL: https://issues.apache.org/jira/browse/HBASE-7123
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Andrew Purtell
>
> The authorize(), permissionGranted(), and requirePermission() methods in
> AccessController have organically grown as both the HBase client API and the
> AccessController itself have evolved, and now have several problems:
> - Code duplication (minor)
> - Unused variants (minor)
> - Signatures optimized for checking certain operations that have a familyMap.
> Unfortunately different operations have different ideas of what type a
> familyMap should be. This leads to runtime type checking and the need to
> convert one family map to another (e.g. {{Map NavigableMap>}} to {{Map>}} (That kind of
> conversion code in a hot path hurts to look at.) There are too many Java
> collection type combinations floating around. Some of this should be
> approached at the client API level too, for example with HBASE-7114.
> - Only one Permission.Action can be checked at a time. We should really
> convert these into a bitmap if multiple actions need checking and pass that
> around instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-7123) Refactor internal methods in AccessController
[
https://issues.apache.org/jira/browse/HBASE-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-7123:
--
Issue Type: Improvement (was: Sub-task)
Parent: (was: HBASE-6096)
> Refactor internal methods in AccessController
> -
>
> Key: HBASE-7123
> URL: https://issues.apache.org/jira/browse/HBASE-7123
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>
> The authorize(), permissionGranted(), and requirePermission() methods in
> AccessController have organically grown as both the HBase client API and the
> AccessController itself have evolved, and now have several problems:
> - Code duplication (minor)
> - Unused variants (minor)
> - Signatures optimized for checking certain operations that have a familyMap.
> Unfortunately different operations have different ideas of what type a
> familyMap should be. This leads to runtime type checking and the need to
> convert one family map to another (e.g. {{Map NavigableMap>}} to {{Map>}} (That kind of
> conversion code in a hot path hurts to look at.) There are too many Java
> collection type combinations floating around. Some of this should be
> approached at the client API level too, for example with HBASE-7114.
> - Only one Permission.Action can be checked at a time. We should really
> convert these into a bitmap if multiple actions need checking and pass that
> around instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-6030) Log AccessDeniedExceptions at INFO level
[ https://issues.apache.org/jira/browse/HBASE-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6030. --- Resolution: Not a Problem Assignee: (was: Andrew Purtell) > Log AccessDeniedExceptions at INFO level > > > Key: HBASE-6030 > URL: https://issues.apache.org/jira/browse/HBASE-6030 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, security >Affects Versions: 0.92.2, 0.94.1, 0.95.2 > Environment: AccessController coprocessor >Reporter: Andrew Purtell >Priority: Minor > > The HBase AccessController can emit a detailed trace of every action, and > whether it succeeded or failed, but this is expected to be used only for > debugging an application in a staging environment. In production a > RegionServer may have thousands of requests per second, logging the audit > trace just isn't viable. However, we might want to log the > AccessDeniedExceptions which result from access failures in the daemon logs > like Hadoop does, e.g. the NameNode log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6192: -- Assignee: (was: Laxman) > Document ACL matrix in the book > --- > > Key: HBASE-6192 > URL: https://issues.apache.org/jira/browse/HBASE-6192 > Project: HBase > Issue Type: Task > Components: documentation, security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Enis Soztutar > Labels: documentaion, security > Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL > Matrix.pdf, HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls, > HBase Security-ACL Matrix.xls, HBase Security-ACL Matrix.xls > > > We have an excellent matrix at > https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf > for ACL. Once the changes are done, we can adapt that and put it in the > book, also add some more documentation about the new authorization features. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-5343) Access control API in HBaseAdmin.java
[ https://issues.apache.org/jira/browse/HBASE-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-5343. --- Resolution: Invalid Assignee: (was: Enis Soztutar) > Access control API in HBaseAdmin.java > --- > > Key: HBASE-5343 > URL: https://issues.apache.org/jira/browse/HBASE-5343 > Project: HBase > Issue Type: Improvement > Components: Client, Coprocessors, security >Affects Versions: 0.92.1, 0.94.0 >Reporter: Enis Soztutar > > To use the access control mechanism added in HBASE-3025, users should either > use the shell interface, or use the coprocessor API directly, which is not > very user friendly. We can add grant/revoke/user_permission commands similar > to the shell interface to HBaseAdmin assuming HBASE-5341 is in. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6101) Insure Observers cover all relevant RPC and lifecycle code paths
[ https://issues.apache.org/jira/browse/HBASE-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6101. --- Resolution: Fixed > Insure Observers cover all relevant RPC and lifecycle code paths > > > Key: HBASE-6101 > URL: https://issues.apache.org/jira/browse/HBASE-6101 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master, regionserver, security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Andrew Purtell > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-5352) ACL improvements
[ https://issues.apache.org/jira/browse/HBASE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-5352. --- Resolution: Fixed Assignee: (was: Enis Soztutar) This umbrella has seen it's day. Will spin out still relevant unfinished subtasks to top level issues. > ACL improvements > > > Key: HBASE-5352 > URL: https://issues.apache.org/jira/browse/HBASE-5352 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.92.1, 0.94.0 >Reporter: Enis Soztutar > > In this issue I would like to open discussion for a few minor ACL related > improvements. The proposed changes are as follows: > 1. Introduce something like > AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so > that clients can check access rights before carrying out the operations. We > need this kind of operation for HCATALOG-245, which introduces authorization > providers for hbase over hcat. We cannot use getUserPermissions() since it > requires ADMIN permissions on the global/table level. > 2. getUserPermissions(tableName)/grant/revoke and drop/modify table > operations should not check for global CREATE/ADMIN rights, but table > CREATE/ADMIN rights. The reasoning is that if a user is able to admin or read > from a table, she should be able to read the table's permissions. We can > choose whether we want only READ or ADMIN permissions for > getUserPermission(). Since we check for global permissions first for table > permissions, configuring table access using global permissions will continue > to work. > 3. Grant/Revoke global permissions - HBASE-5342 (included for completeness) > From all 3, we may want to backport the first one to 0.92 since without it, > Hive/Hcatalog cannot use Hbase's authorization mechanism effectively. > I will create subissues and convert HBASE-5342 to a subtask when we get some > feedback, and opinions for going further. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6192: -- Issue Type: Task (was: Sub-task) Parent: (was: HBASE-5352) > Document ACL matrix in the book > --- > > Key: HBASE-6192 > URL: https://issues.apache.org/jira/browse/HBASE-6192 > Project: HBase > Issue Type: Task > Components: documentation, security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Enis Soztutar >Assignee: Laxman > Labels: documentaion, security > Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL > Matrix.pdf, HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls, > HBase Security-ACL Matrix.xls, HBase Security-ACL Matrix.xls > > > We have an excellent matrix at > https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf > for ACL. Once the changes are done, we can adapt that and put it in the > book, also add some more documentation about the new authorization features. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6096. --- Resolution: Fixed This umbrella has seen it's day. Resolving and spinning out still relevant subtasks to top level issues. > AccessController v2 > --- > > Key: HBASE-6096 > URL: https://issues.apache.org/jira/browse/HBASE-6096 > Project: HBase > Issue Type: Umbrella > Components: security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Andrew Purtell > Attachments: Security-ACL Matrix.pdf > > > Umbrella issue for iteration on the initial AccessController drop. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6098) ACL design changes
[ https://issues.apache.org/jira/browse/HBASE-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6098. --- Resolution: Invalid Invalid "sub-umbrella". I should have just linked the issues here to the parent. > ACL design changes > -- > > Key: HBASE-6098 > URL: https://issues.apache.org/jira/browse/HBASE-6098 > Project: HBase > Issue Type: Sub-task > Components: security >Affects Versions: 0.94.1, 0.95.2 >Reporter: Andrew Purtell > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11095) Add ip restriction in user permissions
[
https://issues.apache.org/jira/browse/HBASE-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984785#comment-13984785
]
Andrew Purtell commented on HBASE-11095:
I've thought about this some. We could build a service authorization engine
supporting fine grained decisionmaking using attributes such as client IP
address (surely the first of many), but consider:
First, Hadoop already has a nascient service authorization framework, see
https://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html
At least when our secure RPC was first committed we picked up support for
service authorization for all of our RPC protocols from the Hadoop libraries.
The current Hadoop code does not support IP filtering but it could.
Second, even if we build our own fine grained service authorization, will
Hadoop someday introduce the same facilities? There is HADOOP-9466. Incubating
projects like Apache Sentry are also working toward this kind of capability.
Third, see HBASE-7123 and HBASE-7254. To the extent that technical debt should
be paid down in the AccessController, we should factor that in.
permissionGranted and the like are where you'd start with today's code to
implement restrictions by IP address.
> Add ip restriction in user permissions
> --
>
> Key: HBASE-11095
> URL: https://issues.apache.org/jira/browse/HBASE-11095
> Project: HBase
> Issue Type: New Feature
> Components: security
>Reporter: Liu Shaohui
>Priority: Minor
>
> For some sensitive data, users want to restrict the from ips of hbase users
> like mysql access control.
> One direct solution is to add the candidated ips when granting user
> permisions.
> {quote}
> grant [ [ [
> ] ] ]
> {quote}
> Any comments and suggestions are welcomed.
> [~apurtell]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10653) Incorrect table status in HBase shell Describe
[
https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984711#comment-13984711
]
Hadoop QA commented on HBASE-10653:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642218/HBASE-10653.2.patch
against trunk revision .
ATTACHMENT ID: 12642218
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9424//console
This message is automatically generated.
> Incorrect table status in HBase shell Describe
> --
>
> Key: HBASE-10653
> URL: https://issues.apache.org/jira/browse/HBASE-10653
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.94.17
>Reporter: Biju Nair
>Assignee: Rekha Joshi
> Labels: HbaseShell, describe
> Fix For: 0.98.0
>
> Attachments: HBASE-10653.1.patch, HBASE-10653.2.patch
>
>
> Describe output of table which is disabled shows as enabled.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10611) Description for hbase:acl table is wrong on master-status#catalogTables
[
https://issues.apache.org/jira/browse/HBASE-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10611:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Integrated to 0.98 and trunk.
Thanks for the patch, Rekha.
> Description for hbase:acl table is wrong on master-status#catalogTables
> ---
>
> Key: HBASE-10611
> URL: https://issues.apache.org/jira/browse/HBASE-10611
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Ted Yu
>Assignee: Rekha Joshi
>Priority: Minor
> Labels: patch
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10611.1.patch, HBASE-10611.1.patch,
> HBASE-10611.2.patch
>
>
> On master-status#catalogTables, I see:
> hbase:acl The .NAMESPACE. table holds information about namespaces.
> In MasterStatusTmpl.jamon, we have:
> {code}
> if (tableName.equals(TableName.META_TABLE_NAME)){
> description = "The hbase:meta table holds references to all User
> Table regions";
> } else {
> description = "The .NAMESPACE. table holds information about
> namespaces.";
> }
> {code}
> The above check doesn't cover hbase:acl table.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9003) TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar
[
https://issues.apache.org/jira/browse/HBASE-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984705#comment-13984705
]
Hadoop QA commented on HBASE-9003:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642412/HBASE-9003.v1.patch
against trunk revision .
ATTACHMENT ID: 12642412
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9423//console
This message is automatically generated.
> TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar
> -
>
> Key: HBASE-9003
> URL: https://issues.apache.org/jira/browse/HBASE-9003
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 0.99.0
>
> Attachments: HBASE-9003.v0.patch, HBASE-9003.v1.patch
>
>
> This is the problem: {{TableMapReduceUtil#addDependencyJars}} relies on
> {{org.apache.hadoop.util.JarFinder}} if available to call {{getJar()}}.
> However {{getJar()}} uses File.createTempFile() to create a temporary file
> under {{hadoop.tmp.dir}}{{/target/test-dir}}. Due HADOOP-9737 the created jar
> and its content is not purged after the JVM is destroyed. Since most
> configurations point {{hadoop.tmp.dir}} under {{/tmp}} the generated jar
> files get purged by {{tmpwatch}} or a similar tool, but boxes that have
> {{hadoop.tmp.dir}} pointing to a different location not monitored by
> {{tmpwatch}} will pile up a collection of jars causing all kind of issues.
> Since {{JarFinder#getJar}} is not a public API from Hadoop (see [~tucu00]
> comment on HADOOP-9737) we shouldn't use that as part of
> {{TableMapReduceUtil}} in order to avoid this kind of issues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
