[jira] [Resolved] (HBASE-19145) Look into hbase-2 client going to hbase-1 server
[ https://issues.apache.org/jira/browse/HBASE-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-19145. -- Resolution: Done > Look into hbase-2 client going to hbase-1 server > > > Key: HBASE-19145 > URL: https://issues.apache.org/jira/browse/HBASE-19145 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0-beta-1 >Reporter: Jerry He >Assignee: Jerry He >Priority: Major > > From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread. > Do we support hbase-2 client going against hbase-1 server? > We seem to be fine mix-and-match the clients and servers within the > hbase-1 releases. And IIRC hbase-1 client is ok against 0.98 server. > Suppose I have a product that depends and bundles HBase client. I > want to upgrade the dependency to hbase-2 so that it can take > advantages of and claim support of hbase-2. > But does it mean that I will need drop the claims that the new version > of the product support any hbase-1 backend? > It has not been an objective. It might work doing basic Client API on a > later branch-1 but will fail doing Admin functions (and figuring if a Table > is online). If it was a small thing to make it > work, lets get it in. > Let's look into it to see what works and what not. Have a statement at least. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker
Jerry He created HBASE-21008:
Summary: HBase 1.x can not read HBase2 hfiles due to
TimeRangeTracker
Key: HBASE-21008
URL: https://issues.apache.org/jira/browse/HBASE-21008
Project: HBase
Issue Type: Bug
Affects Versions: 1.4.6, 2.1.0
Reporter: Jerry He
It looks like HBase 1.x can not open hfiiles written by HBase2 still.
I tested the latest HBase 1.4.6 and 2.1.0. 1.4.6 tried to read and open
regions written by 2.1.0.
{code}
2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1]
regionserver.StoreFile: Error reading timestamp range data from meta --
proceeding without
java.lang.IllegalArgumentException: Timestamp cannot be negative.
minStamp:5783278630776778969, maxStamp:-4698050386518222402
at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
at
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
at
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
at
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
{code}
Or:
{code}
2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0]
handler.OpenRegionHandler: Failed open of
region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting to
roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.EOFException
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.EOFException
at
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
at
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.readFields(TimeRangeTracker.java:170)
at
org.apache.hadoop.hbase.util.Writables.copyWritable(Writables.java:161)
at
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRangeTracker(TimeRangeTracker.java:187)
at
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:197)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
at
[jira] [Created] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
Jerry He created HBASE-20565: Summary: ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4 Key: HBASE-20565 URL: https://issues.apache.org/jira/browse/HBASE-20565 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 1.4.4 Reporter: Jerry He When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see incorrect result. Here is a simple example. One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, ColumnPaginationFilter). We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns are returned. In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19557) Build and release source jars for hbase-shaded-client and others
Jerry He created HBASE-19557: Summary: Build and release source jars for hbase-shaded-client and others Key: HBASE-19557 URL: https://issues.apache.org/jira/browse/HBASE-19557 Project: HBase Issue Type: Improvement Affects Versions: 1.2.6, 1.3.1 Reporter: Jerry He It seems that currently we don't build and release source jars for hbase-shaded-client (and server or mapreduce). IDEs will complain from the dependent users. We should provide them. http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-19171. -- Resolution: Duplicate Fix Version/s: (was: 2.0.0-beta-1) Resolve as dup of the other task [~mdrob] is working on. > Update package references to match new shaded offset in hbase-thirdparty > > > Key: HBASE-19171 > URL: https://issues.apache.org/jira/browse/HBASE-19171 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Priority: Critical > > This has dependency on the parent task, and can only be done after a new > hbase-thirdparty release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty
Jerry He created HBASE-19171: Summary: Update package references to match new shaded offset in hbase-thirdparty Key: HBASE-19171 URL: https://issues.apache.org/jira/browse/HBASE-19171 Project: HBase Issue Type: Sub-task Reporter: Jerry He Priority: Critical Fix For: 2.0.0 This has dependency on the parent task, and can only be done after a new hbase-thirdparty release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19170) [hbase-thirdparty] Change the relocation offset of shaded artifacts
Jerry He created HBASE-19170: Summary: [hbase-thirdparty] Change the relocation offset of shaded artifacts Key: HBASE-19170 URL: https://issues.apache.org/jira/browse/HBASE-19170 Project: HBase Issue Type: Bug Affects Versions: 1.0.1 Reporter: Jerry He Priority: Critical Fix For: 1.0.2 On the dev@hbase list, we conclude that we need to change the relocation offset in hbase-thirdparty to avoid shading conflicts with the other hbase shaded components (hbase-shaded-client and hbase-shaded-mapreduce components). https://lists.apache.org/thread.html/1aa5d1d7f6d176df49e72096926b011cafe1315932515346d06e8342@%3Cdev.hbase.apache.org%3E The suggestion is to use "o.a.h.hbase.thirdparty" in hbase-thirdparty to differentiate between "shaded" for downstream of us vs "thirdparty" for our internal use. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19145) Look into hbase-2 client going to hbase-1 server
Jerry He created HBASE-19145: Summary: Look into hbase-2 client going to hbase-1 server Key: HBASE-19145 URL: https://issues.apache.org/jira/browse/HBASE-19145 Project: HBase Issue Type: Task Affects Versions: 2.0.0-beta-1 Reporter: Jerry He Priority: Major >From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread. Do we support hbase-2 client going against hbase-1 server? We seem to be fine mix-and-match the clients and servers within the hbase-1 releases. And IIRC hbase-1 client is ok against 0.98 server. Suppose I have a product that depends and bundles HBase client. I want to upgrade the dependency to hbase-2 so that it can take advantages of and claim support of hbase-2. But does it mean that I will need drop the claims that the new version of the product support any hbase-1 backend? It has not been an objective. It might work doing basic Client API on a later branch-1 but will fail doing Admin functions (and figuring if a Table is online). If it was a small thing to make it work, lets get it in. Let's look into it to see what works and what not. Have a statement at least. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19003) Make sure all balancer actions respect decommissioned server
[ https://issues.apache.org/jira/browse/HBASE-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-19003. -- Resolution: Done > Make sure all balancer actions respect decommissioned server > > > Key: HBASE-19003 > URL: https://issues.apache.org/jira/browse/HBASE-19003 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0-beta-1 > > > There have been questions raised in HBASE-10367 and other related JIRAs. We > want to make sure all aspects of the balancer respect the draining flag. We > will have a good look, and fix if any violation is found. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19096) Add RowMutions batch support in AsyncTable
Jerry He created HBASE-19096: Summary: Add RowMutions batch support in AsyncTable Key: HBASE-19096 URL: https://issues.apache.org/jira/browse/HBASE-19096 Project: HBase Issue Type: Sub-task Reporter: Jerry He Assignee: Jerry He Batch support for RowMutations has been added in the Table interface, but is not in AsyncTable. This JIRA will add it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-17369) Add ACL to the new region server drain related API
[ https://issues.apache.org/jira/browse/HBASE-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-17369. -- Resolution: Implemented Implemented as part of HBASE-10367. > Add ACL to the new region server drain related API > -- > > Key: HBASE-17369 > URL: https://issues.apache.org/jira/browse/HBASE-17369 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Jerry He >Priority: Critical > > Add ACL to the new region server drain related API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
Jerry He created HBASE-19021:
Summary: Restore a few important missing logics for balancer in 2.0
Key: HBASE-19021
URL: https://issues.apache.org/jira/browse/HBASE-19021
Project: HBase
Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
After looking at the code, and some testing, I see the following things are
missing for balancer to work properly after AMv2.
# hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
Previous default is cluster wide, not by table.
# Servers with no assignments is not added for balance consideration.
# Crashed server is not removed from the in-memory server map in RegionStates,
which affects balance.
# Draining marker is not respected when balance.
Also try to re-enable {{TestRegionRebalancing}}, which has a
{{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19003) Make sure all balancer actions respect decommissioned server
Jerry He created HBASE-19003: Summary: Make sure all balancer actions respect decommissioned server Key: HBASE-19003 URL: https://issues.apache.org/jira/browse/HBASE-19003 Project: HBase Issue Type: Sub-task Reporter: Jerry He There have been questions raised in HBASE-10367 and other related JIRAs. We want to make sure all aspects of the balancer respect the draining flag. We will have a good look, and fix if any violation is found. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-16010) Put draining function through Admin API
[
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-16010.
--
Resolution: Fixed
> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch,
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to
> interact directly with Zookeeper draining node to add and remove draining
> servers.
> For example, in draining_servers.rb:
> {code}
> zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config,
> "draining_servers", nil)
> parentZnode = zkw.drainingZNode
> begin
> for server in servers
> node = ZKUtil.joinZNode(parentZnode, server)
> ZKUtil.createAndFailSilent(zkw, node)
> end
> ensure
> zkw.close()
> end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HBASE-11608) Add synchronous split
[ https://issues.apache.org/jira/browse/HBASE-11608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-11608. -- Resolution: Duplicate Finally, marked as dup. It has been added in HBASE-18229. > Add synchronous split > - > > Key: HBASE-11608 > URL: https://issues.apache.org/jira/browse/HBASE-11608 > Project: HBase > Issue Type: New Feature > Components: Admin >Affects Versions: 0.99.0, 0.98.5 >Reporter: Jerry He > > Users have been asking for this. We have an internal requirement for this as > well. > The goal is a provide a Admin API (and shell command) so that users can > request to split a region or table and get the split completion result > synchronously. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18740) Upgrade Zookeeper version in branch 1.4 and branch-1
Jerry He created HBASE-18740: Summary: Upgrade Zookeeper version in branch 1.4 and branch-1 Key: HBASE-18740 URL: https://issues.apache.org/jira/browse/HBASE-18740 Project: HBase Issue Type: Improvement Reporter: Jerry He Assignee: Jerry He Branch 1.4 and branch 1 are still on Zookeeper 3.4.6. Branch 2 and master branch have upgraded to 3.4.9. There are some important fixes we'd like to have. See the linked JIRAs. Another critical fix is ZOOKEEPER-2146, which can be explored maliciously. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-9417) SecureBulkLoadEndpoint should be folded in core
[ https://issues.apache.org/jira/browse/HBASE-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-9417. - Resolution: Duplicate Fix Version/s: (was: 2.0.0-alpha-3) > SecureBulkLoadEndpoint should be folded in core > --- > > Key: HBASE-9417 > URL: https://issues.apache.org/jira/browse/HBASE-9417 > Project: HBase > Issue Type: Bug > Components: regionserver, security >Reporter: Enis Soztutar >Priority: Critical > > In unsecure bulk loading, the client creates the files to be bulk loaded, and > asks the regionservers to do the operation. Bulk loading is performed by a > move, which would mean that the hbase user has to have WRITE permissions for > the bulk loaded files. If the client who has generated the files is different > than the hbase user, this creates an access denied exception if complete bulk > load is not run as the hbase user. > I think even for unsecure mode, we should mimic what SecureBulkLoadEndpoint > does, where hbase creates a staging directory and the client hands off the > files to that directory with global perms. > Update: Now that HBASE-12052 enables running SecureBulkLoadEndpoint even in > unsecure deployments, we should consider bringing SecureBulkLoad into core > HBase (meaning implement the functionality in RegionServer instead of in the > coprocessor). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18590) branch-1.4 needs a Jenkins commit build job
[ https://issues.apache.org/jira/browse/HBASE-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-18590. -- Resolution: Fixed > branch-1.4 needs a Jenkins commit build job > --- > > Key: HBASE-18590 > URL: https://issues.apache.org/jira/browse/HBASE-18590 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Ted Yu >Priority: Critical > > The current HBase-1.4 job is actually branch-1. > https://builds.apache.org/job/HBase-1.4/ > Need a separate job for branch-1.4. And rename the current job to HBase-1.5. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18590) branch-1.4 needs a Jenkins commit build job
Jerry He created HBASE-18590: Summary: branch-1.4 needs a Jenkins commit build job Key: HBASE-18590 URL: https://issues.apache.org/jira/browse/HBASE-18590 Project: HBase Issue Type: Bug Reporter: Jerry He Priority: Critical The current HBase-1.4 job is actually branch-1. https://builds.apache.org/job/HBase-1.4/ Need a separate job for branch-1.4. And rename the current job to HBase-1.5. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18589) branch-1.4 build compile is broken
Jerry He created HBASE-18589:
Summary: branch-1.4 build compile is broken
Key: HBASE-18589
URL: https://issues.apache.org/jira/browse/HBASE-18589
Project: HBase
Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure: Compilation
failure:
[ERROR]
/hbase-apache/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRegionSnapshotTask.java:[164,46]
error: cannot find symbol
[ERROR] symbol: class SnapshotDescription
[ERROR] location: class HBaseProtos
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18522) Add RowMutations support to Batch
Jerry He created HBASE-18522: Summary: Add RowMutations support to Batch Key: HBASE-18522 URL: https://issues.apache.org/jira/browse/HBASE-18522 Project: HBase Issue Type: Improvement Affects Versions: 1.2.6 Reporter: Jerry He Assignee: Jerry He Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 RowMutations is multiple Puts and/or Deletes atomically on a single row. Current Batch call does not support RowMutations as part of the batch. We should add this missing part. We should be able to batch RowMutations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18123) Hbase indexer
[ https://issues.apache.org/jira/browse/HBASE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-18123. -- Resolution: Invalid > Hbase indexer > - > > Key: HBASE-18123 > URL: https://issues.apache.org/jira/browse/HBASE-18123 > Project: HBase > Issue Type: Task >Affects Versions: 1.2.3 > Environment: Debian / Hadoop 2.6 / Solr 6.4.2 >Reporter: Fred > > Hi all, > I want to extract fields from PDF files and store it into Hbase. Then I want > to link the database with a Solr collection. To do this, I installed > Hbase-indexer. > What is the best way to do it ? > Actually, I can write data into Hbase but not into my Solr collection. When I > launch Hbase-indexer server, I get some errors : > Cannot connect to cluster at myIPaddress:2181/solr: cluster not found/not > ready > Somebody ti o help me ? Thanks in advance. > Fred -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17890) FuzzyRow tests fail if unaligned support is false
Jerry He created HBASE-17890:
Summary: FuzzyRow tests fail if unaligned support is false
Key: HBASE-17890
URL: https://issues.apache.org/jira/browse/HBASE-17890
Project: HBase
Issue Type: Bug
Affects Versions: 1.2.5, 2.0.0
Reporter: Jerry He
When unaligned support is false, FuzzyRow tests fail:
{noformat}
Failed tests:
TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186
expected:<10> but was:<0>
TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but
was:
TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343
expected:<6250> but was:<0>
TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445
expected:<5> but was:<0>
TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
{noformat}
This can be reproduced in the case described in HBASE-17869. Or on a platform
really without unaligned support.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
Jerry He created HBASE-17869: Summary: UnsafeAvailChecker wrongly returns false on ppc Key: HBASE-17869 URL: https://issues.apache.org/jira/browse/HBASE-17869 Project: HBase Issue Type: Bug Affects Versions: 1.2.4 Reporter: Jerry He Assignee: Jerry He Priority: Minor On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK bug. https://bugs.openjdk.java.net/browse/JDK-8165231 This causes some problem for HBase. i.e. FuzzyRowFilter test fails. Fix it by providing a hard-code workaround for the JDK bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reopened HBASE-14161: -- > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-14161. -- Resolution: Duplicate > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-14161. -- Resolution: Fixed > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-14167.
--
Resolution: Duplicate
Fix Version/s: 2.0.0
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14167.11.patch, HBASE-14167-master-v2.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix
[ https://issues.apache.org/jira/browse/HBASE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-17502. -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Pushed. Thanks for the review, Ted. > Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support > matrix > > > Key: HBASE-17502 > URL: https://issues.apache.org/jira/browse/HBASE-17502 > Project: HBase > Issue Type: Task >Reporter: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-17502.patch, HBASE-17502-v2.patch > > > Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin. > HADOOP-10786 fixed the problem in Hadoop 2.6.1. > Let's document it in the Hadoop support matrix. > This was brought up in HBase 1.3.0 RC0 voting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix
Jerry He created HBASE-17502: Summary: Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix Key: HBASE-17502 URL: https://issues.apache.org/jira/browse/HBASE-17502 Project: HBase Issue Type: Task Reporter: Jerry He Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin. HADOOP-10786 fixed the problem in Hadoop 2.6.1. Let's document it in the Hadoop support matrix. This was brought up in HBase 1.3.0 RC0 voting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17370) Fix or provide shell scripts to drain and decommission region server
Jerry He created HBASE-17370: Summary: Fix or provide shell scripts to drain and decommission region server Key: HBASE-17370 URL: https://issues.apache.org/jira/browse/HBASE-17370 Project: HBase Issue Type: Sub-task Reporter: Jerry He 1. Update the existing shell scripts to use the new drain related API. 2 Or provide new shell scripts. 3. Provide a 'decommission' shell tool that puts the server in drain mode and offload the server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17369) Add ACL to the new region server drain related API
Jerry He created HBASE-17369: Summary: Add ACL to the new region server drain related API Key: HBASE-17369 URL: https://issues.apache.org/jira/browse/HBASE-17369 Project: HBase Issue Type: Task Reporter: Jerry He Priority: Critical Add ACL to the new region server drain related API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17322) New API to get the list of draining region servers
[ https://issues.apache.org/jira/browse/HBASE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-17322. -- Resolution: Duplicate > New API to get the list of draining region servers > -- > > Key: HBASE-17322 > URL: https://issues.apache.org/jira/browse/HBASE-17322 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > > In various scenarios it would be useful to have a list of draining region > servers so as to avoid them while doing certain operations such as region > moving during batch rolling upgrades. > Jira to add a method getDrainingServers() in ClusterStatus so that this info > can be retrieved through HBaseAdmin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16796) Correct RpcServer responseSize and requestSize to account for cellScanner payload
[ https://issues.apache.org/jira/browse/HBASE-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-16796. -- Resolution: Duplicate > Correct RpcServer responseSize and requestSize to account for cellScanner > payload > - > > Key: HBASE-16796 > URL: https://issues.apache.org/jira/browse/HBASE-16796 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Jerry He > > The reponseSize and requestSize on the RpcServer don't seem to account for > the cellScanner payload. We should correct them. > The metrics numbers and the logging for reponseTooLarge and TooSlow won't > show up correctly otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17221) Abstract out an interface for RpcServer.Call
Jerry He created HBASE-17221:
Summary: Abstract out an interface for RpcServer.Call
Key: HBASE-17221
URL: https://issues.apache.org/jira/browse/HBASE-17221
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0
RpcServer.Call is a concrete class, but it is marked as:
{noformat}
@InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC,
HBaseInterfaceAudience.PHOENIX})
{noformat}
Let's abstract out an interface out of it for potential consumers that want to
pass it around.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16882) Expose the number of open scanners on region server via metics
Jerry He created HBASE-16882: Summary: Expose the number of open scanners on region server via metics Key: HBASE-16882 URL: https://issues.apache.org/jira/browse/HBASE-16882 Project: HBase Issue Type: Improvement Reporter: Jerry He Assignee: Jerry He Priority: Minor Metrics on the number of open scanners and their live span on the region server are useful in doing server performance diagnostics, and are useful in doing client application diagnostics, profiling as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16843) Manage and secure dynamic jar dir with client API
Jerry He created HBASE-16843: Summary: Manage and secure dynamic jar dir with client API Key: HBASE-16843 URL: https://issues.apache.org/jira/browse/HBASE-16843 Project: HBase Issue Type: Sub-task Reporter: Jerry He Assignee: Jerry He The dynamic jar dir is specified with the property 'hbase.dynamic.jars.dir' The permissions on this dir is a cause of confusion and concern. Let's secure this dir by folding it into root.dir, and provide client API to add, remove jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16822) Enable restore-snapshot and clone-snapshot to use external specified snapshot locatioin
Jerry He created HBASE-16822: Summary: Enable restore-snapshot and clone-snapshot to use external specified snapshot locatioin Key: HBASE-16822 URL: https://issues.apache.org/jira/browse/HBASE-16822 Project: HBase Issue Type: Improvement Reporter: Jerry He Currently restore-snapshot and clone-snapshot only work with the snapshots that are under hbase root.dir. In combination with export-snapshot, this means the snapshot needs to be exported out to another hbase root.dir, or back and forth eventually to a hbase root.dir. There are a few issues with the approach. We've know that export-snapshot has a limitation dealing with secure cluster, where the external user needs to have read access to hbase root.dir data, by-passing table ACL check. The second problem is when we try to use or bring back the exported snapshot for restore/clone. They have to in the target hbase root.dir, and needs write permission to get it in there. Again we will have permission problem. This ticket tries to deal with the second problem, clone and restore from a exported snapshot. The exported snapshots can be on the same cluster but the user may not have write permission to move it to hbase.root.dir. We should have a solution that allow clone/restore snapshot from a external path that keeps snapshot backups. And also do it with security permission in mind. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16798) Integrate MultiRowMutation CPEP into HBase core
Jerry He created HBASE-16798: Summary: Integrate MultiRowMutation CPEP into HBase core Key: HBASE-16798 URL: https://issues.apache.org/jira/browse/HBASE-16798 Project: HBase Issue Type: Sub-task Reporter: Jerry He Integrate MultiRowMutation CPEP into HBase core as discussed in the parent task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16797) Integrate TokenProvider to HBase core
Jerry He created HBASE-16797: Summary: Integrate TokenProvider to HBase core Key: HBASE-16797 URL: https://issues.apache.org/jira/browse/HBASE-16797 Project: HBase Issue Type: Sub-task Reporter: Jerry He Integrate TokenProvider to HBase core as discussed in the parent task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16796) Correct RpcServer responseSize and requestSize to account for cellScanner payload
Jerry He created HBASE-16796: Summary: Correct RpcServer responseSize and requestSize to account for cellScanner payload Key: HBASE-16796 URL: https://issues.apache.org/jira/browse/HBASE-16796 Project: HBase Issue Type: Bug Reporter: Jerry He Assignee: Jerry He The reponseSize and requestSize on the RpcServer don't seem to account for the cellScanner payload. We should correct them. The metrics numbers and the logging for reponseTooLarge and TooSlow won't show up correctly otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16732) Avoid possible NPE in MetaTableLocator
Jerry He created HBASE-16732:
Summary: Avoid possible NPE in MetaTableLocator
Key: HBASE-16732
URL: https://issues.apache.org/jira/browse/HBASE-16732
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
{noformat}
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
at
org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at
org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
at
com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
at
com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
{noformat}
It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16292) Can't start a local instance
[
https://issues.apache.org/jira/browse/HBASE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-16292.
--
Resolution: Duplicate
> Can't start a local instance
>
>
> Key: HBASE-16292
> URL: https://issues.apache.org/jira/browse/HBASE-16292
> Project: HBase
> Issue Type: Bug
>Reporter: Elliott Clark
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Currently master is completely broken.
> {code}
> ./bin/start-hbase.sh
> starting master, logging to
> /Users/elliott/Code/Public/hbase/bin/../logs/hbase-elliott-master-elliott-mbp.out
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m;
> support was removed in 8.0
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m;
> support was removed in 8.0
> {code}
> {code}
> 2016-07-27 10:36:20,992 ERROR [main] regionserver.SecureBulkLoadManager:
> Failed to create or set permission on staging directory
> /user/elliott/hbase-staging
> ExitCodeException exitCode=1: chmod: /user/elliott/hbase-staging: No such
> file or directory
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
> at org.apache.hadoop.util.Shell.run(Shell.java:456)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:728)
> at
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
> at
> org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.start(SecureBulkLoadManager.java:124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:624)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:403)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:221)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:156)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:226)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2417)
> 2016-07-27 10:36:20,994 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterchmod:
> /user/elliott/hbase-staging: No such file or directory
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:221)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:156)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:226)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2417)
> Caused by: java.lang.IllegalStateException: Failed to create or set
> permission on staging directory /user/elliott/hbase-staging
> at
> org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.start(SecureBulkLoadManager.java:141)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:624)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:403)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
[jira] [Created] (HBASE-16647) hbck should do the offline reference repair before online repair
Jerry He created HBASE-16647:
Summary: hbck should do the offline reference repair before online
repair
Key: HBASE-16647
URL: https://issues.apache.org/jira/browse/HBASE-16647
Project: HBase
Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He
{noformat}
hbck
-fixReferenceFiles Try to offline lingering reference store files
Metadata Repair shortcuts
-repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans
-fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps -fixReferenceFiles
-fixTableLocks -fixOrphanedTableZnodes
{noformat}
Bad reference files prevent the region from coming online.
If used in the shortcut combination, the reference files should be fixed before
other online fix.
I have seen repeated '-repair' did not work because bad reference files failed
regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16601) Expand Dynamic Configuration
Jerry He created HBASE-16601: Summary: Expand Dynamic Configuration Key: HBASE-16601 URL: https://issues.apache.org/jira/browse/HBASE-16601 Project: HBase Issue Type: Improvement Reporter: Jerry He Assignee: Jerry He HBase currently supports limited dynamic configuration (modify properties without restarting the servers). Only some compaction, split properties are supported. We should expand to support more and possibly improve and make it more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16599) Expand the use of dynamic jar dir
Jerry He created HBASE-16599: Summary: Expand the use of dynamic jar dir Key: HBASE-16599 URL: https://issues.apache.org/jira/browse/HBASE-16599 Project: HBase Issue Type: Improvement Reporter: Jerry He Dynamic jar dir is currently used for Custom Filer and Comparator only. Let's expand its use so that custom extensions (custom split policy, custom compaction policy, etc) can be more easily deployed. Useful on managed Cloud deployments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16598) Clean up zookeeper useMulti in HBase code
Jerry He created HBASE-16598: Summary: Clean up zookeeper useMulti in HBase code Key: HBASE-16598 URL: https://issues.apache.org/jira/browse/HBASE-16598 Project: HBase Issue Type: Improvement Reporter: Jerry He We have zookeeper useMulti default true since HBase 1.0.0 And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" The benefit of useMulti is obvious. Let's clean up the code to remove useMulti false case so that we don't have to continue with the hassle of maintaining two version of the code (e.g. in replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16581) Optimize Replication queue transfers after server fail over
[
https://issues.apache.org/jira/browse/HBASE-16581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-16581.
--
Resolution: Duplicate
> Optimize Replication queue transfers after server fail over
> ---
>
> Key: HBASE-16581
> URL: https://issues.apache.org/jira/browse/HBASE-16581
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>
> Currently if a region server fails, the replication queue of this server will
> be picked up by another region server. The problem is this queue can
> possibly be huge and contains queues from other multiple or cascading server
> failures.
> We had such a case in production. From zk_dump:
> {code}
> ...
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1467603735059:
> 18748267
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1471723778060:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1468258960080:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1468204958990:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1469701010649:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1470409989238:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1471838985073:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1467142915090:
> 57804890
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1472181000614:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1471464567365:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1469486466965:
>
> /hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1467787339841:
> 47812951
> ...
> {code}
> There were hundreds of wals hanging under this queue, coming from diferent
> region servers, which took a long time to replicate.
> We should have a better strategy which lets live region servers each grep
> part of this nested queue, and replicate in parallel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16582) Add Encryption keystore password for Hadoop credential store support
Jerry He created HBASE-16582:
Summary: Add Encryption keystore password for Hadoop credential
store support
Key: HBASE-16582
URL: https://issues.apache.org/jira/browse/HBASE-16582
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Currently the Encryption keystore password (or keyprovider password) does not
support Hadoop credential store.
{code}
hbase.crypto.keyprovider.parameters
jceks:///path/to/hbase/conf/hbase.jks?password=***
{code}
Let's refactor this parameter a little and add support for Hadoop credential
store
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16581) Optimize Replication queue transfers after server fail over
Jerry He created HBASE-16581:
Summary: Optimize Replication queue transfers after server fail
over
Key: HBASE-16581
URL: https://issues.apache.org/jira/browse/HBASE-16581
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Currently if a region server fails, the replication queue of this server will
be picked up by another region server. The problem is this queue can possibly
be huge and contains queues from other multiple or cascading server failures.
We had such a case in production. From zk_dump:
{code}
...
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1467603735059:
18748267
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1471723778060:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1468258960080:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1468204958990:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1469701010649:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1470409989238:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1471838985073:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1467142915090:
57804890
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1472181000614:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1471464567365:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1469486466965:
/hbase/replication/rs/r01data10-va-pub.xxx.ext,60020,1472680007498/1-r01data10-va-pub.xxx.ext,60020,1455993417125-r01data07-va-pub.xxx.ext,60020,1472680008225-r01data08-va-pub.xxx.ext,60020,1472680007318/r01data10-va-pub.xxx.ext%2C60020%2C1455993417125.1467787339841:
47812951
...
{code}
There were hundreds of wals hanging under this queue, coming from diferent
region servers, which took a long time to replicate.
We should have a better strategy which lets live region servers each grep part
of this nested queue, and replicate in parallel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16370) Exclude jdk tools.jar from Bytecode Version enforcer
Jerry He created HBASE-16370:
Summary: Exclude jdk tools.jar from Bytecode Version enforcer
Key: HBASE-16370
URL: https://issues.apache.org/jira/browse/HBASE-16370
Project: HBase
Issue Type: Improvement
Affects Versions: 1.2.2
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3
Getting this message trying to do a build with -Prelease:
{noformat}
[INFO] Restricted to JDK 1.7 yet jdk.tools:jdk.tools:jar:1.8:system contains
org/relaxng/datatype/DatatypeLibrary.class targeted to JDK 1.8
[WARNING] Rule 3: org.apache.maven.plugins.enforcer.EnforceBytecodeVersion
failed with message:
HBase has unsupported dependencies.
HBase requires that all dependencies be compiled with version 1.7 or earlier
of the JDK to properly build from source. You appear to be using a newer
dependency. You can use
either "mvn -version" or "mvn enforcer:display-info" to verify what version
is active.
Non-release builds can temporarily build with a newer JDK version by setting
the
'compileSource' property (eg. mvn -DcompileSource=1.8 clean package).
Found Banned Dependency: jdk.tools:jdk.tools:jar:1.8
Use 'mvn dependency:tree' to locate the source of the banned dependencies.
[INFO]
{noformat}
My JDK is 1.8. But I wanted to build to target 1.7. So I didn't' have the
-DcompileSource=1.8.
The enforcer checks the jdk tools.jar and causes the error because the system
JDK is 1.8.
This is a valid build/release use case as long as we support both 1.8 and 1.7.
We should exclude jdk tools.jar from the enforcer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16257) Move staging dir to be under hbase root dir
Jerry He created HBASE-16257:
Summary: Move staging dir to be under hbase root dir
Key: HBASE-16257
URL: https://issues.apache.org/jira/browse/HBASE-16257
Project: HBase
Issue Type: Sub-task
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.0.0
The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then defaults
to
{code}
public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
+ System.getProperty("user.name") + "/hbase-staging";
{code}
This default would have problem on local file system standalone case.
We can move the staging dir to be under hbase.rootdir. We are bringing secure
bulkload to the core. It makes sense to bring it under core control as well,
instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-15291) FileSystem not closed in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reopened HBASE-15291: -- > FileSystem not closed in secure bulkLoad > > > Key: HBASE-15291 > URL: https://issues.apache.org/jira/browse/HBASE-15291 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.2, 0.98.16.1 >Reporter: Yong Zhang >Assignee: Yong Zhang > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18, 1.4.0 > > Attachments: HBASE-15291.001.patch, HBASE-15291.002.patch, > HBASE-15291.003.patch, HBASE-15291.004.patch, HBASE-15291.addendum, patch > > > FileSystem not closed in secure bulkLoad after bulkLoad finish, it will > cause memory used more and more if too many bulkLoad . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16149) Log the underlying RPC exception in RpcRetryingCallerImpl
Jerry He created HBASE-16149:
Summary: Log the underlying RPC exception in RpcRetryingCallerImpl
Key: HBASE-16149
URL: https://issues.apache.org/jira/browse/HBASE-16149
Project: HBase
Issue Type: Improvement
Affects Versions: 1.2.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
In RpcRetryingCallerImpl:
{code}
public T callWithRetries(RetryingCallable callable, int callTimeout)
throws IOException, RuntimeException {
...
for (int tries = 0;; tries++) {
try {
...
return callable.call(getTimeout(callTimeout));
...
} catch (Throwable t) {
ExceptionUtil.rethrowIfInterrupt(t);
if (tries > startLogErrorsCnt) {
LOG.info("Call exception, tries=" + tries + ", maxAttempts=" +
maxAttempts + ", started="
+ (EnvironmentEdgeManager.currentTime() - tracker.getStartTime())
+ " ms ago, "
+ "cancelled=" + cancelled.get() + ", msg="
+ callable.getExceptionMessageAdditionalDetail());
}
...
{code}
We log the callable.getExceptionMessageAdditionalDetail() msg. But
callable.getExceptionMessageAdditionalDetail() may not provide the underlying
cause..
For example, in AbstractRegionServerCallable,
{code}
public String getExceptionMessageAdditionalDetail() {
return "row '" + Bytes.toString(row) + "' on table '" + tableName + "' at "
+ location;
}
{code}
Let's add the underlying exception cause to the message as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16010) Put draining function through Admin API
Jerry He created HBASE-16010:
Summary: Put draining function through Admin API
Key: HBASE-16010
URL: https://issues.apache.org/jira/browse/HBASE-16010
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Priority: Minor
Currently, there is no Amdin API for draining function. Client has to interact
directly with Zookeeper draining node to add and remove draining servers.
For example, in draining_servers.rb:
{code}
zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config,
"draining_servers", nil)
parentZnode = zkw.drainingZNode
begin
for server in servers
node = ZKUtil.joinZNode(parentZnode, server)
ZKUtil.createAndFailSilent(zkw, node)
end
ensure
zkw.close()
end
{code}
This is not good in cases like secure clusters with protected Zookeeper nodes.
Let's put draining function through Admin API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15834) Correct Bloom filter documentation in section 96.4 of Reference Guide
[
https://issues.apache.org/jira/browse/HBASE-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-15834.
--
Resolution: Fixed
Fix Version/s: 2.0.0
> Correct Bloom filter documentation in section 96.4 of Reference Guide
> -
>
> Key: HBASE-15834
> URL: https://issues.apache.org/jira/browse/HBASE-15834
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: li xiang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15834.patch.v0, HBASE-15834.patch.v1
>
>
> In section 96.4, the second paragraph from the bottom
> {code}
> Since HBase 0.96, row-based Bloom filters are enabled by default. (HBASE-)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15841) Performance Evaluation tool total rows may not be set correctly
Jerry He created HBASE-15841: Summary: Performance Evaluation tool total rows may not be set correctly Key: HBASE-15841 URL: https://issues.apache.org/jira/browse/HBASE-15841 Project: HBase Issue Type: Bug Reporter: Jerry He Priority: Minor Carried my comment on HBASE-15403 to here: Recently when I ran PerformanceEvaluation, I did notice some problem with the number of rows. hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable1 randomWrite 1 hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable5 randomWrite 5 hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable10 randomWrite 10 hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable10 randomWrite 20 All produced similar number of rows, and on the file system, they look like in similar size as well: hadoop fs -du -h /apps/hbase/data/data/default 786.5 M /apps/hbase/data/data/default/TestTable1 786.0 M /apps/hbase/data/data/default/TestTable10 782.0 M /apps/hbase/data/data/default/TestTable20 713.4 M /apps/hbase/data/data/default/TestTable5 HBase is 1.2.0. Looks like a regression somewhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15808) Reduce potential bulk load intermediate space usage and waste
Jerry He created HBASE-15808: Summary: Reduce potential bulk load intermediate space usage and waste Key: HBASE-15808 URL: https://issues.apache.org/jira/browse/HBASE-15808 Project: HBase Issue Type: Improvement Affects Versions: 1.2.0 Reporter: Jerry He Assignee: Jerry He Priority: Minor If the bulk load input files do not match the existing region boudaries, the files will be splitted. In the unfornate cases where the files need to be splitted multiple times, the process can consume unnecessary space and can even cause out of space. Here is over-simplified example. Orinal size of input files: consumed space: size --> 300GB After a round of splits: consumed space: size + tmpspace1 --> 300GB + 300GB After another round of splits: consumded space: size + tmpspace1 + tmpspace2 --> 300GB + 300GB + 300GB .. Currently we don't do any cleanup in the process. At least all the intermediate tmpspace (not the last one) can be deleted in the process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15592) Print Procedure WAL content
Jerry He created HBASE-15592: Summary: Print Procedure WAL content Key: HBASE-15592 URL: https://issues.apache.org/jira/browse/HBASE-15592 Project: HBase Issue Type: New Feature Reporter: Jerry He Let's have a printer to print the content of Procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15591) ServerCrashProcedure not yielding
Jerry He created HBASE-15591: Summary: ServerCrashProcedure not yielding Key: HBASE-15591 URL: https://issues.apache.org/jira/browse/HBASE-15591 Project: HBase Issue Type: Bug Affects Versions: 1.2.0, 2.0.0 Reporter: Jerry He ServerCrashProcedure is not propagating ProcedureYieldException to the ProcedureExecutor One symptom is that while ServerCrashProcedure is waiting for meta to be up the Procedure WALs get filled up rapidly with all the executions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reopened HBASE-15523: -- Could you provide a patch, [~easyliangjob]? > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: yi liang >Assignee: yi liang >Priority: Minor > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14963) Remove use of Guava Stopwatch from HBase client code
[ https://issues.apache.org/jira/browse/HBASE-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reopened HBASE-14963: -- > Remove use of Guava Stopwatch from HBase client code > > > Key: HBASE-14963 > URL: https://issues.apache.org/jira/browse/HBASE-14963 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Devaraj Das >Assignee: Devaraj Das > Labels: needs_releasenote > Fix For: 2.0.0 > > Attachments: HBASE-14963-branch-1.patch, no-stopwatch.txt > > > We ran into an issue where an application bundled its own Guava (and that > happened to be in the classpath first) and HBase's MetaTableLocator threw an > exception due to the fact that Stopwatch's constructor wasn't compatible... > Might be better to not depend on Stopwatch at all in MetaTableLocator since > the functionality is easily doable without. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15223) Make pubic convertScanToString for Spark
Jerry He created HBASE-15223: Summary: Make pubic convertScanToString for Spark Key: HBASE-15223 URL: https://issues.apache.org/jira/browse/HBASE-15223 Project: HBase Issue Type: Improvement Reporter: Jerry He One way to access HBase from Spark is to use newAPIHadoopRDD, which can take a TableInputFormat as class name. But we are not able to set a Scan object in there, for example to set a HBase filter. In MR, the public API TableMapReduceUtil.initTableMapperJob() or equivalent is used which can take a Scan object. But this call is not used in Spark conveniently. We need to make the TableMapReduceUtil.convertScanToString() public. So that a Scan object can be created, populated and then convert to the property and used by Spark. They are now package private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15201) Add hbase-spark to hbase assembly
Jerry He created HBASE-15201: Summary: Add hbase-spark to hbase assembly Key: HBASE-15201 URL: https://issues.apache.org/jira/browse/HBASE-15201 Project: HBase Issue Type: Improvement Reporter: Jerry He Priority: Minor hbase-spark currently is missing from hbase assembly. We should add it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14898) Correct Bloom filter documentation in the book
Jerry He created HBASE-14898: Summary: Correct Bloom filter documentation in the book Key: HBASE-14898 URL: https://issues.apache.org/jira/browse/HBASE-14898 Project: HBase Issue Type: Bug Reporter: Jerry He Priority: Minor In section 94.4. Bloom Filters: Since HBase 0.96, row-based Bloom filters are enabled by default. (HBASE-) --> in *HBASE-8450* In section 94.4.3. Configuring Server-Wide Behavior of Bloom Filters: io.hfile.bloom.enabled --> *io.storefile.bloom.enabled* Master switch to enable Bloom filters io.hfile.bloom.max.fold --> *io.storefile.bloom.max.fold* io.hfile.bloom.error.rate --> *io.storefile.bloom.error.rate* io.storefile.bloom.block.size --> *default is 128*1024 = 131072* These properties are probably not tuned usually, but should still be fixed in the doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14881) Provide a Put API that uses the provided row without coping
Jerry He created HBASE-14881: Summary: Provide a Put API that uses the provided row without coping Key: HBASE-14881 URL: https://issues.apache.org/jira/browse/HBASE-14881 Project: HBase Issue Type: Improvement Reporter: Jerry He The current available Put API always makes a copy of the rowkey. Let's provide an API that accepts an immutable byte array as rowkey without making a copy. There are cases where the caller of Put has created the immutable byte array (e.g from a serializer) and will not change it for the Put duration. We can avoid making a copy again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without coping
Jerry He created HBASE-14882:
Summary: Provide a Put API that adds the provided family,
qualifier, value without coping
Key: HBASE-14882
URL: https://issues.apache.org/jira/browse/HBASE-14882
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
In the Put API, we have addImmutable()
{code}
/**
* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
* that the underlying arrays won't change. It's intended
* for usage internal HBase to and for advanced client applications.
*/
public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
{code}
But in the implementation the row, family. qualifier and value are still being
copied locally to create kv.
Hopefully we should provide an API that truely uses immutable family, qualifier
and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14548) Expand how table coprocessor jar and dependency path can be specified
Jerry He created HBASE-14548: Summary: Expand how table coprocessor jar and dependency path can be specified Key: HBASE-14548 URL: https://issues.apache.org/jira/browse/HBASE-14548 Project: HBase Issue Type: Improvement Components: Coprocessors Reporter: Jerry He Currently you can specify the location of the coprocessor jar in the table coprocessor attribute. The problem is that it only allows you to specify one jar that implements the coprocessor. You will need to either bundle all the dependencies into this jar, or you will need to copy the dependencies into HBase lib dir. The first option may not be ideal sometimes. The second choice can be troublesome too, particularly when the hbase region sever node and dirs are dynamically added/created. There are a couple things we can expand here. We can allow the coprocessor attribute to specify a directory location, probably on hdfs. We may even allow some wildcard in there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale
[
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He reopened HBASE-14391:
--
> Empty regionserver WAL will never be deleted although the coresponding
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch,
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch,
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there
> are two directories for one host with different timestamp which indicates
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
> @Override
> public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
> for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
> for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
> }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
> throw new IOException("Unable to rename " + file.getPath() + " to "
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
> for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
> }
> }
> }
> LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
> }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
> }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will
> be not deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14275) Backport to 0.98 HBASE-10785 Metas own location should be cached
Jerry He created HBASE-14275: Summary: Backport to 0.98 HBASE-10785 Metas own location should be cached Key: HBASE-14275 URL: https://issues.apache.org/jira/browse/HBASE-14275 Project: HBase Issue Type: Improvement Reporter: Jerry He We've seen similar problem reported on 0.98. It is good improvement to have. This will cover HBASE-10785 and the a later HBASE-11332. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14228) Close BufferedMutator and connection in MultiTableOutputFormat
Jerry He created HBASE-14228: Summary: Close BufferedMutator and connection in MultiTableOutputFormat Key: HBASE-14228 URL: https://issues.apache.org/jira/browse/HBASE-14228 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 1.1.1 Reporter: Jerry He Assignee: Jerry He Priority: Minor Close BufferedMutator and connection in MultiTableRecordWriter of MultiTableOutputFormat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13845) Expire of one region server carrying meta can bring down the master
Jerry He created HBASE-13845:
Summary: Expire of one region server carrying meta can bring down
the master
Key: HBASE-13845
URL: https://issues.apache.org/jira/browse/HBASE-13845
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.1.0
Reporter: Jerry He
There seems to be a code bug that can cause expiration of one region server
carrying meta to bring down the master under certain case.
Here is the sequence of event.
a) The master detects the expiration of a region server on ZK, and starts to
expire the region server.
b) Since the failed region server carries meta, the shutdown handler will call
verifyAndAssignMetaWithRetries() during processing the expired rs.
c) In verifyAndAssignMeta(), there is a logic to verifyMetaRegionLocation
{code}
(!server.getMetaTableLocator().verifyMetaRegionLocation(server.getConnection(),
this.server.getZooKeeper(), timeout)) {
this.services.getAssignmentManager().assignMeta
(HRegionInfo.FIRST_META_REGIONINFO);
} else if
(serverName.equals(server.getMetaTableLocator().getMetaRegionLocation(
this.server.getZooKeeper( {
throw new IOException(hbase:meta is onlined on the dead server
+ serverName);
{code}
If we see the meta region is still alive on the expired rs, we throw an
exception.
We do some retries (default 10x1000ms) for verifyAndAssignMeta.
If we still get the exception after retries, we abort the master.
{code}
2015-05-27 06:58:30,156 FATAL [MASTER_META_SERVER_OPERATIONS-bdvs1163:6-0]
master.HMaster: Master server abort: loaded coprocessors are: []
2015-05-27 06:58:30,156 FATAL [MASTER_META_SERVER_OPERATIONS-bdvs1163:6-0]
master.HMaster: verifyAndAssignMeta failed after10 times retries, aborting
java.io.IOException: hbase:meta is onlined on the dead server
bdvs1164.svl.ibm.com,16020,1432681743203
at
org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:162)
at
org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:184)
at
org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:93)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-05-27 06:58:30,156 INFO [MASTER_META_SERVER_OPERATIONS-bdvs1163:6-0]
regionserver.HRegionServer: STOPPED: verifyAndAssignMeta failed after10 times
retries, aborting
{code}
The problem happens when the expired is slow processing its own expiration or
has a slow death, and is still able to respond to master's meta verification in
the meantime
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
Jerry He created HBASE-13706: Summary: CoprocessorClassLoader should not exempt Hive classes Key: HBASE-13706 URL: https://issues.apache.org/jira/browse/HBASE-13706 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.12, 1.0.1, 2.0.0, 1.1.0 Reporter: Jerry He Priority: Minor CoprocessorClassLoader is used to load classes from the coprocessor jar. Certain classes are exempt from being loaded by this ClassLoader, which means they will be ignored in the coprocessor jar, but loaded from parent classpath instead. One problem is that we categorically exempt org.apache.hadoop. But it happens that Hive packages start with org.apache.hadoop. There is no reason to exclude hive classes from theCoprocessorClassLoader. HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13701) Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load
Jerry He created HBASE-13701: Summary: Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load Key: HBASE-13701 URL: https://issues.apache.org/jira/browse/HBASE-13701 Project: HBase Issue Type: Improvement Reporter: Jerry He HBASE-12052 makes SecureBulkLoadEndpoint work in a non-secure env to solve HDFS permission issues. We have encountered some of the permission issues and have to use this SecureBulkLoadEndpoint to workaround issues. We should probably consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load since it is able to handle both secure Kerberos and non-secure cases. Maintaining two versions of bulk load implementation is also a cause of confusion, and having to explicitly set it is also inconvenient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-13217) Procedure fails due to ZK issue
[
https://issues.apache.org/jira/browse/HBASE-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He reopened HBASE-13217:
--
Procedure fails due to ZK issue
---
Key: HBASE-13217
URL: https://issues.apache.org/jira/browse/HBASE-13217
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12
Reporter: ramkrishna.s.vasudevan
Assignee: Jerry He
Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13217-0.98-v2.patch, HBASE-13217-v2.patch,
HBASE-13217.patch
When ever I try to flush explicitly in the trunk code the flush procedure
fails due to ZK issue
{code}
ERROR: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable
via
stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
at
org.apache.hadoop.hbase.procedure.Procedure.isCompleted(Procedure.java:368)
at
org.apache.hadoop.hbase.procedure.flush.MasterFlushTableProcedureManager.isProcedureDone(MasterFlushTableProcedureManager.java:196)
at
org.apache.hadoop.hbase.master.MasterRpcServices.isProcedureDone(MasterRpcServices.java:905)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:47019)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2073)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
at
org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
{code}
Once this occurs, even on restart of the RS the RS becomes unusable. I have
verified that the ZK remains intact and there is no problem with it. a bit
older version of trunk ( 3months) does not have this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-13217) Procedure fails due to ZK issue
[
https://issues.apache.org/jira/browse/HBASE-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-13217.
--
Resolution: Fixed
Procedure fails due to ZK issue
---
Key: HBASE-13217
URL: https://issues.apache.org/jira/browse/HBASE-13217
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12
Reporter: ramkrishna.s.vasudevan
Assignee: Jerry He
Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13217-0.98-v2.patch, HBASE-13217-v2.patch,
HBASE-13217.patch
When ever I try to flush explicitly in the trunk code the flush procedure
fails due to ZK issue
{code}
ERROR: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable
via
stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
at
org.apache.hadoop.hbase.procedure.Procedure.isCompleted(Procedure.java:368)
at
org.apache.hadoop.hbase.procedure.flush.MasterFlushTableProcedureManager.isProcedureDone(MasterFlushTableProcedureManager.java:196)
at
org.apache.hadoop.hbase.master.MasterRpcServices.isProcedureDone(MasterRpcServices.java:905)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:47019)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2073)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
at
org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
at
org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
{code}
Once this occurs, even on restart of the RS the RS becomes unusable. I have
verified that the ZK remains intact and there is no problem with it. a bit
older version of trunk ( 3months) does not have this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-13598) Make hbase assembly 'attach' to the project
[
https://issues.apache.org/jira/browse/HBASE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He resolved HBASE-13598.
--
Resolution: Fixed
Committed the doc-addendum.
Make hbase assembly 'attach' to the project
---
Key: HBASE-13598
URL: https://issues.apache.org/jira/browse/HBASE-13598
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13598-master.patch, doc-addendum.patch
Currently for hbase assembly, we set 'attach' to 'false':
hbase-assembly/pom.xml:
{code}
!--We do not want assembly attached; run on command-line explicitly
- if you want to do an assembly--
- attachfalse/attach
{code}
The result is that the hbase assembly tarball will not be deployed via 'mvn
install' or 'maven deploy'
There are Apache projects that directly uses the hbase assembly tarball in
their build process. For example, Slider HBase package and Ambari 2.0
Metrics.
Here is the link the maven assembly plug info on the 'attach':
https://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html#attach
{code}
attach:
Controls whether the assembly plugin tries to attach the resulting assembly
to the project.
Type: boolean
Since: 2.2-beta-1
Required: No
User Property: assembly.attach
Default: true
{code}
The assembly will only be built if 'assembly:single' is specified, and then
deployed in 'maven install' or 'maven deploy'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13598) Make hbase assembly 'attach' to the project
Jerry He created HBASE-13598:
Summary: Make hbase assembly 'attach' to the project
Key: HBASE-13598
URL: https://issues.apache.org/jira/browse/HBASE-13598
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Priority: Minor
Currently for hbase assembly, we set 'attach' to 'false':
hbase-assembly/pom.xml:
{code}
!--We do not want assembly attached; run on command-line explicitly
- if you want to do an assembly--
- attachfalse/attach
{code}
The result is that the hbase assembly tarball will not be deployed via 'mvn
install' or 'maven deploy'
There are Apache projects that directly uses the hbase assembly tarball in
their build process. For example, Slider HBase package and Ambari 2.0 Metrics.
Here is the link the maven assembly plug info on the 'attach':
https://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html#attach
{code}
attach:
Controls whether the assembly plugin tries to attach the resulting assembly to
the project.
Type: boolean
Since: 2.2-beta-1
Required: No
User Property: assembly.attach
Default: true
{code}
The assembly will only be built if 'assembly:single' is specified, and then
deployed in 'maven install' or 'maven deploy'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13526) TestRegionServerReportForDuty can be flaky: hang or timeout
Jerry He created HBASE-13526: Summary: TestRegionServerReportForDuty can be flaky: hang or timeout Key: HBASE-13526 URL: https://issues.apache.org/jira/browse/HBASE-13526 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.12, 2.0.0, 1.1.0 Reporter: Jerry He Assignee: Jerry He Priority: Minor This test case is from HBASE-13317. The test uses a custom region server to simulate reportForDuty in a master failover case. This custom RS would start, then the primary master would fail, then the custom RS would reportForDuty to the second master after master failover. The test occasionally will hang or timeout. The root cause is that during first master initialization, the master would assign meta (and create and assign namespace table). It is possible that the meta is assigned to the custom RS, which has started (place a rs node on the ZK), but will not really check-in and be online. Then the master will go thru multiple re-assignment, which can be lengthy and cause trouble. There are a couple of issues I see in the master assignment code: 1. Master puts all the region servers obtained from ZK rs node into the online server list, including those that have not checked-in via RPC. And we will assign meta or other regions based on whole list. 2. When one assign plan fails, we don't exclude the failed server when picking the next destination, which may prolong the assignment process. I will provide a patch to fix the test case. The other issues mentioned are up to discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13345) Fix LocalHBaseCluster so that different region server impl can be used for different slaves
Jerry He created HBASE-13345: Summary: Fix LocalHBaseCluster so that different region server impl can be used for different slaves Key: HBASE-13345 URL: https://issues.apache.org/jira/browse/HBASE-13345 Project: HBase Issue Type: Improvement Reporter: Jerry He Assignee: Jerry He Priority: Minor LocalHBaseCluster and MiniHBaseCluster allow plugging in custom region server implementations. This JIRA will fix a loophole so that different implementations can be used for different slaves. For example, MyRegionServer1 for slave1, MyRegionServer1 for slave2. Or MyRegionServer1 for slave1, All other slaves use the default. This will help targeted testing using MiniHBaseCluster. I am working on a unit test for HBASE-13317. This fix will be useful.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13317) Region server reportForDuty stuck looping if there is a master change
Jerry He created HBASE-13317:
Summary: Region server reportForDuty stuck looping if there is a
master change
Key: HBASE-13317
URL: https://issues.apache.org/jira/browse/HBASE-13317
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.12, 1.0.0, 2.0.0
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0, 1.0.1, 0.98.13
During cluster startup, region server reportForDuty gets stuck looping if there
is a master change.
{noformat}
2015-03-22 11:15:16,186 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf274,6,1427045883965 with port=60020,
startcode=1427048115174
2015-03-22 11:15:16,272 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:16,274 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:19,274 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:19,275 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:19,276 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:22,276 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:22,296 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:22,296 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:25,296 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:25,299 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:25,299 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:28,299 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:28,302 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:28,302 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
{noformat}
What happended is the region server first got
master=bigaperf274,6,1427045883965. Before it was able to report
successfully, the maser changed to bigaperf273,6,1427048108439.
We were supposed to open a new connection to the new master. But we never did,
looping and trying to old address forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13251) Correct 'HBase, MapReduce, and the CLASSPATH' section in HBase Ref Guide
Jerry He created HBASE-13251: Summary: Correct 'HBase, MapReduce, and the CLASSPATH' section in HBase Ref Guide Key: HBASE-13251 URL: https://issues.apache.org/jira/browse/HBASE-13251 Project: HBase Issue Type: Improvement Components: documentation Reporter: Jerry He As [~busbey] pointed out in HBASE-13149, we have a section HBase, MapReduce, and the CLASSPATH in the HBase Ref Guide. http://hbase.apache.org/book.html#hbase.mapreduce.classpath There are duplication, errors and misinformation in the section. Need to cleanup and polish it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13149) HBase server MR tools are broken on Hadoop 2.5+ Yarn
Jerry He created HBASE-13149:
Summary: HBase server MR tools are broken on Hadoop 2.5+ Yarn
Key: HBASE-13149
URL: https://issues.apache.org/jira/browse/HBASE-13149
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.10.1, 1.0.0, 2.0.0
Reporter: Jerry He
Running on the server MR tools is not working on Yarn version 2.5+.
Running org.apache.hadoop.hbase.mapreduce.Export:
{noformat}
Exception in thread main java.lang.NoSuchMethodError:
org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
at
org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
at
org.apache.hadoop.yarn.util.timeline.TimelineUtils.clinit(TimelineUtils.java:47)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.init(ResourceMgrDelegate.java:96)
at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
at
org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
{noformat}
The problem seems to be the jackson jar version. HADOOP-10104 updated jackson
version to 1.9.13. YARN-2092 reported a problem as well.
HBase is using jackson 1.8.8. This version of the jar in the classpath seem to
cause the problem.
Should we upgrade to jackson 1.9.13?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13085) Security issue in the implementatoin of Rest gataway 'doAs' proxy user support
Jerry He created HBASE-13085:
Summary: Security issue in the implementatoin of Rest gataway
'doAs' proxy user support
Key: HBASE-13085
URL: https://issues.apache.org/jira/browse/HBASE-13085
Project: HBase
Issue Type: Bug
Components: REST, security
Affects Versions: 0.98.10, 1.0.0, 2.0.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
When 'hbase.rest.support.proxyuser' is turned on, HBase Rest gateway support
'doAs' proxy user from the Rest client.
The current implementation checks to see if the 'rest server user' is
authorized to impersonate the 'doAs' user (the user in the 'doAs' Rest query
string).
{code}
if (doAsUserFromQuery != null) {
Configuration conf = servlet.getConfiguration();
if (!servlet.supportsProxyuser()) {
throw new ServletException(Support for proxyuser is not configured);
}
UserGroupInformation ugi = servlet.getRealUser();
// create and attempt to authorize a proxy user (the client is attempting
// to do proxy user)
ugi = UserGroupInformation.createProxyUser(doAsUserFromQuery, ugi);
// validate the proxy user authorization
try {
ProxyUsers.authorize(ugi, request.getRemoteAddr(), conf);
} catch(AuthorizationException e) {
throw new ServletException(e.getMessage());
}
servlet.setEffectiveUser(doAsUserFromQuery);
}
{code}
The current implementation allows anyone from the rest client side to
impersonate another user by 'doAs'.
For example, potentially, 'user1' can 'doAs=admin'
The correct implementation should check to see if the rest client user is
authorized to do impersonation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13053) Add support of Visibility Labels in PerformanceEvaluation
Jerry He created HBASE-13053:
Summary: Add support of Visibility Labels in PerformanceEvaluation
Key: HBASE-13053
URL: https://issues.apache.org/jira/browse/HBASE-13053
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.10.1, 1.0.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.0.0, 1.0.1, 0.98.11
Add support of Visibility Labels in PerformanceEvaluation:
During write operations, support adding a visibility expression to KVs.
During read/scan operations, support using visibility authorization.
Here is the usage:
{noformat}
Options:
...
visibilityExp Writes the visibility expression along with KVs. Use for write
commands. Visiblity labels need to pre-exist.
visibilityAuth Specify the visibility auths (comma separated labels) used in
read or scan. Visiblity labels need to pre-exist.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13006) Document visibility label support for groups
Jerry He created HBASE-13006: Summary: Document visibility label support for groups Key: HBASE-13006 URL: https://issues.apache.org/jira/browse/HBASE-13006 Project: HBase Issue Type: Task Reporter: Jerry He Priority: Minor Fix For: 2.0.0 This is to document the changes added from HBASE-12745. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
Jerry He created HBASE-12949:
Summary: Scanner can be stuck in infinite loop if the HFile is
corrupted
Key: HBASE-12949
URL: https://issues.apache.org/jira/browse/HBASE-12949
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.10
Reporter: Jerry He
We've encountered problem where compaction hangs and never completes.
After looking into it further, we found that the compaction scanner was stuck
in a infinite loop. See stack below.
{noformat}
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
{noformat}
We identified the hfile that seems to be corrupted. Using HFile tool shows the
following:
{noformat}
[biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k -m
-f
/user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
org.apache.hadoop.util.PureJavaCrc32
15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
org.apache.hadoop.util.PureJavaCrc32C
15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
deprecated. Instead, use fs.defaultFS
Scanning -
/user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
WARNING, previous row is greater then current row
filename -
/user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
previous -
\x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
current -
Exception in thread main java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:489)
at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
{noformat}
Turning on Java Assert shows the following:
{noformat}
Exception in thread main java.lang.AssertionError: Key
20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
at
org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
{noformat}
It shows that the hfile seems to be corrupted -- the keys don't seem to be
right.
But Scanner is not able to give a meaningful error, but stuck in an infinite
loop in here:
{code}
KeyValueHeap.generalizedSeek()
while ((scanner = heap.poll()) != null) {
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12823) Visibility label security at limited localized level
Jerry He created HBASE-12823: Summary: Visibility label security at limited localized level Key: HBASE-12823 URL: https://issues.apache.org/jira/browse/HBASE-12823 Project: HBase Issue Type: Improvement Components: security Affects Versions: 1.0.0, 2.0.0, 0.98.10 Reporter: Jerry He Fix For: 2.0.0 Currently, if visibility label security is enabled for a HBase instance, after VisibilityController is configured, the cell level visibility label filtering will kick in across the HBase instance. Cell level visibility label filtering has non-negligible performance impact. On the other hand, in many use cases, only a small portion of the overall data needs visibility label protection. If we can support visibility label security at a limited and localized level, we will broaden the use cases and the adoption of this feature. We should be able to support visibility label security at per table or per column family level. This is quite common in many other HBase features. Cell level visibility label filtering will only be enabled and kick in for the tables or column families that the user designates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-8310) HBase snapshot timeout default values and TableLockManger timeout
[ https://issues.apache.org/jira/browse/HBASE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-8310. - Resolution: Won't Fix Clean up JIRAs. Close this a Won't Fix. HBase snapshot timeout default values and TableLockManger timeout - Key: HBASE-8310 URL: https://issues.apache.org/jira/browse/HBASE-8310 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.95.0 Reporter: Jerry He Assignee: Jerry He Priority: Minor Attachments: trunk.patch There are a few timeout values and defaults being used by HBase snapshot. DEFAULT_MAX_WAIT_TIME (6 milli sec, 1 min) for client response TIMEOUT_MILLIS_DEFAULT (6 milli sec, 1 min) for Procedure timeout SNAPSHOT_TIMEOUT_MILLIS_DEFAULT (6 milli sec, 1 min) for region server subprocedure There is also other timeout involved, for example, DEFAULT_TABLE_WRITE_LOCK_TIMEOUT_MS (10 mins) for TakeSnapshotHandler#prepare() We could have this case: The user issues a sync snapshot request, waits for 1 min, and gets an exception. In the meantime the snapshot handler is blocked on the table lock, and the snapshot may continue to finish after 10 mins. But the user will probably re-issue the snapshot request during the 10 mins. This is a little confusing and messy when this happens. To be more reasonable, we should either increase the DEFAULT_MAX_WAIT_TIME or decrease the table lock waiting time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12744) hbase-default.xml lists hbase.regionserver.global.memstore.size twice
Jerry He created HBASE-12744: Summary: hbase-default.xml lists hbase.regionserver.global.memstore.size twice Key: HBASE-12744 URL: https://issues.apache.org/jira/browse/HBASE-12744 Project: HBase Issue Type: Bug Affects Versions: 1.0.0 Reporter: Jerry He Assignee: Jerry He Priority: Minor Fix For: 1.0.0 This seems to be only a problem is branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12745) Visibility Labels: Support user groups visibility labels.
Jerry He created HBASE-12745:
Summary: Visibility Labels: Support user groups visibility labels.
Key: HBASE-12745
URL: https://issues.apache.org/jira/browse/HBASE-12745
Project: HBase
Issue Type: Improvement
Components: security
Affects Versions: 0.99.2, 0.98.9, 1.0.0
Reporter: Jerry He
Assignee: Jerry He
The thinking is that we should support visibility labels to be associated with
user groups.
We will then be able grant visibility labels to a group in addition to
individual users, which provides convenience and usability.
We will use '@group' to denote a group name, as similarly done in
AcccessController.
For example,
{code}
set_auths '@group1', ['SECRET','PRIVATE']
{code}
{code}
get_auth '@group1'
{code}
A user belonging to 'group1' will have all the visibility labels granted to
'group1'
We'll also support super user groups as specified in hbase-site.xml.
The code update will mainly be on the server side VisibilityLabelService
implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12682) compaction thread throttle value is not correct in hbase-default.xml
Jerry He created HBASE-12682:
Summary: compaction thread throttle value is not correct in
hbase-default.xml
Key: HBASE-12682
URL: https://issues.apache.org/jira/browse/HBASE-12682
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0
While doing some compaction tuning and looking up some of the parameters, I
noticed that hbase.regionserver.thread.compaction.throttle default value in the
documentation and in hbase-site.xml seems to be off the mark.
{code}
property
namehbase.regionserver.thread.compaction.throttle/name
value2560/value
descriptionThere are two different thread pools for compactions, one for
large compactions and
the other for small compactions. This helps to keep compaction of lean
tables (such as
systemitemhbase:meta/systemitem) fast. If a compaction is larger
than this threshold, it
goes into the large compaction pool. In most cases, the default value is
appropriate. Default:
2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size
(which defaults to 128).
The value field assumes that the value of
hbase.hregion.memstore.flush.size is unchanged from
the default./description
/property
{code}
It should be in bytes. Or am I miss anyting?
It is not a problem in 0.98 since the property is not in hbase-site.xml over
there. It is obtained dynamically:
{code}
throttlePoint = conf.getLong(hbase.regionserver.thread.compaction.throttle,
2 * maxFilesToCompact * storeConfigInfo.getMemstoreFlushSize());
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12644) Visibility Labels: issue with storing super users in labels table
Jerry He created HBASE-12644: Summary: Visibility Labels: issue with storing super users in labels table Key: HBASE-12644 URL: https://issues.apache.org/jira/browse/HBASE-12644 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.99.2, 0.98.8 Reporter: Jerry He Fix For: 1.0.0, 0.98.9 Super users have all the permissions for ACL and Visibility labels. They are defined in hbase-site.xml. Currently in VisibilityController, we persist super user with their system permission in hbase:labels. This make change in super user difficult. There are two issues: In the current DefaultVisibilityLabelServiceImpl.addSystemLabel, we only add super user when we initially create the 'system' label. No additional update after that even if super user changed. See code for details. Additionally, there is no mechanism to remove any super user from the labels table. We probably should not persist super users in the labels table. They are in hbase-site.xml and can just stay in labelsCache and used from labelsCache after retrieval by Visibility Controller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12441) Export and CopyTable need to be able to keep tags/labels in cells
Jerry He created HBASE-12441: Summary: Export and CopyTable need to be able to keep tags/labels in cells Key: HBASE-12441 URL: https://issues.apache.org/jira/browse/HBASE-12441 Project: HBase Issue Type: Bug Components: mapreduce, security Reporter: Jerry He Export and CopyTable (and possibly other MR tools) currently do not carry over tags/labels in cells. These tools should be able keep tags/labels in cells when they back up the table cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12373) Provide a command to list visibility labels
Jerry He created HBASE-12373: Summary: Provide a command to list visibility labels Key: HBASE-12373 URL: https://issues.apache.org/jira/browse/HBASE-12373 Project: HBase Issue Type: Improvement Affects Versions: 0.99.1, 0.98.7 Reporter: Jerry He Priority: Minor A command to list visibility labels that are in place would be handy. This is also in line with many of the other hbase list commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12346) Scan's default auths behavior under Visibility labels
Jerry He created HBASE-12346:
Summary: Scan's default auths behavior under Visibility labels
Key: HBASE-12346
URL: https://issues.apache.org/jira/browse/HBASE-12346
Project: HBase
Issue Type: Bug
Components: API, security
Affects Versions: 0.99.1, 0.98.7
Reporter: Jerry He
In Visibility Labels security, a set of labels (auths) are administered and
associated with a user.
A user can normally only see cell data during scan that are part of the user's
label set (auths).
Scan uses setAuthorizations to indicates its wants to use the auths to access
the cells.
Similarly in the shell:
{code}
scan 'table1', AUTHORIZATIONS = ['private']
{code}
But it is a surprise to find that setAuthorizations seems to be 'mandatory' in
the default visibility label security setting. Every scan needs to
setAuthorizations before the scan can get any cells even the cells are under
the labels the request user is part of.
The following steps will illustrate the issue:
Run as superuser.
{code}
1. create a visibility label called 'private'
2. create 'table1'
3. put into 'table1' data and label the data as 'private'
4. set_auths 'user1', 'private'
5. grant 'user1', 'RW', 'table1'
{code}
Run as 'user1':
{code}
1. scan 'table1'
This show no cells.
2. scan 'table1', scan 'table1', AUTHORIZATIONS = ['private']
This will show all the data.
{code}
I am not sure if this is expected by design or a bug.
But a more reasonable, more client application backward compatible, and less
surprising default behavior should probably look like this:
A scan's default auths, if its Authorizations attributes is not set explicitly,
should be all the auths the request user is administered and allowed on the
server.
If scan.setAuthorizations is used, then the server further filter the auths
during scan: use the input auths minus what is not in user's label set on the
server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12301) user_permission command does not show global permissions
Jerry He created HBASE-12301:
Summary: user_permission command does not show global permissions
Key: HBASE-12301
URL: https://issues.apache.org/jira/browse/HBASE-12301
Project: HBase
Issue Type: Bug
Components: security, shell
Affects Versions: 0.98.4, 2.0.0
Reporter: Jerry He
It seems that since 0,98 or later, the shell command does not show global
permission anymore, even requested by user with the right privilege.
{code}
hbase(main):004:0 user_permission
UserTable,Family,Qualifier:Permission
hbase default,table1,,: [Permission:
actions=READ,WRITE,EXEC,CREATE,ADMIN]
user2 default,table1,,: [Permission:
actions=READ,WRITE]
hbase default,table2,,: [Permission:
actions=READ,WRITE,EXEC,CREATE,ADMIN]
user2 default,table2,,: [Permission:
actions=READ,WRITE]
{code}
I recall in the older versions, global permissions were shown as permissions on
the hbase:acl table.
Anyway we need a way to show the global permissions as part of user_permission
request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12302) VisibilityClient getAuths does not propagate remote service exception correctly
Jerry He created HBASE-12302:
Summary: VisibilityClient getAuths does not propagate remote
service exception correctly
Key: HBASE-12302
URL: https://issues.apache.org/jira/browse/HBASE-12302
Project: HBase
Issue Type: Bug
Components: Client, security
Affects Versions: 0.98.7, 2.0.0
Reporter: Jerry He
Priority: Minor
Fix For: 2.0.0, 0.98.8
From hbase shell, run 'get_auths' with a non-superuser:
{code}
hbase(main):002:0 get_auths 'user2'
ERROR:
Here is some help for this command:
Get the visibility labels set for a particular user
Syntax : get_auths 'user1'
For example:
hbase get_auths 'user1'
{code}
We should expect a AccessDeniedException from the server.
From a Java client, AccessDeniedException was dumped out, but the end
exception is
{code}
java.util.NoSuchElementException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1124)
at java.util.TreeMap$ValueIterator.next(TreeMap.java:1171)
at
org.apache.hadoop.hbase.security.visibility.VisibilityClient.getAuths(VisibilityClient.java:148)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12168) Document Rest gateway SPNEGO-based authentication for client
Jerry He created HBASE-12168: Summary: Document Rest gateway SPNEGO-based authentication for client Key: HBASE-12168 URL: https://issues.apache.org/jira/browse/HBASE-12168 Project: HBase Issue Type: Task Components: documentation, REST, security Reporter: Jerry He Fix For: 0.98.8, 0.99.1 After HBASE-5050, we seem to support SPNEGO-based authentication from client on Rest gateway. But I had a tough time finding the info. The support is not mentioned in Security book. In the security book, we still have: bq. It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPEGNO HTTP authentication. This is future work. The release note in HBASE-5050 seems to be obsolete as well. e.g. hbase.rest.kerberos.spnego.principal seems to be obsolete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11882) Row level consistency may not be maintained with bulk load and compaction
Jerry He created HBASE-11882: Summary: Row level consistency may not be maintained with bulk load and compaction Key: HBASE-11882 URL: https://issues.apache.org/jira/browse/HBASE-11882 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 1.0.0, 2.0.0 Reporter: Jerry He Priority: Critical Fix For: 1.0.0, 2.0.0 While looking into the TestHRegionServerBulkLoad failure for HBASE-11772, I found the root cause is that row level atomicity may not be maintained with bulk load together with compation. TestHRegionServerBulkLoad is used to test bulk load atomicity. The test uses multiple threads to do bulk load and scan continuously and do compactions periodically. It verifies row level data is always consistent across column families. After HBASE-11591, we added readpoint checks for bulkloaded data using the seqId at the time of bulk load. Now a scanner will not see the data from a bulk load if the scanner's readpoint is earlier than the bulk load seqId. Previously, the atomic bulk load result is visible immediately to all scanners. The problem is with compaction after bulk load. Compaction does not lock the region and it is done one store (column family) at a time. It also compact away the seqId marker of bulk load. Here is an event sequence where the row level consistency is broken. 1. A scanner is started to scan a region with cf1 and cf2. The readpoint is 10. 2. There is a bulk load that loads into cf1 and cf2. The bulk load seqId is 11. Bulk load is guarded by region write lock. So it is atomic. 3. There is a compaction that compacts cf1. It compacts away the seqId marker of the bulk load. 4. The scanner tries to next to row-1001. It gets the bulk load data for cf1 since there is no seqId preventing it. It does not get the bulk load data for cf2 since the scanner's readpoint (10) is less than the bulk load seqId (11). Now we row level consistency is broken in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
