[jira] [Updated] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-11393: -- Attachment: HBASE-11393_v5.patch > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, > HBASE-11393_v5.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14727) Add block cache stats for regions
[
https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984275#comment-14984275
]
Hadoop QA commented on HBASE-14727:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769971/HBASE-14727-v2.patch
against master branch at commit 6cc6d833f2cde47123876e5ca107b522004055ac.
ATTACHMENT ID: 12769971
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1745 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+result = result && (Float.floatToIntBits(getCacheHitRatio())==
Float.floatToIntBits(other.getCacheHitRatio()));
+ result = result && (hasCacheHitRatioLatestNPeriods() ==
other.hasCacheHitRatioLatestNPeriods());
+ new java.lang.String[] { "RegionSpecifier", "Stores",
"Storefiles", "StoreUncompressedSizeMB", "StorefileSizeMB", "MemstoreSizeMB",
"StorefileIndexSizeMB", "ReadRequestsCount", "WriteRequestsCount",
"TotalCompactingKVs", "CurrentCompactedKVs", "RootIndexSizeKB",
"TotalStaticIndexSizeKB", "TotalStaticBloomSizeKB", "CompleteSequenceId",
"DataLocality", "LastMajorCompactionTs", "StoreCompleteSequenceId",
"CacheHitCount", "CacheMissCount", "CacheEvictedBlockCount", "CacheSize",
"CacheBlockCount", "CacheHitRatio", "CacheHitRatioLatestNPeriods", });
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16335//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16335//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16335//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16335//console
This message is automatically generated.
> Add block cache stats for regions
> -
>
> Key: HBASE-14727
> URL: https://issues.apache.org/jira/browse/HBASE-14727
> Project: HBase
> Issue Type: New Feature
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch,
> HBASE-14727.patch, rs-web-ui.png
>
>
> The stats of block cache are calculated at region server level only. So at
> region level, we could not know about the numbers of blocks cached, the sizes
> of cached blocks, the numbers of cache hit, etc. I suggest to add the stats
> of block cache at region level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[
https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984266#comment-14984266
]
Hadoop QA commented on HBASE-11393:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769972/HBASE-11393_v4.patch
against master branch at commit 6cc6d833f2cde47123876e5ca107b522004055ac.
ATTACHMENT ID: 12769972
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1731 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationWALEntryFilters
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16336//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16336//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16336//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16336//console
This message is automatically generated.
> Replication TableCfs should be a PB object rather than a string
> ---
>
> Key: HBASE-11393
> URL: https://issues.apache.org/jira/browse/HBASE-11393
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
> Attachments: HBASE-11393.patch, HBASE-11393_v1.patch,
> HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch
>
>
> We concatenate the list of tables and column families in format
> "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer
> mapping.
> This results in ugly parsing code. We should do this a PB object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-11393: -- Attachment: HBASE-11393_v4.patch > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Patch Available (was: Open) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Attachment: HBASE-14727-v2.patch fix test failure > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Open (was: Patch Available) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12816) GC logs are lost upon Region Server restart if GCLogFileRotation is enabled
[ https://issues.apache.org/jira/browse/HBASE-12816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12816: --- Fix Version/s: (was: 0.98.16) (was: 1.3.0) > GC logs are lost upon Region Server restart if GCLogFileRotation is enabled > --- > > Key: HBASE-12816 > URL: https://issues.apache.org/jira/browse/HBASE-12816 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12816.patch > > > When -XX:+UseGCLogFileRotation is used gc log files end with .gc.0 instead of > .gc. hbase_rotate_log () in hbase-daemon.sh does not handle this correctly > and hence when a RS is restarted old gc logs are lost(overwritten). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14289) Backport HBASE-13965 'Stochastic Load Balancer JMX Metrics' to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14289: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Backport HBASE-13965 'Stochastic Load Balancer JMX Metrics' to 0.98 > --- > > Key: HBASE-14289 > URL: https://issues.apache.org/jira/browse/HBASE-14289 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.17 > > Attachments: 14289-0.98-v2.txt, 14289-0.98-v3.txt, 14289-0.98-v4.txt, > 14289-0.98-v5.txt > > > The default HBase load balancer (the Stochastic load balancer) is cost > function based. The cost function weights are tunable but no visibility into > those cost function results is directly provided. > This issue backports HBASE-13965 to 0.98 branch to provide visibility via JMX > into each cost function of the stochastic load balancer, as well as the > overall cost of the balancing plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12148: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: John Leach > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12273) Generate .tabledesc file during upgrading if missing
[ https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12273: --- Assignee: (was: Yi Deng) > Generate .tabledesc file during upgrading if missing > > > Key: HBASE-12273 > URL: https://issues.apache.org/jira/browse/HBASE-12273 > Project: HBase > Issue Type: Sub-task > Components: Admin >Affects Versions: 1.0.0, 0.98.7 >Reporter: Yi Deng > Labels: upgrade > Fix For: 1.0.3, 0.98.17 > > Attachments: > 1.0-0001-HBASE-12273-Add-a-tool-for-fixing-missing-TableDescr.patch, > 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch > > > Generate .tabledesc file during upgrading if missing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12273) Generate .tabledesc file during upgrading if missing
[ https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12273: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Generate .tabledesc file during upgrading if missing > > > Key: HBASE-12273 > URL: https://issues.apache.org/jira/browse/HBASE-12273 > Project: HBase > Issue Type: Sub-task > Components: Admin >Affects Versions: 1.0.0, 0.98.7 >Reporter: Yi Deng >Assignee: Yi Deng > Labels: upgrade > Fix For: 1.0.3, 0.98.17 > > Attachments: > 1.0-0001-HBASE-12273-Add-a-tool-for-fixing-missing-TableDescr.patch, > 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch > > > Generate .tabledesc file during upgrading if missing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12890: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Provide a way to throttle the number of regions moved by the balancer > - > > Key: HBASE-12890 > URL: https://issues.apache.org/jira/browse/HBASE-12890 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.10 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-12890.patch > > > We have a very large cluster and we frequently add remove quite a few > regionservers from our cluster. Whenever we do this the balancer moves > thousands of regions at once. Instead we provide a configuration parameter: > hbase.balancer.max.regions. This limits the number of regions that are > balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12912) Cache Configuration used during scanner creation
[
https://issues.apache.org/jira/browse/HBASE-12912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12912:
---
Assignee: (was: Andrew Purtell)
> Cache Configuration used during scanner creation
>
>
> Key: HBASE-12912
> URL: https://issues.apache.org/jira/browse/HBASE-12912
> Project: HBase
> Issue Type: Sub-task
>Reporter: John Leach
> Attachments: StoreScannerStall.tiff
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> There is a clear CPU drain and iterator creation when creating store scanners
> under high load. Splice was running a TPCC test of our database and we are
> seeing object creation and CPU waste on the boolean check
> Code Snippet...
> if (store != null && ((HStore)store).getHRegion() != null
> && store.getStorefilesCount() > 1) {
> RegionServerServices rsService =
> ((HStore)store).getHRegion().getRegionServerServices();
> if (rsService == null || !rsService.getConfiguration().getBoolean(
> STORESCANNER_PARALLEL_SEEK_ENABLE, false)) return;
> isParallelSeekEnabled = true;
> executor = rsService.getExecutorService();
> }
> Will attach profile...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12912) Cache Configuration used during scanner creation
[
https://issues.apache.org/jira/browse/HBASE-12912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12912:
---
Fix Version/s: (was: 0.98.16)
(was: 1.3.0)
(was: 2.0.0)
> Cache Configuration used during scanner creation
>
>
> Key: HBASE-12912
> URL: https://issues.apache.org/jira/browse/HBASE-12912
> Project: HBase
> Issue Type: Sub-task
>Reporter: John Leach
> Attachments: StoreScannerStall.tiff
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> There is a clear CPU drain and iterator creation when creating store scanners
> under high load. Splice was running a TPCC test of our database and we are
> seeing object creation and CPU waste on the boolean check
> Code Snippet...
> if (store != null && ((HStore)store).getHRegion() != null
> && store.getStorefilesCount() > 1) {
> RegionServerServices rsService =
> ((HStore)store).getHRegion().getRegionServerServices();
> if (rsService == null || !rsService.getConfiguration().getBoolean(
> STORESCANNER_PARALLEL_SEEK_ENABLE, false)) return;
> isParallelSeekEnabled = true;
> executor = rsService.getExecutorService();
> }
> Will attach profile...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12912) Cache Configuration used during scanner creation
[
https://issues.apache.org/jira/browse/HBASE-12912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-12912.
Resolution: Fixed
Resolved by parent.
> Cache Configuration used during scanner creation
>
>
> Key: HBASE-12912
> URL: https://issues.apache.org/jira/browse/HBASE-12912
> Project: HBase
> Issue Type: Sub-task
>Reporter: John Leach
> Attachments: StoreScannerStall.tiff
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> There is a clear CPU drain and iterator creation when creating store scanners
> under high load. Splice was running a TPCC test of our database and we are
> seeing object creation and CPU waste on the boolean check
> Code Snippet...
> if (store != null && ((HStore)store).getHRegion() != null
> && store.getStorefilesCount() > 1) {
> RegionServerServices rsService =
> ((HStore)store).getHRegion().getRegionServerServices();
> if (rsService == null || !rsService.getConfiguration().getBoolean(
> STORESCANNER_PARALLEL_SEEK_ENABLE, false)) return;
> isParallelSeekEnabled = true;
> executor = rsService.getExecutorService();
> }
> Will attach profile...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13031) Ability to snapshot based on a key range
[ https://issues.apache.org/jira/browse/HBASE-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13031: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Ability to snapshot based on a key range > > > Key: HBASE-13031 > URL: https://issues.apache.org/jira/browse/HBASE-13031 > Project: HBase > Issue Type: Improvement >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-13031-v1.patch, HBASE-13031.patch > > > Posted on the mailing list and seems like some people are interested. A > little background for everyone. > We have a very large table, we would like to snapshot and transfer the data > to another cluster (compressed data is always better to ship). Our problem > lies in the fact it could take many weeks to transfer all of the data and > during that time with major compactions, the data stored in dfs has the > potential to double which would cause us to run out of disk space. > So we were thinking about allowing the ability to snapshot a specific key > range. > Ideally I feel the approach is that the user would specify a start and stop > key, those would be associated with a region boundary. If between the time > the user submits the request and the snapshot is taken the boundaries change > (due to merging or splitting of regions) the snapshot should fail. > We would know which regions to snapshot and if those changed between when the > request was submitted and the regions locked, the snapshot could simply fail > and the user would try again, instead of potentially giving the user more / > less than what they had anticipated. I was planning on storing the start / > stop key in the SnapshotDescription and from there it looks pretty straight > forward where we just have to change the verifier code to accommodate the key > ranges. > If this design sounds good to anyone, or if I am overlooking anything please > let me know. Once we agree on the design, I'll write and submit the patches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13268) Backport the HBASE-7781 security test updates to use the MiniKDC
[ https://issues.apache.org/jira/browse/HBASE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13268: --- Fix Version/s: (was: 0.98.16) > Backport the HBASE-7781 security test updates to use the MiniKDC > > > Key: HBASE-13268 > URL: https://issues.apache.org/jira/browse/HBASE-13268 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell > > Consider backport of the security test updates to use the MiniKDC that are > subtasks of HBASE-7781. Would be good to improve test coverage of security > code in 0.98 branch, as long as neither: > - The changes are a PITA to backport > - The changes break a compatibility requirement > - The changes introduce test instability > > Investigate -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13096) NPE from SecureWALCellCodec$EncryptedKvEncoder#write when using WAL encryption and Phoenix secondary indexes
[
https://issues.apache.org/jira/browse/HBASE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-13096:
---
Fix Version/s: (was: 0.98.16)
0.98.17
> NPE from SecureWALCellCodec$EncryptedKvEncoder#write when using WAL
> encryption and Phoenix secondary indexes
>
>
> Key: HBASE-13096
> URL: https://issues.apache.org/jira/browse/HBASE-13096
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Labels: phoenix
> Fix For: 0.98.17
>
>
> On user@phoenix Dhavi Rami reported:
> {quote}
> I tried using phoenix in hBase with Transparent Encryption of Data At Rest
> enabled ( AES encryption)
> Works fine for a table with primary key column.
> But it doesn't work if I create Secondary index on that tables.I tried to dig
> deep into the problem and found WAL file encryption throws exception when I
> have Global Secondary Index created on my mutable table.
> Following is the error I was getting on one of the region server.
> {noformat}
> 2015-02-20 10:44:48,768 ERROR
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: UNEXPECTED
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:767)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:754)
> at org.apache.hadoop.hbase.KeyValue.getKeyLength(KeyValue.java:1253)
> at
> org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec$EncryptedKvEncoder.write(SecureWALCellCodec.java:194)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:117)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1137)
> at java.lang.Thread.run(Thread.java:745)
> 2015-02-20 10:44:48,776 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> regionserver60020-WAL.AsyncWriter exiting
> {noformat}
> I had to disable WAL encryption, and it started working fine with secondary
> Index. So Hfile encryption works with secondary index but WAL encryption
> doesn't work.
> {quote}
> Parking this here for later investigation. For now I'm going to assume this
> is something in SecureWALCellCodec that needs looking at, but if it turns out
> to be a Phoenix indexer issue I will move this JIRA there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11290: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Unlock RegionStates > --- > > Key: HBASE-11290 > URL: https://issues.apache.org/jira/browse/HBASE-11290 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, > HBASE-11290.draft.patch > > > Even though RegionStates is a highly accessed data structure in HMaster. Most > of it's methods are synchronized. Which limits concurrency. Even simply > making some of the getters non-synchronized by using concurrent data > structures has helped with region assignments. We can go as simple as this > approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13336) Consistent rules for security meta table protections
[ https://issues.apache.org/jira/browse/HBASE-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13336: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Consistent rules for security meta table protections > > > Key: HBASE-13336 > URL: https://issues.apache.org/jira/browse/HBASE-13336 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-13336.patch, HBASE-13336_v2.patch > > > The AccessController and VisibilityController do different things regarding > protecting their meta tables. The AC allows schema changes and disable/enable > if the user has permission. The VC unconditionally disallows all admin > actions. Generally, bad things will happen if these meta tables are damaged, > disabled, or dropped. The likely outcome is random frequent (or constant) > server side op failures with nasty stack traces. On the other hand some > things like column family and table attribute changes can have valid use > cases. We should have consistent and sensible rules for protecting security > meta tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13511) Derive data keys with HKDF
[ https://issues.apache.org/jira/browse/HBASE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13511: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Derive data keys with HKDF > -- > > Key: HBASE-13511 > URL: https://issues.apache.org/jira/browse/HBASE-13511 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.4, 0.98.17 > > > When we are locally managing master key material, when users have supplied > their own data key material, derive the actual data keys using HKDF > (https://tools.ietf.org/html/rfc5869) > DK' = HKDF(S, DK, MK) > where > S = salt > DK = user supplied data key > MK = master key > DK' = derived data key for the HFile > User supplied key material may be weak or an attacker may have some partial > knowledge of it. > Where we generate random data keys we can still use HKDF as a way to mix more > entropy into the secure random generator. > DK' = HKDF(R, MK) > where > R = random key material drawn from the system's secure random generator > MK = master key > (Salting isn't useful here because salt S and R would be drawn from the same > pool, so will not have statistical independence.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13505) Deprecate the "AES" cipher type
[ https://issues.apache.org/jira/browse/HBASE-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13505: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Deprecate the "AES" cipher type > --- > > Key: HBASE-13505 > URL: https://issues.apache.org/jira/browse/HBASE-13505 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.4, 0.98.17 > > > Deprecate the "AES" cipher type. Remove internal references to it and use the > "AES-CTR" name instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13504) Alias current AES cipher as AES-CTR
[ https://issues.apache.org/jira/browse/HBASE-13504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13504: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Alias current AES cipher as AES-CTR > --- > > Key: HBASE-13504 > URL: https://issues.apache.org/jira/browse/HBASE-13504 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.4, 0.98.17 > > > Alias the current cipher with the name "AES" to the name "AES-CTR". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13506) AES-GCM cipher support where available
[ https://issues.apache.org/jira/browse/HBASE-13506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13506: --- Fix Version/s: (was: 0.98.16) 0.98.17 > AES-GCM cipher support where available > -- > > Key: HBASE-13506 > URL: https://issues.apache.org/jira/browse/HBASE-13506 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.4, 0.98.17 > > > The initial encryption drop only had AES-CTR support because authenticated > modes such as GCM are only available in Java 7 and up, and our trunk at the > time was targeted at Java 6. However we can optionally use AES-GCM cipher > support where available. For HBase 1.0 and up, Java 7 is now the minimum so > use of AES-GCM can go in directly. It's probably possible to add support in > 0.98 too using reflection for cipher object initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13896) Multi-actions in hbase-client could fall in dead loop when region moves.
[ https://issues.apache.org/jira/browse/HBASE-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-13896. Resolution: Incomplete Assignee: (was: Victor Xu) Fix Version/s: (was: 0.98.16) Resolving as incomplete. Reopen when there's progress. > Multi-actions in hbase-client could fall in dead loop when region moves. > > > Key: HBASE-13896 > URL: https://issues.apache.org/jira/browse/HBASE-13896 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.13 >Reporter: Victor Xu >Priority: Minor > Attachments: HBASE-13896-0.98-v1.patch > > > The code in AsyncProcess.receiveGlobalFailure() use only one row to update > region cache in hbase-client. When we use HTable.put(List) api to write > some data which are from different regions and some of them are > moved/balanced while writing, the client may fall into a dead loop: > multi-actions fails because some regions moved => update only one region > cache(not the wrong ones) => resubmit => failed again. > It only happens in 0.98 branch, and the master branch is ok. > The patch followed should update all cached region locations when > multi-actions fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13667) Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks
[ https://issues.apache.org/jira/browse/HBASE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13667: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks > > > Key: HBASE-13667 > URL: https://issues.apache.org/jira/browse/HBASE-13667 > Project: HBase > Issue Type: Bug >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Fix For: 1.0.3, 0.98.17 > > > We can backport Split transaction, region merge transaction interfaces to > branch 1.0 and 0.98 without changing coprocessor hooks. Then it should be > compatible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13857) Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
[
https://issues.apache.org/jira/browse/HBASE-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-13857:
---
Fix Version/s: (was: 0.98.16)
0.98.17
> Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
> -
>
> Key: HBASE-13857
> URL: https://issues.apache.org/jira/browse/HBASE-13857
> Project: HBase
> Issue Type: Bug
> Components: regionserver, UI
>Affects Versions: 0.98.0
>Reporter: Lars George
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
>
> The template has this:
> {noformat}
>
> ...
> Slow WAL Append Count
>
>
>
> <% 0 %>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14067) bundle ruby files for hbase shell into a jar.
[ https://issues.apache.org/jira/browse/HBASE-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14067: --- Fix Version/s: (was: 0.98.16) 0.98.17 > bundle ruby files for hbase shell into a jar. > - > > Key: HBASE-14067 > URL: https://issues.apache.org/jira/browse/HBASE-14067 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Sean Busbey > Fix For: 2.0.0, 1.3.0, 0.98.17 > > > We currently package all the ruby scripts for the hbase shell by placing them > in a directory within lib/. We should be able to put these in a jar file > since we rely on jruby. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14129) If any regionserver gets shutdown uncleanly during full cluster restart, locality looks to be lost
[ https://issues.apache.org/jira/browse/HBASE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14129: --- Fix Version/s: (was: 0.98.16) > If any regionserver gets shutdown uncleanly during full cluster restart, > locality looks to be lost > -- > > Key: HBASE-14129 > URL: https://issues.apache.org/jira/browse/HBASE-14129 > Project: HBase > Issue Type: Bug >Reporter: churro morales > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14129.patch > > > We were doing a cluster restart the other day. Some regionservers did not > shut down cleanly. Upon restart our locality went from 99% to 5%. Upon > looking at the AssignmentManager.joinCluster() code it calls > AssignmentManager.processDeadServersAndRegionsInTransition(). > If the failover flag gets set for any reason it seems we don't call > assignAllUserRegions(). Then it looks like the balancer does the work in > assigning those regions, we don't use a locality aware balancer and we lost > our region locality. > I don't have a solid grasp on the reasoning for these checks but there could > be some potential workarounds here. > 1. After shutting down your cluster, move your WALs aside (replay later). > 2. Clean up your zNodes > That seems to work, but requires a lot of manual labor. Another solution > which I prefer would be to have a flag for ./start-hbase.sh --clean > If we start master with that flag then we do a check in > AssignmentManager.processDeadServersAndRegionsInTransition() thus if this > flag is set we call: assignAllUserRegions() regardless of the failover state. > I have a patch for the later solution, that is if I am understanding the > logic correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14049) SnapshotHFileCleaner should optionally clean up after failed snapshots
[ https://issues.apache.org/jira/browse/HBASE-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14049: --- Fix Version/s: (was: 0.98.16) 0.98.17 > SnapshotHFileCleaner should optionally clean up after failed snapshots > -- > > Key: HBASE-14049 > URL: https://issues.apache.org/jira/browse/HBASE-14049 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.13 >Reporter: Andrew Purtell > Fix For: 2.0.0, 1.3.0, 0.98.17 > > > SnapshotHFileCleaner should optionally clean up after failed snapshots rather > than just complain. Add a configuration option that, if set to true > (defaulting to false), instructs SnapshotHFileCleaner to recursively remove > failed snapshot temporary directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14276) NPE when setting up stub for connection to master if secure connect is refused
[
https://issues.apache.org/jira/browse/HBASE-14276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14276:
---
Fix Version/s: (was: 0.98.16)
0.98.17
> NPE when setting up stub for connection to master if secure connect is refused
> --
>
> Key: HBASE-14276
> URL: https://issues.apache.org/jira/browse/HBASE-14276
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.17
>
>
> If an insecure client contacts a secure cluster we can get an NPE when
> setting up the stub for the master connection. We should throw an IOException
> with a clear message, and not retry if possible to distinguish this case.
> Found in 0.98 but might be relevant for later branches
> {noformat}
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> INFO: Process identifier=hconnection-0x3c1e23ff connecting to ZooKeeper
> ensemble=x.x.x.x:2181,x.x.x.x:2181,x.x.x.x:2181
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> makeStub
> INFO: getMaster attempt 1 of 35 failed; retrying after sleep of 100,
> exception=com.google.protobuf.ServiceException: java.lang.NullPointerException
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> makeStub
> INFO: getMaster attempt 2 of 35 failed; retrying after sleep of 200,
> exception=com.google.protobuf.ServiceException: java.io.IOException: Call to
> /x.x.x.x:6 failed on local exception: java.io.EOFException
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> makeStub
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14177) Full GC on client may lead to missing scan results
[ https://issues.apache.org/jira/browse/HBASE-14177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14177: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Full GC on client may lead to missing scan results > -- > > Key: HBASE-14177 > URL: https://issues.apache.org/jira/browse/HBASE-14177 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.12, 0.98.13, 1.0.2 >Reporter: James Estes >Priority: Critical > Labels: dataloss > Fix For: 1.0.3, 0.98.17 > > > After adding a large row, scanning back that row winds up being empty. After > a few attempts it will succeed (all attempts over the same data on an hbase > getting no other writes). > Looking at logs, it seems this happens when there is memory pressure on the > client and there are several Full GCs that happen. Then messages that > indicate that region locations are being removed from the local client cache: > 2015-07-31 12:50:24,647 [main] DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation > - Removed 192.168.1.131:50981 as a location of > big_row_1438368609944,,1438368610048.880c849594807bdc7412f4f982337d6c. for > tableName=big_row_1438368609944 from cache > Blaming the GC may sound fanciful, but if the test is run with -Xms4g -Xmx4g > then it always passes on the first scan attempt. Maybe the pause is enough to > remove something from the cache, or the client is using weak references > somewhere? > More info > http://mail-archives.apache.org/mod_mbox/hbase-user/201507.mbox/%3CCAE8tVdnFf%3Dob569%3DfJkpw1ndVWOVTkihYj9eo6qt0FrzihYHgw%40mail.gmail.com%3E > Test used to reproduce: > https://github.com/housejester/hbase-debugging#fullgctest > I tested and had failures in: > 0.98.12 client/server > 0.98.13 client 0.98.12 server > 0.98.13 client/server > 1.1.0 client 0.98.13 server > 0.98.13 client and 1.1.0 server > 0.98.12 client and 1.1.0 server > I tested without failure in: > 1.1.0 client/server -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14546) Backport stub DNS re-resolution options to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14546: --- Fix Version/s: (was: 0.98.16) 0.98.17 > Backport stub DNS re-resolution options to 0.98 > --- > > Key: HBASE-14546 > URL: https://issues.apache.org/jira/browse/HBASE-14546 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.98.17 > > > HBASE-12943 and HBASE-13067 addresses infinite caching preventing servers > from rejoining a cluster using the same hostname but a different IP address. > HBASE-14544 modifies this to be optional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14611) Backport HBASE-14473 (Compute region locality in parallel) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984159#comment-14984159 ] Andrew Purtell commented on HBASE-14611: Any concerns [~lhofhansl] ? > Backport HBASE-14473 (Compute region locality in parallel) to 0.98 > -- > > Key: HBASE-14611 > URL: https://issues.apache.org/jira/browse/HBASE-14611 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.16 > > Attachments: HBASE-14611-0.98.patch > > > [~eclark] contributed a nice change on HBASE-14473 that scales calculation of > locality balance cost to larger clusters. I'd like to bring this back to 0.98 > for folks running it on larger clusters. The changes require a partial > rewrite of RegionLocationFinder hence a backport issue. > The difference between this backport and RegionLocationFinder on later > branches is we preserve the ability to change the expiration time of cache > items with the configuration parameter > "hbase.master.balancer.regionLocationCacheTime". > The difference between RegionLocationFinder in 0.98 before and after this > change is the expiration time of cache entries will be set according to when > they are written into the cache instead of from when they are last accessed, > which seems better to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13014) Java Tool For Region Moving
[
https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984158#comment-14984158
]
Hadoop QA commented on HBASE-13014:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769950/HBASE-13014-master-v3.patch
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769950
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16334//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16334//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16334//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16334//console
This message is automatically generated.
> Java Tool For Region Moving
>
>
> Key: HBASE-13014
> URL: https://issues.apache.org/jira/browse/HBASE-13014
> Project: HBase
> Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-13014-master-v2.patch,
> HBASE-13014-master-v3.patch, HBASE-13014-master-v3.patch,
> HBASE-13014-master-v3.patch, HBASE-13014-master.patch, HBASE-13014-v2.patch,
> HBASE-13014-v3.patch, HBASE-13014-v4.patch, HBASE-13014-v5.patch,
> HBASE-13014-v6.patch, HBASE-13014.patch
>
>
> As per discussion on HBASE-12989 we should move the functionality of
> region_mover.rb into a Java tool and use region_mover.rb only only as a
> wrapper around it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984102#comment-14984102 ] Nick Dimiduk commented on HBASE-11368: -- bq. any current on going scan will not be able to see the bulk loaded hfiles which is loaded just after the current scan has started. I think that behaviour should be acceptable, right? I believe this should be correct behavior, yes. > Multi-column family BulkLoad fails if compactions go on too long > > > Key: HBASE-11368 > URL: https://issues.apache.org/jira/browse/HBASE-11368 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Qiang Tian > Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch, > key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch > > > Compactions take a read lock. If a multi-column family region, before bulk > loading, we want to take a write lock on the region. If the compaction takes > too long, the bulk load fails. > Various recipes include: > + Making smaller regions (lame) > + [~victorunique] suggests major compacting just before bulk loading over in > HBASE-10882 as a work around. > Does the compaction need a read lock for that long? Does the bulk load need > a full write lock when multiple column families? Can we fail more gracefully > at least? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-13014: --- Attachment: HBASE-13014-master-v3.patch Attaching once more. Hoping for a stable run. > Java Tool For Region Moving > > > Key: HBASE-13014 > URL: https://issues.apache.org/jira/browse/HBASE-13014 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0 > > Attachments: HBASE-13014-master-v2.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master-v3.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master.patch, HBASE-13014-v2.patch, > HBASE-13014-v3.patch, HBASE-13014-v4.patch, HBASE-13014-v5.patch, > HBASE-13014-v6.patch, HBASE-13014.patch > > > As per discussion on HBASE-12989 we should move the functionality of > region_mover.rb into a Java tool and use region_mover.rb only only as a > wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-13014: --- Attachment: (was: HBASE-13014-master-v3.patch) > Java Tool For Region Moving > > > Key: HBASE-13014 > URL: https://issues.apache.org/jira/browse/HBASE-13014 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0 > > Attachments: HBASE-13014-master-v2.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master-v3.patch, > HBASE-13014-master.patch, HBASE-13014-v2.patch, HBASE-13014-v3.patch, > HBASE-13014-v4.patch, HBASE-13014-v5.patch, HBASE-13014-v6.patch, > HBASE-13014.patch > > > As per discussion on HBASE-12989 we should move the functionality of > region_mover.rb into a Java tool and use region_mover.rb only only as a > wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984073#comment-14984073 ] stack commented on HBASE-14575: --- The original author says of the approach "... I'm not sure if it's correct..." I'd be interested in a paragraph on why the current author thinks this patch is (correct). Also, what testing has been done on this approach? (Testing to verify the approach is what the original fellows working on this issue were most concerned about). Giving the patch a quick review, yeah, it complicates HBASE-13082. It might work though given its passing the region lock... I tried to reason about it but was having trouble thinking through scenarios where a close could come in mid-compact or a bulk load. Would appreciate a write up what the current thinking is. Would help w/ this review (and I'm sure would help the fellow trying to land HBASE-13082) Thanks. > Reduce scope of compactions holding region lock > --- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, > 14575-v4.patch, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984069#comment-14984069 ] Abhishek Singh Chouhan commented on HBASE-13014: Will paste the usage output soon. Intent is to replace the region_mover.rb script in bin (this also introduces the no-ack mode and also the timeout option) with this, the ruby script will just be used to call this. I'll create jira for that. Once this is in master and is sufficiently stable we can then replace the ruby script for good(rolling restart will also come to this then), i'll also work on putting this through to 0.98 and branch-1. > Java Tool For Region Moving > > > Key: HBASE-13014 > URL: https://issues.apache.org/jira/browse/HBASE-13014 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0 > > Attachments: HBASE-13014-master-v2.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master-v3.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master.patch, HBASE-13014-v2.patch, > HBASE-13014-v3.patch, HBASE-13014-v4.patch, HBASE-13014-v5.patch, > HBASE-13014-v6.patch, HBASE-13014.patch > > > As per discussion on HBASE-12989 we should move the functionality of > region_mover.rb into a Java tool and use region_mover.rb only only as a > wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984067#comment-14984067 ] Abhishek Singh Chouhan commented on HBASE-13014: Will paste the usage output soon. Intent is to replace the region_mover.rb script in bin (this also introduces the no-ack mode and also the timeout option) with this, the ruby script will just be used to call this. I'll create jira for that. Once this is in master and is sufficiently stable we can then replace the ruby script for good(rolling restart will also come to this then), i'll also work on putting this through to 0.98 and branch-1. > Java Tool For Region Moving > > > Key: HBASE-13014 > URL: https://issues.apache.org/jira/browse/HBASE-13014 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0 > > Attachments: HBASE-13014-master-v2.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master-v3.patch, > HBASE-13014-master-v3.patch, HBASE-13014-master.patch, HBASE-13014-v2.patch, > HBASE-13014-v3.patch, HBASE-13014-v4.patch, HBASE-13014-v5.patch, > HBASE-13014-v6.patch, HBASE-13014.patch > > > As per discussion on HBASE-12989 we should move the functionality of > region_mover.rb into a Java tool and use region_mover.rb only only as a > wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[
https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984064#comment-14984064
]
Hadoop QA commented on HBASE-11393:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769938/HBASE-11393_v3.patch
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769938
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestBlockEvictionFromClient
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16333//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16333//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16333//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16333//console
This message is automatically generated.
> Replication TableCfs should be a PB object rather than a string
> ---
>
> Key: HBASE-11393
> URL: https://issues.apache.org/jira/browse/HBASE-11393
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
> Attachments: HBASE-11393.patch, HBASE-11393_v1.patch,
> HBASE-11393_v2.patch, HBASE-11393_v3.patch
>
>
> We concatenate the list of tables and column families in format
> "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer
> mapping.
> This results in ugly parsing code. We should do this a PB object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14728) TestRowCounter is broken in master
[
https://issues.apache.org/jira/browse/HBASE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984061#comment-14984061
]
Abhishek Singh Chouhan commented on HBASE-14728:
QA gods finally seem pleased :D
> TestRowCounter is broken in master
> --
>
> Key: HBASE-14728
> URL: https://issues.apache.org/jira/browse/HBASE-14728
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-14728.patch, HBASE-14728.patch
>
>
> The runRowCount method simply runs the RowCounter and checks if the exit code
> is zero but does not actually check the row count returned
> {noformat}
> private void runRowCount(String[] args, int expectedCount) throws Exception {
> final RowCounter counter = new RowCounter();
> assertEquals("job failed either due to failure or miscount (see log
> output).", 0,
> ToolRunner.run(TEST_UTIL.getConfiguration(), counter, args));
> }
> {noformat}
> This will always give a false positive provided the job ran without errors
> (irrespective of the count it gives).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14605) Split fails due to 'No valid credentials' error when SecureBulkLoadEndpoint#start tries to access hdfs
[
https://issues.apache.org/jira/browse/HBASE-14605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984058#comment-14984058
]
Hudson commented on HBASE-14605:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1126 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1126/])
HBASE-14605 Addendum restores two methods in SplitTransaction (tedyu: rev
90d920caa64ef67b186314157958e349022e3015)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> Split fails due to 'No valid credentials' error when
> SecureBulkLoadEndpoint#start tries to access hdfs
> --
>
> Key: HBASE-14605
> URL: https://issues.apache.org/jira/browse/HBASE-14605
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 144605-branch-1-v3.txt, 14605-0.98-v5.txt,
> 14605-branch-1-addendum.txt, 14605-branch-1-v4.txt, 14605-branch-1-v5.txt,
> 14605-branch-1.0-v5.txt, 14605-v1.txt, 14605-v2.txt, 14605-v3.txt,
> 14605-v3.txt, 14605-v3.txt, 14605-v4.txt, 14605-v5.txt, 14605.alt
>
>
> During recent testing in secure cluster (with HBASE-14475), we found the
> following when user X (non-super user) split a table with region replica:
> {code}
> 2015-10-12 10:58:18,955 ERROR [FifoRpcScheduler.handler1-thread-9]
> master.HMaster: Region server hbase-4-4.novalocal,60020,1444645588137
> reported a fatal error:
> ABORTING region server hbase-4-4.novalocal,60020,1444645588137: The
> coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> threw an unexpected exception
> Cause:
> java.lang.IllegalStateException: Failed to get FileSystem instance
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:148)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:257)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:160)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:192)
> at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:701)
> at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:608)
> ...
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local host is:
> "hbase-4-4/172.22.66.186"; destination host is: "os-r6-
> okarus-hbase-4-2.novalocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1473)
> at org.apache.hadoop.ipc.Client.call(Client.java:1400)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy18.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:555)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy19.mkdirs(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2775)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2746)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:967)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:963)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> {code}
> The cause was that SecureBulkLoadEndpoint#start tried to create staging dir
> in hdfs as user X but didn't pass authentication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984057#comment-14984057
]
Hudson commented on HBASE-14733:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1126 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1126/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
7bc22570b1fd35883a654ebfc2e3752a8ee46eaa)
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14736) ITBLL debugging search tool OOMEs on big dataset
stack created HBASE-14736:
-
Summary: ITBLL debugging search tool OOMEs on big dataset
Key: HBASE-14736
URL: https://issues.apache.org/jira/browse/HBASE-14736
Project: HBase
Issue Type: Bug
Reporter: stack
I ran an ITBLL on an 80 node cluster sized to do 100B items. The job failed
with 300M undefined items (branch-1). I tried to run the search tool debugging
the loss -- see
https://docs.google.com/document/d/14Tvu5yWYNBDFkh8xCqLkU9tlyNWhJv3GjDGOkqZU1eE/edit#
-- but it OOME'd:
{code}
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:834)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:697)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:2452)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileAsBinaryInputFormat$SequenceFileAsBinaryRecordReader.nextKeyValue(SequenceFileAsBinaryInputFormat.java:119)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.readFileToSearch(IntegrationTestBigLinkedList.java:775)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.readKeysToSearch(IntegrationTestBigLinkedList.java:757)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.run(IntegrationTestBigLinkedList.java:726)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Search.run(IntegrationTestBigLinkedList.java:657)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.runTestFromCommandLine(IntegrationTestBigLinkedList.java:1646)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:123)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1686)
{code}
Its trying to build a sorted set out of the 300M items Dang.
The 10B test passed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14605) Split fails due to 'No valid credentials' error when SecureBulkLoadEndpoint#start tries to access hdfs
[
https://issues.apache.org/jira/browse/HBASE-14605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984053#comment-14984053
]
Hudson commented on HBASE-14605:
FAILURE: Integrated in HBase-1.0 #1105 (See
[https://builds.apache.org/job/HBase-1.0/1105/])
HBASE-14605 Addendum restores two methods in SplitTransaction (tedyu: rev
e1e6c6e3d668a447d8704e4c2f37eb02c9b61e67)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
> Split fails due to 'No valid credentials' error when
> SecureBulkLoadEndpoint#start tries to access hdfs
> --
>
> Key: HBASE-14605
> URL: https://issues.apache.org/jira/browse/HBASE-14605
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 144605-branch-1-v3.txt, 14605-0.98-v5.txt,
> 14605-branch-1-addendum.txt, 14605-branch-1-v4.txt, 14605-branch-1-v5.txt,
> 14605-branch-1.0-v5.txt, 14605-v1.txt, 14605-v2.txt, 14605-v3.txt,
> 14605-v3.txt, 14605-v3.txt, 14605-v4.txt, 14605-v5.txt, 14605.alt
>
>
> During recent testing in secure cluster (with HBASE-14475), we found the
> following when user X (non-super user) split a table with region replica:
> {code}
> 2015-10-12 10:58:18,955 ERROR [FifoRpcScheduler.handler1-thread-9]
> master.HMaster: Region server hbase-4-4.novalocal,60020,1444645588137
> reported a fatal error:
> ABORTING region server hbase-4-4.novalocal,60020,1444645588137: The
> coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> threw an unexpected exception
> Cause:
> java.lang.IllegalStateException: Failed to get FileSystem instance
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:148)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:257)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:160)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:192)
> at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:701)
> at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:608)
> ...
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local host is:
> "hbase-4-4/172.22.66.186"; destination host is: "os-r6-
> okarus-hbase-4-2.novalocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1473)
> at org.apache.hadoop.ipc.Client.call(Client.java:1400)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy18.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:555)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy19.mkdirs(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2775)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2746)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:967)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:963)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> {code}
> The cause was that SecureBulkLoadEndpoint#start tried to create staging dir
> in hdfs as user X but didn't pass authentication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984052#comment-14984052
]
Hudson commented on HBASE-14733:
FAILURE: Integrated in HBase-1.0 #1105 (See
[https://builds.apache.org/job/HBase-1.0/1105/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
ed25a475aa5178d07b46eb24a5fd6e6ab78329d9)
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-11393: -- Attachment: HBASE-11393_v3.patch Fix bug which cause testcase failed. One stupid bug wasted my whole night :( > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14575) Reduce scope of compactions holding region lock
[
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983963#comment-14983963
]
Ted Yu edited comment on HBASE-14575 at 10/31/15 11:51 AM:
---
bq. this patch should remove the read lock in the HRegion.compact()
This is what the patch is doing:
{code}
1787// block waiting for the lock for compaction
1788lock.readLock().lock();
{code}
The above snippet was copied from browser which has plugin for patch files.
In the real patch, you can see '-' sign at the beginning of each line.
was (Author: yuzhih...@gmail.com):
bq. this patch should remove the read lock in the HRegion.compact()
This is what the patch is doing:
{code}
1787// block waiting for the lock for compaction
1788lock.readLock().lock();
{code}
> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch,
> 14575-v4.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope
> of critical section under which compactions hold the region read lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983963#comment-14983963
]
Ted Yu commented on HBASE-14575:
bq. this patch should remove the read lock in the HRegion.compact()
This is what the patch is doing:
{code}
1787// block waiting for the lock for compaction
1788lock.readLock().lock();
{code}
> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch,
> 14575-v4.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope
> of critical section under which compactions hold the region read lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14735) Region may grow too big and can not be split
[
https://issues.apache.org/jira/browse/HBASE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983953#comment-14983953
]
Ted Yu commented on HBASE-14735:
Want to attach a patch ?
> Region may grow too big and can not be split
>
>
> Key: HBASE-14735
> URL: https://issues.apache.org/jira/browse/HBASE-14735
> Project: HBase
> Issue Type: Bug
> Components: Compaction, regionserver
>Affects Versions: 1.1.2, 0.98.15
>Reporter: Shuaifeng Zhou
>Assignee: Shuaifeng Zhou
>
> When a compaction completed, may there are also many storefiles in the store,
> and CompactPriority < 0, then compactSplitThread will do a "Recursive
> enqueue" compaction request instead of request a split:
> {code:title=CompactSplitThread.java|borderStyle=solid}
> if (completed) {
> // degenerate case: blocked regions require recursive enqueues
> if (store.getCompactPriority() <= 0) {
> requestSystemCompaction(region, store, "Recursive enqueue");
> } else {
> // see if the compaction has caused us to exceed max region size
> requestSplit(region);
> }
> {code}
> But in some situation, the "recursive enqueue" request may return null, and
> not build up a new compaction runner. For example, an other compaction of the
> same region is running, and compaction selection will exclude all files older
> than the newest files currently compacting, this may cause no enough files
> can be selected by the "recursive enqueue" request. When this happen, split
> will not be trigged. If the input load is high enough, compactions aways
> running on the region, and split will never be triggered.
> In our cluster, this situation happened, and a huge region more than 400GB
> and 100+ storefiles appeared. Version is 0.98.10, and the trank also have the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983941#comment-14983941
]
Heng Chen commented on HBASE-14703:
---
Oh.. I found two difference point between original code and the patch.
* In original code, in mutateRow, it called callWithRetries; but in
asyncProcess.submit, it called .callWithOutRetries
* timeout setting is different.
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Status: Open (was: Patch Available)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Status: Patch Available (was: Open)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Attachment: HBASE-14703_v1.patch
I update the title of this issue
Try this patch
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch, HBASE-14703_v1.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Summary: not collect stats when call HTable.mutateRow (was: update the
per-region stats twice for the call on return)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14735) Region may grow too big and can not be split
[
https://issues.apache.org/jira/browse/HBASE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983921#comment-14983921
]
Shuaifeng Zhou commented on HBASE-14735:
edit the requestSplit function, remove
-deleted-&& r.getCompactPriority() >= Store.PRIORITY_USER
{code:title=CompactSplitThread.java|borderStyle=solid}
public synchronized boolean requestSplit(final HRegion r) {
// don't split regions that are blocking
if (shouldSplitRegion()) {
byte[] midKey = r.checkSplit();
if (midKey != null) {
requestSplit(r, midKey);
return true;
}
}
return false;
}
{code}
> Region may grow too big and can not be split
>
>
> Key: HBASE-14735
> URL: https://issues.apache.org/jira/browse/HBASE-14735
> Project: HBase
> Issue Type: Bug
> Components: Compaction, regionserver
>Affects Versions: 1.1.2, 0.98.15
>Reporter: Shuaifeng Zhou
>Assignee: Shuaifeng Zhou
>
> When a compaction completed, may there are also many storefiles in the store,
> and CompactPriority < 0, then compactSplitThread will do a "Recursive
> enqueue" compaction request instead of request a split:
> {code:title=CompactSplitThread.java|borderStyle=solid}
> if (completed) {
> // degenerate case: blocked regions require recursive enqueues
> if (store.getCompactPriority() <= 0) {
> requestSystemCompaction(region, store, "Recursive enqueue");
> } else {
> // see if the compaction has caused us to exceed max region size
> requestSplit(region);
> }
> {code}
> But in some situation, the "recursive enqueue" request may return null, and
> not build up a new compaction runner. For example, an other compaction of the
> same region is running, and compaction selection will exclude all files older
> than the newest files currently compacting, this may cause no enough files
> can be selected by the "recursive enqueue" request. When this happen, split
> will not be trigged. If the input load is high enough, compactions aways
> running on the region, and split will never be triggered.
> In our cluster, this situation happened, and a huge region more than 400GB
> and 100+ storefiles appeared. Version is 0.98.10, and the trank also have the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14735) Region may grow too big and can not be split
[
https://issues.apache.org/jira/browse/HBASE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983918#comment-14983918
]
Shuaifeng Zhou commented on HBASE-14735:
A solution is remove "else" switch, give a chance to split after each completed
compaction.
{code:title=CompactSplitThread.java|borderStyle=solid}
if (completed) {
// degenerate case: blocked regions require recursive enqueues
if (store.getCompactPriority() <= 0) {
requestSystemCompaction(region, store, "Recursive enqueue");
}
// see if the compaction has caused us to exceed max region size
requestSplit(region);
{code}
> Region may grow too big and can not be split
>
>
> Key: HBASE-14735
> URL: https://issues.apache.org/jira/browse/HBASE-14735
> Project: HBase
> Issue Type: Bug
> Components: Compaction, regionserver
>Affects Versions: 1.1.2, 0.98.15
>Reporter: Shuaifeng Zhou
>Assignee: Shuaifeng Zhou
>
> When a compaction completed, may there are also many storefiles in the store,
> and CompactPriority < 0, then compactSplitThread will do a "Recursive
> enqueue" compaction request instead of request a split:
> {code:title=CompactSplitThread.java|borderStyle=solid}
> if (completed) {
> // degenerate case: blocked regions require recursive enqueues
> if (store.getCompactPriority() <= 0) {
> requestSystemCompaction(region, store, "Recursive enqueue");
> } else {
> // see if the compaction has caused us to exceed max region size
> requestSplit(region);
> }
> {code}
> But in some situation, the "recursive enqueue" request may return null, and
> not build up a new compaction runner. For example, an other compaction of the
> same region is running, and compaction selection will exclude all files older
> than the newest files currently compacting, this may cause no enough files
> can be selected by the "recursive enqueue" request. When this happen, split
> will not be trigged. If the input load is high enough, compactions aways
> running on the region, and split will never be triggered.
> In our cluster, this situation happened, and a huge region more than 400GB
> and 100+ storefiles appeared. Version is 0.98.10, and the trank also have the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983917#comment-14983917
]
Hadoop QA commented on HBASE-13082:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769932/HBASE-13082_9.patch
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769932
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 38 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1731 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16332//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16332//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16332//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16332//console
This message is automatically generated.
> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt,
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch,
> HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch,
> HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png, gc.png, gc.png, hits.png,
> next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and
> required in the documentation (not picking on Phoenix, this one is my fault
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load.
> RegionScanner operations would keep getting the locks and the
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983914#comment-14983914 ] Sean Busbey commented on HBASE-14725: - +1 > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt, > 14725v2.txt, 14725v4.txt, 14725v4.txt, categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14735) Region may grow too big and can not be split
Shuaifeng Zhou created HBASE-14735:
--
Summary: Region may grow too big and can not be split
Key: HBASE-14735
URL: https://issues.apache.org/jira/browse/HBASE-14735
Project: HBase
Issue Type: Bug
Components: Compaction, regionserver
Affects Versions: 0.98.15, 1.1.2
Reporter: Shuaifeng Zhou
Assignee: Shuaifeng Zhou
When a compaction completed, may there are also many storefiles in the store,
and CompactPriority < 0, then compactSplitThread will do a "Recursive enqueue"
compaction request instead of request a split:
{code:title=CompactSplitThread.java|borderStyle=solid}
if (completed) {
// degenerate case: blocked regions require recursive enqueues
if (store.getCompactPriority() <= 0) {
requestSystemCompaction(region, store, "Recursive enqueue");
} else {
// see if the compaction has caused us to exceed max region size
requestSplit(region);
}
{code}
But in some situation, the "recursive enqueue" request may return null, and not
build up a new compaction runner. For example, an other compaction of the same
region is running, and compaction selection will exclude all files older than
the newest files currently compacting, this may cause no enough files can be
selected by the "recursive enqueue" request. When this happen, split will not
be trigged. If the input load is high enough, compactions aways running on the
region, and split will never be triggered.
In our cluster, this situation happened, and a huge region more than 400GB and
100+ storefiles appeared. Version is 0.98.10, and the trank also have the
problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash
stack created HBASE-14734: - Summary: TestGenerateDelegationToken fails with BindAddress clash Key: HBASE-14734 URL: https://issues.apache.org/jira/browse/HBASE-14734 Project: HBase Issue Type: Bug Components: flakey, test Reporter: stack >From >https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ Error Message Address already in use Stacktrace java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:444) at sun.nio.ch.Net.bind(Net.java:436) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983899#comment-14983899 ] Hudson commented on HBASE-14700: SUCCESS: Integrated in HBase-1.3 #330 (See [https://builds.apache.org/job/HBase-1.3/330/]) HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev 85f2aee07098443c2071b4a60f031fbe9c30486c) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983900#comment-14983900
]
Hudson commented on HBASE-14733:
SUCCESS: Integrated in HBase-1.3 #330 (See
[https://builds.apache.org/job/HBase-1.3/330/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
b8a2caf159f74421c4cb786d3e68c2fe98d0918a)
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13082: --- Status: Patch Available (was: Open) > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch, > HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, > HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png, gc.png, gc.png, hits.png, > next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[
https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983898#comment-14983898
]
Hadoop QA commented on HBASE-14725:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769928/14725v4.txt
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769928
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16330//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16330//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16330//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16330//console
This message is automatically generated.
> Vet categorization of tests so they for sure go into the right
> small/medium/large buckets
> -
>
> Key: HBASE-14725
> URL: https://issues.apache.org/jira/browse/HBASE-14725
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt,
> 14725v2.txt, 14725v4.txt, 14725v4.txt, categorization.patch
>
>
> I tried doing runSmallTests, runMediumTests, etc., and I noticed that some
> tests are larger than our categorization. At least for small tests it means
> they area all running in the one JVM. I also noticed that the categorization
> only takes effect in hbase-server.
> This patch makes it so runSmallTests runs all the small tests only in each
> category. Also moves tests that were larger than small out to medium so they
> don't run in the one JVM anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13082: --- Attachment: HBASE-13082_9.patch Updated patch for Hadoop QA. Previous patch did not apply cleanly. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch, > HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, > HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png, gc.png, gc.png, hits.png, > next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13082: --- Status: Open (was: Patch Available) > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1_WIP.patch, > HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, HBASE-13082_4.patch, > HBASE-13082_9.patch, gc.png, gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983894#comment-14983894 ] Hudson commented on HBASE-14700: FAILURE: Integrated in HBase-Trunk_matrix #416 (See [https://builds.apache.org/job/HBase-Trunk_matrix/416/]) HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983892#comment-14983892
]
Hudson commented on HBASE-14733:
FAILURE: Integrated in HBase-1.1-JDK8 #1669 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1669/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
3dc89fb1d4abe50c783e0b77f9e3e071a0d11f72)
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983888#comment-14983888 ] Hudson commented on HBASE-14700: FAILURE: Integrated in HBase-1.2 #330 (See [https://builds.apache.org/job/HBase-1.2/330/]) HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev f273d147b7ecdc30484e94a8b2fed622e53d7fbd) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983887#comment-14983887
]
Hudson commented on HBASE-14733:
FAILURE: Integrated in HBase-1.1-JDK7 #1581 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1581/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
3dc89fb1d4abe50c783e0b77f9e3e071a0d11f72)
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983889#comment-14983889
]
Hudson commented on HBASE-14733:
FAILURE: Integrated in HBase-1.2 #330 (See
[https://builds.apache.org/job/HBase-1.2/330/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
68eea04745786e829bb1a8a6b4e0a481fa6a7656)
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983880#comment-14983880 ] Hudson commented on HBASE-14700: SUCCESS: Integrated in HBase-1.3-IT #286 (See [https://builds.apache.org/job/HBase-1.3-IT/286/]) HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev 85f2aee07098443c2071b4a60f031fbe9c30486c) * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983879#comment-14983879 ] ramkrishna.s.vasudevan commented on HBASE-11368: This comment https://issues.apache.org/jira/browse/HBASE-11368?focusedCommentId=14693166&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14693166 is getting addressed as part of HBASE-13082. So doing that JIRA would mean that any current on going scan will not be able to see the bulk loaded hfiles which is loaded just after the current scan has started. I think that behaviour should be acceptable, right? > Multi-column family BulkLoad fails if compactions go on too long > > > Key: HBASE-11368 > URL: https://issues.apache.org/jira/browse/HBASE-11368 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Qiang Tian > Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch, > key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch > > > Compactions take a read lock. If a multi-column family region, before bulk > loading, we want to take a write lock on the region. If the compaction takes > too long, the bulk load fails. > Various recipes include: > + Making smaller regions (lame) > + [~victorunique] suggests major compacting just before bulk loading over in > HBASE-10882 as a work around. > Does the compaction need a read lock for that long? Does the bulk load need > a full write lock when multiple column families? Can we fail more gracefully > at least? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983878#comment-14983878 ] ramkrishna.s.vasudevan commented on HBASE-14575: Just trying to understand the scope of this JIRA as it may be related to HBASE-13082. Seeing the patch I can see that the region's reentrant read write lock is passed over through out the compaction logic. In the HRegion#compact() we can see that already we hold the read lock and it is getting released after the compaction is over. In this patch the same lock is again passed in the further flow and acquiring more read locks and releasing it. May be this JIRA is not aimed at this compaction flow whereas some other thread trying to do compaction which does not go through the HRegion.compact()? I may be wrong and missing something. Or may be this patch should remove the read lock in the HRegion.compact() and only do it where ever needed. > Reduce scope of compactions holding region lock > --- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, > 14575-v4.patch, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
