date:20151006

[jira] [Created] (HBASE-14568) Disable hanging test TestStochasticLoadBalancer2

2015-10-06 Thread stack (JIRA)

stack created HBASE-14568:
-

 Summary: Disable hanging test TestStochasticLoadBalancer2
 Key: HBASE-14568
 URL: https://issues.apache.org/jira/browse/HBASE-14568
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


Please address the hanging tests TestStochasticLoadBalancer2 [~ted_yu] You 
introduced it here:

commit ce72ce998f2e9ad23329b48ab8e85912d642fef1
Author: tedyu 
Date:   Mon Aug 10 07:35:19 2015 -0700

HBASE-14200 Separate RegionReplica subtests of TestStochasticLoadBalancer 
into TestStochasticLoadBalancer2

Otherwise, I'd like to just disable it.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14569) Disable hanging test TestNamespaceAuditor

2015-10-06 Thread stack (JIRA)

stack created HBASE-14569:
-

 Summary: Disable hanging test TestNamespaceAuditor
 Key: HBASE-14569
 URL: https://issues.apache.org/jira/browse/HBASE-14569
 Project: HBase
  Issue Type: Sub-task
  Components: test
Reporter: stack
Assignee: Vandana Ayyalasomayajula


The test hung here:  
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//console It hangs 
quite regularly. Any chance of taking a look [~avandana]? Else, I'll just 
disable it so we can get clean builds again. Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14567) Disable hanging test TestMobCompactor and TestMobSweeper

2015-10-06 Thread stack (JIRA)

stack created HBASE-14567:
-

 Summary: Disable hanging test TestMobCompactor and TestMobSweeper
 Key: HBASE-14567
 URL: https://issues.apache.org/jira/browse/HBASE-14567
 Project: HBase
  Issue Type: Sub-task
  Components: test
Reporter: stack
Assignee: Jingcheng Du


These tests hang with some regularity. Please take a look 
[~jingcheng...@intel.com] since I believe they are yours. If not, let me know 
and i'll assign elsewhere. Otherwise, I'll just disable them. Thanks.

They hung here most recently:

 https://builds.apache.org/job/PreCommit-HBASE-Build/15893//console





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14520) Optimize the number of calls for tags creation in bulk load

2015-10-06 Thread Bhupendra Kumar Jain (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946289#comment-14946289
 ] 

Bhupendra Kumar Jain commented on HBASE-14520:
--

Thanks Ted and Anoop for review and commit. 

> Optimize the number of calls for tags creation in bulk load
> ---
>
> Key: HBASE-14520
> URL: https://issues.apache.org/jira/browse/HBASE-14520
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Fix For: 2.0.0
>
> Attachments: HBASE-14520.patch
>
>
> At present, ttl and Visibility expr is one per tsv line i.e. the values and 
> the tags remain same for all the columns present in that line. As per the 
> code, List of tags are created for each cell, Instead of creating new tags 
> for each cell, tags created once for the line can be reused by other cells.  
> Assume 1Million rows and 1000 columns. Currently tags creation will happen 
> for 1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M 
> times. (i.e. one per tsv line). 
> This is applicable in both TsvImporterMapper and TextSortReducer logic. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

2015-10-06 Thread Bhupendra Kumar Jain (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946287#comment-14946287
 ] 

Bhupendra Kumar Jain commented on HBASE-14366:
--

javadoc warnings are not related to the patch.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch, 
> HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14420) Zombie Stomping Session

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946308#comment-14946308
 ] 

Hadoop QA commented on HBASE-14420:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12765316/none_fix.txt
  against master branch at commit 8fd2d6507019f157427c388928d850a65076b9c0.
  ATTACHMENT ID: 12765316

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation, build,
or dev-support patch that doesn't require tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//console

This message is automatically generated.

> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of this one. Am running builds back-to-back 
> on little cluster to turn out the monsters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14549) Simplify scanner stack reset logic

2015-10-06 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946279#comment-14946279
 ] 

Lars Hofhansl commented on HBASE-14549:
---

Looking more at HBASE-5121, I think I do not understand what the issue is. 
We're recreating the scanner heap, how can there be state left over from the 
prior scan. I think as soon as I understand that, I can fix this one and make 
it simpler.


> Simplify scanner stack reset logic
> --
>
> Key: HBASE-14549
> URL: https://issues.apache.org/jira/browse/HBASE-14549
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 14549-0.98.txt
>
>
> Looking at the code, I find that the logic is unnecessarily complex.
> We indicate in updateReaders that the scanner stack needs to be reset. Then 
> almost all store scanner (and derived classes) methods need to check and 
> actually reset the scanner stack.
> Compaction are rare, we should reset the scanner stack in update readers, and 
> hence avoid needing to check in all methods.
> Patch forthcoming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14502) Purge use of jmock and remove as dependency

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946306#comment-14946306
 ] 

stack commented on HBASE-14502:
---

The two test failures are unrelated.

TestProcessBasedCluster does notthing but assert a pom is present.

TestImportExport is trying to find a 0.94 type table but the URL to the 
resource is getting mangled.

Will address the above two in other issues. Meantime let me commit this nice 
cleanup.

> Purge use of jmock and remove as dependency
> ---
>
> Key: HBASE-14502
> URL: https://issues.apache.org/jira/browse/HBASE-14502
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
>Assignee: Gabor Liptak
>  Labels: beginner
> Attachments: HBASE-14502.1.patch
>
>
> jmock is a dependency used by one test only, TestBulkLoad. It looks like you 
> can do anything in mockito that can be done in jmock.  Lets purge it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12615) Document GC conserving guidelines for contributors

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946275#comment-14946275
 ] 

Hudson commented on HBASE-12615:


FAILURE: Integrated in HBase-TRUNK #6880 (See 
[https://builds.apache.org/job/HBase-TRUNK/6880/])
HBASE-12615 Document GC conserving guidelines for contributors (mstanleyjones: 
rev d55f4aee4ff7e952eedbd04565e1b5f7b67379f5)
* src/main/asciidoc/_chapters/developer.adoc


> Document GC conserving guidelines for contributors
> --
>
> Key: HBASE-12615
> URL: https://issues.apache.org/jira/browse/HBASE-12615
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Andrew Purtell
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-12615.patch
>
>
> LinkedIn put up a blog post with a nice concise list of GC conserving 
> techniques we should document for contributors. Additionally, when we're at a 
> point our build supports custom error-prone plugins, we can develop warnings 
> for some of them. 
> Source: 
> http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage
> - Be careful with Iterators
> - Estimate the size of a collection when initializing
> - Defer expression evaluation
> - Compile the regex patterns in advance
> - Cache it if you can
> - String Interns are useful but dangerous
> All good advice and practice that I know we aim for. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14271) Improve Nexus staging instructions

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946274#comment-14946274
 ] 

Hudson commented on HBASE-14271:


FAILURE: Integrated in HBase-TRUNK #6880 (See 
[https://builds.apache.org/job/HBase-TRUNK/6880/])
HBASE-14271 Improve Nexus Staging Instructions  (mstanleyjones: 
rev 8fd2d6507019f157427c388928d850a65076b9c0)
* src/main/asciidoc/_chapters/developer.adoc


> Improve Nexus staging instructions
> --
>
> Key: HBASE-14271
> URL: https://issues.apache.org/jira/browse/HBASE-14271
> Project: HBase
>  Issue Type: Task
>  Components: build, documentation
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14271.patch
>
>
> Refine the Nexus staging instructions a bit. (A promise I made a long time 
> ago.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14502) Purge use of jmock and remove as dependency

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946273#comment-14946273
 ] 

Hadoop QA commented on HBASE-14502:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12765317/HBASE-14502.1.patch
  against master branch at commit 8fd2d6507019f157427c388928d850a65076b9c0.
  ATTACHMENT ID: 12765317

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.util.TestProcessBasedCluster
  org.apache.hadoop.hbase.mapreduce.TestImportExport

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//console

This message is automatically generated.

> Purge use of jmock and remove as dependency
> ---
>
> Key: HBASE-14502
> URL: https://issues.apache.org/jira/browse/HBASE-14502
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
>Assignee: Gabor Liptak
>  Labels: beginner
> Attachments: HBASE-14502.1.patch
>
>
> jmock is a dependency used by one test only, TestBulkLoad. It looks like you 
> can do anything in mockito that can be done in jmock.  Lets purge it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14424) Document that DisabledRegionSplitPolicy blocks manual splits

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946277#comment-14946277
 ] 

Hudson commented on HBASE-14424:


FAILURE: Integrated in HBase-TRUNK #6880 (See 
[https://builds.apache.org/job/HBase-TRUNK/6880/])
HBASE-14424 Document that DisabledRegionSplitPolicy blocks manual splits 
(mstanleyjones: rev 5e60166eac9b607d719ae8a084312e13d65cf074)
* src/main/asciidoc/_chapters/architecture.adoc
* hbase-common/src/main/resources/hbase-default.xml


> Document that DisabledRegionSplitPolicy blocks manual splits
> 
>
> Key: HBASE-14424
> URL: https://issues.apache.org/jira/browse/HBASE-14424
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14424.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12983) HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946276#comment-14946276
 ] 

Hudson commented on HBASE-12983:


FAILURE: Integrated in HBase-TRUNK #6880 (See 
[https://builds.apache.org/job/HBase-TRUNK/6880/])
HBASE-12983 HBase book mentions hadoop.ssl.enabled when it should be 
(mstanleyjones: rev bd9a41a3685ec3f42776b89b756e121c02640b93)
* src/main/asciidoc/_chapters/security.adoc


> HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
> --
>
> Key: HBASE-12983
> URL: https://issues.apache.org/jira/browse/HBASE-12983
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Esteban Gutierrez
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-12983.patch
>
>
> In the HBase book we say the following:
> {quote}
> A default HBase install uses insecure HTTP connections for web UIs for the 
> master and region servers. To enable secure HTTP (HTTPS) connections instead, 
> set *hadoop.ssl.enabled* to true in hbase-site.xml. This does not change the 
> port used by the Web UI. To change the port for the web UI for a given HBase 
> component, configure that port’s setting in hbase-site.xml. These settings 
> are:
> {quote}
> The property should be *hbase.ssl.enabled* instead. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14570) Cleanup hanging TestHBaseFsck

2015-10-06 Thread stack (JIRA)

stack created HBASE-14570:
-

 Summary: Cleanup hanging TestHBaseFsck
 Key: HBASE-14570
 URL: https://issues.apache.org/jira/browse/HBASE-14570
 Project: HBase
  Issue Type: Sub-task
  Components: test
Reporter: stack


This one hangs regularly. Let me at least add timeouts. Looking in log, a bunch 
of tests are potentially hanging tests since they don't see to clean up after 
themselves. Will start watching and just disable likely candidates unless 
someone wants to have a go at fixing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9260) Timestamp Compactions

2015-10-06 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946284#comment-14946284
 ] 

Lars Hofhansl commented on HBASE-9260:
--

Sounds very similar to me.

> Timestamp Compactions
> -
>
> Key: HBASE-9260
> URL: https://issues.apache.org/jira/browse/HBASE-9260
> Project: HBase
>  Issue Type: New Feature
>  Components: Compaction
>Affects Versions: 0.94.10
>Reporter: Adrien Mogenet
>Priority: Minor
>  Labels: features, performance
>
> h1.TSCompactions
> h2.The issue
> One of the biggest issue I currently deal with is compacting big
> stores, i.e. when HBase cluster is 80% full on 4 TB nodes (let say
> with a single big table), compactions might take several hours (from
> 15 to 20 in my case).
> In 'time series' workloads, we could avoid compacting everything
> everytime. Think about OpenTSDB-like systems, or write-heavy,
> TTL based workloads where you want to free space everyday, deleting
> oldest data, and you're not concerned about read latency (i.e. read
> into a single bigger StoreFile).
> > Note: in this draft, I currently consider that we get free space from
> > the TTL behavior only, not really from the Delete operations.
> h2.Proposal and benefits
> For such cases, StoreFiles could be organized and managed in a way
> that would compact:
>   * recent StoreFiles with recent data
>   * oldest StoreFiles that are concerned by TTL eviction
> By the way, it would help when scanning with a timestamp criterion.
> h2.Configuration
>   * {{hbase.hstore.compaction.sortByTS}} (boolean, default=false)
> This indicates if new behavior is enabled or not. Set it to
> {{false}} and compactions will remain the same than current ones.
>   * {{hbase.hstore.compaction.ts.bucketSize}} (integer)
> If `sortByTS` is enabled, tells to HBase the target size of
> buckets. The lower, the more StoreFiles you'll get, but you should
> save more IO's. Higher values will generate less StoreFiles, but
> theses will be bigger and thus compactions could generate more
> IO's.
> h2.Examples
> Here is how a common store could look like after some flushes and
> perhaps some minor compactions:
> {noformat}
>,---, ,---,   ,---,
>|   | |   | ,---, |   |
>|   | |   | |   | |   |
>`---' `---' `---' `---'
> SF1   SF2   SF3   SF4
>\__ __/
>   V
>for all of these Storefiles,
>let say minimum TS is 01/01/2013
>and maximum TS is 31/03/2013
> {noformat}
> Set the bucket size to 1 month, and that's what we have after
> compaction:
> {noformat}
> ,---, ,---,
> |   | |   |
>   ,---, |   | |   |
>   |   | |   | |   |
>   `---' `---' `---'
>SF1   SF2   SF3
>,-,
>|  minimum TS  |  maximum TS  |
>  ,---'
>  | SF1 |  03/03/2013  |  31/03/2013  | most recent, growing
>  | SF2 |  31/01/2013  |  02/03/2013  | old data, "sealed"
>  | SF3 |  01/01/2013  |  30/01/2013  | oldest data, "sealed"
>  '---'
> {noformat}
> h2.StoreFile selection
>   * for minor compactions, current algorithm should already do the
> right job. Pick up `n` eldest files that are small enough, and
> write a bigger file. Remember, TSCompaction are designed for time
> series, so this 'minor selection' should leave "sealed" big old
> files as they are.
>   * for major compactions, when all the StoreFiles have been selected,
> apply the TTL first. StoreFiles that are entirely out of time just
> don't need to be rewritten. They'll be deleted in one time,
> avoiding lots of IO's.
> h2.New issues and trade-offs
>   1. In that case ({{bucketSize=1 month}}), after 1+ year, we'll have lots
>   of StoreFiles (and more generally after `n * bucketSize` seconds) if
>   there is no TTL eviction. In any case, a clever threshold should be
>   implemented to limit the maximum number of StoreFiles.
>   2. If we later add old data that matches timerange of a StoreFile
>   which has already been compacted, this could generate lots of IO's
>   to reconstruct a single StoreFile for this time bucket, perhaps just
>   to merge a few lines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan

2015-10-06 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946298#comment-14946298
 ] 

Lars Hofhansl commented on HBASE-14221:
---

Good find.

Although, isn't there a simpler way to do this, without extending 
KeyValueScanner and adding a new enum of return codes, row state to be 
maintained, etc?

I always thought we can get rid of case #2 above, by piggy packing on the 
comparison of case #1 (and then doing the reset there). Even made a patch for 
that some point; like many things didn't finish it.


> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
>  Issue Type: Sub-task
>  Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14221.patch, HBASE-14221_1.patch, 
> HBASE-14221_1.patch, HBASE-14221_6.patch, withmatchingRowspatch.png, 
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
>   if (ret <= -1) {
> return MatchCode.DONE;
>   } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
>   }
> } else {
>   if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
>   } else if (ret >= 1) {
> return MatchCode.DONE;
>   }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) || 
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
>   this.countPerRow = 0;
>   matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
>   scannerContext.setKeepProgress(true);
>   heap.next(results, scannerContext);
>   scannerContext.setKeepProgress(tmpKeepProgress);
>   nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case.  Was 
> trying to solve this for the MultiCF case but is having lot of cases to 
> solve. But atleast for a single CF case I think these comparison can be 
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a 
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in 
> the heap and so the 3rd comparison can surely be avoided if the 
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that 
> can be avoided if we know that the previous next() call was over due to a new 
> row. Doing all this I found that the compareRows in the profiler which was 
> 19% got reduced to 13%. Initially we can solve for single CF case which can 
> be extended to MultiCF cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14571) Purge TestProcessBasedCluster; it does nothing and then fails

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14571:
--
Attachment: 14571.txt

What I pushed to 0.98+

> Purge TestProcessBasedCluster; it does nothing and then fails
> -
>
> Key: HBASE-14571
> URL: https://issues.apache.org/jira/browse/HBASE-14571
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14571.txt
>
>
> Remove TestProcessBasedCluster. Its an old test... It does nothing currently. 
> It was supposed to do this:
> {code}
>  A basic unit test that spins up a local HBase cluster.
> {code}
> ... but main test got disabled:
>   // DISABLED BECAUSE FLAKEY @Test(timeout=300 * 1000)
>   public void testProcessBasedCluster() throws Exception {
> ... now all that is left is this:
>   @Test
>   public void testHomePath() {
> File pom = new File(HBaseHomePath.getHomePath(), "pom.xml");
> assertTrue(pom.getPath() + " does not exist", pom.exists());
>   }
> ... i.e. assert a pom is present.
> It is flakey too... 
> Failed twice tonight.
> Removing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14571) Purge TestProcessBasedCluster; it does nothing and then fails

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14571.
---
   Resolution: Fixed
Fix Version/s: 0.98.16
   1.1.3
   1.0.3
   1.3.0
   1.2.0
   2.0.0

> Purge TestProcessBasedCluster; it does nothing and then fails
> -
>
> Key: HBASE-14571
> URL: https://issues.apache.org/jira/browse/HBASE-14571
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14571.txt
>
>
> Remove TestProcessBasedCluster. Its an old test... It does nothing currently. 
> It was supposed to do this:
> {code}
>  A basic unit test that spins up a local HBase cluster.
> {code}
> ... but main test got disabled:
>   // DISABLED BECAUSE FLAKEY @Test(timeout=300 * 1000)
>   public void testProcessBasedCluster() throws Exception {
> ... now all that is left is this:
>   @Test
>   public void testHomePath() {
> File pom = new File(HBaseHomePath.getHomePath(), "pom.xml");
> assertTrue(pom.getPath() + " does not exist", pom.exists());
>   }
> ... i.e. assert a pom is present.
> It is flakey too... 
> Failed twice tonight.
> Removing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946252#comment-14946252
 ] 

stack commented on HBASE-14557:
---

bq. It looks like to be related to the recent optimization with NoTagsKeyValue.

So, we back out the above?

> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id : 
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected 
> org.apache.hadoop.hbase.KeyValue, recieved 
> org.apache.hadoop.hbase.NoTagsKeyValue
> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at 
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946251#comment-14946251
 ] 

stack commented on HBASE-14557:
---

bq. ... we cache tagsLen or add a boolean to indicate if tags is present or not

To every KV? Lets not if we can avoid it.

> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id : 
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected 
> org.apache.hadoop.hbase.KeyValue, recieved 
> org.apache.hadoop.hbase.NoTagsKeyValue
> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at 
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14568) Disable hanging test TestStochasticLoadBalancer2

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14568:
--
Assignee: Ted Yu

> Disable hanging test TestStochasticLoadBalancer2
> 
>
> Key: HBASE-14568
> URL: https://issues.apache.org/jira/browse/HBASE-14568
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: Ted Yu
>
> Please address the hanging tests TestStochasticLoadBalancer2 [~ted_yu] You 
> introduced it here:
> commit ce72ce998f2e9ad23329b48ab8e85912d642fef1
> Author: tedyu 
> Date:   Mon Aug 10 07:35:19 2015 -0700
> HBASE-14200 Separate RegionReplica subtests of TestStochasticLoadBalancer 
> into TestStochasticLoadBalancer2
> Otherwise, I'd like to just disable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14571) Purge TestProcessBasedCluster; it does nothing and then fails

2015-10-06 Thread stack (JIRA)

stack created HBASE-14571:
-

 Summary: Purge TestProcessBasedCluster; it does nothing and then 
fails
 Key: HBASE-14571
 URL: https://issues.apache.org/jira/browse/HBASE-14571
 Project: HBase
  Issue Type: Sub-task
  Components: test
Reporter: stack
Assignee: stack


Remove TestProcessBasedCluster. Its an old test... It does nothing currently. 
It was supposed to do this:

{code}
 A basic unit test that spins up a local HBase cluster.
{code}

... but main test got disabled:

  // DISABLED BECAUSE FLAKEY @Test(timeout=300 * 1000)
  public void testProcessBasedCluster() throws Exception {


... now all that is left is this:


  @Test
  public void testHomePath() {
File pom = new File(HBaseHomePath.getHomePath(), "pom.xml");
assertTrue(pom.getPath() + " does not exist", pom.exists());
  }

... i.e. assert a pom is present.

It is flakey too... 

Failed twice tonight.

Removing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945189#comment-14945189
 ] 

stack commented on HBASE-14559:
---

Thanks for noticing lads.

I'm running a little rig here w/ branch-1. The tests in the patch pass with the 
one-liner (upped priorities) where w/o they failed near 100% of the time. At 
this stage in the zombie stomping session, I'm -- ahem -- a little less 
concerned about root cause of a failure and more about just getting stuff going 
again so did not spend much time on why these fixes are needed in branch-1 and 
not on master. 

bq. stack so in these 2 oneliners, you're dropping down the number of high 
priority handlers (to tighten up thread usage?) right?

No sir. HBASE-14290 set the number of handlers down when I noticed tests with 
500 threads running... that failed to run on my local machine because OOME, 
could not create thread.  So, here, I'm upping the handlers on a few tests. I'd 
already done a pass on master -- a few tests there needed more handlers or they 
hung (we need to fix!) -- so was a bit surprised this necessary in branch-1 but 
it looks like you and [~eclark] have identified why.

We should revert HBASE-13635 and HBASE-14322 from branch-1?


> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945153#comment-14945153
 ] 

Hadoop QA commented on HBASE-14366:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12765169/HBASE-14366-0.98.patch
  against 0.98 branch at commit ed4c734b15c83b7f6b8ec1d170abccae9de1b193.
  ATTACHMENT ID: 12765169

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
29 warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15884//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15884//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15884//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15884//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15884//console

This message is automatically generated.

> NPE in case visibility expression is not present in labels table during 
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch, 
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch, 
> HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in 
> labels table during importtsv run. Appropriate exception / message should be 
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at 
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946256#comment-14946256
 ] 

stack commented on HBASE-14479:
---

{code}
kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py  
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//consoleText
Fetching https://builds.apache.org/job/PreCommit-HBASE-Build/15893//consoleText
Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build
Testing patch for HBASE-14479.
Testing patch on branch master.
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck
Hanging test : org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
Hanging test : 
org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
Hanging test : org.apache.hadoop.hbase.mob.mapreduce.TestMobSweeper
Hanging test : org.apache.hadoop.hbase.mob.compactions.TestMobCompactor
Printing Failing tests
{code}

These failed.

I'm just going to disable them all... they fail regularly.  Let me make issues.

> Apply the Leader/Followers pattern to RpcServer's Reader
> 
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2.patch, HBASE-14479-V2.patch, 
> HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but 
> the distribution is just done by round-robin. It is uncertain, especially for 
> long run, whether load is equally divided and resources are used without 
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give 
> priority to low latency (while we just add the requests to queues), and it is 
> possible to reduce throughput of the whole server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946166#comment-14946166
 ] 

Hudson commented on HBASE-14436:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1097 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1097/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev 
6c0f501ece9b7c31f6ce64c85289e55b4f7cc875)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java


> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create 
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
>  Issue Type: Improvement
>  Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following 
> format:
> {code}
>  public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
>  int priority, final Map kvs)
>   throws IOException {
>   ...
>   String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
>   ...
> }
> {code}
> If the 'jarFilePath' is null,  the 'value' will always has the format 
> '|className|priority|'  even if 'kvs' is null, which means no extra arguments 
> for the coprocessor. Then, in the server side, 
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table 
> coprocessors as:
> {code}
>   static List 
> getTableCoprocessorAttrsFromSchema(Configuration conf,
>   HTableDescriptor htd) {
> ...
> try {
>   cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the 
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
>   // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) {  // => cfgSpec will be '|' for the format 
> '|className|priority|'
>   ourConf = new Configuration(false);
>   HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as 
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors, 
> so that the region server will create new Configurations for each region of 
> the table, this will consume a certain number of memory when we have many 
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no 
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in 
> server side which could avoid creating new Configurations for existed such 
> regions after the regions reopened. Discussions and suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12593) Tags and Tag dictionary to work with BB

2015-10-06 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946170#comment-14946170
 ] 

Anoop Sam John commented on HBASE-12593:


This need some change and some what a bigger patch.  Our Tag can work only with 
byte[]. So following the same line of BB backed cell in server side, we have to 
make BB backed Tags as well.. I have a WIP patch.  Missed this some how till 
now.. Let me rebase that patch and put here.

> Tags and Tag dictionary to work with BB
> ---
>
> Key: HBASE-12593
> URL: https://issues.apache.org/jira/browse/HBASE-12593
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
>
> Adding the subtask so that we don't forget it. Came up while reviewing the 
> items required for this parent task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946171#comment-14946171
 ] 

stack commented on HBASE-14451:
---

Killed...


Running org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler
Killed
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test-framework/dev-support/test-patch.sh:
 line 838: 14645 Killed  $MVN clean test 
-Dsurefire.rerunFailingTestsCount=2 -P runAllTests -D${PROJECT_NAME}PatchProcess
We're ok: there is no zombie test

> Move on to htrace-4.0.1 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451.v10.txt, 14451v2.txt, 14451v3.txt, 
> 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14268) Improve KeyLocker

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946172#comment-14946172
 ] 

Hadoop QA commented on HBASE-14268:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12765287/HBASE-14268-V7.patch
  against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
  ATTACHMENT ID: 12765287

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at org.apache.phoenix.pherf.ResultTest.testResult(ResultTest.java:119)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//console

This message is automatically generated.

> Improve KeyLocker
> -
>
> Key: HBASE-14268
> URL: https://issues.apache.org/jira/browse/HBASE-14268
> Project: HBase
>  Issue Type: Improvement
>  Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14268-V5.patch, HBASE-14268-V2.patch, 
> HBASE-14268-V3.patch, HBASE-14268-V4.patch, HBASE-14268-V5.patch, 
> HBASE-14268-V5.patch, HBASE-14268-V6.patch, HBASE-14268-V7.patch, 
> HBASE-14268-V7.patch, HBASE-14268-V7.patch, HBASE-14268-V7.patch, 
> HBASE-14268-V7.patch, HBASE-14268.patch, KeyLockerIncrKeysPerformance.java, 
> KeyLockerPerformance.java, ReferenceTestApp.java
>
>
> 1. In the implementation of {{KeyLocker}} it uses atomic variables inside a 
> synchronized block, which doesn't make sense. Moreover, logic inside the 
> synchronized block is not trivial so that it makes less performance in heavy 
> multi-threaded environment.
> 2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already 
> locked, but it doesn't follow the contract of {{ReentrantLock}} because you 
> are not allowed to freely invoke lock/unlock methods under that contract. 
> That introduces a potential risk; Whenever you see a variable of the type 
> {{RentrantLock}}, you should pay attention to what the included instance is 
> coming from.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9260) Timestamp Compactions

2015-10-06 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946173#comment-14946173
 ] 

Anoop Sam John commented on HBASE-9260:
---

Is this similar to idea in HBASE-14477?

> Timestamp Compactions
> -
>
> Key: HBASE-9260
> URL: https://issues.apache.org/jira/browse/HBASE-9260
> Project: HBase
>  Issue Type: New Feature
>  Components: Compaction
>Affects Versions: 0.94.10
>Reporter: Adrien Mogenet
>Priority: Minor
>  Labels: features, performance
>
> h1.TSCompactions
> h2.The issue
> One of the biggest issue I currently deal with is compacting big
> stores, i.e. when HBase cluster is 80% full on 4 TB nodes (let say
> with a single big table), compactions might take several hours (from
> 15 to 20 in my case).
> In 'time series' workloads, we could avoid compacting everything
> everytime. Think about OpenTSDB-like systems, or write-heavy,
> TTL based workloads where you want to free space everyday, deleting
> oldest data, and you're not concerned about read latency (i.e. read
> into a single bigger StoreFile).
> > Note: in this draft, I currently consider that we get free space from
> > the TTL behavior only, not really from the Delete operations.
> h2.Proposal and benefits
> For such cases, StoreFiles could be organized and managed in a way
> that would compact:
>   * recent StoreFiles with recent data
>   * oldest StoreFiles that are concerned by TTL eviction
> By the way, it would help when scanning with a timestamp criterion.
> h2.Configuration
>   * {{hbase.hstore.compaction.sortByTS}} (boolean, default=false)
> This indicates if new behavior is enabled or not. Set it to
> {{false}} and compactions will remain the same than current ones.
>   * {{hbase.hstore.compaction.ts.bucketSize}} (integer)
> If `sortByTS` is enabled, tells to HBase the target size of
> buckets. The lower, the more StoreFiles you'll get, but you should
> save more IO's. Higher values will generate less StoreFiles, but
> theses will be bigger and thus compactions could generate more
> IO's.
> h2.Examples
> Here is how a common store could look like after some flushes and
> perhaps some minor compactions:
> {noformat}
>,---, ,---,   ,---,
>|   | |   | ,---, |   |
>|   | |   | |   | |   |
>`---' `---' `---' `---'
> SF1   SF2   SF3   SF4
>\__ __/
>   V
>for all of these Storefiles,
>let say minimum TS is 01/01/2013
>and maximum TS is 31/03/2013
> {noformat}
> Set the bucket size to 1 month, and that's what we have after
> compaction:
> {noformat}
> ,---, ,---,
> |   | |   |
>   ,---, |   | |   |
>   |   | |   | |   |
>   `---' `---' `---'
>SF1   SF2   SF3
>,-,
>|  minimum TS  |  maximum TS  |
>  ,---'
>  | SF1 |  03/03/2013  |  31/03/2013  | most recent, growing
>  | SF2 |  31/01/2013  |  02/03/2013  | old data, "sealed"
>  | SF3 |  01/01/2013  |  30/01/2013  | oldest data, "sealed"
>  '---'
> {noformat}
> h2.StoreFile selection
>   * for minor compactions, current algorithm should already do the
> right job. Pick up `n` eldest files that are small enough, and
> write a bigger file. Remember, TSCompaction are designed for time
> series, so this 'minor selection' should leave "sealed" big old
> files as they are.
>   * for major compactions, when all the StoreFiles have been selected,
> apply the TTL first. StoreFiles that are entirely out of time just
> don't need to be rewritten. They'll be deleted in one time,
> avoiding lots of IO's.
> h2.New issues and trade-offs
>   1. In that case ({{bucketSize=1 month}}), after 1+ year, we'll have lots
>   of StoreFiles (and more generally after `n * bucketSize` seconds) if
>   there is no TTL eviction. In any case, a clever threshold should be
>   implemented to limit the maximum number of StoreFiles.
>   2. If we later add old data that matches timerange of a StoreFile
>   which has already been compacted, this could generate lots of IO's
>   to reconstruct a single StoreFile for this time bucket, perhaps just
>   to merge a few lines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14451:
--
Attachment: 14451.v10.txt

> Move on to htrace-4.0.1 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451.v10.txt, 14451.v10.txt, 14451v2.txt, 
> 14451v3.txt, 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 
> 14451v9.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.

2015-10-06 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946174#comment-14946174
 ] 

Anoop Sam John commented on HBASE-14117:


Not able to move it out as subtask and make a top level issue. May be will 
close it and open another improvement issue?

> Check DBEs where fields are being read from Bytebuffers but unused.
> ---
>
> Key: HBASE-14117
> URL: https://issues.apache.org/jira/browse/HBASE-14117
> Project: HBase
>  Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: Jingcheng Du
>
> {code}
> public Cell getFirstKeyCellInBlock(ByteBuff block) {
> block.mark();
> block.position(Bytes.SIZEOF_INT);
> int keyLength = ByteBuff.readCompressedInt(block);
> // TODO : See if we can avoid these reads as the read values are not 
> getting used
> ByteBuff.readCompressedInt(block);
> {code}
> In DBEs many a places we read the integers just to skip them. This JIRA is to 
> see if we can avoid this and rather go position based, as per a review 
> comment in HBASE-12213.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946178#comment-14946178
 ] 

Hadoop QA commented on HBASE-14479:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12765295/HBASE-14479-V2.patch
  against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
  ATTACHMENT ID: 12765295

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//console

This message is automatically generated.

> Apply the Leader/Followers pattern to RpcServer's Reader
> 
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2.patch, HBASE-14479-V2.patch, 
> HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but 
> the distribution is just done by round-robin. It is uncertain, especially for 
> long run, whether load is equally divided and resources are used without 
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give 
> priority to low latency (while we just add the requests to queues), and it is 
> possible to reduce throughput of the whole server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12291) Create Read only buffers where ever possible

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-12291:
---
Issue Type: Bug  (was: Sub-task)
Parent: (was: HBASE-11425)

> Create Read only buffers where ever possible
> 
>
> Key: HBASE-12291
> URL: https://issues.apache.org/jira/browse/HBASE-12291
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>
> This issue is to see if we can really create a Read only buffer in the read 
> path. Later can see if this needs to be BR or our own BB impl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12291) Create Read only buffers where ever possible

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946189#comment-14946189
 ] 

ramkrishna.s.vasudevan commented on HBASE-12291:


[~saint@gmail.com]
Converted this to an issue rather than sub task as suggested.  Your suggestion 
makes sense.

> Create Read only buffers where ever possible
> 
>
> Key: HBASE-12291
> URL: https://issues.apache.org/jira/browse/HBASE-12291
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>
> This issue is to see if we can really create a Read only buffer in the read 
> path. Later can see if this needs to be BR or our own BB impl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946193#comment-14946193
 ] 

ramkrishna.s.vasudevan commented on HBASE-13082:


bq.So, bulk load won't show mid-scan... you have to get to the end? That would 
be fine.
New files added by bulk load during an on going scan won't be included in the 
current scan.  The new files will be included only when a new scan is started.  
Currently during a course of a scan the bulk loaded files can be included too. 
bq.On the patch, can we get more of Lars comments in on what is going on  
Could we get rid of some of these getReaderLocks too... in hstorefile, in 
hstore, etc would be good to not let this stuff out if we can.
Yes we can.  I have a working version of a patch.  Some issues with Merge 
regions and splits.  Working on them. All the locks will be removed that are 
currently in StoreScanner and also the notifyReaders() will also go away 
totally. Just because we are updating the readers during a course of a scan we 
needed all those locks. 

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, gc.png, gc.png, gc.png, hits.png, 
> next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946196#comment-14946196
 ] 

ramkrishna.s.vasudevan commented on HBASE-14117:


Am able to move this to a top level issue and it worked. [~anoop.hbase]

> Check DBEs where fields are being read from Bytebuffers but unused.
> ---
>
> Key: HBASE-14117
> URL: https://issues.apache.org/jira/browse/HBASE-14117
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: Jingcheng Du
>
> {code}
> public Cell getFirstKeyCellInBlock(ByteBuff block) {
> block.mark();
> block.position(Bytes.SIZEOF_INT);
> int keyLength = ByteBuff.readCompressedInt(block);
> // TODO : See if we can avoid these reads as the read values are not 
> getting used
> ByteBuff.readCompressedInt(block);
> {code}
> In DBEs many a places we read the integers just to skip them. This JIRA is to 
> see if we can avoid this and rather go position based, as per a review 
> comment in HBASE-12213.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14117:
---
Issue Type: Bug  (was: Sub-task)
Parent: (was: HBASE-11425)

> Check DBEs where fields are being read from Bytebuffers but unused.
> ---
>
> Key: HBASE-14117
> URL: https://issues.apache.org/jira/browse/HBASE-14117
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: Jingcheng Du
>
> {code}
> public Cell getFirstKeyCellInBlock(ByteBuff block) {
> block.mark();
> block.position(Bytes.SIZEOF_INT);
> int keyLength = ByteBuff.readCompressedInt(block);
> // TODO : See if we can avoid these reads as the read values are not 
> getting used
> ByteBuff.readCompressedInt(block);
> {code}
> In DBEs many a places we read the integers just to skip them. This JIRA is to 
> see if we can avoid this and rather go position based, as per a review 
> comment in HBASE-12213.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks

2015-10-06 Thread Ben Lau (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946197#comment-14946197
 ] 

Ben Lau commented on HBASE-14283:
-

Hey guys, sorry, I should be able to get back to this soon.  Finishing up an 
unrelated project right now.  I didn't know that minor versions in HFiles were 
also non-backwards compatible.  That's one less reason then to make this a 
major version bump.  If anyone has a strong preference for this fix to go into 
a V3.X I can change the patch to use minor version (eg for header size 
calculation) when I have time to do it.  If not I'll leave it as V4 since it's 
a little simpler in the code as a major version bump.  My original intention 
btw if it wasn't clear was that this wouldn't be the only change in a V4, just 
the first change that would go into a V4, whose format/contents is not yet 
meant to be final even when this patch is committed, i.e. V4 would be 
essentially a WIP with more changes suggested and implemented in other tickets 
and eventually released in HBase 2.0.

[~anoop.hbase] I'm down for committing a short-term read-the-header-always fix 
for now and then discussing the longer term solution second.  Which branches do 
you want the patch for?

> Reverse scan doesn’t work with HFile inline index/bloom blocks
> --
>
> Key: HBASE-14283
> URL: https://issues.apache.org/jira/browse/HBASE-14283
> Project: HBase
>  Issue Type: Bug
>Reporter: Ben Lau
>Assignee: Ben Lau
> Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, 
> hfile-seek-before.patch
>
>
> Reverse scans do not work if an HFile contains inline bloom blocks or leaf 
> level index blocks.  The reason is because the seekBefore() call calculates 
> the previous data block’s size by assuming data blocks are contiguous which 
> is not the case in HFile V2 and beyond.
> Attached is a first cut patch (targeting 
> bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes:
> (1) a unit test which exposes the bug and demonstrates failures for both 
> inline bloom blocks and inline index blocks
> (2) a proposed fix for inline index blocks that does not require a new HFile 
> version change, but is only performant for 1 and 2-level indexes and not 3+.  
> 3+ requires an HFile format update for optimal performance.
> This patch does not fix the bloom filter blocks bug.  But the fix should be 
> similar to the case of inline index blocks.  The reason I haven’t made the 
> change yet is I want to confirm that you guys would be fine with me revising 
> the HFile.Reader interface.
> Specifically, these 2 functions (getGeneralBloomFilterMetadata and 
> getDeleteBloomFilterMetadata) need to return the BloomFilter.  Right now the 
> HFileReader class doesn’t have a reference to the bloom filters (and hence 
> their indices) and only constructs the IO streams and hence has no way to 
> know where the bloom blocks are in the HFile.  It seems that the HFile.Reader 
> bloom method comments state that they “know nothing about how that metadata 
> is structured” but I do not know if that is a requirement of the abstraction 
> (why?) or just an incidental current property. 
> We would like to do 3 things with community approval:
> (1) Update the HFile.Reader interface and implementation to contain and 
> return BloomFilters directly rather than unstructured IO streams
> (2) Merge the fixes for index blocks and bloom blocks into open source
> (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ 
> field in the block header in the next HFile version, so that seekBefore() 
> calls can not only be correct but performant in all cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946198#comment-14946198
 ] 

stack commented on HBASE-13082:
---

bq. New files added by bulk load during an on going scan won't be included in 
the current scan. The new files will be included only when a new scan is 
started. Currently during a course of a scan the bulk loaded files can be 
included too.

That makes more sense than our current implementation IMO.

bq. Yes we can. I have a working version of a patch. Some issues with Merge 
regions and splits. Working on them. All the locks will be removed that are 
currently in StoreScanner and also the notifyReaders() will also go away 
totally. Just because we are updating the readers during a course of a scan we 
needed all those locks.

Sweet.

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, gc.png, gc.png, gc.png, hits.png, 
> next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14522) Document and/or disable hsperfdata GC pauses

2015-10-06 Thread Vladimir Rodionov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946211#comment-14946211
 ] 

Vladimir Rodionov commented on HBASE-14522:
---

The work to do is HBase book update. I posted the link to a corresponding 
CASSANDRA JIRA, which discusses all the issues, related to this JVM setting. 
Several JVM performance tools won't work if the flag is enabled, all of them 
are listed in the CASSANDRA JIRA. This JIRA should be committed and new follow 
up documentation JIRA should be created.

> Document and/or disable hsperfdata GC pauses
> 
>
> Key: HBASE-14522
> URL: https://issues.apache.org/jira/browse/HBASE-14522
> Project: HBase
>  Issue Type: Task
>  Components: documentation, Performance
>Reporter: Nick Dimiduk
>Assignee: Lars Francke
> Attachments: HBASE-14522.patch
>
>
> {quote}
> The JVM by default exports statistics by mmap-ing a file in /tmp 
> (hsperfdata). On Linux, modifying a memory mapped file can block until disk 
> I/O completes, which can be hundreds of milliseconds. Since the JVM modifies 
> these statistics during garbage collection and safepoints, this causes pauses 
> that are hundreds of milliseconds long.
> {quote}
> Via [JVM mmap pause|http://www.evanjones.ca/jvm-mmap-pause.html].
> We should add {{-XX:+PerfDisableSharedMem}} to our default options as was 
> apparently done in CASSANDRA-9242 and/or document the presence of this bug so 
> operators know to use tmpfs.
> Hat-tip [~vrodionov]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14458) AsyncRpcClient#createRpcChannel() should check and remove dead channel before creating new one to same server

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946218#comment-14946218
 ] 

Hadoop QA commented on HBASE-14458:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12765299/HBASE-14458%20%281%29.patch
  against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
  ATTACHMENT ID: 12765299

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:red}-1 findbugs{color}.  The patch appears to cause Findbugs 
(version 2.0.3) to fail.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.util.TestProcessBasedCluster
  org.apache.hadoop.hbase.mapreduce.TestImportExport

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15894//testReport/
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15894//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15894//console

This message is automatically generated.

> AsyncRpcClient#createRpcChannel() should check and remove dead channel before 
> creating new one to same server
> -
>
> Key: HBASE-14458
> URL: https://issues.apache.org/jira/browse/HBASE-14458
> Project: HBase
>  Issue Type: Bug
>  Components: IPC/RPC
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14458 (1).patch, HBASE-14458.patch, 
> HBASE-14458.patch
>
>
> I have notice this issue while testing master branch in distributed mode. 
> Reproduction steps:
> 1. Write some data with hbase ltt 
> 2. While ltt is writing execute $graceful_stop.sh --restart --reload [rs] 
> 3. Wait until script start to reload regions to restarted server. In that 
> moment ltt will stop writing and eventually fail. 
> After some digging i have notice that while ltt is working correctly there is 
> single connection per regionserver (lsof for single connection, 27109 is  ltt 
> PID )
> {code}
> java  27109   hbase  143u210579579  0t0TCP 
> hnode1:40423->hnode5:16020 (ESTABLISHED)
> {code}  
> and when in this example hnode5 server is restarted and script starts to 
> reload regions on this server ltt start creating thousands of new tcp 
> connections to this server:
> {code}
> java  27109   hbase *623u  210674415  0t0TCP 
> hnode1:52948->hnode5:16020 (ESTABLISHED)
> java  27109   hbase *624u   210674416  0t0TCP 
> hnode1:52949->hnode5:16020 (ESTABLISHED)
> java  27109   hbase *625u   210674417  0t0TCP 
> hnode1:52950->hnode5:16020 (ESTABLISHED)
> java  27109   hbase *627u   210674419  0t0TCP 
> hnode1:52952->hnode5:16020 (ESTABLISHED)
> java  27109   hbase *628u   210674420  0t0TCP 
> hnode1:52953->hnode5:16020 (ESTABLISHED)
> java  27109   hbase *633u   210674425  0t0TCP 
> hnode1:52958->hnode5:16020 (ESTABLISHED)
> ...
> {code}
> So here is what happened based on some additional logging and debugging:
> - AsyncRpcClient never detected that regionserver is restarted because 
> regions were moved and there was no write/read requests to this server and  
> there is no some sort of heart-bit mechanism implemented

[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946220#comment-14946220
 ] 

ramkrishna.s.vasudevan commented on HBASE-14557:


bq.Then we will have problem setting the OutputValueClass to either of the two 
classes.
Ok.  The problem is that WALCellCodec using KeyValueDecoder that does not write 
tags.  So it will always be NoTagsKeyValue and NoTagsKeyValue is a private 
interface now.
I think this is a critical issue.

> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id : 
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected 
> org.apache.hadoop.hbase.KeyValue, recieved 
> org.apache.hadoop.hbase.NoTagsKeyValue
> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at 
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946223#comment-14946223
 ] 

ramkrishna.s.vasudevan commented on HBASE-14557:


Or we should remove NoTagsKeyValue and always go with KeyValue provided we 
cache tagsLen or add a boolean to indicate if tags is present or not. If we 
don't do it then caching the taglen is a costly operation.  But this will have 
concerns on increasing the heap size occupied by per KV and little more GC (not 
sure on the impact of GC though). 

> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id : 
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected 
> org.apache.hadoop.hbase.KeyValue, recieved 
> org.apache.hadoop.hbase.NoTagsKeyValue
> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at 
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945623#comment-14945623
 ] 

Mikhail Antonov commented on HBASE-14559:
-

Thanks for clarifying [~stack]

regarding tests - I remember we upped the number of high priority handler used 
in all tests (and I admit to have suggested/supported that :), as after finding 
several tests where default number of thread wasn't enough after changes made 
in HBase-13375 bumping the default number of threads looked easy thing to do). 
Should we also lower DEFAULT_REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT back to 
10 as default (it's 20 now on master, as I see, and set higher only in selected 
tests?)

Regarding removing completely - I guess that's up to the judgement of folks 
running big clusters. The way we treat admin user requests _seems_ logical to 
me, but I definitely don't want to argue with production observations using 
logical conclusions :)

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking

2015-10-06 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14497:
---
Attachment: 14497-branch-1-v6.patch

> Reverse Scan threw StackOverflow caused by readPt checking
> --
>
> Key: HBASE-14497
> URL: https://issues.apache.org/jira/browse/HBASE-14497
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 0.98.14, 1.3.0
>Reporter: Yerui Sun
>Assignee: Yerui Sun
> Fix For: 2.0.0
>
> Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, 
> HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, 
> HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, 
> HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, 
> HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, 
> HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, 
> HBASE-14497-master.patch
>
>
> I met stack overflow error in StoreFileScanner.seekToPreviousRow using 
> reversed scan. I searched and founded HBASE-14155, but it seems to be a 
> different reason.
> The seekToPreviousRow will fetch the row which closest before, and compare 
> mvcc to the readPt, which acquired when scanner created. If the row's mvcc is 
> bigger than readPt, an recursive call of seekToPreviousRow will invoked, to 
> find the next closest before row.
> Considering we created a scanner for reversed scan, and some data with 
> smaller rows was written and flushed, before calling scanner next. When 
> seekToPreviousRow was invoked, it would call itself recursively, until all 
> rows which written after scanner created were iterated. The depth of 
> recursive calling stack depends on the count of rows, the stack overflow 
> error will be threw if the count of rows is large, like 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14386) Reset MutableHistogram's min/max/sum after snapshot

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945618#comment-14945618
 ] 

stack commented on HBASE-14386:
---

Any numbers? Do we need to lock? In the past a mistake in metrics making cost 
tens of percents of read latency. Thanks.

> Reset MutableHistogram's min/max/sum after snapshot
> ---
>
> Key: HBASE-14386
> URL: https://issues.apache.org/jira/browse/HBASE-14386
> Project: HBase
>  Issue Type: Bug
>Reporter: binlijin
>Assignee: Oliver
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14386.patch
>
>
> Current MutableHistogram do not reset min/max/sum after snapshot, so we 
> affect by historical data. For example when i monitor the QueueCallTime_mean, 
> i see one host's QueueCallTime_mean metric is high, but when i trace the 
> host's regionserver log i see the QueueCallTime_mean has been lower, but the 
> metric is still high.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945602#comment-14945602
 ] 

stack commented on HBASE-14559:
---

These fixes here made it so branch-1 now passes on my internal rig.

I can't speak to whether we should remove HBASE-13375 completely. It would seem 
to explain why some of the tweaks here were necessary in branch-1 but not in 
master -- that  helps.

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14432) Procedure V2 - enforce ACL on procedure admin tasks

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945514#comment-14945514
 ] 

Hudson commented on HBASE-14432:


FAILURE: Integrated in HBase-1.3 #238 (See 
[https://builds.apache.org/job/HBase-1.3/238/])
HBASE-14432 Procedure V2 - enforce ACL on procedure admin tasks (Stephen 
(syuanjiangdev: rev a6d90bcc97ea6e00d2d75381db0b598ab6c71026)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteColumnFamilyProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteNamespaceProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProcedureProtos.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* hbase-server/pom.xml
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyColumnFamilyProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
* hbase-server/src/test/protobuf/TestProcedure.proto
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AddColumnFamilyProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java


> Procedure V2 - enforce ACL on procedure admin tasks
> ---
>
> Key: HBASE-14432
> URL: https://issues.apache.org/jira/browse/HBASE-14432
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>  Labels: security
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14432.v1-branch-1.patch, 
> HBASE-14432.v1-master.patch
>
>
> In the Procedure class, the owner field is never set. We need to set it so 
> that we can enforce ACLs on admin tasks such as whether a user has privilege 
> to abort a procedure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944558#comment-14944558
 ] 

Mikhail Antonov commented on HBASE-14559:
-

Hm? I didn't know it was reverted on master. Perhaps I missed that jira?

HBASE-13375 /  HBASE-13351? These 2 have to do with rpc priority.

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue

2015-10-06 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944562#comment-14944562
 ] 

ramkrishna.s.vasudevan commented on HBASE-14557:


So is this the problem?
{code}
job.setMapOutputValueClass(KeyValue.class);
{code}

> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id : 
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected 
> org.apache.hadoop.hbase.KeyValue, recieved 
> org.apache.hadoop.hbase.NoTagsKeyValue
> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at 
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944564#comment-14944564
 ] 

Elliott Clark commented on HBASE-14559:
---

HBASE-13635
HBASE-14322

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14553) [0.98] Per column family DFS Replication doesn't work against hadoop-1

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944563#comment-14944563
 ] 

Hudson commented on HBASE-14553:


SUCCESS: Integrated in HBase-0.98 #1144 (See 
[https://builds.apache.org/job/HBase-0.98/1144/])
HBASE-14553 [0.98] Per column family DFS Replication doesn't work (tedyu: rev 
20ebd5b653c8534faf4ddb7d339637c1f0e43f04)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


> [0.98] Per column family DFS Replication doesn't work against hadoop-1
> --
>
> Key: HBASE-14553
> URL: https://issues.apache.org/jira/browse/HBASE-14553
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.16
>
> Attachments: 14553-0.98-v1.txt
>
>
> TestAdmin1#testHFileReplication has been failing against hadoop-1 since (at 
> least) this build:
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1082/testReport/org.apache.hadoop.hbase.client/TestAdmin1/testHFileReplication/
> The cause is that the following method is absent in hadoop-1 (in 
> FSUtils#create()):
> {code}
>   return (FSDataOutputStream) 
> (DistributedFileSystem.class.getDeclaredMethod("create",
> Path.class, FsPermission.class, boolean.class, int.class, 
> short.class, long.class,
> Progressable.class, InetSocketAddress[].class).invoke(backingFs, 
> path, perm, true,
> {code}
> The following fallback doesn't take replication parameter:
> {code}
> return create(fs, path, perm, true);
> {code}
> leading to DFS replication being 3 for the column family which should have 
> DFS replication of 1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14555) Deadlock in MVCC branch-1.2 toString()

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944556#comment-14944556
 ] 

Hudson commented on HBASE-14555:


FAILURE: Integrated in HBase-1.3 #236 (See 
[https://builds.apache.org/job/HBase-1.3/236/])
HBASE-14555 Fix Deadlock in MVCC branch-1.2 toString() (eclark: rev 
12fd46cdcadf7426417397fab46e8835bd848cf7)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java


> Deadlock in MVCC branch-1.2 toString()
> --
>
> Key: HBASE-14555
> URL: https://issues.apache.org/jira/browse/HBASE-14555
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0
>
> Attachments: HBASE-14555-v1.patch, HBASE-14555.patch
>
>
> Just saw this in an IT test.
> {code}
> Thread 75 (PriorityRpcServer.handler=3,queue=1,port=16020):
>   State: BLOCKED
>   Blocked count: 691635
>   Waited count: 1557446
>   Blocked on java.util.LinkedList@32b53d9e
>   Blocked by 81 (PriorityRpcServer.handler=9,queue=1,port=16020)
>   Stack:
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.toString(MultiVersionConcurrencyControl.java:234)
> java.lang.String.valueOf(String.java:2994)
> java.lang.StringBuilder.append(StringBuilder.java:131)
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:209)
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:144)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3191)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2837)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2779)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:697)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:659)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2047)
> 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594)
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2120)
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> java.lang.Thread.run(Thread.java:745)
> Thread 81 (PriorityRpcServer.handler=9,queue=1,port=16020):
>   State: BLOCKED
>   Blocked count: 691858
>   Waited count: 1558138
>   Blocked on java.lang.Object@2a5e9ae8
>   Blocked by 75 (PriorityRpcServer.handler=3,queue=1,port=16020)
>   Stack:
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete(MultiVersionConcurrencyControl.java:191)
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:143)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3191)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2837)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2779)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:697)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:659)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2047)
> 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594)
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2120)
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> java.lang.Thread.run(Thread.java:745)
> Thread 80 (PriorityRpcServer.handler=8,queue=0,port=16020):
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14524) Short-circuit comparison of rows in CellComparator

2015-10-06 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944568#comment-14944568
 ] 

Anoop Sam John commented on HBASE-14524:


bq. For Scans and Gets these objects can sometimes be exactly the same object.
It will be great if u can analyze it and say how often we do this. Who is 
calling it. Then we can see why that really happens.

> Short-circuit comparison of rows in CellComparator
> --
>
> Key: HBASE-14524
> URL: https://issues.apache.org/jira/browse/HBASE-14524
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HBASE-14524.patch, HBASE-14524.patch
>
>
> {{CellComparator#compareRows}} compares two rows. For Scans and Gets these 
> objects can sometimes be exactly the same object.
> I see this (without fully understanding everything that's going on) for the 
> first cell in each row.
> In these circumstances I think it makes sense to short-circuit the comparison 
> (not do a byte comparison and class cast checks) and bail out early if the 
> objects are identical.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944566#comment-14944566
 ] 

Mikhail Antonov commented on HBASE-14559:
-

Oh i see, thanks for pointer.

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14555) Deadlock in MVCC branch-1.2 toString()

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944570#comment-14944570
 ] 

Hudson commented on HBASE-14555:


FAILURE: Integrated in HBase-TRUNK #6876 (See 
[https://builds.apache.org/job/HBase-TRUNK/6876/])
HBASE-14555 Fix Deadlock in MVCC branch-1.2 toString() (eclark: rev 
39042b6a07e3e206b36f3f8ca69ea4a5b2faa4e4)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java


> Deadlock in MVCC branch-1.2 toString()
> --
>
> Key: HBASE-14555
> URL: https://issues.apache.org/jira/browse/HBASE-14555
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0
>
> Attachments: HBASE-14555-v1.patch, HBASE-14555.patch
>
>
> Just saw this in an IT test.
> {code}
> Thread 75 (PriorityRpcServer.handler=3,queue=1,port=16020):
>   State: BLOCKED
>   Blocked count: 691635
>   Waited count: 1557446
>   Blocked on java.util.LinkedList@32b53d9e
>   Blocked by 81 (PriorityRpcServer.handler=9,queue=1,port=16020)
>   Stack:
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.toString(MultiVersionConcurrencyControl.java:234)
> java.lang.String.valueOf(String.java:2994)
> java.lang.StringBuilder.append(StringBuilder.java:131)
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:209)
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:144)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3191)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2837)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2779)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:697)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:659)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2047)
> 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594)
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2120)
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> java.lang.Thread.run(Thread.java:745)
> Thread 81 (PriorityRpcServer.handler=9,queue=1,port=16020):
>   State: BLOCKED
>   Blocked count: 691858
>   Waited count: 1558138
>   Blocked on java.lang.Object@2a5e9ae8
>   Blocked by 75 (PriorityRpcServer.handler=3,queue=1,port=16020)
>   Stack:
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete(MultiVersionConcurrencyControl.java:191)
> 
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:143)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3191)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2837)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2779)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:697)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:659)
> 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2047)
> 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594)
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2120)
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> java.lang.Thread.run(Thread.java:745)
> Thread 80 (PriorityRpcServer.handler=8,queue=0,port=16020):
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14432) Procedure V2 - enforce ACL on procedure admin tasks

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944571#comment-14944571
 ] 

Hudson commented on HBASE-14432:


FAILURE: Integrated in HBase-TRUNK #6876 (See 
[https://builds.apache.org/job/HBase-TRUNK/6876/])
HBASE-14432 Procedure V2 - enforce ACL on procedure admin tasks (syuanjiangdev: 
rev ed4c734b15c83b7f6b8ec1d170abccae9de1b193)
* hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProcedureProtos.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AddColumnFamilyProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteColumnFamilyProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteNamespaceProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyColumnFamilyProcedure.java
* hbase-server/src/test/protobuf/TestProcedure.proto
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
* hbase-server/pom.xml
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


> Procedure V2 - enforce ACL on procedure admin tasks
> ---
>
> Key: HBASE-14432
> URL: https://issues.apache.org/jira/browse/HBASE-14432
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>  Labels: security
> Attachments: HBASE-14432.v1-branch-1.patch, 
> HBASE-14432.v1-master.patch
>
>
> In the Procedure class, the owner field is never set. We need to set it so 
> that we can enforce ACLs on admin tasks such as whether a user has privilege 
> to abort a procedure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14525) Append and increment operation throws NullPointerException on non-existing column families.

2015-10-06 Thread Abhishek Kumar (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kumar updated HBASE-14525:
---
Attachment: HBASE-14525-V1.patch

yes, put ops fails for invalid column family, thanks for pointing this out, 
updated patch attached for review. 

> Append and increment operation throws NullPointerException on non-existing 
> column families.
> ---
>
> Key: HBASE-14525
> URL: https://issues.apache.org/jira/browse/HBASE-14525
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 2.0.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14525-V1.patch, HBASE-14525.patch
>
>
> When performing append operation on non-existing column families, 
> NullPointerException is thrown in hbase shell as shown below:
> {noformat}
> hbase(main):007:0> append 't1', 'r1', 'none:c1', '123'
> ERROR: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HRegion.doGet(HRegion.java:6987)
> at org.apache.hadoop.hbase.regionserver.HRegion.append(HRegion.java:7048)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.append(RSRpcServices.java:580)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2206)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32452)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2133)
> ... 4 more
> {noformat}
> This seems to be caused by absence of check for valid family names as done in 
> other operations like 'Put' in HRegion.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13770) Programmatic JAAS configuration option for secure zookeeper may be broken

2015-10-06 Thread Maddineni Sukumar (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944603#comment-14944603
 ] 

Maddineni Sukumar commented on HBASE-13770:
---

Thanks [~apurtell]  for reviewing and pushing this. 

> Programmatic JAAS configuration option for secure zookeeper may be broken
> -
>
> Key: HBASE-13770
> URL: https://issues.apache.org/jira/browse/HBASE-13770
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: HBASE-13770-0.98.patch, HBASE-13770-0.98.patch, 
> HBASE-13770-v1.patch, HBASE-13770-v2-0.98.patch, HBASE-13770-v2.patch, 
> HBASE-13770-v3-0.98.patch, HBASE-13770-v4-0.98.patch, 
> HBASE-13770-v4-master.patch
>
>
> While verifying the patch fix for HBASE-13768 we were unable to successfully 
> test the programmatic JAAS configuration option for secure ZooKeeper 
> integration. Unclear if that was due to a bug or incorrect test configuration.
> Update the security section of the online book with clear instructions for 
> setting up the programmatic JAAS configuration option for secure ZooKeeper 
> integration.
> Verify it works.
> Fix as necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944604#comment-14944604
 ] 

Mikhail Antonov commented on HBASE-14559:
-

I think there're several things here (or maybe I'm missing something). 

Conceptually it looks logical to me that on large cluster overall 
responsiveness to admin commands would be improved by serving admin commands 
from separate threadpool (is that actually right? if not, we can revert it 
completely?). There were some deadlock-type bugs (these 2 referenced by 
[~eclark]), but that should be fixed? In theory having yet one more threadpool 
to separate admin commands from things like meta lookups might help prevent it, 
but looks overkill.

For the tests, these features are revealed in corner case, as in minicluster 
everything is running as admin, so admin threadpool is overloaded. On current 
master as I'm seeing in HConstants:

bq. public static final int DEFAULT_REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT = 
20;

as was set in HBASE-13351.

[~stack] so in these 2 oneliners, you're dropping down the number of high 
priority handlers (to tighten up thread usage?) right?




> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944553#comment-14944553
 ] 

Elliott Clark commented on HBASE-14559:
---

bq.Is that regression from jira when we made superuser's requests go thru 
high-priority rpc thread pool?
That already had to be backed out a while ago for master. I wonder if we should 
just back it out completely ?

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14347) Add a switch to DynamicClassLoader to disable it

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944573#comment-14944573
 ] 

Hudson commented on HBASE-14347:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1096 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1096/])
HBASE-14347 Add a switch to DynamicClassLoader to disable it (huaxiang 
(matteo.bertozzi: rev fdaecc89f9932f73609ce81e740a6a0e8c839979)
* 
hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java


> Add a switch to DynamicClassLoader to disable it
> 
>
> Key: HBASE-14347
> URL: https://issues.apache.org/jira/browse/HBASE-14347
> Project: HBase
>  Issue Type: Bug
>  Components: Client, defaults, regionserver
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3
>Reporter: Esteban Gutierrez
>Assignee: huaxiang sun
> Fix For: 2.0.0, 1.2.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14347-v001.patch
>
>
> Since HBASE-1936 we have the option to load jars dynamically by default from 
> HDFS or the local filesystem, however hbase.dynamic.jars.dir points to a 
> directory that could be world writable it potentially opens a security 
> problem in both the client side and the RS. We should consider to have a 
> switch to enable or disable this option and it should be off by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14553) [0.98] Per column family DFS Replication doesn't work against hadoop-1

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944572#comment-14944572
 ] 

Hudson commented on HBASE-14553:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1096 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1096/])
HBASE-14553 [0.98] Per column family DFS Replication doesn't work (tedyu: rev 
20ebd5b653c8534faf4ddb7d339637c1f0e43f04)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


> [0.98] Per column family DFS Replication doesn't work against hadoop-1
> --
>
> Key: HBASE-14553
> URL: https://issues.apache.org/jira/browse/HBASE-14553
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.16
>
> Attachments: 14553-0.98-v1.txt
>
>
> TestAdmin1#testHFileReplication has been failing against hadoop-1 since (at 
> least) this build:
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1082/testReport/org.apache.hadoop.hbase.client/TestAdmin1/testHFileReplication/
> The cause is that the following method is absent in hadoop-1 (in 
> FSUtils#create()):
> {code}
>   return (FSDataOutputStream) 
> (DistributedFileSystem.class.getDeclaredMethod("create",
> Path.class, FsPermission.class, boolean.class, int.class, 
> short.class, long.class,
> Progressable.class, InetSocketAddress[].class).invoke(backingFs, 
> path, perm, true,
> {code}
> The following fallback doesn't take replication parameter:
> {code}
> return create(fs, path, perm, true);
> {code}
> leading to DFS replication being 3 for the column family which should have 
> DFS replication of 1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14525) Append and increment operation throws NullPointerException on non-existing column families.

2015-10-06 Thread Abhishek Kumar (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kumar updated HBASE-14525:
---
Summary: Append and increment operation throws NullPointerException on 
non-existing column families.  (was: Append and increment operation throws 
NullPointerException when appending on non-existing column families.)

> Append and increment operation throws NullPointerException on non-existing 
> column families.
> ---
>
> Key: HBASE-14525
> URL: https://issues.apache.org/jira/browse/HBASE-14525
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 2.0.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14525.patch
>
>
> When performing append operation on non-existing column families, 
> NullPointerException is thrown in hbase shell as shown below:
> {noformat}
> hbase(main):007:0> append 't1', 'r1', 'none:c1', '123'
> ERROR: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.HRegion.doGet(HRegion.java:6987)
> at org.apache.hadoop.hbase.regionserver.HRegion.append(HRegion.java:7048)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.append(RSRpcServices.java:580)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2206)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32452)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2133)
> ... 4 more
> {noformat}
> This seems to be caused by absence of check for valid family names as done in 
> other operations like 'Put' in HRegion.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-06 Thread Maddineni Sukumar (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maddineni Sukumar reassigned HBASE-13773:
-

Assignee: Maddineni Sukumar

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945634#comment-14945634
 ] 

stack commented on HBASE-14559:
---

bq. Should we also lower DEFAULT_REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT back 
to 10 as default (it's 20 now on master, as I see, and set higher only in 
selected tests?)

I'm just looking at tests at the moment. I saw priority handlers set to 40 in a 
few instances which seemed excessive. Other tests with many regions had reams 
of handlers just sitting there doing nothing clouding thread dumps where i was 
trying to figure why the test was hung... 

bq. The way we treat admin user requests seems logical to me, but I definitely 
don't want to argue with production observations using logical conclusions

I'm with you. Lets get other input.

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14420) Zombie Stomping Session

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945672#comment-14945672
 ] 

stack commented on HBASE-14420:
---

Going over the last 40 patch builds:

TestReplicationShell hangs three times. Was added to master only. HBASE-13084 
adds it by running all shell commands again plus the new 
replication_admin_test.rb command. I'm going to disable it for now.  
HBASE-14561.

TestHFileOutputFormat2 failed 5 times in last 40 runs. I spent time on it 
yesterday. Seems to be a reliance on test order but was having networking 
issues which complicated my being able to do diagnosis  It seems like an 
ambitious amount of work to get done in a unit test:

{code}
 * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
 * Sets up and runs a mapreduce job that writes hfile output.
 * Creates a few inner classes to implement splits and an inputformat that
 * emits keys and values like those of {@link PerformanceEvaluation}.
{code}

Was added a good while ago, here:

commit e4f8a7419fb4bd0102eaf91e9747de6261e0b5c5
Author: jxiang 
Date:   Fri Feb 21 20:39:21 2014 +

HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat

git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1570702 
13f79535-47bb-0310-9956-ffa450edef68

I'm just going to disable it until someone wants to work on it.

Here is the list of all test failures and their counts:

   2 Hanging test : org.apache.hadoop.hbase.TestNodeHealthCheckChore
   1 Hanging test : org.apache.hadoop.hbase.TestPartialResultsFromClientSide
   2 Hanging test : org.apache.hadoop.hbase.client.TestFromClientSide
   1 Hanging test : 
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
   1 Hanging test : org.apache.hadoop.hbase.client.TestReplicasClient
   3 Hanging test : org.apache.hadoop.hbase.client.TestReplicationShell
   1 Hanging test : org.apache.hadoop.hbase.constraint.TestConstraint
   1 Hanging test : org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd
   1 Hanging test : org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
   2 Hanging test : org.apache.hadoop.hbase.mapreduce.TestCopyTable
   1 Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
   5 Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
   1 Hanging test : org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
   1 Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableInputFormat
   1 Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
   1 Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
   1 Hanging test : org.apache.hadoop.hbase.replication.TestMasterReplication
   1 Hanging test : 
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
   1 Hanging test : 
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
   1 Hanging test : 
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster
   1 Hanging test : 
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager
   1 Hanging test : org.apache.hadoop.hbase.security.access.TestAccessController
   1 Hanging test : org.apache.hadoop.hbase.security.access.TestCellACLs
   1 Hanging test : 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString
   1 Hanging test : 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
   1 Hanging test : 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay
   1 Hanging test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot
   1 Hanging test : org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot
   1 Hanging test : 
org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient
   1 Hanging test : org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot
   1 Hanging test : org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot




> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of

[jira] [Created] (HBASE-14563) Disable zombie TestHFileOutputFormat2

2015-10-06 Thread stack (JIRA)

stack created HBASE-14563:
-

 Summary: Disable zombie TestHFileOutputFormat2
 Key: HBASE-14563
 URL: https://issues.apache.org/jira/browse/HBASE-14563
 Project: HBase
  Issue Type: Sub-task
Reporter: stack
Assignee: stack


Disabling until someone has a chance to look at it.

I watched it in jvisualvm a while. Its starting and stopping clusters multiple 
times and then running mr jobs. Needs a rewrite at least and some shrinking of 
scope on what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14564) Fix and reenable TestHFileOutputFormat2

2015-10-06 Thread stack (JIRA)

stack created HBASE-14564:
-

 Summary: Fix and reenable TestHFileOutputFormat2
 Key: HBASE-14564
 URL: https://issues.apache.org/jira/browse/HBASE-14564
 Project: HBase
  Issue Type: Bug
Reporter: stack


Was disabled as part of the zombie stomping session over in HBASE-14420. Test 
needs a rewrite and/or being split up. Scope of the test needs to be shrunk and 
made more targeted. Currently it does everything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945723#comment-14945723
 ] 

stack commented on HBASE-14563:
---

>From parent issue:
{code}
TestHFileOutputFormat2 failed 5 times in last 40 runs. I spent time on it 
yesterday. Seems to be a reliance on test order but was having networking 
issues which complicated my being able to do diagnosis It seems like an 
ambitious amount of work to get done in a unit test:
 * Simple test for {@link CellSortReducer} and {@link HFileOutputFormat2}.
 * Sets up and runs a mapreduce job that writes hfile output.
 * Creates a few inner classes to implement splits and an inputformat that
 * emits keys and values like those of {@link PerformanceEvaluation}.
Was added a good while ago, here:
commit e4f8a7419fb4bd0102eaf91e9747de6261e0b5c5
Author: jxiang 
Date: Fri Feb 21 20:39:21 2014 +
HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1570702 
13f79535-47bb-0310-9956-ffa450edef68
{code}

The test stops and starts clusters a few times and then runs MR jobs. Needs 
shrinking in size and scope. Needs to be more focused on testing a particular 
issue. 

HBASE-14564 is issue to reenable.

> Disable zombie TestHFileOutputFormat2
> -
>
> Key: HBASE-14563
> URL: https://issues.apache.org/jira/browse/HBASE-14563
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
>
> Disabling until someone has a chance to look at it.
> I watched it in jvisualvm a while. Its starting and stopping clusters 
> multiple times and then running mr jobs. Needs a rewrite at least and some 
> shrinking of scope on what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14563) Disable zombie TestHFileOutputFormat2

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14563:
--
Attachment: 14563.txt

What I pushed to master, branch-1, and branch-1.2.

> Disable zombie TestHFileOutputFormat2
> -
>
> Key: HBASE-14563
> URL: https://issues.apache.org/jira/browse/HBASE-14563
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 14563.txt
>
>
> Disabling until someone has a chance to look at it.
> I watched it in jvisualvm a while. Its starting and stopping clusters 
> multiple times and then running mr jobs. Needs a rewrite at least and some 
> shrinking of scope on what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14420) Zombie Stomping Session

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14420:
--
Status: Patch Available  (was: Open)

Submitting a non-patch

> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of this one. Am running builds back-to-back 
> on little cluster to turn out the monsters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking

2015-10-06 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14497:
---
Fix Version/s: 1.3.0

> Reverse Scan threw StackOverflow caused by readPt checking
> --
>
> Key: HBASE-14497
> URL: https://issues.apache.org/jira/browse/HBASE-14497
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 0.98.14, 1.3.0
>Reporter: Yerui Sun
>Assignee: Yerui Sun
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, 
> HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, 
> HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, 
> HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, 
> HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, 
> HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, 
> HBASE-14497-master.patch
>
>
> I met stack overflow error in StoreFileScanner.seekToPreviousRow using 
> reversed scan. I searched and founded HBASE-14155, but it seems to be a 
> different reason.
> The seekToPreviousRow will fetch the row which closest before, and compare 
> mvcc to the readPt, which acquired when scanner created. If the row's mvcc is 
> bigger than readPt, an recursive call of seekToPreviousRow will invoked, to 
> find the next closest before row.
> Considering we created a scanner for reversed scan, and some data with 
> smaller rows was written and flushed, before calling scanner next. When 
> seekToPreviousRow was invoked, it would call itself recursively, until all 
> rows which written after scanner created were iterated. The depth of 
> recursive calling stack depends on the count of rows, the stack overflow 
> error will be threw if the count of rows is large, like 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking

2015-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945745#comment-14945745
 ] 

Hadoop QA commented on HBASE-14497:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12765206/14497-branch-1-v6.patch
  against branch-1 branch at commit 23079c02bf40c318fff4f77fa9182ebdfb230e90.
  ATTACHMENT ID: 12765206

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s): 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15885//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15885//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15885//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15885//console

This message is automatically generated.

> Reverse Scan threw StackOverflow caused by readPt checking
> --
>
> Key: HBASE-14497
> URL: https://issues.apache.org/jira/browse/HBASE-14497
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 0.98.14, 1.3.0
>Reporter: Yerui Sun
>Assignee: Yerui Sun
> Fix For: 2.0.0
>
> Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, 
> HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, 
> HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, 
> HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, 
> HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, 
> HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, 
> HBASE-14497-master.patch
>
>
> I met stack overflow error in StoreFileScanner.seekToPreviousRow using 
> reversed scan. I searched and founded HBASE-14155, but it seems to be a 
> different reason.
> The seekToPreviousRow will fetch the row which closest before, and compare 
> mvcc to the readPt, which acquired when scanner created. If the row's mvcc is 
> bigger than readPt, an recursive call of seekToPreviousRow will invoked, to 
> find the next closest before row.
> Considering we created a scanner for reversed scan, and some data with 
> smaller rows was written and flushed, before calling scanner next. When 
> seekToPreviousRow was invoked, it would call itself recursively, until all 
> rows which written after scanner created were iterated. The depth of 
> recursive calling stack depends on the count of rows, the stack overflow 
> error will be threw if the count of rows is large, like 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14563) Disable zombie TestHFileOutputFormat2

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14563.
---
   Resolution: Fixed
Fix Version/s: 1.3.0
   1.2.0
   2.0.0

Pushed to branch-1.2+

> Disable zombie TestHFileOutputFormat2
> -
>
> Key: HBASE-14563
> URL: https://issues.apache.org/jira/browse/HBASE-14563
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14563.txt
>
>
> Disabling until someone has a chance to look at it.
> I watched it in jvisualvm a while. Its starting and stopping clusters 
> multiple times and then running mr jobs. Needs a rewrite at least and some 
> shrinking of scope on what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945733#comment-14945733
 ] 

stack commented on HBASE-14563:
---

Looks like some of the tests were disabled in this suite already.

> Disable zombie TestHFileOutputFormat2
> -
>
> Key: HBASE-14563
> URL: https://issues.apache.org/jira/browse/HBASE-14563
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
>
> Disabling until someone has a chance to look at it.
> I watched it in jvisualvm a while. Its starting and stopping clusters 
> multiple times and then running mr jobs. Needs a rewrite at least and some 
> shrinking of scope on what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14420) Zombie Stomping Session

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14420:
--
Attachment: none_fix.txt

A non-fix just to see how patch build is doing. It is currently quiet.

> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies. 
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native 
> threads). Having to do multiple test runs in the hope that we can get a 
> non-zombie-making build or making (arbitrary) rulings that the zombies are 
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier 
> this week. Will hang sub-issues of this one. Am running builds back-to-back 
> on little cluster to turn out the monsters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14511) StoreFile.Writer Meta Plugin

2015-10-06 Thread Devaraj Das (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945771#comment-14945771
 ] 

Devaraj Das commented on HBASE-14511:
-

[~vladrodionov] when you say it doesn't work with MOB, could you say what's not 
working. Is there some test to repro the failure?

> StoreFile.Writer Meta Plugin
> 
>
> Key: HBASE-14511
> URL: https://issues.apache.org/jira/browse/HBASE-14511
> Project: HBase
>  Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14511.v1.patch, HBASE-14511.v2.patch
>
>
> During my work on a new compaction policies (HBASE-14468, HBASE-14477) I had 
> to modify the existing code of a StoreFile.Writer to add additional meta-info 
> required by these new  policies. I think that it should be done by means of a 
> new Plugin framework, because this seems to be a general capability/feature. 
> As a future enhancement this can become a part of a more general 
> StoreFileWriter/Reader plugin architecture. But I need only Meta section of a 
> store file.
> This could be used, for example, to collect rowkeys distribution information 
> during hfile creation. This info can be used later to find the optimal region 
> split key or to create optimal set of sub-regions for M/R jobs or other jobs 
> which can operate on a sub-region level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14561) Disable zombie TestReplicationShell

2015-10-06 Thread stack (JIRA)

stack created HBASE-14561:
-

 Summary: Disable zombie TestReplicationShell
 Key: HBASE-14561
 URL: https://issues.apache.org/jira/browse/HBASE-14561
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


It hung three times in last 40 test runs. Will file issue to reenable it when 
someone has chance to look at why it is hanging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14562) Fix and reenable zombie TestReplicationShell

2015-10-06 Thread stack (JIRA)

stack created HBASE-14562:
-

 Summary: Fix and reenable zombie TestReplicationShell
 Key: HBASE-14562
 URL: https://issues.apache.org/jira/browse/HBASE-14562
 Project: HBase
  Issue Type: Bug
Reporter: stack


Was disabled over in HBASE-14561 because it hangs with some regularity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14561) Disable zombie TestReplicationShell

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945663#comment-14945663
 ] 

stack commented on HBASE-14561:
---

Link to issue to reenable this test when fixed.

> Disable zombie TestReplicationShell
> ---
>
> Key: HBASE-14561
> URL: https://issues.apache.org/jira/browse/HBASE-14561
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
> Attachments: 14561.txt
>
>
> It hung three times in last 40 test runs. Will file issue to reenable it when 
> someone has chance to look at why it is hanging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14561) Disable zombie TestReplicationShell

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14561:
--
Attachment: 14561.txt

This is what I pushed to master.

> Disable zombie TestReplicationShell
> ---
>
> Key: HBASE-14561
> URL: https://issues.apache.org/jira/browse/HBASE-14561
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
> Attachments: 14561.txt
>
>
> It hung three times in last 40 test runs. Will file issue to reenable it when 
> someone has chance to look at why it is hanging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14561) Disable zombie TestReplicationShell

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14561.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 2.0.0

Pushed to master.

> Disable zombie TestReplicationShell
> ---
>
> Key: HBASE-14561
> URL: https://issues.apache.org/jira/browse/HBASE-14561
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 14561.txt
>
>
> It hung three times in last 40 test runs. Will file issue to reenable it when 
> someone has chance to look at why it is hanging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14519) Purge TestFavoredNodeAssignmentHelper, a test for an abandoned feature that can hang

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14519:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

This has been pushed. Resolving.

> Purge TestFavoredNodeAssignmentHelper, a test for an abandoned feature that 
> can hang
> 
>
> Key: HBASE-14519
> URL: https://issues.apache.org/jira/browse/HBASE-14519
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 0.98.16
>
> Attachments: 14519.txt, 14519v2.txt
>
>
> It came in here:
> commit 7a7ab8b8da795177f42e434b1ab1b468e5cd035a
> Author: Devaraj Das 
> Date:   Sun May 12 06:47:39 2013 +
> HBASE-7932. Introduces Favored Nodes for region files. Adds a balancer 
> called FavoredNodeLoadBalancer that will honor favored nodes in the process 
> of balancing but the balance operation is currently a no-op (Devaraj Das)
> git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1481476 
> 13f79535-47bb-0310-9956-ffa450edef68
> I've already purged the other test that came in on this patch...  over in 
> HBASE-14486
> The test hung here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/15823//console
> ... though we seemed to have exited abnormally.
> Will let this issue hang around a while in case someone  disagrees on removal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking

2015-10-06 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945767#comment-14945767
 ] 

Ted Yu commented on HBASE-14497:


Test suite passed:
{code}
Fetching https://builds.apache.org/job/PreCommit-HBASE-Build/15885/consoleFull
Building remotely on H0 (Hadoop 
Tez) in workspace /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build
Testing patch for HBASE-14497.
Testing patch on branch branch-1.
[INFO] Apache HBase .. SUCCESS [2.722s]
[INFO] Apache HBase - Checkstyle . SUCCESS [0.509s]
[INFO] Apache HBase - Resource Bundle  SUCCESS [0.162s]
[INFO] Apache HBase - Annotations  SUCCESS [0.911s]
[INFO] Apache HBase - Protocol ... SUCCESS [11.037s]
[INFO] Apache HBase - Common . SUCCESS [1:28.402s]
[INFO] Apache HBase - Procedure .. SUCCESS [1:52.825s]
[INFO] Apache HBase - Client . SUCCESS [1:20.748s]
[INFO] Apache HBase - Hadoop Compatibility ... SUCCESS [7.393s]
[INFO] Apache HBase - Hadoop Two Compatibility ... SUCCESS [7.066s]
[INFO] Apache HBase - Prefix Tree  SUCCESS [9.676s]
[INFO] Apache HBase - Server . SUCCESS 
[1:36:58.528s]
[INFO] Apache HBase - Testing Util ... SUCCESS [1.222s]
[INFO] Apache HBase - Thrift . SUCCESS [3:20.890s]
[INFO] Apache HBase - Rest ... SUCCESS [9:11.530s]
[INFO] Apache HBase - Shell .. SUCCESS [5:26.924s]
[INFO] Apache HBase - Integration Tests .. SUCCESS [1.363s]
[INFO] Apache HBase - Examples ... SUCCESS [8.626s]
[INFO] Apache HBase - External Block Cache ... SUCCESS [0.606s]
[INFO] Apache HBase - Assembly ... SUCCESS [1.394s]
[INFO] Apache HBase - Shaded . SUCCESS [0.083s]
[INFO] Apache HBase - Shaded - Client  SUCCESS [0.359s]
[INFO] Apache HBase - Shaded - Server  SUCCESS [0.483s]
Printing hanging tests
Printing Failing tests
{code}

> Reverse Scan threw StackOverflow caused by readPt checking
> --
>
> Key: HBASE-14497
> URL: https://issues.apache.org/jira/browse/HBASE-14497
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 0.98.14, 1.3.0
>Reporter: Yerui Sun
>Assignee: Yerui Sun
> Fix For: 2.0.0
>
> Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, 
> HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, 
> HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, 
> HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, 
> HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, 
> HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, 
> HBASE-14497-master.patch
>
>
> I met stack overflow error in StoreFileScanner.seekToPreviousRow using 
> reversed scan. I searched and founded HBASE-14155, but it seems to be a 
> different reason.
> The seekToPreviousRow will fetch the row which closest before, and compare 
> mvcc to the readPt, which acquired when scanner created. If the row's mvcc is 
> bigger than readPt, an recursive call of seekToPreviousRow will invoked, to 
> find the next closest before row.
> Considering we created a scanner for reversed scan, and some data with 
> smaller rows was written and flushed, before calling scanner next. When 
> seekToPreviousRow was invoked, it would call itself recursively, until all 
> rows which written after scanner created were iterated. The depth of 
> recursive calling stack depends on the count of rows, the stack overflow 
> error will be threw if the count of rows is large, like 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14559) branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers

2015-10-06 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945789#comment-14945789
 ] 

Mikhail Antonov commented on HBASE-14559:
-

[~stack] when you traced the tests using excessive number of threads, did they 
timeout because they run slower with lower number of threads, or did they 
deadlock?

I think there're still bugs lurking around in the implementation :( If we have 
number of thread handlers 3 rather than 40, I might expect things running 
noticeably slower, but not the deadlock?

> branch-1 test tweeks; disable assert explicit region lands post-restart and 
> up a few handlers
> -
>
> Key: HBASE-14559
> URL: https://issues.apache.org/jira/browse/HBASE-14559
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14559.branch-1.txt, 14559.master.txt
>
>
> Running branch-1 on internal rig (trying to stabilize branch-1/branch-1.2). 
> Small tweaks get tests to pass. Small one liners that up priority handler 
> count and disable assert that seems wrong -- that we'll always get an explcit 
> region to land on a newly started server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14565) Make ZK connection timeout configurable in MiniZooKeeperCluster

2015-10-06 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14565:
---
Attachment: 14565-v1.txt

> Make ZK connection timeout configurable in MiniZooKeeperCluster
> ---
>
> Key: HBASE-14565
> URL: https://issues.apache.org/jira/browse/HBASE-14565
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14565-v1.txt
>
>
> This request was made by [~swagle] who works on Ambari Metrics System.
> Currently a hardcoded timeout of 30s is used by MiniZooKeeperCluster
> This affects operation of Ambari Metrics System in standalone mode.
> This JIRA is to make the connection timeout configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945844#comment-14945844
 ] 

Hudson commented on HBASE-14563:


SUCCESS: Integrated in HBase-1.3-IT #214 (See 
[https://builds.apache.org/job/HBase-1.3-IT/214/])
HBASE-14563 Disable zombie TestHFileOutputFormat2 (stack: rev 
aeb3a624590be8bd276e58bba9d4debfb3e7759f)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java


> Disable zombie TestHFileOutputFormat2
> -
>
> Key: HBASE-14563
> URL: https://issues.apache.org/jira/browse/HBASE-14563
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14563.txt
>
>
> Disabling until someone has a chance to look at it.
> I watched it in jvisualvm a while. Its starting and stopping clusters 
> multiple times and then running mr jobs. Needs a rewrite at least and some 
> shrinking of scope on what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14561) Disable zombie TestReplicationShell

2015-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945862#comment-14945862
 ] 

Hudson commented on HBASE-14561:


FAILURE: Integrated in HBase-TRUNK #6878 (See 
[https://builds.apache.org/job/HBase-TRUNK/6878/])
HBASE-14561 Disable zombie TestReplicationShell (stack: rev 
fd6acbbf51998a964b6dc0c7d3ee471399a03baa)
* 
hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java


> Disable zombie TestReplicationShell
> ---
>
> Key: HBASE-14561
> URL: https://issues.apache.org/jira/browse/HBASE-14561
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 14561.txt
>
>
> It hung three times in last 40 test runs. Will file issue to reenable it when 
> someone has chance to look at why it is hanging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13478) Document the change of default master ports being used .

2015-10-06 Thread Jonathan Hsieh (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945903#comment-14945903
 ] 

Jonathan Hsieh commented on HBASE-13478:


+1 lgtm.

> Document the change of default master ports being used .
> 
>
> Key: HBASE-13478
> URL: https://issues.apache.org/jira/browse/HBASE-13478
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Srikanth Srungarapu
>Assignee: Misty Stanley-Jones
>Priority: Minor
> Attachments: HBASE-13478.patch
>
>
> In 1.0.x, master by default binds to the region server ports. But in 1.1 and 
> 2.0 branches, we have undone this changes and brought back the usage of old 
> master ports to make the migration from 0.98 -> 1.1 hassle free.  Please see 
> the parent jira for more background.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14565) Make ZK connection timeout configurable in MiniZooKeeperCluster

2015-10-06 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14565:
---
Attachment: 14565-v1.txt

> Make ZK connection timeout configurable in MiniZooKeeperCluster
> ---
>
> Key: HBASE-14565
> URL: https://issues.apache.org/jira/browse/HBASE-14565
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14565-v1.txt
>
>
> This request was made by [~swagle] who works on Ambari Metrics System.
> Currently a hardcoded timeout of 30s is used by MiniZooKeeperCluster
> This affects operation of Ambari Metrics System in standalone mode.
> This JIRA is to make the connection timeout configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14565) Make ZK connection timeout configurable in MiniZooKeeperCluster

2015-10-06 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14565:
---
Attachment: (was: 14565-v1.txt)

> Make ZK connection timeout configurable in MiniZooKeeperCluster
> ---
>
> Key: HBASE-14565
> URL: https://issues.apache.org/jira/browse/HBASE-14565
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14565-v1.txt
>
>
> This request was made by [~swagle] who works on Ambari Metrics System.
> Currently a hardcoded timeout of 30s is used by MiniZooKeeperCluster
> This affects operation of Ambari Metrics System in standalone mode.
> This JIRA is to make the connection timeout configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14565) Make ZK connection timeout configurable in MiniZooKeeperCluster

2015-10-06 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14565:
---
Status: Patch Available  (was: Open)

> Make ZK connection timeout configurable in MiniZooKeeperCluster
> ---
>
> Key: HBASE-14565
> URL: https://issues.apache.org/jira/browse/HBASE-14565
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14565-v1.txt
>
>
> This request was made by [~swagle] who works on Ambari Metrics System.
> Currently a hardcoded timeout of 30s is used by MiniZooKeeperCluster
> This affects operation of Ambari Metrics System in standalone mode.
> This JIRA is to make the connection timeout configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14565) Make ZK connection timeout configurable in MiniZooKeeperCluster

2015-10-06 Thread Ted Yu (JIRA)

Ted Yu created HBASE-14565:
--

 Summary: Make ZK connection timeout configurable in 
MiniZooKeeperCluster
 Key: HBASE-14565
 URL: https://issues.apache.org/jira/browse/HBASE-14565
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu


This request was made by [~swagle] who works on Ambari Metrics System.

Currently a hardcoded timeout of 30s is used by MiniZooKeeperCluster

This affects operation of Ambari Metrics System in standalone mode.

This JIRA is to make the connection timeout configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-13819:
--
Attachment: HBASE-13819_branch-1.patch

Retry

Commit I'd say [~anoop.hbase]  Needs a release note sir.

> Make RPC layer CellBlock buffer a DirectByteBuffer
> --
>
> Key: HBASE-13819
> URL: https://issues.apache.org/jira/browse/HBASE-13819
> Project: HBase
>  Issue Type: Sub-task
>  Components: Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, 
> HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch
>
>
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an 
> on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto 
> certain number of buffers. This jira aims at testing possibility for making 
> this buffers off heap ones. (DBB)  The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are 
> not using unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap 
> writes will be better
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer 
> to the socket channel, it will create a temp DBB and copy data to there and 
> only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can  
> avoid this one more level of copying.
> Will do different perf testing with changed and report back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12615) Document GC conserving guidelines for contributors

2015-10-06 Thread Jonathan Hsieh (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945889#comment-14945889
 ] 

Jonathan Hsieh commented on HBASE-12615:


Wow that is a lot of trailing space removal.

caught this in one of them:
nit : This will output smt like: -> This will output something like:

lgtm +1.

> Document GC conserving guidelines for contributors
> --
>
> Key: HBASE-12615
> URL: https://issues.apache.org/jira/browse/HBASE-12615
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Andrew Purtell
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-12615.patch
>
>
> LinkedIn put up a blog post with a nice concise list of GC conserving 
> techniques we should document for contributors. Additionally, when we're at a 
> point our build supports custom error-prone plugins, we can develop warnings 
> for some of them. 
> Source: 
> http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage
> - Be careful with Iterators
> - Estimate the size of a collection when initializing
> - Defer expression evaluation
> - Compile the regex patterns in advance
> - Cache it if you can
> - String Interns are useful but dangerous
> All good advice and practice that I know we aim for. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13425) Documentation nit in REST Gateway impersonation section

2015-10-06 Thread Jonathan Hsieh (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945902#comment-14945902
 ] 

Jonathan Hsieh commented on HBASE-13425:


+1 lgtm.

> Documentation nit in REST Gateway impersonation section
> ---
>
> Key: HBASE-13425
> URL: https://issues.apache.org/jira/browse/HBASE-13425
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 2.0.0
>Reporter: Jeremie Gomez
>Assignee: Misty Stanley-Jones
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-13425.patch
>
>
> In section "55.8. REST Gateway Impersonation Configuration", there is another 
> property that needs to be set (and thus documented).
> After this sentence ("To enable REST gateway impersonation, add the following 
> to the hbase-site.xml file for every REST gateway."), we should add :
> 
>hbase.rest.support.proxyuser
> true
> 
> It not set, doing a curl call on the rest gateway gives the error "support 
> for proxyuser is not configured".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

2015-10-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945926#comment-14945926
 ] 

stack commented on HBASE-13082:
---

bq. But in case of bulk loaded files, currently in between a scan if a new file 
is bulk loaded it gets included, so after this it will not be. is that 
behavioral change fine?

Sorry, say more [~ram_krish]. So, bulk load won't show mid-scan... you have to 
get to the end? That would be fine.

On the patch, can we get more of Lars comments in on what is going on  
Could we get rid of some of these getReaderLocks too... in hstorefile, in 
hstore, etc would be good to not let this stuff out if we can.



> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, gc.png, gc.png, gc.png, hits.png, 
> next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration

2015-10-06 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14436.
---
   Resolution: Fixed
 Assignee: stack
 Hadoop Flags: Reviewed
Fix Version/s: 0.98.16
   1.1.3
   1.0.3
   1.3.0
   1.2.0
   2.0.0

Pushed to 0.98+

> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create 
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
>  Issue Type: Improvement
>  Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following 
> format:
> {code}
>  public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
>  int priority, final Map kvs)
>   throws IOException {
>   ...
>   String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
>   ...
> }
> {code}
> If the 'jarFilePath' is null,  the 'value' will always has the format 
> '|className|priority|'  even if 'kvs' is null, which means no extra arguments 
> for the coprocessor. Then, in the server side, 
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table 
> coprocessors as:
> {code}
>   static List 
> getTableCoprocessorAttrsFromSchema(Configuration conf,
>   HTableDescriptor htd) {
> ...
> try {
>   cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the 
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
>   // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) {  // => cfgSpec will be '|' for the format 
> '|className|priority|'
>   ourConf = new Configuration(false);
>   HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as 
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors, 
> so that the region server will create new Configurations for each region of 
> the table, this will consume a certain number of memory when we have many 
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no 
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in 
> server side which could avoid creating new Configurations for existed such 
> regions after the regions reopened. Discussions and suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 188 matches

Mail list logo