[jira] [Resolved] (HBASE-20095) Redesign single instance pool in CleanerChore

[Reid Chan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reid Chan resolved HBASE-20095.
---
Resolution: Fixed

> Redesign single instance pool in CleanerChore
> -
>
> Key: HBASE-20095
> URL: https://issues.apache.org/jira/browse/HBASE-20095
> Project: HBase
>  Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: 20095.addendum, 20095.addendum-2.txt, 
> HBASE-20095.master.001.patch, HBASE-20095.master.002.patch, 
> HBASE-20095.master.003.patch, HBASE-20095.master.004.patch, 
> HBASE-20095.master.005.patch, HBASE-20095.master.006.patch, 
> HBASE-20095.master.007.patch, HBASE-20095.master.008.patch, 
> HBASE-20095.master.009.patch, HBASE-20095.master.010.patch, 
> HBASE-20095.master.011.patch, HBASE-20095.master.012.patch, 
> HBASE-20095.master.013.patch, HBASE-20095.master.014.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-20095) Redesign single instance pool in CleanerChore

[Reid Chan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reid Chan reopened HBASE-20095:
---

Reopen to backport branch-2.0

> Redesign single instance pool in CleanerChore
> -
>
> Key: HBASE-20095
> URL: https://issues.apache.org/jira/browse/HBASE-20095
> Project: HBase
>  Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: 20095.addendum, 20095.addendum-2.txt, 
> HBASE-20095.master.001.patch, HBASE-20095.master.002.patch, 
> HBASE-20095.master.003.patch, HBASE-20095.master.004.patch, 
> HBASE-20095.master.005.patch, HBASE-20095.master.006.patch, 
> HBASE-20095.master.007.patch, HBASE-20095.master.008.patch, 
> HBASE-20095.master.009.patch, HBASE-20095.master.010.patch, 
> HBASE-20095.master.011.patch, HBASE-20095.master.012.patch, 
> HBASE-20095.master.013.patch, HBASE-20095.master.014.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20801) Fix broken TestReplicationShell

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-20801:
-

 Summary: Fix broken TestReplicationShell
 Key: HBASE-20801
 URL: https://issues.apache.org/jira/browse/HBASE-20801
 Project: HBase
  Issue Type: Sub-task
  Components: Replication, shell, test
Reporter: Duo Zhang
Assignee: Duo Zhang
 Fix For: HBASE-19064






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[RESULT][VOTE] Merge branch HBASE-19064 back to master

[Duo Zhang]

One week since we start the vote, and now we have 4 binding +1s, a 0, and
no -1s, so the vote passes. Let me close the vote.

There are still small issues for branch HBASE-19064, will merge back to
master after we fix them all.

Thanks all for voting.

2018-06-26 18:20 GMT+08:00 Ashish Singhi :

> SGTM. +1
>
> Regards,
> Ashish
>
> On Tue, Jun 26, 2018 at 12:30 PM, 张铎(Duo Zhang) 
> wrote:
>
> > Here is my +1.
> >
> > As the performance test shows that there is no performance downgrading if
> > we do not enable sync replication, and the feature basically works, I
> think
> > it is OK to merge it back to master branch. We will continue working on
> > polishing and optimizing it for the 3.0 release.
> >
> > 2018-06-26 14:52 GMT+08:00 OpenInx :
> >
> > > Hi guys
> > >
> > > I've tested the write performance on both master branch and
> HBASE-19064
> > > branch.  Here's the JIRA [1].
> > >
> > > The conclusion(Pasted from the comment):
> > > The QPS & AVG latency are similar when comparing the master branch with
> > > async replication and HBASE-19064 with async replication.
> > > BTW the QPS of HBASE-19064 with sync replication dropped about 13%, we
> > will
> > > continue to optimize the sync replication in phrase#2 [2].
> > >
> > > 1. https://issues.apache.org/jira/browse/HBASE-20751
> > > 2. https://issues.apache.org/jira/browse/HBASE-20422
> > >
> > >
> > > On Thu, Jun 21, 2018 at 9:57 PM, Guanghao Zhang 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > 2018-06-21 18:26 GMT+08:00 mengli721014 on 163 dot com <
> > > > mengli721...@163.com
> > > > >:
> > > >
> > > > > 0
> > > > >
> > > > > 使用AquaMail for Android发送
> > > > > http://www.aqua-mail.com
> > > > >
> > > > >
> > > > >
> > > > > 在 2018年6月21日 下午4:21:26 "张铎(Duo Zhang)"  写道:
> > > > >
> > > > > In HBASE-19064 we aim to implement sync replication feature for
> > HBase.
> > > > You
> > > > >> can see the design doc for more details on how it works
> > > > >>
> > > > >> https://docs.google.com/document/d/193D3aOxD-muPIZuQfI4Zo3_
> > > > >> qg6-Nepeu_kraYJVQkiE/edit#
> > > > >>
> > > > >> And now the feature basically works, we have tested it on real
> > > clusters,
> > > > >> and also provide the operational documentation in the ref guide.
> You
> > > can
> > > > >> see the 'Synchronous Replication' section in the ref guide of
> branch
> > > > >> HBASE-19064.
> > > > >>
> > > > >> There are known limitations for this feature, and we have
> > > > >> created HBASE-20422 to track them.
> > > > >>
> > > > >> Please vote:
> > > > >> [+1] Agree
> > > > >> [-1] Disagree
> > > > >> [0] Neutral
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (HBASE-20800) Master orchestrated compactions

[stack (JIRA)]

stack created HBASE-20800:
-

 Summary: Master orchestrated compactions
 Key: HBASE-20800
 URL: https://issues.apache.org/jira/browse/HBASE-20800
 Project: HBase
  Issue Type: Umbrella
  Components: Compaction
Reporter: stack
Assignee: Mohit Goel


An umbrella issue for having compactions go via the Master so we can have a 
centralized arbitrator of cluster i/o. If we put Master in the way, we can do 
stuff like:

 * Ask the Master for current cluster compaction state; what is running, what 
is blocked
 * Master can manage cluster-wide compaction policy and/or throttling/or 
blocking of compaction i/os.
 * Master can schedule when and where compactions run so we can guard against 
the pathological where all RegionServers decide now is the time to major 
compact bringing on a compaction storm.

Other side-benefits might include being able to farm out the compaction work to 
another process -- e.g. the splice machine model of having spark run the 
compactions -- or just to a separate compactor that we might i/o nice.

 * We'll need to figure how to externalize the CompactionRequest so it can be 
passed over RPC.
 * We'll need to have something like a CompactionManager in the Master process 
that keeps up current cluster state.


MOB needs a compaction fabric it can use. Its compactions are currently 
Master-based only and so don't scale. It could make use of this mechanism to 
ask the Master to farm out its compaction requests.

This is an umbrella issue. I thought I'd filed one already on this topic but 
can't find it. Will shut it down if I trip over it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-20799) TestBucketCache#testCacheBlockNextBlockMetadataMissing is flaky

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-20799.

Resolution: Duplicate

> TestBucketCache#testCacheBlockNextBlockMetadataMissing is flaky
> ---
>
> Key: HBASE-20799
> URL: https://issues.apache.org/jira/browse/HBASE-20799
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.5.0, 1.4.5
>Reporter: Andrew Purtell
>Priority: Major
>
> {noformat}
> [ERROR] testCacheBlockNextBlockMetadataMissing[1: blockSize=16,384, 
> bucketSizes=[I@29ee9faa](org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache)
>   Time elapsed: 0.066 s  <<< FAILURE!
> java.lang.AssertionError: expected: 
> java.nio.HeapByteBuffer but 
> was: java.nio.HeapByteBuffer
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache.testCacheBlockNextBlockMetadataMissing(TestBucketCache.java:424)
> {noformat}
> [~zyork] any idea what is going on here?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20799) TestBucketCache#testCacheBlockNextBlockMetadataMissing is flaky

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-20799:
--

 Summary: TestBucketCache#testCacheBlockNextBlockMetadataMissing is 
flaky
 Key: HBASE-20799
 URL: https://issues.apache.org/jira/browse/HBASE-20799
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.5.0
Reporter: Andrew Purtell


{noformat}
[ERROR] testCacheBlockNextBlockMetadataMissing[1: blockSize=16,384, 
bucketSizes=[I@29ee9faa](org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache)
  Time elapsed: 0.066 s  <<< FAILURE!
java.lang.AssertionError: expected: 
java.nio.HeapByteBuffer but 
was: java.nio.HeapByteBuffer
at 
org.apache.hadoop.hbase.io.hfile.bucket.TestBucketCache.testCacheBlockNextBlockMetadataMissing(TestBucketCache.java:424)
{noformat}

[~zyork] any idea what is going on here?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20798) Duplicated threads name of StoreFileOpenerThread

[Zephyr Guo (JIRA)]

Zephyr Guo created HBASE-20798:
--

 Summary: Duplicated threads name of 
StoreFileOpenerThread
 Key: HBASE-20798
 URL: https://issues.apache.org/jira/browse/HBASE-20798
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 2.0.0, 3.0.0
Reporter: Zephyr Guo
Assignee: Zephyr Guo
 Fix For: 3.0.0


{code:title=jstack}
 "StoreFileOpenerThread-info-1" #8994 daemon prio=5 os_prio=0 
tid=0x7fad7a46b000 nid=0x624a waiting on condition [0x7fad7f7ef000]
 "StoreFileOpenerThread-info-1" #8993 daemon prio=5 os_prio=0 
tid=0x7fad71064000 nid=0x6249 waiting on condition [0x7fad7fff7000]
{code}

Duplicated thread names exists in jstack sometimes, because 
StoreFileOpenerThreads are created per region and have same names. 

Suggest adding region name to corresponding thread name in order to distinguish 
StoreFileOpenerThreads, which are created per region. This could be helpful for 
troubleShooting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20797) hbase-spark

[Lars Francke (JIRA)]

Lars Francke created HBASE-20797:


 Summary: hbase-spark 
 Key: HBASE-20797
 URL: https://issues.apache.org/jira/browse/HBASE-20797
 Project: HBase
  Issue Type: Bug
  Components: spark
Affects Versions: 3.0.0
Reporter: Lars Francke


We're running into an issue using the spark integration when using Hadoop 
2.7.2. The problem is this line of code from {{HBaseContext.scala}}

{code:java}
ugi.setAuthenticationMethod(AuthenticationMethod.PROXY)
{code}

I'm not an expert but I think that's wrong code. If we were to create a Proxy 
user then we'd need to use {{UserGroupInformation.createProxyUser(...) }} which 
would also set the realUser etc. Also: I don't think it makes sense to create a 
proxy user on the client side? The chances are good that the user we're 
authenticating as doesn't exen have proxy privileges as it's usually only 
granted to servers.

We've tried to trace where this line of code came from in Git but it was a code 
drop back in Ted's original repo.

The error we're seeing actually occurs when (in a Spark job) we access HDFS 
because KMSClientProvider has code like this:

{code:java}
actualUgi =
(UserGroupInformation.getCurrentUser().getAuthenticationMethod() ==
UserGroupInformation.AuthenticationMethod.PROXY) ? UserGroupInformation
.getCurrentUser().getRealUser() : UserGroupInformation
{code}

But we've never set up the realUser so actualUgi is null which later leads to a 
NullPointerException.

I _think_ the proper fix is to just remove that line as I have no idea what its 
intention is. I can provide a patch but I'd like to get input first. Maybe I'm 
mistaken?




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Looking for JIRA's to contribute to HBase

[Chia-Ping Tsai]

the flaky test is the good starting point for me since we can reproduce the bug 
easily. Also, the test case is usually a good and true scenario in hbase. 

there is a flaky list for your reference.

https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html

Cheers,
chia-ping

On 2018/06/26 22:05:54, Karan Mehta  wrote: 
> Hello everyone,
> 
> I have been using HBase for quite some time and also worked on few small
> JIRA's here
> .
> I am looking forward to JIRA's that can help me understand the practical
> aspects of distributed systems. It would be helpful if anyone can provide
> me list of issues that can be a good starting point for my better
> understanding of code (includes fixing test cases), have potential impact
> and then I will be happy to take up more widely scoped large JIRA's as
> well.
> 
> Regards
> Karan Mehta
> 

Only 3 days left for HBaseConAsia 2018 abstracts!

[Yu Li]

Hi All,

大家好，

The HBaseConAsia 2018 call for proposals is scheduled to close Saturday,
June 30th. If you have an idea for a talk, make sure you get it submitted
ASAP!

HBaseConAsia2018的议题提交将于6月30日（本周六）截止，如果您愿意分享相关主题[1]，请尽快提交议题摘要。

Submit your talks at
https://easychair.org/conferences/?conf=hbaseconasia2018
请通过如下链接提交议题摘要：https://easychair.org/conferences/?conf=hbaseconasia2018

If you need more information, please see
https://hbase.apache.org/hbaseconasia-2018/
更多信息请访问：https://hbase.apache.org/hbaseconasia-2018

- Yu (on behalf of the HBase PMC)

[1] *https://easychair.org/cfp/hbaseconasia-2018
*