[jira] [Created] (HBASE-19438) Doc cleanup after removal of features across Cache/BucketCache

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-19438:
--

 Summary: Doc cleanup after removal of features across 
Cache/BucketCache
 Key: HBASE-19438
 URL: https://issues.apache.org/jira/browse/HBASE-19438
 Project: HBase
  Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Critical
 Fix For: 2.0.0-beta-1






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19437) Wrap the null result from postAppend/postIncrement

[Chia-Ping Tsai (JIRA)]

Chia-Ping Tsai created HBASE-19437:
--

 Summary: Wrap the null result from postAppend/postIncrement 
 Key: HBASE-19437
 URL: https://issues.apache.org/jira/browse/HBASE-19437
 Project: HBase
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


The null from postAppend/postIncrement sparks inconsistent behavior. The 
Table#append and #increment can handle the null result but the batch operation 
throws IllegalStateException.

We should wrap the null from postAppend/postIncrement to empty result since cp 
user should throw IOException rather than return null if they intend to say 
something is broken.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-19297) Nightly job for master timing out in unit tests

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy resolved HBASE-19297.
--
Resolution: Not A Problem

> Nightly job for master timing out in unit tests
> ---
>
> Key: HBASE-19297
> URL: https://issues.apache.org/jira/browse/HBASE-19297
> Project: HBase
>  Issue Type: Task
>  Components: test
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Attachments: results.png
>
>
> master now timing out at 6 hours in master.
> looks like it was making progress still, just in the midst of hte hbase-rest 
> module



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19436) Remove Jenkinsfile from some branches

[Appy (JIRA)]

Appy created HBASE-19436:


 Summary: Remove Jenkinsfile from some branches
 Key: HBASE-19436
 URL: https://issues.apache.org/jira/browse/HBASE-19436
 Project: HBase
  Issue Type: Improvement
Reporter: Appy
Assignee: Appy


Some of the branches in https://builds.apache.org/job/HBase%20Nightly/ don't 
need to be build nightly since they are not in active development. Let's remove 
them to reduce clutter and free up restricted resources. Suggested branches:
HBASE-14070.HLC
(todo: others?)




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Removing hbase-native-client (HBASE-19419)

[Duo Zhang]

Oh, it should be 'silence for months'...

2017-12-06 6:40 GMT+08:00 张铎(Duo Zhang) :

> Hi Josh, not related to this thread but seems the HBASE-14850 branch has
> been silence for silence. What's the current status of the native client
> project?
>
> Thanks.
>
> 2017-12-06 2:45 GMT+08:00 Josh Elser :
>
>> On 12/4/17 11:43 PM, Nick Dimiduk wrote:
>>
>>> On Mon, Dec 4, 2017 at 3:28 PM, Josh Elser  wrote:
>>>
>>> Makes sense to me.

 *soapbox*  I think focusing onhttps://issues.apache.org/jira
 /browse/HBASE-14850 would be a better use of time than trying to
 revitalize the old stuff (which I assume has begun to rot).

>>>
>>> Is the code under hbase-native-client not the same as that referenced by
>>> HBASE-14850? The patches on said ticket's children land code under the
>>> same
>>> path.
>>>
>>
>> That's correct -- separate "efforts". The stuff Andrew is proposing to
>> remove is an older client that wasn't (best I can tell) based on
>> Folly/Wangle like the 14850 stuff is.
>>
>
>

[jira] [Created] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache

[Zach York (JIRA)]

Zach York created HBASE-19435:
-

 Summary: Reopen Files for ClosedChannelException in BucketCache
 Key: HBASE-19435
 URL: https://issues.apache.org/jira/browse/HBASE-19435
 Project: HBase
  Issue Type: Bug
  Components: BucketCache
Affects Versions: 1.3.1, 2.0.0
Reporter: Zach York
Assignee: Zach York


When using the FileIOEngine for BucketCache, the cache will be disabled if the 
connection is interrupted or closed. HBase will then get 
ClosedChannelExceptions trying to access the file. After 60s, the RS will 
disable the cache. This causes severe read performance degradation for 
workloads that rely on this cache. FileIOEngine never tries to reopen the 
connection. This JIRA is to reopen files when the BucketCache encounters a 
ClosedChannelException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19434) create_namespace command for existing namespace does not throw useful error message

[Ted Yu (JIRA)]

Ted Yu created HBASE-19434:
--

 Summary: create_namespace command for existing namespace does not 
throw useful error message
 Key: HBASE-19434
 URL: https://issues.apache.org/jira/browse/HBASE-19434
 Project: HBase
  Issue Type: Bug
Reporter: Romil Choksi
Priority: Minor


create_namespace command for existing namespace does not throw an useful error 
message

Here we try to call create_namespace command twice for same namespace, and 
attempt to verify an error message
from test console log
{code}
2017-11-13 08:19:03,347|INFO|MainThread|hbase.py:191 - runShellCmds()|Writing 
commands to file /hwqe/hadoopqe/artifacts/tmp-844452
2017-11-13 08:19:03,348|INFO|MainThread|hbase.py:194 - runShellCmds()| 
'create_namespace 'testnamespace''
2017-11-13 08:19:03,348|INFO|MainThread|hbase.py:194 - runShellCmds()| 
'create_namespace 'testnamespace''
2017-11-13 08:19:03,348|INFO|MainThread|hbase.py:194 - runShellCmds()| 
'drop_namespace 'testnamespace''
2017-11-13 08:19:03,348|INFO|MainThread|hbase.py:198 - runShellCmds()|Done 
writing commands to file. Will execute them now.
2017-11-13 08:19:03,348|INFO|MainThread|machine.py:145 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|RUNNING: 
/usr/hdp/current/hbase-client/bin/hbase shell 
/hwqe/hadoopqe/artifacts/tmp-844452
2017-11-13 08:19:11,593|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|2017-11-13 08:19:11,592 INFO  
[main] beanutils.FluentPropertyBeanIntrospector: Error when creating 
PropertyDescriptor for public final void 
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
 Ignoring this property.
2017-11-13 08:19:11,628|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|2017-11-13 08:19:11,628 INFO  
[main] impl.MetricsConfig: loaded properties from 
hadoop-metrics2-hbase.properties
2017-11-13 08:19:11,700|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|2017-11-13 08:19:11,700 INFO  
[main] impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2017-11-13 08:19:11,700|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|2017-11-13 08:19:11,700 INFO  
[main] impl.MetricsSystemImpl: HBase metrics system started
2017-11-13 08:19:12,655|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|Took 0.6500 seconds
2017-11-13 08:19:12,784|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|
2017-11-13 08:19:12,784|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|ERROR: testnamespace
2017-11-13 08:19:12,784|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|
2017-11-13 08:19:12,784|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|Create namespace; pass 
namespace name,
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|and optionally a dictionary of 
namespace configuration.
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|Examples:
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|hbase> create_namespace 'ns1'
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|hbase> create_namespace 'ns1', 
{'PROPERTY_NAME'=>'PROPERTY_VALUE'}
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|
2017-11-13 08:19:12,785|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|Took 0.1284 seconds
2017-11-13 08:19:13,016|INFO|MainThread|machine.py:159 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|Took 0.2307 seconds
2017-11-13 08:19:13,541|INFO|MainThread|machine.py:184 - 
run()||GUID=29350822-e327-4a0d-97ec-fc8b0a7d1be2|Exit Code: 0
2017-11-13 08:19:13,575|INFO|MainThread|conftest.py:241 - 
pytest_report_teststatus()|TEST 
"test_ShellCreateNameSpace[CreateExistentNamespace]" FAILED in 10.22 seconds
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19433) ChangeSplitPolicyAction modifies an immutable HTableDescriptor

[Josh Elser (JIRA)]

Josh Elser created HBASE-19433:
--

 Summary: ChangeSplitPolicyAction modifies an immutable 
HTableDescriptor
 Key: HBASE-19433
 URL: https://issues.apache.org/jira/browse/HBASE-19433
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Reporter: Josh Elser
Assignee: Ted Yu
Priority: Critical
 Fix For: 2.0.0-beta-1


{noformat}
2017-12-01 23:18:51,433 WARN  [ChaosMonkeyThread] policies.Policy: Exception 
occurred during performing action: java.lang.UnsupportedOperationException: 
HTableDescriptor is read-only
at 
org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
at 
org.apache.hadoop.hbase.HTableDescriptor.setRegionSplitPolicyClassName(HTableDescriptor.java:333)
at 
org.apache.hadoop.hbase.chaos.actions.ChangeSplitPolicyAction.perform(ChangeSplitPolicyAction.java:54)
at 
org.apache.hadoop.hbase.chaos.policies.PeriodicRandomActionPolicy.runOneIteration(PeriodicRandomActionPolicy.java:59)
at 
org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
at java.lang.Thread.run(Thread.java:745)
{noformat}

Found during some internal testing. Need to make sure this Action, in addition 
to the other, don't fall into the trap of modifying the TableDescriptor 
obtained from Admin.

[~tedyu], want to take a stab at it?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Removing hbase-native-client (HBASE-19419)

[Duo Zhang]

Hi Josh, not related to this thread but seems the HBASE-14850 branch has
been silence for silence. What's the current status of the native client
project?

Thanks.

2017-12-06 2:45 GMT+08:00 Josh Elser :

> On 12/4/17 11:43 PM, Nick Dimiduk wrote:
>
>> On Mon, Dec 4, 2017 at 3:28 PM, Josh Elser  wrote:
>>
>> Makes sense to me.
>>>
>>> *soapbox*  I think focusing onhttps://issues.apache.org/jira
>>> /browse/HBASE-14850 would be a better use of time than trying to
>>> revitalize the old stuff (which I assume has begun to rot).
>>>
>>
>> Is the code under hbase-native-client not the same as that referenced by
>> HBASE-14850? The patches on said ticket's children land code under the
>> same
>> path.
>>
>
> That's correct -- separate "efforts". The stuff Andrew is proposing to
> remove is an older client that wasn't (best I can tell) based on
> Folly/Wangle like the 14850 stuff is.
>

Re: Removing hbase-native-client (HBASE-19419)

[Josh Elser]

On 12/4/17 11:43 PM, Nick Dimiduk wrote:

On Mon, Dec 4, 2017 at 3:28 PM, Josh Elser  wrote:


Makes sense to me.

*soapbox*  I think focusing onhttps://issues.apache.org/jira
/browse/HBASE-14850 would be a better use of time than trying to
revitalize the old stuff (which I assume has begun to rot).


Is the code under hbase-native-client not the same as that referenced by
HBASE-14850? The patches on said ticket's children land code under the same
path.


That's correct -- separate "efforts". The stuff Andrew is proposing to 
remove is an older client that wasn't (best I can tell) based on 
Folly/Wangle like the 14850 stuff is.

Re: Moving 2.0 forward

[Stack]

Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
Christmas day.

Here is the list [1]. We're doing pretty good. We've fixed a load since the
last update including some nice cleanups  two weeks ago (e.g. undoing
client dependency on curator/zk watching making the dependence read-only)
but there are still a few hairy ones hanging out there.

 * HBASE-18946 Stochastic load balancer assigns replica regions to the same
RS
   This one is proving a little tough. Ram has been banging his head on it.
We'll be able to clear a bunch of test failures once this is done.

 * HBASE-19112 Suspect methods on Cell to be deprecated
   Good discussion up on the JIRA. We need to be crystal clear around Cell
and derivatives API and Audience when we ship 2.0.

 * HBASE-17204 Make L2 off heap cache default ON
   We need to try this at least so offheaping can be default.

Making asyncfswal default is also ongoing, HBASE-16890
, making good progress.

I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our unit
test story has gotten better and a few of us are actively working on
flakies [2] but we may make a beta-1 w/o shutting down all failures.

Speak up if you need help on an issue or if you think we are missing items
from our list.

Thanks for all the hard work,

The hbase-2.0.0 RM.


1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
2.
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html

On Thu, Nov 16, 2017 at 3:48 PM, Stack  wrote:

> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> remaining features and apply final polish to API. There will be a beta-2
> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>
> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
> list this morning. See here [1].
>
> There are about ~12 issues in progress with most of these about to land.
> There are 37 TODO. Many of these are tests we need to run, some are related
> to the backup/restore, but a good few are meaty w/o assignees.
>
> The awkward outstanding ones as I see it are the below:
>
> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
> HBASE-17204 Make L2 off heap cache default ON
> HBASE-19112 Suspect methods on Cell to be deprecated
> HBASE-19147 All branch-2 unit tests pass
>
> We need to make progress on the above or punt on them (can't punt on the
> last one though).
>
> Any ideas on what configs we should update in hbase2? Dump ideas into:
> HBASE-19148 Edit of default configuration
>
> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> close behind.
>
> Comments? Thoughts?
>
> Thanks all,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>
> On Tue, Oct 31, 2017 at 2:56 PM, Stack  wrote:
>
>>
>>
>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob  wrote:
>>
>>> Hoping to keep momentum going from our Stack working on alpha4, I tried
>>> to
>>> take a stab at triaging some of the open beta-1 issues.
>>>
>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
>>> then I'm happy to see it pulled back in. Trying to balance optimism with
>>> realism here, and knowing that documentation unfortunately often gets
>>> pushed to the back-burner.
>>>
>>> Also, I poked some folks on unassigned issues that they've filed for
>>> beta-1, especially in the last few days. If issues don't have an owner
>>> they
>>> are unlikely to get worked. I chatted with stack and he agreed to take on
>>> some of the tasks, but there's a lot of surface area to cover.
>>>
>>> If you you're working on issues that are critical for beta-1, please mark
>>> them as such. Then the rest of the community will know to help prioritize
>>> feedback and reviews there.
>>>
>>> Do we have a general theme for the betas like we did with the alphas?
>>>
>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>> well? Continue to work on tests throughout?
>>>
>>>
>> Thanks Mike for helping to kick off the beta-1 train.
>>
>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
>> are going to ship (beta-2 is rolling upgrade and any minor items turned up
>> in testing/burn-in).
>>
>> Thanks,
>> S
>>
>>
>>
>>> Mike
>>>
>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser  wrote:
>>>
>>> > +1 go from my POV.
>>> >
>>> >
>>> > On 10/31/17 10:07 AM, Stack wrote:
>>> >
>>> >> I want to push an alpha-4 today. A few items didn't make it
>>> (HBASE-19092).
>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>> basically
>>> >> done. There may be some changes for beta-1 but hopefully only changes
>>> >> informed by experience trying to port an existing Coprocessor to
>>> hbase2.
>>> >>
>>> >> Shout if there is anything that needs to make alpha-4.
>>> >>
>>> >> Thanks,
>>> >> St.Ack
>>> >>
>>> >>
>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser 
>>> wro

[jira] [Created] (HBASE-19432) Roll the specified writer in HFileOutputFormat2

[Guangxu Cheng (JIRA)]

Guangxu Cheng created HBASE-19432:
-

 Summary: Roll the specified writer in HFileOutputFormat2
 Key: HBASE-19432
 URL: https://issues.apache.org/jira/browse/HBASE-19432
 Project: HBase
  Issue Type: Improvement
Affects Versions: 3.0.0, 2.0.0-beta-1
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng


{code}
// If any of the HFiles for the column families has reached
// maxsize, we need to roll all the writers
if (wl != null && wl.written + length >= maxsize) {
  this.rollRequested = true;
}
{code}
If we always roll all the writers, a large number of small files will be 
generated in the multi family or multi table scene.
So we should only roll the specified writer which HFile has reached maxsize.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)