Re: Moving 2.0 forward

[Stack]

Update. See below.

On Tue, Dec 5, 2017 at 10:43 AM, Stack  wrote:

> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
> Christmas day.
>
>
Still aiming for an hbase2 beta-1 showing up in your christmas stocking.


> Here is the list [1]. We're doing pretty good. We've fixed a load since
> the last update including some nice cleanups  two weeks ago (e.g. undoing
> client dependency on curator/zk watching making the dependence read-only)
> but there are still a few hairy ones hanging out there.
>
>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
> same RS
>This one is proving a little tough. Ram has been banging his head on
> it. We'll be able to clear a bunch of test failures once this is done.
>
>
This should be going in tomorrow.



>  * HBASE-19112 Suspect methods on Cell to be deprecated
>Good discussion up on the JIRA. We need to be crystal clear around Cell
> and derivatives API and Audience when we ship 2.0.
>
>
Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
done in next day or so.



>  * HBASE-17204 Make L2 off heap cache default ON
>We need to try this at least so offheaping can be default.
>
>
Chatted with Anoop and Ram. Because it is so late in the day and because it
critical that the user have a good experience out of the box -- which
requires time and testing -- we are going to punt this from 2.0. Will look
at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
offheap read/write path goodness.



> Making asyncfswal default is also ongoing, HBASE-16890
> , making good progress.
>
>

Good progress here. In perf testing asyncfswal makes us generally faster.
Duo making good progress on last few tests that are in the way of our
making this default for hbase2.

A load of goodness has been landing these last few weeks. Keep up the great
work.

Your hbase-2 RM.




> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
> unit test story has gotten better and a few of us are actively working on
> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>
> Speak up if you need help on an issue or if you think we are missing items
> from our list.
>
> Thanks for all the hard work,
>
> The hbase-2.0.0 RM.
>
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
> lastSuccessfulBuild/artifact/dashboard.html
>
> On Thu, Nov 16, 2017 at 3:48 PM, Stack  wrote:
>
>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>> remaining features and apply final polish to API. There will be a beta-2
>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>
>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>> list this morning. See here [1].
>>
>> There are about ~12 issues in progress with most of these about to land.
>> There are 37 TODO. Many of these are tests we need to run, some are related
>> to the backup/restore, but a good few are meaty w/o assignees.
>>
>> The awkward outstanding ones as I see it are the below:
>>
>> HBASE-18946 Stochastic load balancer assigns replica regions to the same
>> RS
>> HBASE-17204 Make L2 off heap cache default ON
>> HBASE-19112 Suspect methods on Cell to be deprecated
>> HBASE-19147 All branch-2 unit tests pass
>>
>> We need to make progress on the above or punt on them (can't punt on the
>> last one though).
>>
>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>> HBASE-19148 Edit of default configuration
>>
>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>> close behind.
>>
>> Comments? Thoughts?
>>
>> Thanks all,
>> S
>>
>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>
>> On Tue, Oct 31, 2017 at 2:56 PM, Stack  wrote:
>>
>>>
>>>
>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob  wrote:
>>>
 Hoping to keep momentum going from our Stack working on alpha4, I tried
 to
 take a stab at triaging some of the open beta-1 issues.

 I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
 sooner
 then I'm happy to see it pulled back in. Trying to balance optimism with
 realism here, and knowing that documentation unfortunately often gets
 pushed to the back-burner.

 Also, I poked some folks on unassigned issues that they've filed for
 beta-1, especially in the last few days. If issues don't have an owner
 they
 are unlikely to get worked. I chatted with stack and he agreed to take
 on
 some of the tasks, but there's a lot of surface area to cover.

 If you you're working on issues that are critical for beta-1, please
 mark
 them as such. Then the rest of the community will know to help
 prioritize
 feedback and reviews there.

 Do we 

[jira] [Resolved] (HBASE-18978) Align the methods in Table and AsyncTable

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-18978.
---
  Resolution: Fixed
Hadoop Flags: Reviewed

Ok. Resolving this effort for now. We may be back but it will be a new project 
informed by experience and at a cusp of a 3.0 release.

Thanks all for fine effort herein.

All subtasks resolved.

> Align the methods in Table and AsyncTable
> -
>
> Key: HBASE-18978
> URL: https://issues.apache.org/jira/browse/HBASE-18978
> Project: HBase
>  Issue Type: Task
>  Components: asyncclient, Client
>Reporter: Duo Zhang
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19511) Splits causes blocks to be cached again and so such blocks cannot be evicted from bucket cache

[ramkrishna.s.vasudevan (JIRA)]

ramkrishna.s.vasudevan created HBASE-19511:
--

 Summary: Splits causes blocks to be cached again and so such 
blocks cannot be evicted from bucket cache
 Key: HBASE-19511
 URL: https://issues.apache.org/jira/browse/HBASE-19511
 Project: HBase
  Issue Type: Bug
  Components: BucketCache
Affects Versions: 2.0.0-alpha-4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 3.0.0, 2.0.0-beta-1


This is because of similar pattern as in 
https://issues.apache.org/jira/browse/HBASE-8547.
It took a lot of time to debug this and the reason for 
TestBlockeEvictionFromClient was flaky due to this. 
When we create split files the index of the firstKey that we create could 
possibly be the same key as in the case of #testBlockRefCountAfterSplits().
In such cases we were getting the same block to be cached again in the bucket 
cache. As part of HBASE-8547 such cases were being handled. 
{code}
String msg = "Caching an already cached block: " + cacheKey;
   msg += ". This is harmless and can happen in rare cases (see 
HBASE-8547)";
   LOG.warn(msg);
{code}
But this is a tricky case where this log msg will be coming only when block 
with same cachekey was completely cached in the bucket cache. If there is a 
case where the block with the same cachekey was not yet completed written to 
bucket cache (by cache writer threads) this this log msg won't come but the 
ramCache key wil prevent the block from again getting cached.
{code}
if (ramCache.putIfAbsent(cacheKey, re) != null) {
  return;
}
{code}
So when ever the block was getting cached once again and it is already in 
backingMap then we were doing a getBlock() to verify if the block is the same 
block. This was internallly adding to the refcount and so those blocks will 
never get removed from the bucket cache queue. ( there is no one to decrement 
the ref count on such cases). 
So I think for this rare cases it is better we do a copy of the block and then 
check if the block is same as the existing one. This should be harmless and 
should help us in doing proper ref counting 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] The second HBase 1.4.0 release candidate (RC1) is available

[Andrew Purtell]

And here are results for PE.

https://bigdatasf.s3.amazonaws.com/assets/pe_1.2.6_1.4.0RC1.png
https://bigdatasf.s3.amazonaws.com/assets/pe_1.2.6_1.4.0RC1.xlsx

[image: Inline image 1]

Similar to the YCSB results, 1.2.6 and 1.4.0 RC1 are close, but 1.4.0 RC1
shows superior results.

Cluster: 1 master, 5 regionservers, 1 client
AWS Instance Type: c3.4xlarge
Hadoop: 2.7.4
OS: Amazon Linux AMI 2017.09, kernel 4.4.51-40.58.amzn1.x86_64

On Tue, Dec 12, 2017 at 5:50 PM, Andrew Purtell  wrote:

> Here are results from a YCSB comparison between 1.2.6 and 1.4.0 RC1:
>
> https://bigdatasf.s3.amazonaws.com/assets/ycsb_1.2.6_to_1.4.0RC1.png
> https://bigdatasf.s3.amazonaws.com/assets/ycsb_1.2.6_to_1.4.0RC1.xlsx
>
> [image: Inline image 1]
>
> They are pretty closely matched but generally 1.4.0 RC1 shows superior
> results.
>
> [R] and [W] mean different things depending on YCSB workload. They will be
> obvious except for Workload F, where [R] is READ and [W] is UPDATE. I
> didn't include numbers for READ-MODIFY-WRITE.
>
> Cluster: 1 master, 5 regionservers, 1 client
> AWS Instance Type: c3.4xlarge
> Hadoop: 2.7.4
> OS: Amazon Linux AMI 2017.09, kernel 4.4.51-40.58.amzn1.x86_64
>
>
> On Fri, Dec 8, 2017 at 5:50 PM, Andrew Purtell 
> wrote:
>
>> The second HBase 1.4.0​ release candidate (RC1) is available for download
>> at https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.0RC1/ and
>> Maven artifacts are available in the temporary repository
>> https://repository.apache.org/content/repositories/orgapachehbase-1186/ .
>>
>> The git tag corresponding to the candidate is '1.4.0RC1' (10b9b9fae6).
>>
>> A detailed source and binary compatibility report for this release is
>> available for your review at https://dist.apache.org/repos/
>> dist/dev/hbase/hbase-1.4.0RC1/hbase-1.3.1-1.4.0RC1_compatibi
>> lity_report.html ​. All reported compatibility issues should comply with
>> policy but please review them carefully.
>>
>> A list of the 660 issues resolved in this release can be found at
>> https://s.apache.org/OErT .
>>
>> The changes since RC0 are:
>>
>> * [HBASE-19180] - Remove unused imports from AlwaysPasses
>> * [HBASE-19373] - Fix Checkstyle error in hbase-annotations
>> * [HBASE-19435] - Reopen Files for ClosedChannelException in
>> BucketCache
>> * [HBASE-19465] - Required httpcore and httpclient jars not included
>> in binary distribution
>> * [HBASE-19467] - rsgroups shell commands don't properly display
>> elapsed time
>>
>> Please try out the candidate and vote +1/0/-1.
>>
>> This vote will be open for at least 72 hours. Unless objection I will try
>> to close it ​Monday December 18, 2017 if we have sufficient votes.
>>
>> Prior to making this announcement I made the following preflight checks
>> to 1.4.0RC0 (3d571827cb):
>>
>> RAT check passes (7u80)
>> Unit test suite passes (8u131)
>> LTT load 1M rows with 100% verification and 20% updates (8u131)
>> PE randomWrite, randomRead, scanRange100 (8u131)
>> 100M rows ITBLL (8u131)
>>
>> and the following preflight checks to 1.4.0RC1 (10b9b9fae6):
>>
>> RAT check passes (7u80)
>> Unit test suite passes (8u131)
>>
>> Between now and when I want to close the vote I'll write up human
>> readable release notes for the release announcement as promised.
>>
>> I also have agreed to run a scale ITBLL test, a performance comparison
>> with 1.2 using YCSB, a  performance comparison with 1.2 using PE, and an
>> analysis of code and allocation hot spot changes from 1.2, all of which I
>> will publish when available and factor in to my vote.
>>
>> --
>> Best regards,
>> Andrew
>>
>>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[jira] [Created] (HBASE-19510) TestDistributedLogSplitting is flakey for AsyncFSWAL

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-19510:
-

 Summary: TestDistributedLogSplitting is flakey for AsyncFSWAL
 Key: HBASE-19510
 URL: https://issues.apache.org/jira/browse/HBASE-19510
 Project: HBase
  Issue Type: Bug
Reporter: Duo Zhang
Assignee: Duo Zhang
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[ANNOUNCE] Apache HBase 1.1.13 is now available for download

[Nick Dimiduk]

The HBase team is happy to announce the immediate availability of HBase
1.1.13! Download it from an Apache mirror near you,
http://www.apache.org/dyn/closer.lua/hbase/, or wire up through the maven
repo.

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

!!! This is the final release from branch-1.1 !!!

HBase 1.1.13 is the thirteenth and final patch release in the HBase 1.1
line, continuing on the theme of bringing a stable, reliable database to
the Hadoop and NoSQL communities. This release includes over 40 resolved
issues since the 1.1.12 release; the majority of these changes are to build
tooling rather than the product itself. Notable product correctness fixes
include:

HBASE-18665 ReversedScannerCallable invokes getRegionLocations incorrectly
HBASE-19052 FixedFileTrailer should recognize CellComparatorImpl class in
branch-1.x

The full list of fixes included in this release is available at
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12341346
and
and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/1.1.13

The relevant checksums files are available at:

https://www.apache.org/dist/hbase/1.1.13/hbase-1.1.13-src.tar.gz.mds
https://www.apache.org/dist/hbase/1.1.13/hbase-1.1.13-bin.tar.gz.mds

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/1.1.13/hbase-1.1.13-src.tar.gz.asc
https://www.apache.org/dist/hbase/1.1.13/hbase-1.1.13-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at: dev@hbase.apache.org
.

Thanks to all the contributors who made this release possible, and to all
the contributors who made this release line a success!

Cheers,
The HBase Dev Team

Re: [VOTE] The second HBase 1.4.0 release candidate (RC1) is available

[Andrew Purtell]

For those who voted on RC0, are you willing to transfer your vote to RC1?
If so, please cast it. There were 5 changes in RC1 not in RC0. See below
for the list.


On Fri, Dec 8, 2017 at 5:50 PM, Andrew Purtell  wrote:

> The second HBase 1.4.0​ release candidate (RC1) is available for download
> at https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.0RC1/ and Maven
> artifacts are available in the temporary repository
> https://repository.apache.org/content/repositories/orgapachehbase-1186/ .
>
> The git tag corresponding to the candidate is '1.4.0RC1' (10b9b9fae6).
>
> A detailed source and binary compatibility report for this release is
> available for your review at https://dist.apache.org/repos/
> dist/dev/hbase/hbase-1.4.0RC1/hbase-1.3.1-1.4.0RC1_
> compatibility_report.html ​. All reported compatibility issues should
> comply with policy but please review them carefully.
>
> A list of the 660 issues resolved in this release can be found at
> https://s.apache.org/OErT .
>
> The changes since RC0 are:
>
> * [HBASE-19180] - Remove unused imports from AlwaysPasses
> * [HBASE-19373] - Fix Checkstyle error in hbase-annotations
> * [HBASE-19435] - Reopen Files for ClosedChannelException in
> BucketCache
> * [HBASE-19465] - Required httpcore and httpclient jars not included
> in binary distribution
> * [HBASE-19467] - rsgroups shell commands don't properly display
> elapsed time
>
> Please try out the candidate and vote +1/0/-1.
>
> This vote will be open for at least 72 hours. Unless objection I will try
> to close it ​Monday December 18, 2017 if we have sufficient votes.
>
> Prior to making this announcement I made the following preflight checks to
> 1.4.0RC0 (3d571827cb):
>
> RAT check passes (7u80)
> Unit test suite passes (8u131)
> LTT load 1M rows with 100% verification and 20% updates (8u131)
> PE randomWrite, randomRead, scanRange100 (8u131)
> 100M rows ITBLL (8u131)
>
> and the following preflight checks to 1.4.0RC1 (10b9b9fae6):
>
> RAT check passes (7u80)
> Unit test suite passes (8u131)
>
> Between now and when I want to close the vote I'll write up human readable
> release notes for the release announcement as promised.
>
> I also have agreed to run a scale ITBLL test, a performance comparison
> with 1.2 using YCSB, a  performance comparison with 1.2 using PE, and an
> analysis of code and allocation hot spot changes from 1.2, all of which I
> will publish when available and factor in to my vote.
>
> --
> Best regards,
> Andrew
>
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[jira] [Created] (HBASE-19509) RSGroupAdminEndpoint#preCreateTable triggers TableNotFoundException

[Ted Yu (JIRA)]

Ted Yu created HBASE-19509:
--

 Summary: RSGroupAdminEndpoint#preCreateTable triggers 
TableNotFoundException
 Key: HBASE-19509
 URL: https://issues.apache.org/jira/browse/HBASE-19509
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0-alpha-4
Reporter: Ted Yu
Priority: Minor


In a cluster where RS Group endpoint is installed, I noticed the following in 
master log:
{code}
2017-12-13 21:42:14,230 ERROR 
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=2] 
master.TableStateManager: Unable to get table ltt-diff state
org.apache.hadoop.hbase.TableNotFoundException: ltt-diff
at 
org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:175)
at 
org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:132)
at 
org.apache.hadoop.hbase.master.assignment.AssignmentManager.isTableDisabled(AssignmentManager.java:345)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveTables(RSGroupAdminServer.java:409)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.assignTableToGroup(RSGroupAdminEndpoint.java:334)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.preCreateTable(RSGroupAdminEndpoint.java:346)
at 
org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:319)
at 
org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:316)
at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:599)
at 
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:672)
at 
org.apache.hadoop.hbase.master.MasterCoprocessorHost.preCreateTable(MasterCoprocessorHost.java:316)
at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:1738)
at 
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:134)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1734)
at 
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:559)
{code}
It seems RSGroupAdminEndpoint#preCreateTable should take into account that the 
table doesn't exist and avoid the extraneous TableNotFoundException .



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-19212) Align coprocessor in Table and AsyncTable

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-19212.
---
Resolution: Later

One invocation is async and the other sync. The way you call CP is different in 
Table and AsyncTable. We'll live with it.

> Align coprocessor in Table and AsyncTable
> -
>
> Key: HBASE-19212
> URL: https://issues.apache.org/jira/browse/HBASE-19212
> Project: HBase
>  Issue Type: Sub-task
>  Components: API, Coprocessors
>Affects Versions: 2.0.0-alpha-4
>Reporter: Peter Somogyi
>Priority: Minor
> Fix For: 2.0.0-beta-2
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Moving To SLF4J and Log4J2

[Stack]

On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma  wrote:

> Seems like good idea:
> - remove long dead dependency
> - a bit cleaner code
> - hadoop also moved to slf4j
>
> Quickly looking at codebase to get idea of amount of work required, here
> are some numbers:
> - LOG.debug : ~1800
> - LOG.trace : ~500
> - LOG.info: ~3000
>
> Looking at this patch (
> https://issues.apache.org/jira/secure/attachment/
> 12901002/HBASE-19449.1.patch),
> seemed like tedious and repetitive task, was wondering if someone has
> automated it already.
> Looks like this can help reduce a huge part of grunt work:
> https://www.slf4j.org/migrator.html.
>
> But before progressing, as a basic validation, can we see:
> - an example of old vs new log lines (that there is no diff, or we are
> comfortable with what's there)
> - an example of changes in properties file
>
> Maybe starting with hbase-examples module for quick POC.
>
>
I like your suggestion Appy,
S



> -- Appy
>

Re: Can we always add new regions as CLOSED?

[Stack]

On Wed, Dec 13, 2017 at 1:14 AM, Apekshit Sharma  wrote:

> Hi folks,
>
> Was debugging TestTruncateTableProcedure when starting thinking about this.
> (That's one mean test! What nice fault tolerant tests!)
>
> So the specific case: If we fail after adding new regions to meta (
> TRUNCATE_TABLE_ADD_TO_META
>  server/src/main/java/org/apache/hadoop/hbase/master/procedure/
> TruncateTableProcedure.java#L127>),
> then on recovery, AM assumes those regions with null state as offline and
> begins assigning them by itself which is wrong since truncate action is not
> complete (and it'll try to assign them too on recovery, and there are locks
> to avoid simultaneous assigns etc.)
> Simple fix is, add regions with initial state as CLOSED.
>
> Then looking in other places, CreateTableProcedure seems like it should
> suffer the same fate (CREATE_TABLE_ADD_TO_META
>  2e587f5181/hbase-server/src/main/java/org/apache/hadoop/
> hbase/master/procedure/CreateTableProcedure.java#L104>).
> Should we add region as CLOSED there too?
> (Weird part is, it's not failing, looking into it)
>
> So the main question is, shouldn't we always add new regions to meta with
> state as CLOSED?
>
>
I've been here recently. OFFLINE?



> Whatever operation is adding them will also be opening them if needed,
> right? And no operation should be relying on this weird AM assumption to
> complete it's half done job.
>
> Food for thought - Some operations adding regions are: truncate table,
> create table, modify table, clone snapshot, restore snapshot.
>
> Can you imagine a case where not adding a new region as CLOSED makes sense?
>
>
None. There is adding and then AMv2 takes control.

These tests that double-kill to ensure each step recoverable are great,
yeah, at finding dirty bugs.

I think truncate table though a little silly. If you proposed killing it,
I'd +1 it.

St.Ack


> -- Appy
>

[jira] [Created] (HBASE-19508) ReadOnlyConfiguration throws exception if any Configuration in current context calls addDefautlResource

[stack (JIRA)]

stack created HBASE-19508:
-

 Summary: ReadOnlyConfiguration throws exception if any 
Configuration in current context calls addDefautlResource
 Key: HBASE-19508
 URL: https://issues.apache.org/jira/browse/HBASE-19508
 Project: HBase
  Issue Type: Bug
  Components: conf
Reporter: stack
Assignee: stack
 Fix For: 2.0.0-beta-1


Reported by [~psomogyi] up on HBASE-19295, the issue that added CpEnv returning 
a read-only Configuration.

Issue is that hadoop Configuration registers every created Configuration in a 
static REGISTRY on construction. A call to addDefaultResource -- e.g. sqoop 
does this in tests to get its configurations into the mix -- results in a call 
to every registered Configurations loadResource.. which is disallowed in our 
read-only Configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HBASE-19484) The value array written by ExtendedCell#write is out of bounds

[Chia-Ping Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai reopened HBASE-19484:


getValueArray() can return null so we should add the length check before 
writing the value array

> The value array written by ExtendedCell#write is out of bounds
> --
>
> Key: HBASE-19484
> URL: https://issues.apache.org/jira/browse/HBASE-19484
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19484.v0.patch
>
>
> I move the impl of IndividualBytesFieldCell#write to ExtendedCell so as to 
> make it be a default method (see HBASE-19430), but I didn't notice that the 
> value array doesn't be handled correctly.
> {code:title=ExtendedCell}
>   default int write(OutputStream out, boolean withTags) throws IOException {
> // Key length and then value length
> ByteBufferUtils.putInt(out, KeyValueUtil.keyLength(this));
> ByteBufferUtils.putInt(out, getValueLength());
> // Key
> PrivateCellUtil.writeFlatKey(this, out);
> // Value
> out.write(getValueArray());  // <-- here
> // Tags length and tags byte array
> if (withTags && getTagsLength() > 0) {
>   // Tags length
>   out.write((byte)(0xff & (getTagsLength() >> 8)));
>   out.write((byte)(0xff & getTagsLength()));
>   // Tags byte array
>   out.write(getTagsArray(), getTagsOffset(), getTagsLength());
> }
> return getSerializedSize(withTags);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19507) get Mob by rowkey only return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD

[WangYuan (JIRA)]

WangYuan created HBASE-19507:


 Summary: get Mob by rowkey only return error value when run 
compact_mob or major_compact_mob after change MOB_THRESHOLD
 Key: HBASE-19507
 URL: https://issues.apache.org/jira/browse/HBASE-19507
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: WangYuan






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19506) Support variable sized chunks from ChunkCreator

[Anastasia Braginsky (JIRA)]

Anastasia Braginsky created HBASE-19506:
---

 Summary: Support variable sized chunks from ChunkCreator
 Key: HBASE-19506
 URL: https://issues.apache.org/jira/browse/HBASE-19506
 Project: HBase
  Issue Type: Sub-task
Reporter: Anastasia Braginsky


When CellChunkMap is created it allocates a special index chunk (or chunks) 
where array of cell-representations is stored. When the number of 
cell-representations is small, it is preferable to allocate a chunk smaller 
than a default value which is 2MB.

On the other hand, those "non-standard size" chunks can not be used in pool. 
On-demand allocations in off-heap are costly. So this JIRA is about to 
investigate the trade of between memory usage and the final performance. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19505) Disable ByteBufferPool by default at HM

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-19505:
--

 Summary: Disable ByteBufferPool by default at HM
 Key: HBASE-19505
 URL: https://issues.apache.org/jira/browse/HBASE-19505
 Project: HBase
  Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0-beta-1






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Can we always add new regions as CLOSED?

[Apekshit Sharma]

Hi folks,

Was debugging TestTruncateTableProcedure when starting thinking about this.
(That's one mean test! What nice fault tolerant tests!)

So the specific case: If we fail after adding new regions to meta (
TRUNCATE_TABLE_ADD_TO_META
),
then on recovery, AM assumes those regions with null state as offline and
begins assigning them by itself which is wrong since truncate action is not
complete (and it'll try to assign them too on recovery, and there are locks
to avoid simultaneous assigns etc.)
Simple fix is, add regions with initial state as CLOSED.

Then looking in other places, CreateTableProcedure seems like it should
suffer the same fate (CREATE_TABLE_ADD_TO_META
).
Should we add region as CLOSED there too?
(Weird part is, it's not failing, looking into it)

So the main question is, shouldn't we always add new regions to meta with
state as CLOSED?

Whatever operation is adding them will also be opening them if needed,
right? And no operation should be relying on this weird AM assumption to
complete it's half done job.

Food for thought - Some operations adding regions are: truncate table,
create table, modify table, clone snapshot, restore snapshot.

Can you imagine a case where not adding a new region as CLOSED makes sense?

-- Appy