[jira] [Created] (HBASE-18682) TestCompactingMemStore#testFlatteningToCellChunkMap fails in master

[Chia-Ping Tsai (JIRA)]

Chia-Ping Tsai created HBASE-18682:
--

 Summary: TestCompactingMemStore#testFlatteningToCellChunkMap fails 
in master
 Key: HBASE-18682
 URL: https://issues.apache.org/jira/browse/HBASE-18682
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Chia-Ping Tsai
Priority: Minor
 Fix For: 2.0.0


The UT works well on my centos(64-bits) but it always fails on my 
ubuntu(64-bits). 

{code}
Running org.apache.hadoop.hbase.regionserver.TestCompactingMemStore
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.998 sec <<< 
FAILURE! - in org.apache.hadoop.hbase.regionserver.TestCompactingMemStore
testFlatteningToCellChunkMap(org.apache.hadoop.hbase.regionserver.TestCompactingMemStore)
  Time elapsed: 1.64 sec  <<< FAILURE!
java.lang.AssertionError: expected:<920> but was:<952>
at 
org.apache.hadoop.hbase.regionserver.TestCompactingMemStore.testFlatteningToCellChunkMap(TestCompactingMemStore.java:609)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18681) Introduce a new constructor for StoreScanner for MOB

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-18681:
-

 Summary: Introduce a new constructor for StoreScanner for MOB
 Key: HBASE-18681
 URL: https://issues.apache.org/jira/browse/HBASE-18681
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0-alpha-2, 3.0.0
Reporter: Duo Zhang
Assignee: Duo Zhang
 Fix For: 3.0.0, 2.0.0-alpha-3


The mob compactor will create {{StoreScanner}} with several {{StoreFile}} s 
directly, so it can not use the constructors for the normal compaction as they 
all need a {{Store}} instance. But this constructor is only used by UTs and is 
expected to be removed, so let's introduce a new one for mob.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18680) Document things to consider / document when changing dependencies

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-18680:
---

 Summary: Document things to consider / document when changing 
dependencies
 Key: HBASE-18680
 URL: https://issues.apache.org/jira/browse/HBASE-18680
 Project: HBase
  Issue Type: Task
  Components: community, documentation
Reporter: Sean Busbey


While reviewing HBASE-18674 ("upgrade hbase to commons-lang 3"), I realized 
that my questions are the same as I'd give for any dependency change. We should 
try to update our docs to proactively push folks on checking whenever a 
dependency changes (paraphrased):

{quote}
# What's the license of the direct dependency overall?
# Does it contain parts that are under a different license (esp if the overall 
is ALv2)?
# If it is ALv2 does it contain a NOTICE file? If so, do we need to update our 
NOTICE file based on it?
# Does it change the set of transitive dependencies? If so, apply the same 
questions above to that set of changes.
{quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18448) EndPoint example for refreshing HFiles for stores

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-18448.
---
  Resolution: Fixed
Release Note: Adds a new RefreshHFiles Coprocessor Endpoint example. 
Includes client and serverside-endpoint that iterates region Stores to call 
#refreshStoreFiles.

> EndPoint example  for refreshing HFiles for stores
> --
>
> Key: HBASE-18448
> URL: https://issues.apache.org/jira/browse/HBASE-18448
> Project: HBase
>  Issue Type: Sub-task
>  Components: Coprocessors
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Ajay Jadhav
>Assignee: Ajay Jadhav
>Priority: Minor
> Fix For: 1.5.0, 2.0.0-alpha-3
>
> Attachments: HBASE-18448.branch-1.001.patch, 
> HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, 
> HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, 
> HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, 
> HBASE-18448.master.001.patch, HBASE-18448.master.002.patch
>
>
> In the case where multiple HBase clusters are sharing a common rootDir, even 
> after flushing the data from
> one cluster doesn't mean that other clusters (replicas) will automatically 
> pick the new HFile. Through this patch,
> we are exposing the refresh HFiles API which when issued from a replica will 
> update the in-memory file handle list
> with the newly added file.
> This allows replicas to be consistent with the data written through the 
> primary cluster. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.1RC0

[Stack]

+1 from me.

On Wed, Aug 23, 2017 at 3:46 PM, Stack  wrote:

> This is a minor update to hbase-thirdparty, our little side project of
> relocated popular includes such as guava, protobuf, and netty (See [1] for
> more on what hbase-thirdparty is).
>
> hbase-thirdparty 1.0.1RC0 is available at:
>
>  https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty/1.0.1RC0/
>
> Maven artifacts are available in the staging repository:
>
>  https://repository.apache.org/content/repositories/orgapachehbase-1174
>
> Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS file
> http://www-us.apache.org/dist/hbase/KEYS.
>
> I tagged this RC as 1.0.1RC0 at in the hbase-thirdparty repo at
> e07089bee6f51aec65de932b302894507903bd6e https://git-wip-
> us.apache.org/repos/asf/hbase-thirdparty
>
> This minor release includes three fixes:
>
> HBASE-18321 [hbase-thirdparty] Fix generation of META-INF/DEPENDENCIES to
> include dependency list and versions
> HBASE-18666 [hbase-thirdparty] Exclude errorprone annotation
> com.google.errorprone.annotations.CanIgnoreReturnValue
> HBASE-18313 [hbase-thirdparty] Produce src jars/tgz
>
> The first fixes an issue found during VOTE on 1.0.0, the second makes it
> so errorprone build will work back in mainline hbase, and the third has us
> producing src jars when we publish to maven.
>
> VOTE lasts 72 hours.
>
> Thanks,
> St.Ack
>
>
> 1. http://hbase.apache.org/book.html#thirdparty
>

[jira] [Created] (HBASE-18679) YARN may null Counters object and cause an NPE in ITBLL

[Josh Elser (JIRA)]

Josh Elser created HBASE-18679:
--

 Summary: YARN may null Counters object and cause an NPE in ITBLL
 Key: HBASE-18679
 URL: https://issues.apache.org/jira/browse/HBASE-18679
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Trivial
 Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 1.1.13


YARN has a configuration limit to the number of counters that a job can create 
(to avoid some bad job from DDOS'ing the service).

When running ITBLL, we ran into this limit due to the default configuration 
rather low at the time. When YARN notices that the counter limit has been 
exceeded, it nulls out the Counters object obtained by the Job.

Presently in ITBLL, we have a few places where we (reasonably ;)) assume that 
the Counters object would be non-null. Can easily fix this with a null-check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HBASE-17249) Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange reference as the parameter instead of creating a new setColumnFamilyTimeRange instance

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-17249:
---

Reopening. The changes made here contend with another long-standing wish over 
in HBASE-15284 where we want to NOT let users create TimeRanges explicitly.

[~huaxiang] were you just code-reading?

We need to clean up what our intent for TimeRange is. Opening this in meantime. 
Related issue is HBASE-15284. Thanks. 

> Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange 
> reference as the parameter instead of creating a new setColumnFamilyTimeRange 
> instance
> --
>
> Key: HBASE-17249
> URL: https://issues.apache.org/jira/browse/HBASE-17249
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17249-master-001.patch, 
> HBASE-17249-master-002.patch, HBASE-17249-master-003.patch
>
>
> Going through the code, found For Get/Scan's 
> setTimeRange/setColumnFamilyTimeRange, it can use  TimeRange as reference 
> instead of creating a new one.
> Reference:
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L500
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L506
> We can implement this in a similar way as filter:
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L510
> I checked it is same with branch-1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18678) Add more metrics to the Procedure MBean; currently one metric only for a loci of macro churn

[stack (JIRA)]

stack created HBASE-18678:
-

 Summary: Add more metrics to the Procedure MBean; currently one 
metric only for a loci of macro churn
 Key: HBASE-18678
 URL: https://issues.apache.org/jira/browse/HBASE-18678
 Project: HBase
  Issue Type: Improvement
  Components: proc-v2
Reporter: stack


I just noticed the procedurev2 MBean published by the master and how little it 
exposes:

{code}{
name: "Hadoop:service=HBase,name=Master,sub=Procedure",
modelerType: "Master,sub=Procedure",
tag.Context: "master",
tag.Hostname: "ve0524.halxg.cloudera.com",
numMasterWALs: 1
},
{code]

There is a bunch of other stuff we could add in here... to offer insight on the 
Master's reactor



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18677) type in namesapce docs

[Mike Drob (JIRA)]

Mike Drob created HBASE-18677:
-

 Summary: type in namesapce docs
 Key: HBASE-18677
 URL: https://issues.apache.org/jira/browse/HBASE-18677
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Mike Drob
Priority: Trivial


In the docs at http://hbase.apache.org/book.html#_namespace - "Region server 
groups (HBASE-6721) - A namespace/table can be pinned onto a subset of 
RegionServers thus guaranteeing a course level of isolation."

Should be "coarse"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18676) Update branch 1.3 pom version

[Mike Drob (JIRA)]

Mike Drob created HBASE-18676:
-

 Summary: Update branch 1.3 pom version
 Key: HBASE-18676
 URL: https://issues.apache.org/jira/browse/HBASE-18676
 Project: HBase
  Issue Type: Task
  Components: build
Affects Versions: 1.3.1
Reporter: Mike Drob
Priority: Minor
 Fix For: 1.3.2


Branch 1.3 currently has version=1.3.1 set in the poms. I think this should be 
1.3.2-SNAPSHOT, in case anybody tries to deploy nightlies for any reason.

WDYT [~mantonov]?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18509) [HLC] Finishing cleanups

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy resolved HBASE-18509.
--
Resolution: Fixed

> [HLC] Finishing cleanups
> 
>
> Key: HBASE-18509
> URL: https://issues.apache.org/jira/browse/HBASE-18509
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Appy
>Assignee: Amit Patel
> Attachments: HBASE-18509.HBASE-14070.HLC.001.patch, 
> HBASE-18509.HBASE-14070.HLC.002.patch, HBASE-18509.HBASE-14070.HLC.003.patch, 
> HBASE-18509.HBASE-14070.HLC.004.patch, HBASE-18509.HBASE-14070.HLC.005.patch
>
>
> Track all types of cleanups here:
> - (done in 001.patch) --Rename classes to more consistent naming: 
> SystemClock, SystemMonotonicClock, HybridLogicalClock--
> - (done in 001.patch) --Move implementations out from Clock interface. It's a 
> simple interface of 6 fns but very overloaded right now with everything put 
> inside it.--
> - (done in 004.patch) --Maybe encapsulate all clocks in RS/Master into a new 
> class. Then RSServices can just have getClocks() function.--
> {noformat}
> class Clocks {
>   // all 3 types of clocks.
>   // Fns:
>   - update(ClockType, timestamp)
>   - updateAll(List\)
>   - int64 now(ClockType)
>   - List\ nowAll()
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18675) Making {max,min}SessionTimeout configurable for MiniZooKeeperCluster

[Cesar Delgado (JIRA)]

Cesar Delgado created HBASE-18675:
-

 Summary: Making {max,min}SessionTimeout configurable for 
MiniZooKeeperCluster
 Key: HBASE-18675
 URL: https://issues.apache.org/jira/browse/HBASE-18675
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Reporter: Cesar Delgado
Assignee: Cesar Delgado
Priority: Minor


Right now the mini cluster on application developers laptops keep crashing when 
the laptop goes to sleep because Zookeeper times out. We've seen this for a 
while and [~ekoontz] had worked on it before. Now that we tried to upgrade it's 
bitten us so we'd like to push this up as I'm sure we're not the only ones 
getting bitten by this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18674) upgrade hbase to commons-lang3

[Umesh Agashe (JIRA)]

Umesh Agashe created HBASE-18674:


 Summary: upgrade hbase to commons-lang3
 Key: HBASE-18674
 URL: https://issues.apache.org/jira/browse/HBASE-18674
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0-alpha-2
Reporter: Umesh Agashe
Assignee: Umesh Agashe
 Fix For: 2.0.0


upgrade hbase to use commons-lang 3.6



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

[stack]

On Thu, Aug 3, 2017 at 11:05 PM, stack  wrote:

> Thanks Zach for clarification. Let me work up a list and then come back to
> this thread.  Jira needs an edit pass to.
>
>
JIRA editing reflecting incompatibles has been coming along. It is still a
work-in-progress though. Meantime, let me air out here the incompatibles as
I encounter them. The complete list may never be known (smile) and the best
knowledge will only be had on the day before release which will be giving
notice too late for folks to do much about it.

Here is a known breaking incompatible that just went in: HBASE-15982
"Interface ReplicationEndpoint extends Guava's Service". See the release
note for why the breakage unavoidable. Anyone who has implemented our
Replication Endpoint will have a refactor on their hands deploying their
replication machine atop hbase2 (Side note: Estebans' initial work has it
that hbase native replication seems to work going from hbase1 to hbase2 and
even the other way around; more to do). Upside, if there is the will,
now is the time for HBASE-10504 "Define Replication Interface". It is late
in the game but am up for helping out if interest.

Here is a breaking compatible we *could* (maybe) avoid but the amount of
work involved in the workaround and how the patch-up would muddy our
messaging around protobufs in the HBase public API is not worth the price
IMO. What do you all think? The issue is HBASE-15607 "Remove PB references
from Admin for 2.0". The plan is to (re-)commit Ram's change. We'd then
tell users that you cannot administer an hbase2 cluster using an hbase1
client.

Coprocessors will require a refactor to deploy on hbase2. More detail to
follow.

St.Ack






> S
>
> On Aug 3, 2017 23:54, "Zach York"  wrote:
>
> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack  wrote:
>
> > On Wed, Aug 2, 2017 at 5:25 PM, Zach York 
> > wrote:
> >
> > > Do we know what the major pain points for migration are? Can we discuss
> > > that/get a list going?
> > >
> > >
> > Here's a few in outline:
> >
> > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> data
> > whether from HDFS or ZooKeeper or off the wire.
> > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > cluster; no holes in the API or unintelligible serializations.
> > + There is then the little dance that has us rolling restart from an
> > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> > assign regions to the new hbase-2.x regionservers as they come on line.
> > TBD.
> >
> > Is this what you mean sir?
> >
> > S
> >
> >
> > > I think without that knowledge it is hard (for me at least :) ) to
> > > determine where we should set our sights in terms of migration.
> > >
> > > Thanks,
> > > Zach
> > >
> > > On Wed, Aug 2, 2017 at 4:38 PM, Stack  wrote:
> > >
> > > > What are our expectations regards compatibility between hbase1 and
> > > hbase2?
> > > >
> > > > Lets have a chat about it. Here are some goal posts.
> > > >
> > > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> > No
> > > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> > =>
> > > > 2.0?).
> > > > + You do NOT have to upgrade to the latest release of hbase1 to
> migrate
> > > to
> > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > hbase2.
> > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > > watching for region split on RegionServer no longer makes sense given
> > > > Master runs all splits now.
> > > > + An hbase1 client can run against an hbase2 cluster but it will only
> > be
> > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > > admin
> > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > some
> > > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> > > (See
> > > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > > purging this stuff from our Admin API.
> > > >
> > > > What you all think?
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
>
>

[jira] [Reopened] (HBASE-18448) EndPoint example for refreshing HFiles for stores

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John reopened HBASE-18448:


> EndPoint example  for refreshing HFiles for stores
> --
>
> Key: HBASE-18448
> URL: https://issues.apache.org/jira/browse/HBASE-18448
> Project: HBase
>  Issue Type: Sub-task
>  Components: Coprocessors
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Ajay Jadhav
>Assignee: Ajay Jadhav
>Priority: Minor
> Fix For: 3.0.0, 2.0.0-alpha-3
>
> Attachments: HBASE-18448.branch-1.001.patch, 
> HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, 
> HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, 
> HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, 
> HBASE-18448.master.001.patch, HBASE-18448.master.002.patch
>
>
> In the case where multiple HBase clusters are sharing a common rootDir, even 
> after flushing the data from
> one cluster doesn't mean that other clusters (replicas) will automatically 
> pick the new HFile. Through this patch,
> we are exposing the refresh HFiles API which when issued from a replica will 
> update the in-memory file handle list
> with the newly added file.
> This allows replicas to be consistent with the data written through the 
> primary cluster. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18673) Some more unwanted reference to unshaded PB classes

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-18673:
--

 Summary: Some more unwanted reference to unshaded PB classes
 Key: HBASE-18673
 URL: https://issues.apache.org/jira/browse/HBASE-18673
 Project: HBase
  Issue Type: Improvement
Reporter: Anoop Sam John
Priority: Minor


ProtobufLogReader  - Seems using Unshaded CIS which seems a miss
HBaseFsck - Some public methods throw PB ServiceException. Its strange. No code 
within that throws this.
Public exposed PBType class.  I dont know what this type allows the users to 
do. Make their own Type?  If so the Unshaded might be ok.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18672) Drop jdk7 profile from build

[Mike Drob (JIRA)]

Mike Drob created HBASE-18672:
-

 Summary: Drop jdk7 profile from build
 Key: HBASE-18672
 URL: https://issues.apache.org/jira/browse/HBASE-18672
 Project: HBase
  Issue Type: Bug
  Components: build
Reporter: Mike Drob
 Fix For: 2.0.0


We have a jdk7 profile in hbase-annotations/pom.xml - we don't support building 
with jdk7 anyway though so we should remove it. Maybe we can remove the java 8 
profile too, since that's the minimum default?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18671) Support Append/Increment in rest api

[Guangxu Cheng (JIRA)]

Guangxu Cheng created HBASE-18671:
-

 Summary: Support Append/Increment in rest api
 Key: HBASE-18671
 URL: https://issues.apache.org/jira/browse/HBASE-18671
 Project: HBase
  Issue Type: Bug
  Components: REST
Affects Versions: 2.0.0-alpha-2, 3.0.0, 1.4.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng


Recently we need to call append interface through the rest api, but rest api 
has not yet implemented it.
We implemented it and it is stable in the production environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)