[jira] [Created] (HBASE-18682) TestCompactingMemStore#testFlatteningToCellChunkMap fails in master
Chia-Ping Tsai created HBASE-18682: -- Summary: TestCompactingMemStore#testFlatteningToCellChunkMap fails in master Key: HBASE-18682 URL: https://issues.apache.org/jira/browse/HBASE-18682 Project: HBase Issue Type: Bug Components: test Reporter: Chia-Ping Tsai Priority: Minor Fix For: 2.0.0 The UT works well on my centos(64-bits) but it always fails on my ubuntu(64-bits). {code} Running org.apache.hadoop.hbase.regionserver.TestCompactingMemStore Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.998 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestCompactingMemStore testFlatteningToCellChunkMap(org.apache.hadoop.hbase.regionserver.TestCompactingMemStore) Time elapsed: 1.64 sec <<< FAILURE! java.lang.AssertionError: expected:<920> but was:<952> at org.apache.hadoop.hbase.regionserver.TestCompactingMemStore.testFlatteningToCellChunkMap(TestCompactingMemStore.java:609) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18681) Introduce a new constructor for StoreScanner for MOB
Duo Zhang created HBASE-18681: - Summary: Introduce a new constructor for StoreScanner for MOB Key: HBASE-18681 URL: https://issues.apache.org/jira/browse/HBASE-18681 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0-alpha-2, 3.0.0 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0, 2.0.0-alpha-3 The mob compactor will create {{StoreScanner}} with several {{StoreFile}} s directly, so it can not use the constructors for the normal compaction as they all need a {{Store}} instance. But this constructor is only used by UTs and is expected to be removed, so let's introduce a new one for mob. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18680) Document things to consider / document when changing dependencies
Sean Busbey created HBASE-18680: --- Summary: Document things to consider / document when changing dependencies Key: HBASE-18680 URL: https://issues.apache.org/jira/browse/HBASE-18680 Project: HBase Issue Type: Task Components: community, documentation Reporter: Sean Busbey While reviewing HBASE-18674 ("upgrade hbase to commons-lang 3"), I realized that my questions are the same as I'd give for any dependency change. We should try to update our docs to proactively push folks on checking whenever a dependency changes (paraphrased): {quote} # What's the license of the direct dependency overall? # Does it contain parts that are under a different license (esp if the overall is ALv2)? # If it is ALv2 does it contain a NOTICE file? If so, do we need to update our NOTICE file based on it? # Does it change the set of transitive dependencies? If so, apply the same questions above to that set of changes. {quote} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-18448. --- Resolution: Fixed Release Note: Adds a new RefreshHFiles Coprocessor Endpoint example. Includes client and serverside-endpoint that iterates region Stores to call #refreshStoreFiles. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.5.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.1RC0
+1 from me. On Wed, Aug 23, 2017 at 3:46 PM, Stack wrote: > This is a minor update to hbase-thirdparty, our little side project of > relocated popular includes such as guava, protobuf, and netty (See [1] for > more on what hbase-thirdparty is). > > hbase-thirdparty 1.0.1RC0 is available at: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty/1.0.1RC0/ > > Maven artifacts are available in the staging repository: > > https://repository.apache.org/content/repositories/orgapachehbase-1174 > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS file > http://www-us.apache.org/dist/hbase/KEYS. > > I tagged this RC as 1.0.1RC0 at in the hbase-thirdparty repo at > e07089bee6f51aec65de932b302894507903bd6e https://git-wip- > us.apache.org/repos/asf/hbase-thirdparty > > This minor release includes three fixes: > > HBASE-18321 [hbase-thirdparty] Fix generation of META-INF/DEPENDENCIES to > include dependency list and versions > HBASE-18666 [hbase-thirdparty] Exclude errorprone annotation > com.google.errorprone.annotations.CanIgnoreReturnValue > HBASE-18313 [hbase-thirdparty] Produce src jars/tgz > > The first fixes an issue found during VOTE on 1.0.0, the second makes it > so errorprone build will work back in mainline hbase, and the third has us > producing src jars when we publish to maven. > > VOTE lasts 72 hours. > > Thanks, > St.Ack > > > 1. http://hbase.apache.org/book.html#thirdparty >
[jira] [Created] (HBASE-18679) YARN may null Counters object and cause an NPE in ITBLL
Josh Elser created HBASE-18679: -- Summary: YARN may null Counters object and cause an NPE in ITBLL Key: HBASE-18679 URL: https://issues.apache.org/jira/browse/HBASE-18679 Project: HBase Issue Type: Bug Components: integration tests Reporter: Josh Elser Assignee: Josh Elser Priority: Trivial Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 1.1.13 YARN has a configuration limit to the number of counters that a job can create (to avoid some bad job from DDOS'ing the service). When running ITBLL, we ran into this limit due to the default configuration rather low at the time. When YARN notices that the counter limit has been exceeded, it nulls out the Counters object obtained by the Job. Presently in ITBLL, we have a few places where we (reasonably ;)) assume that the Counters object would be non-null. Can easily fix this with a null-check. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (HBASE-17249) Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange reference as the parameter instead of creating a new setColumnFamilyTimeRange instance
[ https://issues.apache.org/jira/browse/HBASE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-17249: --- Reopening. The changes made here contend with another long-standing wish over in HBASE-15284 where we want to NOT let users create TimeRanges explicitly. [~huaxiang] were you just code-reading? We need to clean up what our intent for TimeRange is. Opening this in meantime. Related issue is HBASE-15284. Thanks. > Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange > reference as the parameter instead of creating a new setColumnFamilyTimeRange > instance > -- > > Key: HBASE-17249 > URL: https://issues.apache.org/jira/browse/HBASE-17249 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17249-master-001.patch, > HBASE-17249-master-002.patch, HBASE-17249-master-003.patch > > > Going through the code, found For Get/Scan's > setTimeRange/setColumnFamilyTimeRange, it can use TimeRange as reference > instead of creating a new one. > Reference: > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L500 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L506 > We can implement this in a similar way as filter: > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L510 > I checked it is same with branch-1. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18678) Add more metrics to the Procedure MBean; currently one metric only for a loci of macro churn
stack created HBASE-18678: - Summary: Add more metrics to the Procedure MBean; currently one metric only for a loci of macro churn Key: HBASE-18678 URL: https://issues.apache.org/jira/browse/HBASE-18678 Project: HBase Issue Type: Improvement Components: proc-v2 Reporter: stack I just noticed the procedurev2 MBean published by the master and how little it exposes: {code}{ name: "Hadoop:service=HBase,name=Master,sub=Procedure", modelerType: "Master,sub=Procedure", tag.Context: "master", tag.Hostname: "ve0524.halxg.cloudera.com", numMasterWALs: 1 }, {code] There is a bunch of other stuff we could add in here... to offer insight on the Master's reactor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18677) type in namesapce docs
Mike Drob created HBASE-18677: - Summary: type in namesapce docs Key: HBASE-18677 URL: https://issues.apache.org/jira/browse/HBASE-18677 Project: HBase Issue Type: Improvement Components: documentation Reporter: Mike Drob Priority: Trivial In the docs at http://hbase.apache.org/book.html#_namespace - "Region server groups (HBASE-6721) - A namespace/table can be pinned onto a subset of RegionServers thus guaranteeing a course level of isolation." Should be "coarse" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18676) Update branch 1.3 pom version
Mike Drob created HBASE-18676: - Summary: Update branch 1.3 pom version Key: HBASE-18676 URL: https://issues.apache.org/jira/browse/HBASE-18676 Project: HBase Issue Type: Task Components: build Affects Versions: 1.3.1 Reporter: Mike Drob Priority: Minor Fix For: 1.3.2 Branch 1.3 currently has version=1.3.1 set in the poms. I think this should be 1.3.2-SNAPSHOT, in case anybody tries to deploy nightlies for any reason. WDYT [~mantonov]? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18509) [HLC] Finishing cleanups
[ https://issues.apache.org/jira/browse/HBASE-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy resolved HBASE-18509. -- Resolution: Fixed > [HLC] Finishing cleanups > > > Key: HBASE-18509 > URL: https://issues.apache.org/jira/browse/HBASE-18509 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Amit Patel > Attachments: HBASE-18509.HBASE-14070.HLC.001.patch, > HBASE-18509.HBASE-14070.HLC.002.patch, HBASE-18509.HBASE-14070.HLC.003.patch, > HBASE-18509.HBASE-14070.HLC.004.patch, HBASE-18509.HBASE-14070.HLC.005.patch > > > Track all types of cleanups here: > - (done in 001.patch) --Rename classes to more consistent naming: > SystemClock, SystemMonotonicClock, HybridLogicalClock-- > - (done in 001.patch) --Move implementations out from Clock interface. It's a > simple interface of 6 fns but very overloaded right now with everything put > inside it.-- > - (done in 004.patch) --Maybe encapsulate all clocks in RS/Master into a new > class. Then RSServices can just have getClocks() function.-- > {noformat} > class Clocks { > // all 3 types of clocks. > // Fns: > - update(ClockType, timestamp) > - updateAll(List\) > - int64 now(ClockType) > - List\ nowAll() > } > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18675) Making {max,min}SessionTimeout configurable for MiniZooKeeperCluster
Cesar Delgado created HBASE-18675: - Summary: Making {max,min}SessionTimeout configurable for MiniZooKeeperCluster Key: HBASE-18675 URL: https://issues.apache.org/jira/browse/HBASE-18675 Project: HBase Issue Type: Bug Components: Zookeeper Reporter: Cesar Delgado Assignee: Cesar Delgado Priority: Minor Right now the mini cluster on application developers laptops keep crashing when the laptop goes to sleep because Zookeeper times out. We've seen this for a while and [~ekoontz] had worked on it before. Now that we tried to upgrade it's bitten us so we'd like to push this up as I'm sure we're not the only ones getting bitten by this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18674) upgrade hbase to commons-lang3
Umesh Agashe created HBASE-18674: Summary: upgrade hbase to commons-lang3 Key: HBASE-18674 URL: https://issues.apache.org/jira/browse/HBASE-18674 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0-alpha-2 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0 upgrade hbase to use commons-lang 3.6 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
On Thu, Aug 3, 2017 at 11:05 PM, stack wrote: > Thanks Zach for clarification. Let me work up a list and then come back to > this thread. Jira needs an edit pass to. > > JIRA editing reflecting incompatibles has been coming along. It is still a work-in-progress though. Meantime, let me air out here the incompatibles as I encounter them. The complete list may never be known (smile) and the best knowledge will only be had on the day before release which will be giving notice too late for folks to do much about it. Here is a known breaking incompatible that just went in: HBASE-15982 "Interface ReplicationEndpoint extends Guava's Service". See the release note for why the breakage unavoidable. Anyone who has implemented our Replication Endpoint will have a refactor on their hands deploying their replication machine atop hbase2 (Side note: Estebans' initial work has it that hbase native replication seems to work going from hbase1 to hbase2 and even the other way around; more to do). Upside, if there is the will, now is the time for HBASE-10504 "Define Replication Interface". It is late in the game but am up for helping out if interest. Here is a breaking compatible we *could* (maybe) avoid but the amount of work involved in the workaround and how the patch-up would muddy our messaging around protobufs in the HBase public API is not worth the price IMO. What do you all think? The issue is HBASE-15607 "Remove PB references from Admin for 2.0". The plan is to (re-)commit Ram's change. We'd then tell users that you cannot administer an hbase2 cluster using an hbase1 client. Coprocessors will require a refactor to deploy on hbase2. More detail to follow. St.Ack > S > > On Aug 3, 2017 23:54, "Zach York" wrote: > > This kinda helps, but these seem more like expectations. I was going more > for things like HFile format changed, meta table structure changed, > coprocessor implementations changed (these are just examples, I don't know > if any of these actually changed). > > More technical differences between branch-1 and branch-2 which then can > help us get the right expectations for compatibility. > > On Wed, Aug 2, 2017 at 6:34 PM, Stack wrote: > > > On Wed, Aug 2, 2017 at 5:25 PM, Zach York > > wrote: > > > > > Do we know what the major pain points for migration are? Can we discuss > > > that/get a list going? > > > > > > > > Here's a few in outline: > > > > + There is issue of formats, of hbase-2.x being able to read hbase-1.x > data > > whether from HDFS or ZooKeeper or off the wire. > > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x > > cluster; no holes in the API or unintelligible serializations. > > + There is then the little dance that has us rolling restart from an > > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it > will > > assign regions to the new hbase-2.x regionservers as they come on line. > > TBD. > > > > Is this what you mean sir? > > > > S > > > > > > > I think without that knowledge it is hard (for me at least :) ) to > > > determine where we should set our sights in terms of migration. > > > > > > Thanks, > > > Zach > > > > > > On Wed, Aug 2, 2017 at 4:38 PM, Stack wrote: > > > > > > > What are our expectations regards compatibility between hbase1 and > > > hbase2? > > > > > > > > Lets have a chat about it. Here are some goal posts. > > > > > > > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2. > > No > > > > migration from < hbase-1 (Is this too onerous? Should we support 0.98 > > => > > > > 2.0?). > > > > + You do NOT have to upgrade to the latest release of hbase1 to > migrate > > > to > > > > hbase2; being up on hbase-1.0.0+ will be sufficient. > > > > + You'll have to update your hbase1 coprocessors to deploy them on > > > hbase2. > > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g. > > > > watching for region split on RegionServer no longer makes sense given > > > > Master runs all splits now. > > > > + An hbase1 client can run against an hbase2 cluster but it will only > > be > > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do > > > admin > > > > ops using an hbase1 Admin client against an hbase2 cluster. We have > > some > > > > egregious API violations in branch-1; e.g. we have protobuf in our > API > > > (See > > > > HBASE-15607). The notion is that we can't afford a deprecation cycle > > > > purging this stuff from our Admin API. > > > > > > > > What you all think? > > > > > > > > St.Ack > > > > > > > > > > > >
[jira] [Reopened] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John reopened HBASE-18448: > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18673) Some more unwanted reference to unshaded PB classes
Anoop Sam John created HBASE-18673: -- Summary: Some more unwanted reference to unshaded PB classes Key: HBASE-18673 URL: https://issues.apache.org/jira/browse/HBASE-18673 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Priority: Minor ProtobufLogReader - Seems using Unshaded CIS which seems a miss HBaseFsck - Some public methods throw PB ServiceException. Its strange. No code within that throws this. Public exposed PBType class. I dont know what this type allows the users to do. Make their own Type? If so the Unshaded might be ok. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18672) Drop jdk7 profile from build
Mike Drob created HBASE-18672: - Summary: Drop jdk7 profile from build Key: HBASE-18672 URL: https://issues.apache.org/jira/browse/HBASE-18672 Project: HBase Issue Type: Bug Components: build Reporter: Mike Drob Fix For: 2.0.0 We have a jdk7 profile in hbase-annotations/pom.xml - we don't support building with jdk7 anyway though so we should remove it. Maybe we can remove the java 8 profile too, since that's the minimum default? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18671) Support Append/Increment in rest api
Guangxu Cheng created HBASE-18671: - Summary: Support Append/Increment in rest api Key: HBASE-18671 URL: https://issues.apache.org/jira/browse/HBASE-18671 Project: HBase Issue Type: Bug Components: REST Affects Versions: 2.0.0-alpha-2, 3.0.0, 1.4.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Recently we need to call append interface through the rest api, but rest api has not yet implemented it. We implemented it and it is stable in the production environment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)