[jira] [Created] (HBASE-18410) FilterList Improvement.
Zheng Hu created HBASE-18410: Summary: FilterList Improvement. Key: HBASE-18410 URL: https://issues.apache.org/jira/browse/HBASE-18410 Project: HBase Issue Type: Umbrella Reporter: Zheng Hu Assignee: Zheng Hu FilterList.java is complex now, and we have found some improvements for it. So create an issue to address it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18363) Hbck option to undeploy in memory replica parent region
[ https://issues.apache.org/jira/browse/HBASE-18363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun resolved HBASE-18363. -- Resolution: Invalid the in-memory region state can be fixed with "-fixAssignments". > Hbck option to undeploy in memory replica parent region > > > Key: HBASE-18363 > URL: https://issues.apache.org/jira/browse/HBASE-18363 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.4.0, 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > > We run into cases that with read replica, after split, sometimes, the parent > replica region is left in master's in memory onlineRegion list. This results > in the region got assigned to a region server. Though the root cause will be > fixed by HBASE-18025. We need to enhance hbck tool to fix this in-memory > state. Currently, hbck only allows the fix for primary region (in this case, > the primary region is gone) with fixAssignment option, please see the > following line of code. We will enhance it so it can be applied to replica > region as well. > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java#L2216 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18409) Migrate Client Metrics from codahale to hbase-metrics
Ronald Macmaster created HBASE-18409: Summary: Migrate Client Metrics from codahale to hbase-metrics Key: HBASE-18409 URL: https://issues.apache.org/jira/browse/HBASE-18409 Project: HBase Issue Type: Bug Components: Client, java, metrics Affects Versions: 2.0.0-alpha-1 Reporter: Ronald Macmaster Currently, the metrics for hbase-client are tailored for reporting via a client-side JMX server. The MetricsConnection handles the metrics management and reporting via the metrics platform from codahale. This approach worked well for hbase-1.3.1 when the metrics platform was still relatively young, but it could be improved by using the new hbase-metrics-api. However, now that we have an actual hbase-metrics-api that master, regionserver, zookeeper, and others use, it would be good to also allow the client to leverage the metrics-api. Then, the client could also report its metrics via Hadoop's metrics2 if desired or through another platform that utilizes the hbase-metrics-api. If left alone, client metrics will continue to be only barely visible through a client-side JMX server. The migration to the new metrics-api could be done by simply changing the Metrics data types from codahale types to hbase-metrics types without changing the metrics signatures of MetricsConnection unless completely necessary. The codahale MetricsRegistry would also have to be exchanged for a hbase-metrics MetricsRegistry. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18408) AM consumes CPU and fills up the logs really fast when there is no RS to assign
Enis Soztutar created HBASE-18408:
-
Summary: AM consumes CPU and fills up the logs really fast when
there is no RS to assign
Key: HBASE-18408
URL: https://issues.apache.org/jira/browse/HBASE-18408
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
I was testing something else when I discovered that when there is no RS to
assign a region to (but master is alive), then AM/LB creates GB's of logs.
Logs like this:
{code}
2017-07-18 16:40:00,712 WARN [AssignmentThread] balancer.BaseLoadBalancer:
Wanted to do round robin assignment but no servers to assign to
2017-07-18 16:40:00,712 WARN [AssignmentThread] assignment.AssignmentManager:
unable to round-robin assignment
org.apache.hadoop.hbase.HBaseIOException: unable to compute plans for regions=1
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.acceptPlan(AssignmentManager.java:1725)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1711)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$300(AssignmentManager.java:108)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1587)
2017-07-18 16:40:00,865 WARN [AssignmentThread] balancer.BaseLoadBalancer:
Wanted to do round robin assignment but no servers to assign to
2017-07-18 16:40:00,866 WARN [AssignmentThread] assignment.AssignmentManager:
unable to round-robin assignment
org.apache.hadoop.hbase.HBaseIOException: unable to compute plans for regions=1
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.acceptPlan(AssignmentManager.java:1725)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1711)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$300(AssignmentManager.java:108)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1587)
2017-07-18 16:40:01,019 WARN [AssignmentThread] balancer.BaseLoadBalancer:
Wanted to do round robin assignment but no servers to assign to
2017-07-18 16:40:01,019 WARN [AssignmentThread] assignment.AssignmentManager:
unable to round-robin assignment
org.apache.hadoop.hbase.HBaseIOException: unable to compute plans for regions=1
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.acceptPlan(AssignmentManager.java:1725)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1711)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$300(AssignmentManager.java:108)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1587)
2017-07-18 16:40:01,173 WARN [AssignmentThread] balancer.BaseLoadBalancer:
Wanted to do round robin assignment but no servers to assign to
2017-07-18 16:40:01,173 WARN [AssignmentThread] assignment.AssignmentManager:
unable to round-robin assignment
org.apache.hadoop.hbase.HBaseIOException: unable to compute plans for regions=1
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.acceptPlan(AssignmentManager.java:1725)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1711)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$300(AssignmentManager.java:108)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1587)
{code}
Reproduction is easy:
- Start pseudo-distributed cluster
- Create a table
- kill region server
I have also noticed that we are just spinning CPU in another case consuming
100-200% (but this is in a very old code base from master) in this cycle:
{code}
"ProcedureExecutor-0" #106 daemon prio=5 os_prio=0 tid=0x7fab54851800
nid=0xcf1 runnable [0x7fab4e7b]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.hashCode(Native Method)
at
java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
at
java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:6158)
- locked <0xc4cb62e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6790)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2125)
at org.apache.hadoop.hbase.client.HTable$1.call(HTable.java:425)
at org.apache.hadoop.hbase.client.HTable$1.call(HTable.java:416)
at
org.apache.hadoop.hbase.client.RpcRetryingCalle
[jira] [Created] (HBASE-18407) [C++] Configuration::SetBool should convert bool value to "true" or "false"
Xiaobing Zhou created HBASE-18407: - Summary: [C++] Configuration::SetBool should convert bool value to "true" or "false" Key: HBASE-18407 URL: https://issues.apache.org/jira/browse/HBASE-18407 Project: HBase Issue Type: Sub-task Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Configuration::SetBool internally converts true/false to 1/0 using boost::lexical_cast(value), this results in runtime exception with 'Unexpected value found while conversion to bool.' in Configuration::GetBool since it only recognizes "true" or "false" as string representation of true or false. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18406) In ServerCrashProcedure.java start(MasterProcedureEnv) is a no-op
Alex Leblang created HBASE-18406: Summary: In ServerCrashProcedure.java start(MasterProcedureEnv) is a no-op Key: HBASE-18406 URL: https://issues.apache.org/jira/browse/HBASE-18406 Project: HBase Issue Type: Bug Reporter: Alex Leblang The comments above this method explain that it exists to set configs and return, however, no configs are set in the method. As you can see here: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L210-L214 It is only ever called here: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L142 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [DISCUSS] status of and plans for our hbase-spark integration
Hi Folks! I have taken what I think are the consensus points from this thread and made a first draft on a scope document: https://issues.apache.org/jira/browse/HBASE-18405 Please take a look, give me some feedback, etc. Once we're all in agreement then I can start tracking JIRAs on the gaps before we're ready for inclusion in a release. On 2017-06-21 11:31 (-0500), Sean Busbey wrote: > Hi Folks! > > We've had integration with Apache Spark lingering in trunk for quite > some time, and I'd like us to push towards firming it up. I'm going to > try to cover a lot of ground below, so feel free to respond to just > pieces and I'll write up a doc on things afterwards. > > For background, the hbase-spark module currently exists in trunk, > branch-2, and the 2.0.0-alpha-1 release. Importantly, it has been in > no "ready to use" release so far. It's been in master for ~2 years and > has had a total of nearly 70 incremental changes. Right now it shows > up in Stackâs excellent state of 2.0 doc ( https://s.apache.org/1mB4 ) > as a nice-to-have. Iâd like to get some consensus on either getting it > into release trains or officially move it out of scope for 2.0. > > > > 1) Branch-1 releases > > In July 2015 we started tracking what kind of polish was needed for > this code to make it into our downstream facing release lines in > HBASE-14160. Personally, I think if the module isn't ready for a > branch-1 release than it shouldn't be in a branch-2 release either. > > The only things still tracked as required are some form of published > API docs (HBASE-17766) and an IT that we can run (HBASE-18175). Our Yi > Liang has been working on both of these, and I think we have a good > start on them. > > Is there anything else we ought to be tracking here? I notice the > umbrella "make the connector better" issue (HBASE-14789) has only > composite row key support still open (HBASE-15335). It looks like that > work stalled out last summer after an admirable effort by our Zhan > Zhang. Can this wait for a future minor release? > > Personally, I'd like to see HBASE-17766 and HBASE-18175 closed out and > then our existing support backported to branch-1 in time for whenever > we get HBase 1.4 started. > > 2) What Spark version(s) do we care about? > > The hbase-spark module originally started with support for Spark 1.3. > It currently sits at supporting just 1.6. Our Ted Yu has been > dutifully trying to find consensus on how we handle Spark 2.0 over in > HBASE-16179 for nearly a year. > > AFAICT the Spark community has no more notion of what version(s) their > downstream users are relying on than we do. It appears that Spark 1.6 > will be their last 1.y release and at least the dev community is > largely moving on to 2.y releases now. > > What version(s) do we want to handle and thus encourage our downstream > folks to use? > > Just as a point of reference, Spark 1.6 doesn't have any proper > handling of delegation tokens and our current do-it-ourselves > workaround breaks in the presence of the support introduced in Spark > 2. > > The way I see it, the options are a) ship both 1.6 and 2.y support, b) > ship just 2.y support, c) ship 1.6 in branch-1 and ship 2.y in > branch-2. Does anyone have preferences here? > > Personally, I think I favor option b for simplicity, though I don't > care for more possible delay in getting stuff out in branch-1. > Probably option a would be best for our downstreamers. > > Related, while we've been going around on HBASE-16179 the Apache Spark > community started shipping 2.1 releases and is now in the process of > finalizing 2.2. Do we need to do anything different for these > versions? > > Sparkâs versioning policy suggests ânot unless we want to support > newer APIs or used alpha stuffâ. But I donât have practical experience > with how this plays out in yet. > > http://spark.apache.org/versioning-policy.html > > > 3) What scala version(s) do we care about? > > For those who aren't aware, Scala compatibility is a nightmare. Since > Scala is still the primary language for implementation of Spark jobs, > we have to care more about this than I'd like. (the only way out, I > think, would be to implement our integration entirely in some other > JVM language) > > The short version is that each minor version of scala (we care about) > is mutually incompatible with all others. Right now both Spark 1.6 and > Spark 2.y work with each of Scala 2.10 and 2.11. There's talk of > adding support for Scala 2.12, but it will not happen until after > Spark 2.2. > > (for those looking for a thread on Scala versions in Spark, I think > this is the most recent: https://s.apache.org/IW4D ) > > Personally, I think we serve our downstreamers best when we ship > artifacts that work with each of the scala versions a given version of > Spark supports. It's painful to have to do something like upgrade your > scala version just because the storage layer you want to use requires > a
[jira] [Created] (HBASE-18405) Track scope for HBase-Spark module
Sean Busbey created HBASE-18405: --- Summary: Track scope for HBase-Spark module Key: HBASE-18405 URL: https://issues.apache.org/jira/browse/HBASE-18405 Project: HBase Issue Type: Task Components: spark Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 1.4.0, 2.0.0-beta-1 Start with [\[DISCUSS\] status of and plans for our hbase-spark integration |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18404) Small typo on ACID documentation page
Michael Crutcher created HBASE-18404: Summary: Small typo on ACID documentation page Key: HBASE-18404 URL: https://issues.apache.org/jira/browse/HBASE-18404 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 1.3.1 Reporter: Michael Crutcher Priority: Trivial I noticed a couple of occurrences of the "word" wholely on the ACID semantics doc page (https://hbase.apache.org/acid-semantics.html) This should be "wholly". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: One problem about document change
yep! On Mon, Jul 17, 2017 at 9:19 PM, Guanghao Zhang wrote: > Thanks for your explanation. So if we only push it to master branch, we > should only add master tag to the fix versions of the document issue, right? > > 2017-07-17 22:00 GMT+08:00 Sean Busbey : > >> Contributors and committers usually only worry about making documentation >> changes to the master branch. >> >> When it comes time to spin up release candidates for a new version, the >> release manager usually copies the then-current state of documentation in >> the master branch to the branch they'll be working from (in this case >> branch-2). Sometimes that documentation is also tailored further to the >> release line (e.g. to remove irrelevant feature discussion). >> >> On Sun, Jul 16, 2017 at 9:16 AM, Guanghao Zhang >> wrote: >> >> > Hi, folks >> > >> > Now I am working on the asynchronous admin which is part of asynchronous >> > client. The asynchronous client feature were pushed to branch-2 and >> master >> > branch. But one confuse thing is about the document change. The book site >> > is generated by master branch's code. So the document change should only >> be >> > pushed to master branch? Or it should be pushed to branch-2 and master >> > both, because this feature was merged to branch-2, too? I am not sure >> about >> > this, so look forward to you guys' idea. Thanks. >> > >> > Guanghao Zhang >> > >>
[jira] [Created] (HBASE-18403) [Shell]Truncate permission required
Yun Zhao created HBASE-18403: Summary: [Shell]Truncate permission required Key: HBASE-18403 URL: https://issues.apache.org/jira/browse/HBASE-18403 Project: HBase Issue Type: Improvement Components: shell Reporter: Yun Zhao Priority: Trivial When a user has only (Create) permission to execute truncate, the table will be deleted and not re-created -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18402) Thrift2 should support DeleteFamily and DeleteFamilyVersion type
[
https://issues.apache.org/jira/browse/HBASE-18402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu resolved HBASE-18402.
--
Resolution: Not A Problem
In method ThriftUtilities#deleteFromThrift, if we do not pass a qualifier for
the TDelete , then the handler will regard the Delete as a DeleteFamily (or
DeleteFamilyVersion ), So we do not need a DeleteFamily or DeleteFamilyVersion.
> Thrift2 should support DeleteFamily and DeleteFamilyVersion type
> -
>
> Key: HBASE-18402
> URL: https://issues.apache.org/jira/browse/HBASE-18402
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0-alpha-1
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> Currently, our thrift2 only support two delete types, Actually, there are
> four delete types.and we should support the other delete type: DeleteFamily
> and DeleteFamilyVersion.
> {code}
> /**
> * Specify type of delete:
> * - DELETE_COLUMN means exactly one version will be removed,
> * - DELETE_COLUMNS means previous versions will also be removed.
> */
> enum TDeleteType {
> DELETE_COLUMN = 0,
> DELETE_COLUMNS = 1
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18402) Thrift2 should support DeleteFamily and DeleteFamilyVersion type
Zheng Hu created HBASE-18402:
Summary: Thrift2 should support DeleteFamily and
DeleteFamilyVersion type
Key: HBASE-18402
URL: https://issues.apache.org/jira/browse/HBASE-18402
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 2.0.0-alpha-1
Reporter: Zheng Hu
Assignee: Zheng Hu
Currently, our thrift2 only support two delete type, Actually, there are four
delete types.and we should support the other delete type: DeleteFamily and
DeleteFamilyVersion.
{code}
/**
* Specify type of delete:
* - DELETE_COLUMN means exactly one version will be removed,
* - DELETE_COLUMNS means previous versions will also be removed.
*/
enum TDeleteType {
DELETE_COLUMN = 0,
DELETE_COLUMNS = 1
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
