[jira] [Created] (HBASE-15223) Make pubic convertScanToString for Spark
Jerry He created HBASE-15223: Summary: Make pubic convertScanToString for Spark Key: HBASE-15223 URL: https://issues.apache.org/jira/browse/HBASE-15223 Project: HBase Issue Type: Improvement Reporter: Jerry He One way to access HBase from Spark is to use newAPIHadoopRDD, which can take a TableInputFormat as class name. But we are not able to set a Scan object in there, for example to set a HBase filter. In MR, the public API TableMapReduceUtil.initTableMapperJob() or equivalent is used which can take a Scan object. But this call is not used in Spark conveniently. We need to make the TableMapReduceUtil.convertScanToString() public. So that a Scan object can be created, populated and then convert to the property and used by Spark. They are now package private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15207) Stuck balancer
[ https://issues.apache.org/jira/browse/HBASE-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15207. --- Resolution: Cannot Reproduce Resolving as cannot reproduce. I think balancer is fine, not stuck. It just looked that way because of the log spew caused by the subtask. > Stuck balancer > -- > > Key: HBASE-15207 > URL: https://issues.apache.org/jira/browse/HBASE-15207 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 1.2.0 >Reporter: stack > > Balancer seems to have gotten stuck in 1.2.0RC1 soon after Master joins > running cluster (previous Master had been killed by chaos monkey). > Investigate. At least fix the crazy logging which made me notice the stuck > balancer. > Last night my logs filled with this (10x256MB log files): > > 2016-02-01 11:25:26,958 DEBUG > [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer: > Lowest locality region server with non zero regions is > ve0542.halxg.cloudera.com with locality 0.0 > 2016-02-01 11:25:26,958 DEBUG > [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer: > Lowest locality region index is 0 and its region server contains 1 regions > ... > Added by this: > commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9 > Author: tedyu> Date: Thu Aug 13 09:11:59 2015 -0700 > HBASE-13376 Improvements to Stochastic load balancer (Vandana > Ayyalasomayajula) > Looks like balancer got stuck. Logging at ten lines a millisecond. > Here is lead up. Nothing in particular jumps out. Rerun doesn't show this. > {code} > 2016-01-28 05:56:22,572 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,572 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,572 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,572 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Lowest locality region server with non zero > regions is ve0540.halxg.cloudera.com with locality 0.0 > 2016-01-28 05:56:22,572 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Lowest locality region index is 0 and its region > server contains 1 regions > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Lowest locality region server with non zero > regions is ve0540.halxg.cloudera.com with locality 0.0 > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Lowest locality region index is 0 and its region > server contains 1 regions > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions. > 2016-01-28 05:56:22,573 DEBUG > [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] > balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions. > > {code} > Nothing else is happening on this master > Happens just after a Master joins cluster after being killed by a monkey. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] HBase 1.2.0 RC1
-1 The log spew doesn't happen always but if it did, as per Andrew, could fill a partition and mess up a cluster. HBASE-15218 though an oldie, is a baddie. The combo sinks the RC for me. On up side, I've been running cluster tests and 1.2.0 is looking good. St.Ack On Fri, Jan 29, 2016 at 7:29 AM, Sean Busbeywrote: > Hi Folks! > > I'm pleased to announce the second release candidate for HBase 1.2.0. > > Artifacts are available here: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC1/ > > As of this vote, the relevant md5 hashes are: > > a338ca93cd4c495f03bcff2d457222ef hbase-1.2.0-bin.tar.gz > 955cf9908ae7fef12e3b1447ce8dd035 hbase-1.2.0-src.tar.gz > > Maven artifacts are available in this staging repository: > > https://repository.apache.org/content/repositories/orgapachehbase-1127/ > > All artifacts are signed with my code signing key 0D80DB7C, available in > the project KEYS file: > > http://www.apache.org/dist/hbase/KEYS > > these artifacts correspond to commit hash > > 46fc1d876bd604f2f71f8692d79978055a095a7a > > which signed tag 1.2.0RC1 currently point to > > > https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=5696635f2f87da6777878b3755a17e0fa639a5c4 > > HBase 1.2.0 is the second minor release in the HBase 1.x line, continuing > on > the theme of bringing a stable, reliable database to the Hadoop and NoSQL > communities. This release includes roughly 250 resolved issues not covered > by previous 1.x releases. > > Notable new features include: > - JDK8 is now supported > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported > - per column-family time ranges for scan (HBASE-14355) > - daemons respond to SIGHUP to reload configs (HBASE-14529) > - region location methods added to thrift2 proxy (HBASE-13698) > - table-level sync that sends deltas (HBASE-13639) > - client side metrics via JMX (HBASE-12911) > > The full list of issues can be found at: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12332062 > > To see the changes since the prior release candidate, you can use the > following git command on > a up-to-date checkout of the hbase repository: > > git log 1.2.0RC0..1.2.0RC1 > > Please take a few minutes to verify the release[1] and vote on releasing > it: > > [ ] +1 Release this package as Apache HBase 1.2.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... > > Vote will be subject to Majority Approval[2] and will close at 4:00PM UTC > on Friday, Feb 5th, 2015[3]. > > [1]: http://www.apache.org/info/verification.html > [2]: https://www.apache.org/foundation/glossary.html#MajorityApproval > [3]: to find this in your local timezone see: > http://s.apache.org/hbase-1.2.0-rc1-close >
Successful: HBase Generate Website
Build status: Successful If successful, the website and docs have been generated. If failed, skip to the bottom of this email. Use the following commands to download the patch and apply it to a clean branch based on origin/asf-site. If you prefer to keep the hbase-site repo around permanently, you can skip the clone step. git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git cd hbase-site wget -O- https://builds.apache.org/job/hbase_generate_website/127/artifact/website.patch.zip | funzip > 6f6a8ed71fe98b83e8a8db974fc15b0d8597b174.patch git fetch git checkout -b asf-site-6f6a8ed71fe98b83e8a8db974fc15b0d8597b174 origin/asf-site git am 6f6a8ed71fe98b83e8a8db974fc15b0d8597b174.patch At this point, you can preview the changes by opening index.html or any of the other HTML pages in your local asf-site-6f6a8ed71fe98b83e8a8db974fc15b0d8597b174 branch, and you can review the differences by running: git diff origin/asf-site There are lots of spurious changes, such as timestamps and CSS styles in tables. To see a list of files that have been added, deleted, renamed, changed type, or are otherwise interesting, use the following command: git diff --name-status --diff-filter=ADCRTXUB origin/asf-site To see only files that had 100 or more lines changed: git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}' When you are satisfied, publish your changes to origin/asf-site using this command: git push origin asf-site-6f6a8ed71fe98b83e8a8db974fc15b0d8597b174:asf-site Changes take a couple of minutes to be propagated. You can then remove your asf-site-6f6a8ed71fe98b83e8a8db974fc15b0d8597b174 branch: git checkout asf-site && git branch -d asf-site-6f6a8ed71fe98b83e8a8db974fc15b0d8597b174 If failed, see https://builds.apache.org/job/hbase_generate_website/127/console
[jira] [Reopened] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reopened HBASE-15204: Reopening for addressing comments from Stack. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Successful: HBase Generate Website
Build status: Successful If successful, the website and docs have been generated. If failed, skip to the bottom of this email. Use the following commands to download the patch and apply it to a clean branch based on origin/asf-site. If you prefer to keep the hbase-site repo around permanently, you can skip the clone step. git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git cd hbase-site wget -O- https://builds.apache.org/job/hbase_generate_website/128/artifact/website.patch.zip | funzip > 9c832109458ced70d0db701e7f58ef41b1df0fc7.patch git fetch git checkout -b asf-site-9c832109458ced70d0db701e7f58ef41b1df0fc7 origin/asf-site git am 9c832109458ced70d0db701e7f58ef41b1df0fc7.patch At this point, you can preview the changes by opening index.html or any of the other HTML pages in your local asf-site-9c832109458ced70d0db701e7f58ef41b1df0fc7 branch, and you can review the differences by running: git diff origin/asf-site There are lots of spurious changes, such as timestamps and CSS styles in tables. To see a list of files that have been added, deleted, renamed, changed type, or are otherwise interesting, use the following command: git diff --name-status --diff-filter=ADCRTXUB origin/asf-site To see only files that had 100 or more lines changed: git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}' When you are satisfied, publish your changes to origin/asf-site using this command: git push origin asf-site-9c832109458ced70d0db701e7f58ef41b1df0fc7:asf-site Changes take a couple of minutes to be propagated. You can then remove your asf-site-9c832109458ced70d0db701e7f58ef41b1df0fc7 branch: git checkout asf-site && git branch -d asf-site-9c832109458ced70d0db701e7f58ef41b1df0fc7 If failed, see https://builds.apache.org/job/hbase_generate_website/128/console
[RESULT] Re: [VOTE] HBase 1.2.0 RC1
This vote fails with a single +1 and no other votes. Thanks to everyone who evaluated the release candidate! -Sean On Fri, Jan 29, 2016 at 9:29 AM, Sean Busbeywrote: > Hi Folks! > > I'm pleased to announce the second release candidate for HBase 1.2.0. > > Artifacts are available here: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC1/ > > As of this vote, the relevant md5 hashes are: > > a338ca93cd4c495f03bcff2d457222ef hbase-1.2.0-bin.tar.gz > 955cf9908ae7fef12e3b1447ce8dd035 hbase-1.2.0-src.tar.gz > > Maven artifacts are available in this staging repository: > > https://repository.apache.org/content/repositories/orgapachehbase-1127/ > > All artifacts are signed with my code signing key 0D80DB7C, available in > the project KEYS file: > > http://www.apache.org/dist/hbase/KEYS > > these artifacts correspond to commit hash > > 46fc1d876bd604f2f71f8692d79978055a095a7a > > which signed tag 1.2.0RC1 currently point to > > > https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=5696635f2f87da6777878b3755a17e0fa639a5c4 > > HBase 1.2.0 is the second minor release in the HBase 1.x line, continuing > on > the theme of bringing a stable, reliable database to the Hadoop and NoSQL > communities. This release includes roughly 250 resolved issues not covered > by previous 1.x releases. > > Notable new features include: > - JDK8 is now supported > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported > - per column-family time ranges for scan (HBASE-14355) > - daemons respond to SIGHUP to reload configs (HBASE-14529) > - region location methods added to thrift2 proxy (HBASE-13698) > - table-level sync that sends deltas (HBASE-13639) > - client side metrics via JMX (HBASE-12911) > > The full list of issues can be found at: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12332062 > > To see the changes since the prior release candidate, you can use the > following git command on > a up-to-date checkout of the hbase repository: > > git log 1.2.0RC0..1.2.0RC1 > > Please take a few minutes to verify the release[1] and vote on releasing > it: > > [ ] +1 Release this package as Apache HBase 1.2.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... > > Vote will be subject to Majority Approval[2] and will close at 4:00PM UTC > on Friday, Feb 5th, 2015[3]. > > [1]: http://www.apache.org/info/verification.html > [2]: https://www.apache.org/foundation/glossary.html#MajorityApproval > [3]: to find this in your local timezone see: > http://s.apache.org/hbase-1.2.0-rc1-close > -- Sean
[jira] [Created] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
Elliott Clark created HBASE-15222: - Summary: Use HDR histograms rather than hadoop or yammer's Key: HBASE-15222 URL: https://issues.apache.org/jira/browse/HBASE-15222 Project: HBase Issue Type: Bug Reporter: Elliott Clark Running the benchmarks now, but it looks like the results are pretty extreme. The locking in our histograms is pretty extreme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
Josh Elser created HBASE-15221: -- Summary: HTableMultiplexer improvements (stale region locations and resource leaks) Key: HBASE-15221 URL: https://issues.apache.org/jira/browse/HBASE-15221 Project: HBase Issue Type: Bug Components: Client Reporter: Josh Elser Assignee: Josh Elser Priority: Critical Fix For: 2.0.0, 1.2.1, 1.1.4, 0.98.18 It looks like HTableMultiplexer has a couple of issues. Upon failing to send a Put to the appropriate RS, the Put is re-queued back into the system. Normally this is fine as such an exception is transient and the Put would eventually succeed. However, in the case where the Put was rejected because of a NotServingRegionException (e.g. split, balance, merge), the re-queuing of the Put will end up using the same cached HRegionLocation. This means that the Put will just be repeatedly sent back to the same RS over and over again, eventually being dropped on the floor. Need to invalidate the location cache (or make sure we refresh it) when we re-queue the Put. The internal ClusterConnection is also leaked. If a user creates many HTableMultiplexers, they'll eventually run into issues (memory, zk connections, etc) because they'll never get cleaned up. HTableMultiplexer needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15094) Selection of WAL files eligible for incremental backup is broken
[ https://issues.apache.org/jira/browse/HBASE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-15094. --- Resolution: Fixed As a part of patch v30 > Selection of WAL files eligible for incremental backup is broken > > > Key: HBASE-15094 > URL: https://issues.apache.org/jira/browse/HBASE-15094 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > We should add to a selection WAL files which have been copied over by > different backup sessions and which are newer than the most recent files > processed for a given table. Currently, we look for WAL files only in a files > system (WALs and OldWALs directories) > Scenario > Full Backup T1 > Full Backup T2 > Created WAL1 > Incremental Backup T1 (WAL1 becomes eligible for deletion) > some time later > Incremental Backup T2 (WAL1 can be deleted by this time and won't get it into > WAL file list for T2) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15219) Canary tool does not return non-zero exit when one of region stuck state
Vishal Khandelwal created HBASE-15219: - Summary: Canary tool does not return non-zero exit when one of region stuck state Key: HBASE-15219 URL: https://issues.apache.org/jira/browse/HBASE-15219 Project: HBase Issue Type: Bug Affects Versions: 0.98.16 Reporter: Vishal Khandelwal 2016-02-05 12:24:18,571 ERROR [pool-2-thread-7] tool.Canary - read from region CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418. column family 0 failed org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Fri Feb 05 12:24:15 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@54c9fea0, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418. is not online on isthbase02-dnds1-3-crd.eng.sfdc.net,60020,1454669984738 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2852) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4468) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2984) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31186) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) -bash-4.1$ echo $? 0 Below code prints the error but it does sets/returns the exit code. Due to this tool can't be integrated with nagios or other alerting. Ideally it should return error for failures. as pre the documentation: This tool will return non zero error codes to user for collaborating with other monitoring tools, such as Nagios. The error code definitions are: private static final int USAGE_EXIT_CODE = 1; private static final int INIT_ERROR_EXIT_CODE = 2; private static final int TIMEOUT_ERROR_EXIT_CODE = 3; private static final int ERROR_EXIT_CODE = 4; org.apache.hadoop.hbase.tool.Canary.RegionTask public Void read() { try { table = connection.getTable(region.getTable()); tableDesc = table.getTableDescriptor(); } catch (IOException e) { LOG.debug("sniffRegion failed", e); sink.publishReadFailure(region, e); ... return null; } -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
stack created HBASE-15224: - Summary: Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213 Key: HBASE-15224 URL: https://issues.apache.org/jira/browse/HBASE-15224 Project: HBase Issue Type: Sub-task Reporter: stack Remove an escape valve no longer needed after HBASE-15213. Remove so folks don't ever have to worry their pretty-little heads about it. Let the bulk of HBASE-15031 remain, the issue that added "hbase.increment.fast.but.narrow.consistency" because it mostly cleanup we want to keep. -- This message was sent by Atlassian JIRA (v6.3.4#6332)