[jira] [Resolved] (HBASE-13098) HBase Connection Control
[ https://issues.apache.org/jira/browse/HBASE-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi resolved HBASE-13098. --- Resolution: Not A Problem Number of connection/requests to a table/namespace can be controlled using quota. HBase Connection Control Key: HBASE-13098 URL: https://issues.apache.org/jira/browse/HBASE-13098 Project: HBase Issue Type: New Feature Components: security Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13098.patch, HBase Connection Control.pdf It is desirable to set the limit on the number of client connections permitted to the HBase server by controlling with certain system variables/parameters. Too many connections to the HBase server imply too many queries and MR jobs running on HBase. This can slow down the performance of the system and lead to denial of service. Hence such connections need to be controlled. Using too many connections may just cause thrashing rather than get more useful work done. This is kind off inspired from http://www.ebaytechblog.com/2014/08/21/quality-of-service-in-hadoop/#.VO2JXXyUe9y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13720) Mob files are not encrypting in mob compaction and Sweeper
Jingcheng Du created HBASE-13720: Summary: Mob files are not encrypting in mob compaction and Sweeper Key: HBASE-13720 URL: https://issues.apache.org/jira/browse/HBASE-13720 Project: HBase Issue Type: Sub-task Reporter: Jingcheng Du Assignee: Jingcheng Du The mob files are not encrypted. Part of the issue was fixed in HBASE-13693. Still we have more places need the encryption too, for example the writer used in mob file compaction and Sweeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13719) Asynchronous scanner -- cache size-in-bytes bug fix
Eshcar Hillel created HBASE-13719: - Summary: Asynchronous scanner -- cache size-in-bytes bug fix Key: HBASE-13719 URL: https://issues.apache.org/jira/browse/HBASE-13719 Project: HBase Issue Type: Bug Reporter: Eshcar Hillel Hbase Streaming Scan is a feature recently added to trunk. In this feature, an asynchronous scanner pre-loads data to the cache based on its size (both row count and size in bytes). In one of the locations where the scanner polls an item from the cache, the variable holding the estimated byte size of the cache is not updated. This affects the decision of when to load the next batch of data. A bug fix patch is attached - it comprises only local changes to the ClientAsyncPrefetchScanner.java file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13721) Improve shell scan performances when using LIMIT
Jean-Marc Spaggiari created HBASE-13721:
---
Summary: Improve shell scan performances when using LIMIT
Key: HBASE-13721
URL: https://issues.apache.org/jira/browse/HBASE-13721
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.1.0
Reporter: Jean-Marc Spaggiari
Assignee: Jean-Marc Spaggiari
When doing a scan which is expected to return the exact same number of rows as
the LIMIT we give, we still scan the entire table until we return the row(s)
and then test the numbers of rows we have. This can take a lot of time.
Example:
scan 'sensors', { COLUMNS = ['v:f92acb5b-079a-42bc-913a-657f270a3dc1'],
STARTROW = '000a', LIMIT = 1 }
This is because we will break on the limit condition AFTER we ask for the next
row. If there is none, we scan the entire table than exit.
Goal of this patch is to handle this specific case without impacting the others.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13722) Avoid non static method from BloomFilterUtil
Anoop Sam John created HBASE-13722: -- Summary: Avoid non static method from BloomFilterUtil Key: HBASE-13722 URL: https://issues.apache.org/jira/browse/HBASE-13722 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Trivial Fix For: 2.0.0 This is an unused method and slipped into this Util class from ByteBloomFilter during the cleanup. boolean contains(byte[] buf, ByteBuffer bloom) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
HBASE-12451
Hi all, Anyone to look at HBASE-12451 ? Sound very valid to me. I will try to have a look but someone else might also want to look at it. JM
[DISCUSSION] Merge of the hbase-11339 mob branch into master.
Hi folks, The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (10k-10MB) to be stored so that write amplification is reduced when compared to the normal I/O path. At a high level, it provides alternate flush and compaction mechanisms that segregates large cells into a separate area where they are not subject to potentially frequent compaction and splits that can be encountered in the normal I/O path. A more detailed design doc can be found on the hbase-11339 jira. Jingcheng Du has been working on the mob feature for a while and Anoop, Ram and I have been shepherding him through the design revisions and implementation of the feature in the hbase-11339 branch.[2] The branch we are proposing to merge into master is compatible with HBase's core functionality including snapshots, replication, shell support, behaves well with table alters, bulk loads and does not require external MR processes. It has been documented, and subject to many integration test runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection. Performance testing of the feature shows what can be a 2x-3x throughput improvement for workloads that contain mobs. These results can be seen on the hbase 2.0 panel discussion slides from hbasecon (once published). Recently there have been some hfile encryption related shortcomings that we could address in branch or in master. Earlier iterations of the feature has been tested in production by users that Jingcheng has been responsible for. A version has also been deployed at users I have been responsible for. Some of the folks from Huawei (ashutosh) have also been submitting the recent encryption bug reports against the hbase-11339 branch so there is some evidence of usage by them. The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the feature and feel it is a good time to call a merge vote. Ive posted a megapatch version for folks who want to peruse the code. [3] What do you all think? Thanks, Jingcheng, Jon, Ram, and Anoop. [1] https://issues.apache.org/jira/browse/HBASE-11339 [2] https://github.com/apache/hbase/tree/hbase-11339 [3] https://reviews.apache.org/r/34475/ -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh
[jira] [Created] (HBASE-13724) ReplicationSource dies under certain conditions reading a sequence file
churro morales created HBASE-13724:
--
Summary: ReplicationSource dies under certain conditions reading a
sequence file
Key: HBASE-13724
URL: https://issues.apache.org/jira/browse/HBASE-13724
Project: HBase
Issue Type: Bug
Reporter: churro morales
A little background,
We run our server in -ea mode and have seen quite a few replication sources
silently die over the past few months.
Note: the stacktrace I posted below comes from a regionserver running 0.94 but
quickly looking at this issue, I believe this will happen in 98 too.
Should we harden replication source to deal with these types of assertion
errors by catching throwables, should we be dealing with this at the sequence
file reader level? Still looking into the root cause of this issue but when
manually shutdown our regionservers the regionserver that recovered its queue
replicated that log just fine. So in our case a simple retry would've worked
just fine.
{code}
2015-05-08 11:04:23,348 ERROR
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unexpected
exception in ReplicationSource,
currentPath=hdfs://hm6.xxx.flurry.com:9000/hbase/.logs/x.yy.flurry.com,60020,1426792702998/x.atl.flurry.com%2C60020%2C1426792702998.1431107922449
java.lang.AssertionError
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:121)
at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1489)
at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1479)
at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1474)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:583)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:373)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] The 1st HBase 1.1.0.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Verified signatures Checked basic shell commands against local instance Built against hadoop 2.6.0 Ran test suite Cheers On Tue, May 19, 2015 at 8:12 PM, Enis Söztutar e...@apache.org wrote: +1. Checked sigs, crcs. Run local mode. Simple smoke tests Checked layout, jars and java files Run LTT with local mode Build hadoop2 versions deployed pseudo mode Enis On Mon, May 18, 2015 at 11:21 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.1.0.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.1.0.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1081/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.1.0.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [VOTE] The 1st HBase 0.98.12.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 checked crcs Run local mode + some command with the shell create/put/get/scan/disable/delete Checked source changes and jars layout Build from source and run some unit test (Admin, ACL, snapshot) Run local mode (secure/unsecure) + some command with the shell create/put/get/scan/disable/delete On Tue, May 19, 2015 at 8:12 PM, Enis Söztutar e...@apache.org wrote: +1. Checked sigs, crcs. Run local mode. Simple smoke tests Checked layout, jars and java files Run LTT with local mode Build with hadoop1 and hadoop2 Run pseudo mode Enis On Mon, May 18, 2015 at 11:19 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 0.98.12.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/0.98.12.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1079/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 0.98.12.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-13726) stop using Hadoop's IOUtils
Sean Busbey created HBASE-13726: --- Summary: stop using Hadoop's IOUtils Key: HBASE-13726 URL: https://issues.apache.org/jira/browse/HBASE-13726 Project: HBase Issue Type: Task Reporter: Sean Busbey Assignee: Sean Busbey In HBaseFsck we make use of Hadoop's IOUtils for ignore-errors-while-closing. All of these methods (in the way we call them) behave the same as commons-io's IOUtils.closeQuietly. One of the methods in the Hadoop version also uses a parameter that isn't in org.apache.hadoop. We already have commons-io as a dependency in this module, we should just use the commons-io version since it is stable and more limited in surface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13729) old hbase.regionserver.global.memstore.upperLimit is ignored if present
Esteban Gutierrez created HBASE-13729: - Summary: old hbase.regionserver.global.memstore.upperLimit is ignored if present Key: HBASE-13729 URL: https://issues.apache.org/jira/browse/HBASE-13729 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 1.1.0, 1.0.1, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Critical If hbase.regionserver.global.memstore.upperLimit is present we should use it instead of hbase.regionserver.global.memstore.size the current implementation of HeapMemorySizeUtil.getGlobalMemStorePercent() asumes that if hbase.regionserver.global.memstore.size is not defined thenit should use the old configuration, however it should be the other way around. This has a large impact specially if doing a rolling upgrade of a cluster when the memstore upper limit has been changed from the default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] The 1st HBase 1.0.1.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Checked sums and signature Checked tarball layouts Built from source RAT checks pass Unit tests pass (7u79) Ran LTT with local mode, no unexpected messages in logs, latency stable after warmup On Mon, May 18, 2015 at 11:20 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.0.1.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.0.1.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1080/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.0.1.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Reopened] (HBASE-13618) ReplicationSource is too eager to remove sinks
[ https://issues.apache.org/jira/browse/HBASE-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar reopened HBASE-13618: --- ReplicationSource is too eager to remove sinks -- Key: HBASE-13618 URL: https://issues.apache.org/jira/browse/HBASE-13618 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 2.0.0, 0.98.13, 1.0.2, 1.1.1 Attachments: 13618-v2.txt, 13618.txt Looking at the replication for some other reason I noticed that the replication source might be a bit too eager to remove sinks from the list of valid sinks. The current logic allows a sink to fail N times (default 3) and then it will be remove from the sinks. But note that this failure count is never reduced, so given enough runtime and some network glitches _every_ sink will eventually be removed. When all sink are removed the source pick new sinks and the counter is set to 0 for all of them. I think we should change to reset the counter each time we successfully replicate something to the sink (which proves the sink isn't dead). Or we could decrease the counter each time we successfully replication, that might be better - if we consistently fail more attempts than we succeed the sink should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13618) ReplicationSource is too eager to remove sinks
[ https://issues.apache.org/jira/browse/HBASE-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-13618. --- Resolution: Fixed Changed resolution to Fixed. ReplicationSource is too eager to remove sinks -- Key: HBASE-13618 URL: https://issues.apache.org/jira/browse/HBASE-13618 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 2.0.0, 0.98.13, 1.0.2, 1.1.1 Attachments: 13618-v2.txt, 13618.txt Looking at the replication for some other reason I noticed that the replication source might be a bit too eager to remove sinks from the list of valid sinks. The current logic allows a sink to fail N times (default 3) and then it will be remove from the sinks. But note that this failure count is never reduced, so given enough runtime and some network glitches _every_ sink will eventually be removed. When all sink are removed the source pick new sinks and the counter is set to 0 for all of them. I think we should change to reset the counter each time we successfully replicate something to the sink (which proves the sink isn't dead). Or we could decrease the counter each time we successfully replication, that might be better - if we consistently fail more attempts than we succeed the sink should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13728) Remove use of Hadoop's GenericOptionsParser
Sean Busbey created HBASE-13728: --- Summary: Remove use of Hadoop's GenericOptionsParser Key: HBASE-13728 URL: https://issues.apache.org/jira/browse/HBASE-13728 Project: HBase Issue Type: Task Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker GenericOptionsParser has been IA.Private for all of Hadoop 2 (handled in HADOOP-6668) we shouldn't be using it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13725) [book] Pseudo-Distributed Local Install can link to hadoop instructions
Nick Dimiduk created HBASE-13725:
Summary: [book] Pseudo-Distributed Local Install can link to
hadoop instructions
Key: HBASE-13725
URL: https://issues.apache.org/jira/browse/HBASE-13725
Project: HBase
Issue Type: Improvement
Reporter: Nick Dimiduk
Priority: Minor
The below is no longer true, we can link to
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
{quote}
Hadoop Configuration
This procedure assumes that you have configured Hadoop and HDFS on your local
system and or a remote system, and that they are running and available. It also
assumes you are using Hadoop 2. Currently, the documentation on the Hadoop
website does not include a quick start for Hadoop 2, but the guide at
link:http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide
is a good starting point.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13727) Codehaus repository is out of service
Andrew Purtell created HBASE-13727: -- Summary: Codehaus repository is out of service Key: HBASE-13727 URL: https://issues.apache.org/jira/browse/HBASE-13727 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1 Reporter: Andrew Purtell Assignee: Andrew Purtell The Codehaus repository is now out of service and this can break our builds, as found by BIGTOP-1874. Let's remove the dead repo entry from our POMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] The 1st HBase 0.98.12.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Checked sums and signature Checked tarball layouts Built from source RAT checks pass Unit tests pass (7u79) Ran LTT with local mode, no unexpected messages in logs, latency stable after warmup On Mon, May 18, 2015 at 11:19 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 0.98.12.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/0.98.12.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1079/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 0.98.12.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [VOTE] The 1st HBase 1.1.0.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Checked sums and signature Checked tarball layouts Built from source RAT checks pass Unit tests pass (7u79) Ran LTT with local mode, no unexpected messages in logs, latency stable after warmup On Mon, May 18, 2015 at 11:21 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.1.0.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.1.0.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1081/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.1.0.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [VOTE] The 1st HBase 1.1.0.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Checked sums Checked RAT Passed unit test Ran LTT on a test cluster On Wed, May 20, 2015 at 9:20 AM, Ted Yu yuzhih...@gmail.com wrote: +1 Verified signatures Checked basic shell commands against local instance Built against hadoop 2.6.0 Ran test suite Cheers On Tue, May 19, 2015 at 8:12 PM, Enis Söztutar e...@apache.org wrote: +1. Checked sigs, crcs. Run local mode. Simple smoke tests Checked layout, jars and java files Run LTT with local mode Build hadoop2 versions deployed pseudo mode Enis On Mon, May 18, 2015 at 11:21 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.1.0.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.1.0.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1081/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.1.0.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-13732) TestHBaseFsck#testParallelWithRetriesHbck fails intermittently
Stephen Yuan Jiang created HBASE-13732:
--
Summary: TestHBaseFsck#testParallelWithRetriesHbck fails
intermittently
Key: HBASE-13732
URL: https://issues.apache.org/jira/browse/HBASE-13732
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 1.1.0, 2.0.0, 1.2.0
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Priority: Minor
Fix For: 2.0.0, 1.2.0, 1.1.1
TestHBaseFsck#testParallelWithRetriesHbck failed intermittently (especially in
Windows environment) with java.io.IOException: Duplicate hbck - Abort
{noformat}
java.util.concurrent.ExecutionException: java.io.IOException: Duplicate hbck -
Abort
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testParallelWithRetriesHbck(TestHBaseFsck.java:644)
Caused by: java.io.IOException: Duplicate hbck - Abort
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:484)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:53)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck$2RunHbck.call(TestHBaseFsck.java:635)
at
org.apache.hadoop.hbase.util.TestHBaseFsck$2RunHbck.call(TestHBaseFsck.java:628)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{noformat}
HBASE-13591 tried to address this issue. It did improve the pass rate in Linux
environment (after the fix, I could not repro in my machine); however, the test
still failed intermittently in Windows environment during testing of 1.1
release.
Looking at the code, it uses the ExponentialBackoffPolicy (starting with 200ms
sleep time after first failed attempt to acquire the lock in ZK, then 400ms,
then 800ms, etc.) in between retries. Therefore, even the first hbck run
completes, the second hbck run would still fail due to long sleep time.
the proposal to fix the problem is to use ExponentialBackoffPolicyWithLimit and
cap the max sleep time to some small number (eg. 5 seconds, it should be
configurable). This would make the test more robust.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] The 1st HBase 1.0.1.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Checked sums Checked RAT. Passed unit test. Ran LTT on a test clsuter. On Wed, May 20, 2015 at 2:59 PM, Andrew Purtell apurt...@apache.org wrote: +1 Checked sums and signature Checked tarball layouts Built from source RAT checks pass Unit tests pass (7u79) Ran LTT with local mode, no unexpected messages in logs, latency stable after warmup On Mon, May 18, 2015 at 11:20 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.0.1.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.0.1.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1080/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.0.1.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-13730) Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98
Andrew Purtell created HBASE-13730: -- Summary: Backport HBASE-13576 (Failure in checking one region should not fail the entire HBCK operation) to 0.98 Key: HBASE-13730 URL: https://issues.apache.org/jira/browse/HBASE-13730 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.98.14 Backport HBASE-13576 - Failure in checking one region should not fail the entire HBCK operation - to 0.98 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13715) Backport HBASE-13057 - Provide client utility to easily enable and disable table replication
[ https://issues.apache.org/jira/browse/HBASE-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-13715. Resolution: Done Fix Version/s: (was: 1.0.2) Assignee: (was: Dima Spivak) I'm going to resolve this as Done because I can pick back the original commit from branch-1 cleanly to branch-1.0. Let me do that and update the fix versions on the original issue. Backport HBASE-13057 - Provide client utility to easily enable and disable table replication Key: HBASE-13715 URL: https://issues.apache.org/jira/browse/HBASE-13715 Project: HBase Issue Type: Sub-task Components: Replication Reporter: Dima Spivak Looks like we got this useful functionality in 0.98 and branch-1, but missed branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-13712) Backport HBASE-13199 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-13712: Let me reopen this since there's a patch for 0.98 pending. Can close once there's a disposition for that patch either way. Backport HBASE-13199 to branch-1 Key: HBASE-13712 URL: https://issues.apache.org/jira/browse/HBASE-13712 Project: HBase Issue Type: Bug Affects Versions: 1.1.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.13, 1.2.0 Attachments: HBASE-13712-0.98.patch, HBASE-13712-branch-1.patch HBASE-13199 is practically a requirement for large clusters trying to use Canary; we should port it to branch-1 so that it's usable on clusters with 10k regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
This is a useful feature, Jon. I went over the mega-patch and left some comments on review board. I noticed that hbck was not included in the patch. Neither did I find a sub-task of HBASE-11339 that covers hbck. Do you or Jingcheng plan to add MOB-aware capability for hbck ? Cheers On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh j...@cloudera.com wrote: Hi folks, The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (10k-10MB) to be stored so that write amplification is reduced when compared to the normal I/O path. At a high level, it provides alternate flush and compaction mechanisms that segregates large cells into a separate area where they are not subject to potentially frequent compaction and splits that can be encountered in the normal I/O path. A more detailed design doc can be found on the hbase-11339 jira. Jingcheng Du has been working on the mob feature for a while and Anoop, Ram and I have been shepherding him through the design revisions and implementation of the feature in the hbase-11339 branch.[2] The branch we are proposing to merge into master is compatible with HBase's core functionality including snapshots, replication, shell support, behaves well with table alters, bulk loads and does not require external MR processes. It has been documented, and subject to many integration test runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection. Performance testing of the feature shows what can be a 2x-3x throughput improvement for workloads that contain mobs. These results can be seen on the hbase 2.0 panel discussion slides from hbasecon (once published). Recently there have been some hfile encryption related shortcomings that we could address in branch or in master. Earlier iterations of the feature has been tested in production by users that Jingcheng has been responsible for. A version has also been deployed at users I have been responsible for. Some of the folks from Huawei (ashutosh) have also been submitting the recent encryption bug reports against the hbase-11339 branch so there is some evidence of usage by them. The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the feature and feel it is a good time to call a merge vote. Ive posted a megapatch version for folks who want to peruse the code. [3] What do you all think? Thanks, Jingcheng, Jon, Ram, and Anoop. [1] https://issues.apache.org/jira/browse/HBASE-11339 [2] https://github.com/apache/hbase/tree/hbase-11339 [3] https://reviews.apache.org/r/34475/ -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh
Re: [VOTE] The 1st HBase 0.98.12.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 Verified sums and sigs Built from source against hadoop versions 2.2.0, 2.3.0, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2 (went overboard, new to this :)) checked rat -- Regards, Appy
[jira] [Created] (HBASE-13735) race condition for web interface during master start up
Sean Busbey created HBASE-13735:
---
Summary: race condition for web interface during master start up
Key: HBASE-13735
URL: https://issues.apache.org/jira/browse/HBASE-13735
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.1
Reporter: Sean Busbey
Priority: Minor
loaded the master web page while the master was starting up and managed to hit
a HTTP 500 with a NPE.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.parse(MasterAddressTracker.java:236)
at
org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterInfoPort(MasterAddressTracker.java:86)
at
org.apache.hadoop.hbase.tmpl.master.BackupMasterStatusTmplImpl.renderNoFlush(BackupMasterStatusTmplImpl.java:53)
at
org.apache.hadoop.hbase.tmpl.master.BackupMasterStatusTmpl.renderNoFlush(BackupMasterStatusTmpl.java:113)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:309)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:373)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:364)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:81)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1351)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13733) failed MiniZooKeeperCluster startup not shut down ZK servers
Stephen Yuan Jiang created HBASE-13733:
--
Summary: failed MiniZooKeeperCluster startup not shut down ZK
servers
Key: HBASE-13733
URL: https://issues.apache.org/jira/browse/HBASE-13733
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 1.1.0, 2.0.0, 1.2.0
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
MiniZooKeeperCluster#startup() starts servers one-by-one, if everything is
good, it would declare success of start:
{code}
public int startup(File baseDir, int numZooKeeperServers)
...
// running all the ZK servers
for (int i = 0; i numZooKeeperServers; i++) {
...=== could throw exception in the loop and end the startup
// Start up this ZK server
standaloneServerFactory.startup(server);
...
standaloneServerFactoryList.add(standaloneServerFactory);
zooKeeperServers.add(server);
}
...
started = true;
...
}
{code}
However, if exception throws in the middle of start up (eg. some servers
already started), the MiniZooKeeperCluster#shutdown() would not shut down them
and clean up resources.
{code}
public void shutdown() throws IOException {
if (!started) {
return;
}
...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Is there any comparison between HBASE-11339 and HDFS-7240? Is their 'Object' a super set of our 'Medium Object'? 2015-05-21 10:38 GMT+08:00 Ted Yu yuzhih...@gmail.com: This is a useful feature, Jon. I went over the mega-patch and left some comments on review board. I noticed that hbck was not included in the patch. Neither did I find a sub-task of HBASE-11339 that covers hbck. Do you or Jingcheng plan to add MOB-aware capability for hbck ? Cheers On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh j...@cloudera.com wrote: Hi folks, The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (10k-10MB) to be stored so that write amplification is reduced when compared to the normal I/O path. At a high level, it provides alternate flush and compaction mechanisms that segregates large cells into a separate area where they are not subject to potentially frequent compaction and splits that can be encountered in the normal I/O path. A more detailed design doc can be found on the hbase-11339 jira. Jingcheng Du has been working on the mob feature for a while and Anoop, Ram and I have been shepherding him through the design revisions and implementation of the feature in the hbase-11339 branch.[2] The branch we are proposing to merge into master is compatible with HBase's core functionality including snapshots, replication, shell support, behaves well with table alters, bulk loads and does not require external MR processes. It has been documented, and subject to many integration test runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection. Performance testing of the feature shows what can be a 2x-3x throughput improvement for workloads that contain mobs. These results can be seen on the hbase 2.0 panel discussion slides from hbasecon (once published). Recently there have been some hfile encryption related shortcomings that we could address in branch or in master. Earlier iterations of the feature has been tested in production by users that Jingcheng has been responsible for. A version has also been deployed at users I have been responsible for. Some of the folks from Huawei (ashutosh) have also been submitting the recent encryption bug reports against the hbase-11339 branch so there is some evidence of usage by them. The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the feature and feel it is a good time to call a merge vote. Ive posted a megapatch version for folks who want to peruse the code. [3] What do you all think? Thanks, Jingcheng, Jon, Ram, and Anoop. [1] https://issues.apache.org/jira/browse/HBASE-11339 [2] https://github.com/apache/hbase/tree/hbase-11339 [3] https://reviews.apache.org/r/34475/ -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh
[jira] [Created] (HBASE-13734) Improper timestamp checking with VisibilityScanDeleteTracker
Anoop Sam John created HBASE-13734: -- Summary: Improper timestamp checking with VisibilityScanDeleteTracker Key: HBASE-13734 URL: https://issues.apache.org/jira/browse/HBASE-13734 Project: HBase Issue Type: Bug Components: security Reporter: Anoop Sam John Assignee: Anoop Sam John 3 issues 1. When VC is used and all the put cells and delete cells are not having any visibility associated with them, we are not correctly checking put cells ts against that of delete cell resulting in deletion of cells coming in after the delete ts 2. Have a row r1 with 2 cells of same TS but different visibility. In order to delete both cells we have to apply 2 deletes with these 2 visibility being set to Delete. We are trying to do this using delete full row option or delete cf way. But only one cell is getting deleted. 3. Same case as in #2 when I try to delete using family version delete, only one cell is getting deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13736) Add delay for the first execution of ExpiredMobFileCleanerChore and MobFileCompactorChore
Jingcheng Du created HBASE-13736: Summary: Add delay for the first execution of ExpiredMobFileCleanerChore and MobFileCompactorChore Key: HBASE-13736 URL: https://issues.apache.org/jira/browse/HBASE-13736 Project: HBase Issue Type: Sub-task Components: mob Reporter: Jingcheng Du Assignee: Jingcheng Du Currently, the ExpiredMobFileCleanerChore and MobFileCompactorChore are executed immediately when the master is started. At that time most of regions are not online, running these chore is a waste. We need to add a delay for them, now in the patch, we will use the interval as the initial delay. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] The 1st HBase 0.98.12.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 * checked sigs, crcs * checked apache-rat:check * checked LICENSE.txt and NOTICE.txt * successfully ran through upgrade from 0.98.12 on 4 worker cluster. On Mon, May 18, 2015 at 1:19 PM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 0.98.12.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/0.98.12.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1079/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 0.98.12.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Sean
Re: [VOTE] The 1st HBase 1.0.1.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 * checked sigs, crcs * checked apache-rat:check * checked LICENSE.txt and NOTICE.txt * successfully ran through upgrade from 1.0.1 on 4 worker cluster. On Mon, May 18, 2015 at 1:20 PM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.0.1.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.0.1.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1080/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.0.1.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Sean
Re: [VOTE] The 1st HBase 1.1.0.1 patch release candidate (RC0) is available, vote closing 5/20/2015
+1 * checked sigs, crcs * checked apache-rat:check * checked LICENSE.txt and NOTICE.txt * successfully ran through upgrade from 1.1.0 on 4 worker cluster. On Mon, May 18, 2015 at 1:21 PM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 1.1.0.1 patch release candidate (RC0) is available for download at http://people.apache.org/~apurtell/1.1.0.1RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1081/ Signed with my code signing key D5365CCD. Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on May 20 on whether or not we should release this as 1.1.0.1. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Sean
[jira] [Created] (HBASE-13723) In table.rb scanners are never closed.
Jean-Marc Spaggiari created HBASE-13723: --- Summary: In table.rb scanners are never closed. Key: HBASE-13723 URL: https://issues.apache.org/jira/browse/HBASE-13723 Project: HBase Issue Type: Bug Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Looking at table.rb, it seems that scanners are never closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
HBASE-13721 ready for review
Pretty simple. Thanks, JM
