Two kinds of data inconsistency
Not sure whether there already had similar discussion on it, sorry for re-raising if yes. 1. data inconsistency between master and peer clusters: a). write a keyvalue KV1 at a specific coordinate (row, cf, col, ts) with value V1 to master cluster A, and since there is active scanner while flushing, KV1's memstoreTS not set to 0 in the resultant hfile F1 b). write KV1 once again with the same coordinate (row, cf, col, ts) but with different value V2, and no active scanner while flushing this time, KV1's memstoreTS is set to 0 in the resultant hfile F2 c). two KV1 are replicated to peer cluster serially, no active scanner when flushing and they are flushed to two different hfiles both with memstoreTS=0 now, a client reads KV1 from the master cluster will find the value is V1 (since its memstoreTS is larger), and when it reads KV1 from peer cluster will find the value is V2 (since memstoreTS are equal but the latter's seqID is larger) 2. data inconsistency in different time phases: a). write a keyvalue KV1 at a specific coordinate (row, cf, col, ts) with value V1 to master cluster A, and since there is active scanner while flushing, KV1's memstoreTS is not set to 0 in the resultant hfile F1 b). write KV1 once again with the same coordinate (row, cf, col, ts) but with different value V2, and no active scanner while flushing this time, KV1's memstoreTS is set to 0 in the resultant hfile F2 reading KV1 now will find the value is V1 (since its memstoreTS is larger) c). after a while a compact including F1(but not F2) occurs and KV1's memstoreTS is set to 0 since no active scanner reading KV1 now will find the value is V2 (since memstoreTS are equal but the latter's seqID is larger) Keeping mvcc untouched during a keyvalue's whole lifecycle (during flush/compact, or failover/HLog-replay) can avoid above two kinds of data inconsistency, any opinion?
[jira] [Created] (HBASE-11255) Negative request num in region load
Liu Shaohui created HBASE-11255:
---
Summary: Negative request num in region load
Key: HBASE-11255
URL: https://issues.apache.org/jira/browse/HBASE-11255
Project: HBase
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
We found that the request number of region is negative in long-running hbase
cluster.
The is because of improper cast in HRegionServer#createRegionLoad
{code}
... .setReadRequestsCount((int)r.readRequestsCount.get())
.setWriteRequestsCount((int) r.writeRequestsCount.get())
{code}
The patch is simple and just removes the cast.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-11234.
Resolution: Fixed
> FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
>
>
> Key: HBASE-11234
> URL: https://issues.apache.org/jira/browse/HBASE-11234
> Project: HBase
> Issue Type: Bug
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4
>
> Attachments: 11234-94.patch, 11234-96.patch,
> 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch
>
>
> As Ted found,
> With this change:
> {noformat}
> Index:
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
> ===
> ---
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
>(revision 1596579)
> +++
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
>(working copy)
> @@ -51,6 +51,7 @@
> import org.apache.hadoop.hbase.filter.FilterList.Operator;
> import org.apache.hadoop.hbase.filter.PageFilter;
> import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
> +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
> import org.apache.hadoop.hbase.io.hfile.CacheConfig;
> import org.apache.hadoop.hbase.io.hfile.HFileContext;
> import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
> @@ -90,6 +91,7 @@
> CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
> HFileContextBuilder hcBuilder = new HFileContextBuilder();
> hcBuilder.withBlockSize(2 * 1024);
> +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
> HFileContext hFileContext = hcBuilder.build();
> StoreFile.Writer writer = new StoreFile.WriterBuilder(
> TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
> {noformat}
> I got:
> java.lang.AssertionError:
> expected:
> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
> at
> org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
> After debugging, it seems the method of
> FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
> hfilescanner#seekBefore returns wrong result.
> The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11256) [Visibility Controller] Enhance IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour
ramkrishna.s.vasudevan created HBASE-11256: -- Summary: [Visibility Controller] Enhance IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour Key: HBASE-11256 URL: https://issues.apache.org/jira/browse/HBASE-11256 Project: HBase Issue Type: Test Reporter: ramkrishna.s.vasudevan Attachments: HBASE-11039_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
+1 Downloaded, checked hash, checked doc. Loaded data into standaalone mode. Checked it made it. Checked UI. All seems fine. Put it up on my little test cluster and ran my blockcache loadings (I had to copy in hadoop 2.4.x libs). It started fine over data written by trunk. Seems fine. No unusual behaviors that I could see on cursory review. St.Ack On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell wrote: > The 2nd HBase 0.98.3 release candidate (RC1) is available for download at > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts are also > available in the temporary repository > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > Signed with my code signing key D5365CCD. > > The issues resolved in this release can be found here: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 > -0800 GMT) on June 1 on whether or not we should release this as 0.98.3. > Three +1 votes from PMC will be required to release. > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
[jira] [Created] (HBASE-11257) [0.89-fb] Remove the timestamp from the annotation of Put
Gaurav Menghani created HBASE-11257: --- Summary: [0.89-fb] Remove the timestamp from the annotation of Put Key: HBASE-11257 URL: https://issues.apache.org/jira/browse/HBASE-11257 Project: HBase Issue Type: Improvement Reporter: Gaurav Menghani Assignee: Gaurav Menghani Priority: Minor David and I, recently found out, when discussing the C++ client with @csliu from the Search Team, that we have an extraneous timestamp field in the Put. Actually, that field is used when we create a Put object like: Put p = new Put(row, ts); And then, if you do: p.add(cf, qualifier, value); it would use ts as a timestamp for the KeyValue for the cf, qualifier. If you did not specify it, it will use HConstants.LATEST_TIMESTAMP. One can also do this, where you explicitly state the timestamp to be used: p.add(cf, qualifier, ts, value) In either case, when the add() method is called, the KeyValue is constructed, and it has the proper timestamp. Therefore, once you have created the family map with all these KeyValues, you don't need to send the ts field provided during the construction. All the KVs will have the correct timestamp embedded by the time it will be sent across. This diff removes the timestamp field from the Put object. This will save us some network bandwidth, hopefully :) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Two kinds of data inconsistency
Unification of mvcc and seqId would pave way for solving such inconsistencies. See HBASE-8763 Cheers On Tue, May 27, 2014 at 3:16 AM, 冯宏华 wrote: > Not sure whether there already had similar discussion on it, sorry for > re-raising if yes. > > 1. data inconsistency between master and peer clusters: > a). write a keyvalue KV1 at a specific coordinate (row, cf, col, ts) > with value V1 to master cluster A, and since there is active scanner while > flushing, KV1's memstoreTS not set to 0 in the resultant hfile F1 > b). write KV1 once again with the same coordinate (row, cf, col, ts) but > with different value V2, and no active scanner while flushing this time, > KV1's memstoreTS is set to 0 in the resultant hfile F2 > c). two KV1 are replicated to peer cluster serially, no active scanner > when flushing and they are flushed to two different hfiles both with > memstoreTS=0 > > now, a client reads KV1 from the master cluster will find the value is > V1 (since its memstoreTS is larger), and when it reads KV1 from peer > cluster will find the value is V2 (since memstoreTS are equal but the > latter's seqID is larger) > > 2. data inconsistency in different time phases: >a). write a keyvalue KV1 at a specific coordinate (row, cf, col, ts) > with value V1 to master cluster A, and since there is active scanner while > flushing, KV1's memstoreTS is not set to 0 in the resultant hfile F1 > b). write KV1 once again with the same coordinate (row, cf, col, ts) but > with different value V2, and no active scanner while flushing this time, > KV1's memstoreTS is set to 0 in the resultant hfile F2 > > reading KV1 now will find the value is V1 (since its memstoreTS is > larger) > > c). after a while a compact including F1(but not F2) occurs and KV1's > memstoreTS is set to 0 since no active scanner > > reading KV1 now will find the value is V2 (since memstoreTS are equal > but the latter's seqID is larger) > > Keeping mvcc untouched during a keyvalue's whole lifecycle (during > flush/compact, or failover/HLog-replay) can avoid above two kinds of data > inconsistency, any opinion? >
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
+1 Downloaded, checked signature, ran tests, checked doc. Ran ITs for visibility labels, tags and encryption (HFile and WAL). All looks good. -Anoop- On Tue, May 27, 2014 at 10:20 PM, Stack wrote: > +1 > > Downloaded, checked hash, checked doc. > > Loaded data into standaalone mode. Checked it made it. Checked UI. All > seems fine. > > Put it up on my little test cluster and ran my blockcache loadings (I had > to copy in hadoop 2.4.x libs). It started fine over data written by trunk. > Seems fine. No unusual behaviors that I could see on cursory review. > > St.Ack > > > > > On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell > wrote: > > > The 2nd HBase 0.98.3 release candidate (RC1) is available for download at > > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts are > also > > available in the temporary repository > > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > > > Signed with my code signing key D5365CCD. > > > > The issues resolved in this release can be found here: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > > > Please try out the candidate and vote +1/-1 by midnight Pacific Time > (00:00 > > -0800 GMT) on June 1 on whether or not we should release this as 0.98.3. > > Three +1 votes from PMC will be required to release. > > > > -- > > Best regards, > > > >- Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > >
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
Downloaded, checked signature, poked around with a mini-cluster + shell, ui looks good, built against phoenix. +1 --- Jesse Yates @jesse_yates jyates.github.com On Tue, May 27, 2014 at 10:09 AM, Anoop John wrote: > +1 > Downloaded, checked signature, ran tests, checked doc. Ran ITs for > visibility labels, tags and encryption (HFile and WAL). All looks good. > > -Anoop- > > On Tue, May 27, 2014 at 10:20 PM, Stack wrote: > > > +1 > > > > Downloaded, checked hash, checked doc. > > > > Loaded data into standaalone mode. Checked it made it. Checked UI. All > > seems fine. > > > > Put it up on my little test cluster and ran my blockcache loadings (I had > > to copy in hadoop 2.4.x libs). It started fine over data written by > trunk. > > Seems fine. No unusual behaviors that I could see on cursory review. > > > > St.Ack > > > > > > > > > > On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell > > wrote: > > > > > The 2nd HBase 0.98.3 release candidate (RC1) is available for download > at > > > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts are > > also > > > available in the temporary repository > > > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > > > > > Signed with my code signing key D5365CCD. > > > > > > The issues resolved in this release can be found here: > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > > > > > Please try out the candidate and vote +1/-1 by midnight Pacific Time > > (00:00 > > > -0800 GMT) on June 1 on whether or not we should release this as > 0.98.3. > > > Three +1 votes from PMC will be required to release. > > > > > > -- > > > Best regards, > > > > > >- Andy > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein > > > (via Tom White) > > > > > >
Re: Two kinds of data inconsistency
Yes, that's one of HBASE-8763 intention. The issues could happen when region moves and we bump up memstore readpoint which would cause the scanner returns inconsistent data as well. One solution is that we could keep mvcc number around but it can't cover all cases such as in replication & data recovery where we apply changes based on the ordering of log change sequence id. Using log sequence id to replace mvcc, we will have a consistent view against snapshot if a client has the read point when opening the scanner. Thanks, -Jeffrey On 5/27/14 9:57 AM, "Ted Yu" wrote: >Unification of mvcc and seqId would pave way for solving such >inconsistencies. >See HBASE-8763 > >Cheers > > >On Tue, May 27, 2014 at 3:16 AM, 冯宏华 wrote: > >> Not sure whether there already had similar discussion on it, sorry for >> re-raising if yes. >> >> 1. data inconsistency between master and peer clusters: >> a). write a keyvalue KV1 at a specific coordinate (row, cf, col, ts) >> with value V1 to master cluster A, and since there is active scanner >>while >> flushing, KV1's memstoreTS not set to 0 in the resultant hfile F1 >> b). write KV1 once again with the same coordinate (row, cf, col, ts) >>but >> with different value V2, and no active scanner while flushing this time, >> KV1's memstoreTS is set to 0 in the resultant hfile F2 >> c). two KV1 are replicated to peer cluster serially, no active scanner >> when flushing and they are flushed to two different hfiles both with >> memstoreTS=0 >> >> now, a client reads KV1 from the master cluster will find the value is >> V1 (since its memstoreTS is larger), and when it reads KV1 from peer >> cluster will find the value is V2 (since memstoreTS are equal but the >> latter's seqID is larger) >> >> 2. data inconsistency in different time phases: >>a). write a keyvalue KV1 at a specific coordinate (row, cf, col, ts) >> with value V1 to master cluster A, and since there is active scanner >>while >> flushing, KV1's memstoreTS is not set to 0 in the resultant hfile F1 >> b). write KV1 once again with the same coordinate (row, cf, col, ts) >>but >> with different value V2, and no active scanner while flushing this time, >> KV1's memstoreTS is set to 0 in the resultant hfile F2 >> >> reading KV1 now will find the value is V1 (since its memstoreTS is >> larger) >> >> c). after a while a compact including F1(but not F2) occurs and KV1's >> memstoreTS is set to 0 since no active scanner >> >> reading KV1 now will find the value is V2 (since memstoreTS are equal >> but the latter's seqID is larger) >> >> Keeping mvcc untouched during a keyvalue's whole lifecycle (during >> flush/compact, or failover/HLog-replay) can avoid above two kinds of >>data >> inconsistency, any opinion? >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Gathering opinions and recommendations
Hi, all We are planning to upgrade HBase 0.94.6/Hadoop2.0 to either 0.96.2/Hadoop2.4 or to 0.98.2/Hadoop2.4. But I have a concern ... The 'stable' is still 0.94.19? Does it mean that both 0.96. and 0.98 are not recommended for production yet? And which one should we take 0.96 or 0.98? I know that 0.96 branch is in bug-fix--only mode right now. One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? Rolling or not? Thank you, in advance and Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or notificati...@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: [VOTE] The 1st hbase 0.94.20 release candidate is available for download
+1 Checked hash. Layout looks good (did we mean to get rid of the security folder?) Docs look good. Simple PE tests on standalone seem to do the right thing. St.Ack On Fri, May 23, 2014 at 10:35 PM, lars hofhansl wrote: > Just to make you folks work over the long weekend... > > The 1st 0.94.20 RC is available for download at > http://people.apache.org/~larsh/hbase-0.94.20-rc0/ > Signed with my code signing key: C7CFE328 > > HBase 0.94.20 is a small bug fix release with 12 fixes: > [HBASE-10936] - Add zeroByte encoding test > [HBASE-10958] - [dataloss] Bulk loading with seqids can prevent some > log entries from being replayed > [HBASE-0] - Ability to load FilterList class is dependent on > context classloader > [HBASE-11143] - Improve replication metrics > [HBASE-11188] - "Inconsistent configuration" for SchemaMetrics is > always shown > [HBASE-11212] - Fix increment index in KeyValueSortReducer > [HBASE-11225] - Backport fix for HBASE-10417 'index is not incremented > in PutSortReducer#reduce()' > [HBASE-11247] - [0.94] update maven-site-plugin to 3.3 > [HBASE-11008] - Align bulk load, flush, and compact to require > Action.CREATE > [HBASE-9] - Update ExportSnapShot to optionally not use a tmp file > on external file system > [HBASE-11128] - Add -target option to ExportSnapshot to export with a > different name > [HBASE-11134] - Add a -list-snapshots option to SnapshotInfo > > The list of changes is also available here: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326654 > > Since Apache Jenkins appears to be out on vacation, I built this on my own > machine. I ran the full test suite a few times. > > Please try out the RC, check out the doc, take it for a spin, etc, and > vote +1/-1 by EOD May 30th on whether we should release this as 0.94.20. > > Thanks. > > -- Lars >
Re: Gathering opinions and recommendations
Isn’t HDP 2.1 shipping 0.98.x ? On May 27, 2014, at 8:36 PM, Vladimir Rodionov wrote: > Hi, all > > We are planning to upgrade HBase 0.94.6/Hadoop2.0 to either 0.96.2/Hadoop2.4 > or to 0.98.2/Hadoop2.4. > But I have a concern ... The 'stable' is still 0.94.19? Does it mean that > both 0.96. and 0.98 are not recommended for > production yet? And which one should we take 0.96 or 0.98? I know that 0.96 > branch is in bug-fix--only mode right now. > One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? Rolling > or not? > > Thank you, in advance > > and > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: vrodio...@carrieriq.com > > > > Confidentiality Notice: The information contained in this message, including > any attachments hereto, may be confidential and is intended to be read only > by the individual or entity to whom this message is addressed. If the reader > of this message is not the intended recipient or an agent or designee of the > intended recipient, please note that any review, use, disclosure or > distribution of this message or its attachments, in any form, is strictly > prohibited. If you have received this message in error, please immediately > notify the sender and/or notificati...@carrieriq.com and delete or destroy > any copy of this message and its attachments. >
Re: Gathering opinions and recommendations
And so is CDH5.0.1. Vladimir, I will say go with 0.98 instead of 0.96 since 0.98 will get more fixes/features. But test your applications with it first. JM 2014-05-27 16:10 GMT-04:00 Michael Segel : > Isn’t HDP 2.1 shipping 0.98.x ? > > On May 27, 2014, at 8:36 PM, Vladimir Rodionov > wrote: > > > Hi, all > > > > We are planning to upgrade HBase 0.94.6/Hadoop2.0 to either > 0.96.2/Hadoop2.4 or to 0.98.2/Hadoop2.4. > > But I have a concern ... The 'stable' is still 0.94.19? Does it mean > that both 0.96. and 0.98 are not recommended for > > production yet? And which one should we take 0.96 or 0.98? I know that > 0.96 branch is in bug-fix--only mode right now. > > One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? > Rolling or not? > > > > Thank you, in advance > > > > and > > > > Best regards, > > Vladimir Rodionov > > Principal Platform Engineer > > Carrier IQ, www.carrieriq.com > > e-mail: vrodio...@carrieriq.com > > > > > > > > Confidentiality Notice: The information contained in this message, > including any attachments hereto, may be confidential and is intended to be > read only by the individual or entity to whom this message is addressed. If > the reader of this message is not the intended recipient or an agent or > designee of the intended recipient, please note that any review, use, > disclosure or distribution of this message or its attachments, in any form, > is strictly prohibited. If you have received this message in error, please > immediately notify the sender and/or notificati...@carrieriq.com and > delete or destroy any copy of this message and its attachments. > > > >
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
I tried the test suite 3 times and 3 times failed on this: Results : Failed tests: testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile) Tests in error: testScannerSelection[0](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) testScannerSelection[1](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) testScannerSelection[2](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) testBlocksScanned(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction) Tests run: 840, Failures: 1, Errors: 6, Skipped: 2 Am I the only one facing that? JM 2014-05-27 14:27 GMT-04:00 Jesse Yates : > Downloaded, checked signature, poked around with a mini-cluster + shell, ui > looks good, built against phoenix. > > +1 > > --- > Jesse Yates > @jesse_yates > jyates.github.com > > > On Tue, May 27, 2014 at 10:09 AM, Anoop John > wrote: > > > +1 > > Downloaded, checked signature, ran tests, checked doc. Ran ITs for > > visibility labels, tags and encryption (HFile and WAL). All looks good. > > > > -Anoop- > > > > On Tue, May 27, 2014 at 10:20 PM, Stack wrote: > > > > > +1 > > > > > > Downloaded, checked hash, checked doc. > > > > > > Loaded data into standaalone mode. Checked it made it. Checked UI. > All > > > seems fine. > > > > > > Put it up on my little test cluster and ran my blockcache loadings (I > had > > > to copy in hadoop 2.4.x libs). It started fine over data written by > > trunk. > > > Seems fine. No unusual behaviors that I could see on cursory review. > > > > > > St.Ack > > > > > > > > > > > > > > > On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell > > > wrote: > > > > > > > The 2nd HBase 0.98.3 release candidate (RC1) is available for > download > > at > > > > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts > are > > > also > > > > available in the temporary repository > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > > > > > > > Signed with my code signing key D5365CCD. > > > > > > > > The issues resolved in this release can be found here: > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > > > > > > > Please try out the candidate and vote +1/-1 by midnight Pacific Time > > > (00:00 > > > > -0800 GMT) on June 1 on whether or not we should release this as > > 0.98.3. > > > > Three +1 votes from PMC will be required to release. > > > > > > > > -- > > > > Best regards, > > > > > > > >- Andy > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein > > > > (via Tom White) > > > > > > > > > >
[jira] [Created] (HBASE-11258) [0.89-fb] Pull in Integration Tests from open source 0.96/trunk
Elliott Clark created HBASE-11258: - Summary: [0.89-fb] Pull in Integration Tests from open source 0.96/trunk Key: HBASE-11258 URL: https://issues.apache.org/jira/browse/HBASE-11258 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.89-fb Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.89-fb -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
I checked out src and tried it and passes for me mighty JMS: --- T E S T S --- Running org.apache.hadoop.hbase.regionserver.TestStoreFile 2014-05-27 14:12:06.218 java[8992:1903] Unable to load realm info from SCDynamicStore Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.961 sec Results : Tests run: 12, Failures: 0, Errors: 0, Skipped: 0 St.Ack On Tue, May 27, 2014 at 1:30 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wrote: > I tried the test suite 3 times and 3 times failed on this: > Results : > > Failed tests: > > testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile) > > Tests in error: > > > testScannerSelection[0](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testScannerSelection[1](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testScannerSelection[2](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > testBlocksScanned(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > > > testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction) > > Tests run: 840, Failures: 1, Errors: 6, Skipped: 2 > > > Am I the only one facing that? > > JM > > > 2014-05-27 14:27 GMT-04:00 Jesse Yates : > > > Downloaded, checked signature, poked around with a mini-cluster + shell, > ui > > looks good, built against phoenix. > > > > +1 > > > > --- > > Jesse Yates > > @jesse_yates > > jyates.github.com > > > > > > On Tue, May 27, 2014 at 10:09 AM, Anoop John > > wrote: > > > > > +1 > > > Downloaded, checked signature, ran tests, checked doc. Ran ITs for > > > visibility labels, tags and encryption (HFile and WAL). All looks good. > > > > > > -Anoop- > > > > > > On Tue, May 27, 2014 at 10:20 PM, Stack wrote: > > > > > > > +1 > > > > > > > > Downloaded, checked hash, checked doc. > > > > > > > > Loaded data into standaalone mode. Checked it made it. Checked UI. > > All > > > > seems fine. > > > > > > > > Put it up on my little test cluster and ran my blockcache loadings (I > > had > > > > to copy in hadoop 2.4.x libs). It started fine over data written by > > > trunk. > > > > Seems fine. No unusual behaviors that I could see on cursory > review. > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell > > > > > wrote: > > > > > > > > > The 2nd HBase 0.98.3 release candidate (RC1) is available for > > download > > > at > > > > > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts > > are > > > > also > > > > > available in the temporary repository > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > > > > > > > > > Signed with my code signing key D5365CCD. > > > > > > > > > > The issues resolved in this release can be found here: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > > > > > > > > > Please try out the candidate and vote +1/-1 by midnight Pacific > Time > > > > (00:00 > > > > > -0800 GMT) on June 1 on whether or not we should release this as > > > 0.98.3. > > > > > Three +1 votes from PMC will be required to release. > > > > > > > > > > -- > > > > > Best regards, > > > > > > > > > >- Andy > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > > Hein > > > > > (via Tom White) > > > > > > > > > > > > > > >
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
These tests passed locally (on MacBook). Cheers On Tue, May 27, 2014 at 1:30 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wrote: > I tried the test suite 3 times and 3 times failed on this: > Results : > > Failed tests: > > testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile) > > Tests in error: > > > testScannerSelection[0](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testScannerSelection[1](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testScannerSelection[2](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > testBlocksScanned(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > > > testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction) > > Tests run: 840, Failures: 1, Errors: 6, Skipped: 2 > > > Am I the only one facing that? > > JM > > > 2014-05-27 14:27 GMT-04:00 Jesse Yates : > > > Downloaded, checked signature, poked around with a mini-cluster + shell, > ui > > looks good, built against phoenix. > > > > +1 > > > > --- > > Jesse Yates > > @jesse_yates > > jyates.github.com > > > > > > On Tue, May 27, 2014 at 10:09 AM, Anoop John > > wrote: > > > > > +1 > > > Downloaded, checked signature, ran tests, checked doc. Ran ITs for > > > visibility labels, tags and encryption (HFile and WAL). All looks good. > > > > > > -Anoop- > > > > > > On Tue, May 27, 2014 at 10:20 PM, Stack wrote: > > > > > > > +1 > > > > > > > > Downloaded, checked hash, checked doc. > > > > > > > > Loaded data into standaalone mode. Checked it made it. Checked UI. > > All > > > > seems fine. > > > > > > > > Put it up on my little test cluster and ran my blockcache loadings (I > > had > > > > to copy in hadoop 2.4.x libs). It started fine over data written by > > > trunk. > > > > Seems fine. No unusual behaviors that I could see on cursory > review. > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell > > > > > wrote: > > > > > > > > > The 2nd HBase 0.98.3 release candidate (RC1) is available for > > download > > > at > > > > > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts > > are > > > > also > > > > > available in the temporary repository > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > > > > > > > > > Signed with my code signing key D5365CCD. > > > > > > > > > > The issues resolved in this release can be found here: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > > > > > > > > > Please try out the candidate and vote +1/-1 by midnight Pacific > Time > > > > (00:00 > > > > > -0800 GMT) on June 1 on whether or not we should release this as > > > 0.98.3. > > > > > Three +1 votes from PMC will be required to release. > > > > > > > > > > -- > > > > > Best regards, > > > > > > > > > >- Andy > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > > Hein > > > > > (via Tom White) > > > > > > > > > > > > > > >
Re: Gathering opinions and recommendations
Inlined. > > 2014-05-27 16:10 GMT-04:00 Michael Segel : > > > Isn’t HDP 2.1 shipping 0.98.x ? > Yes. > > One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? > > Rolling or not? > 0.96 -> 1.0 probably won't be supported unless there is explicit interest and work for it. I doubt that will happen. The reason is hfile v3 is default in 1.0, but 0.96 code base cannot read hfile v3 files. This means that rolling restarts is tricky because if a new server writes a new hfile, old servers cannot read this, the region won't be opened and will keep bouncing until it hits a new server. In theory it should work, but I won't recommend doing it. 0.98 -> 1.0 should work with rolling restarts unless we do an explicit decision to drop the support for some reason. I don't imagine there will be a compelling enough reason to drop support for now. https://issues.apache.org/jira/browse/HBASE-11164 tracks this. Enis > > > > > > Thank you, in advance > > > > > > and > > > > > > Best regards, > > > Vladimir Rodionov > > > Principal Platform Engineer > > > Carrier IQ, www.carrieriq.com > > > e-mail: vrodio...@carrieriq.com > > > > > > > > > > > > Confidentiality Notice: The information contained in this message, > > including any attachments hereto, may be confidential and is intended to > be > > read only by the individual or entity to whom this message is addressed. > If > > the reader of this message is not the intended recipient or an agent or > > designee of the intended recipient, please note that any review, use, > > disclosure or distribution of this message or its attachments, in any > form, > > is strictly prohibited. If you have received this message in error, > please > > immediately notify the sender and/or notificati...@carrieriq.com and > > delete or destroy any copy of this message and its attachments. > > > > > > > >
Re: Gathering opinions and recommendations
> The 'stable' is still 0.94.19? Lets fix this. 0.98.3 becomes new 'stable'? On Tue, May 27, 2014 at 2:40 PM, Enis Söztutar wrote: > Inlined. > ... > > > One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? > > > Rolling or not? > > > > 0.96 -> 1.0 probably won't be supported unless there is explicit interest > and work for it. I doubt that will happen. The reason is hfile v3 is > default in 1.0, but 0.96 code base cannot read hfile v3 files. This means > that rolling restarts is tricky because if a new server writes a new hfile, > old servers cannot read this, the region won't be opened and will keep > bouncing until it hits a new server. In theory it should work, but I won't > recommend doing it. > > Our book actually says this http://hbase.apache.org/book.html#d3219e3534 (Yah!). St.Ack
Re: Gathering opinions and recommendations
Enis, for customer already on 0.96, can they rolling restart to 0.98, and then from 0.98 rolling to 1.0? Demai On Tue, May 27, 2014 at 2:40 PM, Enis Söztutar wrote: > Inlined. > > > > > > 2014-05-27 16:10 GMT-04:00 Michael Segel : > > > > > Isn’t HDP 2.1 shipping 0.98.x ? > > > > Yes. > > > > One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? > > > Rolling or not? > > > > 0.96 -> 1.0 probably won't be supported unless there is explicit interest > and work for it. I doubt that will happen. The reason is hfile v3 is > default in 1.0, but 0.96 code base cannot read hfile v3 files. This means > that rolling restarts is tricky because if a new server writes a new hfile, > old servers cannot read this, the region won't be opened and will keep > bouncing until it hits a new server. In theory it should work, but I won't > recommend doing it. > > 0.98 -> 1.0 should work with rolling restarts unless we do an explicit > decision to drop the support for some reason. I don't imagine there will be > a compelling enough reason to drop support for now. > https://issues.apache.org/jira/browse/HBASE-11164 tracks this. > > Enis > > > > > > > > > > Thank you, in advance > > > > > > > > and > > > > > > > > Best regards, > > > > Vladimir Rodionov > > > > Principal Platform Engineer > > > > Carrier IQ, www.carrieriq.com > > > > e-mail: vrodio...@carrieriq.com > > > > > > > > > > > > > > > > Confidentiality Notice: The information contained in this message, > > > including any attachments hereto, may be confidential and is intended > to > > be > > > read only by the individual or entity to whom this message is > addressed. > > If > > > the reader of this message is not the intended recipient or an agent or > > > designee of the intended recipient, please note that any review, use, > > > disclosure or distribution of this message or its attachments, in any > > form, > > > is strictly prohibited. If you have received this message in error, > > please > > > immediately notify the sender and/or notificati...@carrieriq.com and > > > delete or destroy any copy of this message and its attachments. > > > > > > > > > > > > >
Re: Gathering opinions and recommendations
On Tue, May 27, 2014 at 2:47 PM, Demai Ni wrote: > Enis, > > for customer already on 0.96, can they rolling restart to 0.98, and then > from 0.98 rolling to 1.0? > I think that should work. Enis > > Demai > > > On Tue, May 27, 2014 at 2:40 PM, Enis Söztutar wrote: > > > Inlined. > > > > > > > > > > 2014-05-27 16:10 GMT-04:00 Michael Segel : > > > > > > > Isn’t HDP 2.1 shipping 0.98.x ? > > > > > > > Yes. > > > > > > One more question: future upgrade from 0.96 -> 1.0 and 0.98 -> 1.0? > > > > Rolling or not? > > > > > > > 0.96 -> 1.0 probably won't be supported unless there is explicit interest > > and work for it. I doubt that will happen. The reason is hfile v3 is > > default in 1.0, but 0.96 code base cannot read hfile v3 files. This means > > that rolling restarts is tricky because if a new server writes a new > hfile, > > old servers cannot read this, the region won't be opened and will keep > > bouncing until it hits a new server. In theory it should work, but I > won't > > recommend doing it. > > > > 0.98 -> 1.0 should work with rolling restarts unless we do an explicit > > decision to drop the support for some reason. I don't imagine there will > be > > a compelling enough reason to drop support for now. > > https://issues.apache.org/jira/browse/HBASE-11164 tracks this. > > > > Enis > > > > > > > > > > > > > > Thank you, in advance > > > > > > > > > > and > > > > > > > > > > Best regards, > > > > > Vladimir Rodionov > > > > > Principal Platform Engineer > > > > > Carrier IQ, www.carrieriq.com > > > > > e-mail: vrodio...@carrieriq.com > > > > > > > > > > > > > > > > > > > > Confidentiality Notice: The information contained in this message, > > > > including any attachments hereto, may be confidential and is intended > > to > > > be > > > > read only by the individual or entity to whom this message is > > addressed. > > > If > > > > the reader of this message is not the intended recipient or an agent > or > > > > designee of the intended recipient, please note that any review, use, > > > > disclosure or distribution of this message or its attachments, in any > > > form, > > > > is strictly prohibited. If you have received this message in error, > > > please > > > > immediately notify the sender and/or notificati...@carrieriq.com and > > > > delete or destroy any copy of this message and its attachments. > > > > > > > > > > > > > > > > > > >
Re: [VOTE] The 1st hbase 0.94.20 release candidate is available for download
Thanks Stack. The security folder is still there on purpose. I just changed the pom to always pull the secure files into the build (not just for the secure target). -- Lars From: Stack To: HBase Dev List ; lars hofhansl Sent: Tuesday, May 27, 2014 1:02 PM Subject: Re: [VOTE] The 1st hbase 0.94.20 release candidate is available for download +1 Checked hash. Layout looks good (did we mean to get rid of the security folder?) Docs look good. Simple PE tests on standalone seem to do the right thing. St.Ack On Fri, May 23, 2014 at 10:35 PM, lars hofhansl wrote: > Just to make you folks work over the long weekend... > > The 1st 0.94.20 RC is available for download at > http://people.apache.org/~larsh/hbase-0.94.20-rc0/ > Signed with my code signing key: C7CFE328 > > HBase 0.94.20 is a small bug fix release with 12 fixes: > [HBASE-10936] - Add zeroByte encoding test > [HBASE-10958] - [dataloss] Bulk loading with seqids can prevent some > log entries from being replayed > [HBASE-0] - Ability to load FilterList class is dependent on > context classloader > [HBASE-11143] - Improve replication metrics > [HBASE-11188] - "Inconsistent configuration" for SchemaMetrics is > always shown > [HBASE-11212] - Fix increment index in KeyValueSortReducer > [HBASE-11225] - Backport fix for HBASE-10417 'index is not incremented > in PutSortReducer#reduce()' > [HBASE-11247] - [0.94] update maven-site-plugin to 3.3 > [HBASE-11008] - Align bulk load, flush, and compact to require > Action.CREATE > [HBASE-9] - Update ExportSnapShot to optionally not use a tmp file > on external file system > [HBASE-11128] - Add -target option to ExportSnapshot to export with a > different name > [HBASE-11134] - Add a -list-snapshots option to SnapshotInfo > > The list of changes is also available here: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326654 > > Since Apache Jenkins appears to be out on vacation, I built this on my own > machine. I ran the full test suite a few times. > > Please try out the RC, check out the doc, take it for a spin, etc, and > vote +1/-1 by EOD May 30th on whether we should release this as 0.94.20. > > Thanks. > > -- Lars >
Re: Gathering opinions and recommendations
On Tue, May 27, 2014 at 2:59 PM, Enis Söztutar wrote: > On Tue, May 27, 2014 at 2:47 PM, Demai Ni wrote: > > > Enis, > > > > for customer already on 0.96, can they rolling restart to 0.98, and then > > from 0.98 rolling to 1.0? > > > > I think that should work. > Agree that that is expected to work. It does in our testing. Let us know if issue. We need to fix it if there is a problem. St.Ack
[jira] [Created] (HBASE-11259) Compression.java different compressions load system classpath differently causing errors
Enoch Hsu created HBASE-11259: - Summary: Compression.java different compressions load system classpath differently causing errors Key: HBASE-11259 URL: https://issues.apache.org/jira/browse/HBASE-11259 Project: HBase Issue Type: Improvement Affects Versions: 0.99.0 Reporter: Enoch Hsu Assignee: Enoch Hsu Priority: Minor Fix For: 0.99.0 I stumbled upon this issue when testing Snappy compression on hbase tables with a webserver. On the webserver the system classpath of the server JVM did not include the path to hadoop-common.jar and since it is hardcoded to only retrieve the system classpath it ran into a NoClassDefFoundException using hte following call ClassLoader.getSystemClassLoader() However LZ4 compression works using this private method getClassLoaderForCodec() which attempts to load the threads classpath first and the system classpath last. I propose to change all the ClassLoader.getSystemClassLoader() calls to getClassLoaderForCodec() to allow for consistent behavior in loading compression classes and to also make it easier for users to make sure they do not run into RuntimeExceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Gathering opinions and recommendations
Stack and Enis, thanks. We may eventually come out a way to get customers directly from 0.96 to 1.0, although probably just wrap two-rolling-start (96->98, 98->1.0) into one command. will report back if it works. Demai On Tue, May 27, 2014 at 3:26 PM, Stack wrote: > On Tue, May 27, 2014 at 2:59 PM, Enis Söztutar wrote: > > > On Tue, May 27, 2014 at 2:47 PM, Demai Ni wrote: > > > > > Enis, > > > > > > for customer already on 0.96, can they rolling restart to 0.98, and > then > > > from 0.98 rolling to 1.0? > > > > > > > I think that should work. > > > > Agree that that is expected to work. It does in our testing. Let us know > if issue. We need to fix it if there is a problem. > St.Ack >
Re: Gathering opinions and recommendations
Thanks, everyone I think we will go with 0.98.2 -Vladimir On Tue, May 27, 2014 at 3:41 PM, Demai Ni wrote: > Stack and Enis, > > thanks. We may eventually come out a way to get customers directly from > 0.96 to 1.0, although probably just wrap two-rolling-start (96->98, > 98->1.0) into one command. will report back if it works. > > Demai > > > On Tue, May 27, 2014 at 3:26 PM, Stack wrote: > > > On Tue, May 27, 2014 at 2:59 PM, Enis Söztutar > wrote: > > > > > On Tue, May 27, 2014 at 2:47 PM, Demai Ni wrote: > > > > > > > Enis, > > > > > > > > for customer already on 0.96, can they rolling restart to 0.98, and > > then > > > > from 0.98 rolling to 1.0? > > > > > > > > > > I think that should work. > > > > > > > Agree that that is expected to work. It does in our testing. Let us > know > > if issue. We need to fix it if there is a problem. > > St.Ack > > >
[jira] [Created] (HBASE-11260) hbase-default.xml refers to hbase.regionserver.global.memstore.upperLimit which is deprecated
Ted Yu created HBASE-11260:
--
Summary: hbase-default.xml refers to
hbase.regionserver.global.memstore.upperLimit which is deprecated
Key: HBASE-11260
URL: https://issues.apache.org/jira/browse/HBASE-11260
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Trivial
I noticed the following in log:
{code}
2014-05-27 16:50:46,978 WARN [main] hbase.HBaseConfiguration(78):
hbase.regionserver.global.memstore.upperLimit is deprecated by
hbase.regionserver.global.memstore.size
{code}
hbase.regionserver.global.memstore.size should be used instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: [DISCUSSION] Update on HBASE-10070 / Merge into trunk
Agreed with Devaraj. I do not think that these changes will destabilize the code base much. All of the code committed to branch has been unit tested and integration tests, both specific to region replicas and others have been running for some time. Specifically, various large scale data ingestion + verification tests, bulk load tests, hbck tests, shell tests, replication, mapreduce based tests, snapshot tests, etc has been running on this code base. For testing the feature itself, HBASE-10572, HBASE-10616, and HBASE-10818 adds tests for get/multi-get/scan and bulk load which has been running with CM. HBASE-10817 adds a test for region replicas + replication. HBASE-10791 changes PerformanceEvaluation to be able to work with region replicas. Nicolas is also working on further perf testing and improvements. For 1.0, I think it should be fine to include this as an experimental feature. Otherwise, it will be very hard to add this to the 1.x line. Are there any more concerns? If not, I would like to raise a VOTE soon. Enis On Sat, May 24, 2014 at 10:40 AM, Devaraj Das wrote: > Thanks Stack. On the testing, we have been adding unit tests for all > patches. We have also tested the patches quite extensively on real clusters > via IT tests (existing and newly added) and issues discovered have been > fixed. AFAICT, and from the tests run, this feature shouldn't destabilize > the mainline. There are some limitations when region-replica is used - > better integration with hbck (HBASE-10674) and support for split/merge (and > these are being worked on as part of phase 2). > On May 23, 2014 9:56 PM, "Stack" wrote: > > > I think merge sooner rather than later. > > > > What sort of testing has been done on 10070 and what can you say about > how > > well the feature works? > > > > Where do you see risk, if any, of it destabilizing the mainline > especially > > what with us coming up to a 1.0. > > > > Thanks, > > St.Ack > > > > > > > > On Wed, May 21, 2014 at 5:08 PM, Enis Söztutar wrote: > > > > > Hi, > > > > > > We would like to give an update on the status of HBASE-10070 work, and > > open > > > up discussion for how we can do further development. > > > > > > We seem to be at a point where we have the core functionality of the > > > region replica, as described in HBASE-10070 working. As pointed out > > > under the section "Development Phases" in the design doc posted on the > > > jira HBASE-10070, this work was divided into two broad phases. The > first > > > phase introduces region replicas concept, the new consistency model, > and > > > corresponding RPC implementations. All of the issues for Phase 1 can be > > > found under [3]. Phase 2 is still in the works, and contains the > proposed > > > changes listed under [4]. > > > > > > With all the issues committed in HBASE-10070 branch in svn, we think > that > > > the "phase-1" is complete. The user documentation on HBASE-10513 gives > an > > > accurate picture of what has been done in phase-1 and what the impact > of > > > using this feature is, APIs etc. We have added > > > a couple of IT tests as part of this work and we have tested the work > > > we did in "phase-1" of the project quite extensively in Hortonworks' > > > infrastructure. > > > > > > In summary, with the code in branch, you can create tables with region > > > replicas, do gets / multi gets and scans using TIMELINE consistency > with > > > high availability. Region replicas periodically scan the files of the > > > primary and pick up flushed / committed files. The RPC paths / > > assignment, > > > balancing etc are pretty stable. However some more performance analysis > > and > > > tuning is needed. More information can be found in [1] and [2]. > > > > > > > > > As a reminder, the development has been happening in the branch - > > > hbase-10070 (https://github.com/apache/hbase/tree/hbase-10070). We > have > > > been pretty diligent about more than one committer's +1 on the branch > > > commits and for almost all the subtasks in HBASE-10070 there is more > than > > > one +1 except for test fix issues or more trivial issues, where there > > maybe > > > only one +1. Enis/Nicolas/Sergey/Devaraj/Nick are the main > contributors > > > of code in the phase-1 and many patches have been reviewed already by > > > people outside > > > this group (mainly Stack, Jimmy) > > > > > > For Phase 2, we think that we can deliver on the proposed design > > > incrementally over the next couple of months. However, we think that it > > > might be ok to merge the changes from phase 1 first, then do a > > > commit-as-you-go approach for remaining items. Therefore, we would like > > to > > > propose to merge the branch to trunk, and continue the work over > there. > > > This might also result in more reviews. > > > > > > Alternatively, we can continue the work in the branch, and do the merge > > at > > > the end of Phase 2, but that will make the review process a bit more > > > tricky, which is why we would like the
Re: [VOTE] The 2nd HBase 0.98.3 release candidate (RC1) is available, vote closing 6/1/2014
I ran the unit test suite 25 times on JDK 7 and did not see this. I am currently doing the same on JDK 6. No unit test failures so far. On Tue, May 27, 2014 at 1:30 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wrote: > I tried the test suite 3 times and 3 times failed on this: > Results : > > Failed tests: > > testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile) > > Tests in error: > > > testScannerSelection[0](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testScannerSelection[1](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testScannerSelection[2](org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange) > > > testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > testBlocksScanned(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > > > testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction) > > Tests run: 840, Failures: 1, Errors: 6, Skipped: 2 > > > Am I the only one facing that? > > JM > > > 2014-05-27 14:27 GMT-04:00 Jesse Yates : > > > Downloaded, checked signature, poked around with a mini-cluster + shell, > ui > > looks good, built against phoenix. > > > > +1 > > > > --- > > Jesse Yates > > @jesse_yates > > jyates.github.com > > > > > > On Tue, May 27, 2014 at 10:09 AM, Anoop John > > wrote: > > > > > +1 > > > Downloaded, checked signature, ran tests, checked doc. Ran ITs for > > > visibility labels, tags and encryption (HFile and WAL). All looks good. > > > > > > -Anoop- > > > > > > On Tue, May 27, 2014 at 10:20 PM, Stack wrote: > > > > > > > +1 > > > > > > > > Downloaded, checked hash, checked doc. > > > > > > > > Loaded data into standaalone mode. Checked it made it. Checked UI. > > All > > > > seems fine. > > > > > > > > Put it up on my little test cluster and ran my blockcache loadings (I > > had > > > > to copy in hadoop 2.4.x libs). It started fine over data written by > > > trunk. > > > > Seems fine. No unusual behaviors that I could see on cursory > review. > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > On Mon, May 26, 2014 at 2:18 PM, Andrew Purtell > > > > > wrote: > > > > > > > > > The 2nd HBase 0.98.3 release candidate (RC1) is available for > > download > > > at > > > > > http://people.apache.org/~apurtell/0.98.3RC1/ and Maven artifacts > > are > > > > also > > > > > available in the temporary repository > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1022 > > > > > > > > > > Signed with my code signing key D5365CCD. > > > > > > > > > > The issues resolved in this release can be found here: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326765 > > > > > > > > > > Please try out the candidate and vote +1/-1 by midnight Pacific > Time > > > > (00:00 > > > > > -0800 GMT) on June 1 on whether or not we should release this as > > > 0.98.3. > > > > > Three +1 votes from PMC will be required to release. > > > > > > > > > > -- > > > > > Best regards, > > > > > > > > > >- Andy > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > > Hein > > > > > (via Tom White) > > > > > > > > > > > > > > > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-11261) Handle splitting of regions that have region_replication greater than one
Devaraj Das created HBASE-11261: --- Summary: Handle splitting of regions that have region_replication greater than one Key: HBASE-11261 URL: https://issues.apache.org/jira/browse/HBASE-11261 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects
[ https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones resolved HBASE-11250. - Resolution: Fixed > Document: CLONE - Fix jersey serialization/deserialization of json objects > -- > > Key: HBASE-11250 > URL: https://issues.apache.org/jira/browse/HBASE-11250 > Project: HBase > Issue Type: Bug > Components: documentation, REST >Affects Versions: 0.92.2 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Blocker > > Stargate uses the default json marshaller/unmarshaller in natural mode. In > this mode the unmarshaller has trouble unmarshalling json instances. > This patch fixes this issue by using jackson as the marshaller/unmarshaller > instead. > I've also updated all the model unit tests to test json > serialization/deserialization. Backwards compatibilty can be verified by > modify the test base class to use the original marshaller/unmarshaller and > see that model tests pass. > The patch is backward compatible except for StorageClusterStatusModel, which > is broken anyway. It only shows one node in the liveNodes field. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6139) Add troubleshooting section for CentOS 6.2 page allocation failure issue
[ https://issues.apache.org/jira/browse/HBASE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones resolved HBASE-6139. Resolution: Won't Fix Centos 6.2 is very very old now. I don't think this info is worth adding to the Ref Guide anymore. Please reopen if you object. > Add troubleshooting section for CentOS 6.2 page allocation failure issue > > > Key: HBASE-6139 > URL: https://issues.apache.org/jira/browse/HBASE-6139 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Andrew Purtell >Assignee: Misty Stanley-Jones > > Tim Robertson reports: > bq. HBase CentOS version 6.2 reports kernel: java: page allocation failure. > order:4, mode:0x20. Any ideas anyone? > Then: > bq. echo 360448 > /proc/sys/vm/min_free_kbytes appears to stop page > allocation failure using HBase on CentOS 6.2 > If this is the proper fix for this condition, we should document it. > @Tim, how did you arrive at 360448? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-9729) mvn site fails without install first
[
https://issues.apache.org/jira/browse/HBASE-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones resolved HBASE-9729.
Resolution: Cannot Reproduce
Assignee: Misty Stanley-Jones
I can't reproduce this and think it must have been fixed long ago.
> mvn site fails without install first
>
>
> Key: HBASE-9729
> URL: https://issues.apache.org/jira/browse/HBASE-9729
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Enis Soztutar
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-9729-0.patch
>
>
> mvn site fails on hadoopqa runs, and also on fresh checkouts. The problem
> seems to be that mvn site somehow does not trigger a correct reactor
> ordering. hbase-server is built before other components, and thus throws
> dependency errors because the other modules are not build yet.
> An example from
> https://builds.apache.org/job/PreCommit-HBASE-Build/7491//consoleFull:
> {code}
> /home/jenkins/tools/maven/latest/bin/mvn compile site -DskipTests
> -DHBasePatchProcess >
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/patchSiteOutput.txt
> 2>&1
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-8100) Wiki Thrift Documentation For Scan Methods
[ https://issues.apache.org/jira/browse/HBASE-8100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones resolved HBASE-8100. Resolution: Cannot Reproduce This seems to have been fixed some time since reporting. > Wiki Thrift Documentation For Scan Methods > -- > > Key: HBASE-8100 > URL: https://issues.apache.org/jira/browse/HBASE-8100 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Jesse Anderson >Assignee: Misty Stanley-Jones > Labels: noob > > The documentation on the Wiki for the Scan methods in Thrift are wrong. The > wiki page is http://wiki.apache.org/hadoop/Hbase/ThriftApi. The openScanner, > getScannerResult and closeScanner methods aren't methods in the Thrift > interface. They should be scannerOpen, scannerGet, and scannerClose. Some > of the method paramaters might need to be changed too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-3696) No durability when running with LocalFileSystem
[ https://issues.apache.org/jira/browse/HBASE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-3696. -- Resolution: Implemented Resolving as implemented by HBASE-11218 (which should go in soon) at least for standalone. Data loss on fs has been doc'd also. Hopefully HBASE-11218 will do pseudo distributed mode too. If not, doc will need tweaking. Can do in new issue. > No durability when running with LocalFileSystem > --- > > Key: HBASE-3696 > URL: https://issues.apache.org/jira/browse/HBASE-3696 > Project: HBase > Issue Type: Bug > Components: documentation, wal >Reporter: Todd Lipcon > > LocalFileSystem in Hadoop doesn't currently implement sync(), so when we're > running in that case, we don't have any durability. This isn't a huge deal > since it isn't a realistic deployment scenario, but it's probably worth > documenting. It caused some confusion for a user when a table disappeared > after killing a standalone instance that was hosting its data in the local FS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-6575) Add SPM for HBase to Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones resolved HBASE-6575. Resolution: Fixed > Add SPM for HBase to Ref Guide > -- > > Key: HBASE-6575 > URL: https://issues.apache.org/jira/browse/HBASE-6575 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Otis Gospodnetic >Assignee: Misty Stanley-Jones >Priority: Minor > Attachments: HBASE-6575.patch > > > Ref Guide should point users to SPM for HBase in monitoring section(s). -- This message was sent by Atlassian JIRA (v6.2#6252)
