Subject changed to not clutter the RC thread. I've been doing testing with HDFS WAL replication of 2 and the default (3) and it has not made a huge difference. Probably about 10%. As easy as it is to point at that and say that because we are writing 1.5x the data then we must have 1.5x the slowdown, I don't think that is the case here.
We'll run a couple more rounds of CI with different settings and report back. On Wed, Feb 19, 2014 at 2:39 PM, Keith Turner <[email protected]> wrote: > On Wed, Feb 19, 2014 at 2:21 PM, Sean Busbey <[email protected] > >wrote: > > > -1 (if we're still counting votes) due to #3 below > > > > Here's where I am ATM: > > > > * Verified data integrity for data written in 1.4.4 after upgrade (for a > > smattering of rfile options, built from source dist with hadoop 2 > profile) > > * 2x Continuous Ingest 24hr w/verification[1] (built from source dist > with > > hadoop 2 profile) > > * once for each of RC1 and RC2 > > * no agitation due to ACCUMULO-2382 discovered after the fact (NN > > failover still present) > > * Significantly fewer cells written than when I last ran on the same > > cluster with 1.4.5-6593a9+agitation (7B vs 31B) > > * functional tests of binary distro pass on Hadoop 2, given > workarounds[2] > > > > 1) None of the issues I ran into running tests look like blockers; > they've > > all been filed at this point. > > > > 2) The significant decrease in write throughput might be concerning, but > I > > don't know if this was already in 1.5.0 so I'm not flagging it. > > > > There is a difference in replication between 1.4 and 1.5. 1.4 used to > replicated data to two loggers. 1.5 using HDFS defaults will replicate > walogs to 3 datanodes. ACCUMULO-1083 [1] has some discussion and numbers > about this. There is also ACCUMULO-1905 [2], 1.4 would not call hsync. > > [1] : https://issues.apache.org/jira/browse/ACCUMULO-1083 > [2] : https://issues.apache.org/jira/browse/ACCUMULO-1905 > > > > > > 3) the release notes need to have things broken out by version. Otherwise > > you're asking an ops person to go back and look at the 1.5.0 release > notes > > to determine how 1.5.1 impacts them. For comparison, both Avro and > Jackson > > (which I consider good exemplars for projects) break out their release > > notes to the bugfix[3]. > > > > 4) I'm a little concerned that no one has done a Hadoop 1 test yet. > > > > [1]: Cluster Specs > > OS: CentOS 6.4 > > Hadoop: CDH 4.5.0 (2.0.0+cdh4.5.0) > > ZK: CDH 4.5.0 (3.4.5+cdh4.5.0) > > Size: 2 Masters, 5 Workers, HDFS in HA+QJM, 5 ZKs > > > > [2]: Run on single node (backed by the same cluster Bill mentioned > earlier) > > OS: CentOS 6.4 > > Hadoop: CDH 4.5.0 (2.0.0+cdh4.5.0) > > ZK: CDH 4.5.0 (3.4.5+cdh4.5.0) > > Size: 2 Masters, 5 Workers, HDFS in HA+QJM, 3 ZKs > > > > [3]: e.g. > > http://svn.codehaus.org/jackson/tags/1.8/1.8.9/release-notes/VERSION > > https://github.com/apache/avro/blob/trunk/CHANGES.txt > > > > > > On Wed, Feb 19, 2014 at 12:23 PM, Mike Drob <[email protected]> wrote: > > > > > I went back and looked at our release governance page[1] and it does > > > explicitly state that votes will be 72 hours. So I was out of line when > > > asking you to extend it and I'm not sure that the extension is valid at > > > this point anyway. Lack of bylaws makes this a messy process. > > > > > > In light of this I am changing my vote from +1 to +0, since I did not > > vote > > > in the original time frame. > > > > > > [1]: http://accumulo.apache.org/governance/releasing.html#releasing > > > > > > > > > On Sat, Feb 15, 2014 at 7:17 PM, Josh Elser <[email protected]> > > wrote: > > > > > > > Alright, given the snow, holiday, and the lack of bylaws stating > that I > > > > cannot do this: > > > > > > > > I'm extending the VOTE on 1.5.1-RC2 until 02/19/2014 1900 EST (this > > > > extends the original duration to a week for those keeping track). > This > > is > > > > expected to provide an additional two full work days for people to > > > inspect > > > > the release. > > > > > > > > Let's get some good feedback before then, folks. > > > > > > > > - Josh > > > > > > > > > > > > On 2/15/14, 6:29 PM, Christopher wrote: > > > > > > > >> Either way works for me. > > > >> > > > >> I was just suggesting a more formal approach in the absence of > bylaws > > > >> that explicitly permit extensions. The general concern, I suppose, > is > > > >> that vote extensions could be used to manipulate to a desired > outcome > > > >> in a majority approval scheme... so having the vote conditions fixed > > > >> at the time it is announced prevents that. I don't think that's a > > > >> serious concern, though... especially since we all have the same > goal > > > >> of producing a quality release, and preventing one that falls short > of > > > >> that. > > > >> > > > >> With the bylaws in place, things are simpler, because we'd have > > > >> already agreed on those bylaws, and wouldn't need to do anything > > > >> silly, like vote on whether to allow a vote extension in the first > > > >> place (which would get obnoxious). > > > >> > > > >> -- > > > >> Christopher L Tubbs II > > > >> http://gravatar.com/ctubbsii > > > >> > > > >> > > > >> On Sat, Feb 15, 2014 at 6:00 PM, Billie Rinaldi > > > >> <[email protected]> wrote: > > > >> > > > >>> On Sat, Feb 15, 2014 at 1:57 PM, Christopher <[email protected]> > > > >>> wrote: > > > >>> > > > >>> A somewhat more formal way of "extending" the vote would be to > > simply > > > >>>> retract/cancel this vote (or let it lapse with no votes), and just > > > >>>> re-issue another vote with identical artifacts at a more opportune > > > >>>> time. I point this out for two reasons: > > > >>>> > > > >>>> 1) I don't want to undermine Josh's work to create this release > > > >>>> candidate. He shouldn't have to do that again if nothing has > changed > > > >>>> and we just need more time to review. And, > > > >>>> > > > >>>> 2) The vote was called with a 72hr. notice, and changing that > after > > > >>>> the fact is probably a bit questionable. We can achieve the same > > > >>>> effect without modifying the characteristics of the vote, by > simply > > > >>>> calling a new vote, identical to this one, later. > > > >>>> > > > >>>> > > > >>> I'm not sure that extending the vote is questionable. I think it > > would > > > >>> be > > > >>> fine if Josh just said the vote deadline is extended to X (perhaps > an > > > >>> additional 72 hours, or maybe event one week from the original post > > > since > > > >>> many people have Monday off). Some Apache projects explicitly > > mention > > > >>> that > > > >>> votes may be extended in their bylaws [1], so that's something we > > could > > > >>> consider when we write ours. > > > >>> > > > >>> But if people would feel more comfortable if Josh reposted the > vote, > > > I'm > > > >>> sure he could do that. :-) > > > >>> > > > >>> [1]: https://hc.apache.org/bylaws.html > > > >>> > > > >>> -- > > > >>> Christopher L Tubbs II > > > >>> http://gravatar.com/ctubbsii > > > >>> > > > >>> > > > >>> On Fri, Feb 14, 2014 at 6:09 PM, Christopher <[email protected]> > > > >>> wrote: > > > >>> > > > >>>> More time would be great. I'll still try to finish up some testing > > by > > > >>>>> tomorrow, but I can't make any guarantees. > > > >>>>> > > > >>>>> -- > > > >>>>> Christopher L Tubbs II > > > >>>>> http://gravatar.com/ctubbsii > > > >>>>> > > > >>>>> > > > >>>>> On Fri, Feb 14, 2014 at 12:43 PM, Josh Elser < > [email protected] > > > > > > >>>>> > > > >>>> wrote: > > > >>>> > > > >>>>> If people want some extra time given the impact of snow, please > > > inform. > > > >>>>>> > > > >>>>> I'm > > > >>>> > > > >>>>> ok with extending this a few days if it means people will give it > > > more > > > >>>>>> > > > >>>>> love. > > > >>>> > > > >>>>> > > > >>>>>> > > > >>>>>> On 2/12/14, 6:50 PM, Josh Elser wrote: > > > >>>>>> > > > >>>>>>> > > > >>>>>>> All, > > > >>>>>>> > > > >>>>>>> Please consider the following candidate as Apache Accumulo > 1.5.1 > > > >>>>>>> > > > >>>>>>> Git artifacts: The staging repository was built from the branch > > > >>>>>>> "1.5.1-rc2" (c810f51b). No accompanying git tag was created yet > > (as > > > >>>>>>> it > > > >>>>>>> would be the same exact thing as providing the above SHA1). > > > >>>>>>> > > > >>>>>>> Maven Staged Repo: > > > >>>>>>> > > > >>>>>>> https://repository.apache.org/content/repositories/ > > > >>>> orgapacheaccumulo-1001 > > > >>>> > > > >>>>> > > > >>>>>>> Source tarball: > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> http://repository.apache.org/content/repositories/ > > > >>>> orgapacheaccumulo-1001/org/apache/accumulo/accumulo/1.5. > > > >>>> 1/accumulo-1.5.1-src.tar.gz > > > >>>> > > > >>>>> > > > >>>>>>> > > > >>>>>>> Binary tarball: > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> http://repository.apache.org/content/repositories/ > > > >>>> orgapacheaccumulo-1001/org/apache/accumulo/accumulo/1.5. > > > >>>> 1/accumulo-1.5.1-bin.tar.gz > > > >>>> > > > >>>>> > > > >>>>>>> > > > >>>>>>> Changes since 1.5.1-RC1: ACCUMULO-1908, ACCUMULO-1935, > > > ACCUMULO-2299, > > > >>>>>>> ACCUMULO-2329, ACCUMULO-2331, ACCUMULO-2332, ACCUMULO-2334, > > > >>>>>>> ACCUMULO-2337, ACCUMULO-2342, ACCUMULO-2344, ACCUMULO-2356, > > > >>>>>>> > > > >>>>>> ACCUMULO-2360 > > > >>>> > > > >>>>> > > > >>>>>>> Changes since 1.5.0: > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a= > > > >>>> commitdiff;h=d277321d176b71753d391f896f09dc9785173cb0 > > > >>>> > > > >>>>> > > > >>>>>>> > > > >>>>>>> Keys: http://www.apache.org/dist/accumulo/KEYS > > > >>>>>>> > > > >>>>>>> Testing: > > > >>>>>>> > > > >>>>>>> Manual testing and verification of fixes since RC1 and 12hr CI > > with > > > >>>>>>> verification performed. All previously mentioned testing done > for > > > >>>>>>> RC1. > > > >>>>>>> > > > >>>>>>> This vote will be open for the next 72 hours. > > > >>>>>>> > > > >>>>>>> Upon successful completion of this vote, a 1.5.1 gpg-signed Git > > tag > > > >>>>>>> > > > >>>>>> will > > > >>>> > > > >>>>> be created from c810f51b and the above staging repository will be > > > >>>>>>> promoted. > > > >>>>>>> > > > >>>>>>> - Josh > > > >>>>>>> > > > >>>>>> > > > >>>> > > > > > >
