On Mon, Nov 4, 2013 at 9:13 PM, Andrew Purtell <apurt...@apache.org> wrote:
> When last we spoke on dev@ about 0.98 I suggested branching one month > after > the 0.96.0 release. There was no comment or objection. That would be late > November then. No problem to branch sooner. > > Sorry none of us at Intel could be there in person. For 0.98, to add to the > below list there are a couple of security related issues we're aiming for > the release, based on tags, coprocessors, and HFile v3: cell ACLs, > visibility labels, and transparent encryption. > > Branch end of Nov. sounds about right for getting the above in. Good on you Andrew, St.Ack On Mon, Nov 4, 2013 at 9:13 PM, Andrew Purtell <apurt...@apache.org> wrote: > When last we spoke on dev@ about 0.98 I suggested branching one month > after > the 0.96.0 release. There was no comment or objection. That would be late > November then. No problem to branch sooner. > > Sorry none of us at Intel could be there in person. For 0.98, to add to the > below list there are a couple of security related issues we're aiming for > the release, based on tags, coprocessors, and HFile v3: cell ACLs, > visibility labels, and transparent encryption. > > > > On Saturday, November 2, 2013, Stack wrote: > > > Below are rough minutes from the developer meetup last Thursday week down > > at HortonWorks: http://www.meetup.com/hackathon/events/144366512/ > > > > The agenda was fast moving and the notes I kept were sparse (pardon me). > > Hopefully the below at least conveys some flavor of what transpired. > > > > > > Below is proposed agenda with discussion filled in in between in italics. > > > > Git/Gerrit< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.bi2n1jnf8wr9 > > > > > > > Lieutenants< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ts3zsiollp5s > > > > > > > MTTR< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2 > > > > > > > Distributed log > > replay< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.5nxjf2azyr63 > > > > > > > Region online for > > writes< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.7h4u62q0yu1t > > > > > > > Client< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.dxyzoaydy7up > > > > > > > 0.98< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.jny6o6f3j3x8 > > > > > > > 1.0< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.4sjuoukolha1 > > > > > > > Release passes IT tests for time period before > > release?< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ijcofohq0rk1 > > > > > > > Sequenceid< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.14eyjkb3glf1 > > > > > > > MVCC< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.hbv260gur4ui > > > > > > > Multiwal< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.tab1edfjmpgq > > > > > > > Speculative Read< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.3xr7mrnpmpby > > > > > > > Favored node finish > > up< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.2s9y1srcix0z > > > > > > > Compactions< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.23pauxcj0ux8 > > > > > > > Stripe?< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.xsncn8yoed07 > > > > > > > Master Redo< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2 > > > > > > > Secondary indices from > > phoenix< > > > https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.mk56oao2p3vj > > > > > > > > > Git/Gerrit > > > > Move to this? Or do Git without gerrit ? > > > > All were +1 on going to GIT. A gentleman Accumulo fellow present > > volunteered to pass us his script for how they made the transition. > > DISCUSSION already started on dev list. > > > > Lieutenants/Component Owners > > > > Review. How is this going? What can we do to improve? Not enough > reviews > > going on. > > > > The list of Component Owners needs an edit. It is stale. We need to > > revive/renew/refresh this initiative. > > > > More Friction Committing > > > > Not enough reviews of stuff going in and this is a db after all? +1 > > (eclark) +1 from Dave > > > > Generally agreed. Pointed at Lieutenants/Component Owners role. Was > > thought there should be some sort of automated performance test as part > of > > hadoopqa. A benchmark proving no degradation of a claimed improvement is > > required. > > > > Better was adding some set of general micro benchmarks... and then do > them > > on your machine before and after and paste findings. > > > > Compat testing > > > > Now that 0.96.0 is out the door, we should be careful about changes that > > are compat-breaking. Any ideas on how to best avoid/detect such changes? > > Also maybe have a discussion about how wire compat/protobufs changes how > > we do things, how to best utilize it, pitfalls to avoid, what it can’t > do? > > > > Formalize compat matrix? Go over the matrix we had 1 year ago. Do we want > > to support all combinations, or only between two major releases with some > > rolling upgrade model? How to avoid current situation where client does > not > > scale -- found at last minute running on 80 node cluster! > > > > Test framework do this? But would have to be a non-apache context > because > > no resources there to do it. There are the jdiff and compare. We have to > > do rolling upgrade tests. Deprecations in .protos too. Remove after a > > release. Just go version numbers. Write up the matrix. How to avoid > issues > > like the recent client not scaling. Need perf tests. Need > > microbenchmarks. Modular so can pull out and simulate rpc. Tests like > > that. Standup simple server done in proto > > MTTRDistributed log replay > > > > Relax semantics; it is ok to allow out-of-order edits? Or fix? > > > > Distributed log replay going to be done in 0.98 using tags. > > Region online for writes > > > > Lets just do this. > > > > Client > > > > - Short term perf/scalability regression (0.96) time frame > > > > - Long term (0.98/1.x/2.x) time frame. > > > > - Asynchbase on 0.96 - almost there. > > > > Should we remove support for setting timestamp. Lars says set it in > > HTableDescription. Sequence number and ts. Sequenceid Perhaps add new > > coordinate. Out-of-order deletes. Ts w/ the value. Make it configurable > > and do optimization. > > > > *Aditya brought up new c-client effort.* > > 0.98 > > > > - tags. > > > > - reverse scan > > > > - issue with cells? (intel guys at mtg?) > > > > Where is the branch? > > 1.0 > > > > is 0.98 == 1.0? > > > > - when to drop hadoop1 support? next year? > > > > > > 98.x becomes 1.0 Security not as a CP but in codepath. Do check in the > > code. Make security first class. But security was always good for dev'ing > > CP. Make it a required CP. Integrated w/ simple testing. Enable > > Authorization by default. Permissive mode needs to be added. Small perf > > price. > > > > > > We need better defaults -- especially for 1.0. Revisit the docs, the > > refguide. An edit. Questions, the docbook.... Formatting is a pain. > > Docathon? > > ….but no one would show up. Drop hadoop1 post 1.0. > > > > New release adoption > > > > - What can we do for 0.96 adoption > > > > - People are still using 0.90 > > > > Within versions, update... You are in the middle of a rolling upgrade. > Can > > we make it easier for folks to update? Advertising that upgrade to 0.96 > is > > successful. Rolling upgrade needs to be in the master, not out in a > script. > > Release passes IT tests for time period before release? > > > > 48 hours Real honest, regionserver, datanodes, zookeepers all being > > killed. Committers releasing it for a time. Failing a 48 hour test is > > enough to sink an RC. We have a bar and we should keep raising it. > > > > Elliott will put up bar for test release. > > > > SequenceidMVCC > > > > Unify sequenceid and mvcc > > > > Do we need locks at all? > > MultiwalSpeculative Read > > > > Raise awareness with hdfs brothers and sisters > > Favored node finish up > > > > Add consideration to the balancer > > > > Need to fix balancer. Not AM job. Its in AM and in Balancer. > > > > > > CompactionsStripe? > > > > Get some empirical data to see if different types of compactions are > > better. > > Master Redo > > > > Jimmy says evolve instead of rewrite > > Secondary indices from phoenix? > > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
