On Fri, Mar 20, 2015 at 2:59 PM, Jerry He <jerry...@gmail.com> wrote:
> On Wed, Mar 18, 2015 at 7:38 AM, Jonathan Hsieh <j...@cloudera.com> wrote: > > > We are now considered a mature project and as a "middle turtle" project > we > > need to consider the projects that we will inconvenience because > > they depend on us. All the folks in this discussion are hbase devs and or > > sophisticated hbase users. I agree with one of Sean's thoughts -- let us > > validate if this is a concern for users or not by have a discussion on > > user@ > > to see how our users feel about it. Maybe this is no big deal for them. > > > This probably can serve as a head-up for the user community as well. > > > > > > > Let's say that this is a concern for the users. I like Lars's pragmatic > > critieria -- if the dep change doesn't cause problems for intra version > > interoperability then, let's let it in and let's update our policy. If > it > > does cause problem's let's keep it out and stick to our guns. > > > > To be more concrete there are two scenarios we could test: > > 1) an mr job that uses hbase 1.0 client (and it's hadoop dep) needs to > > work out of the box against 1.1 hbase servers (and it's hadoop deps) or a > > mixed hbase 1.1- hbase 1.0 environment (e.g. assume we're in the middle > of > > rolling upgrading hbase) without failing due to dependency compat checks. > > > > See the coverage matrix below > > > > 2) an mr job that uses hbase 1.0 client (and it's hadoop dep) needs to > work > > out of the box against 1.1 hbase servers while the hdfs before and after > it > > is upgraded (either rolling or shutdown-restart upgrade) > > If both succeed despite dep changes, we are in in the clear. If it fails > > then we do not allow the dep change. > > > > (hbase 1.0 client w/ jackson 1.8 + hadoop 2.4 ) go against (hbase 1.1 > server w/ jackson 1.9 + hadoop 2.4) > > (hbase 1.0 client w/ jackson 1.8 + hadoop 2.4 ) go against (hbase 1.1 > server w/ jackson 1.9 + hadoop 2.6) > > (hbase 1.0 client w/ jackson 1.8 + hadoop 2.6 ) go against (hbase 1.1 > server w/ jackson 1.9 + hadoop 2.6) > > (hbase 1.0 client w/ jackson 1.8 + hadoop 2.4 ) go against (mixed of hbase > 1.0 w/ jackson 1.8 + hbase 1.1 server w/ Jackson 1.9 + hadoop 2.6) > > Is this the coverage you meant? > > > Yes -- with emphasis on the last one because of our rolling upgrade promise for hbase 1.0->1.1 > > > To avoid and simplify this problem on our internal builds, we have done > the > > exercise of harmonizing all to one version of jackson/guava/etc or > shading > > jars for the combination of hadoop and hbase (and other systems) that we > be > > ship. I'd like to be in a world where this work is upstream. At the > > moment, we don't know whether an upgrade to 2.6 hadoop including its > > Jackson 1.9 dep will have MR jobs fail or succeed because of classpath > > ordering issues. It sounds like it is an may be an issue but we should > > test to make sure and decide based on that. > > > > Jon. > > > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh
