Hi Kevin, This is a very good point. Compatibility is one of the main requirements for a Hadoop release these days. This is particularly important for rolling upgrades that we have in Hadoop, when components from different versions run on the same cluster. So incompatible changes in a dot release are considered bugs and should be either fixed or reverted.
> if a 2.7.3 namenode would be compatible with a 2.7.4 datanode? Well the rolling upgrade procedure assumes that we upgrade NameNodes first then DataNodes. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html So 2.7.4 NameNode must work with 2.7.3 DataNodes. I do not know if it will work in reverse: old NN with new DNs. It might, but not guaranteed. > Are all hadoop 2.7.x components compatible with one another? We provide backward compatibility so that you can upgrade from older version to newer without cluster down time. But this does not mean you can arbitrary mix and match different components of different versions. Hope this is sufficient for your use case. Thanks, --Konstantin On Fri, Jul 14, 2017 at 4:48 PM, Kevin Monroe <kevin.mon...@canonical.com> wrote: > Hey folks, I have some reservation with this going into bigtop-1.2.1. > > @Konst, I maintain the Bigtop bits related to Canonical's Juju tool. This > tool allows you to deploy various Bigtop roles (namenode, datanode, etc) > and scale them as needed. Currently, I've coded these bits so that all > bigtop-1.2.x roles are compatible -- for example, an existing bigtop-1.2.0 > namenode (hadoop 2.7.3) would allow registration from a bigtop-1.2.1 > datanode (potentially hadoop 2.7.4). > > You mentioned that 2.7.4 would be compat from a MR perspective, but could > you tell me if a 2.7.3 namenode would be compatible with a 2.7.4 datanode? > Forgive my ignorance, but i've never mixed hadoop point releases in an > hdfs/mr environment. This is important to me because I have users with > long running deployments that will eventually scale their datanodes, which > would pick up the new bigtop-1.2.1 packages. > > I can work around this by insisting NN and DN bigtop repos must match. I > can also add docs to warn against mismatching components. However, I'd > rather not impose limitations when they're not needed. Are all hadoop > 2.7.x components compatible with one another? > > Thanks for any clarifications! > -Kevin > > On Fri, Jul 14, 2017 at 12:48 AM, Konstantin Boudnik <c...@apache.org> > wrote: > > > Totally, that's why I've asked about the git branch. Once I have I'll try > > to do it on a separate branch. Perhaps might even figure out if there's a > > bug in that functionality. > > -- > > Regards, > > Cos > > > > On July 13, 2017 9:42:39 PM PDT, Evans Ye <evan...@apache.org> wrote: > > >Cos, > > > > > >Can we build from git? > > >Last time i tried and it doea not work. > > >Would be great if you xan light me up a bit. > > > > > >MrAsanjar . <afsan...@gmail.com>於 2017年7月14日 週五，上午10:38寫道： > > > > > >> where is source for hadoop 2.7.4? can't find it in github. > > >> need it to verify JIRA > > >https://issues.apache.org/jira/browse/HADOOP-13579 > > >> > > >> On Thu, Jul 13, 2017 at 7:52 PM, Konstantin Boudnik <c...@apache.org> > > >> wrote: > > >> > > >> > On Thu, Jul 13, 2017 at 05:50PM, Konstantin Boudnik wrote: > > >> > > Certainly having an official release is much preferable. However, > > >we > > >> can > > >> > try > > >> > > to go with a branch for now. Say, I can change the patch to start > > >> > building > > >> > > from the git instead, so we can start testing it on a separate > > >branch > > >> > for now. > > >> > > > >> > Is there at least a branch where the work is being done? I don't > > >see > > >> 2.7.4 > > >> > in > > >> > the github. Please advise. > > >> > > > >> > > Once the release is ready (and if 1.2.1 is still unreleased on > > >our end) > > >> > we'll > > >> > > move to the official bits. > > >> > > > > >> > > Cos > > >> > > > > >> > > On Thu, Jul 13, 2017 at 05:33PM, Konstantin Shvachko wrote: > > >> > > > Thank you, Evans. Would be really good to include 2.7.4. > > >> > > > Do you guys need the actual Hadoop release for inclusion into a > > >> BigTop > > >> > > > release? > > >> > > > > > >> > > > Thanks, > > >> > > > --Konst > > >> > > > > > >> > > > On Wed, Jul 12, 2017 at 10:52 PM, Evans Ye <evan...@apache.org> > > >> wrote: > > >> > > > > > >> > > > > I'm actually OK to have 2.7.4 included in 1.2.1, if that's > > >ready in > > >> > master. > > >> > > > > Right now 1.2.1 release is blocked by building up a build > > >slave > > >> > docker > > >> > > > > image for SuSE 42.1. > > >> > > > > Once the obstacle removed. The release process will proceed. > > >> > > > > If 2.7.4 is ready then, we can try to include it. > > >> > > > > > > >> > > > > > > >> > > > > 2017-07-13 8:55 GMT+08:00 Konstantin Boudnik > > ><c...@apache.org>: > > >> > > > > > > >> > > > > > Oh, one more question: is there a JIRA ticket tracking > > >2.7.4 > > >> > release? I > > >> > > > > > couldn't find it somehow. Thanks > > >> > > > > > > > >> > > > > > Cos > > >> > > > > > > > >> > > > > > On Wed, Jul 12, 2017 at 05:45PM, Konstantin Boudnik wrote: > > >> > > > > > > Thanks Konstantin! > > >> > > > > > > > > >> > > > > > > I would venture the guess it includes mostly bug fixes > > >and > > >> > perhaps some > > >> > > > > > > minor performance improvements? I guess as such it won't > > >be > > >> that > > >> > much > > >> > > > > > > different from 2.7.3 we currently support. At least from > > >the > > >> > APIs and > > >> > > > > UX > > >> > > > > > point > > >> > > > > > > of view. Now, a few weeks might be an over-stretch (what > > >is it > > >> > by your > > >> > > > > > > estimation?) and we might not be willing to hold current > > >> release > > >> > that > > >> > > > > > long, > > >> > > > > > > but this would RM's call (hey Evans!), > > >> > > > > > > > > >> > > > > > > Adding this to the master (presumably for 1.3 or even > > >1.2.2) is > > >> > easy > > >> > > > > > though > > >> > > > > > > and should be done so we can upgrade the stack to the > > >latest > > >> > 2.7.x line > > >> > > > > > until > > >> > > > > > > 2.8 (or whatever would be the next one) line is mature > > >enough. > > >> > At any > > >> > > > > > rate, I > > >> > > > > > > have created BIGTOP-2839 to track this. > > >> > > > > > > > > >> > > > > > > Thoughts? > > >> > > > > > > Cos > > >> > > > > > > > > >> > > > > > > On Wed, Jul 12, 2017 at 01:17PM, Konstantin Shvachko > > >wrote: > > >> > > > > > > > Hi guys, > > >> > > > > > > > > > >> > > > > > > > We plan to release Hadoop 2.7.4 in the next few weeks. > > >> > > > > > > > I was wondering if you could include it into Bigtop. > > >> > > > > > > > It would be really valuable to add Bigtop testing for > > >the > > >> > release. > > >> > > > > > > > I believe you already have support for earlier Hadoop > > >2.7 > > >> > release. > > >> > > > > > > > Hope it wouldn't be too much trouble to integrate > > >2.7.4. > > >> > > > > > > > > > >> > > > > > > > Thanks, > > >> > > > > > > > --Konst > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > >> > > > >> > > > >> > > >
