Thank you for taking this on! On Mon, 20 Jan 2025 at 06:40, Istvan Toth <st...@cloudera.com.invalid> wrote:
> Thanks to everyone who participated in the discussion and suggested > improvements like additional testing. > > This has been merged on all branches, and I have also updated the docs to > include the new policy. > > Istvan > > On Tue, Dec 17, 2024 at 2:51 PM Nick Dimiduk <ndimi...@apache.org> wrote: > > > I've come around to be in favor of bumping the default version for > > branch-2.6 and maintaining it up on the latest backward-compatible minor > > releases that we can. I arrive at this conclusion because we do not have > > the bandwidth as a community to make more frequent minor or major > releases. > > That means the only way that we can stay ahead of the aggressive CVE > train > > coming out of our massive dependency stack is to aggressively upgrade the > > deps. > > > > I've given my +1 on HBASE-28980 for branch-2.6 > > > > Thanks, > > Nick > > > > On 2024/11/15 06:26:08 Istvan Toth wrote: > > > With Duo's fix for HBASE-28965 , there are no more known blockers. > > > > > > Should I go ahead with making 3.4.1 the default for branch-2.5 and > > > branch-2.6 ? > > > > > > On Tue, Nov 5, 2024 at 7:16 AM Istvan Toth <st...@cloudera.com> wrote: > > > > > > > I've just committed the - hopefully - last test change, so the > > nightlies > > > > will not always fail on the packaging tests. > > > > > > > > All non-released branches (i.e. 2.7+ now default to building with > > 3.4.1). > > > > > > > > I wanted to revisit updating the default version on the released > (2.5, > > > > 2.6) branches, because Nick has expressed concerns about it. > > > > > > > > The dev tests are good (the test failures seem to be "normal" > > flakies). As > > > > we've not updated the default version, we're not yet running the > > packaging > > > > tests > > > > on branch-2.5 and 2.6 against the older Hadoop versions, but we do on > > the > > > > rest of branches, and if we update it, we will also run them on 2.5 > > and 2.6. > > > > > > > > Without repeating the arguments I have already made for it, I want to > > add > > > > a new one: > > > > > > > > Security and CVEs are getting more and more emphasis, which is great, > > but > > > > has some drawbacks. > > > > While the proliferation of static analyzers leads to a lot of > > frustrating > > > > CVE witch hunts and false positives, the majority of users cannot > > evaluate > > > > the actual security impact, > > > > or even if they can, they are tied by inflexible policies. > > > > > > > > We at Phoenix had recent security discussions with Trino, and ended > up > > > > having to dependencyManage the transitive Hadoop dependencies in our > > shaded > > > > uberjar to address their concerns. > > > > (Which is an antipattern) > > > > > > > > While updating the Hadoop version in a patch release undoubtedly > > increases > > > > the risk of regressions, IMO we are protected by the Hadoop backwards > > > > compatibility promises, and we have added a reasonable number of > tests > > to > > > > catch any issues. I am confident in our test coverage when building > > HBase > > > > with any of the supported HBase versions. We also have the packaging > > > > (smoke) tests for running the official binaries against earlier > Hadoop > > > > clusters, which would catch (some of) the Hadoop wire api > > incompatibilities > > > > . > > > > > > > > However, IMO not updating Hadoop is net negative to the project's > > health, > > > > as the binary (maven or assembly) releases are used primarily either > by > > > > other libraries interfacing with HBase, or by new users, POC > clusters, > > etc. > > > > and these are the use cases where the transitive CVEs can prevent > > projects > > > > from adding (or maintaining as in the case of Trino) HBase support, > or > > > > discourage new users from adopting HBase. > > > > > > > > Obviously, I have a very skewed view of HBase users, but I think that > > most > > > > production HBase users either use a vendor or cloud provider version, > > or > > > > have enough in-house expertise to rebuild HBase with their Hadoop > > version > > > > if something goes wrong (despite the Hadoop backwards compatibility > > > > promises) > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 30, 2024 at 12:41 PM Istvan Toth <st...@cloudera.com> > > wrote: > > > > > > > >> Thanks! > > > >> > > > >> I will backport the test changes, but keep the default Hadoop > version. > > > >> > > > >> We will have more information then. > > > >> > > > >> Istvan > > > >> > > > >> On Wed, Oct 30, 2024 at 10:22 AM Nick Dimiduk <ndimi...@apache.org> > > > >> wrote: > > > >> > > > >>> On Mon, Oct 28, 2024 at 11:00 AM Istvan Toth > > <st...@cloudera.com.invalid> > > > >>> wrote: > > > >>> > > > > >>> > I have looked at branch-2.5, but the nightly looks off there, as > it > > > >>> runs > > > >>> > the packaging tests with Hadoop 3.1.1, which it doesn't even > > > >>> officially > > > >>> > support anymore. > > > >>> > > > > >>> > What should we do with branch-2.5 ? > > > >>> > > > > >>> > I think that it would not be a lot of extra work to backport > > > >>> everything, > > > >>> > both the backwards compatibility tests and defaulting Hadoop to > > 3.4.1. > > > >>> > We just have to update the version in the pom, and add 3.2.4 to > the > > > >>> list of > > > >>> > versions to test for backwards compatibility and integration (and > > > >>> remove > > > >>> > 3.1.1). > > > >>> > > > > >>> > I would prefer to have uniform tests and default to Hadoop 3.4.1 > > on all > > > >>> > active branches. > > > >>> > Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x > > support > > > >>> may > > > >>> > be useful for users for migration and CVE mitigation purposes. > > > >>> > > > > >>> > WDYT ? > > > >>> > > > >>> branch-2.5's default hadoop3 version is 3.2.4. That's a big > > dependency > > > >>> to change for a patch release. I don't think that we can get away > > with > > > >>> that change and maintain our compatibility obligations. I'm not up > to > > > >>> speed on the current state of CVEs for this older (EOL?) version, > so > > > >>> we have that dimension to consider. If the newer version is > "drop-in" > > > >>> compatible (and only if), then I have no issue with moving that > > > >>> release line forward. Ultimately it's the release manager for 2.5 > to > > > >>> make a determination, so I defer to Andrew's assessment. > > > >>> > > > >>> I am in favor of backing-porting the improved testing coverage > you've > > > >>> added to branch-2.5. It would be great to understand if branch-2.5 > > (1) > > > >>> compiled against 3.2.4 will run on Hadoop 3.4.1 and (2) builds and > > > >>> tests out on 3.4.1. That will give the more security-minded users > > > >>> additional confidence in bumping their hadoop dependency on their > > own. > > > >>> > > > >>> Thanks, > > > >>> Nick > > > >>> > > > >>> > On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth <st...@cloudera.com> > > > >>> wrote: > > > >>> > > > > >>> > > We could also move the default to 3.4.1 directly. > > > >>> > > We already test for 3.4.0 in the nightly job. > > > >>> > > > > > >>> > > On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) < > > palomino...@gmail.com > > > >>> > > > > >>> > > wrote: > > > >>> > > > > > >>> > >> And seems hadoop 3.4.1 is out. we could see whether to bump to > > this > > > >>> > >> version later? > > > >>> > >> > > > >>> > >> Istvan Toth <st...@cloudera.com.invalid> 于2024年10月21日周一 > > 20:56写道： > > > >>> > >> > > > > >>> > >> > I have merged the new tests to the nightly Jenkins runs on > > master. > > > >>> > >> > > > > >>> > >> > They have identified another 3.4.0 incompatibility: > > > >>> > >> > HBASE-28929 < > > https://issues.apache.org/jira/browse/HBASE-28929> > > > >>> > >> > > > > >>> > >> > I will hold off backporting the test changes until > > HBASE-28929 is > > > >>> > >> resolved. > > > >>> > >> > > > >>> > > > > > >>> > > > > > >>> > > -- > > > >>> > > *István Tóth* | Sr. Staff Software Engineer > > > >>> > > *Email*: st...@cloudera.com > > > >>> > > cloudera.com <https://www.cloudera.com> > > > >>> > > [image: Cloudera] <https://www.cloudera.com/> > > > >>> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > [image: > > > >>> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > > [image: > > > >>> > > Cloudera on LinkedIn] < > https://www.linkedin.com/company/cloudera > > > > > > >>> > > ------------------------------ > > > >>> > > ------------------------------ > > > >>> > > > > > >>> > > > > >>> > > > > >>> > -- > > > >>> > *István Tóth* | Sr. Staff Software Engineer > > > >>> > *Email*: st...@cloudera.com > > > >>> > cloudera.com <https://www.cloudera.com> > > > >>> > [image: Cloudera] <https://www.cloudera.com/> > > > >>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > [image: > > > >>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> > [image: > > > >>> Cloudera > > > >>> > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > >>> > ------------------------------ > > > >>> > ------------------------------ > > > >>> > > > >> > > > >> > > > >> -- > > > >> *István Tóth* | Sr. Staff Software Engineer > > > >> *Email*: st...@cloudera.com > > > >> cloudera.com <https://www.cloudera.com> > > > >> [image: Cloudera] <https://www.cloudera.com/> > > > >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > >> ------------------------------ > > > >> ------------------------------ > > > >> > > > > > > > > > > > > -- > > > > *István Tóth* | Sr. Staff Software Engineer > > > > *Email*: st...@cloudera.com > > > > cloudera.com <https://www.cloudera.com> > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > ------------------------------ > > > > ------------------------------ > > > > > > > > > > > > > -- > > > *István Tóth* | Sr. Staff Software Engineer > > > *Email*: st...@cloudera.com > > > cloudera.com <https://www.cloudera.com> > > > [image: Cloudera] <https://www.cloudera.com/> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > Cloudera > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > ------------------------------ > > > ------------------------------ > > > > > > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ >
