Thank you for taking this on!

On Mon, 20 Jan 2025 at 06:40, Istvan Toth <st...@cloudera.com.invalid>
wrote:

> Thanks to everyone who participated in the discussion and suggested
> improvements like additional testing.
>
> This has been merged on all branches, and I have also updated the docs to
> include the new policy.
>
> Istvan
>
> On Tue, Dec 17, 2024 at 2:51 PM Nick Dimiduk <ndimi...@apache.org> wrote:
>
> > I've come around to be in favor of bumping the default version for
> > branch-2.6 and maintaining it up on the latest backward-compatible minor
> > releases that we can. I arrive at this conclusion because we do not have
> > the bandwidth as a community to make more frequent minor or major
> releases.
> > That means the only way that we can stay ahead of the aggressive CVE
> train
> > coming out of our massive dependency stack is to aggressively upgrade the
> > deps.
> >
> > I've given my +1 on HBASE-28980 for branch-2.6
> >
> > Thanks,
> > Nick
> >
> > On 2024/11/15 06:26:08 Istvan Toth wrote:
> > > With Duo's fix for HBASE-28965 , there are no more known blockers.
> > >
> > > Should I go ahead with making 3.4.1 the default for branch-2.5 and
> > > branch-2.6 ?
> > >
> > > On Tue, Nov 5, 2024 at 7:16 AM Istvan Toth <st...@cloudera.com> wrote:
> > >
> > > > I've just committed the - hopefully - last test change, so the
> > nightlies
> > > > will not always fail on the packaging tests.
> > > >
> > > > All non-released branches (i.e. 2.7+ now default to building with
> > 3.4.1).
> > > >
> > > > I wanted to revisit updating the default version on the released
> (2.5,
> > > > 2.6) branches, because Nick has expressed concerns about it.
> > > >
> > > > The dev tests are good (the test failures seem to be "normal"
> > flakies). As
> > > > we've not updated the default version, we're not yet running the
> > packaging
> > > > tests
> > > > on branch-2.5 and 2.6 against the older Hadoop versions, but we do on
> > the
> > > > rest of branches, and if we update it, we will also run them on 2.5
> > and 2.6.
> > > >
> > > > Without repeating the arguments I have already made for it, I want to
> > add
> > > > a new one:
> > > >
> > > > Security and CVEs are getting more and more emphasis, which is great,
> > but
> > > > has some drawbacks.
> > > > While the proliferation of static analyzers leads to a lot of
> > frustrating
> > > > CVE witch hunts and false positives, the majority of users cannot
> > evaluate
> > > > the actual security impact,
> > > > or even if they can, they are tied by inflexible policies.
> > > >
> > > > We at Phoenix had recent security discussions with Trino, and ended
> up
> > > > having to dependencyManage the transitive Hadoop dependencies in our
> > shaded
> > > > uberjar to address their concerns.
> > > > (Which is an antipattern)
> > > >
> > > > While updating the Hadoop version in a patch release undoubtedly
> > increases
> > > > the risk of regressions, IMO we are protected by the Hadoop backwards
> > > > compatibility promises, and we have added a reasonable number of
> tests
> > to
> > > > catch any issues. I am confident in our test coverage when building
> > HBase
> > > > with any of the supported HBase versions. We also have the packaging
> > > > (smoke) tests for running the official binaries against earlier
> Hadoop
> > > > clusters, which would catch (some of) the Hadoop wire api
> > incompatibilities
> > > > .
> > > >
> > > > However, IMO not updating Hadoop is net negative to the project's
> > health,
> > > > as the binary (maven or assembly) releases are used primarily either
> by
> > > > other libraries interfacing with HBase, or by new users, POC
> clusters,
> > etc.
> > > > and these are the use cases where the transitive CVEs can prevent
> > projects
> > > > from adding (or maintaining as in the case of Trino) HBase support,
> or
> > > > discourage new users from adopting HBase.
> > > >
> > > > Obviously, I have a very skewed view of HBase users, but I think that
> > most
> > > > production HBase users either use a vendor or cloud provider version,
> > or
> > > > have enough in-house expertise to rebuild HBase with their Hadoop
> > version
> > > > if something goes wrong (despite the Hadoop backwards compatibility
> > > > promises)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 30, 2024 at 12:41 PM Istvan Toth <st...@cloudera.com>
> > wrote:
> > > >
> > > >> Thanks!
> > > >>
> > > >> I will backport the test changes, but keep the default Hadoop
> version.
> > > >>
> > > >> We will have more information then.
> > > >>
> > > >> Istvan
> > > >>
> > > >> On Wed, Oct 30, 2024 at 10:22 AM Nick Dimiduk <ndimi...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> On Mon, Oct 28, 2024 at 11:00 AM Istvan Toth
> > <st...@cloudera.com.invalid>
> > > >>> wrote:
> > > >>> >
> > > >>> > I have looked at branch-2.5, but the nightly looks off there, as
> it
> > > >>> runs
> > > >>> > the packaging tests with Hadoop 3.1.1, which it  doesn't even
> > > >>> officially
> > > >>> > support anymore.
> > > >>> >
> > > >>> > What should we do with branch-2.5 ?
> > > >>> >
> > > >>> > I think that it would not be a lot of extra  work to backport
> > > >>> everything,
> > > >>> > both the backwards compatibility tests and defaulting Hadoop to
> > 3.4.1.
> > > >>> > We just have to update the version in the pom, and add 3.2.4 to
> the
> > > >>> list of
> > > >>> > versions to test for backwards compatibility and integration (and
> > > >>> remove
> > > >>> > 3.1.1).
> > > >>> >
> > > >>> > I would prefer to have uniform tests and default to Hadoop 3.4.1
> > on all
> > > >>> > active branches.
> > > >>> > Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x
> > support
> > > >>> may
> > > >>> > be useful for users for migration and CVE mitigation purposes.
> > > >>> >
> > > >>> > WDYT ?
> > > >>>
> > > >>> branch-2.5's default hadoop3 version is 3.2.4. That's a big
> > dependency
> > > >>> to change for a patch release. I don't think that we can get away
> > with
> > > >>> that change and maintain our compatibility obligations. I'm not up
> to
> > > >>> speed on the current state of CVEs for this older (EOL?) version,
> so
> > > >>> we have that dimension to consider. If the newer version is
> "drop-in"
> > > >>> compatible (and only if), then I have no issue with moving that
> > > >>> release line forward. Ultimately it's the release manager for 2.5
> to
> > > >>> make a determination, so I defer to Andrew's assessment.
> > > >>>
> > > >>> I am in favor of backing-porting the improved testing coverage
> you've
> > > >>> added to branch-2.5. It would be great to understand if branch-2.5
> > (1)
> > > >>> compiled against 3.2.4 will run on Hadoop 3.4.1 and (2) builds and
> > > >>> tests out on 3.4.1. That will give the more security-minded users
> > > >>> additional confidence in bumping their hadoop dependency on their
> > own.
> > > >>>
> > > >>> Thanks,
> > > >>> Nick
> > > >>>
> > > >>> > On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth <st...@cloudera.com>
> > > >>> wrote:
> > > >>> >
> > > >>> > > We could also move the default to 3.4.1 directly.
> > > >>> > > We already test for 3.4.0 in the nightly job.
> > > >>> > >
> > > >>> > > On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) <
> > palomino...@gmail.com
> > > >>> >
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > >> And seems hadoop 3.4.1 is out. we could see whether to bump to
> > this
> > > >>> > >> version later?
> > > >>> > >>
> > > >>> > >> Istvan Toth <st...@cloudera.com.invalid> 于2024年10月21日周一
> > 20:56写道:
> > > >>> > >> >
> > > >>> > >> > I have merged the new tests to the nightly Jenkins runs on
> > master.
> > > >>> > >> >
> > > >>> > >> > They have identified another 3.4.0 incompatibility:
> > > >>> > >> > HBASE-28929 <
> > https://issues.apache.org/jira/browse/HBASE-28929>
> > > >>> > >> >
> > > >>> > >> > I will hold off backporting the test changes until
> > HBASE-28929 is
> > > >>> > >> resolved.
> > > >>> > >>
> > > >>> > >
> > > >>> > >
> > > >>> > > --
> > > >>> > > *István Tóth* | Sr. Staff Software Engineer
> > > >>> > > *Email*: st...@cloudera.com
> > > >>> > > cloudera.com <https://www.cloudera.com>
> > > >>> > > [image: Cloudera] <https://www.cloudera.com/>
> > > >>> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > [image:
> > > >>> > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > [image:
> > > >>> > > Cloudera on LinkedIn] <
> https://www.linkedin.com/company/cloudera
> > >
> > > >>> > > ------------------------------
> > > >>> > > ------------------------------
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > *István Tóth* | Sr. Staff Software Engineer
> > > >>> > *Email*: st...@cloudera.com
> > > >>> > cloudera.com <https://www.cloudera.com>
> > > >>> > [image: Cloudera] <https://www.cloudera.com/>
> > > >>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > [image:
> > > >>> > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> [image:
> > > >>> Cloudera
> > > >>> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > >>> > ------------------------------
> > > >>> > ------------------------------
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> *István Tóth* | Sr. Staff Software Engineer
> > > >> *Email*: st...@cloudera.com
> > > >> cloudera.com <https://www.cloudera.com>
> > > >> [image: Cloudera] <https://www.cloudera.com/>
> > > >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > >> ------------------------------
> > > >> ------------------------------
> > > >>
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: st...@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: st...@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------
> > >
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: st...@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>

Reply via email to