[DISCUSS] Move default Hadoop version to 3.3.x

[Wei-Chiu Chuang]

Hi HBase devs,

Over the past few years HBase supports the default Hadoop version of 3.2.x
but it also works on Hadoop 3.3.x.

I'm wondering if it makes sense to move the current default hadoop.version
 from 3.2.4 to
3.3.x.

Why?

1. From a stability and security point of view, Hadoop 3.3 is the most up
to date release line. And all HBase tests pass using 3.3.x. There hasn't
been a new Hadoop 3.2.x release for over a year.

2. We have a feature (using HBase on Ozone) that depends on an API in
Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
hadoop version to 3.3.6 will save a lot of hassle.

Thoughts?

Best,
Weichiu

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Tak Lon (Stephen) Wu]

coming back here, I created the
https://github.com/apache/hbase/tree/HBASE-27740 and will send PRs
accordingly...

-Stephen

On Thu, Jun 22, 2023 at 3:14 PM Wei-Chiu Chuang  wrote:

> I am +1 to use a feature branch.
>
> On Tue, Jun 20, 2023 at 10:20 AM Tak Lon (Stephen) Wu 
> wrote:
>
> > or maybe we create a new feature branch hadoop-33-ozone that has these
> > interfaces and ozone related support, then we put all the feature changes
> > into this feature branch and merge later?
> >
> > The problem I see is that it's hard and very confusing to maintain two
> > hadoop3 profiles, and I can see sooner or later hadoop 3.2.x could be
> EOL.
> >
> > Thanks,
> > Stephen
> >
> >
> >
> > On Fri, Jun 16, 2023 at 11:32 AM Viraj Jasani 
> wrote:
> >
> > > How about using a new hadoop 3.3 profile for features that are
> explicitly
> > > present in 3.3 (like FileSystem changes)? When the time comes, we
> switch
> > to
> > > 3.3 profile by default and drop old hadoop 3 profile that supports
> 3.2.x
> > > versions as of today?
> > >
> > >
> > > On Fri, Jun 16, 2023 at 7:11 AM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > >> In general, in HBase, we will use the last patch release of the oldest
> > >> supported hadoop release line as our default hadoop dependency.
> > >>
> > >> For example, since we claim that 3.x will support hadoop 3.2.x and
> > >> 3.3.x, then we will declare the default hadoop version as 3.2.4.
> > >>
> > >> I think we can discuss whether to move up to 3.3.6 as the default
> > >> version, if there are no compatibility issues when communicating with
> > >> 3.2.x hadoop clusters.
> > >>
> > >> But if we want to use the features which are only provided in 3.3.6,
> > >> then we should be careful as this means our users can not build hbase
> > >> with 3.2.x any more, which means we have dropped the support for
> > >> 3.2.x.
> > >>
> > >> Thanks.
> > >>
> > >> Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
> > >> >
> > >> > Hi HBase devs,
> > >> >
> > >> > Over the past few years HBase supports the default Hadoop version of
> > >> 3.2.x
> > >> > but it also works on Hadoop 3.3.x.
> > >> >
> > >> > I'm wondering if it makes sense to move the current default
> > >> hadoop.version
> > >> >  from
> 3.2.4
> > >> to
> > >> > 3.3.x.
> > >> >
> > >> > Why?
> > >> >
> > >> > 1. From a stability and security point of view, Hadoop 3.3 is the
> most
> > >> up
> > >> > to date release line. And all HBase tests pass using 3.3.x. There
> > hasn't
> > >> > been a new Hadoop 3.2.x release for over a year.
> > >> >
> > >> > 2. We have a feature (using HBase on Ozone) that depends on an API
> in
> > >> > Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the
> > default
> > >> > hadoop version to 3.3.6 will save a lot of hassle.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > Best,
> > >> > Weichiu
> > >>
> > >
> >
>

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Duo Zhang]

In general, in HBase, we will use the last patch release of the oldest
supported hadoop release line as our default hadoop dependency.

For example, since we claim that 3.x will support hadoop 3.2.x and
3.3.x, then we will declare the default hadoop version as 3.2.4.

I think we can discuss whether to move up to 3.3.6 as the default
version, if there are no compatibility issues when communicating with
3.2.x hadoop clusters.

But if we want to use the features which are only provided in 3.3.6,
then we should be careful as this means our users can not build hbase
with 3.2.x any more, which means we have dropped the support for
3.2.x.

Thanks.

Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
>
> Hi HBase devs,
>
> Over the past few years HBase supports the default Hadoop version of 3.2.x
> but it also works on Hadoop 3.3.x.
>
> I'm wondering if it makes sense to move the current default hadoop.version
>  from 3.2.4 to
> 3.3.x.
>
> Why?
>
> 1. From a stability and security point of view, Hadoop 3.3 is the most up
> to date release line. And all HBase tests pass using 3.3.x. There hasn't
> been a new Hadoop 3.2.x release for over a year.
>
> 2. We have a feature (using HBase on Ozone) that depends on an API in
> Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
> hadoop version to 3.3.6 will save a lot of hassle.
>
> Thoughts?
>
> Best,
> Weichiu

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Viraj Jasani]

How about using a new hadoop 3.3 profile for features that are explicitly
present in 3.3 (like FileSystem changes)? When the time comes, we switch to
3.3 profile by default and drop old hadoop 3 profile that supports 3.2.x
versions as of today?


On Fri, Jun 16, 2023 at 7:11 AM 张铎(Duo Zhang)  wrote:

> In general, in HBase, we will use the last patch release of the oldest
> supported hadoop release line as our default hadoop dependency.
>
> For example, since we claim that 3.x will support hadoop 3.2.x and
> 3.3.x, then we will declare the default hadoop version as 3.2.4.
>
> I think we can discuss whether to move up to 3.3.6 as the default
> version, if there are no compatibility issues when communicating with
> 3.2.x hadoop clusters.
>
> But if we want to use the features which are only provided in 3.3.6,
> then we should be careful as this means our users can not build hbase
> with 3.2.x any more, which means we have dropped the support for
> 3.2.x.
>
> Thanks.
>
> Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
> >
> > Hi HBase devs,
> >
> > Over the past few years HBase supports the default Hadoop version of
> 3.2.x
> > but it also works on Hadoop 3.3.x.
> >
> > I'm wondering if it makes sense to move the current default
> hadoop.version
> >  from 3.2.4 to
> > 3.3.x.
> >
> > Why?
> >
> > 1. From a stability and security point of view, Hadoop 3.3 is the most up
> > to date release line. And all HBase tests pass using 3.3.x. There hasn't
> > been a new Hadoop 3.2.x release for over a year.
> >
> > 2. We have a feature (using HBase on Ozone) that depends on an API in
> > Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
> > hadoop version to 3.3.6 will save a lot of hassle.
> >
> > Thoughts?
> >
> > Best,
> > Weichiu
>

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Tak Lon (Stephen) Wu]

or maybe we create a new feature branch hadoop-33-ozone that has these
interfaces and ozone related support, then we put all the feature changes
into this feature branch and merge later?

The problem I see is that it's hard and very confusing to maintain two
hadoop3 profiles, and I can see sooner or later hadoop 3.2.x could be EOL.

Thanks,
Stephen



On Fri, Jun 16, 2023 at 11:32 AM Viraj Jasani  wrote:

> How about using a new hadoop 3.3 profile for features that are explicitly
> present in 3.3 (like FileSystem changes)? When the time comes, we switch to
> 3.3 profile by default and drop old hadoop 3 profile that supports 3.2.x
> versions as of today?
>
>
> On Fri, Jun 16, 2023 at 7:11 AM 张铎(Duo Zhang) 
> wrote:
>
>> In general, in HBase, we will use the last patch release of the oldest
>> supported hadoop release line as our default hadoop dependency.
>>
>> For example, since we claim that 3.x will support hadoop 3.2.x and
>> 3.3.x, then we will declare the default hadoop version as 3.2.4.
>>
>> I think we can discuss whether to move up to 3.3.6 as the default
>> version, if there are no compatibility issues when communicating with
>> 3.2.x hadoop clusters.
>>
>> But if we want to use the features which are only provided in 3.3.6,
>> then we should be careful as this means our users can not build hbase
>> with 3.2.x any more, which means we have dropped the support for
>> 3.2.x.
>>
>> Thanks.
>>
>> Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
>> >
>> > Hi HBase devs,
>> >
>> > Over the past few years HBase supports the default Hadoop version of
>> 3.2.x
>> > but it also works on Hadoop 3.3.x.
>> >
>> > I'm wondering if it makes sense to move the current default
>> hadoop.version
>> >  from 3.2.4
>> to
>> > 3.3.x.
>> >
>> > Why?
>> >
>> > 1. From a stability and security point of view, Hadoop 3.3 is the most
>> up
>> > to date release line. And all HBase tests pass using 3.3.x. There hasn't
>> > been a new Hadoop 3.2.x release for over a year.
>> >
>> > 2. We have a feature (using HBase on Ozone) that depends on an API in
>> > Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
>> > hadoop version to 3.3.6 will save a lot of hassle.
>> >
>> > Thoughts?
>> >
>> > Best,
>> > Weichiu
>>
>

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Wei-Chiu Chuang]

I am +1 to use a feature branch.

On Tue, Jun 20, 2023 at 10:20 AM Tak Lon (Stephen) Wu 
wrote:

> or maybe we create a new feature branch hadoop-33-ozone that has these
> interfaces and ozone related support, then we put all the feature changes
> into this feature branch and merge later?
>
> The problem I see is that it's hard and very confusing to maintain two
> hadoop3 profiles, and I can see sooner or later hadoop 3.2.x could be EOL.
>
> Thanks,
> Stephen
>
>
>
> On Fri, Jun 16, 2023 at 11:32 AM Viraj Jasani  wrote:
>
> > How about using a new hadoop 3.3 profile for features that are explicitly
> > present in 3.3 (like FileSystem changes)? When the time comes, we switch
> to
> > 3.3 profile by default and drop old hadoop 3 profile that supports 3.2.x
> > versions as of today?
> >
> >
> > On Fri, Jun 16, 2023 at 7:11 AM 张铎(Duo Zhang) 
> > wrote:
> >
> >> In general, in HBase, we will use the last patch release of the oldest
> >> supported hadoop release line as our default hadoop dependency.
> >>
> >> For example, since we claim that 3.x will support hadoop 3.2.x and
> >> 3.3.x, then we will declare the default hadoop version as 3.2.4.
> >>
> >> I think we can discuss whether to move up to 3.3.6 as the default
> >> version, if there are no compatibility issues when communicating with
> >> 3.2.x hadoop clusters.
> >>
> >> But if we want to use the features which are only provided in 3.3.6,
> >> then we should be careful as this means our users can not build hbase
> >> with 3.2.x any more, which means we have dropped the support for
> >> 3.2.x.
> >>
> >> Thanks.
> >>
> >> Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
> >> >
> >> > Hi HBase devs,
> >> >
> >> > Over the past few years HBase supports the default Hadoop version of
> >> 3.2.x
> >> > but it also works on Hadoop 3.3.x.
> >> >
> >> > I'm wondering if it makes sense to move the current default
> >> hadoop.version
> >> >  from 3.2.4
> >> to
> >> > 3.3.x.
> >> >
> >> > Why?
> >> >
> >> > 1. From a stability and security point of view, Hadoop 3.3 is the most
> >> up
> >> > to date release line. And all HBase tests pass using 3.3.x. There
> hasn't
> >> > been a new Hadoop 3.2.x release for over a year.
> >> >
> >> > 2. We have a feature (using HBase on Ozone) that depends on an API in
> >> > Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the
> default
> >> > hadoop version to 3.3.6 will save a lot of hassle.
> >> >
> >> > Thoughts?
> >> >
> >> > Best,
> >> > Weichiu
> >>
> >
>