In favour of dropping hadoop 2 support . Another point is the lack of security and vulnerability fixes in hadoop2.
On Wed, Jun 28, 2023 at 12:17 PM Clint Wylie <cwy...@apache.org> wrote: > obvious +1 from me > > On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <g...@apache.org> wrote: > > > > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the > very > > next release (which I assume will be Druid 27) but the one after that, > > likely late 2023 timeframe. > > > > In 2021, we had a discussion about moving away from Hadoop 2: > > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For > > various reasons, it didn't seem like the right time. However, I believe > now > > is the right time: > > > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There is > now > > a Hadoop 3 build profile, as well as convenience binaries on > > https://druid.apache.org/downloads.html. > > > > 2) We have SQL-based ingest with MSQ tasks, which provides a built-in / > > scalable / robust alternative to using Hadoop at all. > > > > 3) It has been an additional two years. Hadoop 2 is that much older, that > > much more time has passed since it was superseded by Hadoop 3, and people > > have had that much more time to migrate. > > > > 4) The original main reason for wanting to move away from Hadoop 2 is > still > > relevant. It keeps us on various old dependencies, including an ancient > > version of Guava, which in turn has been keeping us on an ancient version > > of Calcite. The Calcite community has graciously decided to support this > > old version of Guava for at least one release, but plans to drop support > by > > Calcite 1.36, leaving us back in the same position. Managing this > situation > > is time-consuming for both Druid and Calcite maintainers. > > > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in > 2021, > > such as reworking Hadoop support to be purely extension based, and > > reworking extensions to be more isolated from each other. However, these > > are both substantially more complex than dropping support, and in the two > > years since the original thread, these more complex solutions have not > been > > implemented. So, I think we need to move on with the simpler solution of > > dropping support. > > > > Gian > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > > -- Thanks Karan
