obvious +1 from me On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <g...@apache.org> wrote: > > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the very > next release (which I assume will be Druid 27) but the one after that, > likely late 2023 timeframe. > > In 2021, we had a discussion about moving away from Hadoop 2: > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For > various reasons, it didn't seem like the right time. However, I believe now > is the right time: > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There is now > a Hadoop 3 build profile, as well as convenience binaries on > https://druid.apache.org/downloads.html. > > 2) We have SQL-based ingest with MSQ tasks, which provides a built-in / > scalable / robust alternative to using Hadoop at all. > > 3) It has been an additional two years. Hadoop 2 is that much older, that > much more time has passed since it was superseded by Hadoop 3, and people > have had that much more time to migrate. > > 4) The original main reason for wanting to move away from Hadoop 2 is still > relevant. It keeps us on various old dependencies, including an ancient > version of Guava, which in turn has been keeping us on an ancient version > of Calcite. The Calcite community has graciously decided to support this > old version of Guava for at least one release, but plans to drop support by > Calcite 1.36, leaving us back in the same position. Managing this situation > is time-consuming for both Druid and Calcite maintainers. > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in 2021, > such as reworking Hadoop support to be purely extension based, and > reworking extensions to be more isolated from each other. However, these > are both substantially more complex than dropping support, and in the two > years since the original thread, these more complex solutions have not been > implemented. So, I think we need to move on with the simpler solution of > dropping support. > > Gian
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
