Given the replies on this thread, I think it's appropriate that we do the following now:
1) Announce in the upcoming Druid 27 release that Hadoop 2 support is deprecated and will be removed in Druid 28 2) Remove Hadoop 2 support from the master branch, since Druid 27 has been branched off already, and the next release (28) is meant to not have it. Does anyone have some spare cycles to do (2)? Gian On 2023/06/28 06:42:08 Gian Merlino wrote: > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the very > next release (which I assume will be Druid 27) but the one after that, > likely late 2023 timeframe. > > In 2021, we had a discussion about moving away from Hadoop 2: > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For > various reasons, it didn't seem like the right time. However, I believe now > is the right time: > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There is now > a Hadoop 3 build profile, as well as convenience binaries on > https://druid.apache.org/downloads.html. > > 2) We have SQL-based ingest with MSQ tasks, which provides a built-in / > scalable / robust alternative to using Hadoop at all. > > 3) It has been an additional two years. Hadoop 2 is that much older, that > much more time has passed since it was superseded by Hadoop 3, and people > have had that much more time to migrate. > > 4) The original main reason for wanting to move away from Hadoop 2 is still > relevant. It keeps us on various old dependencies, including an ancient > version of Guava, which in turn has been keeping us on an ancient version > of Calcite. The Calcite community has graciously decided to support this > old version of Guava for at least one release, but plans to drop support by > Calcite 1.36, leaving us back in the same position. Managing this situation > is time-consuming for both Druid and Calcite maintainers. > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in 2021, > such as reworking Hadoop support to be purely extension based, and > reworking extensions to be more isolated from each other. However, these > are both substantially more complex than dropping support, and in the two > years since the original thread, these more complex solutions have not been > implemented. So, I think we need to move on with the simpler solution of > dropping support. > > Gian > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
