+1 -Ayush
> On 05-Apr-2020, at 12:43 AM, Wei-Chiu Chuang <weic...@apache.org> wrote: > > Hi Hadoop devs, > > I spent a good part of the past 7 months working with a dozen of colleagues > to update the guava version in Cloudera's software (that includes Hadoop, > HBase, Spark, Hive, Cloudera Manager ... more than 20+ projects) > > After 7 months, I finally came to a conclusion: Update to Hadoop 3.3 / > 3.2.1 / 3.1.3, even if you just go from Hadoop 3.0/ 3.1.0 is going to be > really hard because of guava. Because of Guava, the amount of work to > certify a minor release update is almost equivalent to a major release > update. > > That is because: > (1) Going from guava 11 to guava 27 is a big jump. There are several > incompatible API changes in many places. Too bad the Google developers are > not sympathetic about its users. > (2) guava is used in all Hadoop jars. Not just Hadoop servers but also > client jars and Hadoop common libs. > (3) The Hadoop library is used in practically all software at Cloudera. > > Here is my proposal: > (1) shade guava into hadoop-thirdparty, relocate the classpath to > org.hadoop.thirdparty.com.google.common.* > (2) make a hadoop-thirdparty 1.1.0 release. > (3) update existing references to guava to the relocated path. There are > more than 2k imports that need an update. > (4) release Hadoop 3.3.1 / 3.2.2 that contains this change. > > In this way, we will be able to update guava in Hadoop in the future > without disrupting Hadoop applications. > > Note: HBase already did this and this guava update project would have been > much more difficult if HBase didn't do so. > > Thoughts? Other options include > (1) force downstream applications to migrate to Hadoop client artifacts as > listed here > https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/DownstreamDev.html > but > that's nearly impossible. > (2) Migrate Guava to Java APIs. I suppose this is a big project and I can't > estimate how much work it's going to be. > > Weichiu --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
