Hi Sean, Thanks for clarifying. I re-read SPARK-2420 and now have a better understanding. >From a user perspective, what would you recommend to build Spark with Hive >0.12 / 0.13+ libraries moving forward and deploy to production cluster that >runs on a older version of Hadoop (e.g. 2.2 or 2.4) ? My concern is that there's going to be a lag for technology adoption and since Spark is moving fast, the libraries may always be newer. Protobuf is one good example, shading. From a biz point of view, if there is no benefit to upgrade the library, the chances that this will happen with a higher priority is low due to stability concern and re-running the entire test suite. Just by observation, there's still a lot of ppl running Hadoop 2.2 instead of 2.4 or 2.5 and the release and upgrade is depending on other big players such as Cloudera, Hortonwork, etc for their distro. Not to mention the process of upgrading. Is there any benefit to use Guava 14 in Spark? I believe there is usually some competitive reason why Spark choose Guava 14, however, I'm not sure if anyone raise that in the conversation so I don't know if that is necessary. Looking forward to seeing Hive on Spark to work soon. Please let me know if there's any help or feedback I can provide. Thanks Sean.
> From: so...@cloudera.com > Date: Mon, 21 Jul 2014 18:36:10 +0100 > Subject: Re: Hive From Spark > To: user@spark.apache.org > > I haven't seen anyone actively 'unwilling' -- I hope not. See > discussion at https://issues.apache.org/jira/browse/SPARK-2420 where I > sketch what a downgrade means. I think it just hasn't gotten a looking > over. > > Contrary to what I thought earlier, the conflict does in fact cause > problems in theory, and you show it causes a problem in practice. Not > to mention it causes issues for Hive-on-Spark now. > > On Mon, Jul 21, 2014 at 6:27 PM, Andrew Lee <alee...@hotmail.com> wrote: > > Hive and Hadoop are using an older version of guava libraries (11.0.1) where > > Spark Hive is using guava 14.0.1+. > > The community isn't willing to downgrade to 11.0.1 which is the current > > version for Hadoop 2.2 and Hive 0.12.
