[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583356#comment-15583356 ]
Xuefu Zhang commented on HIVE-14029: ------------------------------------ [~spena], Keeping b/c is a good thing in general. Before we take the effort (which seems a lot) to do it, I think we should clearly understand and define what b/c is in this case. Spark is rapidly releasing w/o much b/c in mind. So far, Hive on Spark has once depended on Spark 1.2, 1.3, 1.4, 1.5, and 1.6. I'm not sure what versions of Spark Hive has been released with, but one thing is clear, Spark isn't b/c between these releases. Before Spark community has a good sense of keeping b/c in their APIs, it's going to be very hard and burdensome for Hive to maintain support for different Spark releases, not to mention the library dependency issues we have had. I'm okay to start thinking of a shim layer to support multiple versions of Spark, but it sounds daunting to me due to the dynamics of Spark project. > Update Spark version to 2.0.0 > ----------------------------- > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug > Reporter: Ferdinand Xu > Assignee: Ferdinand Xu > Labels: Incompatible, TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, > HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)