Is there any document to explain how to build the hive jars for spark?

2014-12-11 Thread Yi Tian

Hi, all

We found some bugs in hive-0.12, but we could not wait for hive 
community fixing them.


We want to fix these bugs in our lab and build a new release which could 
be recognized by spark.


As we know, spark depends on a special release of hive, like:

|
  org.spark-project.hive
  hive-metastore
  ${hive.version}

|

The different between |org.spark-project.hive| and |org.apache.hive| was 
described by Patrick:


|There are two differences:

1. We publish hive with a shaded protobuf dependency to avoid
conflicts with some Hadoop versions.
2. We publish a proper hive-exec jar that only includes hive packages.
The upstream version of hive-exec bundles a bunch of other random
dependencies in it which makes it really hard for third-party projects
to use it.
|

Is there any document to guide us how to build the hive jars for spark?

Any help would be greatly appreciated.

​


Re: Is there any document to explain how to build the hive jars for spark?

2014-12-14 Thread Michael Armbrust
The modified version of hive can be found here:
https://github.com/pwendell/hive

On Thu, Dec 11, 2014 at 5:47 PM, Yi Tian  wrote:
>
> Hi, all
>
> We found some bugs in hive-0.12, but we could not wait for hive community
> fixing them.
>
> We want to fix these bugs in our lab and build a new release which could
> be recognized by spark.
>
> As we know, spark depends on a special release of hive, like:
>
> |
>   org.spark-project.hive
>   hive-metastore
>   ${hive.version}
> 
> |
>
> The different between |org.spark-project.hive| and |org.apache.hive| was
> described by Patrick:
>
> |There are two differences:
>
> 1. We publish hive with a shaded protobuf dependency to avoid
> conflicts with some Hadoop versions.
> 2. We publish a proper hive-exec jar that only includes hive packages.
> The upstream version of hive-exec bundles a bunch of other random
> dependencies in it which makes it really hard for third-party projects
> to use it.
> |
>
> Is there any document to guide us how to build the hive jars for spark?
>
> Any help would be greatly appreciated.
>
> ​
>