After manually copying hive 0.13.1 jars to local maven repo, I got the
following errors when building spark-hive_2.10 module :
[ERROR]
/homes/xx/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala:182:
type mismatch;
found : String
required: Array[String]
[ERROR] val proc: CommandProcessor =
CommandProcessorFactory.get(tokens(0), hiveconf)
[ERROR]
^
[ERROR]
/homes/xx/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:60:
value getAllPartitionsForPruner is not a member of org.apache.
hadoop.hive.ql.metadata.Hive
[ERROR] client.getAllPartitionsForPruner(table).toSeq
[ERROR] ^
[ERROR]
/homes/xx/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:267:
overloaded method constructor TableDesc with alternatives:
(x$1: Class[_ <: org.apache.hadoop.mapred.InputFormat[_, _]],x$2:
Class[_],x$3: java.util.Properties)org.apache.hadoop.hive.ql.plan.TableDesc
<and>
()org.apache.hadoop.hive.ql.plan.TableDesc
cannot be applied to (Class[org.apache.hadoop.hive.serde2.Deserializer],
Class[(some other)?0(in value tableDesc)(in value tableDesc)], Class[?0(in
value tableDesc)(in value tableDesc)], java.util.Properties)
[ERROR] val tableDesc = new TableDesc(
[ERROR] ^
[WARNING] Class org.antlr.runtime.tree.CommonTree not found - continuing
with a stub.
[WARNING] Class org.antlr.runtime.Token not found - continuing with a stub.
[WARNING] Class org.antlr.runtime.tree.Tree not found - continuing with a
stub.
[ERROR]
while compiling:
/homes/xx/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
during phase: typer
library version: version 2.10.4
compiler version: version 2.10.4
The above shows incompatible changes between 0.12 and 0.13.1
e.g. the first error corresponds to the following method
in CommandProcessorFactory :
public static CommandProcessor get(String[] cmd, HiveConf conf)
Cheers
On Mon, Jul 28, 2014 at 1:32 PM, Steve Nunez <snu...@hortonworks.com> wrote:
> So, do we have a short-term fix until Hive 0.14 comes out? Perhaps adding
> the hive-exec jar to the spark-project repo? It doesn¹t look like there¹s
> a release date schedule for 0.14.
>
>
>
> On 7/28/14, 10:50, "Cheng Lian" <lian.cs....@gmail.com> wrote:
>
> >Exactly, forgot to mention Hulu team also made changes to cope with those
> >incompatibility issues, but they said that¹s relatively easy once the
> >re-packaging work is done.
> >
> >
> >On Tue, Jul 29, 2014 at 1:20 AM, Patrick Wendell <pwend...@gmail.com>
> >wrote:
> >
> >> I've heard from Cloudera that there were hive internal changes between
> >> 0.12 and 0.13 that required code re-writing. Over time it might be
> >> possible for us to integrate with hive using API's that are more
> >> stable (this is the domain of Michael/Cheng/Yin more than me!). It
> >> would be interesting to see what the Hulu folks did.
> >>
> >> - Patrick
> >>
> >> On Mon, Jul 28, 2014 at 10:16 AM, Cheng Lian <lian.cs....@gmail.com>
> >> wrote:
> >> > AFAIK, according a recent talk, Hulu team in China has built Spark SQL
> >> > against Hive 0.13 (or 0.13.1?) successfully. Basically they also
> >> > re-packaged Hive 0.13 as what the Spark team did. The slides of the
> >>talk
> >> > hasn't been released yet though.
> >> >
> >> >
> >> > On Tue, Jul 29, 2014 at 1:01 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >> >
> >> >> Owen helped me find this:
> >> >> https://issues.apache.org/jira/browse/HIVE-7423
> >> >>
> >> >> I guess this means that for Hive 0.14, Spark should be able to
> >>directly
> >> >> pull in hive-exec-core.jar
> >> >>
> >> >> Cheers
> >> >>
> >> >>
> >> >> On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell <pwend...@gmail.com
> >
> >> >> wrote:
> >> >>
> >> >> > It would be great if the hive team can fix that issue. If not,
> >>we'll
> >> >> > have to continue forking our own version of Hive to change the way
> >>it
> >> >> > publishes artifacts.
> >> >> >
> >> >> > - Patrick
> >> >> >
> >> >> > On Mon, Jul 28, 2014 at 9:34 AM, Ted Yu <yuzhih...@gmail.com>
> >>wrote:
> >> >> > > Talked with Owen offline. He confirmed that as of 0.13,
> >>hive-exec is
> >> >> > still
> >> >> > > uber jar.
> >> >> > >
> >> >> > > Right now I am facing the following error building against Hive
> >> 0.13.1
> >> >> :
> >> >> > >
> >> >> > > [ERROR] Failed to execute goal on project spark-hive_2.10: Could
> >>not
> >> >> > > resolve dependencies for project
> >> >> > > org.apache.spark:spark-hive_2.10:jar:1.1.0-SNAPSHOT: The
> >>following
> >> >> > > artifacts could not be resolved:
> >> >> > > org.spark-project.hive:hive-metastore:jar:0.13.1,
> >> >> > > org.spark-project.hive:hive-exec:jar:0.13.1,
> >> >> > > org.spark-project.hive:hive-serde:jar:0.13.1: Failure to find
> >> >> > > org.spark-project.hive:hive-metastore:jar:0.13.1 in
> >> >> > > http://repo.maven.apache.org/maven2 was cached in the local
> >> >> repository,
> >> >> > > resolution will not be reattempted until the update interval of
> >> >> > maven-repo
> >> >> > > has elapsed or updates are forced -> [Help 1]
> >> >> > >
> >> >> > > Some hint would be appreciated.
> >> >> > >
> >> >> > > Cheers
> >> >> > >
> >> >> > >
> >> >> > > On Mon, Jul 28, 2014 at 9:15 AM, Sean Owen <so...@cloudera.com>
> >> wrote:
> >> >> > >
> >> >> > >> Yes, it is published. As of previous versions, at least,
> >>hive-exec
> >> >> > >> included all of its dependencies *in its artifact*, making it
> >> unusable
> >> >> > >> as-is because it contained copies of dependencies that clash
> >>with
> >> >> > >> versions present in other artifacts, and can't be managed with
> >> Maven
> >> >> > >> mechanisms.
> >> >> > >>
> >> >> > >> I am not sure why hive-exec was not published normally, with
> >>just
> >> its
> >> >> > >> own classes. That's why it was copied, into an artifact with
> >>just
> >> >> > >> hive-exec code.
> >> >> > >>
> >> >> > >> You could do the same thing for hive-exec 0.13.1.
> >> >> > >> Or maybe someone knows that it's published more 'normally' now.
> >> >> > >> I don't think hive-metastore is related to this question?
> >> >> > >>
> >> >> > >> I am no expert on the Hive artifacts, just remembering what the
> >> issue
> >> >> > >> was initially in case it helps you get to a similar solution.
> >> >> > >>
> >> >> > >> On Mon, Jul 28, 2014 at 4:47 PM, Ted Yu <yuzhih...@gmail.com>
> >> wrote:
> >> >> > >> > hive-exec (as of 0.13.1) is published here:
> >> >> > >> >
> >> >> > >>
> >> >> >
> >> >>
> >>
> >>
> http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-exec%7C
> >>0.13.1%7Cjar
> >> >> > >> >
> >> >> > >> > Should a JIRA be opened so that dependency on hive-metastore
> >>can
> >> be
> >> >> > >> > replaced by dependency on hive-exec ?
> >> >> > >> >
> >> >> > >> > Cheers
> >> >> > >> >
> >> >> > >> >
> >> >> > >> > On Mon, Jul 28, 2014 at 8:26 AM, Sean Owen
> >><so...@cloudera.com>
> >> >> > wrote:
> >> >> > >> >
> >> >> > >> >> The reason for org.spark-project.hive is that Spark relies on
> >> >> > >> >> hive-exec, but the Hive project does not publish this
> >>artifact
> >> by
> >> >> > >> >> itself, only with all its dependencies as an uber jar. Maybe
> >> that's
> >> >> > >> >> been improved. If so, you need to point at the new hive-exec
> >>and
> >> >> > >> >> perhaps sort out its dependencies manually in your build.
> >> >> > >> >>
> >> >> > >> >> On Mon, Jul 28, 2014 at 4:01 PM, Ted Yu <yuzhih...@gmail.com
> >
> >> >> wrote:
> >> >> > >> >> > I found 0.13.1 artifacts in maven:
> >> >> > >> >> >
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
> >>
> http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-metasto
> >>re%7C0.13.1%7Cjar
> >> >> > >> >> >
> >> >> > >> >> > However, Spark uses groupId of org.spark-project.hive, not
> >> >> > >> >> org.apache.hive
> >> >> > >> >> >
> >> >> > >> >> > Can someone tell me how it is supposed to work ?
> >> >> > >> >> >
> >> >> > >> >> > Cheers
> >> >> > >> >> >
> >> >> > >> >> >
> >> >> > >> >> > On Mon, Jul 28, 2014 at 7:44 AM, Steve Nunez <
> >> >> > snu...@hortonworks.com>
> >> >> > >> >> wrote:
> >> >> > >> >> >
> >> >> > >> >> >> I saw a note earlier, perhaps on the user list, that at
> >>least
> >> >> one
> >> >> > >> >> person is
> >> >> > >> >> >> using Hive 0.13. Anyone got a working build configuration
> >>for
> >> >> this
> >> >> > >> >> version
> >> >> > >> >> >> of Hive?
> >> >> > >> >> >>
> >> >> > >> >> >> Regards,
> >> >> > >> >> >> - Steve
> >> >> > >> >> >>
> >> >> > >> >> >>
> >> >> > >> >> >>
> >> >> > >> >> >> --
> >> >> > >> >> >> CONFIDENTIALITY NOTICE
> >> >> > >> >> >> NOTICE: This message is intended for the use of the
> >> individual
> >> >> or
> >> >> > >> >> entity to
> >> >> > >> >> >> which it is addressed and may contain information that is
> >> >> > >> confidential,
> >> >> > >> >> >> privileged and exempt from disclosure under applicable
> >>law.
> >> If
> >> >> the
> >> >> > >> >> reader
> >> >> > >> >> >> of this message is not the intended recipient, you are
> >>hereby
> >> >> > >> notified
> >> >> > >> >> that
> >> >> > >> >> >> any printing, copying, dissemination, distribution,
> >> disclosure
> >> >> or
> >> >> > >> >> >> forwarding of this communication is strictly prohibited.
> >>If
> >> you
> >> >> > have
> >> >> > >> >> >> received this communication in error, please contact the
> >> sender
> >> >> > >> >> immediately
> >> >> > >> >> >> and delete it from your system. Thank You.
> >> >> > >> >> >>
> >> >> > >> >>
> >> >> > >>
> >> >> >
> >> >>
> >>
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
