[
https://issues.apache.org/jira/browse/CRUNCH-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Durfey updated CRUNCH-659:
----------------------------------
Attachment: CRUNCH-659_v1.patch
added patch. updates hive to 2.1 and hadoop to 2.6.0. everything seems to build
just fine after these two bumps. the logging differences i think will require
both log4j and log4j2 properties files to be provided. the bridge dependency i
mentioned routes log4j statements into log4j2 log system. so, log4j2 is still
necessary.
> Upgrade to Hive 2.x
> -------------------
>
> Key: CRUNCH-659
> URL: https://issues.apache.org/jira/browse/CRUNCH-659
> Project: Crunch
> Issue Type: Task
> Reporter: Stephen Durfey
> Assignee: Stephen Durfey
> Attachments: CRUNCH-659_v1.patch
>
>
> I've been working on CRUNCH-340 to finish implementing the HCatSource and
> HCatTarget. It seems to be in a better place now that crunch only supports
> hadoop 2. I was looking to target as high of a version of hive/hcat as
> possible with minimal impact on the code base and dependencies.
> Hive 2.3.1 is out now. That relies upon hadoop 2.7.2, but HBase doesn't bump
> up to that version until HBase 2.x. Trying to run with hadoop 2.7.2 causes
> test failures in crunch-hbase. I'm not sure if that is going to cause runtime
> issues as the minicluster wouldn't even start due to a package name change in
> hadoop-hdfs (for the class StorageType) that's causing a no class found
> error.
> Hive 2.1.0 relies upon Hadoop 2.6.0, and that plays nice with HBase 1.x.
> However, the class StructField (inside TupleObjectInspector for ORC files)
> has a new abstract method added to it that would need to be implemented that
> was introduced after 2.x of Hive. Other than that everything runs fine.
> Currently Crunch is on 0.13.1 of Hive, so it's pretty far behind. I'm just
> kind of looking for feedback on the version bumps that should be targeted for
> my changes in CRUNCH-340. I wanted to take care of those first in a separate
> JIRA before introducing new code against a higher Hive version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
