[
https://issues.apache.org/jira/browse/CRUNCH-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246684#comment-16246684
]
Stephen Durfey commented on CRUNCH-659:
---------------------------------------
Another (annoying) thing I forgot to list here is that Hive switched over to
log4j2 after 2.x. It will be a little awkward having log4j and log4j2 in the
same project. They look for different files, but the format between the two is
very different. There is a bridge jar to route log4j 1.x statements through the
log4j 2 system \[1\], but I haven't looked into how that works from a
configuration perspective.
\[1\] https://logging.apache.org/log4j/log4j-2.2/log4j-1.2-api/index.html
> Upgrade to Hive 2.x
> -------------------
>
> Key: CRUNCH-659
> URL: https://issues.apache.org/jira/browse/CRUNCH-659
> Project: Crunch
> Issue Type: Task
> Reporter: Stephen Durfey
>
> I've been working on CRUNCH-340 to finish implementing the HCatSource and
> HCatTarget. It seems to be in a better place now that crunch only supports
> hadoop 2. I was looking to target as high of a version of hive/hcat as
> possible with minimal impact on the code base and dependencies.
> Hive 2.3.1 is out now. That relies upon hadoop 2.7.2, but HBase doesn't bump
> up to that version until HBase 2.x. Trying to run with hadoop 2.7.2 causes
> test failures in crunch-hbase. I'm not sure if that is going to cause runtime
> issues as the minicluster wouldn't even start due to a package name change in
> hadoop-hdfs (for the class StorageType) that's causing a no class found
> error.
> Hive 2.1.0 relies upon Hadoop 2.6.0, and that plays nice with HBase 1.x.
> However, the class StructField (inside TupleObjectInspector for ORC files)
> has a new abstract method added to it that would need to be implemented that
> was introduced after 2.x of Hive. Other than that everything runs fine.
> Currently Crunch is on 0.13.1 of Hive, so it's pretty far behind. I'm just
> kind of looking for feedback on the version bumps that should be targeted for
> my changes in CRUNCH-340. I wanted to take care of those first in a separate
> JIRA before introducing new code against a higher Hive version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
