[
https://issues.apache.org/jira/browse/HIVE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888624#comment-13888624
]
jay vyas commented on HIVE-6336:
--------------------------------
Maybe we need to broaden the context of this JIRA: I think the issue is that
its not clear if one can run an embedded hive query , in local mode, from
inside of maven on Hadoop 2x.
The RegexSerDe (and even the datanucleus) issues are symptoms of, probably, an
incomplete attempt to setup an embedded maven query....
Any recipes for embedded maven based hive testing on 2x? that will be
sufficient to solve this and other similar issues.
> Issue is hive 12 datanucleus incompatability with
> org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-6336
> URL: https://issues.apache.org/jira/browse/HIVE-6336
> Project: Hive
> Issue Type: Wish
> Components: HiveServer2
> Affects Versions: 0.12.0
> Environment: Hadoop 2.2 local derby Meatastore embedded
> Reporter: Nigel Savage
> Priority: Blocker
> Labels: HADOOP
>
> There is an with hive 12 datanucleus incompatability which seems to have
> invompatibility with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> The main question:
> *IF hive 0.12.0 and datanucleus are compatabile, then what is the version of
> datanucleus I should be using with Hive 12 and Hadoop 2.2?*
> The error which Im getting (this blocks me from properly running hive queries
> invoked from the "test" phase of a maven project)
> *To reproduce*
> I have hadoop and hive running as a pseudo cluster local mode and derby as
> the metastore
> I have the following environment variables
> {noformat}
> HADOOP_HOME=/home/ubu/hadoop
> JAVA_HOME=/usr/lib/jvm/java-7-oracle
> {noformat}
> I have the RegexSerDe declared in the hive-site.xml
> {noformat}
> <property>
> <name>hive.aux.jars.path</name>
> <value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
> <description>This JAR file available to all users for
> alljobs</description>
> </property>
> {noformat}
> If I run with
> {noformat}
> <datanucleus.version>3.0.2</datanucleus.version>
> {noformat}
> I get the following 1 exception only
> 'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'
> HOWEVER, If I run with
> {noformat}
> <datanucleus.version>3.2.0-release</datanucleus.version>
> {noformat}
> I get the following 1 exception exception only
> java.lang.ClassNotFoundException:
> org/apache/hadoop/hive/contrib/serde2/RegexSerDe
> EXPLANATION
> The RegexSerDe class is picked up at run time but the datanucleus Map class
> is not available, I have checked in the datanucleus-core 3.0.2 jar and it is
> missing, Upgrading to the first datanucleus above 3.0.2 that includes the
> Map class throws the ClassNotFoundException for RegexSerDe.
> The earlier *3.0.2* datanucleus, code fails with the missing Map class but
> the RegexSerDe class is found, then when I upgrade to the
> 3.2.0-release the Map class is found but for some unkown reason the code/Hive
> no longer finds the RegexSerDe class
> I started using the same datanucleus dependencies found in this hive pom
> http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
> below are the dependencies my latest attempts to get a functioning pom
> {noformat}
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-server</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-client</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <!-- misc -->
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-lang3</artifactId>
> <version>3.1</version>
> </dependency>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>${guava.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.derby</groupId>
> <artifactId>derby</artifactId>
> <version>${derby.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> <version>${datanucleus.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> <version>${datanucleus-rdbms.version}</version>
> </dependency>
> <dependency>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo-api</artifactId>
> <version>3.0.1</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> <version>${datanucleus.jdo.version}</version>
> <exclusions>
> <exclusion>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo2-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> </exclusion>
> <exclusion>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <!-- hadoop -->
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${hadoop.version}</version>
> </dependency>
> <!-- hive -->
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-common</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-serde</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-exec</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-jdbc</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-contrib</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-metastore</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-cli</artifactId>
> <version>${hive.version}</version>
> <exclusions>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.jolbox</groupId>
> <artifactId>bonecp</artifactId>
> <version>${bonecp.version}</version>
> </dependency>
> <!-- logging -->
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> <version>${slf4j.version}</version>
> </dependency>
> <!-- SL4J Binding provided at runtime -->
> <dependency>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> <version>1.2.12</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> <version>${slf4j.version}</version>
> <scope>provided</scope>
> </dependency>
> <!-- Unit test artifacts -->
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.11</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.hamcrest</groupId>
> <artifactId>hamcrest-all</artifactId>
> <version>1.3</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.mrunit</groupId>
> <artifactId>mrunit</artifactId>
> <version>1.0.0</version>
> <classifier>hadoop2</classifier>
> </dependency>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
