[
https://issues.apache.org/jira/browse/HIVE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HIVE-6336:
---------------------------
Description:
There is an with hive 12 datanucleus incompatability which seems to have
invompatibility with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
The main question:
IF hive 0.12.0 and datanucleus are compatabile, then what is the version of
datanucleus I should be using with Hive 12 and Hadoop 2.2?
The error which Im getting (this blocks me from properly running hive queries
invoked from the "test" phase of a maven project)
To reproduce:
I have hadoop and hive running as a pseudo cluster local mode and derby as the
metastore
I have the following environment variables
{noformat}
HADOOP_HOME=/home/ubu/hadoop
JAVA_HOME=/usr/lib/jvm/java-7-oracle
{noformat}
I have the RegexSerDe declared in the hive-site.xml
{noformat}
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
<description>This JAR file available to all users for
alljobs</description>
</property>
{noformat}
If I run with
{noformat}
<datanucleus.version>3.0.2</datanucleus.version>
{noformat}
I get the following 1 exception only
'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'
HOWEVER, If I run with
{noformat}
<datanucleus.version>3.2.0-release</datanucleus.version>
{noformat}
I get the following 1 exception exception only
java.lang.ClassNotFoundException:
org/apache/hadoop/hive/contrib/serde2/RegexSerDe
EXPLANATION
The RegexSerDe class is picked up at run time but the datanucleus Map class is
not available, I have checked in the datanucleus-core 3.0.2 jar and it is
missing, Upgrading to the first datanucleus above 3.0.2 that includes the Map
class throws the ClassNotFoundException for RegexSerDe.
The earlier *3.0.2* datanucleus, code fails with the missing Map class but the
RegexSerDe class is found, then when I upgrade to the
3.2.0-release the Map class is found but for some unkown reason the code/Hive
no longer finds the RegexSerDe class
I started using the same datanucleus dependencies found in this hive pom
http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
below are the dependencies my latest attempts to get a functioning pom
{noformat}
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>0.96.0-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.96.0-hadoop2</version>
</dependency>
<!-- misc -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>${derby.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>${datanucleus.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
<version>${datanucleus-rdbms.version}</version>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>${datanucleus.jdo.version}</version>
<exclusions>
<exclusion>
<groupId>javax.jdo</groupId>
<artifactId>jdo2-api</artifactId>
</exclusion>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- hadoop -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!-- hive -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-common</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-serde</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-contrib</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-cli</artifactId>
<version>${hive.version}</version>
<exclusions>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
</exclusion>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.jolbox</groupId>
<artifactId>bonecp</artifactId>
<version>${bonecp.version}</version>
</dependency>
<!-- logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- SL4J Binding provided at runtime -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.12</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- Unit test artifacts -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.0.0</version>
<classifier>hadoop2</classifier>
</dependency>
{noformat}
was:
Issue is hive 12 datanucleus incompatability with
org.apache.hadoop.hive.contrib.serde2.RegexSerDe
I have hadoop and hive running as a pseudo cluster local mode and derby as the
metastore
I have the following environment variables
HADOOP_HOME=/home/ubu/hadoop
JAVA_HOME=/usr/lib/jvm/java-7-oracle
I have the RegexSerDe declared in the hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
<description>This JAR file available to all users for
alljobs</description>
</property>
If I run with <datanucleus.version>3.0.2</datanucleus.version> I get the
following 1 exception only
java.lang.ClassNotFoundException:
org.datanucleus.store.types.backed.Map
If I run with <datanucleus.version>3.2.0-release</datanucleus.version> I get
the following 1 exception exception only
java.lang.ClassNotFoundException:
org/apache/hadoop/hive/contrib/serde2/RegexSerDe
basically the RegexSerDe class is picked up at run time but the datanucleus Map
class is not available, I have checked in the datanucleus-core 3.0.2 jar and it
is missing
upgrading to the first datanucleus above 3.0.2 that includse the Map class
throws the ClassNotFoundException for RegexSerDe
that is with the earlier 3.0.2 datanucleus the code fails with the missing Map
class but the RegexSerDe class is found, then when I upgrade to the
3.2.0-release the Map class is found but for some unkown reason the code/Hive
no longer finds the RegexSerDe class
if hive 0.12.0 and datanucleus are compatabile what is the version of
datanucleus I should be using with Hive 12 and Hadoop 2.2, thanks for your time
and effort
I started using the same datanucleus dependencies found in this hive pom
http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
below are the dependencies my latest attempts to get a functioning pom
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>0.96.0-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.96.0-hadoop2</version>
</dependency>
<!-- misc -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>${derby.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>${datanucleus.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
<version>${datanucleus-rdbms.version}</version>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>${datanucleus.jdo.version}</version>
<exclusions>
<exclusion>
<groupId>javax.jdo</groupId>
<artifactId>jdo2-api</artifactId>
</exclusion>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- hadoop -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!-- hive -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-common</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-serde</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-contrib</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-cli</artifactId>
<version>${hive.version}</version>
<exclusions>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
</exclusion>
<exclusion>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.jolbox</groupId>
<artifactId>bonecp</artifactId>
<version>${bonecp.version}</version>
</dependency>
<!-- logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- SL4J Binding provided at runtime -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.12</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!-- Unit test artifacts -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.0.0</version>
<classifier>hadoop2</classifier>
</dependency>
> Issue is hive 12 datanucleus incompatability with
> org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-6336
> URL: https://issues.apache.org/jira/browse/HIVE-6336
> Project: Hive
> Issue Type: Wish
> Components: HiveServer2
> Affects Versions: 0.12.0
> Environment: Hadoop 2.2 local derby Meatastore embedded
> Reporter: Nigel Savage
> Priority: Blocker
> Labels: HADOOP
>
> There is an with hive 12 datanucleus incompatability which seems to have
> invompatibility with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> The main question:
> IF hive 0.12.0 and datanucleus are compatabile, then what is the version of
> datanucleus I should be using with Hive 12 and Hadoop 2.2?
> The error which Im getting (this blocks me from properly running hive queries
> invoked from the "test" phase of a maven project)
> To reproduce:
> I have hadoop and hive running as a pseudo cluster local mode and derby as
> the metastore
> I have the following environment variables
> {noformat}
> HADOOP_HOME=/home/ubu/hadoop
> JAVA_HOME=/usr/lib/jvm/java-7-oracle
> {noformat}
> I have the RegexSerDe declared in the hive-site.xml
> {noformat}
> <property>
> <name>hive.aux.jars.path</name>
> <value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
> <description>This JAR file available to all users for
> alljobs</description>
> </property>
> {noformat}
> If I run with
> {noformat}
> <datanucleus.version>3.0.2</datanucleus.version>
> {noformat}
> I get the following 1 exception only
> 'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'
> HOWEVER, If I run with
> {noformat}
> <datanucleus.version>3.2.0-release</datanucleus.version>
> {noformat}
> I get the following 1 exception exception only
> java.lang.ClassNotFoundException:
> org/apache/hadoop/hive/contrib/serde2/RegexSerDe
> EXPLANATION
> The RegexSerDe class is picked up at run time but the datanucleus Map class
> is not available, I have checked in the datanucleus-core 3.0.2 jar and it is
> missing, Upgrading to the first datanucleus above 3.0.2 that includes the
> Map class throws the ClassNotFoundException for RegexSerDe.
> The earlier *3.0.2* datanucleus, code fails with the missing Map class but
> the RegexSerDe class is found, then when I upgrade to the
> 3.2.0-release the Map class is found but for some unkown reason the code/Hive
> no longer finds the RegexSerDe class
> I started using the same datanucleus dependencies found in this hive pom
> http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
> below are the dependencies my latest attempts to get a functioning pom
> {noformat}
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-server</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-client</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <!-- misc -->
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-lang3</artifactId>
> <version>3.1</version>
> </dependency>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>${guava.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.derby</groupId>
> <artifactId>derby</artifactId>
> <version>${derby.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> <version>${datanucleus.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> <version>${datanucleus-rdbms.version}</version>
> </dependency>
> <dependency>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo-api</artifactId>
> <version>3.0.1</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> <version>${datanucleus.jdo.version}</version>
> <exclusions>
> <exclusion>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo2-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> </exclusion>
> <exclusion>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <!-- hadoop -->
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${hadoop.version}</version>
> </dependency>
> <!-- hive -->
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-common</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-serde</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-exec</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-jdbc</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-contrib</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-metastore</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-cli</artifactId>
> <version>${hive.version}</version>
> <exclusions>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.jolbox</groupId>
> <artifactId>bonecp</artifactId>
> <version>${bonecp.version}</version>
> </dependency>
> <!-- logging -->
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> <version>${slf4j.version}</version>
> </dependency>
> <!-- SL4J Binding provided at runtime -->
> <dependency>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> <version>1.2.12</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> <version>${slf4j.version}</version>
> <scope>provided</scope>
> </dependency>
> <!-- Unit test artifacts -->
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.11</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.hamcrest</groupId>
> <artifactId>hamcrest-all</artifactId>
> <version>1.3</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.mrunit</groupId>
> <artifactId>mrunit</artifactId>
> <version>1.0.0</version>
> <classifier>hadoop2</classifier>
> </dependency>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
