Re: ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory

[Poorvi Ahirwal]

Is there any other  method to use instead of
HcatInputFormat.setInput(job,"db","table")..? Because this is dependent on
many jars which are not there in hadoop-1.2.1 and hive-0.13.1. Or any other
way of reading data from hive table with mapreduce program.?

Thanks

On Wed, Sep 24, 2014 at 12:14 AM, hadoop hive  wrote:

> I am talking bout property file
> On Sep 23, 2014 5:18 PM, "Poorvi Ahirwal" 
> wrote:
>
>> you are saying about mapred program or some property file?
>>
>> On Tue, Sep 23, 2014 at 4:14 AM, hadoop hive 
>> wrote:
>>
>>> Comment the jdo properties from mapred and hdfs
>>> On Sep 23, 2014 4:42 PM, "Poorvi Ahirwal" 
>>> wrote:
>>>
 Hi,
 I am executing a mapreduce program with hcatalog and hive database.
 Even if the jars are included its showing this error:

 Exception in thread "main" java.io.IOException:
 com.google.common.util.concurrent.UncheckedExecutionException:
 javax.jdo.JDOFatalUserException: Class
 org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
 NestedThrowables:
 java.lang.ClassNotFoundException:
 org.datanucleus.api.jdo.JDOPersistenceManagerFactory
 at
 org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
 at
 org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
 ..
 Please help

 thanks

>>>
>>

Re: Takes long time at map 100% Reduce 100%

[unmesha sreeveni]

Do you have any type of calculation in Driver?

On Wed, Sep 24, 2014 at 10:19 AM, Rohith Sharma K S <
rohithsharm...@huawei.com> wrote:

> Is it your custom job or any mapreduce-example jobs?
> How many mappers and reducers are running?
>
> Check application master container logs why job is not finished.
>
> Thanks & Regards
> Rohith Sharma K S
>
>
> -Original Message-
> From: Karthiksrivasthava [mailto:karthiksrivasth...@gmail.com]
> Sent: 24 September 2014 09:44
> To: user@hive.apache.org
> Subject: Takes long time at map 100% Reduce 100%
>
> Hi ,
>
> My MapReduce program takes almost 10 minutes to finish the job  after it
> reaches map 100% reduce 100% ..
>
> Thanks
> Karthik
>



-- 
*Thanks & Regards *


*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/

RE: Takes long time at map 100% Reduce 100%

[Rohith Sharma K S]

Is it your custom job or any mapreduce-example jobs?
How many mappers and reducers are running?

Check application master container logs why job is not finished.

Thanks & Regards
Rohith Sharma K S


-Original Message-
From: Karthiksrivasthava [mailto:karthiksrivasth...@gmail.com] 
Sent: 24 September 2014 09:44
To: user@hive.apache.org
Subject: Takes long time at map 100% Reduce 100%

Hi ,

My MapReduce program takes almost 10 minutes to finish the job  after it 
reaches map 100% reduce 100% ..  

Thanks
Karthik

Takes long time at map 100% Reduce 100%

[Karthiksrivasthava]

Hi ,

My MapReduce program takes almost 10 minutes to finish the job  after it 
reaches 
map 100% reduce 100% ..  

Thanks 
Karthik

Hiveserver Error : Hiveserver.java.readResults Error: File /tmp/***/pipeout not found

[Leena Gupta]

Hello,

I am trying to run a query on Hive from a reporting tool but it fails with
a File not found exception. I am using hiverserver1.
>From the /tmp/user/hive.log file, I can see that the Map reduce jobs
completed fine, but in the end of the file I see the following error. Any
pointers on what might be causing this issue ? Here is a snippet from the
log file:


2014-09-23 15:32:39,712 ERROR service.HiveServer
(HiveServer.java:readResults(392)) - File
/tmp/user/6e878275-de0c-46fd-a718-53e9bbf75dd32652576951214936161.pipeout
not found.
java.io.FileNotFoundException:
/tmp/user/6e878275-de0c-46fd-a718-53e9bbf75dd32652576951214936161.pipeout
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:120)
at java.io.FileReader.(FileReader.java:55)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.readResults(HiveServer.java:390)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.fetchN(HiveServer.java:442)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$fetchN.getResult(ThriftHive.java:692)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$fetchN.getResult(ThriftHive.java:676)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)

Thanks!

Issues after upgrading to hive0.13 from hive0.12

[Ankita Bakshi]

1. The following query works in Hive0.12 but not is hive0.13. Note, in
hive0.12 the following query will not compile if we remove one of the Day
column from group by.

SELECT DAY, daypart, 'DAY' AS DIMENSION, `day` AS VALUE,  SUM(`total`) AS T
  FROM H_ANKITEST GROUP BY  DAY, `day` ,  daypart

 FAILED: SemanticException [Error 10021]: Line 2:41 Repeated key in GROUP
BY 'day'

2. UNION ALL fails with the IndexOutOfBoundsException exception. Note. this
is reproducible when we have investigationId column which is an INT type in
UNION ALL, but it succeeds for other columns which are of INT type.

*Query*:

SELECT daypart, DAY, 'Z_AIRPLANEMODE' AS CUBENAME, DIMENSION, VALUE

FROM (

SELECT DAY, daypart, 'WEEK' AS DIMENSION, `week` AS VALUE,
SUM(`total`) AS T

   FROM H_ANKITEST GROUP BY  DAY, daypart, `week`

  ORDER BY T DESC LIMIT 1000

   UNION ALL

   SELECT DAY, daypart, 'INVESTIGATIONID' AS DIMENSION, `investigationId`
AS VALUE,  SUM(`total`) AS T

 FROM H_ANKITEST GROUP BY  DAY, daypart, `investigationId`

 ORDER BY T DESC LIMIT 1000

 )

 OUTER


*Plan for the failed task in Hive0.13:*

 2014-09-23 21:51:15,723 INFO org.apache.hadoop.hive.ql.exec.MapOperator:*
dump TS struct<_col1:string,_col0:string,_col2:string,_col3:string> // col4
is missing*

Id =5

  

Id =0

  

Id =1

  

Id =2

  

Id =3

  

  <\Children>

  Id = 2 null<\Parent>

<\FS>

  <\Children>

  Id = 1 null<\Parent>

<\SEL>

  <\Children>

  Id = 4

Id =4

  null

  <\Children>

  Id = 5 null<\Parent>

<\TS>Id = 0 null<\Parent>

<\UNION>

  <\Children>

  Id = 5 null<\Parent>

<\TS>

  <\Children>

  <\Parent>

<\MAP>


*Plan for the successful task in Hive0.12*

2014-09-23 22:05:22,022 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
dump TS struct<_col0:string,_col1:string,_col2:string,_col3:string,
*_col4:bigint*>

Id =5

  

Id =4

  

Id =2

  

Id =1

  

Id =0

  Id = 1 null<\Parent>

<\FS>

  <\Children>

  Id = 2 null<\Parent>

<\SEL>

  <\Children>

  Id = 3

Id =3

  null

  <\Children>

  Id = 5 null<\Parent>

<\TS>Id = 4 null<\Parent>

<\UNION>

  <\Children>

  Id = 5 null<\Parent>

<\TS>

  <\Children>

<\MAP>



*Exception*:

java.lang.RuntimeException: Error in configuring object

at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)

at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)

... 9 more

Caused by: java.lang.RuntimeException: Error in configuring object

at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)

at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)

... 14 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)

... 17 more

Caused by: java.lang.RuntimeException: Map operator initialization failed

at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)

... 22 more

Caused by: java.lang.IndexOutOfBoundsException: Index: 4, Size: 4

at java.util.ArrayList.rangeCheck(ArrayList.java:635)

at java.util.ArrayList.get(ArrayList.java:411

Re: bug in hive

[Shushant Arora]

Hi Alan
1.When writer takes exclusive lock , hive won't allow to write anyone(even
the session which holds lock) to write in table.
Do I  need to pass lock handle to read query or I am missing here something.
2.Or you mean to insert using hadoop filesystem not using hive ?

On Tue, Sep 23, 2014 at 8:13 PM, Alan Gates  wrote:

> Shushant,
>
> Creating a patched jar that would include the lock functionality you want
> is unlikely to work.  Wouldn't the following workflow work for you:
>
> 1. Writer locks the table explicitly via LOCK TABLE
> 2. Writer inserts
> 3. Writer unlocks the table explicitly via UNLOCK TABLE
>
> If you're using ZK for your locking I think the client dying (as opposed
> to ending the session) should cause the lock to expire.  If not, you may
> have to assure the unlock happens in your application.  Hope that helps.
>
> Alan.
>
>   Shushant Arora 
>  September 20, 2014 at 8:00
> Hi Alan
>
> I have 0.10 version of hive deployed in my org's cluster, I cannot update
> that because of org's policy.
> How can I achieve exclusive lock functionality while inserting in dynamic
> partition on hive 0.10 ?
> Does calling hive scripts via some sort of java api with patched jar
> included will help ?
> Moreover hive does not release locks in 0.10 when hive session is killed .
> User has to explicitly unlock a table.
> Can i specify any sort of max expiry time while taking a lock.
>
> Thanks
> Shushant
>
>
>   Alan Gates 
>  September 20, 2014 at 7:41
>  Up until Hive 0.13 locks in Hive were really advisory only, since as you
> note any user can remove any other user's lock.  In Hive 0.13 a new type of
> locking was introduced, see
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-LockManager
> This new locking is automatic and ignores both LOCK and UNLOCK commands.
> Note that it is off by default, you have to configure Hive to use the new
> DbTxnManager to get turn on this locking.  In 0.13 it still has the bug you
> describe as far as acquiring the wrong lock for dynamic partitioning, but I
> believe I've fixed that in 0.14.
>
> Alan.
>
>
>   Shushant Arora 
>  September 20, 2014 at 5:39
>
> Hive version 0.9 and later has a bug
>
>
>
> While inserting in a hive table Hive takes an exclusive lock. But if table
> is partitioned , and insert is in dynamic partition , it will take shared
> lock on table but if all partitions are static then hive takes exclusive
> lock on partitions in which data is being inserted
>
> and shared lock on table.
>
> https://issues.apache.org/jira/browse/HIVE-3509
>
>
> 1.What if I want to take exclusive lock on table while inserting in
> dynamic partition ?
>
>
> I tried to take explicit lock using :
>
> LOCK TABLE tablename EXCLUSIVE;
>
>
> But it made table to be disabled.
>
> I cannot even read from table anymore even is same session until I do
>
> unlock table tablename in another session;
>
>
> 2. moreover whats lock level in hive , I mean any user can remove any
> other users lock. that too seems buggy.
>
>
> Thanks
>
> Shushant
>
>
>
>
> --
> Sent with Postbox 
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory

[hadoop hive]

I am talking bout property file
On Sep 23, 2014 5:18 PM, "Poorvi Ahirwal"  wrote:

> you are saying about mapred program or some property file?
>
> On Tue, Sep 23, 2014 at 4:14 AM, hadoop hive  wrote:
>
>> Comment the jdo properties from mapred and hdfs
>> On Sep 23, 2014 4:42 PM, "Poorvi Ahirwal" 
>> wrote:
>>
>>> Hi,
>>> I am executing a mapreduce program with hcatalog and hive database. Even
>>> if the jars are included its showing this error:
>>>
>>> Exception in thread "main" java.io.IOException:
>>> com.google.common.util.concurrent.UncheckedExecutionException:
>>> javax.jdo.JDOFatalUserException: Class
>>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
>>> NestedThrowables:
>>> java.lang.ClassNotFoundException:
>>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>>> at
>>> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
>>> at
>>> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
>>> ..
>>> Please help
>>>
>>> thanks
>>>
>>
>

Re: Oracle backend for Hive Metastore

[hadoop hive]

You can import the metastore db in oracle directly.

Inside oracle db
@/path/to/oracle-hive-0.12.0.sql
 On Sep 23, 2014 9:08 PM, "Rahul Channe"  wrote:

> Hi Alan,
>
> I am using version 0.12.0, I did not create tables explicitly.
>
> Temporarily I modified the hive-metastore jar to update long data type to
> clob and it worked
>
> Not sure if it's hive bug
>
> On Tuesday, September 23, 2014, Alan Gates  wrote:
>
>> What version of Hive are you using?
>>
>> Did you explicitly create the tables in the metastore via the Oracle
>> script or depend on DataNucleus to do it for you?
>>
>> Alan.
>>
>>   Rahul Channe
>>  September 22, 2014 at 10:47
>> Hi All,
>>
>> I am using oracle as hive metastore. I could see the schema created in
>> oracle after logging into hive and executing create database command
>> successfully.
>>
>> When i am trying to create table in hive its throwing error, please help
>>
>> hive>  create table test ( name string,age int) row format delimited
>> fields terminated by ",";
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(
>> message:javax.jdo.JDODataStoreException: An exception was thrown while
>> adding/validating class(es) : ORA-01754: a table may contain only one
>> column of type LONG
>>
>> java.sql.SQLException: ORA-01754: a table may contain only one column of
>> type LONG
>>
>> at
>> oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
>> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
>> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
>> at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
>> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
>> at
>> oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:946)
>> at
>> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1160)
>> at
>> oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1679)
>>
>>
>> --
>> Sent with Postbox 
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>

Re: Unxpected exception: Unxpected exception: org.apache.hadoop.io.LongWritable cannot be cast to java.lang.Long

[Jason Dere]

What does your GenericUDF look like?
What version of Hive? Does the query work without the UDF?


On Sep 22, 2014, at 3:28 PM, Dan Fan  wrote:

> Dear hive users: 
> 
> Quick question about hive longwritable convert to long. 
> I have a generic udf called protected_column, which works well as following: 
> 
> Select protect_column(auction_id_64, ‘auction_id_64’, vp_bitmap) from table ; 
> And works well when I run 
> Select * from ( select protect_column(auction_id_64, ‘auction_id_64', 
> vp_bitmap) from tableA union all select protect_column(auction_id_64, 
> ‘auction_id_64', vp_bitmap) from tableB) tmp ;  
> 
> However, the generic udf would fail if I use it in the join.
> 
> 
> 
> SELECT * 
> FROM   (SELECT Protect_column(auction_id_64, 'auction_id_64', vp_bitmap) 
> FROM   view_agg_dw_impressions_tmp t 
>LEFT OUTER JOIN table_A pub 
> ON ( pub.id = t.publisher_id ) 
>LEFT OUTER JOIN table_B inv 
> ON ( inv.id = t.inventory_source_id ) 
> WHERE  t.buyer_member_id = 1541 
>AND dh = '2014-09-19 12') feed; 
> 
> 
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unxpected 
> exception: Unxpected exception: org.apache.hadoop.io.LongWritable cannot be 
> cast to java.lang.Long
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:234)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
> at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:136)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
> 
> It fails if I run the following too. 
> 
> 
> SELECT Protect_column(auction_id_64, 'auction_id_64', vp_bitmap) 
> FROM   (SELECT auction_id_64, vp_bitmap
> FROM   view_agg_dw_impressions_tmp t 
>LEFT OUTER JOIN table_A pub 
> ON ( pub.id = t.publisher_id ) 
>LEFT OUTER JOIN table_B inv 
> ON ( inv.id = t.inventory_source_id ) 
> WHERE  t.buyer_member_id = 1541 
>AND dh = '2014-09-19 12') feed; 
> 
> Any one can help explaining why hive wanna convert longwritable to long and 
> why the udf works fine on select query and union all query but not join 
> queries ? 
> 
> Thanks 
> 
> Dan


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Querying A table which JDBC

[Nitin Pawar]

can you share hiveserver2 heap size and your table size ?

On Tue, Sep 23, 2014 at 11:31 PM, Shiang Luong 
wrote:

> Ritesh thanks for your response.
>
> Where do I download and place the jars?
> Do you mean on the hive server itself?  I believe the files are already
> there since I can query the same table via command line.
> It feels like the serde is not being sent along with the query? or I need
> to get the jar sent out to the distributed cache?
> I even tried running:
>
> myStatment.execute("add JAR /usr/lib/hive/extra_libs/test.jar");
>
> That didn't work.  I'm not sure just shooting out thoughts.
>
> Thanks,
>
> Shiang
>
> On Mon, Sep 22, 2014 at 10:52 PM, Ritesh Kumar Singh <
> riteshoneinamill...@gmail.com> wrote:
>
>> try downloading the jar files and put it in the libraries folder
>>
>> On Tue, Sep 23, 2014 at 10:58 AM, Shiang Luong 
>> wrote:
>>
>>> Hi All,
>>>
>>> I'm new to hive.  I'm having some problems querying an hive table with
>>> JDBC.  It fails when it is trying to run an map reduce job.  It can't seem
>>> to find the serde jar file.  When I query it through the command line it
>>> works fine.  Anyone have any hints on how I can get it working with JDBC?
>>>
>>> Thanks in advance.
>>>
>>> Shiang
>>>
>>
>>
>
>
> --
> Shiang Luong
> Software Engineer in Test | OpenX
> 888 East Walnut Street, 2nd Floor | Pasadena, CA 91101
> o: +1 (626) 466-1141 x | m: +1 (626) 512-2165 | shiang.lu...@openx.com
> OpenX ranked No. 7 in Forbes’ America’s Most Promising Companies
>



-- 
Nitin Pawar

Re: Querying A table which JDBC

[Shiang Luong]

Ritesh thanks for your response.

Where do I download and place the jars?
Do you mean on the hive server itself?  I believe the files are already
there since I can query the same table via command line.
It feels like the serde is not being sent along with the query? or I need
to get the jar sent out to the distributed cache?
I even tried running:

myStatment.execute("add JAR /usr/lib/hive/extra_libs/test.jar");

That didn't work.  I'm not sure just shooting out thoughts.

Thanks,

Shiang

On Mon, Sep 22, 2014 at 10:52 PM, Ritesh Kumar Singh <
riteshoneinamill...@gmail.com> wrote:

> try downloading the jar files and put it in the libraries folder
>
> On Tue, Sep 23, 2014 at 10:58 AM, Shiang Luong 
> wrote:
>
>> Hi All,
>>
>> I'm new to hive.  I'm having some problems querying an hive table with
>> JDBC.  It fails when it is trying to run an map reduce job.  It can't seem
>> to find the serde jar file.  When I query it through the command line it
>> works fine.  Anyone have any hints on how I can get it working with JDBC?
>>
>> Thanks in advance.
>>
>> Shiang
>>
>
>


-- 
Shiang Luong
Software Engineer in Test | OpenX
888 East Walnut Street, 2nd Floor | Pasadena, CA 91101
o: +1 (626) 466-1141 x | m: +1 (626) 512-2165 | shiang.lu...@openx.com
OpenX ranked No. 7 in Forbes’ America’s Most Promising Companies

Hiveserver crash with RStudio (using RJDBC)

[Nathalie Blais]

Hello,

We are currently experiencing a severe reproducible hiveserver2 crash when 
using the RJDBC connector in RStudio (please refer to the description below for 
the detailed test case).  We have a hard time pinpointing the source of the 
problem and we are wondering whether this is a known issue or we have a glitch 
in our configuration; we would sincerely appreciate your input on this case.

Case
Severe Hiveserver2 crash when returning "a certain" volume of data (really not 
that big) to RStudio through RJDBC

Config Versions
Hadoop Distribution: Cloudera - cdh5.0.1p0.47
Hiverserver2: 0.12
RStudio: 0.98.1056
RJDBC: 0.2-4

How to Reproduce

1.   In a SQL client application (Aqua Data Studio was used for the purpose 
of this example), create Hive test table

a.   create table test_table_connection_crash(col1 string);

2.   Load data into table (data file attached)

a.   LOAD DATA INPATH '/user/test/testFile.txt' INTO TABLE 
test_table_connection_crash;

3.   Verify row count

a.   select count(*) nbRows from test_table_connection_crash;

b.  720 000 rows

4.   Display all rows

a.   select * from test_table_connection_crash order by col1 desc

b.  All the rows are returned by the Map/Reduce to the client and displayed 
properly in the interface

5.   Open RStudio

6.   Create connection to Hive

a.   library(RJDBC)

b.  drv <- JDBC(driverClass="org.apache.hive.jdbc.HiveDriver", 
classPath=list.files("D:/myJavaDriversFolderFromClusterInstall/", 
pattern="jar$", full.names=T), identifier.quote="`")

c.   conn <- dbConnect(drv, 
"jdbc:hive2://server_name:1/default;ssl=true;sslTrustStore=C:/Progra~1/Java/jdk1.7.0_60/jre/lib/security/cacerts;trustStorePassword=pswd",
 "user", "password")

7.   Verify connection with a small query

a.   r <- dbGetQuery(conn, "select * from test_table_connection_crash order 
by col1 desc limit 100")

b.  print(r)

c.   100 rows are returned to RStudio and properly displayed in the console 
interface

8.   Remove the limit and try the original query (as performed in the SQL 
client application)

a.   r <- dbGetQuery(conn, "select * from test_table_connection_crash order 
by col1 desc")

b.  Query starts running

c.   *** hiveserver crash ***

Worst comes to worst, in the eventuality that RStudio desktop client cannot 
handle such an amount of data, we might expect the desktop application to 
crash; not the whole hiveserver2.

Please let us know whether or not you are aware of any issues of the kind.  
Also, please do not hesitate to request any configuration file you might need 
to examine.

Thank you very much!

Best regards,

Nathalie


[dna_signature]

Nathalie Blais
B.I. Developer | Technology Group
Ubisoft Montreal





testFile.7z
Description: testFile.7z

Re: Oracle backend for Hive Metastore

[Rahul Channe]

Hi Alan,

I am using version 0.12.0, I did not create tables explicitly.

Temporarily I modified the hive-metastore jar to update long data type to
clob and it worked

Not sure if it's hive bug

On Tuesday, September 23, 2014, Alan Gates  wrote:

> What version of Hive are you using?
>
> Did you explicitly create the tables in the metastore via the Oracle
> script or depend on DataNucleus to do it for you?
>
> Alan.
>
>   Rahul Channe 
>  September 22, 2014 at 10:47
> Hi All,
>
> I am using oracle as hive metastore. I could see the schema created in
> oracle after logging into hive and executing create database command
> successfully.
>
> When i am trying to create table in hive its throwing error, please help
>
> hive>  create table test ( name string,age int) row format delimited
> fields terminated by ",";
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(
> message:javax.jdo.JDODataStoreException: An exception was thrown while
> adding/validating class(es) : ORA-01754: a table may contain only one
> column of type LONG
>
> java.sql.SQLException: ORA-01754: a table may contain only one column of
> type LONG
>
> at
> oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
> at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
> at
> oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:946)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1160)
> at
> oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1679)
>
>
> --
> Sent with Postbox 
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Oracle backend for Hive Metastore

[Alan Gates]

What version of Hive are you using?

Did you explicitly create the tables in the metastore via the Oracle 
script or depend on DataNucleus to do it for you?


Alan.


Rahul Channe 
September 22, 2014 at 10:47
Hi All,

I am using oracle as hive metastore. I could see the schema created in 
oracle after logging into hive and executing create database command 
successfully.


When i am trying to create table in hive its throwing error, please help

hive>  create table test ( name string,age int) row format delimited 
fields terminated by ",";
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. 
MetaException(message:javax.jdo.JDODataStoreException: An exception 
was thrown while adding/validating class(es) : ORA-01754: a table may 
contain only one column of type LONG


java.sql.SQLException: ORA-01754: a table may contain only one column 
of type LONG


at 
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
at 
oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:946)
at 
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1160)
at 
oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1679)




--
Sent with Postbox 

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: bug in hive

[Alan Gates]

Shushant,

Creating a patched jar that would include the lock functionality you 
want is unlikely to work.  Wouldn't the following workflow work for you:


1. Writer locks the table explicitly via LOCK TABLE
2. Writer inserts
3. Writer unlocks the table explicitly via UNLOCK TABLE

If you're using ZK for your locking I think the client dying (as opposed 
to ending the session) should cause the lock to expire.  If not, you may 
have to assure the unlock happens in your application.  Hope that helps.


Alan.


Shushant Arora 
September 20, 2014 at 8:00
Hi Alan

I have 0.10 version of hive deployed in my org's cluster, I cannot 
update that because of org's policy.
How can I achieve exclusive lock functionality while inserting in 
dynamic partition on hive 0.10 ?
Does calling hive scripts via some sort of java api with patched jar 
included will help ?
Moreover hive does not release locks in 0.10 when hive session is 
killed . User has to explicitly unlock a table.

Can i specify any sort of max expiry time while taking a lock.

Thanks
Shushant


Alan Gates 
September 20, 2014 at 7:41
Up until Hive 0.13 locks in Hive were really advisory only, since as 
you note any user can remove any other user's lock.  In Hive 0.13 a 
new type of locking was introduced, see 
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-LockManager  
This new locking is automatic and ignores both LOCK and UNLOCK 
commands.  Note that it is off by default, you have to configure Hive 
to use the new DbTxnManager to get turn on this locking.  In 0.13 it 
still has the bug you describe as far as acquiring the wrong lock for 
dynamic partitioning, but I believe I've fixed that in 0.14.


Alan.


Shushant Arora 
September 20, 2014 at 5:39

Hive version 0.9 and later has a bug

While inserting in a hive table Hive takes an exclusive lock. But if 
table is partitioned , and insert is in dynamic partition , it will 
take shared lock on table but if all partitions are static then hive 
takes exclusive lock on partitions in which data is being inserted


and shared lock on table.

https://issues.apache.org/jira/browse/HIVE-3509


1.What if I want to take exclusive lock on table while inserting in 
dynamic partition ?



I tried to take explicit lock using :

LOCK TABLE tablename EXCLUSIVE;


But it made table to be disabled.

I cannot even read from table anymore even is same session until I do

unlock table tablename in another session;


2. moreover whats lock level in hive , I mean any user can remove any 
other users lock. that too seems buggy.



Thanks

Shushant





--
Sent with Postbox 

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: how to read array enclosed within square brackets

[Ankita Bakshi]

Thanks Xuefu. I was hoping I don't have to do that. We create this file by
serializing java array so was hoping that it would be a common case to
handle arrays with square brackets.

Thanks,
Ankita

On Mon, Sep 22, 2014 at 7:55 PM, Xuefu Zhang  wrote:

> Hive doesn't know it needs to skip your square brackets, so you numbers
> are really [1, 2, and 3]. [1 and 3] cannot be parsed to numbers, so they
> become null.
>
> I think you interpret the second column as [1, 2, 3] of type string. Then
> you can  remove the brackets, and use a UDF (write your own if there isn't
> one) to generate integer array from the striped string.
>
>
> On Mon, Sep 22, 2014 at 6:12 PM, Ankita Bakshi 
> wrote:
>
>> Hi,
>>
>> I have '|' delimited file where arrays are serialized with square
>> brackets. I am trying create a hive table to parse this file.
>>
>> Example:
>>
>> first|[1,2,3]|100
>>
>> second|[11,12,13]|200
>>
>>
>> Create External Table H_histoTest(dim1 string, hist ARRAY,
>> measure1 bigint)
>>
>> ROW FORMAT DELIMITED FIELDS
>>
>> TERMINATED BY '|'
>>
>> COLLECTION ITEMS TERMINATED BY ','
>>
>> LINES TERMINATED BY '\n'
>>
>> LOCATION '/user/ankita/hive/histoTest';
>>
>>
>> hive> select * from H_histoTest;
>>
>> first [null,2,null] 100
>>
>> second [null,12,null] 200
>>
>>
>>
>> If I remove the square brackets than the array is parsed correctly.
>>
>> first|1,2,3|100
>>
>> second|11,12,13|200
>>
>>
>> hive> select * from H_histoTest;
>>
>> first [1,2,3] 100
>>
>> second [11,12,13] 200
>>
>>
>> Let me know if I am missing something.
>>
>>
>> Thanks,
>> Ankita
>>
>
>

Re: ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory

[Poorvi Ahirwal]

you are saying about mapred program or some property file?

On Tue, Sep 23, 2014 at 4:14 AM, hadoop hive  wrote:

> Comment the jdo properties from mapred and hdfs
> On Sep 23, 2014 4:42 PM, "Poorvi Ahirwal" 
> wrote:
>
>> Hi,
>> I am executing a mapreduce program with hcatalog and hive database. Even
>> if the jars are included its showing this error:
>>
>> Exception in thread "main" java.io.IOException:
>> com.google.common.util.concurrent.UncheckedExecutionException:
>> javax.jdo.JDOFatalUserException: Class
>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
>> NestedThrowables:
>> java.lang.ClassNotFoundException:
>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>> at
>> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
>> at
>> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
>> ..
>> Please help
>>
>> thanks
>>
>

Re: ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory

[hadoop hive]

Comment the jdo properties from mapred and hdfs
On Sep 23, 2014 4:42 PM, "Poorvi Ahirwal"  wrote:

> Hi,
> I am executing a mapreduce program with hcatalog and hive database. Even
> if the jars are included its showing this error:
>
> Exception in thread "main" java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> javax.jdo.JDOFatalUserException: Class
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
> NestedThrowables:
> java.lang.ClassNotFoundException:
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
> at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
> at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
> ..
> Please help
>
> thanks
>

ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory

[Poorvi Ahirwal]

Hi,
I am executing a mapreduce program with hcatalog and hive database. Even if
the jars are included its showing this error:

Exception in thread "main" java.io.IOException:
com.google.common.util.concurrent.UncheckedExecutionException:
javax.jdo.JDOFatalUserException: Class
org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
NestedThrowables:
java.lang.ClassNotFoundException:
org.datanucleus.api.jdo.JDOPersistenceManagerFactory
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
..
Please help

thanks

how to convert LazyLong to Long in hive generic udf ?

[Dan Fan]

Hi Hive users:

Does anyone know how to convert LazyLong to Long in hive generic udf  ?


Thanks

Dan

hive thrift sdk not work

[flreey]

hello,i use
https://github.com/apache/hive/blob/trunk/service/if/TCLIService.thrift to
gen golang sdk, but it ddn't work. The golang sdk can only opensession. I
compare my code within python sdk and didn't find any error usage. below is
my golang code.


> socket, err := thrift.NewTSocket("localhost:11000")
if err != nil {
fmt.Printf("%s\n", err)
return
}
trans := thrift.NewTBufferedTransport(socket, 1024)

protocol := thrift.NewTBinaryProtocol(trans, true, true)
client := hive.NewTCLIServiceClientProtocol(trans, protocol, protocol)
trans.Open()
defer trans.Close()

sReq := hive.NewTOpenSessionReq()
sReq.ClientProtocol = 0

session, err := client.OpenSession(sReq)
fmt.Printf("!%#v %s\n", session, err)

exeReq := hive.NewTExecuteStatementReq()
exeReq.SessionHandle = session.SessionHandle
exeReq.Statement = "USE default"

result, err := client.ExecuteStatement(exeReq)
fmt.Printf("result: %s !! %s\n", result, err)



and the output is :

!&hive.TOpenSessionResp{Status:(*hive.TStatus)(0xc21001e460),
ServerProtocolVersion:0,
SessionHandle:(*hive.TSessionHandle)(0xc212b0),
Configuration:map[string]string{}} %!s()

result:  !! Required field 'guid' is unset!
Struct:THandleIdentifier(guid:null, secret:null)

Required field 'guid' is unset! Struct:THandleIdentifier(guid:null,
secret:null) is return by the server. I have try other hive functions,
always the same error.

Is there something wrong with my code or thirft didn't support golang yet?

Re:

[Poorvi Ahirwal]

sorry i forgot to mention. i am not using eclipse

On Tue, Sep 23, 2014 at 12:14 AM, Karthiksrivasthava <
karthiksrivasth...@gmail.com> wrote:

> Try to include hive-site.xml and yarn-site.xml to eclipse and include yarn
> application class path in yarn-site.xml
>
> Thanks
> Karthik
> > On Sep 23, 2014, at 2:06, Poorvi Ahirwal 
> wrote:
> >
> > Hi,
> > I am executing a mapreduce program with hcatalog and hive database. Even
> if the jars are included its showing this error:
> >
> > Exception in thread "main" java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> javax.jdo.JDOFatalUserException: Class
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
> > NestedThrowables:
> > java.lang.ClassNotFoundException:
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
> >   at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
> >   at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
> > ..
> > Please help
> >
> > thanks
>

Re:

[Karthiksrivasthava]

Try to include hive-site.xml and yarn-site.xml to eclipse and include yarn 
application class path in yarn-site.xml 

Thanks 
Karthik
> On Sep 23, 2014, at 2:06, Poorvi Ahirwal  wrote:
> 
> Hi,
> I am executing a mapreduce program with hcatalog and hive database. Even if 
> the jars are included its showing this error:
> 
> Exception in thread "main" java.io.IOException: 
> com.google.common.util.concurrent.UncheckedExecutionException: 
> javax.jdo.JDOFatalUserException: Class 
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
> NestedThrowables:
> java.lang.ClassNotFoundException: 
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory
>   at 
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
>   at 
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
> ..
> Please help
> 
> thanks

[no subject]

[Poorvi Ahirwal]

Hi,
I am executing a mapreduce program with hcatalog and hive database. Even if
the jars are included its showing this error:

Exception in thread "main" java.io.IOException:
com.google.common.util.concurrent.UncheckedExecutionException:
javax.jdo.JDOFatalUserException: Class
org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found.
NestedThrowables:
java.lang.ClassNotFoundException:
org.datanucleus.api.jdo.JDOPersistenceManagerFactory
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:88)
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:64)
..
Please help

thanks