java.io.FileNotFoundException: File does not exist (nexr-hive-udf-0.2-SNAPSHOT.jar)
Hi, Please help! I am using hiveserver2 on HIVE 0.13 on Hadoop 2.4.1, also nexr-hive-udf-0.2-SNAPSHOT.jar I can run query from CLI, e.g. hive SELECT add_months(sysdate(), +12) FROM DUAL; Execution completed successfully MapredLocal task succeeded OK 2015-12-17 Time taken: 7.393 seconds, Fetched: 1 row(s) hive-site.xml (added) property namehive.aux.jars.path/name value$HIVE_HOME/nexr-hive-udf-0.2-SNAPSHOT.jar,$HIVE_HOME/csv-serde-1.1.2-0.11.0-all.jar/value /property hive-env.sh (added) export HIVE_AUX_JARS_PATH=$HIVE_HOME/lib/csv-serde-1.1.2-0.11.0-all.jar:$HIVE_HOME/lib/nexr-hive-udf-0.2-SNAPSHOT.jar However, if it is accessed via hiveserver2, I got the following error, please help. Regards Arthur 14/12/17 16:47:52 WARN conf.Configuration: file:/tmp/hive_2014-12-17_16-47-51_096_5821374687950910377-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. Execution log at: /tmp/hduser_20141217164747_80b15b85-7820-4e3a-88ea-afffa131ff5a.log java.io.FileNotFoundException: File does not exist: hdfs://mycluster/hadoop_data/hadoop_data/tmp/mapred/staging/hduser1962118853/.staging/job_local1962118853_0001/libjars/nexr-hive-udf-0.2-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://mycluster/hadoop_data/hadoop_data/tmp/mapred/staging/hduser1962118853/.staging/job_local1962118853_0001/libjars/nexr-hive-udf-0.2-SNAPSHOT.jar)' Execution failed with exit status: 1 Obtaining error information
Re: java.io.FileNotFoundException: File does not exist (nexr-hive-udf-0.2-SNAPSHOT.jar)
Hi Arthur Have a look at here https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions *RegardsMuthupandi.K* Think before you print. On Wed, Dec 17, 2014 at 2:53 PM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote: Hi, Please help! I am using *hiveserver2 *on HIVE 0.13 on Hadoop 2.4.1, also nexr-hive-udf-0.2-SNAPSHOT.jar I can run query from CLI, e.g. hive SELECT add_months(sysdate(), +12) FROM DUAL; Execution completed successfully MapredLocal task succeeded OK 2015-12-17 Time taken: 7.393 seconds, Fetched: 1 row(s) hive-site.xml (added) property namehive.aux.jars.path/name value$HIVE_HOME/nexr-hive-udf-0.2-SNAPSHOT.jar,$HIVE_HOME/csv-serde-1.1.2-0.11.0-all.jar/value /property hive-env.sh (added) export HIVE_AUX_JARS_PATH=$HIVE_HOME/lib/csv-serde-1.1.2-0.11.0-all.jar:$HIVE_HOME/lib/nexr-hive-udf-0.2-SNAPSHOT.jar However, if it is accessed via *hiveserver2, *I got the following error, please help. Regards Arthur 14/12/17 16:47:52 WARN conf.Configuration: file:/tmp/hive_2014-12-17_16-47-51_096_5821374687950910377-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. Execution log at: /tmp/hduser_20141217164747_80b15b85-7820-4e3a-88ea-afffa131ff5a.log java.io.FileNotFoundException: File does not exist: hdfs://mycluster/hadoop_data/hadoop_data/tmp/mapred/staging/hduser1962118853/.staging/job_local1962118853_0001/libjars/nexr-hive-udf-0.2-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://mycluster/hadoop_data/hadoop_data/tmp/mapred/staging/hduser1962118853/.staging/job_local1962118853_0001/libjars/nexr-hive-udf-0.2-SNAPSHOT.jar )' Execution failed with exit status: 1 Obtaining error information
Is it possible to use count in Where clause ?
Hi, I am quite new to Hive. I just wonder if a query like this is possible in hive? select * from events where events.pid = '123' having count(events.id) 2 or more precisely is it possible to use having count(events.id) 2 in Hive SQL ? best, /Shahab
Hive 0.14 SQL Inserts Complex Data Types
With 0.14 is it possible to use the INSERT...VALUES statement to insert data into a table with complex types like arrays, maps, etc? We will never be loading data from a file which is why I ask.
Row Delimiter in Create table
Hi, I am trying to create table for a text file who has a row delimiter other than new line character. I know hive create table does not support anything other than new line. I have columns where in data contains new line so i specified a new line character in my sqoop. What are the best options as far as i googled or looked at the manual a transformation is required but my table is real huge. Thanks, G
Re: relationship between a hive query and a tez dag
DDL operations are done from the client, not as part of a Tez DAG. There still is no linkage between the DAGs even if they access the same tables or somehow depend on each other. Hive manages making sure queries are executed in the order given. Tez just executed each query. Alan. Xiaoyong Zhu mailto:xiaoy...@microsoft.com December 16, 2014 at 15:41 Thanks Alan! Then what if different hive queries have some links (e.g. query 1 creates table abc and query 2 reads data from abc) will this be in 1 DAG or 2 DAGs? If 2, then is there any link/relatioinship between the 2 DAGs? Xiaoyong *From:*Alan Gates [mailto:ga...@hortonworks.com] *Sent:* Wednesday, December 17, 2014 5:27 AM *To:* user@hive.apache.org *Subject:* Re: relationship between a hive query and a tez dag Several. Each Hive query becomes one Tez DAG. Alan. -- Sent with Postbox http://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. Alan Gates mailto:ga...@hortonworks.com December 16, 2014 at 13:26 Several. Each Hive query becomes one Tez DAG. Alan. -- Sent with Postbox http://www.getpostbox.com -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: relationship between a hive query and a tez dag
I see, thanks for the explaination! Xiaoyong From: Alan Gates [mailto:ga...@hortonworks.com] Sent: Thursday, December 18, 2014 8:05 AM To: Xiaoyong Zhu Cc: user@hive.apache.org Subject: Re: relationship between a hive query and a tez dag DDL operations are done from the client, not as part of a Tez DAG. There still is no linkage between the DAGs even if they access the same tables or somehow depend on each other. Hive manages making sure queries are executed in the order given. Tez just executed each query. Alan. [cid:image001.jpg@01D01A99.C2B8BDF0] Xiaoyong Zhumailto:xiaoy...@microsoft.com December 16, 2014 at 15:41 Thanks Alan! Then what if different hive queries have some links (e.g. query 1 creates table abc and query 2 reads data from abc) will this be in 1 DAG or 2 DAGs? If 2, then is there any link/relatioinship between the 2 DAGs? Xiaoyong From: Alan Gates [mailto:ga...@hortonworks.com] Sent: Wednesday, December 17, 2014 5:27 AM To: user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: relationship between a hive query and a tez dag Several. Each Hive query becomes one Tez DAG. Alan. -- Sent with Postboxhttp://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. [cid:image002.jpg@01D01A99.C2B8BDF0] Alan Gatesmailto:ga...@hortonworks.com December 16, 2014 at 13:26 Several. Each Hive query becomes one Tez DAG. Alan. -- Sent with Postboxhttp://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Row Delimiter in Create table
Afaik, it was restricted by implementation of hadoop. But now hadoop-2 supports custom delimiter, hopefully it also can be implemented in hive. I'm not sure but currently possible way of do that is setting textinputformat.record.delimiter in table properties. Thanks, Navis 2014-12-18 6:20 GMT+09:00 Gayathri Swaroop g3.vasude...@gmail.com: Hi, I am trying to create table for a text file who has a row delimiter other than new line character. I know hive create table does not support anything other than new line. I have columns where in data contains new line so i specified a new line character in my sqoop. What are the best options as far as i googled or looked at the manual a transformation is required but my table is real huge. Thanks, G
