Re: Hive Installation issues: NoClassDefFoundError
This should be a JAR files conflict issue . Please check if $HADOOP_HOME/lib and $HIVE_HOME/lib both have thilibthrift-*.jar , if do have, remove one of them and retry. 2013/6/24 Kannu Mehta > Hi All, > > I'm trying to run Hadoop and Hive in a pseudo-distributed mode on my local > machine. I've already got hadoop running but when I try to run Hive, it > gives me the following error > > `Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/thrift/TException > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:266) > at org.apache.hadoop.util.RunJar.main(RunJar.java:149) > Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TException > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > ... 3 more ` > > I have tried to debug the problem but couldn't find what's missing. My > HADOOP_HOME, HADOOP_CLASSPATH are being set properly. Also, > HADOOP_CLASSPATH contains libthrift-0.9.0.jar which has the class shown in > the error above. > > Any suggestion what am I missing here? > > -- > Regards, > Kannu Mehta > -- Best wishs！ Fangkun.Cao
Re: What is HIVE_PLAN?
The plan will be serialized to the default hdfs instance , and put in
distributed cache.
So please have a check of the distributed cache local directory of every
tasktracker
Common like :
* {hadoop.tmp.dir}/mapred/local/taskTracker*
2013/6/7 Li jianwei
> Hi FangKun:
> Thanks for your reply!
> I ran the "select count(*)" again, and check the JobConf, find the
> property you mentioned, they were as following:
> *hive.exec.plan *hdfs://
> 192.168.1.112:9100/tmp/hive-cyg_server/hive_2013-06-07_12-56-10_656_195237350266205704/-mr-10003/e1438d71-2497-4834-a89e-8b2e7d78448d
> *hive.exec.scratchdir */tmp/hive-cyg_server
> when hive was running, I browsed the HDFS filesystem, the file specified
> by *hive.exec.plan* was there with permission rwsr-xr-x, but I didn't
> find any file had "HIVE_PLAN" in its name under any subdir of *
> hive.exec.scratchdir*. I also set the permission of *hive.exec.scratchdir*to
> rwxrwxrwx.
> Is it not the problem in HDFS? According to the java exception, it is the
> native java method java.io.FileInputStream.open which can not access the
> file, which probably is in the local filesystem of the tasktracker node.
>
> --
> Date: Fri, 7 Jun 2013 12:09:24 +0800
> Subject: Re: What is HIVE_PLAN?
> From: caofang...@gmail.com
> To: user@hive.apache.org
>
>
> It's kept in JobConf as part of the plan file name.
> Check the link below
> http://hdfs-namenode:50030/jobconf.jsp?jobid=job_201306070901_0001
>
> and find * hive.exec.plan * and * hive.exec.scratchdir*
> Do you have proper Read and Write permissions ?
>
>
>
>
> 2013/6/7 Li jianwei
>
> Hi, everyone:
> I have set up a hadoop cluster on THREE windows7 machines with Cygwin, and
> made several test, which were all passed, with hadoop-test-1.1.2.jar and
> hadoop-examples-1.1.2.jar.
> Then I tried to run Hive 0.10.0 on my cluster ( also in Cygwin ). I could
> create tables, show them, load data into them and "select *" from them. But
> when I tried "select count(*)" from my table, I've got the following
> exception. *My question is: what is that HIVE_PLANxx file? how is it
> created? where is it placed?*
> Would anyone give me some infomation?
> ..
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=
> Starting Job = job_201306070901_0001, Tracking URL =
> http://hdfs-namenode:50030/jobdetails.jsp?jobid=job_201306070901_0001
> Kill Command = C:\hadoop-1.1.2\\bin\hadoop.cmd job -kill
> job_201306070901_0001
> Hadoop job information for Stage-1: number of mappers: 13; number of
> reducers: 1
> 2013-06-07 09:02:19,296 Stage-1 map = 0%, reduce = 0%
> 2013-06-07 09:02:51,745 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201306070901_0001 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://hdfs-namenode:50030/jobdetails.jsp?jobid=job_201306070901_0001
> Examining task ID: task_201306070901_0001_m_14 (and more) from job
> job_201306070901_0001
>
> Task with the most failures(4):
> -
> Task ID:
> task_201306070901_0001_m_06
>
> URL:
>
> http://hdfs-namenode:50030/taskdetails.jsp?jobid=job_201306070901_0001&tipid=task_201306070901_0001_m_06
> -
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: java.io.FileNotFoundException: *
> HIVE_PLANc632c8e2-257d-4cd4-b833-a09c7d249b2c* (Access is denied)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:226)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:536)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:197)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.io.FileNotFoundException: *
> HIVE_PLANc632c8e2-257d-4cd4-b833-a09c7d249b2c* (Access is denied)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.
Re: Hive Header line in Select query help?
There's a issue working on this : https://issues.apache.org/jira/browse/HIVE-4346 2013/6/7 Nitin Pawar > if you do not have hiverc file then you can create one and use it with > --hiveconf along side your hive execution > you also set the same value just before your select query starts > > > On Thu, Jun 6, 2013 at 10:13 PM, Matouk IFTISSEN < > matouk.iftis...@ysance.com> wrote: > >> I use Hortonworks on windows (HDInsight) and I dont’ find the file >> hiverc , where can I fin it? And where I add >> set hive.cli.print.header=true; >> >> Thanks; >> >> *De :* Nitin Pawar [mailto:nitinpawar...@gmail.com] >> *Envoyé :* jeudi 6 juin 2013 18:31 >> >> *À :* user@hive.apache.org >> *Objet :* Re: Hive Header line in Select query help? >> >> >> >> why don't you set the variable in your script or in the hiverc file ? >> then it will work all the time >> >> >> >> On Thu, Jun 6, 2013 at 9:43 PM, Matouk IFTISSEN < >> matouk.iftis...@ysance.com> wrote: >> >> I use a script to query Hive and store results in Local file, but no >> headers after execution of the select query how to do this? thanks >> >> >> >> *De :* Nitin Pawar [mailto:nitinpawar...@gmail.com] >> *Envoyé :* jeudi 6 juin 2013 18:02 >> *À :* user@hive.apache.org >> *Objet :* Re: Hive Header line in Select query help? >> >> >> >> if you look at the setting >> >> it says "hive.cli" that means it is supposed to work on cli only. >> >> >> >> Which interface you want to use it ? >> >> >> >> On Thu, Jun 6, 2013 at 9:12 PM, Matouk IFTISSEN < >> matouk.iftis...@ysance.com> wrote: >> >> Hello Ivry bady, >> >> >> >> I want to know if is there a way to export headers line in select query >> to store the result in file from local or HDFS directory? >> >> like this query results : >> >> >> >> set hive.cli.print.header=true; >> >> >> >> INSERT OVERWRITE LOCAL DIRECTORY 'C:\resultats\alerts_http_500\par_heure' >> >> SELECT heure_actuelle,nb_erreur_500, pct_erreurs_500 , >> >> heure_precedentele, nb_erreur_500, hp.pct_erreurs_500 >> >> FROM Ma_table; >> >> >> >> the set hive.cli.print.header=true; does not work it works only in the >> CLI. >> >> >> >> *thanks for your answers ;)* >> >> *Matouk* >> >> >> >> >> >> >> >> -- >> Nitin Pawar >> >> >> >> >> >> -- >> Nitin Pawar >> > > > > -- > Nitin Pawar > -- Best wishs！ Fangkun.Cao
Re: What is HIVE_PLAN?
It's kept in JobConf as part of the plan file name. Check the link below http://hdfs-namenode:50030/jobconf.jsp?jobid=job_201306070901_0001 and find * hive.exec.plan * and * hive.exec.scratchdir* Do you have proper Read and Write permissions ? 2013/6/7 Li jianwei > Hi, everyone: > I have set up a hadoop cluster on THREE windows7 machines with Cygwin, and > made several test, which were all passed, with hadoop-test-1.1.2.jar and > hadoop-examples-1.1.2.jar. > Then I tried to run Hive 0.10.0 on my cluster ( also in Cygwin ). I could > create tables, show them, load data into them and "select *" from them. But > when I tried "select count(*)" from my table, I've got the following > exception. *My question is: what is that HIVE_PLANxx file? how is it > created? where is it placed?* > Would anyone give me some infomation? > .. > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks determined at compile time: 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer= > In order to limit the maximum number of reducers: > set hive.exec.reducers.max= > In order to set a constant number of reducers: > set mapred.reduce.tasks= > Starting Job = job_201306070901_0001, Tracking URL = > http://hdfs-namenode:50030/jobdetails.jsp?jobid=job_201306070901_0001 > Kill Command = C:\hadoop-1.1.2\\bin\hadoop.cmd job -kill > job_201306070901_0001 > Hadoop job information for Stage-1: number of mappers: 13; number of > reducers: 1 > 2013-06-07 09:02:19,296 Stage-1 map = 0%, reduce = 0% > 2013-06-07 09:02:51,745 Stage-1 map = 100%, reduce = 100% > Ended Job = job_201306070901_0001 with errors > Error during job, obtaining debugging information... > Job Tracking URL: > http://hdfs-namenode:50030/jobdetails.jsp?jobid=job_201306070901_0001 > Examining task ID: task_201306070901_0001_m_14 (and more) from job > job_201306070901_0001 > > Task with the most failures(4): > - > Task ID: > task_201306070901_0001_m_06 > > URL: > > http://hdfs-namenode:50030/taskdetails.jsp?jobid=job_201306070901_0001&tipid=task_201306070901_0001_m_06 > - > Diagnostic Messages for this Task: > java.lang.RuntimeException: java.io.FileNotFoundException: * > HIVE_PLANc632c8e2-257d-4cd4-b833-a09c7d249b2c* (Access is denied) > at > org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:226) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:536) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:197) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Unknown Source) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.io.FileNotFoundException: * > HIVE_PLANc632c8e2-257d-4cd4-b833-a09c7d249b2c* (Access is denied) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(Unknown Source) > at java.io.FileInputStream.(Unknown Source) > at > org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:217) > ... 12 more > > > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > MapReduce Jobs Launched: > Job 0: Map: 13 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL > Total MapReduce CPU Time Spent: 0 msec > -- Best wishs！ Fangkun.Cao
Re: Where is exception stacktrace and root cause for UDF
Hi you can find the Exception log at map or reduce task logs .
http://your_task_tracker:50060/tasklog?taskid=attempt_20130*_*_r_00_0&all=true
2013/3/21 Harsh J
> They should be viewable if you enable the DEBUG mode.
>
> hive -hiveconf hive.root.logger=DEBUG,console
>
> On Thu, Mar 21, 2013 at 1:31 AM, Marc Limotte wrote:
> > Hi.
> >
> > I'm trying to understand what happens to Exceptions that are thrown by
> > custom UDFs. I have a UDF that throws a RuntimeException with useful
> > debugging information, but I don't see it in any logs. I'm running my
> > commands using the hive cli (Hive 0.9.0). Instead I see:
> >
> >> FAILED: Error in semantic analysis: Line 3:7 Wrong arguments '600':
> >> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute
> method …
> >> evaluate(...) on object ...
> >> Command exited with non-zero status 10
> >
> >
> > Despite the "semantic analysis" error, I know it's getting to the
> > evaluate(…) method because of other log messages.
> >
> > Looking in org.apache.hadoop.hive.ql.exec.FunctionRegistry, I find the
> > invoke method, which contains:
> >
> > public static Object invoke(Method m, Object thisObject, Object...
> > arguments)
> > throws HiveException {
> > Object o;
> > try {
> > o = m.invoke(thisObject, arguments);
> > } catch (Exception e) {
> > ...
> > throw new HiveException("Unable to execute method " + m + " "
> > + " on object " + thisObjectString + " with arguments "
> > + argumentString.toString(), e);
> > }
> > return o;
> > }
> >
> > So it appears that my RuntimeException is wrapped in HiveException. But
> > then it looks like only the Exception message is printed by the hive cli.
> > Does the stack trace and root cause get printed in some log? I didn't
> find
> > it anywhere.
> >
> >
> > Marc
> >
>
>
>
> --
> Harsh J
>
--
Best wishs！
Fangkun.Cao
Re: 0.8.0 -> 0.9.0 mysql schema upgrade
Hi Sam William: Check this issue: https://issues.apache.org/jira/browse/HIVE-3649 and http://svn.apache.org/repos/asf/hive/tags/release-0.10.0/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql 2013/1/5 Sam William > Looks like this column is not even there in the 0.8/0.9 schema files . I > have no idea, how I have it in my schema . I just set a default 'false' > value and I m fine now. > Sam > > On Jan 4, 2013, at 2:22 PM, Sam William wrote: > > > When I upgraded to 0.9.0, Im getting an exception when I try to create > tables > > > > FAILED: Error in metadata: javax.jdo.JDODataStoreException: Insert of > object "org.apache.hadoop.hive.metastore.model.MStorageDescriptor@4774e78a" > using statement "INSERT INTO `SDS` > (`SD_ID`,`NUM_BUCKETS`,`LOCATION`,`INPUT_FORMAT`,`CD_ID`,`OUTPUT_FORMAT`,`SERDE_ID`,`IS_COMPRESSED`) > VALUES (?,?,?,?,?,?,?,?)" failed : Field 'IS_STOREDASSUBDIRECTORIES' > doesn't have a default value > > NestedThrowables: > > java.sql.SQLException: Field 'IS_STOREDASSUBDIRECTORIES' doesn't have a > default value > > > > > > The upgrade script from 0.8 to 0.9 doesnt have anything ? What am I > missing ? > > > > Sam William > > sa...@stumbleupon.com > > > > > > > > Sam William > sa...@stumbleupon.com > > > > -- Best wishs！ Fangkun.Cao
