I have added all needed jars to eclipse class path including hive-metastore.jar libthrift.jar hive-core.jar libfb303.jar jdo*-api.jar slf4j-api.jar ,can you tell me how can I register these jars in pig script? use setProperty? or others?
At 2015-03-04 02:34:05, "Daniel Dai" <da...@hortonworks.com> wrote: >HCatSchema is in hcatalog-core.jar and it is missing in the backend. You >will need to register several jars in pig script: > >hcatalog-core.jar > >hive-metastore.jar >libthrift.jar >hive-core.jar >libfb303.jar >jdo*-api.jar >slf4j-api.jar > > > > > > > >When you run in command line, bin/pig script does that for you. But in >eclipse, you will need to do it by yourself. > >Daniel > >On 3/3/15, 4:40 AM, "李运田" <cumt...@163.com> wrote: > >>2015-03-03 19:54:43,072 INFO [main] >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for >>application appattempt_1421806758143_0252_000001 >>2015-03-03 19:54:43,623 WARN [main] org.apache.hadoop.conf.Configuration: >>job.xml:an attempt to override final parameter: >>mapreduce.job.end-notification.max.retry.interval; Ignoring. >>2015-03-03 19:54:43,650 WARN [main] org.apache.hadoop.conf.Configuration: >>job.xml:an attempt to override final parameter: >>mapreduce.job.end-notification.max.attempts; Ignoring. >>2015-03-03 19:54:43,778 WARN [main] >>org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop >>library for your platform... using builtin-java classes where applicable >>2015-03-03 19:54:43,798 INFO [main] >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: >>2015-03-03 19:54:43,798 INFO [main] >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, >>Service: , Ident: >>(org.apache.hadoop.yarn.security.AMRMTokenIdentifier@307c9a57) >>2015-03-03 19:54:43,833 INFO [main] >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max >>attempts: 2 for application: 252. Attempt num: 1 is last retry: false >>2015-03-03 19:54:43,841 INFO [main] >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred >>newApiCommitter. >>2015-03-03 19:54:43,985 WARN [main] org.apache.hadoop.conf.Configuration: >>job.xml:an attempt to override final parameter: >>mapreduce.job.end-notification.max.retry.interval; Ignoring. >>2015-03-03 19:54:43,999 WARN [main] org.apache.hadoop.conf.Configuration: >>job.xml:an attempt to override final parameter: >>mapreduce.job.end-notification.max.attempts; Ignoring. >>2015-03-03 19:54:44,535 WARN [main] >>org.apache.hadoop.hdfs.BlockReaderLocal: The short-circuit local reads >>feature cannot be used because libhadoop cannot be loaded. >>2015-03-03 19:54:44,719 INFO [main] >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in >>config null >>2015-03-03 19:54:44,844 INFO [main] >>org.apache.hadoop.service.AbstractService: Service >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; >>cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: >>java.io.IOException: Deserialization error: >>org.apache.hcatalog.data.schema.HCatSchema >>org.apache.hadoop.yarn.exceptions.YarnRuntimeException: >>java.io.IOException: Deserialization error: >>org.apache.hcatalog.data.schema.HCatSchema >> at >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRApp >>Master.java:473) >> at >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.jav >>a:374) >> at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1456 >>) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at >>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation. >>java:1548) >> at >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRApp >>Master.java:1453) >> at >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1386) >>Caused by: java.io.IOException: Deserialization error: >>org.apache.hcatalog.data.schema.HCatSchema >> at >>org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.jav >>a:59) >> at org.apache.pig.impl.util.UDFContext.deserialize(UDFContext.java:192) >> at >>org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupUDFCont >>ext(MapRedUtil.java:173) >> at >>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputForm >>at.setupUdfEnvAndStores(PigOutputFormat.java:229) >> at >>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputForm >>at.getOutputCommitter(PigOutputFormat.java:275) >> at >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRApp >>Master.java:471) >> ... 8 more >>Caused by: java.lang.ClassNotFoundException: >>org.apache.hcatalog.data.schema.HCatSchema >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> >> >> >> >> >> >> >>在 2015-02-28 09:37:29,"李运田" <cumt...@163.com> 写道: >> >>pigServer.registerQuery("tmp = load 'pig' using >>org.apache.hcatalog.pig.HCatLoader();"); >>pigServer.registerQuery("tmp = foreach tmp generate id;"); >>when I execute this,I cant get any error >>but,When I execute >>pigServer.registerQuery("store tmp into 'hive' using >>org.apache.hcatalog.pig.HCatStorer();"); >>I get error like this::: >>org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to >>open iterator for alias tmp >> at org.apache.pig.PigServer.openIterator(PigServer.java:880) >> at org.gradle.PigHiveHCat.run(PigHiveHCat.java:68) >> at org.gradle.PigHiveHCat.main(PigHiveHCat.java:28) >>Caused by: java.io.IOException: Job terminated with anomalous status >>FAILED >> at org.apache.pig.PigServer.openIterator(PigServer.java:872) >> ... 2 more >>.......................................................................... >>................................. >>pigServer.registerQuery("a = LOAD '/user/hadoop/pig.txt' ;"); >>pigServer.store("a", "/user/hadoop/pig1.txt"); >>pigServer.registerQuery("store a into '/user/hadoop/pig2.txt';"); >>........................................this is OK,so I think something >>about Hcatalog is wrong........................... >> >> >>At 2015-02-28 01:21:12, "Alan Gates" <alanfga...@gmail.com> wrote: >>What error message are you getting? >> >>Alan. >> >> >>李运田 >>February 26, 2015 at 18:58 >>I want to use hcatalog in eclipse to deal with tables in hive. >>but I cant store table into hive:: >>pigServer.registerQuery("tmp = load 'pig' using >>org.apache.hcatalog.pig.HCatLoader();"); >>pigServer.registerQuery("tmp = foreach tmp generate id;"); >>pigServer.registerQuery("store tmp into 'hive' using >>org.apache.hcatalog.pig.HCatStorer();"); >>I can store into file:: >>pigServer.registerQuery("a = LOAD '/user/hadoop/pig.txt' ;"); >>pigServer.store("a", "/user/hadoop/pig1.txt"); >>pigServer.registerQuery("store a into '/user/hadoop/pig2.txt';"); >>perhaps ,the hcatalog jars are wrong? >> >> >