I have added all needed jars to eclipse class path including
hive-metastore.jar
libthrift.jar
hive-core.jar
libfb303.jar
jdo*-api.jar
slf4j-api.jar
,can you tell me how can I register these jars in pig script?
use setProperty? or others?
At 2015-03-04 02:34:05, "Daniel Dai" <da...@hortonworks.com> wrote:
>HCatSchema is in hcatalog-core.jar and it is missing in the backend. You
>will need to register several jars in pig script:
>
>hcatalog-core.jar
>
>hive-metastore.jar
>libthrift.jar
>hive-core.jar
>libfb303.jar
>jdo*-api.jar
>slf4j-api.jar
>
>
>
>
>
>
>
>When you run in command line, bin/pig script does that for you. But in
>eclipse, you will need to do it by yourself.
>
>Daniel
>
>On 3/3/15, 4:40 AM, "李运田" <cumt...@163.com> wrote:
>
>>2015-03-03 19:54:43,072 INFO [main]
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
>>application appattempt_1421806758143_0252_000001
>>2015-03-03 19:54:43,623 WARN [main] org.apache.hadoop.conf.Configuration:
>>job.xml:an attempt to override final parameter:
>>mapreduce.job.end-notification.max.retry.interval; Ignoring.
>>2015-03-03 19:54:43,650 WARN [main] org.apache.hadoop.conf.Configuration:
>>job.xml:an attempt to override final parameter:
>>mapreduce.job.end-notification.max.attempts; Ignoring.
>>2015-03-03 19:54:43,778 WARN [main]
>>org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>>library for your platform... using builtin-java classes where applicable
>>2015-03-03 19:54:43,798 INFO [main]
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
>>2015-03-03 19:54:43,798 INFO [main]
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
>>Service: , Ident:
>>(org.apache.hadoop.yarn.security.AMRMTokenIdentifier@307c9a57)
>>2015-03-03 19:54:43,833 INFO [main]
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max
>>attempts: 2 for application: 252. Attempt num: 1 is last retry: false
>>2015-03-03 19:54:43,841 INFO [main]
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred
>>newApiCommitter.
>>2015-03-03 19:54:43,985 WARN [main] org.apache.hadoop.conf.Configuration:
>>job.xml:an attempt to override final parameter:
>>mapreduce.job.end-notification.max.retry.interval; Ignoring.
>>2015-03-03 19:54:43,999 WARN [main] org.apache.hadoop.conf.Configuration:
>>job.xml:an attempt to override final parameter:
>>mapreduce.job.end-notification.max.attempts; Ignoring.
>>2015-03-03 19:54:44,535 WARN [main]
>>org.apache.hadoop.hdfs.BlockReaderLocal: The short-circuit local reads
>>feature cannot be used because libhadoop cannot be loaded.
>>2015-03-03 19:54:44,719 INFO [main]
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in
>>config null
>>2015-03-03 19:54:44,844 INFO [main]
>>org.apache.hadoop.service.AbstractService: Service
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED;
>>cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
>>java.io.IOException: Deserialization error:
>>org.apache.hcatalog.data.schema.HCatSchema
>>org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
>>java.io.IOException: Deserialization error:
>>org.apache.hcatalog.data.schema.HCatSchema
>> at
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRApp
>>Master.java:473)
>> at
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.jav
>>a:374)
>> at
>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>> at
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1456
>>)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>java:1548)
>> at
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRApp
>>Master.java:1453)
>> at
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1386)
>>Caused by: java.io.IOException: Deserialization error:
>>org.apache.hcatalog.data.schema.HCatSchema
>> at
>>org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.jav
>>a:59)
>> at org.apache.pig.impl.util.UDFContext.deserialize(UDFContext.java:192)
>> at
>>org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupUDFCont
>>ext(MapRedUtil.java:173)
>> at
>>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputForm
>>at.setupUdfEnvAndStores(PigOutputFormat.java:229)
>> at
>>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputForm
>>at.getOutputCommitter(PigOutputFormat.java:275)
>> at
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRApp
>>Master.java:471)
>> ... 8 more
>>Caused by: java.lang.ClassNotFoundException:
>>org.apache.hcatalog.data.schema.HCatSchema
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>>
>>
>>
>>
>>
>>
>>
>>在 2015-02-28 09:37:29，"李运田" <cumt...@163.com> 写道：
>>
>>pigServer.registerQuery("tmp = load 'pig' using
>>org.apache.hcatalog.pig.HCatLoader();");
>>pigServer.registerQuery("tmp = foreach tmp generate id;");
>>when I execute this，I cant get any error
>>but,When I execute
>>pigServer.registerQuery("store tmp into 'hive' using
>>org.apache.hcatalog.pig.HCatStorer();");
>>I get error like this:::
>>org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
>>open iterator for alias tmp
>> at org.apache.pig.PigServer.openIterator(PigServer.java:880)
>> at org.gradle.PigHiveHCat.run(PigHiveHCat.java:68)
>> at org.gradle.PigHiveHCat.main(PigHiveHCat.java:28)
>>Caused by: java.io.IOException: Job terminated with anomalous status
>>FAILED
>> at org.apache.pig.PigServer.openIterator(PigServer.java:872)
>> ... 2 more
>>..........................................................................
>>.................................
>>pigServer.registerQuery("a = LOAD '/user/hadoop/pig.txt' ;");
>>pigServer.store("a", "/user/hadoop/pig1.txt");
>>pigServer.registerQuery("store a into '/user/hadoop/pig2.txt';");
>>........................................this is OK,so I think something
>>about Hcatalog is wrong...........................
>>
>>
>>At 2015-02-28 01:21:12, "Alan Gates" <alanfga...@gmail.com> wrote:
>>What error message are you getting?
>>
>>Alan.
>>
>>
>>李运田
>>February 26, 2015 at 18:58
>>I want to use hcatalog in eclipse to deal with tables in hive.
>>but I cant store table into hive::
>>pigServer.registerQuery("tmp = load 'pig' using
>>org.apache.hcatalog.pig.HCatLoader();");
>>pigServer.registerQuery("tmp = foreach tmp generate id;");
>>pigServer.registerQuery("store tmp into 'hive' using
>>org.apache.hcatalog.pig.HCatStorer();");
>>I can store into file::
>>pigServer.registerQuery("a = LOAD '/user/hadoop/pig.txt' ;");
>>pigServer.store("a", "/user/hadoop/pig1.txt");
>>pigServer.registerQuery("store a into '/user/hadoop/pig2.txt';");
>>perhaps ,the hcatalog jars are wrong?
>>
>>
>
