Hi,
I am trying to run a pig script from java, with multiple threads. I am getting
below error. Is PigRunner threadsafe. Can we run simultaneously.
Below is the code that use (taken from Oozie PigRunnerMain, ofcourse)..
protected int runPigJob(String[] args, String pigLog,
boolean resetSecurityManager, boolean retrieveStats)
throws Exception {
PigStats stats = PigRunner.run(args, null);
int code = stats.getReturnCode();
String jobIds = getHadoopJobIds(stats);
if (jobIds != null && !jobIds.isEmpty()) {
System.out.println("Hadoop Job IDs executed by Pig: " +
jobIds);
//TODO
}
return code;
}
Failed to parse: null
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at
com.vzw.mobile.common.pig.udf.CustomMultiPigRunner2.runPigJob(CustomMultiPigRunner2.java:268)
at
com.vzw.mobile.common.pig.udf.PigRunnable2.call(CustomMultiPigRunner2.java:417)
at
com.vzw.mobile.common.pig.udf.PigRunnable2.call(CustomMultiPigRunner2.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
at
org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2210)
at
org.apache.hive.hcatalog.common.HCatContext.setConf(HCatContext.java:66)
at
org.apache.hive.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:214)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
at
org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:886)
at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 18 more
