[
https://issues.apache.org/jira/browse/PIG-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705130#comment-13705130
]
Rohini Palaniswamy commented on PIG-3376:
-----------------------------------------
The stacktrace you posted is not complete. Please post the full stack trace. If
it is not in your pig log, you can go to the JobTracker UI for the hadoop job
launched and get the stacktrace clicking on the map attempt.
> Jython UDF invocation failure when importing java class in Cluster mode
> -----------------------------------------------------------------------
>
> Key: PIG-3376
> URL: https://issues.apache.org/jira/browse/PIG-3376
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: local, cluster
> Reporter: Sergey
>
> Hi, I'm testing my scripts in local mode then I run them in production using
> oozie.
> Locally everything works fine. My pig version is 0.11
> When I run the same script in cluster mode, I do get exception on line where
> jython udf is invoked. Here is my UDF, see it imports java class. This class
> is IN runtime, I give 100%:
> {code}
> from ru.pig.geo import GSMCellCalculator
> @outputSchema("centerLon:double, centerLat:double")
> def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
> print lon, lat, maxDist, angleWidth, azimuth
> spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat,
> float(maxDist), float(angleWidth), float(azimuth))
> return spatialCoord.longitude, spatialCoord.latitude
> {code}
> Here is a part of script:
> {code}
> register '$geoSpatialUdfs' using jython as udf;
> /*
> some code goes here....
> */
> gsmCellProj = FOREACH gsmCellFixed GENERATE
> branchId,
> cellId, lac,
> lon, lat,
> (int)azimuth, (int)midDist,
> (int)maxDist,
> cellType, (int)angWidth, gen,
> startAng,
> angWidthFixed, startAngFixed,
>
> FLATTEN(udf.calculateDropShapedCenter(lon, lat, midDist, angWidth, azimuth));
> {code}
> Here is the log STDOUT:
> {code}
> -07-10 17:33:30,729 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2013-07-10 17:33:30,729 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2013-07-10 17:33:30,737 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2013-07-10 17:33:30,737 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2013-07-10 17:33:31,562 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201307101220_0154
> 2013-07-10 17:33:31,562 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201307101220_0154
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Processing aliases
> cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Processing aliases
> cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - detailed locations: M:
> gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
> C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
> cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - detailed locations: M:
> gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
> C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
> cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
> Heart beat
> 2013-07-10 17:33:51,196 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 14% complete
> 2013-07-10 17:33:51,196 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 14% complete
> 2013-07-10 17:33:54,210 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 34% complete
> 2013-07-10 17:33:54,210 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 34% complete
> 2013-07-10 17:33:57,229 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 50% complete
> 2013-07-10 17:33:57,229 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 50% complete
> 2013-07-10 17:34:13,307 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:34:13,307 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> Heart beat
> 2013-07-10 17:34:31,899 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:34:31,899 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> Heart beat
> 2013-07-10 17:34:49,480 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:34:49,480 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:35:08,576 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:35:08,576 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> Heart beat
> 2013-07-10 17:35:26,170 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
> stop immediately on failure.
> 2013-07-10 17:35:26,170 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
> stop immediately on failure.
> 2013-07-10 17:35:26,170 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job job_201307101220_0154 has failed! Stop running all dependent jobs
> 2013-07-10 17:35:26,170 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job job_201307101220_0154 has failed!
> {code}
> Here is the log with error:
> {code}
> org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
> executing function
> at
> org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:376)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:354)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackage
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
