Sergey created PIG-3376:
---------------------------
Summary: Jython UDF invocation failure when importing java class
in Cluster mode
Key: PIG-3376
URL: https://issues.apache.org/jira/browse/PIG-3376
Project: Pig
Issue Type: Bug
Affects Versions: 0.11
Environment: local, cluster
Reporter: Sergey
Hi, I'm testing my scripts in local mode then I run them in production using
oozie.
Locally everything works fine. My pig version is 0.11
When I run the same script in cluster mode, I do get exception on line where
jython udf is invoked. Here is my UDF, see it imports java class. This class is
IN runtime, I give 100%:
{code}
from ru.pig.geo import GSMCellCalculator
@outputSchema("centerLon:double, centerLat:double")
def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
print lon, lat, maxDist, angleWidth, azimuth
spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat,
float(maxDist), float(angleWidth), float(azimuth))
return spatialCoord.longitude, spatialCoord.latitude
{code}
Here is a part of script:
{code}
register '$geoSpatialUdfs' using jython as udf;
/*
some code goes here....
*/
gsmCellProj = FOREACH gsmCellFixed GENERATE
branchId,
cellId, lac,
lon, lat,
(int)azimuth, (int)midDist,
(int)maxDist,
cellType, (int)angWidth, gen, startAng,
angWidthFixed, startAngFixed,
FLATTEN(udf.calculateDropShapedCenter(lon, lat, midDist, angWidth, azimuth));
{code}
Here is the log STDOUT:
{code}
-07-10 17:33:30,729 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-07-10 17:33:30,729 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-07-10 17:33:30,737 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2013-07-10 17:33:30,737 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2013-07-10 17:33:31,562 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201307101220_0154
2013-07-10 17:33:31,562 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201307101220_0154
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases
cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases
cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M:
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M:
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
Heart beat
2013-07-10 17:33:51,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 14% complete
2013-07-10 17:33:51,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 14% complete
2013-07-10 17:33:54,210 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 34% complete
2013-07-10 17:33:54,210 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 34% complete
2013-07-10 17:33:57,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2013-07-10 17:33:57,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2013-07-10 17:34:13,307 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
2013-07-10 17:34:13,307 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
Heart beat
2013-07-10 17:34:31,899 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
2013-07-10 17:34:31,899 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
Heart beat
2013-07-10 17:34:49,480 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
2013-07-10 17:34:49,480 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
2013-07-10 17:35:08,576 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
2013-07-10 17:35:08,576 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 83% complete
Heart beat
2013-07-10 17:35:26,170 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop
immediately on failure.
2013-07-10 17:35:26,170 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop
immediately on failure.
2013-07-10 17:35:26,170 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201307101220_0154 has failed! Stop running all dependent jobs
2013-07-10 17:35:26,170 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201307101220_0154 has failed!
{code}
Here is the log with error:
{code}
org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error executing
function
at
org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:376)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:354)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackage
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
