[ https://issues.apache.org/jira/browse/PIG-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704725#comment-13704725 ]
Rohini Palaniswamy commented on PIG-3376: ----------------------------------------- Can you give the full stacktrace from the map job? > Jython UDF invocation failure when importing java class in Cluster mode > ----------------------------------------------------------------------- > > Key: PIG-3376 > URL: https://issues.apache.org/jira/browse/PIG-3376 > Project: Pig > Issue Type: Bug > Affects Versions: 0.11 > Environment: local, cluster > Reporter: Sergey > > Hi, I'm testing my scripts in local mode then I run them in production using > oozie. > Locally everything works fine. My pig version is 0.11 > When I run the same script in cluster mode, I do get exception on line where > jython udf is invoked. Here is my UDF, see it imports java class. This class > is IN runtime, I give 100%: > {code} > from ru.pig.geo import GSMCellCalculator > @outputSchema("centerLon:double, centerLat:double") > def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth): > print lon, lat, maxDist, angleWidth, azimuth > spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat, > float(maxDist), float(angleWidth), float(azimuth)) > return spatialCoord.longitude, spatialCoord.latitude > {code} > Here is a part of script: > {code} > register '$geoSpatialUdfs' using jython as udf; > /* > some code goes here.... > */ > gsmCellProj = FOREACH gsmCellFixed GENERATE > branchId, > cellId, lac, > lon, lat, > (int)azimuth, (int)midDist, > (int)maxDist, > cellType, (int)angWidth, gen, > startAng, > angWidthFixed, startAngFixed, > > FLATTEN(udf.calculateDropShapedCenter(lon, lat, midDist, angWidth, azimuth)); > {code} > Here is the log STDOUT: > {code} > -07-10 17:33:30,729 [JobControl] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths to process : 1 > 2013-07-10 17:33:30,729 [JobControl] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths to process : 1 > 2013-07-10 17:33:30,737 [JobControl] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths (combined) to process : 1 > 2013-07-10 17:33:30,737 [JobControl] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths (combined) to process : 1 > 2013-07-10 17:33:31,562 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - HadoopJobId: job_201307101220_0154 > 2013-07-10 17:33:31,562 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - HadoopJobId: job_201307101220_0154 > 2013-07-10 17:33:31,563 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Processing aliases > cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj > 2013-07-10 17:33:31,563 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Processing aliases > cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj > 2013-07-10 17:33:31,563 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - detailed locations: M: > gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] > C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R: > cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14] > 2013-07-10 17:33:31,563 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - detailed locations: M: > gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] > C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R: > cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14] > 2013-07-10 17:33:31,563 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - More information at: > http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154 > 2013-07-10 17:33:31,563 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - More information at: > http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154 > Heart beat > 2013-07-10 17:33:51,196 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 14% complete > 2013-07-10 17:33:51,196 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 14% complete > 2013-07-10 17:33:54,210 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 34% complete > 2013-07-10 17:33:54,210 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 34% complete > 2013-07-10 17:33:57,229 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 50% complete > 2013-07-10 17:33:57,229 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 50% complete > 2013-07-10 17:34:13,307 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > 2013-07-10 17:34:13,307 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > Heart beat > 2013-07-10 17:34:31,899 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > 2013-07-10 17:34:31,899 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > Heart beat > 2013-07-10 17:34:49,480 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > 2013-07-10 17:34:49,480 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > 2013-07-10 17:35:08,576 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > 2013-07-10 17:35:08,576 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 83% complete > Heart beat > 2013-07-10 17:35:26,170 [main] WARN > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > stop immediately on failure. > 2013-07-10 17:35:26,170 [main] WARN > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > stop immediately on failure. > 2013-07-10 17:35:26,170 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - job job_201307101220_0154 has failed! Stop running all dependent jobs > 2013-07-10 17:35:26,170 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - job job_201307101220_0154 has failed! > {code} > Here is the log with error: > {code} > org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error > executing function > at > org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:376) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:354) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackage > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira