[
https://issues.apache.org/jira/browse/PIG-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904718#comment-15904718
]
Nandor Kollar commented on PIG-5179:
------------------------------------
[~kellyzly] how about 5179_2.patch? Since only taskId is problematic, I created
a subclass SparkExecutableManager inheriting from HadoopExecutableManager, made
writeErrorToHDFS protected, and I used
{code}TaskContext.get().attemptNumber(){code} instead of
{code}TaskAttemptID.forName(taskId).getTaskID().getId(){code}
> BigData_Streaming_1 is failing
> ------------------------------
>
> Key: PIG-5179
> URL: https://issues.apache.org/jira/browse/PIG-5179
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Fix For: spark-branch
>
> Attachments: PIG-5179_1.patch, PIG-5179_2.patch
>
>
> BigData_Streaming_1 is failing, because Spark uses HadoopExecutableManager,
> and looks like spark task id is not a valid task attempt id.
> {code}
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2083:
> Error while trying to get next result in POStream.
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextHelper(POStream.java:309)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextTuple(POStream.java:171)
> ... 19 more
> Caused by: java.lang.IllegalArgumentException: TaskAttemptId string : 1 is
> not properly formed
> at
> org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:201)
> at
> org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.writeErrorToHDFS(HadoopExecutableManager.java:177)
> at
> org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:107)
> at
> org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:259)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextHelper(POStream.java:281)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
