[
https://issues.apache.org/jira/browse/PIG-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907104#comment-15907104
]
Nandor Kollar edited comment on PIG-5179 at 3/13/17 10:01 AM:
--------------------------------------------------------------
[~kellyzly] added PIG-5179_3.patch, replaced long to int as you told. It seems
that org.apache.spark.TaskContext#taskAttemptId is deprecated since Spark
1.3.0, and is replaced by org.apache.spark.TaskContext#attemptNumber, so we
should use the latter, from the TaskContext.scala:
{code}
@deprecated("use attemptNumber", "1.3.0")
def attemptId(): Long
{code}
was (Author: nkollar):
[~kellyzly] uploaded PIG-5179_3.patch. It seems that
org.apache.spark.TaskContext#taskAttemptId is deprecated since Spark 1.3.0, and
is replaced by org.apache.spark.TaskContext#attemptNumber, so we should use the
latter, from the TaskContext.scala:
{code}
@deprecated("use attemptNumber", "1.3.0")
def attemptId(): Long
{code}
> BigData_Streaming_1 is failing
> ------------------------------
>
> Key: PIG-5179
> URL: https://issues.apache.org/jira/browse/PIG-5179
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Fix For: spark-branch
>
> Attachments: PIG-5179_1.patch, PIG-5179_2.patch, PIG-5179_3.patch
>
>
> BigData_Streaming_1 is failing, because Spark uses HadoopExecutableManager,
> and looks like spark task id is not a valid task attempt id.
> {code}
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2083:
> Error while trying to get next result in POStream.
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextHelper(POStream.java:309)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextTuple(POStream.java:171)
> ... 19 more
> Caused by: java.lang.IllegalArgumentException: TaskAttemptId string : 1 is
> not properly formed
> at
> org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:201)
> at
> org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.writeErrorToHDFS(HadoopExecutableManager.java:177)
> at
> org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:107)
> at
> org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:259)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextHelper(POStream.java:281)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
