[
https://issues.apache.org/jira/browse/PIG-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945708#comment-13945708
]
Aniket Mokashi commented on PIG-3825:
-------------------------------------
[~cheolsoo], I didn't like this approach myself a lot. But, it seems there is
no non-hacky way of solving this. I will give it another try to make it the way
you suggested.
We are using hadoop-1.0.4 and 2.0.5 with pig_13. Auto local mode works for both
the cases well. If jobs are not storing anything (when you are writing to s3
for example), you should check if your local job is getting required
configuration or not.
> Stats collection needs to be changed for hadoop2 (with auto local mode)
> -----------------------------------------------------------------------
>
> Key: PIG-3825
> URL: https://issues.apache.org/jira/browse/PIG-3825
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.13.0
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Fix For: 0.13.0
>
> Attachments: PIG-3825.patch
>
>
> With auto local mode, hadoop 2 getJob, getMapTaskReports,
> getReduceTaskReports api cause NumberFormatException with stack similar to
> following-
> (I have also noticed that getMapTaskReports causes memory leak and OOMs for
> long running jobs as hadoop 2 map task reports are quite bulky, but that is a
> separate issue)
> {noformat}
> Caused by: java.lang.NumberFormatException: For input string: "local154006779"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> at java.lang.Long.parseLong(Long.java:410)
> at java.lang.Long.parseLong(Long.java:468)
> at
> org.apache.hadoop.mapreduce.JobID.toClusterTimeStamp(JobID.java:172)
> at org.apache.hadoop.mapreduce.JobID.getAppId(JobID.java:167)
> at
> org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:79)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.<init>(ClientServiceDelegate.java:114)
> at org.apache.hadoop.mapred.ClientCache.getClient(ClientCache.java:68)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:550)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:182)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:586)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:584)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:584)
> at
> org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:638)
> at
> org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:632)
> at
> org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:318)
> at
> org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:308)
> at
> org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:240)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:363)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:374)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
