[
https://issues.apache.org/jira/browse/PIG-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944578#comment-13944578
]
Cheolsoo Park commented on PIG-3825:
------------------------------------
[~aniket486], I am not sure it's a good idea to always ignore exceptions. Can't
we explicitly use "local<number>" in the JobID to identify the local mode and
handle it differently? I am worried that with your approach, not only
NumberFormatException but also any other exceptions are swallowed. That might
cover up some real problems.
Can you also tell me which Hadoop version you're using? I tried EMR 2.2, but
auto local mode doesn't seem to work at all. Basically, jobs never store
anything. I am going to test it with EMR 2.3 as soon as it's released.
> Stats collection needs to be changed for hadoop2 (with auto local mode)
> -----------------------------------------------------------------------
>
> Key: PIG-3825
> URL: https://issues.apache.org/jira/browse/PIG-3825
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.13.0
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Fix For: 0.13.0
>
> Attachments: PIG-3825.patch
>
>
> With auto local mode, hadoop 2 getJob, getMapTaskReports,
> getReduceTaskReports api cause NumberFormatException with stack similar to
> following-
> (I have also noticed that getMapTaskReports causes memory leak and OOMs for
> long running jobs as hadoop 2 map task reports are quite bulky, but that is a
> separate issue)
> {noformat}
> Caused by: java.lang.NumberFormatException: For input string: "local154006779"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> at java.lang.Long.parseLong(Long.java:410)
> at java.lang.Long.parseLong(Long.java:468)
> at
> org.apache.hadoop.mapreduce.JobID.toClusterTimeStamp(JobID.java:172)
> at org.apache.hadoop.mapreduce.JobID.getAppId(JobID.java:167)
> at
> org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:79)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.<init>(ClientServiceDelegate.java:114)
> at org.apache.hadoop.mapred.ClientCache.getClient(ClientCache.java:68)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:550)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:182)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:586)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:584)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:584)
> at
> org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:638)
> at
> org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:632)
> at
> org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:318)
> at
> org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:308)
> at
> org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:240)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:363)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:374)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
