[
https://issues.apache.org/jira/browse/PIG-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555165#comment-14555165
]
Thejas M Nair commented on PIG-4556:
------------------------------------
I believe s3 is used in non-local modes as well.
Change looks good to me. (for my reference, since it took me few mins to figure
out, the real change is in order in which params are passed to
ConfigurationUtil.mergeConf).
> Local mode is broken in some case by PIG-4247
> ---------------------------------------------
>
> Key: PIG-4556
> URL: https://issues.apache.org/jira/browse/PIG-4556
> Project: Pig
> Issue Type: Bug
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.15.0
>
> Attachments: PIG-4556-1.patch, PIG-4556-2.patch
>
>
> HExecutionEngine.getS3Conf is wrong. It should only return s3 config.
> Currently it will return all the properties, including *-site.xml even in
> local mode. In one particular case, mapred-site.xml contains
> "mapreduce.application.framework.path", this will going to the local mode
> config, thus we see the exception:
> {code}
> Message: java.io.FileNotFoundException: File
> file:/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:111)
> at
> org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:460)
> at org.apache.hadoop.fs.FilterFs.resolvePath(FilterFs.java:157)
> at org.apache.hadoop.fs.FileContext$24.next(FileContext.java:2137)
> at org.apache.hadoop.fs.FileContext$24.next(FileContext.java:2133)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2133)
> at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:595)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:753)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
> at
> org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:194)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
