[ https://issues.apache.org/jira/browse/HADOOP-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-10969. ------------------------------------- Resolution: Invalid > RawLocalFileSystem.setPermission throws Exception on windows > ------------------------------------------------------------ > > Key: HADOOP-10969 > URL: https://issues.apache.org/jira/browse/HADOOP-10969 > Project: Hadoop Common > Issue Type: Bug > Environment: hadoop 2.3.0, Windows Environment, Development using > Eclipse, Lenevo Laptop > Reporter: Venkatesh > Priority: Blocker > > I'm an application developer. We recently moved from CDH4.7 to CDH5.1. The > hadoop version have been from 1.x to 2.x. In order to perform development on > Eclipse (on WINDOWS), the following class was created > public class WindowsLocalFileSystem extends LocalFileSystem { > public WindowsLocalFileSystem() { > super(); > } > @Override > public boolean mkdirs(Path f, FsPermission permission) throws > IOException { > final boolean result = super.mkdirs(f); > this.setPermission(f, permission); > return result; > > } > @Override > public void setPermission(Path p, FsPermission permission) > throws IOException { > try { > super.setPermission(p, permission); > } catch (final IOException e) { > System.err.println("Cant help it, hence ignoring > IOException setting persmission for path \"" + p + > "\": " + e.getMessage()); > } > } > } > This class was used in MapReduce Job as > if (RUN_LOCAL) { > conf.set("fs.default.name", "file:///"); > conf.set("mapred.job.tracker", "local"); > conf.set("fs.file.impl", > > "org.scif.bdp.mrjobs.WindowsLocalFileSystem"); > conf.set( > "io.serializations", > > "org.apache.hadoop.io.serializer.JavaSerialization," > + > "org.apache.hadoop.io.serializer.WritableSerialization"); > } > It worked fine on CDH4.7. Now the same code when compiled on CDH5.1 works but > when I try to execute it throws the following stacktrace > Exception in thread "main" java.lang.NullPointerException > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010) > at org.apache.hadoop.util.Shell.runCommand(Shell.java:451) > at org.apache.hadoop.util.Shell.run(Shell.java:424) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656) > at org.apache.hadoop.util.Shell.execCommand(Shell.java:745) > at org.apache.hadoop.util.Shell.execCommand(Shell.java:728) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633) > at > org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:467) > at > com.scif.bdp.common.WindowsLocalFileSystem.setPermission(WindowsLocalFileSystem.java:26) > at > com.scif.bdp.common.WindowsLocalFileSystem.mkdirs(WindowsLocalFileSystem.java:17) > at > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313) > at com.scif.bdp.mrjobs.DeDup.run(DeDup.java:55) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at com.scif.bdp.mrjobs.DeDup.main(DeDup.java:59) > (Note DeDup is my MR class to remove duplicates) > Upon investigation the only change I saw was the change in method > .setPermission(). It invokes Native.POSIX.chmod as against Native.chmod -- This message was sent by Atlassian JIRA (v6.2#6252)