> Using ls to figure out permissions is a bad design anyway, so I would > not be surprised if this hardcode was reported as a bug.
Of course, I have no idea why it was implemented like this. I assume it was written at some point in time where Java didn't provide the needed APIS (?) Implementing the permission check without relying in ls at all is also a solution for the problem I have :) > LLoyd > > On 11 December 2015 at 09:19, Samuel <samuelri...@gmail.com> wrote: > > Hi, > > > > I am experiencing some crashes when using spark over local files (mainly for > > testing). Some operations fail with > > > > java.lang.RuntimeException: Error while running command to get file > > permissions : java.io.IOException: Cannot run program "/bin/ls": error=2, No > > such file or directory > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:206) > > at org.apache.hadoop.util.Shell.run(Shell.java:188) > > at > > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > at org.apache.hadoop.util.Shell.execCommand(Shell.java:467) > > at org.apache.hadoop.util.Shell.execCommand(Shell.java:450) > > at > > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593) > > at > > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51) > > at > > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514) > > at > > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489) > > at > > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292) > > > > etcetera... > > > > Which seems to be related to Shell.java in org.apache.hadoop-util, that uses > > ls -ld to figure out file permissions (that is in > > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead of just > > calling ls, Shell .java calls /bin/ls, which is usually available, but in > > certain circumstances might not. Regardless of the reasons not to have ls in > > /bin, hardcoding the directory bans users from using the standard mechanisms > > to decide which binaries to run in their systems (in this case, $PATH), so I > > wonder if there is a particular reason why that path has been hardcoded to > > an absolute path instead to something resolvable using$PATH. > > > > Or in other words, is this a bug or a feature? > > > > Best > > > > -- > > Samuel -- Samuel --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
