Hello Samuel, Issue HADOOP-11935 tracks an improvement to re-implement this code path using either JNI to OS syscalls or the JDK 7 java.nio.file APIs (probably the latter).
https://issues.apache.org/jira/browse/HADOOP-11935 For right now, I don't see a viable workaround besides ensuring that the command is accessible at /bin/ls on your system. --Chris Nauroth On 12/11/15, 8:05 AM, "Namikaze Minato" <lloydsen...@gmail.com> wrote: >My question was, which spark command are you using, and since you >already did the analysis, which function of Shell.java is this spark >code using? > >Regards, >LLoyd > >On 11 December 2015 at 15:43, Samuel <samuelri...@gmail.com> wrote: >> I am not using hadoop-util directly, it is Spark code what uses it >> (i,e. not directly under my control). >> >> Regarding ls, for my particular use case it is fine if you use "ls" >> instead of "/bin/ls". >> >> However, I do agree that using ls to fetch file permissions is >> incorrect, so a better solution (in terms of code quality) would be >> not to use ls at all. >> >> >> On 11 December 2015 at 14:56, Namikaze Minato <lloydsen...@gmail.com> >>wrote: >>> So what you ultimately need is a piece of java code listing the rwx >>> permissions for user, group and others that is not using ls >>> internally, is that correct? >>> If "RawLocalFileSystem" is not HDFS, do you really need to use >>> hadoop-util for that? >>> Can you tell us more about your use case? >>> >>> Regards, >>> LLoyd >>> >>> >>> >>> >>> On 11 December 2015 at 13:05, Samuel <samuelri...@gmail.com> wrote: >>>>> Using ls to figure out permissions is a bad design anyway, so I would >>>>> not be surprised if this hardcode was reported as a bug. >>>> >>>> Of course, I have no idea why it was implemented like this. I assume >>>> it was written at some point in time where Java didn't provide the >>>> needed APIS (?) >>>> >>>> Implementing the permission check without relying in ls at all is also >>>> a solution for the problem I have :) >>>> >>>>> LLoyd >>>>> >>>>> On 11 December 2015 at 09:19, Samuel <samuelri...@gmail.com> wrote: >>>>> > Hi, >>>>> > >>>>> > I am experiencing some crashes when using spark over local files >>>>>(mainly for >>>>> > testing). Some operations fail with >>>>> > >>>>> > java.lang.RuntimeException: Error while running command to get file >>>>> > permissions : java.io.IOException: Cannot run program "/bin/ls": >>>>>error=2, No >>>>> > such file or directory >>>>> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) >>>>> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:206) >>>>> > at org.apache.hadoop.util.Shell.run(Shell.java:188) >>>>> > at >>>>> > >>>>>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:3 >>>>>81) >>>>> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:467) >>>>> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:450) >>>>> > at >>>>> > >>>>>org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem >>>>>.java:593) >>>>> > at >>>>> > >>>>>org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem. >>>>>java:51) >>>>> > at >>>>> > >>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermiss >>>>>ionInfo(RawLocalFileSystem.java:514) >>>>> > at >>>>> > >>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermissi >>>>>on(RawLocalFileSystem.java:489) >>>>> > at >>>>> > >>>>>org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$an >>>>>on$1$$anonfun$12.apply(newParquet.scala:292) >>>>> > >>>>> > etcetera... >>>>> > >>>>> > Which seems to be related to Shell.java in org.apache.hadoop-util, >>>>>that uses >>>>> > ls -ld to figure out file permissions (that is in >>>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that >>>>>instead of just >>>>> > calling ls, Shell .java calls /bin/ls, which is usually available, >>>>>but in >>>>> > certain circumstances might not. Regardless of the reasons not to >>>>>have ls in >>>>> > /bin, hardcoding the directory bans users from using the standard >>>>>mechanisms >>>>> > to decide which binaries to run in their systems (in this case, >>>>>$PATH), so I >>>>> > wonder if there is a particular reason why that path has been >>>>>hardcoded to >>>>> > an absolute path instead to something resolvable using$PATH. >>>>> > >>>>> > Or in other words, is this a bug or a feature? >>>>> > >>>>> > Best >>>>> > >>>>> > -- >>>>> > Samuel >>>> >>>> >>>> >>>> >>>> -- >>>> Samuel >> >> >> >> -- >> Samuel > >--------------------------------------------------------------------- >To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >For additional commands, e-mail: user-h...@hadoop.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
