> My question was, which spark command are you using, and since you > already did the analysis, which function of Shell.java is this spark > code using?
Sorry, I misunderstood you. It was something using RawLocalFileSystem to load parquet files. The problem seemed to go away after I upgraded to spark 1.5.2 (that was in spark 1.4.0). I don't have the code with me now, but if the information is useful in any way I can provide an example later. Otherwise, there seems to be a ticket already to reimplement that code path, so that is good enough for me. Best > > Regards, > LLoyd > > On 11 December 2015 at 15:43, Samuel <samuelri...@gmail.com> wrote: > > I am not using hadoop-util directly, it is Spark code what uses it > > (i,e. not directly under my control). > > > > Regarding ls, for my particular use case it is fine if you use "ls" > > instead of "/bin/ls". > > > > However, I do agree that using ls to fetch file permissions is > > incorrect, so a better solution (in terms of code quality) would be > > not to use ls at all. > > > > > > On 11 December 2015 at 14:56, Namikaze Minato <lloydsen...@gmail.com> wrote: > >> So what you ultimately need is a piece of java code listing the rwx > >> permissions for user, group and others that is not using ls > >> internally, is that correct? > >> If "RawLocalFileSystem" is not HDFS, do you really need to use > >> hadoop-util for that? > >> Can you tell us more about your use case? > >> > >> Regards, > >> LLoyd > >> > >> > >> > >> > >> On 11 December 2015 at 13:05, Samuel <samuelri...@gmail.com> wrote: > >>>> Using ls to figure out permissions is a bad design anyway, so I would > >>>> not be surprised if this hardcode was reported as a bug. > >>> > >>> Of course, I have no idea why it was implemented like this. I assume > >>> it was written at some point in time where Java didn't provide the > >>> needed APIS (?) > >>> > >>> Implementing the permission check without relying in ls at all is also > >>> a solution for the problem I have :) > >>> > >>>> LLoyd > >>>> > >>>> On 11 December 2015 at 09:19, Samuel <samuelri...@gmail.com> wrote: > >>>> > Hi, > >>>> > > >>>> > I am experiencing some crashes when using spark over local files > >>>> > (mainly for > >>>> > testing). Some operations fail with > >>>> > > >>>> > java.lang.RuntimeException: Error while running command to get file > >>>> > permissions : java.io.IOException: Cannot run program "/bin/ls": > >>>> > error=2, No > >>>> > such file or directory > >>>> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) > >>>> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:206) > >>>> > at org.apache.hadoop.util.Shell.run(Shell.java:188) > >>>> > at > >>>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > >>>> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:467) > >>>> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:450) > >>>> > at > >>>> > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593) > >>>> > at > >>>> > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51) > >>>> > at > >>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514) > >>>> > at > >>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489) > >>>> > at > >>>> > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292) > >>>> > > >>>> > etcetera... > >>>> > > >>>> > Which seems to be related to Shell.java in org.apache.hadoop-util, > >>>> > that uses > >>>> > ls -ld to figure out file permissions (that is in > >>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead > >>>> > of just > >>>> > calling ls, Shell .java calls /bin/ls, which is usually available, but > >>>> > in > >>>> > certain circumstances might not. Regardless of the reasons not to have > >>>> > ls in > >>>> > /bin, hardcoding the directory bans users from using the standard > >>>> > mechanisms > >>>> > to decide which binaries to run in their systems (in this case, > >>>> > $PATH), so I > >>>> > wonder if there is a particular reason why that path has been > >>>> > hardcoded to > >>>> > an absolute path instead to something resolvable using$PATH. > >>>> > > >>>> > Or in other words, is this a bug or a feature? > >>>> > > >>>> > Best > >>>> > > >>>> > -- > >>>> > Samuel > >>> > >>> > >>> > >>> > >>> -- > >>> Samuel > > > > > > > > -- > > Samuel -- Samuel --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
