Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-12 Thread Namikaze Minato
Then it is enough :) Regards, LLoyd On 11 December 2015 at 23:27, Samuel wrote: >> My question was, which spark command are you using, and since you >> already did the analysis, which function of Shell.java is this spark >> code using? > > Sorry, I misunderstood you. It was something using RawLo

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Samuel
Thanks for the information. If you are already aware of the problem, that is enough for me :) Best On 11 December 2015 at 18:49, Chris Nauroth wrote: > Hello Samuel, > > Issue HADOOP-11935 tracks an improvement to re-implement this code path > using either JNI to OS syscalls or the JDK 7 java.ni

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Samuel
> My question was, which spark command are you using, and since you > already did the analysis, which function of Shell.java is this spark > code using? Sorry, I misunderstood you. It was something using RawLocalFileSystem to load parquet files. The problem seemed to go away after I upgraded to sp

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Chris Nauroth
Hello Samuel, Issue HADOOP-11935 tracks an improvement to re-implement this code path using either JNI to OS syscalls or the JDK 7 java.nio.file APIs (probably the latter). https://issues.apache.org/jira/browse/HADOOP-11935 For right now, I don't see a viable workaround besides ensuring that th

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Namikaze Minato
My question was, which spark command are you using, and since you already did the analysis, which function of Shell.java is this spark code using? Regards, LLoyd On 11 December 2015 at 15:43, Samuel wrote: > I am not using hadoop-util directly, it is Spark code what uses it > (i,e. not directly

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Samuel
I am not using hadoop-util directly, it is Spark code what uses it (i,e. not directly under my control). Regarding ls, for my particular use case it is fine if you use "ls" instead of "/bin/ls". However, I do agree that using ls to fetch file permissions is incorrect, so a better solution (in ter

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Namikaze Minato
So what you ultimately need is a piece of java code listing the rwx permissions for user, group and others that is not using ls internally, is that correct? If "RawLocalFileSystem" is not HDFS, do you really need to use hadoop-util for that? Can you tell us more about your use case? Regards, LLoyd

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Samuel
> Using ls to figure out permissions is a bad design anyway, so I would > not be surprised if this hardcode was reported as a bug. Of course, I have no idea why it was implemented like this. I assume it was written at some point in time where Java didn't provide the needed APIS (?) Implementing t

Re: Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Namikaze Minato
Using ls to figure out permissions is a bad design anyway, so I would not be surprised if this hardcode was reported as a bug. LLoyd On 11 December 2015 at 09:19, Samuel wrote: > Hi, > > I am experiencing some crashes when using spark over local files (mainly for > testing). Some operations fail

Dependency on absolute path /bin/ls in Shell.java

2015-12-11 Thread Samuel
Hi, I am experiencing some crashes when using spark over local files (mainly for testing). Some operations fail with java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "/bin/ls": error=2, No such file or directory at j