On 2 May 2016, at 19:24, Gourav Sengupta <gourav.sengu...@gmail.com<mailto:gourav.sengu...@gmail.com>> wrote:
Jorn, what aspects are you speaking about ? My response was absolutely pertinent to Jinan because he will not even face the problem if he used Scala. So it was along the lines of helping a person to learn fishing that giving him a fish. Sorry: it will. This is a compatibility issue with Hadoop s3a as implemented in hadoop-aws.jar and amazon s3 libraries, something you can replicate in ScalaContext.textFile() This is a classpath problem, not a language issue And by the way your blinkered and biased response missed the fact that SPARK WAS WRITTEN AND IS WRITTEN IN SCALA. And runs in the JVM, including much of the core ASF libraries, google guava, Hadoop core, Zookeeper, etc. Oh, and hadoop has some C native libraries which you should have for Unix performance, and will need for Windows. There are bits of bash and python around, and even groovy in some of the spark-assembly JARs, though that's an accident which has been corrected in recent versions. Languages are tools: the more you know, the more tools you have at your disposal. Now, if you really want to make the problem go away, a patch for https://issues.apache.org/jira/browse/HADOOP-13062 would be nice; I promise I will review it
